cloud_storage: simplify topic tiered storage enablement into one on/off toggle (remove legacy "archival" mode)

[jcsp]

Archival pre-dates the general tiered storage feature in Redpanda: it is the write path only, writing a stream of log segments into S3.

Currently there are independent read + write properties for S3 access on each topic.

remote.read=true,remote.write=true is an easy state to explain for a topic: it means we'll use tiered storage, spilling some data into S3 and sending reads there when necessary. Both properties false also clearly makes sense. Anything else is hard to explain and reason about, including the "archival" mode where remote.write is true but remote.read is not.

We should collapse the overall property for controlling use of tiered storage into a single true/false, and not have this individual toggling of the read & write paths.

• If someone really wants to write an unbounded history of a partition into S3, they can still do that by enabling tiered storage and setting very high limits for retention.bytes/retention.ms: this isn't a separate feature/mode, it's just an odd use of the general tiered storage feature.
• If someone really wants to write data to S3 but not read it from there (i.e. always keep on-disk copies too) then they can set local retention properties that keep data in local storage forever.

If we wanted to explicitly support the use case of writing segment to S3 and never deleting them, but not considering those segments in S3 to be part of the readable partition range (e.g. for someone that wants an unbounded-size audit trail of a partition), we could add that back in as a first class feature that would basically disable deleting segments when they fall out of the retention period.

For administrative/support situations where we might want to e.g. disable writes temporarily, we need more structured & tested paths for doing so:

• cloud_storage: robust handling of enabling/disable tiered storage #6627
• cloud_storage: maintenance admin API endpoints #6628

JIRA Link: CORE-1036

[mmedenjak]

Related - redpanda-data/docs#596