Support custom endpoint for S3 Chunk Store

[mizeng]

Is your feature request related to a problem? Please describe.
I use Ceph Object Store which has S3 compatible API. However it uses Path style S3 URL like (http://s3endpoint/BUCKET), and it can not support by current Loki.

Loki put the bucket prefixed to the endpoint (Virtual-hosted Style)as the final endpoint (http://BUCKET.s3endpoint), which can not be used here. And I always get error like below:

RequestError: send request failed\ncaused by: Put http://lokitest.<ceph object store>/fake/6c773d64ab8ab7c2%3A16b870e34df%3A16b870e34df%3A5feb2045: dial tcp: lookup lokitest.<object store> on xxxxx:53: no such host

For the details of the 2 styles, reference can be found here

Describe the solution you'd like
Provide a new config to let user choose use Path style or Virtual-hosted Style.
Thanks to the answer in aws/aws-sdk-go#1585.

[mizeng]

Filed PR in Cortex: cortexproject/cortex#1476

[mizeng]

PR in Cortex merged, need to port back to Loki.

[mizeng]

port back to Loki PR: #705

[ahrkrak]

@mizeng would love to know if you've got Loki now working with Ceph as we're about to try doing the same thing...

[mizeng]

@ahrkrak Ceph Object Store? Yes, it is working.

[ThoreKr]

@mizeng do you run the rgw on http or https?

[ningyougang]

My loki configuration of support ceph as below, worked well

...
  schema_config:
    configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
  server:
    http_listen_port: 3100
  storage_config:
    boltdb_shipper:
      active_index_directory: /data/loki/boltdb-shipper-active
      cache_location: /data/loki/boltdb-shipper-cache
      cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
      shared_store: s3
    aws:
      bucketnames: test1
      endpoint: 10.106.153.207:6780
      region: default
      access_key_id: test1
      secret_access_key: test1
      insecure: true
      sse_encryption: false
      http_config:
        idle_conn_timeout: 90s
        response_header_timeout: 0s
        insecure_skip_verify: false
      s3forcepathstyle: true
...

[vadlakiran]

@mizeng @ningyougang can you please help me to access the ceph object storage with loki