at least 2 live replicas required across different availability zones, could only find 1

[mdvaliyullah]

Running mimir in monolithic mode across 2 Data centres with replication factor 2, unable to push metrics to mimir getting the below error on mimir-proxy as shown below. Please advise

at least 2 live replicas required across different availability zones, could only find 1

target: all,overrides-exporter

server:
  log_level: info

distributor:
  pool:
    health_check_ingesters: true
  ring:
    heartbeat_period: 2s
    heartbeat_timeout: 10s  
    kvstore:
      store: memberlist

querier:
  lookback_delta: 1m
  query_store_after: 0s
  query_ingesters_within: 0s
  shuffle_sharding_ingesters_enabled: true

ingester_client:
  grpc_client_config:
    # Configure the client to allow messages up to 100MB.
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600
    grpc_compression: gzip

ingester:  
  ring:
    final_sleep: 0s
    num_tokens: 512
    tokens_file_path: /data/tokens
    heartbeat_period: 2s
    heartbeat_timeout: 10s
    unregister_on_shutdown: true    
    kvstore:
      store: memberlist
    replication_factor: 2
    instance_availability_zone: "region-details"
    zone_awareness_enabled: true

limits:
  max_global_series_per_user: 0
  out_of_order_time_window: 30m
  compactor_block_upload_enabled: true
  
blocks_storage:
  storage_prefix: blocks
  bucket_store:
    sync_dir: /data/tmp/mimir/tsdb-sync
    sync_interval: 5m
    ignore_blocks_within: 0s
    metadata_cache:
      bucket_index_content_ttl: 1m
      tenants_list_ttl: 1m
      tenant_blocks_list_ttl: 1m
      metafile_doesnt_exist_ttl: 1m    
  tsdb:
    dir: /data/ingester
    flush_blocks_on_shutdown: true
    ship_interval: 1m

compactor:
  data_dir: /data/compactor
  sharding_ring:
    heartbeat_period: 2s
    heartbeat_timeout: 10s  
    kvstore:
      store: memberlist
  cleanup_interval: 5m   

store_gateway:
  sharding_ring:
    replication_factor: 2
    tokens_file_path: /data/tokens
    heartbeat_period: 2s
    heartbeat_timeout: 10s
    instance_availability_zone: "region-details"   
    zone_awareness_enabled: true 
    kvstore:
      store: memberlist     

ruler:
  rule_path: /data/ruler
  poll_interval: 2s  
  #alertmanager_url: http://127.0.0.1:8080/alertmanager
  ring:
    heartbeat_period: 2s
    heartbeat_timeout: 10s
    kvstore:
      store: memberlist

ruler_storage:
  storage_prefix: "ruler"

memberlist:
  join_members : [ host_members_details] 

[pracucci]

👋 Hi!

with replication factor 2

Replication factor 2 is not officially supported by Mimir and there are some known quirks. You can learn more here:
#3844 (comment)

at least 2 live replicas required across different availability zones, could only find 1

That being said, I don't think it's the reason of the issue you're experiencing. The happy path for replication factor = 2 still works.

Based on the information you provided, I've the feeling the the ingesters in the 2 datacenters are not joining the same ring. If you open the ingester ring page, do you see all ingesteres registered to the same ring? If not there's a misconfiguration in memberlist and ingesters in the 2 datacenters can't directly talk each other.

[hendrikkiedrowski]

We also have sort of the same Issue here. We have 4 Zones and a replication Factor of 4 and if two zones fail (which is what we wanted to archive) the cluster breaks. We have deployed that via Helm

[dimitarvdimitrov]

mimir needs quorum to progress with reads and writes. Quorum is understood as half + 1. 2/4 is unfortunately not quorum. 3/4 would be. But that's the same tolerance as having RF=3. Consider changing the replication factor to 3 or 5.

[hendrikkiedrowski]

So if i understand this correctly i could create 8 Logical zones with a RF of 7 and could loose 4 of the zones? Because if my math would be right here it need to have 4/7 for quorum. Do i also understand correctly that the number of zones dont have to be equal to the RF?

[dimitarvdimitrov]

So if i understand this correctly i could create 8 Logical zones with a RF of 7 and could loose 4 of the zones? Because if my math would be right here it need to have 4/7 for quorum

that's right

Do i also understand correctly that the number of zones dont have to be equal to the RF?

The number of zones must be greater than or equal to the replication factor. This applies to store-gateways as well as ingesters.

[hendrikkiedrowski]

Thank you so much for clarification! @dimitarvdimitrov