Infinite loop caused in transport gem when `reload_on_failure` is true and more than 1 host is provided.

[estolfo]

The perform_request method will get itself into an infinite loop when reload_on_failure is set to true and more than 1 host is provided.

The issue is in the rescue block:

if @options[:reload_on_failure] and tries < connections.all.size
  logger.warn "[#{e.class}] Reloading connections (attempt #{tries} of #{connections.all.size})" if logger
  reload_connections! and retry
end

On the first try, tries is 1 and connections.all.size is greater than 1 so the if condition is met. reload_connections! will then call sniffer.hosts that then also calls the perform_request method. tries is reset at 1 when this method is called so this rescue block is reached and and the cycle continues, infinitely.

The key to getting the SystemStackError: stack level too deep error and not a timeout error is to set the sniffer_timeout high enough so that the timeout doesn't interrupt the infinite loop before the SystemStackError: stack level too deep is thrown. This is why the error was originally not able to be reproduced.

I was able to reproduce the error with the following client settings:

      {
        hosts: ['http://unavabilable:9200', 'http://unavabilable:9201'],
        reload_on_failure: true,
        sniffer_timeout: 50
      }

> client.info  
# SystemStackError: stack level too deep

Please see this issue for further history: #177

[estolfo]

@eherot as a workaround for now, try adjusting sniffer_timeout so that you get a timeout error before a SystemStackError.

[eherot]

Thanks for the workaround! I think I might have a better solution for our particular setup...

It seems that if the client is connected to Elasticsearch through an ELB, connections.all might be (eventually, see below) taking on the host list as reported back from the ES /_nodes/http endpoint.

What makes this extra fun/annoying is that it doesn't happen until DEFAULT_RELOAD_AFTER is exceeded, which meant, in our case, that it would consistently run for about 45 minutes before landing itself in this infinite loop and then crashing (not from SystemStackError but by running out of memory).

As a fix I'm going to turn off reload_connections since it's not getting back the right host list anyway (and it also probably isn't needed because ELB is doing that job).

[estolfo]

@eherot That makes sense as a resolution for your setup. Is the DEFAULT_RELOAD_AFTER setting in the fluent-plugin-elasticsearch gem?

[eherot]

Yes, thankfully both Reload Connections (the boolean) and Reload After are configurable in the plugin. Amusingly the documentation for reload_connections specifically says, "This can be an issue if your Elasticsearch cluster is behind a Reverse Proxy, as Fluentd process may not have direct network access to the Elasticsearch nodes." Too bad I didn't notice this sooner!

[estolfo]

Thanks for clarifying. I hope others will see this thread and benefit from the info!

[estolfo]

Resolved in master with this commit: #696