Set default node.max_local_storage_nodes to 1

[jordansissel]

A number of users over the years have been punished by unintentionally running two ES nodes simultaneously on the same server and then losing data because of mixed and lost data between nodes/0 and nodes/1 directories.

One such example: https://groups.google.com/d/msg/logstash-users/1JKLLa4YUw4/g0vcCcxmeiMJ

This occurs most frequently by accident, I think, under some weird situation where someone is running Elasticsearch, intends to restart it, and either has ES still running when the new node comes up or has some race condition during restart that causes 2 nodes to exist temporarily so the nodes/0 directory is locked.

It would be very helpful for new users and operators to have node.max_local_storage_nodes default to 1.

[untergeek]

👍 I think it should be an explicit setting if you want more than one instance to share the same tree.

[dakrone]

@jordansissel this would have the unfortunate downside of not allowing anyone to start up multiple ES nodes on their computer for development or testing, without changing the setting :(

[untergeek]

@dakrone I understand the frustration, but this setting is something that would be easy to leave commented out with the default of 1 in the elasticsearch.yml file. The absence of this setting (defaulting to 1) bites a lot of people using Elasticsearch in production, and many more using Logstash.

I feel that this setting should favor production environments first, and developers second. Developers should not have a problem explicitly enabling this setting in order to run multiple ES nodes for development or testing.

[kimchy]

why does this setting really is more prominent in the case of logstash users compared to others?

we talked in several issues that it might make sense to have a prod flag when starting a process, with potentially different defaults to settings that apply to that use case.

[untergeek]

I'm not sure how frequently non-time-series users of Elasticsearch run multiple nodes per box, but it is not uncommon with deployments involving time-series indices.

I would love a "prod" flag, with node.max_local_storage_nodes: 1 (among other things) as the default behavior. That would satisfy me completely with regards to this ticket. We could easily set that as the default behavior in Logstash (and we would).

[jordansissel]

@kimchy I'm not sure how each user arrives at the problem (multiple simultaneous data nodes on the same server). Quickly restarting Elasticsearch, perhaps repeatedly? or accidentally running it from multiple terminals? I dunno. It's usually newbies who hit this, and I suspect that it happens several hours into a new-user "WTF is going on!?" -kind of situation where the user is grasping at straws trying to make the computer do things, so it may be very difficult to learn what actually caused it :\

[vjanelle]

@jordansissel Running two copies of logstash with the internal ES perhaps?

[damm]

This is set by the Chef cookbook by default. https://github.com/elastic/cookbook-elasticsearch/blob/master/libraries/resource_configure.rb#L102

Thank you for whomever did it. It really should be set by default.

[pickypg]

Perhaps instead of making it to 1, it should be limited to a more sane default, like 5 rather than 50 (the current). 5 would be inline with our current default port range for convenient development.

We should consider setting this when we install via the RPM/DEB installers.

[dakrone]

Resolved in 5.0 by #19964

[Ankiewicz]

If you find you have two, how do you kill them?

`ls -al /usr/local/var/elasticsearch/elasticsearch_Jonathan
total 0
drwxr-xr-x 2 Jonathan admin 68 Jan 20 23:32 .
drwxr-xr-x 4 Jonathan admin 136 Jan 20 23:32 ..

`