Adding a GeoIP2 database in $ES_CONFIG/ingest-geoip results in node not starting if using default filename

[jakommo]

Elasticsearch version (bin/elasticsearch --version): Version: 7.9.3, Build: default/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z, JVM: 15

Plugins installed: []

JVM version (java -version): Bundled

OS version (uname -a if on a Unix-like system): Linux es-jre2 5.9.1-arch1-1 #1 SMP PREEMPT Sat, 17 Oct 2020 13:30:37 +0000 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

When adding a GeoIP DB file with the same name as the ones that are shipped by default, startup fails with:

[2020-10-27T11:45:52,157][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-jre2] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-City.mmdb] to not exist in [/home/jakob/ELASTIC/elasticsearch-7.9.3/config/ingest-geoip]

According to the docs, the custom GeoIP DB should be placed in $ES_CONFIG/ingest-geoip, but there is no mention that they can not have the same name as the default files (which are also the default names if one download updated files from maxmind).

I think this is related to the breaking change discussed in #39495.

Not sure what the correct approach to this is. Either we should update the docs to reflect that it can not have the same file name or we should disable the check.

Steps to reproduce:

jakob@es-jre2  ~/ELASTIC/elasticsearch-7.9.3  mkdir config/ingest-geoip                 

 jakob@es-jre2  ~/ELASTIC/elasticsearch-7.9.3  cp ./modules/ingest-geoip/GeoLite2-City.mmdb config/ingest-geoip

 jakob@es-jre2  ~/ELASTIC/elasticsearch-7.9.3  ES
[2020-10-27T11:45:48,403][INFO ][o.e.n.Node               ] [es-jre2] version[7.9.3], pid[314972], build[default/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.9.1-arch1-1/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/15/15+36-1562]
[2020-10-27T11:45:48,406][INFO ][o.e.n.Node               ] [es-jre2] JVM home [/home/jakob/ELASTIC/elasticsearch-7.9.3/jdk]
[2020-10-27T11:45:48,407][INFO ][o.e.n.Node               ] [es-jre2] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-17741669071348173933, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/home/jakob/ELASTIC/elasticsearch-7.9.3, -Des.path.conf=/home/jakob/ELASTIC/elasticsearch-7.9.3/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2020-10-27T11:45:50,161][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [aggs-matrix-stats]
[2020-10-27T11:45:50,161][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [analysis-common]
[2020-10-27T11:45:50,162][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [constant-keyword]
[2020-10-27T11:45:50,162][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [flattened]
[2020-10-27T11:45:50,162][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [frozen-indices]
[2020-10-27T11:45:50,163][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [ingest-common]
[2020-10-27T11:45:50,163][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [ingest-geoip]
[2020-10-27T11:45:50,163][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [ingest-user-agent]
[2020-10-27T11:45:50,163][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [kibana]
[2020-10-27T11:45:50,164][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [lang-expression]
[2020-10-27T11:45:50,164][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [lang-mustache]
[2020-10-27T11:45:50,164][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [lang-painless]
[2020-10-27T11:45:50,165][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [mapper-extras]
[2020-10-27T11:45:50,165][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [parent-join]
[2020-10-27T11:45:50,165][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [percolator]
[2020-10-27T11:45:50,165][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [rank-eval]
[2020-10-27T11:45:50,166][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [reindex]
[2020-10-27T11:45:50,166][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [repository-url]
[2020-10-27T11:45:50,166][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [search-business-rules]
[2020-10-27T11:45:50,167][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [searchable-snapshots]
[2020-10-27T11:45:50,167][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [spatial]
[2020-10-27T11:45:50,167][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [tasks]
[2020-10-27T11:45:50,167][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [transform]
[2020-10-27T11:45:50,168][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [transport-netty4]
[2020-10-27T11:45:50,168][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [vectors]
[2020-10-27T11:45:50,168][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [wildcard]
[2020-10-27T11:45:50,169][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-analytics]
[2020-10-27T11:45:50,169][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-async]
[2020-10-27T11:45:50,169][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-async-search]
[2020-10-27T11:45:50,170][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-autoscaling]
[2020-10-27T11:45:50,170][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-ccr]
[2020-10-27T11:45:50,170][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-core]
[2020-10-27T11:45:50,170][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-data-streams]
[2020-10-27T11:45:50,171][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-deprecation]
[2020-10-27T11:45:50,171][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-enrich]
[2020-10-27T11:45:50,171][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-eql]
[2020-10-27T11:45:50,171][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-graph]
[2020-10-27T11:45:50,171][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-identity-provider]
[2020-10-27T11:45:50,172][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-ilm]
[2020-10-27T11:45:50,172][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-logstash]
[2020-10-27T11:45:50,172][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-ml]
[2020-10-27T11:45:50,172][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-monitoring]
[2020-10-27T11:45:50,172][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-ql]
[2020-10-27T11:45:50,173][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-rollup]
[2020-10-27T11:45:50,173][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-security]
[2020-10-27T11:45:50,173][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-sql]
[2020-10-27T11:45:50,173][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-stack]
[2020-10-27T11:45:50,173][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-voting-only-node]
[2020-10-27T11:45:50,174][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-watcher]
[2020-10-27T11:45:50,174][INFO ][o.e.p.PluginsService     ] [es-jre2] no plugins loaded
[2020-10-27T11:45:50,210][INFO ][o.e.e.NodeEnvironment    ] [es-jre2] using [1] data paths, mounts [[/home (/dev/mapper/vg0-home)]], net usable_space [256.8gb], net total_space [829.3gb], types [ext4]
[2020-10-27T11:45:50,211][INFO ][o.e.e.NodeEnvironment    ] [es-jre2] heap size [1gb], compressed ordinary object pointers [true]
[2020-10-27T11:45:50,238][INFO ][o.e.n.Node               ] [es-jre2] node name [es-jre2], node ID [JvPMhAMTS3Wlh_RMaOGRvQ], cluster name [elasticsearch]
uncaught exception in thread [main]
[2020-10-27T11:45:52,157][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-jre2] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-City.mmdb] to not exist in [/home/jakob/ELASTIC/elasticsearch-7.9.3/config/ingest-geoip]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.3.jar:7.9.3]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.3.jar:7.9.3]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.3.jar:7.9.3]
Caused by: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-City.mmdb] to not exist in [/home/jakob/ELASTIC/elasticsearch-7.9.3/config/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:78) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:127) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:109) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.node.Node.<init>(Node.java:400) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.3.jar:7.9.3]
java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-City.mmdb] to not exist in [/home/jakob/ELASTIC/elasticsearch-7.9.3/config/ingest-geoip]
Likely root cause: java.io.IOException: expected database [GeoLite2-City.mmdb] to not exist in [/home/jakob/ELASTIC/elasticsearch-7.9.3/config/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.assertDatabaseExistence(IngestGeoIpPlugin.java:150)
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.loadDatabaseReaders(IngestGeoIpPlugin.java:102)
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:76)
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:127)
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:109)
	at org.elasticsearch.node.Node.<init>(Node.java:400)
	at org.elasticsearch.node.Node.<init>(Node.java:277)
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
For complete error details, refer to the log at /home/jakob/ELASTIC/elasticsearch-7.9.3/logs/elasticsearch.log
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.3.jar:7.9.3]
	... 6 more
Caused by: java.io.IOException: expected database [GeoLite2-City.mmdb] to not exist in [/home/jakob/ELASTIC/elasticsearch-7.9.3/config/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.assertDatabaseExistence(IngestGeoIpPlugin.java:150) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.loadDatabaseReaders(IngestGeoIpPlugin.java:102) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:76) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:127) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:109) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.node.Node.<init>(Node.java:400) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.3.jar:7.9.3]
	... 6 more

 jakob@es-jre2  ~/ELASTIC/elasticsearch-7.9.3  mv config/ingest-geoip/GeoLite2-City.mmdb config/ingest-geoip/MyGeoLite2-City.mmdb

 jakob@es-jre2  ~/ELASTIC/elasticsearch-7.9.3  ES
[2020-10-27T11:46:13,637][INFO ][o.e.n.Node               ] [es-jre2] version[7.9.3], pid[315295], build[default/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.9.1-arch1-1/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/15/15+36-1562]
[2020-10-27T11:46:13,641][INFO ][o.e.n.Node               ] [es-jre2] JVM home [/home/jakob/ELASTIC/elasticsearch-7.9.3/jdk]
[2020-10-27T11:46:13,641][INFO ][o.e.n.Node               ] [es-jre2] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-4118470711344022869, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/home/jakob/ELASTIC/elasticsearch-7.9.3, -Des.path.conf=/home/jakob/ELASTIC/elasticsearch-7.9.3/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2020-10-27T11:46:15,324][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [aggs-matrix-stats]
[2020-10-27T11:46:15,324][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [analysis-common]
[2020-10-27T11:46:15,324][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [constant-keyword]
[2020-10-27T11:46:15,325][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [flattened]
[2020-10-27T11:46:15,325][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [frozen-indices]
[2020-10-27T11:46:15,326][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [ingest-common]
[2020-10-27T11:46:15,326][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [ingest-geoip]
[2020-10-27T11:46:15,326][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [ingest-user-agent]
[2020-10-27T11:46:15,327][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [kibana]
[2020-10-27T11:46:15,327][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [lang-expression]
[2020-10-27T11:46:15,327][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [lang-mustache]
[2020-10-27T11:46:15,327][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [lang-painless]
[2020-10-27T11:46:15,328][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [mapper-extras]
[2020-10-27T11:46:15,328][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [parent-join]
[2020-10-27T11:46:15,328][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [percolator]
[2020-10-27T11:46:15,329][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [rank-eval]
[2020-10-27T11:46:15,329][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [reindex]
[2020-10-27T11:46:15,329][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [repository-url]
[2020-10-27T11:46:15,330][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [search-business-rules]
[2020-10-27T11:46:15,330][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [searchable-snapshots]
[2020-10-27T11:46:15,330][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [spatial]
[2020-10-27T11:46:15,330][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [tasks]
[2020-10-27T11:46:15,330][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [transform]
[2020-10-27T11:46:15,331][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [transport-netty4]
[2020-10-27T11:46:15,331][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [vectors]
[2020-10-27T11:46:15,331][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [wildcard]
[2020-10-27T11:46:15,331][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-analytics]
[2020-10-27T11:46:15,331][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-async]
[2020-10-27T11:46:15,332][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-async-search]
[2020-10-27T11:46:15,332][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-autoscaling]
[2020-10-27T11:46:15,332][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-ccr]
[2020-10-27T11:46:15,332][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-core]
[2020-10-27T11:46:15,333][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-data-streams]
[2020-10-27T11:46:15,333][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-deprecation]
[2020-10-27T11:46:15,333][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-enrich]
[2020-10-27T11:46:15,333][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-eql]
[2020-10-27T11:46:15,334][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-graph]
[2020-10-27T11:46:15,334][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-identity-provider]
[2020-10-27T11:46:15,334][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-ilm]
[2020-10-27T11:46:15,334][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-logstash]
[2020-10-27T11:46:15,335][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-ml]
[2020-10-27T11:46:15,335][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-monitoring]
[2020-10-27T11:46:15,335][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-ql]
[2020-10-27T11:46:15,335][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-rollup]
[2020-10-27T11:46:15,336][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-security]
[2020-10-27T11:46:15,336][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-sql]
[2020-10-27T11:46:15,336][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-stack]
[2020-10-27T11:46:15,336][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-voting-only-node]
[2020-10-27T11:46:15,336][INFO ][o.e.p.PluginsService     ] [es-jre2] loaded module [x-pack-watcher]
[2020-10-27T11:46:15,337][INFO ][o.e.p.PluginsService     ] [es-jre2] no plugins loaded
[2020-10-27T11:46:15,375][INFO ][o.e.e.NodeEnvironment    ] [es-jre2] using [1] data paths, mounts [[/home (/dev/mapper/vg0-home)]], net usable_space [256.8gb], net total_space [829.3gb], types [ext4]
[2020-10-27T11:46:15,375][INFO ][o.e.e.NodeEnvironment    ] [es-jre2] heap size [1gb], compressed ordinary object pointers [true]
[2020-10-27T11:46:15,407][INFO ][o.e.n.Node               ] [es-jre2] node name [es-jre2], node ID [MkfqQQ9_TMGhdORwIcyhsw], cluster name [elasticsearch]
[2020-10-27T11:46:18,065][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [es-jre2] [controller/315525] [Main.cc@114] controller (64 bit): Version 7.9.3 (Build 6c27059cb8397a) Copyright (c) 2020 Elasticsearch BV
[2020-10-27T11:46:18,625][INFO ][o.e.x.s.a.s.FileRolesStore] [es-jre2] parsed [0] roles from file [/home/jakob/ELASTIC/elasticsearch-7.9.3/config/roles.yml]
[2020-10-27T11:46:19,436][INFO ][o.e.t.NettyAllocator     ] [es-jre2] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2020-10-27T11:46:19,490][INFO ][o.e.d.DiscoveryModule    ] [es-jre2] using discovery type [zen] and seed hosts providers [settings]
[2020-10-27T11:46:19,835][WARN ][o.e.g.DanglingIndicesState] [es-jre2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2020-10-27T11:46:20,159][INFO ][o.e.n.Node               ] [es-jre2] initialized
[2020-10-27T11:46:20,159][INFO ][o.e.n.Node               ] [es-jre2] starting ...
[2020-10-27T11:46:20,288][INFO ][o.e.t.TransportService   ] [es-jre2] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2020-10-27T11:46:20,429][WARN ][o.e.b.BootstrapChecks    ] [es-jre2] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-10-27T11:46:20,436][INFO ][o.e.c.c.ClusterBootstrapService] [es-jre2] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2020-10-27T11:46:23,443][INFO ][o.e.c.c.Coordinator      ] [es-jre2] setting initial configuration to VotingConfiguration{MkfqQQ9_TMGhdORwIcyhsw}
[2020-10-27T11:46:23,580][INFO ][o.e.c.s.MasterService    ] [es-jre2] elected-as-master ([1] nodes joined)[{es-jre2}{MkfqQQ9_TMGhdORwIcyhsw}{NmgLb04NT7y5z4mt7q70YQ}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=33294389248, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{es-jre2}{MkfqQQ9_TMGhdORwIcyhsw}{NmgLb04NT7y5z4mt7q70YQ}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=33294389248, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}
[2020-10-27T11:46:23,618][INFO ][o.e.c.c.CoordinationState] [es-jre2] cluster UUID set to [7Vs1zHUaQQ-AocUQYJVRRA]
[2020-10-27T11:46:23,646][INFO ][o.e.c.s.ClusterApplierService] [es-jre2] master node changed {previous [], current [{es-jre2}{MkfqQQ9_TMGhdORwIcyhsw}{NmgLb04NT7y5z4mt7q70YQ}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=33294389248, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2020-10-27T11:46:23,676][INFO ][o.e.h.AbstractHttpServerTransport] [es-jre2] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2020-10-27T11:46:23,676][INFO ][o.e.n.Node               ] [es-jre2] started

After renaming the file it starts up fine and I can also referance the new file in a pipeline.
Note: I used the file from the module here to simplify the repro, but also tried with a downloaded GeoIP db file.

Provide logs (if relevant):

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[joegallo]

Related to #36949, which added the assertDatabaseExistence checks.

[jasontedor]

Not sure what the correct approach to this is. Either we should update the docs to reflect that it can not have the same file name or we should disable the check.

I want to clarify that this is deliberate, because those are not custom databases. We ensure that the built-in databases are loaded from the ingest-geoip module path, and custom databases are treated as configuration files from the config directory. One reason we did this was to ensure that users didn't have stale database files sitting around from the time when these database files were unpackaged from the ingest-geoip plugin into the configuration directory, but we transitioned to bundling them as part of the distribution when we made ingest-geoip a module instead of a plugin. This check ensures we don't load stale databases, but also that we aren't leniently and silently ignore them, which would be surprising to a user.

[jakommo]

Thanks for the background @jasontedor. Still not sure I understand what the correct approach would be then?

Are you saying this should only be used for custom databases but not to "update" to a more recent MaxMind DB?
I created this after helping a user that wanted to update the GeoLite2-City.mmdb to get more recent IP data, without updating Elasticsearch.

If we fail this on purpose for the reasons you outlined, then I think we should mention it in the docs, because currently they suggest that GeoLite2-City.mmdb is the default value for database_file in the config dir.

[jasontedor]

@jakommo I think a user could "update" these databases by treating them as custom, namely giving them a name that distinguishes them from the built-in databases such as GeoLite2-2020-10-29-City.mmdb and then referring to them in the processor definition:

    ...
    {
      "geoip": {
        "field": ...
        "database_file": "GeoLite2-2020-10-29-City.mmdb",
        ...
      }
    }
    ...

Note that the extension must be -(City|Country|ASN).mmdb.

I opened #64340.