process does not come up on ubuntu machine

[singalshashank]

Hi Team,

I tried running the embedded postgresql from my machine. It works fine on local ubuntu machine but on remote CI setup, I always get the following logs as it doesn't work.

Extract /home/teamcity/.embedpostgresql/postgresql-9.4.1-1-linux-x64-binaries.tar.gz START
Extract /home/teamcity/.embedpostgresql/postgresql-9.4.1-1-linux-x64-binaries.tar.gz START
[postgres error]LOG:  database system was shut down at 2015-05-19 05:57:27 UTC
[postgres error] LOG:  database system is ready to accept connections
[postgres error] LOG:  autovacuum launcher started
[postgres error] ERROR [2015-05-19 05:57:29,259] ru.yandex.qatools.embed.postgresql.PostgresProcess: Failed to read PID file (File '/tmp/embedpostgres-db-c9be464c-8cf5-435e-8f9a-76fb642e8232/postmaster.pid' does not exist)
Extract /home/teamcity/.embedpostgresql/postgresql-9.4.1-1-linux-x64-binaries.tar.gz START
FATAL:  could not open relation mapping file "global/pg_filenode.map": No such file or directory
[postgres error] FATAL:  could not open relation mapping file "global/pg_filenode.map": No such file or directory
[postgres error] JDBC URL is jdbc:postgresql://127.0.0.1:57039/test?user=test&password=test
FATAL:  could not open relation mapping file "global/pg_filenode.map": No such file or directory
[postgres error] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 14.986 sec <<< FAILURE!
Running com.......Test
[05:57:31][com.......Test] testJdbi...
[05:57:31][testJdbi...] org.postgresql.util.PSQLException: FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory
[05:57:31]
[testJdbi...] org.postgresql.util.PSQLException: FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory
    at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:420)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:195)
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
    at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:127)
    at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
    at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
    at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:41)
    at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
    at org.postgresql.Driver.makeConnection(Driver.java:414)
    at org.postgresql.Driver.connect(Driver.java:282)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:270)
    at com....testJdbi...setUp(Jdbi..Test.java:70)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
    at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
    at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
[05:57:31][testJdbi...] java.lang.NullPointerException
[05:57:31]
[testJdbi...] java.lang.NullPointerException
    at com....Jdbi..Test.tearDown(Jdbi..Test.java:85)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
    at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
    at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)

Regards,
Shashank

[singalshashank]

The CI setup also runs on an ubuntu machine.

[smecsia]

@singalshashank Thank you. Please indicate the version of Ubuntu, you are using on CI machine.

[singalshashank]

@smecsia
Please find below the details:
Distributor ID: Ubuntu
Description: Ubuntu 14.04.2 LTS
Release: 14.04
Codename: trusty

[singalshashank]

Is anyone still working on it?

[smecsia]

@singalshashank sorry, I was busy last two weeks. I actually could not reproduce this.
You're not running remote CI as a root, right? Can you try to run Postgres within Docker container on that machine?

[adamturski]

The same error I had when the embedded-postgresql process was not ended/killed and I tried run another one.
In that situation I had similiar errors like 'base/123 subdirectory missing', 'invalid TimeZone format' ...

After killing the process, everything was fine.

[smecsia]

@trak17 Your problem is a very rare and it is hard to reproduce and inspect the reasons. When you launch Postgres many times sequentially on single system, it (probably) caches something in the system files. Process killing does not always help, but system reboot solves the issue. Unfortunately I have no time to investigate this closer.
I'd recommend you to use the clean and isolated environment for your tests, where postgres would run normally every time.

[pboos]

We are currently getting this same error on Ubuntu.
It works fine under Mac OS X though. So I wonder what the difference could be.

Ubuntu: Ubuntu 14.04.4 LTS
Postgres: PRODUCTION (9.5)

SEVERE: Failed to read PID file (File '/tmp/postgresql-embed-ec29b9ac-03b0-4f55-ad06-bd2584ada682/db-content-1e43fdb5-ad0f-4d50-9b2d-37b1cabd0177/postmaster.pid' does not exist)

[pboos]

Sorry. Deleted previous comments of mine. They were a wrong way. Extraction seems to be fine. Just deleting afterwards seems to leave files behind.

The above error is thrown here in PostgresProcess.java:187

[pboos]

I think I traced down the error in our case. The starting process fails with the following error:

pg_ctl: cannot be run as root
Please log in (using, e.g., "su") as the (unprivileged) user that will
own the server process.

Running it as non root then gives the following error for initdb:

initdb: invalid locale settings; check LANG and LC_* environment variables

Solution (at least for us)

1. run as non root user
2. export LC_ALL=en_US.UTF-8

Reasons:
pg_ctl does refuse to run as root.
initdb: invalid locale settings; check LANG and LC_* environment variables

I was testing by running mvn test in the postgresql-embedded git repository on our server. And with the above I got the tests to run correctly.

[smecsia]

@pboos Thank you for the detailed research! Actually embedded postgres works well on Ubuntu (and this is proved by our CI server, which launches every PR and master build within Docker container with Ubuntu). There were different problems with launching on Mac and Win, but I believe the most of them have been resolved.
Your problem concerns PostgreSQL server directly (which is known to not start up under privileged user). Nevertheless, I hoped this problem has been solved by #29 but I had no time to confirm this. Probably, pg_ctl works under privileged user only on Windows, but not on Linux.

P.S. I'll add the notes about LANG and non-privileged user to README.

[corydolphin]

As a follow up, is there anything that can be done to print out more detailed failure messages?

It took me a while to find this post to solve my problem. And a little longer to read it closely enough to notice the locale issue!

[smecsia]

@corydolphin thanks for follow up. All available logging options and the locale configuration are mentioned in current readme. For the newer versions you don't need to configure locale yourself (it is done by default), so I don't think it is the case anymore.