Use Java 16+ unix sockets

[overheadhunter]

I learned via a downstream bug report, that this library transitively depends on a several JNR libs. Which is fine unless certain configuration problems arise. 😉

When looking at the dependency tree I noticed, that the only reason why dbus-java requires those is for accessing unix sockets. Now that we have native support for unix sockets in Java 16+, I'd ask whether it is possible to replace jnr-unixsocket and get rid of a whole bunch of dependencies.

I know, this would increases the required Java version significantly. But as the current version is mature and stable, I would propose to schedule this for a major release that could live in parallel to the current version. No changes in the public API, so users can switch between versions depending on their needs.

If you think this would be feasible from a maintainers perspective, I would offer our help to implement this.

[hypfvieh]

I already read about that feature in newer Java versions.
I'm not sure that the internal implementation supports all features required by dbus-java (e.g. setting certain flags on the socket like SO_PASSCRED etc).

As this library still continues to support older versions of Java, using the native implementation is not possible.
Maintaining two different version is also nothing I'm planning to do (takes too much time and effort).

Increasing the required Java version was already on my todo list.
I also always prefer to use the Java LTS versions (1.8 is LTS, 11 is LTS and 17 will be LTS).
The next Java version required by dbus-java would be 11, so any other project can still use this library without requiring a bleeding edge Java version.

One thing I could think of is to change the unix socket parts to use service loader to switch the "unix socket backend".
This is something I already do to support file descriptor passing using a third party library.
I'll have to investigate that as well as the supported socket options (like said before).

[overheadhunter]

I'm not sure that the internal implementation supports all features required by dbus-java (e.g. setting certain flags on the socket like SO_PASSCRED etc).

Ok that needs to be checked, of course.

Maintaining two different version is also nothing I'm planning to do (takes too much time and effort).

That's what I assumed. But I still had a little hope that (judging from the low number of open issues), the current version is mature and doesn't require that much attention, so focus can shift to a new version. 😉

One thing I could think of is to change the unix socket parts to use service loader to switch the "unix socket backend".
This is something I already do to support file descriptor passing using a third party library.
I'll have to investigate that as well as the supported socket options (like said before).

While certainly good software design, this might not be that much less effort, since it would require at least two, maybe three new libraries:

1. The API definition of said service (and possibly the JEP 380 implementation as a MR-JAR)
2. The JNR implementation providing the service
3. (if not already part of 1., the JEP 380-based service provider)

[hypfvieh]

That's what I assumed. But I still had a little hope that (judging from the low number of open issues)

Low number of issues does not mean there are no bugs or that there is no room for improvement.
There are parts which could certainly need some more attention (rewriting), but I'm pretty busy lately...

As far as I can see in the description of JEP 380, it seems that it does not support all feature required by dbus-java:

Non-Goals

It is not a goal to support features that are not common across the major Unix platforms and Windows. This includes Linux-specific features such as the abstract filesystem-independent namespace. It also includes features that are generally supported on Unix but unsupported on Windows, such as socket pairs. Support for these features can be revisited in the future if needed and, in the case of the missing Windows features, if the Windows platform evolves to support them. An exception to this non-goal is support for peer credentials, which can be implemented as a JDK specific socket option on platforms that support it. Other socket options may be investigated as follow up work, possibly also as JDK specific options, after this JEP is completed.

One issue would be: dbus-java supports abstract unix sockets, which is not part of JEP 380 because windows does not support that.
But I will take a look at that if I find some spare time to do it.

[brett-smith]

I have some interest in this. I am using the embedded bus on Windows, and so have to use the TCP transport on this OS. It is noticeably slower communication between the client and server when compared to running on OS X or Linux both using domain sockets. So frankly any alternative on Windows would be great.

I had actually started on implementation on Windows that used name pipes, but while this whould be relatively easy with JNA (there is code already written), I couldn't find enough documentation on JNR to fit in with dbus-java and lost interest!

I can also confirm that you don't have to use an abstract path on Linux, you can connect to the system or session dbus fine or create your own embedded bus using a real path. OS X doesn't support abstract paths either.

[hypfvieh]

When just using the library as "client", abstract sockets are not really needed. But when using the DBusDaemon class, abstract might be required.
In the current version, this is not handled at all. If you use the DBusDaemon on OS X with a bus address indicating an abstract unixsocket, it will probably throw an exception.

I'll have to dig into this when I have some spare time left.
Handling the abstract socket stuff should be improved.
Switching the 'transport' of DBus is already supported (e.g. using TCP instead of UnixSocket), so maybe I can improve that part to allow using another implementation of unixsocket... I'll have to check that.

Also I still want to support older Java versions (especially 11) when 17 will be released next month.
I'm also a fan of using newer Java versions, but when providing libraries always forcing the users to use bleeding edge Java is not a good idea. In case someone uses this library on some IOT devices, it might be difficult to update/switch to a recent Java version.

I few years ago when I started with dbus-java there were still frameworks for IOT forcing the developer to provide their plugins with Java 1.7 compatibility. That was really annoying, so my policy is to at least support the previous Java LTS version.
In this case this would mean I would bump the required Java version to 11 shortly after 17 is released.

[brett-smith]

OS X does indeed throw an exception if you try to use an abstract, but I am successfully using DBusDaemon (and a client) with domain sockets with a non-abstract path so am not too bothered by this.

As another vote for having Java 16 domain socket support ... I have a research project scheduled to see if the application I use dbus-java with can be compiled with GraalVM native-image. One of the sticking points is almost certainly going to be dbus-java itself because of it's use of JNR.

The last time I looked, neither JNA or JNR were fully supported. Graal SDK does have it's own FFI, but that would mean a rewrite of jnr-unixsocket. However, Graal do have an experimental Java 16 build that presumably can access this built-in domain socket support. So if that is possible, then that would give you Graal support for free

[hypfvieh]

I just started to take a closer look to native unixsockets in Java 16.
There are some more 'special handlings' beside the 'abstract' unix socket which is not required.

Especially the authentication stuff (SO_PASSCRED vs SO_PEERCRED) and the handling of all of this on BSD and OS X.
For BSD / OS X, there is some special treatment for the credential part relying heavily on jnr-posix (see FreeBSDHelper).
I removed / commented the code in SASL / NativeUnixTransport which calls FreeBSDHelper. I don't know if all of this is required when using native unixsockets and I don't have FreeBSD or OS X to test that.

More changes were required due to the fact that the Java 16 unix sockets uses NIO while the current implementation uses the old socket API.
All methods which required Socket / InputStream / OutputStream were changed to support SocketChannel, including the SASL stuff.
This will also breaks the SPI/ServiceLoader stuff used to support dbus-java-nativefd.

My first tests using TestAll/TestLowLevel on an Ubuntu Linux machine worked.
More tests on other OSes would be appreciated.

I'm not sure about including or not including jnr-unixsocket as dependency.
At the moment I have a few ideas:

1. use ServiceLoader (or something like that) and provide the backend separately (e.g. dbus-java-jnr-unixsockets / dbus-java-native-unixsockets), the developer has to choose one of them
2. use ServiceLoader, include dbus-java-jnr-unixsockets by default, so developer can exclude this and include dbus-java-native-unixsockets if needed
3. Provide both implementations in dbus-java, using native version if suitable JRE (16+) is found, use jnr otherwise
4. Provide both implementations, use jnr per default and native only when certain property (or similar) is set

I uploaded the current development state to a new branch (java16-unixsocket) which is work-in-progress.
To use the new unixsocket implementation, you need Java 16 (or higher).
If you run it from your development IDE, you can force it to use the new implementation by setting the system property dbus.java.disable.jnr-unixsocket to true (look at the TestAll/TestLowLevel).
If you run it in your application you may exclude jnr-unixsocket from the dependencies (not tested yet).

[brett-smith]

Thanks for making a start on this. Ive given it a try on Linux, Windows and Mac. The only real success so far I've had is connecting to the system bus on Linux. As the primary purpose of dbus-java, this works as well as it ever did.

However, I can't seem to get EmbeddedDBusDaemon working at all with either TCP or domain sockets on any operating system.

There were a few tweaks I needed to get it to compile in my environment. module-info.java needed updating. All the JNR modules needed to be made static (optional), and 3 new JDK modules needed to be added - java.security.sasl, jdk.security.auth and jdk.net.

It seemed I needed to add listen=true to the address set on the embedded daemon. That got me a little further. Maybe this was always required, but I didn't used to have to do this on stable dbus-java.

On Windows, I noticed the path is stil /tmp/dbus-XXXXX, does this need to be a window path? If so the system property java.io.tmpdir may be a way to contstruct that path in a cross platform way.

The main problem now seems to be starting the embedded daemon, then getting a connection to export some services on (in the same JVM). This results in two exceptions, one on the daemon thread and on the main thread where I am trying to connect to the daemon.

java.lang.RuntimeException: java.net.BindException: Address already in use
	at org.freedesktop.dbus.bin.EmbeddedDBusDaemon.startInForeground(EmbeddedDBusDaemon.java:58)
	at java.base/java.lang.Thread.run(Thread.java:831)
Caused by: java.net.BindException: Address already in use
	at java.base/sun.nio.ch.UnixDomainSockets.bind0(Native Method)
	at java.base/sun.nio.ch.UnixDomainSockets.bind(UnixDomainSockets.java:111)
	at java.base/sun.nio.ch.ServerSocketChannelImpl.unixBind(ServerSocketChannelImpl.java:319)
	at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:292)
	at java.base/java.nio.channels.ServerSocketChannel.bind(ServerSocketChannel.java:223)
	at org.freedesktop.dbus.connections.transports.NativeUnixSocketTransport.connectImpl(NativeUnixSocketTransport.java:62)
	at org.freedesktop.dbus.connections.transports.AbstractTransport.connect(AbstractTransport.java:109)
	at org.freedesktop.dbus.bin.EmbeddedDBusDaemon.startSocket(EmbeddedDBusDaemon.java:84)
	at org.freedesktop.dbus.bin.EmbeddedDBusDaemon.listen(EmbeddedDBusDaemon.java:73)
	at org.freedesktop.dbus.bin.EmbeddedDBusDaemon.startInForeground(EmbeddedDBusDaemon.java:55)
	... 1 more
05-09-2021 00:05:20 [DBusConnection] ERROR IncomingMessageThread - FatalException in connection thread.
org.freedesktop.dbus.exceptions.FatalDBusException: Underlying transport returned EOF (1)
	at org.freedesktop.dbus.connections.AbstractConnection.readIncoming(AbstractConnection.java:1149)
	at org.freedesktop.dbus.connections.IncomingMessageThread.run(IncomingMessageThread.java:39)
05-09-2021 00:05:20 [DBus Sender Thread-1] INFO  DBusConnection - Setting reply to MethodCall(0,1) { Path=>/org/freedesktop/DBus, Interface=>org.freedesktop.DBus, Member=>Hello, Destination=>org.freedesktop.DBus } { } as an error

Regarding your points as to how this could be plugged in, my preference would be for ServiceLoader to be used. This fits in with JPMS and opens the possibility of adding further novel transports as add-ons. There would be nothing stopping you using ServiceLoader internally in dbus-java to discover all supported / available transports.

[hypfvieh]

Thanks for the feedback. I have to admit I didn't look at modules-info (and I still don't really understand JPMS).
I added your suggestions regarding it in my branch.

EmbeddedDbusDaemon is broken due to my refactoring. I will fix it and implement the ServiceLoader stuff as well.

The thing with the '/tmp' path is simply wrong, and is also wrong in upstream. Instead of using java.io.tmpdir, "/tmp" is used hardcoded. I will fix this as well and post an update here when you can pull the branch again.

[hypfvieh]

Alright, it has taken some more time than I excepted, but I just pushed my latest changes.

The library has been splitted into multiple parts:

• dbus-java (core library)
• dbus-java-transport-jnr-unixsocket (jnr-unixsocket implementation)
• dbus-java-transport-native-unixsocket (Java 16+ unix socket implementation)
• dbus-java-transport-tcp (TCP implementation)
• dbus-java-tests (containing all unit test to allow running them against each transport)

The system property stuff used before has been removed.

So the big change is that you have to add at least on of the transport modules as well as dbus-java itself. Adding both, jnr and native unix socket will cause dbus-java to throw an exception because there can only be one protocol provider at the same time. In this case both provide support for 'unix' protocol, so you have to choose one of them (otherwise it would be more or less random which gets used during runtime). Combining jnr or native with tcp is still possible.

Maybe I'll rename dbus-java to dbus-java-core to make clear that it does not work without any of the transport implementations.

All parts now contain a module-info.java file and will also require at least Java 11 (Java 16+ is required when using native transport).

During my refactoring I saw that there are not so many tests running with the TCP version. Most tests use unix socket or fail due to authentication issues. E.g. when running TestPeer2Peer using TCP on both ends (EmbeddedDbusDaemon + DBusConnection), authentication always fails because SASL switches from AUTHENTICATED state to FAILED state.. I'm not sure whats wrong at that point, but as those tests where never executed using TCP, I just disabled them for TCP right now.
Maybe someone has some idea what's wrong here.

All unit tests are working with native/jnr when using linux. No other OSes tested yet.
I also hope I fixed the broken DBusDaemon ;)

Feedback welcome!

[brett-smith]

Thanks, its looking a lot more polished already, but I am still seeing more or less the same thing as I saw before though, which sounds like the same case as TestPeer2Peer.

I have an app that is in two parts. The service part exports a DBus interface on the system or session bus if it's there, but it tries to start an embedded bus if one doesn't exist (i.e. usually Mac and Windows). It then tries to export the service on this embedded bus. So it's all happening in the same JVM.

FWIW, the same now seems to happen with both JNR and native domain sockets so hopefully it's just an effect of the refactoring rather than a limitation of native sockets.

I'll post if I can find why.

Edit: Found one thing. TestPeer2Peer can be made to work, if you change TcpTransport SASL auth mode to AUTH_ANON.

    TcpTransport(BusAddress _address, int _timeout) {
        super(_address);
        timeout = _timeout;
//        setSaslAuthMode(SASL.AUTH_SHA);
        setSaslAuthMode(SASL.AUTH_ANON);
    }

Edit 2: While that makes TestPeer2Peer complete, it doesn't completely help in my application. It gets further, but then hangs trying to obtain a connection to the embedded bus (again using TCP).

[hypfvieh]

Using SASL.AUTH_ANON is not a proper solution as the DBus specifications says that you have to use the SHA method when connecting to DBus using TCP. And because it is possible to run the original dbus daemon in TCP mode, dbus-java should be able to connect to it.

I did some more investigation and it seems that SASL is somehow broken when using TCP. If you use the current master branch and change the CONNECTION_ADDRESS in TestPeer2Peer to a TCP based address, the test will always fail.
For me it looks like the SASL response send in doResponse when using AUTH_SHA should be OK not CONTINUE.
When changing the return value to OK TestPeer2Peer passes on master and passes on the branch (please pull branch, master is still unmodified).

Some of the other tests still fail or get stuck (e.g. TestTwoPart). Maybe I'll take a look at that later.

[overheadhunter]

Feedback welcome!

Any chance you could publish this as an alpha version? This would simplify creating adjusted test versions of downstream projects.

@purejava @swiesend What do you think? Would this change help to modularize and reduce dependencies of kdewallet/secret-service libs?

[purejava]

@purejava @swiesend What do you think? Would this change help to modularize and reduce dependencies of kdewallet/secret-service libs?

Besides depending on dbus-java, kdewallet has only JUnit and SLF4J as further dependencies. So, on modularizing kdewallet, the dependencies for one part/module would be significantly reduced.

[purejava]

I add this change to my to do list.