System not trying second address

[morvael]

Describe the bug
I have a system configured with 2 addresses. If the first one is not responding during launch of application it will not try second system and will eventually quit (because the init code waits up to 60 seconds for successful login or other meaningful response).

Expected behavior
System tries second address in 30 seconds.

system information:

• OS: Linux
• Java version: JDK21
• QFJ Version: 3.0.1

Additional context

20260613-12:20:11: Session FIXT.1.1:xxx->yyy schedule is daily, 05:30:00-UTC - 17:30:00-UTC (daily, 07:30:00-CEST - 19:30:00-CEST)
20260613-12:20:11: Created session: FIXT.1.1:xxx->yyy
20260613-12:20:11: Configured socket addresses for session: [/127.0.0.1:10134, /127.0.0.1:10135]
20260613-12:20:11: MINA session created: local=/127.0.0.1:43448, class org.apache.mina.transport.socket.nio.NioSocketSession, remote=view-localhost/127.0.0.1:10134
20260613-12:20:11: Disconnecting: Encountered END_OF_STREAM
20260613-12:20:41: MINA session created: local=/127.0.0.1:56316, class org.apache.mina.transport.socket.nio.NioSocketSession, remote=view-localhost/127.0.0.1:10134
20260613-12:20:41: Disconnecting: Encountered END_OF_STREAM

I think it may be caused by no callbacks reaching me because Session.next has this code:

if (message == EventHandlingStrategy.END_OF_STREAM) {
  disconnect(ENCOUNTERED_END_OF_STREAM, false);
  return;
}

and disconnect has this code:

if (logonReceived || logonSent) {
  try {
    application.onLogout(sessionID);

so I'm never notified of this. But that doesn't explain why it only tries the first socket address again and again.

[morvael]

Maybe the problem is with IoSessionInitiator.pollConnectFuture() which always resets nextSocketAddressIndex to 0? But this is just my wild guess.

[morvael]

This is the stacktrace where it set nextSocketAddressIndex from 1 to 0 and attempted to reconnect to the same bad address:

edit: sorry uploading images failed so here's link: https://ibb.co/VW2DLFBX

[chrjohn]

I need to look at this after the weekend. Setting SocketConnectTimeout should help with the long delay though

[morvael]

Disconnect is immediate, but there's no callback about this and in 30 seconds there's the next connection attempt, sadly again to first address. I think this is some special disconnect, maybe FW Reject. It's part of disaster recovery tests.

[chrjohn]

there's no callback about this

StateListener.onDisconnect() should get called. Maybe also .onConnectException(), but probably not since I cannot see any exception in the log.

[the-thing]

@chrjohn How similar is this to #402? I think the underlaying cause might the same or just the same failure.

The author of the other issue claims that the failover mechanism works for unresolved address which makes sense since quickfix.mina.initiator.IoSessionInitiator.ConnectTask#getNextSocketAddress could throw an IllegalArgumentException when hostname is null from the previous, unresolved attempt. This will skip quickfix.mina.initiator.IoSessionInitiator.ConnectTask#pollConnectFuture call.

morvael might be onto something here.

[chrjohn]

@the-thing looks like a similar issue to me. Any idea how we could detect the timeout?

[the-thing]

I will have a look soon, but it seems like there are two things that we have to confirm.

• quickfix.SessionStateListener#onDisconnect is getting called (I think it does)
• check round robin for multiple addresses

[morvael]

I admit I'm maybe not using SessionStateListener, so only the round robin thing is a problem.

[the-thing]

@the-thing looks like a similar issue to me. Any idea how we could detect the timeout?

I think the timeout is already being detected.

[the-thing]

This is probably the breaking change.

#152

[morvael]

Yeah, I do a full recycle of the client between days so they would already start from host 0. However my scenario is that it restarts from 0 in one lifecycle without any other commands. These two states should be somehow distinguished.

[morvael]

I guess some kind of alternative would be to create client with just one host and on specific events change the host in the configuration (rebuilding the client if that's required), but this seems too heavy. Maybe add a config switch allowing/disallowing this kind of round robin ignoring?

[the-thing]

I guess some kind of alternative would be to create client with just one host and on specific events change the host in the configuration (rebuilding the client if that's required), but this seems too heavy. Maybe add a config switch allowing/disallowing this kind of round robin ignoring?

Config is probably not the best, because we will have 2 broken functionalities driven by a configuration.

@chrjohn @morvael

We could write custom round robin mechanics, but this requires a bit of thinking and some changes to get it right so I would avoid it for the time being. Here is what I'm thinking.

Currently we use different String resasons for disconnect reasons. Add an additional flag indicating if session was disconnected by END_OF_STREAM (initially boolean, maybe enum later for more reasons if required). The flag is stored in the quickfix.Session state. When new connection is made we check the flag and decide when the index should advance and / or be reset.

The implication of this is that if a session failovers during the day to Host2 and then sessions is disconnect by EndTime and later connected again it will still connect to Host2, but I guess this is expected behavior...