feat: Live-update subway status on subway alerts page

[joshlarson]

Screen.Recording.2025-08-04.at.1.47.05.PM.mov

---

No ticket, but this is pretty cool, isn't it?

Ticket: Live-update system status on subway alerts page

[thecristen]

This is really cool!!!

[thecristen]

nit: I think you can use DotcomWeb.Endpoint.subscribe("alerts") here, as I believe the endpoint has all these (subscribe/broadcast/etc) functions available

[joshlarson]

Huh - I didn't think it would work, but it seems to work.

[joshlarson]

Oooh actually the thing that worked was using DotcomWeb.Endpoint for the subway_status_updated message. This suggestion isn't working for the reason that I thought: DotcomWeb.Endpoint is started after SubwayCache, which is causing SubwayCache to fail to start.

** (Mix) Could not start application dotcom: Dotcom.Application.start(:normal, []) returned an error: shutdown: failed to start child: Dotcom.SystemStatus.SubwayCache
    ** (EXIT) an exception was raised:
        ** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

            (stdlib 7.0.2) :ets.lookup(DotcomWeb.Endpoint, :pubsub_server)
            (dotcom 0.0.1) deps/phoenix/lib/phoenix/endpoint.ex:541: DotcomWeb.Endpoint.config/2
            (dotcom 0.0.1) lib/dotcom_web/endpoint.ex:4: DotcomWeb.Endpoint.pubsub_server!/0
            (dotcom 0.0.1) lib/dotcom_web/endpoint.ex:4: DotcomWeb.Endpoint.subscribe/2
            (dotcom 0.0.1) lib/dotcom/system_status/subway_cache.ex:36: Dotcom.SystemStatus.SubwayCache.init/1
            (stdlib 7.0.2) gen_server.erl:2276: :gen_server.init_it/2
            (stdlib 7.0.2) gen_server.erl:2236: :gen_server.init_it/6
            (stdlib 7.0.2) proc_lib.erl:333: :proc_lib.init_p_do_apply/3

But.... if I swap those in application.ex so that SubwayCache is started after DotcomWeb.Endpoint, then it does seem to work.

I thought it was considered virtuous to start the Endpoint last - am I doing evil by swapping them around?

[thecristen]

nah you're right, well I don't think it's evil per se, but we may as well stick with the convention of having Endpoint last

[thecristen]

question: Why not start this in test too?

[joshlarson]

Because having it running causes some funky errors, which I think are related to the lack of existence of Dotcom.Utils.DateTime.Mock.

If I adopt this suggestion, and also have this run in test, then I get this error 👇, because init tries to call a function that calls now/0.

** (Mix) Could not start application dotcom: Dotcom.Application.start(:normal, []) returned an error: shutdown: failed to start child: Dotcom.SystemStatus.SubwayCache
    ** (EXIT) an exception was raised:
        ** (Mox.UnexpectedCallError) no expectation defined for Dotcom.Utils.DateTime.Mock.now/0 in process #PID<0.538.0> with args []
            (mox 1.2.0) lib/mox.ex:903: Mox.__dispatch__/4
            (dotcom 0.0.1) lib/dotcom/system_status.ex:22: Dotcom.SystemStatus.subway_status/0
            (dotcom 0.0.1) lib/dotcom/system_status/subway_cache.ex:31: Dotcom.SystemStatus.SubwayCache.init/1
            (stdlib 7.0.2) gen_server.erl:2276: :gen_server.init_it/2
            (stdlib 7.0.2) gen_server.erl:2236: :gen_server.init_it/6
            (stdlib 7.0.2) proc_lib.erl:333: :proc_lib.init_p_do_apply/3

If I leave init as-is, then the alerts repo tests start failing for the same underlying reason, but in a different way:

11:16:32.216 [error] GenServer Dotcom.SystemStatus.SubwayCache terminating
** (Mox.UnexpectedCallError) no expectation defined for Dotcom.Utils.DateTime.Mock.now/0 in process #PID<0.525.0> with args []
    (mox 1.2.0) lib/mox.ex:903: Mox.__dispatch__/4
    (dotcom 0.0.1) lib/dotcom/system_status.ex:22: Dotcom.SystemStatus.subway_status/0
    (dotcom 0.0.1) lib/dotcom/system_status/subway_cache.ex:42: Dotcom.SystemStatus.SubwayCache.handle_info/2
    (stdlib 7.0.2) gen_server.erl:2434: :gen_server.try_handle_info/3
    (stdlib 7.0.2) gen_server.erl:2420: :gen_server.handle_msg/3
    (stdlib 7.0.2) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Last message: :alerts_updated

☝️ happens the first time the alerts cache store tries to publish an :alerts_updated message. That doesn't cause any tests to fail the first time, but then the second, third, etc times :alerts_updated messages are published, we get errors like 👇

     ** (exit) exited in: GenServer.call(Alerts.Cache.Store, {:update, [%Alerts.Alert{id: "orange_alert", active_period
: [], banner: "", cause: "", created_at: nil, description: "", effect: :unknown, header: "", image: nil, image_alternat
ive_text: nil, informed_entity: %Alerts.InformedEntitySet{activities: MapSet.new([]), direction_id: MapSet.new([nil]),
entities: [%Alerts.InformedEntity{activities: MapSet.new([]), direction_id: nil, facility: nil, route: "Orange", route_
type: nil, stop: nil, trip: nil}], facility: MapSet.new([nil]), route: MapSet.new(["Orange"]), route_type: MapSet.new([
nil]), stop: MapSet.new([nil]), trip: MapSet.new([nil])}, lifecycle: :unknown, priority: :low, severity: 5, updated_at:
 ~U[2025-09-15 14:21:30.221355Z], url: ""}, %Alerts.Alert{id: "red_alert", active_period: [], banner: "", cause: "", cr
eated_at: nil, description: "", effect: :unknown, header: "", image: nil, image_alternative_text: nil, informed_entity:
 %Alerts.InformedEntitySet{activities: MapSet.new([]), direction_id: MapSet.new([nil]), entities: [%Alerts.InformedEnti
ty{activities: MapSet.new([]), direction_id: nil, facility: nil, route: "Red", route_type: nil, stop: nil, trip: nil}],
 facility: MapSet.new([nil]), route: MapSet.new(["Red"]), route_type: MapSet.new([nil]), stop: MapSet.new([nil]), trip:
 MapSet.new([nil])}, lifecycle: :unknown, priority: :low, severity: 5, updated_at: ~U[2025-09-15 14:21:30.221355Z], url
: ""}, %Alerts.Alert{id: "blue_alert", active_period: [], banner: "", cause: "", created_at: nil, description: "", effe
ct: :unknown, header: "", image: nil, image_alternative_text: nil, informed_entity: %Alerts.InformedEntitySet{activitie
s: MapSet.new([]), direction_id: MapSet.new([nil]), entities: [%Alerts.InformedEntity{activities: MapSet.new([]), direc
tion_id: nil, facility: nil, route: "Blue", route_type: nil, stop: nil, trip: nil}], facility: MapSet.new([nil]), route
: MapSet.new(["Blue"]), route_type: MapSet.new([nil]), stop: MapSet.new([nil]), trip: MapSet.new([nil])}, lifecycle: :u
nknown, priority: :low, severity: 5, updated_at: ~U[2025-09-15 14:21:30.221355Z], url: ""}], nil}, 5000)
         ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name,
 possibly because its application isn't started
     code: Store.update([orange_alert, red_alert, blue_alert], nil)
     stacktrace:
       (elixir 1.18.4) lib/gen_server.ex:1121: GenServer.call/3
       (dotcom 0.0.1) lib/alerts/cache/store.ex:34: Alerts.Cache.Store.update/2
       test/alerts/repo_test.exs:53: (test)

---

I'm sure there's a more elegant way to do this, but blocking the genserver in test did seem to work! 😅

[thecristen]

ah, I think we typically just add this to the test module to mock that:

setup _ do
  stub_with(Dotcom.Utils.DateTime.Mock, Dotcom.Utils.DateTime)

  :ok
end

[joshlarson]

That stub would somehow need to get initialized before the application starts in the test. I haven't figured out yet how to do that!

I tried adding stub_with(Dotcom.Utils.DateTime.Mock, Dotcom.Utils.DateTime) to the individual *_case.ex setup blocks, but those get executed after init is called, which means that I can't get around

** (Mix) Could not start application dotcom: Dotcom.Application.start(:normal, []) returned an error: shutdown: failed to start child: Dotcom.SystemStatus.SubwayCache
    ** (EXIT) an exception was raised:
        ** (Mox.UnexpectedCallError) no expectation defined for Dotcom.Utils.DateTime.Mock.now/0 in process #PID<0.538.0> with args []
            (mox 1.2.0) lib/mox.ex:903: Mox.__dispatch__/4
            (dotcom 0.0.1) lib/dotcom/system_status.ex:22: Dotcom.SystemStatus.subway_status/0
            (dotcom 0.0.1) lib/dotcom/system_status/subway_cache.ex:31: Dotcom.SystemStatus.SubwayCache.init/1
            (stdlib 7.0.2) gen_server.erl:2276: :gen_server.init_it/2
            (stdlib 7.0.2) gen_server.erl:2236: :gen_server.init_it/6
            (stdlib 7.0.2) proc_lib.erl:333: :proc_lib.init_p_do_apply/3

---

Reverting this change combined with the stub_with suggestion looks promising, but we still need to add the stub_with clause to any test whose execution path goes through Alerts.Cache.Store. We can shortcut some of the pain by adding the setup clause to ConnCase, but some of the tests that touch Alerts.Cache.Store use ExUnit.Case, which we can't modify.

I don't love this for two reasons:

1. I actually thought that this change was good, and I'm not keen on reverting it.
2. The way that it fails makes it flaky and hard to debug - the first failure crashes the GenServer, which doesn't raise an error; the second failure then crashes and raises an error because the GenServer doesn't exist anymore, but gives no clue about when or why the GenServer crashed to begin with. This'll crop up again any time we add a new test and forget to add the stub_with somewhere where we should have added it, but then the error we get will be flaky and not easily tied to the offending test.

Would you be up for pairing on this at some point? Hopefully there's a better solution that I'm not thinking of right now?

[joshlarson]

I was kind of torn about doing this versus a @cacheable decoration. There were a few reasons I went this route, but none of them are dealbreakers:

• I knew how to use GenServers in this way, and (to me at least), the code was pretty easy to write and read.
• I wasn't sure about redis serializing and deserializing subway status in a way that would preserve status equality when things aren't changing (thus avoiding spamming open sockets with noop subway_status_updated messages)
• I wasn't sold on the idea of having to bust the subway status cache whenever alerts change - needing to do that kind of manipulation feels like it defeats the purpose of the @cacheable decoration to begin with.
• There's no real need for a TTL, since status will be recomputed whenever alerts are updated.

There are some potential downsides that I can see:

• This is storing subway status state in a different way than any of the other pieces of state, which creates cognitive load.
• Each dotcom node needs to store subway status (performance problem?)
• If for some reason the alerts fetcher doesn't run on a particular node, then sockets connected to that node will have outdated subway status data.

I tested out ☝️ locally and in deployed environments and it seemed to work, so I'm not convinced that the downsides are actual problems, but I did want to list them for full transparency.

What do you all think?

[thecristen]

One question I have is I'm not sure we need this intermediate layer... couldn't the LiveView itself subscribe to the alerts, then recompute subway_status() in its handle_info callback?

In answer to your actual question, I think the GenServer's a fine way to handle this.

[joshlarson]

(Talked offline, but want to post this here also)

This was a potentially premature optimization - without SubwayCache, the alerts fetcher will notify every connected browser session every time an alerts fetch happens (typically once per minute). With SubwayCache, the connected browser sessions will only get notified when subway status itself changes.

[thecristen]

ah, did we run into a using-different-versions-of-libretranslate problem again? 😬

[joshlarson]

Uggggh. I'm pinned on version 1.7.2. What's everyone else using??