feat: add PostgreSQL output mode - #493
Conversation
|
@xoolive CI is failing because that ARM runner has rustc 1.93.1, and the sqlx 0.9.0 I added for Postgres needs 1.94. I tried dropping sqlx back to 0.8 to stay on 1.93, but it doesn't work: sqlx 0.8 pulls libsqlite3-sys 0.30 for its sqlite support, and rs1090 already uses rusqlite 0.39 (libsqlite3-sys 0.37). Both want to link the sqlite3 native lib, so cargo won't resolve it. sqlx 0.9.0 is the version that lines those up, so we should keep it. Simplest fix: pin the toolchain. A |
|
Does it work locally if you pin the version? Also I remain to be convinced with the postgresql use case... (so I appreciate it is gated behind a feature 😁) I see you didn't try to make something like a trajectory table with random features you would arbitrarily select, which I appreciate, but then what do you gain if you have almost no expressivity in the queries? or am I completely wrong? Note that we use a redis pub/sub here for our tangram use case which delegates all these things we don't want to deal with at the application level... 😅 Any opinion @abc8747 ? |
| } | ||
|
|
||
| if let Some(pool) = &postgres_pool { | ||
| insert_message(pool, &postgres_table, &msg).await; |
There was a problem hiding this comment.
I haven't used Postgres in a while, but won't inserting each message row by row in a hot decode loop cause problems, especially if there is network latency? For reference, we are currently dealing with ~1000-2000 messages/sec
There was a problem hiding this comment.
That was a concern as well (that I kept for second round opinion 😅)
| serde = { version = "1.0.228", features = ["derive"] } | ||
| serde_json = "1.0.149" | ||
| soapysdr = { version = "0.5.0", optional = true } | ||
| sqlx = { version = "0.9.0", default-features = false, features = [ |
|
I also have some reservations about adding postgres directly to jet1090, but I’d like to understand your use case. Fyi, in Tangram we needed to be able to filter by icao24, altitude, groundspeed etc., so we stored the decoded messages published on Redis into a Delta Lake with buffered / batched writes. |
|
Oh I haven't checked about PostgreSQL in a while and didn't know about the JSON queries. Maybe I should have read more before answering |
|
Hey both! I wasn't aware of Tangram, so I will definitely take a look! Agree that batching would be smarter here @abc8747, I implemented it without to follow the Redis implementation. To be fair, I've been running this branch on my jet1090 instance (which aggregates across two receivers in Germany and England), and I haven't seen any issues even without batching. Regarding my use case, I mainly just wanted access to the data (past the history expiry time) to analyze, collect interesting metrics, as well as support playback. It seems like most of these things can be done via Tangram already, and maybe I would just need to implement a custom plugin for anything I'm missing 😄 |
|
Awesome, let's not discard your idea just yet though! How about you have a look at tangram, look at plugins (maybe consider if they're worth sharing?) and if you still consider the postgresql approach necessary, we can still add it (gated behind the feature as you did) About performance, when I aggregate data from about 10 radarcapes without proper coverage, things start to get complicated to be honest |
3 participants
Adds PostgreSQL as an output mode, alongside the existing outputs.
When
--postgres-urlis set, every decoded message is inserted into a PostgreSQL table. The full message is stored asJSONB, withtimestamp,icao24, anddfpulled into their own columns for easy filtering. The table is created automatically if it doesn't exist, and its name defaults tojet1090(configurable with--postgres-table).