A system that predicts whether a flight will be late, explains why flights get delayed, and answers questions about it in plain English.
Built on 1.6 million real flights from official US government records.
Imagine you could ask an airline's operations team three questions:
- "Will this flight be late?" — the system gives a probability, not a guess
- "Why do flights on this route run late?" — it breaks the delay down by cause
- "Which airline is most reliable?" — it ranks them from real data
This project answers all three. You can ask it in ordinary language and get an answer back in seconds.
The important part is what it won't do: if it doesn't have enough data to answer honestly, it says so instead of making something up.
Ask anyone why flights get delayed and they'll say weather. I analysed 319,395 delayed flights to check.
They're wrong.
| What causes delays | Share of all delay time |
|---|---|
| A previous flight ran late (knock-on effect) | 39.8% |
| Airline operations — crew, maintenance, baggage | 33.2% |
| Air traffic control and airport congestion | 19.9% |
| Weather | 7.0% |
| Security | 0.2% |
The biggest cause of flight delays is other flight delays.
An aircraft lands late in the morning, so its next flight leaves late, and that pushes the one after it. The delay travels through the day like a traffic jam. Weather is a distant fourth. Security is almost nothing.
And when a flight is delayed, how long is it? Half of all delays are under 42 minutes. But the worst 10% run past two and a half hours. Quoting the average alone (72 minutes) would misrepresent a typical delay by 70% — so the system reports the range, not one number.
flowchart LR
A["📥 Download<br/>1.6M real flights<br/>from US government"]
B["🔧 Clean and prepare<br/>the data<br/>(runs on Spark)"]
C["🧠 Train a model<br/>to spot delay<br/>patterns"]
D["📏 Check the model<br/>is honest<br/>about itself"]
E["💬 Answer questions<br/>in plain English"]
A --> B --> C --> D --> E
style A fill:#E8EEF5,stroke:#1F3B5C,color:#12212F
style B fill:#E8EEF5,stroke:#1F3B5C,color:#12212F
style C fill:#E8EEF5,stroke:#1F3B5C,color:#12212F
style D fill:#FFF4E6,stroke:#C2410C,color:#12212F
style E fill:#E8EEF5,stroke:#1F3B5C,color:#12212F
Step 1 — Get the data. The US government publishes every domestic flight: when it was scheduled, when it actually arrived, and who flew it. That's about 550,000 flights a month.
Step 2 — Prepare it. 1.6 million flights is too much for a normal computer to handle in memory, so this runs on Apache Spark — software designed to split big jobs across many processors. It works out things like how busy an airport is at each hour and how a route has been performing recently.
Step 3 — Train the model. The system learns from past flights which combinations of airport, airline, time of day and season tend to run late.
Step 4 — Check it's honest. This is the step most projects skip, and it's the one I care most about. A model that says "70% chance of delay" should be right about 70% of the time. Most models aren't, so I measure it and correct it.
Step 5 — Answer questions. Ask "how bad is JFK to LAX?" and get a real answer. Ask about an airline that isn't in the data and it tells you that, instead of inventing a figure.
The system can never make up a number.
This sounds obvious. It isn't — it's the single hardest thing about building anything with AI, because language models are very good at producing confident-sounding answers to questions they can't actually answer.
Here's how it's prevented. When you ask a question, the AI is only allowed to choose which lookup to run and phrase the result. Every figure comes from the data itself. If the data isn't there, the lookup refuses:
You: What about Emirates?
System: 'Emirates' is not in this dataset. It covers US reporting carriers only, so international airlines such as Emirates, Qatar Airways or Lufthansa do not appear at all — including their US routes.
Same for a route with too little history:
Only 12 flights on record for this route, below the 30-flight minimum for reliable statistics. No delay figures can be given.
That refusal lives in the code, not in an instruction to the AI. Telling an AI "please don't make things up" is a request. Making it structurally impossible is a guarantee.
The model scores 0.67 on a measure called AUC, where 1.0 is perfect and 0.5 is a coin flip. That sounds mediocre. It isn't — it's the honest answer.
This model predicts delays before the plane has left the gate. It doesn't know whether the incoming aircraft is late, whether the crew arrived, or what the weather will do. Published academic research on this same dataset gets the same range.
If I had scored above 0.90, that would mean I had a bug, not a breakthrough. The usual way people accidentally hit 0.95 is by letting the model see the departure delay — but a plane that left 40 minutes late obviously arrives late. That's arithmetic, not prediction. I deliberately removed it.
Being able to explain why I rejected a great-looking score is more meaningful than the score itself.
Raw machine-learning models are good at ranking — putting likely delays above unlikely ones. They're often bad at the actual numbers. A model might rank perfectly while its "70% chance" really means 45%.
I measured this and corrected it:
| Before correction | After correction | |
|---|---|---|
| Ranking ability (AUC) | 0.6723 | 0.6722 — unchanged |
| Honesty of the probabilities | 0.0400 | 0.0157 — 2.5× better |
The ranking didn't change at all. What changed is that the numbers now mean what they say. Every decision built on top of this model depends on that.
And the decision threshold follows from cost, not convention. Most systems flag anything above 50%. But missing a real delay costs an airline far more than a false alarm — roughly 5 times more. So the system calculates the cut-off that minimises actual cost, and lands on 17%, not 50%. That catches 72% of real delays.
flowchart TB
subgraph ingest [" Ingestion "]
BTS[("US DOT BTS<br/>on-time performance<br/>1.66M flights")]
end
subgraph spark [" Distributed processing "]
SP["<b>PySpark feature pipeline</b><br/>temporal · congestion · rolling windows<br/>leakage-safe expanding aggregates<br/>broadcast joins"]
PQ[("Parquet feature store<br/>partitioned by year/month")]
end
subgraph models [" Modelling "]
CLF["<b>Delay classifier</b><br/>LightGBM + isotonic calibration<br/>strict temporal split"]
EVAL["<b>Evaluation framework</b><br/>reliability curves · cost-weighted<br/>threshold · PSI drift"]
REG[("MLflow registry<br/>SQLite backend")]
end
subgraph serve [" Serving "]
TOOLS["<b>Tool layer</b><br/>refuses below 30 flights"]
ROUTER["<b>Zero-token router</b><br/>regex intent matching"]
AGENT["<b>LangGraph agent</b><br/>tool-calling only"]
API["FastAPI"]
DASH["Streamlit dashboard"]
end
BTS --> SP --> PQ
PQ --> CLF --> EVAL --> REG
PQ --> TOOLS
REG --> TOOLS
TOOLS --> ROUTER
ROUTER -.->|"only when rules<br/>cannot answer"| AGENT
AGENT --> TOOLS
ROUTER --> DASH
TOOLS --> API
classDef store fill:#1F3B5C,stroke:#0F2338,color:#fff
classDef proc fill:#E8EEF5,stroke:#1F3B5C,color:#12212F
class BTS,PQ,REG store
class SP,CLF,EVAL,TOOLS,ROUTER,AGENT,API,DASH proc
sequenceDiagram
participant U as User
participant R as Router
participant T as Tool layer
participant M as Model
participant L as LLM
U->>R: "How bad is JFK to LAX?"
R->>R: regex match - route pattern
R->>T: route_performance("JFK","LAX")
T->>T: check 30+ flights on record
T-->>R: {delay_rate: 0.138, flights: 2250}
R-->>U: formatted answer - 0 tokens spent
U->>R: "What about Emirates?"
R->>T: carrier_performance("Emirates")
T-->>R: ToolError - not a US reporting carrier
R-->>U: honest refusal - 0 tokens spent
U->>R: "Compare weekend and weekday patterns"
R->>R: no rule matches
R->>L: fall through to the agent
L->>T: model chooses a tool
T->>M: predict_proba
M-->>T: calibrated probability
T-->>L: tool result - the only source of numbers
L-->>U: phrased answer
Trained on 1,626,052 flights. Split by date, never randomly: train Jan 1 – Mar 6, validate Mar 6 – 19, test Mar 19 – 31.
| Metric | Uncalibrated | Calibrated |
|---|---|---|
| ROC AUC | 0.6723 | 0.6722 |
| Average precision | 0.3454 | 0.3399 |
| Brier score | 0.1505 | 0.1481 |
| Expected calibration error | 0.0400 | 0.0157 |
| Cost-optimal threshold | 0.14 | 0.17 |
| Precision / recall at threshold | 0.267 / 0.719 | 0.257 / 0.766 |
| Layer | Technology | Engineering shown |
|---|---|---|
| Ingestion | Python, requests |
Streaming download, column trimming (250 MB → 28 MB on disk) |
| Features | PySpark (containerised) | Window functions, broadcast joins, partitioned Parquet, leakage-safe expanding aggregates |
| Classifier | LightGBM, scikit-learn | Temporal split, native categorical handling, isotonic calibration |
| Forecasting | statsmodels SARIMA | Weekly seasonality, honest backtest (sMAPE alongside MAPE) |
| Evaluation | custom module | Reliability curves, cost-weighted thresholds, PSI drift |
| Tracking | MLflow | Runs, params, artifacts, model registry |
| Agent | LangGraph | Tool-calling with a hard no-hallucination boundary |
| Cost control | rules engine | Most questions answered at zero tokens |
| Serving | FastAPI, Streamlit | Typed REST, 422 on insufficient data, interactive dashboard |
| Infrastructure | Docker Compose | Reproducible Spark job |
Temporal split, never random. Shuffling flight data randomly puts flights from after the test period into training. Every route and carrier aggregate is also shifted one day back, so a flight's features can never include its own outcome.
Calibration is measured, then corrected. Raw gradient boosting ranks well but its probabilities lie. Since the entire cost analysis depends on 0.30 meaning 30%, isotonic calibration is applied and the improvement is logged as evidence.
The threshold comes from cost. Missing a delay is priced at 5× a false alarm,
so optimal_threshold() minimises expected cost rather than maximising F1.
The cheapest LLM call is the one you never make. A regex intent router answers route lookups, airline comparisons, cause breakdowns and model-health questions at zero API cost. The language model is reached only for genuinely conversational questions.
No Java needed — Spark runs in a container. No Python downgrade — everything else runs on 3.13.
python -m pip install -r requirements.txt -r requirements-dev.txt
# 1. Fetch public flight data (~100 MB, no API key required)
python scripts/download_data.py --months 2024-01 2024-02 2024-03
# 2. Build the feature store (one-shot Spark job)
docker compose run --rm spark
# 3. Train, calibrate and register the model
python -m ml.train_delay_model
# 4. Explore
mlflow ui --backend-store-uri sqlite:///mlflow.db # localhost:5000
uvicorn api.main:app --reload # localhost:8000/docs
streamlit run dashboard/app.py # localhost:8501RUNBOOK.md documents the expected output of every step and how to tell when
something has gone wrong.
- Weather features are not populated. The pipeline joins NOAA observations and the columns exist, but no weather extract has been downloaded yet.
- Three months of data means rolling features are thin at the start of January.
- The 5:1 cost ratio is a working default, not a claim about any airline's real economics.
- SARIMA orders are fixed, not searched per route.
- Delay-duration prediction is not built. The model predicts whether a flight will be late, not how many minutes.
- US domestic flights only. No international carrier appears in this data.
NOTES.md tracks build status and every bug hit along the way.
Ten bugs are documented in NOTES.md. This one is worth reading.
Spark writes its output partitioned by year and month. When pandas reads those
files back, it converts the partition columns to a category type instead of
numbers. So month silently became a categorical feature during training.
At prediction time it was passed as a plain integer, the encoding no longer matched, and every prediction came back identical — an accuracy score of exactly 0.5, with no error message anywhere.
Nothing crashed. Nothing warned. The score landing on exactly 0.5 is the only reason it was caught — had the encoding been nearly right, the model would have quietly degraded and still looked fine.
Fixed by routing training and prediction through one shared loader so the two can never disagree. That class of bug — the kind that produces a plausible wrong answer instead of an error — is the one worth designing against.
Public-domain US Department of Transportation on-time performance data. It records flights, not passengers, so it contains no personal information.
SECURITY.md documents what was checked before publishing and the known
limitations of running this outside localhost.
MIT