Flight Delay Predictor — Pipeline Overview

1. Problem & Data

Binary classification: predict whether a US domestic flight departs 15+ minutes late (DEP_DEL15), using the Kaggle dataset 2019 Airline Delays and Cancellations (~3M+ flight records).

XGBoost

Binary Classification

Class imbalance ~81/19

2. Feature Engineering

Historical delay averages (carrier, airport, day-of-week, time-block) were self-computed from the training data rather than using the dataset's built-in historical columns, which risk leakage from future information.

Categorical variables were encoded via .cat.codes rather than one-hot encoding — appropriate for a tree-based model that handles high-cardinality categories natively.

Class imbalance (~19% delayed) was addressed via scale_pos_weight, correcting the loss function's bias toward the majority class without resampling.

3. The Feature-Scoping Decision (v1 → v2)

v1 used 21 features, including PLANE_AGE, CONCURRENT_FLIGHTS, monthly flight-volume counts, and PREVIOUS_AIRPORT. Once building the live API, the question became: could a real user actually provide these? No — they require aircraft-tracking infrastructure or data no free API exposes.

    Rather than silently imputing placeholder values, the model was retrained on 15 deployable features. Performance barely moved, evidence those 6 features carried little marginal signal beyond the historical priors already present.
  

	v1 (21 features)	v2 (15 features)
Recall (delayed)	0.64	0.65
Precision (delayed)	0.31	0.33

4. Interpretability & Error Analysis (SHAP)

SHAP (TreeExplainer) identified DEP_TIME_BLK as the most decisive feature by a wide margin — consistent with real airline operations, where delays cascade through the day as aircraft and crew schedules compound.

Manual error analysis on the confusion matrix found a concrete pattern:

False positives cluster in evening blocks with high historical delay rates and worse-than-average weather — the model over-triggers on evenings that run fine anyway.
False negatives cluster in morning blocks with low historical rates and good weather — delays here come from causes with no signal in the data (mechanical, staffing, one-offs).

    Diagnosis: the model leans on time-of-day as a blunt proxy instead of distinguishing "evening, clear skies" from "evening, storming." This became the basis for the v3 experiments below.
  

5. v3 Experiments — Testing the Diagnosis

Three concrete fixes were tested against the time-of-day over-reliance found above, rather than left as untested theory:

Explicit interaction feature — added an "evening AND meaningful rain/snow" flag so the model wouldn't need to infer the interaction on its own. Result: precision and recall were statistically unchanged.
Dropped the redundant time-block prior — removed timeblock_avg_delay, which is derived directly from DEP_TIME_BLK, to test whether the model was "double-counting" it. Result: also unchanged.
Threshold tuning — plotted a full precision-recall curve rather than trusting the default 0.5 cutoff.

Finding: both feature-engineering fixes tested clean and negative — tree-based models like XGBoost already discover feature interactions on their own, so handing one to it explicitly added no new information. This is a real result: it shows the ~0.33 precision ceiling is closer to a data limitation (no signal for mechanical/staffing/one-off delays) than a fixable feature-engineering gap.

Threshold tuning, by contrast, produced a real, usable gain with zero retraining risk:

Threshold	Precision	Recall
0.50 (previous default)	0.33	0.65
0.60 (deployed)	0.40	0.40
0.65	0.45	0.28

0.60 was chosen as the operating point: a meaningful precision gain (fewer false alarms) while recall stays usable, rather than collapsing at higher thresholds.

6. Live API & Enrichment

The API (FastAPI) only asks users for what they'd actually know: month, day of week, departure time block, distance group, carrier, and departing airport.

Everything else is fetched server-side: live weather from Open-Meteo (via a 96-airport coordinate table), and precomputed historical priors looked up from reference tables. Unknown carriers or airports are explicitly rejected rather than silently mispredicted.

7. MLOps Stack

Component	Tool	Purpose
Experiment tracking	MLflow (SQLite backend)	Logs params, metrics, and model artifacts per run for comparison across versions
Interpretability	SHAP (TreeExplainer)	Global feature importance and the basis for the v3 error-analysis experiments
Testing	Pytest + FastAPI TestClient	Health check, valid predictions, rejection of unknown categories
CI/CD	GitHub Actions	Runs tests on every push; deploys to the VM via SSH on success
Containerization	Docker + Docker Compose	Reproducible environment, identical locally and in deployment
Deployment	Azure VM (Ubuntu, Docker)	Publicly reachable live API, manually started/stopped to manage student credit

8. Known Limitations (documented honestly)

96-airport scope: the model only predicts for airports present in the 2019 training data — a structural limit of categorical encoding, not fixable by fetching live data for new airports.
Precision (0.40 at the deployed threshold): tested against two feature-engineering fixes that did not move it, suggesting the remaining gap is closer to a data limitation than a modeling one.
Weather fallback: if Open-Meteo is unreachable, the API falls back to neutral default weather values rather than failing the request.
Deployment uptime: hosted on a manually-managed VM (Azure Student credits), not an always-on production service.
US-only: scoped to US domestic flights due to no accessible bulk historical dataset for other regions at the time of writing.

Tech Stack

Python

XGBoost

Pandas

FastAPI

Pydantic

SHAP

MLflow

Pytest

Docker

GitHub Actions

Azure VM

Open-Meteo API

Chart.js

How it was built