The itch
Heatwaves are one of the deadliest weather events in India, and Uttar Pradesh — a state of over 200 million people — feels them hard every summer. When I joined the Climate Resilient Observatory as a Data Scientist Intern, the question on the table wasn't "can we forecast temperature," it was "can we forecast the specific days temperature crosses into dangerous territory, early enough for someone to actually do something about it." Those are very different problems.
Standard forecasting models are trained and evaluated to minimize average error across all days. That objective quietly optimizes them to be good at predicting normal weather and mediocre at predicting the rare, extreme days — which is exactly backwards from what you want in an early-warning system. A model can be highly accurate overall and still miss the one week a year that actually kills people.
What already existed
Most operational weather forecasting relies on numerical weather prediction (NWP) — physics-based simulations run at national or regional scale. They're good at general forecasts but coarse, and they're not built to specifically flag rare extreme-heat events at a granularity a local government body can act on. On the machine learning side, most public heatwave-prediction work I found treated it as a standard regression problem — predict the temperature, then threshold it — which under-weights exactly the tail events that matter most, because there are so few of them in the training data.
How I actually built it
Before any modeling, I had to trust the data, and government climate sensor data in the wild is messy — gaps, sensor drift, duplicate readings. I built automated SQL-based data quality checks to validate historical climate observations before they ever touched a model, because a heatwave model trained on bad data is worse than useless — it's confidently wrong.
For the forecasting core, I designed a multivariate LSTM pipeline — LSTM specifically because heatwaves aren't a single-variable phenomenon; they emerge from the interaction of temperature, humidity, wind, and pressure trends over multiple preceding days, and LSTMs are built to capture exactly that kind of sequential, multivariate dependency that a simple regression model would flatten away.
Where XGBoost comes in
Raw LSTM output on real government data still carried systematic bias — a consistent under-prediction pattern I found when I validated forecasts against real-time observations. Instead of throwing more layers at the LSTM, I trained an XGBoost model specifically on the LSTM's residuals — the errors, not the raw temperatures — to learn and correct that systematic bias. That two-stage approach (sequence model first, error-correction model second) cut the systematic error down and made the final forecasts meaningfully more trustworthy against live data, without needing a bigger or more opaque single model.
The numbers
- 0.82 recall on extreme heat event detection — explicitly optimized to catch the rare, dangerous days, not just perform well on average
- Multivariate LSTM pipeline running on live CRO climate monitoring data across Uttar Pradesh
- XGBoost residual correction layered on top, validated against real-time government data to reduce systematic error
- Automated SQL data quality checks on historical climate observations before modeling
Why it matters
A forecasting model that lives in a notebook doesn't save anyone. Part of this work was turning model output into structured reports that non-technical stakeholders — the people who actually issue heat advisories — could read and act on quickly. Optimizing for recall on the days that matter, and being honest about the pipeline's error and how it was corrected, was the whole point.