Spotify Listening Analytics

BigQuery · Databricks · PySpark · Delta Lake · Next.js · TypeScript

An end-to-end pipeline over 13,005 listening sessions and 42,809 tracks. Python ingestion lands raw history in BigQuery, a SQL layer cleans it, and 17 Delta tables in Databricks read BigQuery directly through a Unity Catalog foreign catalog via Lakehouse Federation — no data is copied between platforms.

Sessions are modeled with a gaps-and-islands window pattern, which turns a flat stream of play events into bounded listening sessions. Two behavioral measures come out of that. The classification thresholds are set from observed percentile distributions — the first attempt used an arbitrary cutoff and misclassified over 90% of sessions, which is the kind of thing you only catch by checking the output against what you know about your own listening.

Results are pre-computed and exported through the Databricks SQL Statement Execution API to a Next.js dashboard on Vercel. The warehouse never sits in the request path, so a cold or quota-limited cluster can't take the site down.