What a Music Streaming Company Can Teach Wall Street About Data Lakes

When Spotify engineers talk data, you expect playlists, not limit orders. But their latest open-source project, Random Access Parquet (RAP), is making waves in finance circles.

The problem they tackled is one every trading firm knows: you've got terabytes of historical data in a lake, but your live apps need a single record in milliseconds. The usual fix? Copy that data into a separate database. That costs money, and it gets worse as data piles up.

The Data Lake's Dirty Secret

Data lakes are the go-to for analytics and AI. They hold petabytes in open formats like Parquet and Iceberg. But there's a catch: query engines like Trino and BigQuery are built for scanning huge chunks, not for grabbing one key. Ask them "what's the last trade for XYZ?" and they'll slog through thousands of files, with all the planning and metadata overhead slowing things down.

Spotify says their lake on Google Cloud Storage stores exabytes, while their operational Bigtable holds petabytes. That's a ton of duplication, and the cost of copying between them keeps climbing. In capital markets, it's the same story: tick data, order books, and trade histories sit in lakes, but low-latency services need instant access to individual records.

RAP: An External Index Layer

RAP tackles this by adding an index on top of Parquet files. The index maps query keys—like a user ID or, in our case, a ticker symbol—straight to the file and row offset where that record lives. Instead of scanning everything, a query hits the index first, then does a targeted read on object storage.

The neat part? The underlying Parquet files never change. As new data lands in Iceberg tables, a background builder generates append-only index segments. No rewriting, no copying. The same dataset keeps serving analytics, ML pipelines, and notebooks, while also powering latency-sensitive online apps.

Why Stock Market Firms Should Care

In trading, pulling up a single record without copying data is huge. Imagine a compliance check that needs a specific order's details, or a risk system that must fetch the latest position for one account. These are point queries, and they usually require either a separate database or a painfully slow scan.

With RAP, an exchange or brokerage could keep all historical data in one lake, use the same files for daily analytics, and still serve sub-second point queries for live dashboards or customer apps. No more duplicating into Elasticsearch or Redis just to answer "what's the current bid for AAPL?"

Indexing Strategies for Market Data

Spotify's implementation includes a few tricks to cut storage operations. One is sorting data by the query key—like ordering trades by symbol and timestamp—so accessing a single symbol touches fewer files. Another is co-locating related records, which helps when you're fetching a sequence of trades for the same symbol.

They also interleave value columns so a single contiguous read can grab multiple attributes. For a stock quote, that might be price, volume, and bid-ask spread all in one read. They even support covering indexes, which can answer some queries without touching the Parquet files at all.

These optimizations come at a cost: slightly larger files or indexes. But they can shrink a point query down to a single range read of a few kilobytes.

Secondary Indexes Without Rewriting Files

One feature that stands out for market data is secondary indexes. You might want to query by buyer ID, seller ID, or a composite key like symbol plus order type. RAP lets you add these at the service layer, no rewriting of Parquet files, no changes to your pipeline.

For hash-based lookups, you get exact-match performance. For range queries—say, "show me all trades between 10:00 and 10:05"—a sorted index works better. And with Z-ordering or Hilbert curves, you can improve data locality for those secondary dimensions, making the whole system faster.

The Bigger Picture

Spotify isn't the only one pushing data lakes beyond analytics. Google Cloud recently described an Apache Iceberg-based lakehouse for AI, also aiming to cut data duplication. But RAP's approach is distinct: a dedicated external index for point queries, while staying fully compatible with existing Parquet and Iceberg tables.

The data engineering community has noticed. Andrew Lamb sees RAP as a way to extend open data formats to interactive workloads. Vikas Singh points out that as cloud object storage gets faster, the bottleneck shifts to query planning and metadata access—and RAP tackles that directly by pre-computing indexes.

For the stock market, the implication is simple: if you can serve real-time point queries from your lake without a separate operational store, you save money and reduce complexity. And you keep one source of truth.

What This Means for Trading Infrastructure

Picture a small brokerage that stores all its historical trade data in a lake. Today, to power a customer dashboard showing recent transactions, they'd probably copy that data into PostgreSQL or MongoDB. That's extra infrastructure, extra latency, extra cost.

With something like RAP, the same lake could serve both the analytics team and the live dashboard. The index would let the dashboard fetch a single transaction in milliseconds, while the analytics team continues heavy scans over the full dataset. No duplication, no sync issues.

Granted, RAP is early days and not a full product yet. But the core ideas are solid, and they fit a broader trend of making data lakes more operationally capable.

The Bottom Line

Market data is massive, time-sensitive, and increasingly stored in lakes. Serving low-latency point queries without copying data is a headache many firms face. Spotify's RAP offers a possible fix: an external index that makes Parquet files queryable in real time, while keeping open formats and avoiding duplication.

Whether it's compliance lookups, real-time pricing, or customer-facing portfolio views, fetching a single record fast from your lake could change the game for finance. And it's coming from a music streaming company—proof that great data engineering ideas don't care where they start.