Stream processing and windowing

Batch jobs run over data that's already been collected. Stream processing works on data as it comes in, so you get answers in seconds and not hours. Tools like Flink, Spark Structured Streaming and Kafka Streams take care of a lot of the painful parts for you.

A stream never ends, so if you want to count something, you have to chop it into windows. The simplest kind is a tumbling window, where each window is the same size and they don't overlap.

There are also two kinds of time to keep straight. Event time is when something happened. Processing time is when your system found out. Phones lose signal and networks get slow, so events can show up late or out of order. Stream engines use something called watermarks to guess when a window's probably done.

• Ask how fresh the data needs to be before you jump to streaming.
• Say what kind of window you'd use and how big.
• Talk about what happens with late events, and when a worker restarts.

Counting rides per city, per minute

A dashboard shows how many rides started in each city in each minute. About 50,000 ride events come in a second, worldwide, and the numbers should show up within 10 seconds.

1. Events go into Kafka, keyed by city, so every event for one city lands on the same partition and the same worker.
2. Each worker keeps a running count in memory for every (city, minute) pair it's seeing. That's tiny: a few thousand cities times a couple of open minutes.
3. Windows are by event time, the time on the phone when the ride started, not when the event arrived.
4. A phone in a tunnel sends its event 40 seconds late. The watermark says 'I've probably seen everything up to 30 seconds ago', so the 12:01 window is still open when it arrives and the count is right.
5. When the watermark passes the end of a minute, the worker writes the final count to a database the dashboard reads from, then forgets that window.
6. An event that shows up 10 minutes late is past the allowed lateness. You can drop it, or send it to a side output and fix the count in a nightly batch job.
7. Every few seconds the engine saves a checkpoint of the counts and the Kafka offsets together. If a worker dies, a new one restores the checkpoint and replays from those offsets, so nothing is counted twice.

Things interviewers tend to ask once you've brought this up.

What if one city gets way more events than the others?

That partition becomes the bottleneck. Split the hot key, for example by city plus a random suffix 0 to 9, count each piece separately, then add the ten counts together in a second step.

How long should you wait for late events?

It's a trade between freshness and accuracy. Look at how late events actually arrive. If 99.9% show up within 30 seconds, wait 30 seconds and handle the rest separately.

Why not just run a batch job every minute?

You can, and it's simpler. It struggles when the data is big, because each run has to scan everything again, and when you need answers in seconds. Lots of systems use both: streaming for fast numbers, batch for the final numbers.

Where does the state live?

In the worker, usually in memory or a local store like RocksDB, saved to durable storage in checkpoints. That's what lets the engine restart without losing counts.

• Using processing time when the question is about when things happened.
• Not saying what happens to late events.
• Writing every single event to a database instead of aggregating first.
• Forgetting that a restarted worker can double count unless state and offsets are saved together.
• Reaching for streaming when a 'refresh every hour' batch job is all anyone needs.

• Message queues and event streamsKeep services from depending on each other, and handle traffic spikes.
• Caching strategiesMake reads faster, and figure out what to do when data changes.
• Geospatial indexingFind what's nearby with map grids like geohashes and H3.