← Back to home

Streaming prototype

Real-Time Crypto Arbitrage Prototype

A streaming prototype for studying market-data ingestion, window alignment, freshness, alerting, and observability behavior.

Kafka · Flink · WebSockets · Docker 2025 GitHub repo ↗

SUMMARY

Coinbase and Binance WebSocket feeds aligned through Kafka and Flink windows, then routed into alerts and observability.

Problem

Market-data prototypes need to handle bursty streams, windowed computation, stale prices, and alert paths without pretending latency is free.

What I built

I built a streaming path from exchange WebSockets into Kafka, Flink tumbling windows, an alerter, and monitoring surfaces.

Architecture / system design

The architecture separates ingestion, durable stream transport, stream processing, alerting, and observability so each failure mode has a place to surface.

Failure modes / what broke

The likely breaks are stale data, exchange disconnects, duplicate events, and windows that report opportunities too late to be meaningful.

Proof / metrics / tests

The original section is kept as a prototype case study; it does not claim production trading performance.

Lessons learned

Streaming systems are state systems. The hard part is less the formula and more the freshness, ordering, and observability around it.

field notes

Expanded field notes

THE STREAMING LESSON

A streaming-systems prototype: exchange APIs, message bus, windowed computation, alerter, dashboard - four decoupled services so each piece can fail or restart on its own. The point isn't to trade money; it's to practice the alignment problem at the heart of every multi-source streaming pipeline.

A streaming pipeline that watches BTC trade on two exchanges at once and spots the moments they disagree. WebSocket feeds from Coinbase and Binance land in Kafka; Flink tumbling windows align the two streams in time and compute the spread; an alerter pushes live notifications to Discord while a Streamlit dashboard charts spreads and alert history.

The system is decomposed into four services — producer, stream processor, alerter, dashboard — each independently deployable and orchestrated with Docker Compose, so any piece can fail or restart without taking down the pipeline.

DESIGN NOTE

Two exchanges tick at different rates, so comparing "latest price vs latest price" lies to you. Tumbling windows force both streams onto the same clock before the spread is computed — the alignment, not the math, is the hard part.

Arbitrage pipeline architecture coinbase ws binance ws kafka flink tumbling windows alerter dashboard discord spread > threshold four services · docker compose
fig. 4 — two exchange feeds aligned in Flink windows; alerts fan out to Discord and the live dashboard.

hover or tab through any node to trace the flow

contact

Open to backend systems, AI infrastructure, and product engineering roles.