Stream Processing with Windowed CRDTs
Create decentralized stream processing system that scales globally with Windowed CRDTs
Distributed SystemsStreamingCRDTs
Challenge
Traditional stream processing systems require statically defined coordination, which becomes a bottleneck as data scales. This limits scalability and limits the ability to process data in real-time.
Approach
Creating a decentralized system using Windowed CRDTs as the underlying data structure, utilizing strong eventual consistency to achieve a high throughput and low latency, with fully decentralized recovery.
Impact
Reduced latency by 3.8x and increased throughput by 11x and 5x failure recovery compared to state of the art systems.
Tech Stack
ScalaKafkaKubernetesGoogle Cloud
Metrics
Before:4,890 ms
After:1,280 ms
Improvement: 73.8%