How to Build a Reliable Kafka Data Processing Pipeline, Focusing on Contention, Uptime and Latency

  Переглядів 1,704

InfoQ

InfoQ

3 місяці тому

InfoQ Dev Summit Boston, a two-day conference of actionable advice from senior software developers hosted by InfoQ, will take place on June 24-25, 2024 Boston, Massachusetts.
Deep-dive into 20+ talks from senior software developers over 2 days with parallel breakout sessions. Clarify your immediate dev priorities and get practical advice to make development decisions easier and less risky.
Register now: bit.ly/47tNEWv
---------------------------------------------------------------------------------------------------
Video with transcript included on InfoQ: bit.ly/496fhXv
Lily Mara shares how OneSignal improved the performance and maintainability of its highest-throughput HTTP endpoints (backed by a Kafka consumer in Rust) by making it an asynchronous system. She shares how metrics changed when the system went from sync to async, and what unique sharding strategies were used to maximize concurrency and performance, while maintaining consistency for Kafka consumers.
#ApacheKafka #Performance #Reliability
------------------------------------------------------------------------------------------------------
Follow InfoQ:
- Mastodon: techhub.social/@infoq
- Twitter: / infoq
- LinkedIn: / infoq
- Facebook: / infoqdotcom
- Instagram: @infoqdotcom

КОМЕНТАРІ: 2
@andmal8
@andmal8 2 місяці тому
Very nice presentation!
@BR-lx7py
@BR-lx7py 19 днів тому
I guess this is a long way of saying that Kafka is not really a message queue
Reliable Architectures through Observability
49:00
InfoQ
Переглядів 2 тис.
How Netflix Ensures Highly-Reliable Online Stateful Systems
49:32
😨Новая Война в GTA 5 Online #shorts
00:40
King Dm
Переглядів 1,7 млн
#1 How to Build a Virtual Machine from Scratch
31:09
Philip Bohun
Переглядів 58 тис.
Apache Kafka in 6 minutes
6:48
James Cutajar
Переглядів 925 тис.
Managing 238M Memberships at Netflix
50:05
InfoQ
Переглядів 2,8 тис.
System Design: Why is Kafka fast?
5:02
ByteByteGo
Переглядів 1 млн
The BEST library for building Data Pipelines...
11:32
Rob Mulla
Переглядів 66 тис.
Amazon DynamoDB Distributed Transactions at Scale
49:08
InfoQ
Переглядів 1,4 тис.
The Rise of the Serverless Data Architectures
46:58
InfoQ
Переглядів 1,5 тис.
The PA042 SAMSUNG S24 Ultra phone cage turns your phone into a pro camera!
0:24
Which Phone Unlock Code Will You Choose? 🤔️
0:12
Game9bit
Переглядів 6 млн
Нужно ли чистить ПК от пыли?
0:59
CompShop Shorts
Переглядів 168 тис.