How to Build a Reliable Kafka Data Processing Pipeline, Focusing on Contention, Uptime and Latency

  Переглядів 1,681

InfoQ

InfoQ

2 місяці тому

Want to attend #Qcon or #DevSummit for free? Here's your chance!
Write an article for #InfoQ and you could win a ticket! Show off your expertise and contribute to the tech community.
Join the contest now: www.infoq.com/articles/infoq-...
💪 Knowledge is power!
#ArticleContest #TechArticles #WinATicket
---------------------------------------------------------------------------------------------------
Video with transcript included on InfoQ: bit.ly/496fhXv
Lily Mara shares how OneSignal improved the performance and maintainability of its highest-throughput HTTP endpoints (backed by a Kafka consumer in Rust) by making it an asynchronous system. She shares how metrics changed when the system went from sync to async, and what unique sharding strategies were used to maximize concurrency and performance, while maintaining consistency for Kafka consumers.
#ApacheKafka #Performance #Reliability
------------------------------------------------------------------------------------------------------
Follow InfoQ:
- Mastodon: techhub.social/@infoq
- Twitter: / infoq
- LinkedIn: / infoq
- Facebook: / infoqdotcom
- Instagram: @infoqdotcom

КОМЕНТАРІ: 2
@andmal8
@andmal8 2 місяці тому
Very nice presentation!
@BR-lx7py
@BR-lx7py 13 днів тому
I guess this is a long way of saying that Kafka is not really a message queue
Reliable Architectures through Observability
49:00
InfoQ
Переглядів 2 тис.
The BEST library for building Data Pipelines...
11:32
Rob Mulla
Переглядів 66 тис.
How Netflix Really Uses Java
50:31
InfoQ
Переглядів 19 тис.
#1 How to Build a Virtual Machine from Scratch
31:09
Philip Bohun
Переглядів 57 тис.
They Enabled Postgres Partitioning and their Backend fell apart
31:52
Hussein Nasser
Переглядів 36 тис.
Apache Kafka in 6 minutes
6:48
James Cutajar
Переглядів 921 тис.
Amazon DynamoDB Distributed Transactions at Scale
49:08
InfoQ
Переглядів 1,4 тис.
Design a High-Throughput Logging System | System Design
8:23
Interview Pen
Переглядів 39 тис.
Why flat earthers scare me
8:05
Sabine Hossenfelder
Переглядів 223 тис.
System Design: Why is Kafka fast?
5:02
ByteByteGo
Переглядів 1 млн
Как открыть дверь в Jaecoo J8? Удобно?🤔😊
0:27
Суворкин Сергей
Переглядів 970 тис.
Как открыть дверь в Jaecoo J8? Удобно?🤔😊
0:27
Суворкин Сергей
Переглядів 970 тис.
КУПИЛ SAMSUNG GALAXY S24 ULTRA ЗА 88000 РУБЛЕЙ!
27:29
DimaViper
Переглядів 66 тис.