Event-Driven Architectures Done Right, Apache Kafka • Tim Berglund • Devoxx Poland 2021

  Переглядів 179,650

Devoxx Poland

Devoxx Poland

День тому

Subscribe to our channel: youtube.pl/c/DevoxxPoland?sub...
Far from a controversial choice, Kafka is now a technology developers and architects are adopting with enthusiasm. And it’s often not just a good choice, but a technology enabling meaningful improvements in complex, evolvable systems that need to respond to the world in real time. But surely it's possible to do wrong! In this talk, we'll look at common mistakes in event-driven systems built on top of Kafka:
-Deploying Kafka when an event-driven architecture is not the best choice.
-Ignoring schema management. Events are the APIs of event-driven systems!
-Writing bespoke consumers when stream processing is a better fit.
-Using stream processing when you really need a database.
-Trivializing the task of elastic scaling in all parts of the system.
It's highly likely for medium- and large-scale systems that an event-first perspective is the most helpful one to take, but it's early days, and it's still possible to get this wrong. Come to this talk for a survey of mistakes not to make.
Tim Berglund is a teacher, author, and technology leader with Confluent, where he serves as the Senior Director of Developer Advocacy. He can frequently be found at speaking at conferences in the United States and all over the world. He is the co-presenter of various training videos on topics ranging from Git to Distributed Systems to Apache Kafka. He tweets as @tlberglund, blogs very occasionally at timberglund.com, and lives in Littleton, CO, USA with his wife, their three children having grown up.
Topics covered:
-What is Event-Driven Architecture
-Data Mesh Principles
-Scaling
-State management
Recorded at Devoxx Poland 2021
Twitter: / devoxxpl
Instagram: / devoxxpl
Join us also here:
Devflix: devflix.pl
#Devoxx #DevoxxPoland #IT #Development #SoftwareDevelopment

КОМЕНТАРІ: 76
@alexanderpodkopaev6691
@alexanderpodkopaev6691 Рік тому
"If it is small program, you don't call it a monolith. Architecturally it IS one. I want to give you permission just embrace that" - BRILLIANT! Noted for further discussions with middle developers, thanks a lot Tim!
@DevoxxPoland
@DevoxxPoland Рік тому
Good catch 👍
@atkinpaul
@atkinpaul Рік тому
Thanks! I hope you do engage more with Data Mesh. Zhamak seems to de-prioritize operational data flows but there is a massive opportunity to converge operational and analytic pipelines when you use an event driven architecture combined with polyglot persistence.
@ChandraShekhar-by3cd
@ChandraShekhar-by3cd Рік тому
Thanks for the Amazing talk.
@darrenhorwitz1860
@darrenhorwitz1860 Рік тому
this guy is brilliant, I love his sense of humor and what a great talk !!!
@marekchodak6705
@marekchodak6705 Рік тому
I usually watch on 1.5x speed but he is so good to listen i kinda dont want to
@DevoxxPoland
@DevoxxPoland Рік тому
Wow. One of the best compliment that I saw. Thanks a lot!!!
@arielguzman2875
@arielguzman2875 2 місяці тому
Or you miss most of it at that speed 😅
@KirBirger
@KirBirger 6 місяців тому
Thanks for this. There's a shortage of honest content out there that doesn't simply evangelize concepts and approaches.
@mosespeter9711
@mosespeter9711 Рік тому
This is one of the best talk I've ever listened to
@DevoxxPoland
@DevoxxPoland Рік тому
Tim at his best
@AlexanderSergeenko
@AlexanderSergeenko Рік тому
Thank you for the great talk!
@DevoxxPoland
@DevoxxPoland Рік тому
Our pleasure!
@joshzwicker
@joshzwicker Рік тому
Asked event-driven-architecture into your heart 😄👍nice metaphor
@DevoxxPoland
@DevoxxPoland Рік тому
😁
@tanertasim3637
@tanertasim3637 Місяць тому
This presentation is just gold!! Very clean and precise explanations! Thanks!
@mehrdadk.6816
@mehrdadk.6816 Рік тому
The reason for developers think when using event-driven is a database replacement can be , there is no example that uses the combination in Confluent materials. Maybe this can be getting done
@TheRedbeardster
@TheRedbeardster 3 місяці тому
T-shirt really rocks!
@user-sg5qk6wy3p
@user-sg5qk6wy3p 6 місяців тому
Given the event log is the system of record, and lets say we want the customer's address updated in the user service. Does that mean we have to fetch the current version of the user from the event log? Seems challenging and requires specific technologies. Alternatively you'll use the DB in the user service. By doing that, the DB has effectively become the system of record, right?
@3enny3oy
@3enny3oy Рік тому
The last 15 minutes of 2001 a Space Odyssey 😂 I’d say that’s where all the meaning of the movie is. And yeah, don’t be afraid of a monolith if you’re building something small and isolated. If you can get something up and running to do the job with little up front effort or admin overhead, it’s probably fine to spend the extra effort to migrate to an event driven architecture in the future once you’re actually generating value. If you over engineer up front it will be far longer before you start to generate business value and it will probably cost more in the long run. Build fast, fail fast and iterate. Just avoid decisions which will lock you down and make iteration difficult. Be pragmatic and agile.
@nullah100
@nullah100 Рік тому
Nice and Informative
@DevoxxPoland
@DevoxxPoland Рік тому
Thanks
@flynnblu6992
@flynnblu6992 11 місяців тому
You had me at persnickety.
@shashanksharma8254
@shashanksharma8254 11 днів тому
This is how to make complex things Simple enough.
@diegorosadossantos8493
@diegorosadossantos8493 Рік тому
Thanks for sharing your knowledge 😃
@DevoxxPoland
@DevoxxPoland Рік тому
Our pleasure...
@vanivari359
@vanivari359 Рік тому
Don't want to complain too much, but from an expert like him i kinda expected a bit more than: "only use EDA if you need it", "Use DBs if they are a good idea", "the event schema is kinda important because stuff changes" and "stuff is hard, don't implement your own scaling or state management". I was already suspicious after the claim that there are 5 ways to fail with EDA because i've seen at least 10. And the 5 in the talk are for sure valid points, but not the stuff (except scaling) projects typically struggle with. The real challenge is stuff like eventual consistency and the lack of transactions - issues which could have been discussed pretty well with that reference architecture. But hey, now i know that he wanted to buy tailored shirts once... but didn't. ;)
@KirBirger
@KirBirger 6 місяців тому
I'm not sure what you want here. The guy broke down some concepts, and explained the pros and cons of some approaches. It's up to you to look critically at your use case and determine if you "need" it.
@samanthamccarthy9765
@samanthamccarthy9765 4 місяці тому
exactly what do you want here .
@benisrood
@benisrood 2 місяці тому
​​​@@samanthamccarthy9765 Hi Samantha, I am guessing that the OP wanted Tim to forgo the digressions-and digressions upon digressions! 😂-and associations. Instead Tim could have just methodically iterated through the various examples and cases he wanted to cover. Not Tim's style, I grant you, but it was more than a little unnecessary this time, and makes it harder to follow the points he is making and wanting to connect together. Not a bad talk, and absolutely a necessary talk, but I can also imagine that the next time Tim gives this same presentation he will probably refine the way he gives it. Maybe it was also situational, perhaps the Poles were a tough crowd 😂 and Tim was nervous!
@smaug9833
@smaug9833 Рік тому
Another pitfall of event driven architecture is that one service derives the whole context from the message it consumes. But sometimes, there is a business need for a service to maintain its own context in addition to what it reads from the topics. I know this is not the way to go for purely event driven services, but sometimes you are forced into a situation where you have to do this.
@alexanderpodkopaev6691
@alexanderpodkopaev6691 Рік тому
Yeah, something like this one: A user payed for an order, then decided to change shipping address and placed another order. So, where 1st order will be shipped in this system? Because shipping service maintains its user DB , I assume shipping address is not included into order or payment event. With such design, I guess chances that address change event will come and would be processed BEFORE order payment for order#1 event would be high for any reasonably loaded site causing shipping to wrong address. During promo or BlackFriday - it's almost guaranteed.
@cxmais
@cxmais Рік тому
do things right, don’t do things wrong
@thepracticaldev
@thepracticaldev Рік тому
Nice intro!
@DevoxxPoland
@DevoxxPoland Рік тому
Glad you liked it
@RaviChandraEnaganti
@RaviChandraEnaganti Рік тому
Isn't storing/maintaining user data within the Shipping service an anti pattern? Each micro service is supposed to handle one domain?
@immrsv
@immrsv Рік тому
IMHO, the shipping service isn't 'maintaining' the data so much as it's just keeping a cached view of the data. All the rules around the handling of User data are still kept within the User service.
@kitkarson4226
@kitkarson4226 Рік тому
It is called materialized view . It is not an anti pattern
@user-dq7vo6dy7g
@user-dq7vo6dy7g 5 місяців тому
The materialized view thing is obviously bullshit. I think the point it to avoid coupling between shipping service and user service. If the user service is down for example, you could not ship orders because you cannot get hold of the addresses. Or if the interface of the user service changes, you might have to change the shipping service in tandem. And as @immrsv writes the shipping service still doesn't handle the user domain, he just saves the data that is relevant for it's own domain. I think it's also reasonable to allow those cross-service calls. You just need to consider the pros and cons of either. A EDA purist would probably put it all on the event bus.
@juilipanse-kanade9583
@juilipanse-kanade9583 Рік тому
thanks for sharing!
@DevoxxPoland
@DevoxxPoland Рік тому
My pleasure!
@debugin1227
@debugin1227 Рік тому
How do you handle data security in this scenario where a one business unit should not be able to see / consume the data of another business unit in a company? Some users might be able to see qty/price in their plant, other users maybe qty only.. how does this work in these events?
@dinoscheidt
@dinoscheidt Рік тому
You can use Crypto Shredding (field level encryption). Can / is also be used for personal identifiable data. Meaning: Users that do not have access to a key due to their role, don’t get the data. Thats what key vaults are useful for.
@joelkorpela2706
@joelkorpela2706 Рік тому
@@dinoscheidt That's a fascinating idea for data access control when multiple services consume the same event. Thanks!
@rdean150
@rdean150 Рік тому
If the universe of consumers/permission permutations is finite and manageable, the approach I've seen taken is for the owners of the original topic to create consumers of their own topic that then publish filtered subsets of the messages to new topics. The downstream consumers would then listen to the filtered topic instead of the original topic. This can potentially be configured dynamically. This is surely not the best solution, and may be difficult to scale, but for relatively limited use-cases, it gets the job done.
@BosonCollider
@BosonCollider 11 місяців тому
Just a note that Something can be logically event-driven while being a monolith from a devops perspective. For example, many programs in Go, Elixir, or using Streams in Python anyio/trio are event/message queue driven. They just happen all to live in the same runtime and don't require Kafka to act as a message bus. Imho using language native multitasking in this way is a good way to build a system that can be refactored into distributed systems. And especially for the case of Elixir, the BEAM is just so reliable that you will probably not need Kubernetes or anything fancy to keep it going. Just put it in a freebsd or Rocky/RHEL VM with a managed database and you will have very good uptime.
@yangyun6221
@yangyun6221 Рік тому
Same in Mobile App Developing and small companies. Every people say use MVVM, but your code has only 10 views. And small companies pay lots of money buying managing SAAS tools while they only have 5 people.
@DevoxxPoland
@DevoxxPoland Рік тому
Thanks.
@gerritlikestohike
@gerritlikestohike Рік тому
A nice talk but I have to admit the takeaways are very little here. Real pitfalls which you need to takle when writing a real QRS or DDD event drive architecture have not really been adressed. So if any1 is looking for answers to those you don`t have to watch this.
@DevoxxPoland
@DevoxxPoland Рік тому
Thanks for your feedback.
@sarwanhakm7517
@sarwanhakm7517 2 місяці тому
the real pitfalls as he mentioned here are I think scaling and state management which is hard to figure out on your own and hence he recommends tools to be uses.
@petersuvara
@petersuvara Рік тому
Otherwise known as GOTO hell :)
@vernetto
@vernetto Рік тому
I was really interested in Kafka, but I will look for some other video where the speaker goes straight to the point without a lot of blabla
@DevoxxPoland
@DevoxxPoland Рік тому
Sometimes introduction is necessary for better understanding and explanation
@mugishaalainchristian2613
@mugishaalainchristian2613 11 місяців тому
This man is hilarious haha :)
@lhxperimental
@lhxperimental Рік тому
Good talk for some reason I was reminded of Louis CK as I watched.
@ibgib
@ibgib Рік тому
I've enjoyed the first 30 min, and I always jump the gun on these comments (am I the only one? )...but at a 29:50 you mention more evolutionary architecture. Well I've been developing a web3 protocol since before bitcoin that came from the event sourcing side of things, with the idea that you can apply transforms as data ("event" analog) that live on the same Merkle DAG timelines to produce more "living" data. Think git applying semantic diffs (but not text-centric) to create/recreate branches per use case, as opposed to applying events strictly to rehydrate aggregate state. I see that the speaker is a git guru, and I actually reimplemented much of their internal DAG storage mechanism (even completely coincidentally using the ^ for a delimiter). Anyway it's hard to express to computer people because most are already enamored with bitcoin or existing event sourcing paradigms, but both share the same difficulty when it comes to scale: that a "single source of truth" is a very funny thing, and that we can produce more biological (evolutionary) code which even unifies microservices with the future of ANNs/RNNs/Transformers/IoT, and allows for sovereign node implementations (like the speaker's continual reminder that RDbs are good). It's just really good at unifying/simplifying what kubernetes and other orchestration systems do, with the ability to have consensus structure/specs data "on chain" and able to be shared alongside the data itself. Anyway, it's hard to find people who know ES and git internals extensively, and if you're interested you can look at ibgib on Github (prototype) & GitLab (low level DAG graphing lib). Also I have a few (crappy!) YT videos. Perhaps I could explain the evolution from ES design more in depth if you'd care to connect. Thanks for the video! (And getting this far in this comment hah)
@isorry123_
@isorry123_ Рік тому
i also around ~22m was thinking why not just replace database and notification system with a blockchain ledger. write and read directly to it.
@ibgib
@ibgib Рік тому
@@isorry123_ yes exactly. But the devil's in the details of how you decide to allow for performance/trust/availability/etc tradeoffs. For example, a blockchain with a hard-coded consensus algorithm like bitcoin has only one "shape" that it can take WRT how it performs & where it centralizes. Ethereum, both pre- and post-Merge, will also be a one size fits all, and this will always generate lock-in with e.g. requiring solidity to enforce their beliefs on how the math guarantees their level of trust. The same goes for every blockchain-based approach that I've looked at, though if you take the entire blockchain ecosystem as a whole, you get a wide range of dynamics. My approach literally uses the biological paradigm by enabling version control-like dynamics - diffing + DAG like in git, but with the diff transforms themselves as metadata as well. These transforms are not text-diffs but rather semantic diffs, each transform itself being a node in the DAG. So in the event sourcing world, this is like having a redundancy of storing both the events and the hydrated state, which sounds like a waste. But if you use them in context of version control, where you apply the same transforms to **different** branches, you come up with different results (something not wanted in event sourcing). So it ends up looking like biological organisms, having the transform dna "genotypes" and the expressed "phenotypes" of the state when actually being consumed in an app. Interestingly, this has many positive after effects like idempotent CRDT behavior. So when I go to "synchronize" two spaces (merging multiple timeline branches), I can simply look at which transforms have been applied...though only when order is not important. (Ordering implies that you should have a localized space or the standard barrier/critical section/etc parallel constructs. But I digress... My point is that this enables consensus specification to be "on chain" in a consistent manner, whereas with all approaches I've seen host their code on github with no plans to move. (Mine also is on git because I don't have time to implement the version control cli while also doing the entire rest of the shiny UX stuff that I suck at) But the point is that git itself is a blockchain (DAG) under the hood, so right off you're duplicating code, increasing surface area, and decreasing flexibility and agility because you have to edit the code in git as opposed to the code already existing in a granular fashion in its own "version control". This flexibility is absolutely necessary to actually implement the unification that you're talking about. There are a couple projects that are somewhat close (IPFS/IPLD/etc and w3c's rdf triples addressing protocols which have changed over the years), but I don't see them getting the whole picture.
@LarsRyeJeppesen
@LarsRyeJeppesen Рік тому
Lol I knew he would say ,"React"
@samlee4490
@samlee4490 27 днів тому
tough crowd
@sergeykichuk2586
@sergeykichuk2586 6 місяців тому
What you described it is not Event Driven Architecture! Correct me if I'm wrong but what you described here as reference architecture with event logs and database as views is more like Event sourcing and CQRS where source of true is your events!
@sarwanhakm7517
@sarwanhakm7517 2 місяці тому
@sergeykichuk2586 this is the bigger picture described as Event-Driven Architecture, ES and CQRS are definitely in play here but he simply avoids explaining them. ES here is when materializing/projecting user data into shipping and CQRS is obvious, the requests.
@sergey5551
@sergey5551 Рік тому
if we will skip "water" in the speech, video will be way shorter......
@VasuNori1
@VasuNori1 2 місяці тому
this talk can be condensed to 10 min. too much random stuff.. isn't the audience mostly engineers? or a bunch of know-nothing newbies?
@mjj0829
@mjj0829 Рік тому
I think he knows a lot and the agenda was appropriate but man he talks a lot. What he covered in this 50m can be said in 20 or less. I wish he actually put more depth into it rather than small sideway talks. I watched it at 1.75x and skipped a few times.
@a_guy_called_Jerry
@a_guy_called_Jerry Місяць тому
hard audience it is
@bernaridho
@bernaridho Рік тому
The word ARCHITECTURE is overused. Too many things are named with Architecture.
@rdean150
@rdean150 Рік тому
Perhaps, but in this instance it is definitely appropriate. This topic is about a complex arrangement of interactions between independent processes running on separate hardware but working collectively in a concerted fashion to achieve a task. The roles of the various components, the manner in which they interact and contracts they hold between each other, and the spectrum of implications that the particular arrangement results in - this can aptly be described as the system's architecture.
@DevoxxPoland
@DevoxxPoland Рік тому
Good point
@jopsuey
@jopsuey Рік тому
Ultra bored
@ftnsco
@ftnsco Рік тому
Omg 🥱🥱🥱 this video has many annoying points
@DevoxxPoland
@DevoxxPoland Рік тому
What do you mean by annoying?
@calebvear7381
@calebvear7381 Рік тому
@@DevoxxPoland I think zero is referring to some of the tangents that came up. Eg he is about to tell us something and then decided to explain his tshirt instead.
@xamax4
@xamax4 5 місяців тому
@@calebvear7381 breaking the line of thoughts
@JosiahWarren
@JosiahWarren 3 місяці тому
I was so tired throughout the presentattion and learned nothing new. Congrats🫤
Event-Driven Architecture (EDA) vs Request/Response (RR)
12:00
Confluent
Переглядів 20 тис.
ФОКУС С ЧИПСАМИ (секрет)
00:44
Masomka
Переглядів 2,6 млн
3. Apache Kafka Fundamentals | Apache Kafka Fundamentals
24:14
Confluent
Переглядів 430 тис.
Flink vs Kafka Streams/ksqlDB: Comparing Stream Processing Tools
55:56
The Many Meanings of Event-Driven Architecture • Martin Fowler • GOTO 2017
50:06
Про Kafka (основы)
49:23
Владимир Богдановский
Переглядів 338 тис.
Event-Driven Architecture: Explained in 7 Minutes!
7:18
Alex Hyett
Переглядів 67 тис.
What is Apache Kafka®?
11:42
Confluent
Переглядів 328 тис.
ИГРОВОЙ ПК c WILDBERRIES за 40 тысяч рублей
30:17
Ремонтяш
Переглядів 477 тис.
Лучший телефон на андроиде?
0:25
Опросный
Переглядів 62 тис.