Data Consistency in Microservices Architecture (Grygoriy Gonchar)

  Переглядів 112,871

Devoxx

Devoxx

День тому

While we go with microservices we bring one of the consequence which is using multiple datastores. With single data source, it’s way much easier to guaranty that given entity such as customer, order or payment will have the same state across the system. The challenges arise when the related data is spread across multiple microservices. This talk represents my architecture experience gathered in three organizations from different industries on making data consistent across multiple microservices. The solutions are based and inspired on the known facts about distributed systems such as CAP theorem or FLP impossibility, protocols such as 2PC or Raft, patterns such as Sagas or Event Sourcing.
Voxxed Days Microservices 2018: 2 days conference (+1 optional workshops day) only on Microservices.
Follow us on :
Website : voxxeddays.com/microservices (bit.ly/vxdmicro)
Twitter : / vxdmicroservice
Linkedin : / voxxed-days-microservi...
Keep in touch : t.co/pxf7cHZOpl
#developers #conference #microservices

КОМЕНТАРІ: 49
@mehrdadk.6816
@mehrdadk.6816 Рік тому
Approaches for dealing with data inconsistency 1. saga 2. reconcilation 3. application event log 4. 2PC/XA 5. event first
@arisskarpetis
@arisskarpetis 4 роки тому
Moving functions that require data consistency in the same Microservice was great information.
@amanbains5474
@amanbains5474 4 роки тому
Why didn’t they ask any event failure type rollback questions in microservices !! This is a really great presentation !!Too much content here in approx 30 minutes !! I have gone through this 5 times !!! Excellent stufff !!
@grygoriygonchar6994
@grygoriygonchar6994 4 роки тому
Thanks for your warm feedback! And sorry for making it that concise, that's not just you who shared that.
@mehrdadk.6816
@mehrdadk.6816 Рік тому
In germany people tend to be quite. stupid developers who don't listen and yet make a mistake and leave the company
@PointlessMuffin
@PointlessMuffin 4 роки тому
Data consistency is number one priority ;)
@VarunKalia26
@VarunKalia26 4 роки тому
Haha I see what you did there ;)
@AjayKumar-fd9mv
@AjayKumar-fd9mv 3 роки тому
Thanks for this great talk
@aarnashreyasvinay2643
@aarnashreyasvinay2643 4 роки тому
Hi Grygoriy Gonchar, thanks for the nice video on data consistency. I have a question : Where is the "Application Event Log" containing the order details (Request, Payment, Flight etc) stored ? I mean in which service ? In the example you mentioned, in "makemytrip" there are 5 services, where is the "Application Event Log" Maintained ? Is it maintained by a separate microservice (SAGA) or in one of the 5 microservices (Order, Payment, Hotel, Flight, Transfer) ?
@romantsyupryk3009
@romantsyupryk3009 4 роки тому
Thanks so much for this tutorial.
@raymondyoo5461
@raymondyoo5461 9 місяців тому
Wow, this is such an amazing speech. Thanks a lot for sharing your wisdom 👍👍 23:36 here is the summary of the lecture but it would not be so easy if you haven’t followed the whole journey
@Joseph-oz7tx
@Joseph-oz7tx 3 місяці тому
Thanks, Grygoriy, that's a great speech
@talgatsabyrov9957
@talgatsabyrov9957 3 роки тому
Эх, наших всегда можно распознать по акценту :) Классное видео, большое спасибо :)
@deniskoval4908
@deniskoval4908 2 роки тому
говорит на русском, английскими словами)
@Endvvell
@Endvvell Рік тому
НАУ може пишатися такими випускниками як він
@tobiowolawi5044
@tobiowolawi5044 3 роки тому
there isn't a fully exact way you can achieve complete data consistency with micro-service without a level of decoupling trust me. using some kind of locking mechanism and consumer acknowledgement could help to some extent though. for example, use ACID features to lock a transaction on your service db, send and event to a queue like rabbitmq or kafka and wait for acknowledgement before persisting your data in your service before unlocking or if error occurs then unlock and report the error. this same scenario for an event listener as well
@pinkylover911
@pinkylover911 2 роки тому
the presenter is very well educated, please add his name for reference :) thanks for the great talk
@TV-lv4mf
@TV-lv4mf 2 роки тому
Great sharing video and thank for tutorial friend, nice meet you ~~^^ ^^
@sajhak
@sajhak 4 роки тому
Awesome presentation. No questions from the audience, made me surprise! Anyway, He mention in Kafka vs RabbitMQ, with Kafka there are no retries out of the box. How come that happen?
@grygoriygonchar6994
@grygoriygonchar6994 4 роки тому
In Kafka, there is no easy way to retry processing a specific message without stop moving the cursor and blocking the whole partition processing. Sure you can retry processing a message easily by not moving the cursor forward, but that would delay processing the next messages in that partition. In contrast in message brokers like RabbitMQ you can do that.
@rdean150
@rdean150 2 роки тому
What he said ^ I'll just add that different use cases have different requirements for strict ordering of event processing. In some applications, it is critical that the next message not be processed until after the current message has been processed, and hence the cursor should not be allowed to move and the whole partition's queue blocked until the issue is resolved. In other use cases, one or two retries can be attempted and then the failure recorded and the processing allowed to move on past the failed message. And in some use cases, a given event should only be processed one time and the queue continues, regardless of whether it succeeded or failed. All of these approaches are easily achieved using Kafka's offset counter / cursor. But the developer needs to make a conscious decision about which approach is appropriate for a their use case.
@AmanNidhi
@AmanNidhi 4 роки тому
nice presentation
@dfhwze
@dfhwze 4 роки тому
Yeah but bad audience. There is a sea of questions sailed by!
@himanitrainingandconsultan3388
@himanitrainingandconsultan3388 4 роки тому
Need more examples
@nitinkulkarni7942
@nitinkulkarni7942 4 роки тому
@18:21, how will the "orders" event cancelled if payments did not go thru or vice-versa. In other words what happens if number 4 errors out in one of the 2 places
@grygoriygonchar6994
@grygoriygonchar6994 4 роки тому
It would be a payment service responsibility to publish an event into Orders steam/topic to inform other services that event is canceled OR it would be an order service responsibility to monitor some Payments steam/topic to detect failed payment and cancel the order. Depending on where do you want to have this complexity and if you want payments service know a bit about orders or orders to know about the payment. Looks like payments already know about orders so 1st scenario looks slightly better to me
@nishanth2884
@nishanth2884 3 роки тому
@@grygoriygonchar6994 Isn't that similar to a compensating transaction in the saga pattern except that this is event driven ?
@rdean150
@rdean150 2 роки тому
Or the Order event processor only responds to Payment Succeeded events in the first place. Or possibly it responds to the same Order Initiated event that triggers the Payment processor by creating/storing an Order in a pending / incomplete state. When it encounters a Payment Succeeded event, the Order event processor will load the pending order and advance it to an actionable state and possibly publish a new event that will signal a different processor to act upon. If the Order event processor receives a Payment Failed event, it will load the pending order and update it to a rejected state. This does introduce a potential race condition, however, that an Order event processor could encounter a Payment Succeeded event before one of its sibling processors has successfully finished processing the Order Initiated event, and hence there is no pending order for it to load and update. So I think the first approach may be the better one. But the downside of only responding to Payment Succeeded events is that if the payment failed, there will be no record of the Order existing at all, other than in the event log itself.
@VarunKalia26
@VarunKalia26 4 роки тому
I wonder if anyone if the audience was even listening to this guy? How come you cannot have a question about this topic?
@grygoriygonchar6994
@grygoriygonchar6994 4 роки тому
The audience asked many questions afterward in person :)
@SopheakSem
@SopheakSem 4 роки тому
Does it affect system performance?
@vadergrd
@vadergrd 3 роки тому
usually consistency and availability are a compromise , so in a way yes ... more consistency is less
@konstantinchvilyov9602
@konstantinchvilyov9602 5 років тому
Спасибо, коротко, по делу, понятным английским. А на русском есть то же самое или похожее?
@grygoriygonchar6994
@grygoriygonchar6994 5 років тому
Из моих докладов, к сожалению, нет
@konstantinchvilyov9602
@konstantinchvilyov9602 5 років тому
@@grygoriygonchar6994 Спасибо. Может, есть на украинском? :)
@konstantinchvilyov9602
@konstantinchvilyov9602 4 роки тому
@@grygoriygonchar6994 Спасибо, текст доклада очень помоает ebaytech.berlin/data-consistency-in-microservices-architecture-bf99ba31636f
@yoloswaggins2161
@yoloswaggins2161 4 роки тому
paid*
@riansyahtohamba8215
@riansyahtohamba8215 2 роки тому
21:15 deal with incositency
@sameersarmah1446
@sameersarmah1446 4 роки тому
Too much content in too little time
@gaatutube
@gaatutube 4 роки тому
I agree ... and some of the patterns were not clear segregated. For example, he talked about reconciliation first ... but later seemed to talk about CQRS etc (stuff in the SAGA pattern). Is reconciliation part of the saga pattern or is it a different pattern altogether? Was confused on this.
@countchivas
@countchivas 3 роки тому
ZZZZZZZZZzzzzzzzzzzzzzzz
@sbera87
@sbera87 Місяць тому
This is how you not do a presentation.. a slide should illustrate the idea. Too much talk
@vanmierra6927
@vanmierra6927 2 роки тому
Blockchain
@jayantprakash6425
@jayantprakash6425 11 місяців тому
good content but poorly presented
@bahtiyarozdere9303
@bahtiyarozdere9303 3 роки тому
No code, no believe.
@harouna_gn
@harouna_gn Рік тому
😂
@epi9820
@epi9820 Рік тому
difficult to understand his English. painful. sorry. Otherwise, it is good
@srianshworld3942
@srianshworld3942 3 роки тому
Too much content in too little time
Этого От Него Никто Не Ожидал 😂
00:19
Глеб Рандалайнен
Переглядів 10 млн
What are Distributed CACHES and how do they manage DATA CONSISTENCY?
13:29
The hardest part of microservices is your data
46:05
Red Hat Summit
Переглядів 177 тис.
Database per Service Pattern in Microservices
22:49
Arpit Bhayani
Переглядів 9 тис.
Microservices explained - the What, Why and How?
18:30
TechWorld with Nana
Переглядів 781 тис.
How to do Distributed Transactions the RIGHT way? Microservices
13:06
Airpods’un Gizli Özelliği mi var?
0:14
Safak Novruz
Переглядів 1,5 млн
Наушники Ой🤣
0:26
Listen_pods
Переглядів 421 тис.
Игровой ноутбук за 100тр в МВИДЕО
0:58
KOLBIN REVIEW
Переглядів 524 тис.