Very interesting talk, but set playback to 1.25x speed. It transforms this talk.

Agenda [3:58] ACID is not an option [5:04] (Scenario: Customer has a credit limit) [5:13] Transactions in a monolithic architecture [5:58] Concurrent transaction for the same customer will be serialized [7:15] Microservices [9:00] Microservice architecture [10:30] Loose coupling = encapsulated data [11:13] 2PC is not an option [13:41] Overview of sagas [15:01] Use Sagas instead of 2PC [15:14] Create Order Saga [16:22] Rollback using compensating transactions [18:20] Saga: Every Ti has a Ci [19:08] Create Order Saga - rollback [20:30] Sagas complicate API design [22:03] Request initiate the saga. When to send back the response? Option #1: Send response when saga completes - Response specifies the outcome - Reduced availability Option #2: Send response immediately after creating the saga (recommended) - Improved availability - Response does not specify the outcome. Client must poll or be notified Revised Create Order API [24:41] createOrder() returns id of newly created order NOT fully validated getOrder(id) called periodically by client to get outcome of validation Minimal impact on UI [25:22] UI hides asynchronous API from the user Saga will usually appear instantaneous ( UI displays "processing" popup Server can push notification to UI Sagas complicate the business logic [26:33] Coordinating sagas [28:33] How to sequence the saga transactions? [28:41] After the completion of transaction Ti "something" must decide what step to execute next Success: which T(i+1) - branching Failure: C(i-1) Option#1: Choreography-based coordination (distributed) [29:45] Option#2: Orchestration-based coordination (centralized) [30:16] Saga orchestrators are state machines [30:48] Implicit vs. explicit orchestrator [31:47] Event-based, implicit orchestrator [32:38] Explicit orchestration [34:05] Create Order Saga (code sample) [35:12] Initializing the saga [37:20] Handling a reply [37:58] Customer Service - command handling [38:28] Transactional messaging [39:18] About Saga orchestrator ⇔ participant communication [39:22] Messaging channels [40:52] Create Order Saga messaging [41:31] Messaging must be transactional [41:59] 2PC still isn't an option [43:06] Use database table as a message queue [43:22] Publishing message using polling [44:33] Transaction log tailing [46:17] About transaction log tailing [46:57] MySQL master-slave replication protocol DynamoDB table streams Summary [48:13]

I incorporated a company in 2007 who's goal was simply to proclaim the wonderfulness of MicroServices. Since (and before) then, I have never seen such a flawless talk related to MicroServices. Every other "concern" about distributed systems, Service-Oriented Architecture, Statelessness, Events, Administration, Security, and so forth have all been so mundane. And frankly, all of those other concerns can be mocked out in fairly generic frameworks. This talk touches on the actual MEAT of Software Development which is different for every customer and is the actual fun challenge. It may be stuff that experienced MicroService people know, but it has never been spoken so eloquently. I appreciate this talk very much.

Great speaker , articulate and simplifies the concept

excellent, I have looking for something like this for days. really thanks to Chris Richardson

I'd only add some minute or two about idempotency in some key moments of the process. Despite of that, this is an excellent talk.

Actually I'm readying that book. very informative book!

Great Speaker! Knows how to simplify ideas.

really pragmatic talk. thank you!

I readed alot about Saga design pattern and I developed two years a system than implement Saga design pattern and I still learned new things from this video. Thank you.

This is a real talk on real use cases on real problems

Very well explained! Thanks!

Thx for the upload, good stuff.

Very good explanation of the concepts.

This was great 👍 . Very easy to follow.

It's been really informative demonstration about how the Saga works entirely, when it comes to publish a message to the message broker (queue) in order to make them transactional , we can design them to get ACK back then consider it as done thus give it a go

Had already been implementing this principle in some of my designs before learning that it had a name . what a saga :D

Very interesting talk! One question I had near the end of the talk - it seems like using the DB’s change log as the mechanism to publish to the message broken has a large drawback: it tightly couples the internal DB scheme to the API a service exposes to the Saga (which defeats one purpose of microservices, since you want to avoid leaking your DB). I didn’t quite follow why having each individual service coordinate a 2PC between its own DB and a message broker was problematic. This seems much less bad than using distributed TXNs to implement the entire Saga, since the 2PC is an implementation detail of each service. The only single point of failure is the message broker, and I think this is true regardless of your approach?

Excellent talk. Learned a lot. Thanks

Nicely explained, Thank you.

Great talk, like that you explained very important part as processing data and sending event atomically. Other resources usually do not explain this important part. But I want to mention one moment, as usually messaging systems guarantee “at least once” delivery, we have to think about idempotency. And if we implement it, we don’t need complicated atomic save and send mechanism anymore, we now can save data and then try to send message to the next queue and ack/commit current queue. For example If we have server failure after saving to DB and before sending to next queue and acking current, then the other instance will get and process current message (here idempotency come into play as we already saved data to DB and another instance will do the same processing cycle with the message - save to db and send to next queue)

A masterclass! Thank you.

Great talk, one thing I realized after this talk is that 2PC is an option.

Old wine in new bottle. I have been doing this design pattern of State Machine Orchestrator in my SOA EDA , with compensating transactions to rollback what services had done till the point of failure. The failure event is either broadcast to all the services or to the Orchestrator so it can co-ordinate the cancellation. As I listened to the talk it was like going through my design document. Message table to message broker has a drawback when the load is high. As it is DB spefic to pull data out to push on message broker it introduces a bottleneck in the DB. Ideal for this is using a distrbuted cache DB or like he mentioned the replication and use the replicated data to message broker. Excellent talk. It touched all the touchpoints Newbies to Micorservices without much experience may think this is something rocketscience, but it as been there for long in a different avatar.

The Saga and Outbox implementation remind me a lot of NServiceBus.

surprisingly interesting talk and could really relate. great learning reference

Thanks for sharing

Great talk. What I understood from this talk is microservice consideration and implementation have to be chosen very carefully. Otherwise a perfectly running system that maintains ACID can be runied in a very short period. I guess microservices are good when a system with low transaction volume can be refactored to be cloud native.

With 2x speed, it's a very knowledge 25 minutes talk!

great talk, got a lot of useful information

13:08 correct me if I'm wrong but that transaction example with a single database doesn't work unless you have specified a serialized isolation level (one cpu core to execute all transactions). Otherwise it suffers from write skew even with Snapshot Isolation transaction isolation level which is by default I guess on many relational DB systems. You still might have two transactions that would confirm that there is enough credit at the same time, and at the same time execute insertion of two new orders.

Awesome.. really good. Thank you :)

Thanks!!!

Great talk. Really useful

Hi, i understand that 2PC is out of the scope between two microservices, but inside a single microservices you can use it. For example when you have to write a record on DB and send a message to a broker to start another micorservice.

nice talk by Satya Nadella

Thank you very much

This is dope 😍

That opening was nuts!

Thanks buddy :D

I feel like "Saga Orchestration" is completely against the logic behind the microservices which is building independent components and teams. In this scenario a team has to maintain the orchestration point of two services which also has to be aware of the business logic of both services. I know sometimes orchestration is inevitable but I think in this situation it's better to use choreography.

Great talk. Learned a lot. Question, at 29:50, isn't it a bad idea to use choreography under any circumstance since it no longer loosely couples microservices (which is one of the main benefits of a microservices architecture)? I know he mentions the following centralized approach which is better. But what I'm curious about is, "shouldnt one avoid (at all costs) doing choreography and not even say that is a viable option in the first place?"

Good and clear presentation. Nice talk. I am surprised about using tables for recording messages. If you are using Saga coordinator then it could be the one verifying that an operation was successful and generate message for the next operation.

His microservices talks are the most informative and concrete out of anything else I've watched. Definitely getting the book! There's also a lot of information on his website ➠ microservices.io

Good presentation and still this is just showing one aspect chipping off the iceberg. Moving from a monolith to distributed remote services really introduces a ton of complexity and impacts latency. Done badly, and most of them will be, companies move from one ball of mud to another. Its when something goes wrong that is the killer... and you have to scramble around for a magnet to find the needle in a haystack. There are still lots of companies that haven't matured in terms of building and packaging services for a monolith, let alone scaling that to a distributed service architecture which isn't any easier. Lots of opportunities for IT staff in the coming years. I still think that modelling vertical slices based on "entities" is a mistake in presentations. Customer or even order, for example, means completely different things to different roles in an organisation.

Excellent, it took last 15mints to understand 1hour repeat and see. Its 4 years before, now i want to see what are latest changes or implications update in saga, any suggestions?? Please replay to comment

Do I have any advantage of using a message broker when using a message table unless I want to fan out my messages?

Chris the way you are referring to implicit orchestration through events, I am confused then what is choreography?

Really enjoyed the talk, but imagine if they these talks used a richer language like F#, Rust, Scala, etc that have enums and traits instead of all the class based fluff ❤

Great talk! I hope Sagas don't manifest themselves as ServiceBus. Whereas Sagas are some state machine implementation, people would abuse them until it becomes another ball of mud. I would rather prefer microservices to be along the following lines: - Define your bounded context well so that they are relatively independent and will have consistent state within the context - If you must have other services decide a consistent state (those shouldn't be lot if you have followed the above), rely on synchronous calls to other services. With next generation protocols like gRPC performance impact is negligible - Leverage service mesh framework like Istio for coordination

one alternative approach to the deleting the messages in the message table is : 1. mark the processed messages as processed = 1 2. select only those messages for which processed = 0 and order them according to the timestamp

Not sure I get it. How is this not just a distributed transaction where you've added the concept of compensating actions.

10:50 each services has its own databases. 12:30 how to maintain data consistency for each databases?

These are very old and good known problems of any distributed system. Main point is when you go from monolith to microservoces you go from single instance to distributed with all of its problems and complications.

AFAIK It is kind of antipattern to use database table as a message queue

the money transfer case, I think this kind of function should be designed as in one microservice and done inside a single DB transaction. If there is no option to do so, then probably an intermediate cash pool (fake, intermediate account) is needed

Great talk! One question though: If you use a DB table to buffer the messages to be sent to the message broker, haven't you just moved the problem of exactly-once-delivery? What if the mechanism that dispatches the messages from your table to the broker crashes/rolls back?

how awesome would it be to get an architectural/code review from Chris Richardson?

Interesting progression from "Monolith has this problem X" then "Microservices solved the issue X" but then we have problem Y and then "We solve problem Y by taking away some of the good parts of Microservices", which ends up in losing the benefits of both worlds of Monolith and Microservices.

Started as a good talk, then slightly became messy after the 'theory of sagas' and when all problems were 'interesting' (but without solution), then, when it came to 'practical' part, everything turned into a complete mess with reinventing message brokers on top of specific features of databases. Anyway, we will probably get another 'open-source' framework :)

40:50 ... Hmm... "assuming that the message broker has durable at-least-once delivery". Yes... assuming that. - and that it's up. I'm not sure I buy that async ReST doesn't work just because you can make the problem go away by introducing a magical component. If you have to run this yourself, is it easier to have high availability on the message-broker than on your service? why? ... and why can't the client just have retry logic a part of the saga state machine?

why not use an aggregate microservice for transactional operration?

You could have a Saga Service that handles all sagas?

Can somebody explain to me why is it not a good idea to send msg inside a txn? if a msg was not transmitted then isnt it perfectly ok to rollback the transaction? Wont tailing the commit log of db get us the same behavior?

I am extremely confused by this orchestration based saga approach. Doesn't this present with the same problems that we set out to solve in the first place? For example, what if the saga has 5 steps and for some reason, after making the API call for the 3rd step (say to the CustomerService) and waiting for the API response, the orchestrator service goes down. How will the Saga Orchestrator after restarting know where to start from? It can at best start and try to perform 3rd step again but the 3rd step might have already been processed by CustomerService which means we might be trying to perform same action twice.

How we rollback if compensation transaction also fails

Choreography looks much better to me and more loosely coupled

For the last part, "Transactional messaging", why can't we utilize out of the box solutions like Kafka instead of using database commit log tailing?

Had to set it to 1.5 speed make it more consumable

The Saga seems to me like an AWS step Functions? Is that correct?

Why not just have a separate "Transaction Service" that tracks the steps and whether they fail or not?

I'm using gateway api as a middleman for my microservices

Read more and get code samples at microservices.io/patterns/data/saga.html

genius :-)

why are we trying to do this ? to make systems less complex , to have a smaller focused development teams ? to reduce time to production ? I feel we are making systems more complex , we are adding many more smaller databases , we are re-inventing the transaction mechanism such a waste of time , please do a video where you really need to have microservices and where not, its become a fashion to bounce the terms microservices , REST, K8S

Distributed monolith FTW...

how can I get that PPT ??

It's better to understand the current

@11:00 why would somebody make multiple databases for this? seems utterly crazy is this because one would naively be following the microservices dogma?

At 8:37 , not monolith . It’s rather many monoliths

Why not use TC/C ?

1.5 work for me .. :)

Intro music name, plz ...

talk starts at 15:16

Please don't roll your own workflow. See Durable Task Framework and Azure Durable Functions.

Everyone says this is a great talk. Honestly, I didn't see anything new. Little bits from this concept and that concept cobbled together. Talk is cheap. I want to see a working code.

CreateOrderSaga? Really? Reinventing distributed transaction manager under a new name. The only difference is the sequential rollback invocation. Some silly stuff.

Unfortunately the approach presented here is not perfect when comes to scalability: Order.state == PENDING is basically a database write lock in disguise.

If only there was some way for a database to expose a consistent interface layer while still allowing the "private data" (ie, actual tables) to be modified "under the covers". Oh wait, there is. Views. Stored procedures. Functions. You do not need an application level API layer to provide data model independence. Views/procs/functions *are an API layer*.

Will not work. The compensating transaction can fail also. In addition, when you commit (order created) then other processes can make decisions based on this fact. Those processes will perform some other business processes (transactions). Then , you decided to reverse order . But, the train left the station already. How will you rewind all other transactions?

(y) msilva

pro tips 1.25x sounds normal.

10mins and still didn't found saga.

MassTransit to the rescue 2021

This does mot work where ot maaters. It will eventially work where nobody cares.

david blaine's father?

Precisely what stored procedures used to do so well with transactions wrapper. Anything that went wrong would rollback the whole thing. We keep reinventing the wheel with this kind of rubbish - “saga” a new word for same old solved problems

for peuple watching in 2023. don’t use saga! use a database that support multi/shard transaction!

Microservices isn't an architecture. Decoupling is. Microservices is nothing new. Decoupling has been around forever. (This is nothing related to the video, just felt like saying it.).

I stopped watching when he says use database as a temporary message queue. WTF