What is a MESSAGE QUEUE and Where is it used?
ΠΠ΅ΡΠ΅Π³Π»ΡΠ΄ΡΠ² 933,972
ΠΠ΅Π½Ρ ΡΠΎΠΌΡMessaging Queues are widely used in asynchronous systems. Message processing in an asynchronous fashion allows the client to relieve itself from waiting for a task to complete and, hence, can do other jobs during that time. It also allows a server to process its jobs in the order it wants to.
Messaging Queues provide useful features such as persistence, routing, and task management. We will be discussing the benefits of a message queue in future videos.
A system having a message queue can move to higher-level requirements while abstracting implementation details of message delivery and event handling to the messaging queue.
The 'queue' is just a name for this data structure. In practice, it could be storing messages using any policy. Some examples of message queues are Kafka and RabbitMQ. They are widely used for various purposes, such as command query request segregation (CQRS) and event sourcing.
00:00 Pizza Shop Example
01:40 Benefits of Asynchronous Processing
02:44 Scaling Shops (Horizontally)
03:10 Fault Tolerance
04:09 Features of a message queue
08:38 Encapsulation
09:38 Thank you!
Code: github.com/coding-parrot/Low-...
Looking to ace your next interview? Try this System Design video course! π₯
interviewready.io
With video lectures, architecture diagrams, capacity planning, API contracts, and evaluation tests. It's a complete package.
Asynchronous Request-Response:
stackoverflow.com/questions/1...
www.enterpriseintegrationpatte...
Message Queue:
blog.codepath.com/2013/01/06/a...
highscalability.com/blog/2012/...
www.cloudamqp.com/blog/2014-1...
www.rabbitmq.com/getstarted.html
Load Balancing:
β’ What is CONSISTENT HAS...
You can find me here:
/ gkcs0
www.quora.com/profile/Gaurav-...
github.com/gkcs/
#message-queue #system-design #scalability #server
@UlfAslak 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Notes to self: * Servers are processing jobs in parallel. * A server can crash. The jobs running on the crashed server still needs to get processed. * A notifier constantly polls the status of each server and if a server crashes it takes ALL unfinished jobs (listed in some database) and distributes it to the rest of the servers. Because distribution uses a load balancer (with consistent hashing) duplicate processing will not occur as job_1 which might be processing on server_3 (alive) will land again on server_3, and so on. * This "notifier with load balancing" is a "Message Queue".
@ahalyapotti9905 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Very good notes π
@mahmoudelrabee2456 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
I don't understand why there might be a duplicate ? the notifier will just query the tasks that handled by the cracked server and distribute them!
@sidharthsinghal1920 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
@@mahmoudelrabee2456 That is true if we store the server id also. This is explained as the first approach at @4:52
@codekicks2171 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Consistent hashing and load balancer are 2 different things not same
@user-eq4oy6bk5p 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
The alternative approach is, instead of assigning individual task to server, you can let servers poll from the queue. In this case, your message queue is decoupled from application servers since message queue doesn't need to know anything about servers.
@Montisero 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
This guy reminds me of that friend that tries to explain and wrap up the whole semester for you 30 minutes right before the exam, because you didn't attend any lecture since the beginning.
@dipak2704 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
great comment.... the way he is explaining its really wonderful
@cizzlen07 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Lmao
@sweet1251 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
dipak sonawane r
@hrishihatolkar1944 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Exactly :)
@sammcalilly107 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
those people are saints
@anulik1985 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Great job explaining everything in a way anyone can understand. You are a natural teacher! Please continue teaching and sharing your knowledge!
@gkcs 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Thank you!
@mokonzi2K6 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Message queue is getting lots of spot light in the industry, please keep making the video on this topic i like you way of teaching. thanks again.
@corybeaver7562 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Former teacher turned Linux engineer here. Very well done explanation of this concept. Easy to follow with great usage of visuals and ongoing metaphor!
@rskrao 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
I like the sheer excitement with which the topic is delivered!! Kudos!!
@yusufahmed2233 Π ΡΠΊ ΡΠΎΠΌΡ
Explained the whole thing in literally the first 40 seconds. Truly amazing work!
@PanKonKeso99 6 ΠΌΡΡΡΡΡΠ² ΡΠΎΠΌΡ
Brother, I've been researching about message queues and I was so confused until I saw your video, thanks alot!
@AhmadSayeed-plus 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
I am a java dev. For learning purpose I am planning to make queue system. But from your video i got idea that JMS is something I should learn now. Thanks Gaurav.
@irynasherepot9882 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Thank you! Your tutorials are great! My college did not have System Design and Analysis class, and your videos helped to learn a lot.
@gwho 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
great clear presentation! one tip: use higher contrast ink and board. So that means more lighting to make the board more lighter/white and/or using dark ink like black or brown so we can see it more clearly. great content. subscribed. keep up the good work!
@ish1285 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
This was the best video so far. The way you explained the entire stuff without saying Message Queue the whole time awakened my grey cells. Thanks a lot!
@mayankvora8116 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Amazing explanation Gaurav. The best way to explain any concept is with practical examples, and you did the same.
@dilawarmulla6293 6 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
There are very less resource available for system design in youtube so please complete series . I like the way you teach. Thanks for making videos for us. God bless you
@gkcs 6 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Thanks Sameer :)
@Thrillseeker666 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
"Very less", Very few. Indian English is hilarious though.
@hfontanez98 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
EXCELLENT job in all areas: Simplifying the use cases for explaining easily to non-experts, very VERY close examples to real-world instead of using hypothetical cases, starting from simple (in memory) to more complex approach (with database), and avoiding super technical jargon; yet not shying away from technical details (i.e. load balancing, hashing, etc.) Well done Gaurav!! I enjoyed this video.
@gwho 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
well said.
@rishabhagarwal9871 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Hi Gaurav, This is the one of the best real life example (with pizza shop) showing the need of asynchronous request/response system. Thanks for the great video. Really loved it.
@gkcs 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Thanks!
@ralphmoran 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Again, thanks so much for taking the time to make this video. I've learned a lot from you. Keep it up!
@gkcs 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
π
@pramodkharade5373 Π ΡΠΊ ΡΠΎΠΌΡ
The concept has been explained very clearly. It would be great if you would come with practical implementation using NodeJS with any MQ
@stillthinking6357 4 ΠΌΡΡΡΡΡ ΡΠΎΠΌΡ
know a days People like Bhaiya and didi on linkdln those who are giving lecture on system design other bla bla.. even don't know how to code, have more subscribers than this genuine talent. Hats of bro.π
@timothy6966 Π ΡΠΊ ΡΠΎΠΌΡ
Your channel is amazing. Your explanations are some of the best Iβve ever heard/seen. Good job man.
@ShubhamSingh-ku2ow 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Bro, keep bringing up more videos like this. We are all such a big fan of yours. ALL the best! π PS: Amazon has SQS ;)
@rahilsinha_1407 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Great Explanations of Sytem Design. Made fall in love with each concept.
@rajeev812 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Gaurav, this is really a nice and knowledgeable tutorial you have made, in very sort time you have explained this topic very clearly. Can you try to make it as a practical ( how to implement RabbitMQ)
@mahendrajadav2286 ΠΡΡΡΡΡ ΡΠΎΠΌΡ
"Now I know everything about how to run a restaurant; I will be starting my own restaurant very soon." Thanks to the legend Gaurav.
@barimohammedabdul5166 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Explained the concept really well , easy to understand
@vaibhavpali 9 ΠΌΡΡΡΡΡΠ² ΡΠΎΠΌΡ
kya to samjaya hai Gaurav, maza aa gaya.. Such precise explanation is so rare on online tutorials.
@NohandleReqd 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Are we just going to ignore the fact that 9 / 11 were pointing at the same server and THAT was the server that crashed! :P Your lectures are fun man!
@MrZyman 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Illuminati confirmed! Well done.
@dianecoffee4467 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Excellent and lite tutorial on messaging and load balancing
@udayayyagari9160 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Gaurav, fantastic job with explaining the key tenants of high-level system design; how about bringing in some tooling and technology stack to support each of the architectures.
@sat1478 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
You are the best teacher. Thank you very much.
@cyrilrajc 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
If you could cover on the concepts of real time operating systems- which includes tasks, task states, ,message queues and mailboxes etc..., It will be definetely helpful for me and for those who are interested in embedded computing and RTOS...
@pratikbhandarkar25 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Great video! This video explains the system design of a pizza shop very well. However, it spends a lot of time explaining load balancers and the notifier and very few minutes are dedicated to discussing message queues. I am more interested in the actual use of message queues in real-world systems like pizza shop here.
@voleti19 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Appreciate your work!! thanks for sharing.
@chiragr1336 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
@Gaurav Sen Very informative video! I request you to make a video or videos about every component of any system design. Different problems employ different components. If we first study all the different components and their properties then at least we can stay thinking in right direction. Thanks for your video series!
@manoharsingh6050 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
When to use a Topic vs a Que would have been a nice addition.
@vigneshwarpadmanaban7816 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Amazing work Gaurav, Keep inspiring!
@AmanSharma-hi3fd 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Thanks a lot for the content. Even I discussed the same heartbeat like mechanism (Got the idea from the video :P) in an interview. It was really helpful!
@zararosenrot 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Thanks for the great explanation man!!
@navjhs 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Nice explanation. You are actually doing excellent work. Thank you.
@MubashirAR 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
The only youtuber that keeps his video at 9:59
@gkcs 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Hahaha
@danielk8452 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
HAHAHAHAHAHAHA
@rtwkkumar20 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
haha :?
@hairypaulsack 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
back in the golden days of youtube everybody made videos to make videos, not to make money
@kailashc1321 Π ΡΠΊ ΡΠΎΠΌΡ
Excellent explanation. Thanks
@prativmukherjee5081 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Short.. To the point.. Brilliant π
@Llink4n 4 ΠΌΡΡΡΡΡ ΡΠΎΠΌΡ
You explained it so well, thank you
@user-ld8ec2sj9i Π ΡΠΊ ΡΠΎΠΌΡ
Thanks for the valuable video! It really helps me to understand what the message queue is :)
@susantagouda8213 Π ΡΠΊ ΡΠΎΠΌΡ
Very good job. I had so many confusion you just cleared
@Ayesha_siddiqah 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Brilliant:-) good job. Really appreciable. Kindly make complete series on system design and soon start video series on python programming:-):-)
@nomib_k2 Π ΡΠΊ ΡΠΎΠΌΡ
so smart the way he is explaining. keep it up man
@ashish1004tiwari 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Man, You are awesome.. the way you explain is truly great.π
@KB-ce5sh 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Damn man you are an amazing teacher. Subscribed!
@elvinvalette9928 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Very clear explanation. Thank you
@roooooot9545 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
this is awesome explanation
@sagarmittal8392 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
amazing series!
@cyrilrajc 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Thats an excellent explanation Gaurav.. Thank you.
@abShar0705 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
banks, thats a fairly common use case that uses messaging services/queues extensively. batch processing transactions, after the card scheme authorises your tx and the request is send to the issuing bank, the entire xml message is stacked up on a messaging queue that is being constantly listened to by the issuing bank messaging service
@KTechy- 8 ΠΌΡΡΡΡΡΠ² ΡΠΎΠΌΡ
great explantaion!
@gasannazer6556 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Pretty well explained.
@naufilshaikh09 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Really nice explanation, impressive.
@mukeshbarman 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
In a typical System design interview: Dabbe banao dabbe.. :D
@TheGugustar 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Everyday you learn something new! I always thought PS3 stood for PlaStation 3, but here I learn that it stands for PizzaShop 3.
@rohitdhankar360 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Excellent stuff - Cheers !!
@sankalparora9374 Π ΡΠΊ ΡΠΎΠΌΡ
Amazing explanation! Thanks!
@gkcs Π ΡΠΊ ΡΠΎΠΌΡ
You're welcome!
@shalinifefar4750 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Hey Gaurav, Thank you so much for the great explanation. Your videos are awesome. Keep up the great work.π
@gkcs 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Thanks!
@NealMadlani Π ΡΠΊ ΡΠΎΠΌΡ
I've learnt a heck of a lot about how pizza shops work
@anindachatterjee8904 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Nicely and simply explained
@ravishankarjoshi2952 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
We would like to have videos on object-oriented design patterns too like observer pattern etc. Please keep making videos. :D
@redpillsatori3020 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
OOP is overrated and people are finally moving towards functional programming with NodeJS and React as some common examples
@user-mw5bm2xh9x 10 ΠΌΡΡΡΡΡΠ² ΡΠΎΠΌΡ
thanks for the informative presentation
@gkcs 9 ΠΌΡΡΡΡΡΠ² ΡΠΎΠΌΡ
Glad it was helpful!
@staronkar 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Please create a video over HermesJMS. This video was really helpful in getting the basics of MQ
@emmanueldsouza 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Great video. Would like a part 2
@gkcs 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Maybe one on Apache Kafka and it's features :)
@abheeg5041 6 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
for each system design if a detailed explaination is there, then, it will be helpful for all becoz i think i can learn system designs from you thank you for your extraordinary work
@gkcs 6 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Thanks David :)
@jeevithamary7781 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Good Teaching!!! Keep it up with great work
@michaelajayi2117 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Dude, you should get a Ph.D. for this... made it easy to understand. Thanks for sharing.
@juanmamani2110 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Cool! Thanks for sharing.
@misaelpereira9679 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Your charisma and you way of teaching gain me to be your subscriber :)
@kalidindiprashanth7363 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Super!!! Thank you Bro...
@VikramThakur8 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Gaurav! Its good conceptually. Thanks
@dataman4503 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Great explanation
@muhammadsalmansanaullah102 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Nicely done Gaurav!! Once again... :)
@madinabonualisherova8935 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Well explained!
@vishalberi1259 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Hey Gaurav, appreciate your awesome work. my point is as you gave example if a pizza shop is down. what i think there will be no server on shop side and a shop has a client id only servers must be at remote location. orders object must have a client id. there must be process on server side that check heartbeat of client with an associated client id. if that client id down ( clients spawn a heartbeat thread to server and server checks heartbeat from client) then it will do the rest of things your explained done by your notifier component to assign the orders ( to client having nearest to client which is down)
@gkcs 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Thanks Vishal! Have a look at the full playlist. I do speak of it as you mentioned
@gauravmaithani4707 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
"Filling up the coke can" 2:20 haha. Love your videos bro.
@SankethShetty 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
his expressions are hilarious !
@juanandrescastillosoria2936 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
This tutoria is GREAT
@dheerajsree1166 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Hi Gaurav, In the video where we discussed about duplicating order no. '3' between S3 and S2, the S2 is still up and running right? So will there be a case where load from S2 should be distributed to other servers? Will we be doing load balancing even if the Server is active?
@curious1731 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Very giid analogies and articulation of topics..gr8 wrk.
@ShaanGola 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Hi Gaurav, I'm glad you have discuss about RabbitMq. I just came over to explore more about this topic ...I just want to know more about how to monitor microservices in rabbitmq..I'm able to know more about the application architecture and implementation after watching your videos. can we have ellaborate discussion over this i'm stuck somewhere in rabbitmq.
@iitgupta2010 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
this is the first time I found the wrong explanation. 1. Once you have notified and it found that server 3 is dead then it can give that specific task to "assigner" node which basically divides the task and share between other nearest server to that location (it is important). 2. Each shop has to maintain his own task queue and "assigner node" would add those task in that queue based on priority ( but this will be least among its own task, as it requires to completes own task first then other. ) 3. there is no use of centralized queue, until n unless you provide a feature to the pizza shop and based on user location pizza shop automatically (user did not select the shop which is always the case in dominoes at least) assign this task to the nearest shop. 4. Load balancer (if you were talking about the actual load balancer between servers) is no use for the assignment, as its responsibility to just equally divides the task across multiple servers within the same region for which this load balancer responsible for.
@parvezmulla3324 6 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Nice explanation! Can you suggest resources to follow for system design?
@amruthlal1394 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Hey Gaurav, you explained the concepts behind services provided by a message/task queue neatly. Your system design series is extremely useful for beginners like me. Keep up the good work. I have a small suggestion regarding the pizza example. In the scenario of a pizza shop node failure, the load balancer may consider other factors such as geographic proximity of a particular shop to the clients location. Hence the dynamic reallocation of requests by load balancer also includes some business logic. This makes the example a little bit complex for explaining the purpose of message queue. Please do consider finding out more simple examples which serves the purpose without additional complexity in future videos
@gkcs 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
That's a good point. I took up the example since I wanted to point the simple way of going about designing a system. π
@AnirbanRoyChowdhury1987 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Really good video. Khub valo laglo dekhe.
@TheNumberEight 6 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
I'm self taught and system design is not an area that I seem to be learning from documentation (and that makes sense, it isn't relevant to synatx or features of most technologies). Thank you for providing this material! You're decent to good at teaching, but mostly you're providing resources that are difficult to acquire outside a classroom or without a mentor.
@rajchoudhary4349 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Nice narration..!! My pizza just got delivered under 30min
@shantanusinha3609 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Great Analogy!
@adityajoshi8794 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Hi Gaurav, loved your explanation. But I think, here you've mixed up related but different concepts. You are actually creating a storm topology with input from a kafka queue. The order queue is a message queue (say kafka). Storm topology is polling messages from the order queue & is assigning the tasks to one of the worker nodes (pizza shops) & is internally keeping track of task status using a task queue & using hearbeats (by using a zookeeper) to check whether nodes (pizza shops) are alive or not & if not assign the unfinished tasks on that worker node to a different worker node.
@gkcs 3 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Yes. What about the reallocation of tasks though?
@saraogi 4 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
Thanks for such a great video, but one small doubt, the problem of duplicates how will it be handled when we are adding a new server, like on removal it was simple we will distribute the load to others so no chance of duplicates.
@stillranking 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
It's an amazing talk. Here's a follow-up question. What if the message queue goes down to be a single failure? Should we have the message queue with hot backup such as Active-Passive mode?
@sarahdaniel6862 2 ΡΠΎΠΊΠΈ ΡΠΎΠΌΡ
That is where the persistence of data comes into play. In order to keep reliability of the messages high, most message queues offer the ability to persist all messages to disk until they have been received and completed by the consumer(s). Even if the applications or the message queue itself happens to crash, the messages are safe and will be accessible to consumers as soon as the system is operational.
@suneelbabu6186 Π ΡΠΊ ΡΠΎΠΌΡ
Thank you so much for good video
@vaibhavkulkarni7932 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Very well explainedπ ..If possible please make a video or two on Kafka Queues( involving Producer n Consumer ) ..This would serve as an example for for few of your previous videos on MQ,Load Balancing,Event Driven Systems..This would help them put together..
@gkcs 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
I have this in my task list. It'll take time to get to it though :)
@vaibhavkulkarni7932 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Cool!!!!
@ankitsharma-wm4zj 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Hi Gaurav, Can you please explain this in more details like what are the classes hierarchy used to make such application ? Your teaching way is too good please explain this also. (Y)
@manishbhunwal1 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Please confirm if my video understanding related to queues are correct : When S3 crashed, we wanted to re-distribute the order no 9 and 11 to other pizza stores (s0 to s2).But server did not have record or order no's which were assigned to Pizza store S3. Initially we thought of using 'Load Balancer' for this purpose. (This is clear to me, so moving to queue now) After that we improved our solution by using 'Message queues'. If we use 'Message queues' then we need to have one 'Queue' for each Pizza Store (Both at client and server side applications). Each pizza store will listen to its own 'Message queue'. Whenever a new Pizza is added to a 'Queue' (belonging to S3). when this Message is received by Pizza store S3 and acknowledged and then S3 starts processing it. If S3 is 'SUCCESSFULLY' able to complete it then message will be removed from the Queue at server side. If S3 crashes then heart beats are not received and all messages stored in the Queue [belonging to Store S3] will be added to other store's Queues. If S3 throws error/Exception (Enough Cheese not available for Pizza) during processing then message is still on server queue [Not a valid use case for current discussion]
@hvanmegen 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
As a European developer that has seen too much low-quality programming disasters from projects outsourced to low-wage Asian country developers, I must say that it's refreshing to see an Indian engineer that has actually studied for this stuff and knows what he's talking about, using the correct terminology... I'm subscribing; you're creating great content.
@srinathrachapudi 5 ΡΠΎΠΊΡΠ² ΡΠΎΠΌΡ
Low-quality programmers are everywhere not just in Asia.
Gaurav Sen
ΠΠ΅ΡΠ΅Π³Π»ΡΠ΄ΡΠ² 682 ΡΠΈΡ.
Hussein Nasser
ΠΠ΅ΡΠ΅Π³Π»ΡΠ΄ΡΠ² 138 ΡΠΈΡ.
Π£ΠΠ ΠΠΠΠ Π‘ΠΠΠΠΠΠ― - ΠΠ ΠΠ¨Π
ΠΠ΅ΡΠ΅Π³Π»ΡΠ΄ΡΠ² 1,1 ΠΌΠ»Π½
ΠΠΈ - Π£ΠΊΡΠ°ΡΠ½Π°
ΠΠ΅ΡΠ΅Π³Π»ΡΠ΄ΡΠ² 570 ΡΠΈΡ.
ΠΠΈ - Π£ΠΊΡΠ°ΡΠ½Π°
ΠΠ΅ΡΠ΅Π³Π»ΡΠ΄ΡΠ² 1,5 ΠΌΠ»Π½
Gaurav Sen
ΠΠ΅ΡΠ΅Π³Π»ΡΠ΄ΡΠ² 1,8 ΠΌΠ»Π½
Gaurav Sen
ΠΠ΅ΡΠ΅Π³Π»ΡΠ΄ΡΠ² 200 ΡΠΈΡ.
sudoCODE
ΠΠ΅ΡΠ΅Π³Π»ΡΠ΄ΡΠ² 77 ΡΠΈΡ.
WeAreDevelopers
ΠΠ΅ΡΠ΅Π³Π»ΡΠ΄ΡΠ² 13 ΡΠΈΡ.
Gaurav Sen
ΠΠ΅ΡΠ΅Π³Π»ΡΠ΄ΡΠ² 965 ΡΠΈΡ.
Π£ΠΠ ΠΠΠΠ Π‘ΠΠΠΠΠΠ― - ΠΠ ΠΠ¨Π
ΠΠ΅ΡΠ΅Π³Π»ΡΠ΄ΡΠ² 1,1 ΠΌΠ»Π½