How Discord Stores TRILLIONS of Messages

  Переглядів 651,863

Coding with Lewis

Coding with Lewis

День тому

Have you ever wondered how Discord is able to store trillions of user messages without breaking a sweat? In this video, we'll take a deep dive into the technologies and strategies that Discord's software engineers use to manage such a massive amount of data.
SOURCE: / discord
Discord is one of the most popular communication apps in the world, with over 250 million users and counting. With such a large user base, Discord's servers receive an enormous amount of messages every day. To store all these messages, Discord employs a distributed database system that runs on hundreds of servers worldwide.
This system uses a variety of modern database technologies, including Cassandra, Redis, Elasticsearch and most recently: ScyllaDB to ensure that messages are stored reliably and efficiently. These databases are designed to handle huge amounts of data and provide fast read and write capabilities, making it possible for Discord to scale to millions of users without compromising performance.
So if you've ever wondered how your messages are stored on Discord, or if you're just curious about how modern database technology works, this video is a must-watch. Join us as we explore the world of distributed databases and learn how Discord's software engineers keep trillions of messages organized and accessible for millions of users around the world.
👻 MY NEW STARTUP: www.caspr.ai/
MY NEWSLETTER 💌
thebetter.dev
------
CONNECT WITH ME ON SOCIAL
📸 Instagram:
/ lewismenelaws
🎚TikTok:
/ lewismenelaws
🐣 Twitter:
/ lewismenelaws
--
My gear 💻
liinks.co/lewismenelaws
-----
TIMESTAMPS ⏰
0:00 Intro
0:38 The BackStory
1:46 Cassandra Troubles
4:09 Change Architectures
6:28 Data Services
8:57 The BIG Migration
10:35 Where It's at now
11:40 What we can learn from this
12:46 Thanks for watching and subscribe :)

КОМЕНТАРІ: 589
@CodingWithLewis
@CodingWithLewis Рік тому
Thanks for watching this video :) What company would you like me to cover next?
@infinitelancer4612
@infinitelancer4612 Рік тому
Reddit
@ExplodeCode
@ExplodeCode Рік тому
Do How ExplodeCode is cool. naa do snapchat or instagram or something based with pictures
@adill6295
@adill6295 Рік тому
Rust roadmap
@akvamsikrishna5535
@akvamsikrishna5535 Рік тому
How telegram makes money and how it stores as much as data like discord and much more (you too will get more questions once you start to think how telegram is able to do it 😅)
@akvamsikrishna5535
@akvamsikrishna5535 Рік тому
And 😅 I want you and forrest knight to have a video together 🥺
@sorteslyngel2k
@sorteslyngel2k Рік тому
How the hell is discord even making money. That infrastructure sounds pretty expensive...
@agentnull5242
@agentnull5242 Рік тому
Nitro 😭
@agentnull5242
@agentnull5242 Рік тому
Bloody expensive too
@lucachiapello6454
@lucachiapello6454 Рік тому
Data
@tonik2558
@tonik2558 Рік тому
Venture capital is one hell of a drug
@MtTheToto
@MtTheToto Рік тому
They are not making money. Welcome to capitalism
@jamoncitovideos
@jamoncitovideos Рік тому
This types of videos are great. It's not common to face scenarios that have this massive scale and it's great to get to know how they have been handled before by other engineers
@puspamadak
@puspamadak 10 місяців тому
The engineering was truly awesome. Have to salute the engineers as well as SyllaDB team for such a great product.
@ryojikn
@ryojikn Рік тому
This is amazing, please keep going. Such an amazing source of knowledge and inspiration in a perfect method for better engineers
@staticobjx7232
@staticobjx7232 Рік тому
Hi Lewis, I discovered your channel thru YT Shorts, but I like this format a little better, not only it dives into architectural problems and solutions, in my opinion it is not rushed. Also, awesome graphics! Thanks a lot and more power to your channel.
@5095smf
@5095smf Рік тому
Nice! I love databases and learning more about them, great video.
@coolxify
@coolxify Рік тому
😂 same
@nerduser1000
@nerduser1000 Рік тому
The type of content I love to watch thank you lewis! Had this question for Reddit how reddit saved and would love to know how discord works
@ThatBubbledGuy
@ThatBubbledGuy Рік тому
Your channel needs to be more popular. This content is the only thing that motivates me to write some code after running into an error.
@wonkywonky6307
@wonkywonky6307 Рік тому
This is fantastic! Congratulations to all those amazing engineers and congratulations to you for making this great video. Thank you for bringing this content to us.
@Luxcium
@Luxcium 10 місяців тому
I am so happy 😁 to see how it goes with your channel and how much your topic is related to more advanced topics
@demin-e
@demin-e Рік тому
No © symbol next to “Rust”? Lawsuit is already being filled
@cocoiyeah
@cocoiyeah Рік тому
Kudos to the ScyllaDB team! I've been a fan of their product since 2019, and in my opinion, it's better than Apache Cassandra. However, their open-source license limits usage to only five nodes.
@Jecsham
@Jecsham Рік тому
the ydeserve being paid for that product
@dominikmankowski1636
@dominikmankowski1636 Рік тому
Not really. The 5 nodes limit applies only to ScyllaDB Manger, not the OSS database.
@pietrodc0
@pietrodc0 9 місяців тому
72 nodes per 140mln users = ~1.9 users per node. 5 nodes limit = ~7.72mln users = you must have money for DB licenses 😅
@OT-tn7ci
@OT-tn7ci 8 місяців тому
​@pietrodcfab but that's not how it scales
@nemzyxt
@nemzyxt 9 місяців тому
Whoa! This is so enlightening, thanks Lewis! The engineers are doing really great work bts
@My.MusicChannel
@My.MusicChannel Рік тому
This video was perfect omg!! Liked & subscribed, keep it up ❤️❤️
@walterrodriguez9550
@walterrodriguez9550 Рік тому
great breakdown and description, thank you!
@AdoSean
@AdoSean 11 місяців тому
Really fascinating video and well made. Thank you!
@BrendAn_RSA
@BrendAn_RSA 11 місяців тому
This is my first video on your channel. Loving it. Awesome video :)
@TNTISGOOD
@TNTISGOOD 2 місяці тому
agreed, my first video too! :D
@xenofenus
@xenofenus 8 місяців тому
This was an incredibly well made video!
@TheMassgames
@TheMassgames Рік тому
This types of videos are great, and your animations are really on point
@bvorg
@bvorg Рік тому
Everything is perfect. Thanks much Lewis!
@AnushR27
@AnushR27 Рік тому
Great video Lewis 😃. Got to know so much information about Discord databases. Thankyou!
@def1nt
@def1nt 11 місяців тому
Very cool format and interesting stories! I wish Discord to find frontend devs as good as their backend team
@ryanzspaulding
@ryanzspaulding 4 місяці тому
I like your funny words magic man. Definitely didn’t understand about 95% of this video but it was really fascinating and well presented
@vaggelis_best
@vaggelis_best Рік тому
11:24 Poor Descord engineers😢 They are stressing even HARD
@Raul-pg1pf
@Raul-pg1pf Рік тому
lmao
@Keys_02
@Keys_02 Рік тому
Great content, please I want to learn about the internet from a developer's perspective what book are you going to recommend for me?
@joker-wr8pt
@joker-wr8pt Рік тому
Sir you're my inspiration these types are useful please bring more like this like how big system works so that we can have real-time understanding
@Skeffles
@Skeffles 3 місяці тому
Great video. Fascinating to see how they solved this!
@Fingiorstyle
@Fingiorstyle Рік тому
Thank you for this video! Please create more
@denitechYT
@denitechYT Рік тому
Aha so now we know how! Great video man!
@justingolden21
@justingolden21 Рік тому
Love the video and also the short takeaways at the end
@smixqse
@smixqse Рік тому
i love the way the subtitles interpreted scylladb: s db, mes db, solid b, sillydb, facilitydb, silladb, sil a db
@thewhitefalcon8539
@thewhitefalcon8539 11 місяців тому
still a DB vs no longer a DB
@peckychicken
@peckychicken Рік тому
Wow discord are really committed to keeping their app up to date
@JoeyClover
@JoeyClover 11 місяців тому
The concern you described at the end is a well known anti-pattern called the Golden Hammer. Don't get too comfortable with the tools you know and use.
@NaviYT
@NaviYT Рік тому
I was today years old when I learned that Lewis makes long form content.
@joshburger2217
@joshburger2217 10 місяців тому
as a dev, just thinking about a migration of that scale is anxiety inducing
@somnathroy102
@somnathroy102 Рік тому
Thank god yt recommended this. I learned a lot.
@AhmedMi1ad
@AhmedMi1ad 9 місяців тому
Wow I really enjoyed the video, so insightful.
@MistaWu
@MistaWu 7 місяців тому
That was cool… awesome breakdown…🔥🔥
@YosephTeki
@YosephTeki 11 місяців тому
wow very very insightfull !!! thanks for making this video
@awsomevideoperson
@awsomevideoperson 10 місяців тому
I wonder if the issue with the normal Cassandra is really the GC, or the memory pressure Java currently has due go reference types. I suspect user defined primitives and fibers will really make Java a stronger contender in the high performance space
@po6577
@po6577 Рік тому
this story are amazing, great motivation to improve myself!
@avonzo
@avonzo 8 місяців тому
Great video. Great content. Thank you.
@varshard0
@varshard0 9 місяців тому
So informative and entertaining.
@javadevelopment6551
@javadevelopment6551 9 місяців тому
Great video, and I'm left with one question how to find and use the right tools at the right place and at the right time 😅 That something comes with experience or any beginning can also master that?
@nathan0401_
@nathan0401_ 11 місяців тому
This was very fascinating to watch
@costelinha1867
@costelinha1867 Рік тому
Bold of you to mention the Rust language and use it's logo in this video. Considering what the Rust Foundation has been up to recently.... AND YOU SHOW A PICTURE OF THE LOGO IN A DIFFERENT COLOR... oh my god, I hope the foundation doesn't see this.
@CodingWithLewis
@CodingWithLewis Рік тому
🤣🤣🤣
@pemessh
@pemessh Рік тому
Wow this is an awesome informative video. Please do keep them coming!!
@-hackers_industry
@-hackers_industry Рік тому
Could you do "A roadpath to become a software engineer" some time soon?
@tanmaypanadi1414
@tanmaypanadi1414 9 місяців тому
the editing on this channel has exponenetially increased in quality 🤩
@frogery
@frogery 6 місяців тому
love the animations on these videos
@orlovchik
@orlovchik Рік тому
thanks for the video!!! the question was in my mind this morning and, what a great recommendation, you gave me a full answer! but I have one more question why discord mobile app is so laggy and slow, while its web and desktop versions works fine
@weiSane
@weiSane Рік тому
I'm waiting for the day discord switches to vector databases when they have to store zetabytes of messages 😂
@avsync-live
@avsync-live Рік тому
I can see it happening. I’m sure the singularity will shake up technology a lot, and it does seem like vector databases will continue to become more popular
@felipheleal
@felipheleal Рік тому
Excellent video 👏🏼👍🏼
@MultiSciGeek
@MultiSciGeek 6 місяців тому
This is better than ASMR. Fell asleep in 8 minutes!
@shloktadilkar5536
@shloktadilkar5536 Рік тому
Very nice video Lewis
@IgnitedIce81
@IgnitedIce81 Рік тому
Never realized how incredible the engineering is on Discord
@mukithasan9684
@mukithasan9684 10 місяців тому
Today i learned a lot about database management how behind the scenes work
@roshangamage7749
@roshangamage7749 Рік тому
Great video ❤
@lisibethklair
@lisibethklair 7 місяців тому
every time I think about discord I get a horrendously awkward flashback to my first situationship :/ when he asked “do u have discord” I replied “about you staying or about you leaving?” 💀
@MyNameIsSalo
@MyNameIsSalo 3 місяці тому
have never put much thought into discord but they really are at the peak of social media, almost 0 downtime ever and always provides a consistent streamlined experience. I can imagine that they will end up doing many many more database migrations in the future as discord dominates the social media space. Wonder if there will ever be a permanent solution of a database which is truely infinitely expandable.
@bobtheoutbuilder8552
@bobtheoutbuilder8552 Рік тому
I can imagine reading discord messages from 47 years ago.
@nickhtt
@nickhtt 3 місяці тому
What a nice video. Thank you very much :)
@TomWhi
@TomWhi 11 місяців тому
Interesting video. I feel sorry for that underused blackboard though… it looked like it was going to be written on about 5 times with not a single mark ever made on it!
@bajagainsatish4828
@bajagainsatish4828 Рік тому
Awesome dude!!!
@ojasmittal8381
@ojasmittal8381 10 місяців тому
This guy basically turned Discord's blog into an interesting video 10/10.
@SusGus-rf8gm
@SusGus-rf8gm Рік тому
A few years ago, I don't remember why exactly but I had issues my phone plan and I was stuck with worse than dial up speeds for a while. Im talking low kilobytes per second, maybe even bytes. Discord was literally the only "online" app that still worked pretty much normally (besides embeds). That thing is optimized to death
@harrywang4769
@harrywang4769 9 місяців тому
yeah that's not it. the size of the data being sent and received has no correlation between how optimized the app is. your internet connection works for discord because the amount of data sent by discord is very small. it has nothing to do with app optimization.
@zuman_one
@zuman_one 4 місяці тому
Please Keep making such system design videos because I don’t like to read long articles 😂
@bideshbanerjee5506
@bideshbanerjee5506 Рік тому
I may be wrong but I heard from someone in a livestream that data service layer can be implemented in golang and the latency difference would still be unnoticeable
@ninjanape
@ninjanape 10 місяців тому
Great video. Subbed.
@IRazerIGhost
@IRazerIGhost Рік тому
Great video!
@CodingWithLewis
@CodingWithLewis Рік тому
Thanks for the visit
@skiper7795
@skiper7795 Рік тому
I love discord. Best comunicator out there
@Joshuahendrix
@Joshuahendrix Рік тому
Really enjoyed this one ☝️
@farhan-momin
@farhan-momin Рік тому
what a video! learning + entertainment
@allezvenga7617
@allezvenga7617 Рік тому
Thanks for your sharing
@oldschoolgaming6538
@oldschoolgaming6538 7 місяців тому
To be honest, Im glad they store them, I kinda use dc as a pseudo gdrive.
@musikSkool
@musikSkool 5 місяців тому
Discord is the perfect combination of Twitter, Facebook, an in-game chat bar, and even Instagram. If you pay them a little you can even treat it like UKposts or Twitch.
@blackbriarmead1966
@blackbriarmead1966 Рік тому
I love how you made the letters of coalescence alternate purple and yellow... like Moira's ultimate in Overwatch. This was intentional, right?
@CodingWithLewis
@CodingWithLewis Рік тому
Hmmmmm 🤔
@TheDingsBoms
@TheDingsBoms 9 місяців тому
Inspiring!
@willmil1199
@willmil1199 7 місяців тому
Love this channel
@varunshah3445
@varunshah3445 10 місяців тому
I have a questuon ... If they had written the data services first , would they still migrate from cassandra ? Considering that daya services was their bigger improvement ?
@shuaibalghazali3405
@shuaibalghazali3405 9 місяців тому
Just so amazing
@jlficken
@jlficken 7 місяців тому
I don't move to the "shiny new object" unless there's a very good reason to do so. I firmly believe "If it ain't broke...don't fix it until it is.". Then again I'm all alone in my little corner so experimenting just isn't something I have time for either.
@Terracronz
@Terracronz Рік тому
Cool video, earned a sub
@vanilla-plus
@vanilla-plus 10 місяців тому
Interesting colour choice on the Coalesce... What an interesting hypothesis.
@broneverasked
@broneverasked Рік тому
Lewis: *thinking he says NoSQL bad* Me: *says NoSquirrel*
@keflatspiral4633
@keflatspiral4633 9 місяців тому
i was thinking what do happen to the people that were working there ( for example on cassandra db) after migration? all went home and new people came in ?
@shruthimohanan669
@shruthimohanan669 Рік тому
This video makes me think Oh woww!! and got the shit out of me thinking about the developer stress all at the same time.
@DemiImp
@DemiImp Рік тому
You just altered the color of the rust logo. That is in violation of their trademark!!! 😂
@cjbht19minaret
@cjbht19minaret 9 місяців тому
Instead of storing strings why don’t servers store a hybrid of string / references to a set of strings stored in central locations. The references numbers could serve as a barcode of sorts to search for messages faster. Conversation could be pieced together with pieces of sentences or even full sentences. With the cutback of the amount of data being stored, more of the string storages could be stored at more nodes increasing performance. AI could be used to compile common phrases or sentences and assign it to a reference code or number.
@darpitrangari8207
@darpitrangari8207 Рік тому
Can you go into a little more details as to why they were facing difficulties?
@simonkimel3339
@simonkimel3339 Рік тому
This shows that Discord developers are really capable, yet the auto update on Linux doesn't work.
@Tudas
@Tudas Рік тому
Because Linux is literally completely irrelevant, atleast on the consumer market.
@oh-noe
@oh-noe Рік тому
@@Tudas lol so dumb
@lucass8119
@lucass8119 10 місяців тому
@@Tudas You would think Discord, a company that relies on 100% of their own servers running Linux and is a consumer of open-source products like Apache Cassandra, would put just a tad more care into the linux version.
@Nitidus
@Nitidus 10 місяців тому
​@@TudasI mean, there's the Steam Deck now. It definitely has become a LOT less irrelevant compared to a couple of years ago.
@umaradam3788
@umaradam3788 10 місяців тому
​@@Tudaswhat are you talking about. The entire consumer market is reliant on Linux. Facts.
@sebobebo10
@sebobebo10 Рік тому
I read the title and immediately thought, “I wonder how much storage space I can take up by just spamming emojis…”
@infinitelancer4612
@infinitelancer4612 Рік тому
That's awesome
@CodingWithLewis
@CodingWithLewis Рік тому
More than awesome!
@god_of_everything9644
@god_of_everything9644 7 місяців тому
Got urself a sub sir
@xeridea
@xeridea 2 місяці тому
Any info on performance with Cassandra with Data Services?
@adill6295
@adill6295 Рік тому
Can you make a video on rust, or how to learn rust, free courses or tools.
@404statuscode
@404statuscode Рік тому
Make a video on how Google sheets can handle Sheets that are thousands of rows long without even the user noticing it. For example How it can load the data faster(even with my crappy internet). I know that they will be using compressions and things. But a video or a short would be nice explaining it.
@martinmc0950
@martinmc0950 Рік тому
It’s just text. Your average UKposts video is harder to load
@Tudas
@Tudas Рік тому
​@@martinmc0950 It's not just text lmao, did you ever see Sheets that are fking huge? But Excel does that even better, NASA literally uses Excel for huge ass calculations.
@martinmc0950
@martinmc0950 Рік тому
@@Tudas just because the sheets are big doesn’t mean it’s not text??? Although I see where you’re coming from with the whole calculation thing, using a computer for math is always going to be effective
@rohithgupta9672
@rohithgupta9672 6 місяців тому
I did build multiple grpc channels/servers in my org.It's neither straight forward nor fit for all use cases, GRPC is primarily used for streaming like a stub but load balancing such endpoints is a nightmare. Not sure why they were using GRPC in such system though. Whenever you create a GRPC connection it will block a thread until the connection is closed. If you create a one connection per channel you wont have enough threads in the VM of the instance running the GRPC client service. I think it is not a good idea to always have a open connection with database.
@kumalalalalalalaa7045
@kumalalalalalalaa7045 Рік тому
Love these videos
How Instagram Stores BILLIONS of Videos
11:39
Coding with Lewis
Переглядів 214 тис.
How a Random Programmer FIXED GTA 5
13:09
Coding with Lewis
Переглядів 578 тис.
The Wiiiiiiiiiiiiiiide Gaming Setup
23:53
Linus Tech Tips
Переглядів 1 млн
These Apps Cost $140. So I Rebuilt them for FREE.
14:18
Coding with Lewis
Переглядів 102 тис.
I Trained an AI with 10,000 Memes
14:52
Coding with Lewis
Переглядів 163
Am I Overreacting? Live with Dadvocate
The Dadvocate
Переглядів 267
What's the Discord HQ like? // office tour + meet the team!
16:23
Chloe Shih
Переглядів 45 тис.
The Bug that Broke Reddit for 314 Minutes
14:42
Coding with Lewis
Переглядів 45 тис.
How Discord Managed To Store BILLIONS Of Messages With This Database
9:00
The purest coding style, where bugs are near impossible
10:25
Coderized
Переглядів 817 тис.
Inside a Google data center
5:28
Google Workspace
Переглядів 21 млн
Наушники Ой🤣
0:26
Listen_pods
Переглядів 339 тис.
Портативная PS 5 🎮 #ps5 #expressly
0:22
ExpresSLY Shorts
Переглядів 216 тис.
С Какой Высоты Разобьётся NOKIA3310 ?!😳
0:43
Вы поможете украсть ваш iPhone
0:56
Romancev768
Переглядів 152 тис.
Which Phone Unlock Code Will You Choose? 🤔️
0:12
Game9bit
Переглядів 6 млн