Next video: "we lost all our data"

“How’s the edit going? I’m holding your file server” was the best line 😂. And then when Jake walked over with a second node….*chefs kiss*

This is lowkey one of the most interesting videos they have done on this channel in a long time, in terms of tangible, world-changing technology. The utility of that AI search tool is so unfathomably insane

"I'm holding your server" "Is it on Wi-Fi?" Absolutely amazing

cannot lose data, until Linus somehow manages to drop it eventually

I get it now. You need two of everything for proper redundancy. Elijah can drop stuff when Linus isn't there. Elijah can mess stuff up when Dennis isn't there. Good hire, LMG.

As a senior data center engineer in a multi billion dollar enterprise I really enjoy such content (with a bit of envy), while we still have to deal with the old school netapp metro clusters, HPE XP8 or IBMs SAN volume controllers you guys can play with the innovative stuff, even if its totally overkill for your usecase.

This is easily one of the most beneficial and practical uses of AI I've seen. Eliminating the tedious work of sifting through thousands of hours of content to find something relevant? Yes, please.

My favorite LTT videos are the ones with Jake explaining networking that I do not understand at all

THE INTRO HAS RETURNED. Thank you Dan.

I must say, Linus Tech Tips has truly outdone themselves this year! The improvement in content quality is simply outstanding. It's incredibly noticeable how much work you've put into enhancing everything from writing to video production. The videos now are on a whole new level of coolness! Guys, you were good before, but now you're next-next-next level! Kudos to the LTT team for continuously raising the bar and delivering top-notch content.

Jake looks fantastic and I'm so proud of him and the effort he has clearly put in. Good on you man.

My home desktop cannot lose data as well - in the sense that if something were to fail, I'm too broke to afford redundant storage. Enterprise reliability is nuts, it's great to watch :)

Okay… the server was neat but HOLY COW that sorta AI search program thing at the end was unbelievable. One of the coolest products I’ve ever seen. That’s an actual game changer for folks who have a lotttttt of data and through which it’s not easy to search.

As someone who built software like Weka for ~8 years, I appreciate how excited you are about these insanely complex systems and the performance they bring.

"What could go wrong?" "I mean, a lot." Ah, it's always a good sign when those words are being said on the _other_ side of the screen 😂 Glad to see Jake back again! Had been thinking it's been a while

Videos like this are proof that Jake is a very valuable employee. Guy knows his IT stuff.

That Epyc CPU could hold the entire system memory of a Win 2000 machine in its L3 cache. Can you even imagine how fast it would be if your entire kernel AND workload is in CPU cache, and main memory isn't even being used!? That's wild.

Having an engineer like Jake around can make a world of difference for your company

so much energy you got there, mate. Thanks

8:25 linus gesturing with the server sent chills down my spine.

THE MAGIC OF BUYING TWO OF THEM: like all proper nerds of this vintage, the idea of anything being dual is just ruddy exciting: dual cpu, dual hard disk, dual gpu, dual dual server with dual dual dual dual psu…

That software they use to search for videos is insane! It's pretty impressive at how accurate their searches were and how the thumbnails matched.

The concept of cluster voting was developped by Digital Equipment Corp in the 1980s for its VMS clusters. The votes and quorum mechanism did not handle just failure of a node, but more importantly prevented a node that left the cluster from continuing to function (thus proventing it from accessing drives without enforcing locks/synchrinization with the other nodes). (called cluster partitioning). In the case of a 2 node cluster, you could designate a drive as quorum disk and give it votes, so any node in the cluster that still saw that/those drives would get the votes and could then continue. (in VMS clusters, drives could either be served by a node (and accessed from other nodes via network) or be standalone with direct hardware links to each node. (Digital developped extention to SCSI (called DSSI) to allow multiple hosts to access the same SCSI bus so multiple nodes coul write directly to the same physical drives). (later on, I beleive SCSI did get that functionality). The freezing of a node to prevent partitioning is very clitical to prevent a node that lost its connection to the other nodes from continuing to write to disks or perform tasks without the synchronization with the other cluster nodes. (in a cluster, there was a single lock database so when a process on a node too a lock on a file (or area of file) or a resource, that lock would exist on all nodes so processes on other nodes would not get that lock until first one released it). From a failover POV: You can have 2 nodes doing work since the locking is synchronized, so when one node fails, the other one continues and new network connectiosn all go to the second node instead of being split between the two. The other way is to have node 1 take and get lock and do all the work. Node 2 requests the lock but is put on hold because lock is unavailable. When node 1 goes down, node 2 gets the lcok and then processes all the work. Tandem NonStop (used for mission critical stuff such as Interac/credit card) has different fault tolerance. In the same chassis, a process would run on a single CPU/RAM, but a copy would run on a designated CPU/RAM, but basically have it writes to disk/network disabled. Both processes got the same data from network/disk. So the backup was a copy of the main one, and shoudl main one go down, the backup tok over right where the backup failed since the backup had identical RAM, process state and conections to network etc. However, while VMS clusters could be spread across multiple buildings (and up to 96 nodes), the Tandem Nonstop was within a single chassis in 1 computer room. so fault tolerant within chassis, but not disaster tolerant. A mere smb file server just blindly executes writes/reads from any node to any area of disk/file and when you have different windows instances accessing the same time, the last one writing to it wins.

I’m so glad you finally made this video. I have always wondered how the editors find relevant archive footage. You’re telling me it was just from memory this whole time?!

Last time you lost data it was not because of lack of failsafes, but rather the fact that no one was keeping an eye on things. This may prevent failures when something goes wrong, but only if you actually respond to those failures in time. Make sure to have some sort of alert system this time around.

It would be interesting to see Ceph running on this thing. CERN uses Ceph for data from the LHC.

That axle AI is crazy! Can't wait to see more historical clips showing up in new videos!

To Add to the split brain talk, you need an odd number of nodes to guarantee they can reach quorum. Running only 3 nodes also creates issues when there is a failure as you now can't reach quorum and need to immediately get the node back up. When running 5//7/9 nodes outages are not nearly as urgent .

You're killing me linus 8:27 like holy you made my heart drop.

20:40 OK that is definitely the best feature so far, this era of information there are lots, but to actually get one that is relevant is the true power.

really enjoying the intro being back!

Great to see professional server stuff shown to the public. One small correction though: even though the ConnectX-6 cards have 2x200G ports, the card itself is limited to a total of 200G by its chip, and both ports have to share the total bandwidth.

Love the server hardware videos.. keep em coming! Supermicro web browsing, just for fun do exist

See you guys in a year when the server loses data

Dad, and son nerding out over servers. Warms my heart

It makes me so happy seeing the intro in so many new videos. It's the best intro on youtube, and screw viewer retention, I will watch it!

I love the server videos, so much crazy tech being used is amazing to watch

These networking server infrastructure videos are my guilty pleasure

THE INTRO IS BACK PEOPLE!

I REALLY want a Floatplane extra for this which dives DEEP into this stuff. I don't care if it's 3,5 hours long like the FP Exclusive of Kyle. THIS. IS. AWESOME!

This is the kind of stuff that is exciting for me in IT. Although I would not go this extreme and aim for maximum value for money, it is fun to watch.

Haha, that's amazing. Congrats LTT, Supermicro, and Weka for this killer demo!

...the ending is pure gold: "..see all the lights, they never stopped blinking." :D such a powerful statement, so true for any debugging process no matter who you are!

I absolutely love these server oriented videos! Please never shy away from doing more! I almost wish you guys had a specialized channel for this kind of stuff and just let Jake geek out. Great stuff, LTT!

I remember the first video from Linus Tech Tips that got me hooked to the channel - the first Petabyte project. Even after all these years it is still as entertaining as it ever was to watch these guys do what they enjoy. Thanks to Linus Media Group, I am now pursuing a career in IT. I feel grateful to have come across this channel because it has formed who I am as a person now. Keep up the good work.

The one thing I really take away from this one: "New-new-new-new-new-new-new-new-new-Whonnock goes REAL BRRRRRRRRRRRRRRRRR!" The amount of knowledge and expertise involved in these projects exceeds my level of computing soooo far at this point^^ Still fun to watch!

I just decommissioned something similar at work about 2 months ago these are awesome and super reliable. The ones we decommissioned were used for high availability to radiology records in a major hospital, ran for 7 years with no issue and was only replaced for a standard life cycle upgrade of the system!

this is a great video to show how AI could be useful in an SMB - Small to Medium Business, not the shared drive :D I kinda wish you hadn't buried it 20 minutes into the video, but you had to, so we could see the technology that makes it reasonable to do. This is actually one of the most relevant videos of yours for me in what I do in my professional life. Good work.

I am a small homelabber, and I love only these types of videos from LMG....I watch all of their server room/ servers/ storage kind of videos. I first thought they were gonna use Ceph but looks like Weka is super cool too.

That AI footage search is crazy cool. That type of stuff is definitely the positive side of AI/machine learning hardware and tech etc.

LOVE THE RETURN OF THE INTRO

That time jump at the beginning hooked me like a fish and I watched the whole video I love infrastructure content and other business related IT! I could watch it all day.

I'd love a video on the evolution of Whonnock over time. I think that's good for learning how to evolve our home server storage over time. Fun fact: one of the Whonnock videos was the first LTT video I ever saw.

Seeing the software made me really wish for someone to build a consumer storage classifier. Like just that part, you could point it to a file in your regular file system, give it somewhere to store all the generated information, and be able to manage all your pictures and videos.

Genuinely need so much more of jake and this awesome nerdy server content.

Some C-suite at Supermicro when Linus and Jake pulled the servers. "Why do we work with these guys?" Then again, it proves how good the setup is!

the truly nice feature to high availability is not even just the redundancy but also the fact that when you want to make upgrades or any kind of changes or even just performing a lot of software updates you can pull each machine 1 by 1 and do them in a localized setting so as to not create bottlenecks otherwise for everyone trying to read/write data constantly. something that Valve Corp hasn't even learned as in 2024 there are still a lot of downtime across all services on Steam every Tuesday.

15:43 Jake is referring to the fact that the latency accounts for the fact that they are using DPDK (Data Plane Development Kitty) to facilitate RDMA (Remote Direct Memory Access): basically running RDMA through software, and accounting for the latency as a result (I.e, in the userspace rather than at a kennel level). This is pretty fantastic!

Obligatory ceph mention

Now, for true redundancy, do a nightly mirror/backup over fiber optic or microwave link to an identical setup in the labs building. Call it Whonnock 10_2. God forbid another UPS or power bus catch fire in the server room. Or Linus pull 3 servers for a fail over demo by mistake.

It would be kinda sick to see an educational series on IT engineering and deployment given all of the amazing professionals working on projects like this on the front and back end.

One thing I love about these server videos is that, unlike the high end PC builds you make, I might actually interact with one of these! Obviously not of my own, but as a professional software engineer deploying stuff for an employer. Even just making decisions based on what hardware bottlenecks will appear for different types of services! And the best part is not having to spend your own money on the outcome 😂

Oh wow! It's an actual use for large language models and training that's ethical, reasonable, and useful! Incredible!

3 years from now: How we lost all our data

Holy frick... that AI scanning is hella dope. Excited for the road ahead for ya'll.

Jake is the kind of employee that brings real value to an organization. Vote party for maximum raise at performance review time! 👏🏾👏🏾💯🔥

This really sounds like an unsinkable ship! Good job guys!

Also check out Ceph I'd say. It's neat.

I don't know why, but I forget how much networking/nas that Jake knows. When he's the one setting most of it up. Good job Jake.

Thank you Jake, for calling me awesome!! :D

Linus: "Here it is, the final form" _Less then 1 year later_ Linus:

"They don't know what HA means" Me.. working in IT for 25 years: "HA means nothing if its not distributed and works as a failover cluster in a DR event" TLDR; It's still not secure, it's not protected against floods, fires, or other disasters. It still lives in a closet. The system needs to be distributed physically before it can be called HA.

God I love the intro coming back to LTT videos! That cut from the intermission bleep to the intro had me lmao 🤣 1:38

First time hearing about Weka. Literal game changer. Very impressive.

Jake and server upgrade video ? My favorite ngl more jake pls

I love when LTT does server stuff. Keep em coming PLEASE

So glad the intro is back!

Honestly that Ai clip searching feature is the first useful, non-harmful to creators, implementation of AI I have ever seen.

Thank you for including the intro again :D all i got to say atm :P

Jake with a full beard is a much better look for him!

Best video ending ever, please tell us we are awesome more often. I love being an awesome viewer!

I have no idea about servers or storage but i still enjoy watching these

Whonnock 10 Last whonnock ever, until whonnock 11

12:41 I laughed way harder than i should have at that linus pillow

That was an amazing video for me. Amazing solution from hardware to file system to performance and even AI

Can you guys do a recap video of the realistic timeline/data requirements for an ongoing business that compliments this video by recapping every video like this one in your past(like all 7 or so with the title "we don't have enough space", "winnock server died", "we're upgrading again!") You've done many of these videos with the same theme, and a long term analysis of "Did we meet the goals we set out to achieve" with these upgrades over the past decade would be very fascinating and useful for the industry. Obv you are a special case since you output video but it would be cool to get some insight on both the long term decision making and how it's changed every time you do one of these.

Let’s get some popcorn and watch Linus drop it

It's been a while since I've been wow'd by anything here. The Axle AI thing deserves its own piece.

The capabilities of WCA are mind-blowing, and the potential for AI-assisted search is really promising.

You should try an IBM server sometimes!

Have a great day everyone!

I absolutely love the PTM7950. I use it on my laptop and on my GPU and temps dropped significantly compared to using thermal paste. It's like magic.

26:52 "Is this on Wi-Fi?" -Mark 2024 🤣

I love the not jank era of Linus tech tips. This is more like a proper corporate setup than home lab setup.

Grate video

This video totally blew my mind... I didn't know technology was thus fast yet

This is a legitimately insane trio of cool new technologies that all add up to be greater than the sum of their parts: Supermicro's insane node architecture, BLAZING fast AND large capacity NVMe drives running ON A FILESYSTEM MEANT FOR NVMe, and that insanely cool AI detection software. I can easily see how just four rack units worth of servers, including these HA servers, GPU rendering servers, email/AD servers, and cold storage servers could conceivably run the entire infrstructure a major VFX house, versus previously needing a literal datacenter full of equipment. 1024 cores, 24TB of RAM, and 3PB of flash in 4U work of rackspace running at 100GB/s WITH a filesystem (over a network using non-optimal NICs) is WILD.

Jake is looking great. Looks like he’s been hitting the gym quite a bit

They got WEKA?!? I'm so jealous! I've always wanted to use this but you just can't run it for the homelab!!!! P.S. How do you backup a weka-system?