More bottlenecks than a Coca Cola factory.

60 HDDs in RAID 0 is the definition of "all gas no brakes"

I had this colleague who told this story: I had boss who did not understand anything about computer or electronics. When ever I was trouble shooting something he would come and watch over my shoulder and comment "Have you checked the power?" This was very annoying as he did not understand anything. What made it doubly annoying that his advice was spot on so many times. Check the power!!

This is a bonkers crazy setup. It's so mad I just had to watch, 100%. Kudos to you for getting it to work at all. Your persistence is inspiring.

A petabyte and a raspberry pi, the crossover we didn’t know we needed. This is sick.

the PetaPi seems like a winner, I can only imagine the folks at 45 drives watching this with a mix of awe and horror.

Pushing the limits of "hackey" tech is what most hardware and software engineers should be shooting for. Very well done. Kudos to 45 Drives!

Fascinating! A lot of work! The issue it seems is having to tether to the hard-disk through a band shared with other drives, rather than universal. There is a technique where using data-points written to each hard-drive mathematical computations can rewrite and measure the data for inference machiningly- without even using the shared io-bus anymore! You’d see a slowdown in computation monologue but all speed ahead on as many drives as you want!

Woah.

Do I need a petabyte of storage? No. Would I mount one in my rack? Yes.

Even though the idea is crazy, Jeff knows his craft. The tips at time 09:00 on how to select hard drives for the task are priceless!

Jeff you never cease to deliver, you're an absolute legend, great video!!

How about one controller to one pi, then stripe 4 pis together, that should increase the throughput and scale down the issue to a more managable chunk, and still a pi project.

"I was going to unbox this on camera but Fedex already did" As someone who's had multiple expensive items destroyed by Fedex and actively avoids doing business with companies that ship with them, I felt that...

As someone who is about to endeavor on a new NAS project, this was a fun watch! Thanks!

It's impressive that this works at all.

I think i heard Seagate in the US having a heart attack watching drives being juggled. I live in Taiwan.

Linus would be proud

You are an absolute madman and I'm rooting for you the whole way

Jeff another wonderful video. Keep up the great work.

1942: "We need to figure out a solution to digitally store dozens of bytes at a time. Vaccuum tubes, maybe? This is going to cost us millions, but it will be worth it to finally have accurate artillery range tables!" 2022: "I'm going to hook up this petabyte of data storage to this cheap single board computer!"

I always like to think that every time Jeff publishes a new video, the raspberry pi design team feels a disturbance in the force.

This is an awesome project. Makes me wanna build a large storage server.

As an HDD engineer usually drives are built to compensate for vibration due to certain fan RPMs. Especially if we have a big customer we'll optimize things for the frequencies of vibration in their trays.

I hope LTT sees this. Great stuff as always, Jeff!

Oh my goodness, that juggling of those drives.😳

I just completed a 24 X 18T build that's almost 1/2 PB. I bought the drives a few at a time all Segate recertified it seems to be the sweet spot in price for me. I had to upgrade and rebuild things several times. Your video was just like my experience switching OS, FS, etc. I had endless drives/arrays just dropping out, mostly on startup. I tried Fedora, Centos, Open Suse, Suse JeOs, and Ubuntu Server all let me down for one reason or another. I tried them with various shares, raid arrays and file systems; plus not all would run my app. I ultimately got it working with Ubuntu workstation and the Ubuntu share - no samba. I'm not happy with ZFS. I had to kill the swap and add more RAM (which meant a new motherboard) to keep ZFS cache from crashing. I solved the dropouts problem by putting the drives really close to the host board and using short/expensive data cables that are all the same length. I'm using a very old Athlon FX-8300 8 core and 64 gig of Ram. I found a great last generation Adaptec 52445 raid card new old stock. I had to install 2 power supplies and rewire one to all molex to get enough amperage on the 5 volt rail. New 1200 watt supplies have plenty of 12v but almost no 5v power. I also upgraded to 2.5 gbps network card. The write speed is NOT stellar. With Raid0 it goes real fast at first 450mbps filling all that cache, but slows down to about 150mbps. With a single JBOD I only get 130mbps, two drives at the same time still go 130 each, and I can transfer to 4 drives at the same time before it bogs down the network and write speed drops to 70 each drive or 280 total. I already got 4 sas expanders and plan to continue adding drives (and power supplies) up to 2.5 PB. My box is an old IBM 2401 tape drive converted to rack space. I yell at the You Tube screen, not my computers. That's not true I also yell at my computer at work (it's windows).

This tutorial helped me so much!! Thanks!

I saw you posted some on the homelab subreddit last week, THIS is what you were hiding from us?? What a fun idea. Can't wait to see your next project!

You tried all the things I wanted to see. You know your audience!

I think Arthur C Clarke hypothesized that this would be enough to store a few people's minds into it

Well, pairing €20.000 - €24.000 worth of drives with a 100 dollar raspberry seemed like a solid plan to start with. Having it run a 1pb raid0 config while hosting that storage seems like a task made for this little arm processor 😂 Nice to see you try out such extreme things with the raspberry😀

Glad to see Red Shirt Jeff back. I was wondering what happened to him. Geez, your projects are soooo extreme and cutting edge. Your troubleshooting processes are very informative and helpful to your viewers, who otherwise wouldn't have a clue where to look. We never see that on "How to" setup videos, where everything just automagically works. BTW, I recently used Styrofoam to separate 2 prototype boards while testing them out. It got hotter than expected and the foam sagged, allowing the power rail of one board to touch the other board, which fried as soon as I powered up the following morning. Don't try this at home, folks. Cardboard insulation good. Styrofoam bad. Thanks for sharing Jeff. I'm fully confident that you'll get your bandwidth up on the 60 HDD RAID. Looks to me like you're already nearly there. Instead of installing it inside a server rack, perhaps a locked cage would be a better idea to protect it from Red Shirt Jeff. 😎

Jeff, you have carved yourself a niche channel in an overcrowded tech community, taking the Pi to new heights in every video. Keep it up. Loving every video. I'm left astounded on the Pi's capabilities and untapped potential.

I run something similar but without the pci-express stuff. Only usb2sata adapters. I am using a german one, that has a power supply that gives 2 amps for 12 volt circuit of the hdd, and 2 amps for the 5 volt circuit, therefor not using power from the PI usb at all. 2 harddrives per pi, running in simple raid1, xfs. I mostly use nfs instead of samba and for the most part it saturates 1GB connection (exception being many small files... but it's common). Less so with writing, where sometimes goes a bit under (as it has to write on 2 different drives and sometimes it falls to 700-800ish MB/s), but other than that, pretty practical. I've been using this setup since the pi4 came out and am very happy with it, especially when it comes to the electric bill. I tried connecting all of them to a single pi, but there were stability issues at times. Might have been the usb hub I was using, or some of the cables, so reverted back to 2 harddrives per pi, one in each usb3 port.

Hey Jeff, I’ve been using your ansible roles for almost 10 years. Love seeing you around HN and the other traps, great to see your UKposts channel is doing so well. Very cool projects!

When I first saw the title I figured this would be one of your massive multiprocessor/multi-Pi projects combined with a massive amount of storage. Something closer to 20 TB per Raspberry Pi. At 60 Drives and 60 Raspberry Pi's, that would still be way beyond any normal homebrew project.

This showed up in my recommendations. I have no clue what happened here and I know nothing about data managment or IT in general. But I watched it to the end not realizing that this vid is 22 minutes long 😅good content 9/10

Hu, I know that modern cases have that 1 stud so you can place your motherboard in on the stud which makes it easier to align the rest of the screws, but even then every motherboard I have installed has had 8 - 9 screw holes, sometimes there is a heatsink or something in the way of having all 9 screw holes, but 8 is the minium I have seen, this is only for full ATX though not ITX / mATX.

Many Pi’s running something like Minio might allow for some interesting single box hardware redundancy. Also might be able to get over the 1Gb limitation since you’d have many pi’s each with their own 1Gb connections.

Jeff Geerling: My storage setup registers on the Richter Scale

OK, this is the kind of cool content I like seeing appear on my birthday! (even if I didn't see it until 5 days later.) Just ONE of those drives is more storage than everything in my house, and I do Unixy networky stuff for a living. :)

Ripps out high end setup for a PI! You Monster! Though a Fun Project to play with a pi, maybe in the future with a PI 14 hope you enjoy your new Perabyte Server

Maybe the CABD order is like the firing sequence of a 4-cylinder petrol engine lol 🤔

One thing I saw you doing when wiring up the NAS was that you connected the molex adapters to two sata power connectors on the same line. If the psu has another line, try connecting to that, as each line can draw a limited amount of current and there might be an issue there. Then again, the issue might be anywhere else but that's a thing you can easily try

Jeff, I'm catching up with thx, since I've admired your ability for a while. I am an old geek and you do magic. Reminds me of my S-100 days with CromixOS.

My God! 20TB 60HDD I never dreamed of such a volume

Be careful with those sata to molex adapters...they are prone to fires!

Amazing content as always, keep up the great work!

Yet another absurd carage setup. Just love it!

Awesome video & project!

Well Jeff, one thing's for sure, if you and others aren't pushing the pi to bleed on the edge, no progress will ever be made in this direction. I'm not sure what kind of useful stuff this direction will yield, but it surely will yield something. Keep on pushing ya madlad!

It almost hurts to watch the board being removed. I've been wishing for a storeage like this for decades. Still a very interesting project.

Ive always wondered if there was a way to make a external ram board like hp has one that adds 4 ram slots on a pcie. Ive ran into issues of bottlenecked performance not because of cpu but because ram capped out. The pi has only max of 8gb so i would assume max speed on drives could only be met with maybe 6 to 10 drives at most. If you were to add an additional 128gb i wonder if that would help out. Plus the raid cards you had i couldnt tell if they had their own memory or passthrough.

Jeff - I mourn you missed seizing the opportunity to officially name this The Pi-tabyte Project - a portmanteau teed-up for a long drive, but you duffed it, lol. - great vid, keep 'em coming, I'm having a grand time doing some of your projects. Cheers.

This is amazing. Good job charming 45drives into sending the case! They must be Red Shirt fans 🤣

Even before doing a raid or zfs test; I’d have run a single drive test (either benchmark or simple linear read/write). Loop that 60 times and see success. Then repeat with two drives in parallel and loop 30 times. Redo with three in parallel 20 times, etc etc. When you start seeing failures you really will see the cause-effect point. There is technically no reason why this won’t work with 60 drives - if you ignore performance. Any bugs that are exposed that can be fixed will simply improve the base users world. This is an awesome test that pushes the RPi and kernel to the limit. Making this work at that limit helps all of us just running one drive. Plus, the errors you saw, as bad as they were, should somehow restart the drive (without a reboot).

Next week: A 1024 CPU server running off a 1.44 MB floppy disk.

The real impressive thing ist that he was able to get a pi compute module 4 I am trying to find one now for almost a year but the only place I even found one listed was on Amazon in the us, it is sold out since then, here in europe I did not even find a seller who listed it

My NAS is a pi4 with a desoldered USB chip to expose the PCIe bus. It's connected to a PCIe switch(to improve transmission and prevent crashes) and then to a cheap ASMedia SATA controller. The kernel is also patched to force PCIe gen 1 to prevent crashes(this is probably the same thing that happened to you in the video btw). PCIe gen1 is still faster than gigabit so it doesn't matter. Over samba or NFS I can get the full gigabit speed even on large data transfers so no bottlenecks there. Would I recommend this setup? no. Does it work? hell yeah (longest uptime was like 90 days or something and then a power outage killed it, I need to get(or more likely make) a UPS, I know). If anyone wants photos lmk

Peta-Pi go BRRRRR Im genuinely surprised this worked, very impressive. As for what to do with it, a video on downloading/hosting a local copy of Wikipedia would be pretty cool.

The slow storage movement. Bytes aged to perfection, not for those in a hurry but for folks who appreciate a good long wait for quality unbalanced spectacle in design.

Thanks for this idea to store my stuff.

Absolutely fascinating and completely, totally barmy! What an immense amount of work you obviously put in to this - research, sweet talking 45Drives, research, RAID solutions, research, talking to 45Drives techs, research, etc. I am *extremely* impressed, both by you AND by 45Drives for their courage! And my abiding thought? "Chassis" is pronounced "shassey", not "chassey"! Pfft! So I looked up chassis in Cambridge Dictionary to back my obviously accurate opinion and ... well, blow me, North Americans really do say "chassey"! Every day is a learning day! Pedantry isn't good

60x USB-connected HDDs... I'm telling you, a missed opportunity :D

I have a Pi running RAID1 with 2 External SSDs and on times I have to reboot as it seems to forget that the disk exists. That's Pi 4 and Samba and a Window share. So to get as far as you did its amazing.

i was guessing and hoping this was the project in progress when i saw the rack update video :D

I have smaller versions of those EXOS drives. >200 drives, over the past 5 years, I’ve had 4 failures. 3 covered under RMA. 1 had just expired.

Could we please get an updated video with the pi 5 and see how the two setups compare

This guy is the only human capable of winning in a debate with Data from star trek.

"I was thinking about unboxing it on camera but fedex already did". Best punchline :'-)

**CM4:** Hey Jeff! What are we going to do today? **Jeff:** You will handle 60 enterprise grade hard drives. **CM4:** Oof

Hi there! I work for Infomaniak, and I am managing storage networks. We offer backup services and connected drive (we call it kDrive, it's a bit like google drive, just we protect your data and they are hosted in Switzerland). Adding 96 Exos 20TB HDD in my swift storage cluster is what I do every day. I manage "moderately large" Swift clusters. On them, we add 6 2U servers at a time, each of them holding 16 HDD (so 6*16 = 96 HDDs). In total, I calculated that I am managing more than 5000 spinning drives in our OpenStack swift clusters, which amounts probably around 100PB (so not just 1 PB like you're doing...). Just this week, I added 12 HDD storage nodes, and 6 proxies (with 2x SSD each, plus the system drive), and 9x NVMe storage nodes (10 NVMe each, to be used in a Ceph cluster). Your idea of a Raspbery-Pi is fun, but instead of one RPi for all HDDs, I would setup one RPi *PER* HDD, and then it all makes sense, and you may have decent speed. BTW the casing you bought seems of very bad quality compared to what we get from HPe or Lenovo. I've been running this service for nearly 5 years now, and we never lost a single bit of data... :)

It was only about time that the new storage measurement of reference was going to be the petapite... I'll show myself out :D

Jeff doesn't have the word "why" in his vocabulary :) (Just kidding, things like this are fun, which is "why" enough for me).

45drives marketing team is on a roll

Integrating a Pi compute module with Ethernet into each hard drive can make scaling easier. It lets each drive connect to a network independently, simplifying data handling in big storage setups.

Very interesting watching "The PP Project"! :D

As a St. Vincent fan, I'm calling it "Pietabyte" whether anybody else does or not.

I'd almost be interested in seeing each raid card assigned to 1 PI and then them clustered together.

this is amazing

Hi - im interrested in the "working mat" on your table - where i can get this? And thanks for the entertaining clips!

I am new to the channel and this was my first video of yours! On simple question...how does one...learn about everything you just did and said in this video. It’s all so interesting but so much of it is over my head...

I would really like to see you build a cluster based file system like gluster or ceph, run that on a pi cluster with 2 to 4 drives per pi, then show performance

Next step: 3.14PB on a PI

OMG the original hardware is so beautiful it brought a tear to my eye to see it removed. Given the resources I would have 2 or 3, one for local redundancy and one for off site but given net speeds the off site would mostly be pointless. And no I don't need one, I just want one so bad it feels like I need it :)

Naive guess: regarding the harddrives going offline/faulting when using other schemes than sequential/linear, it might just be a power issue where addressing that many discs trough the PCIE cards might draw too much current and the cards get below a voltage threshold on the expansion board.

How many times did you recompile the kernel?

If lsblk starts to run out of letters and shows drives as "sdaa, sdab, sdac" etc. you know you have a data hoarding problem.

Hey! 45 Drives is here in my home town!

I am afraid someone addicted to this work is called a petaphile person! 😱 But great, that there are people doing these kind of projects!

Would love to see this with a RISC V processor

"I grabbed a small piece of cardboard to insulate the boards from each other. At least temporarily." Yeah. Sure. I think red shirt Jeff was trying the break back into the room.

I built my first datahoarder set up using nothing but 8tb SSDs XD a whole bunch of them. solves the issues of "spinning" in an easy way

Super Projekt !

They must really trust you to borrow $35k+ worth of disks

*happy linus noises*

Yay! Tiny computers doing huge things!!! 🔥🔥🔥

I'M SO INTERESTED IN THIS PETABYTE THING

dang, fedex unboxed it first huh. not surprised