Raspberry Pi Cluster Ep 1 - Introduction to Clustering
Переглядів 705,870
День томуIn this video, I introduce the concept of Raspberry Pi clustering, and cluster computing in general. Technology like Beowulf clusters and Kubernetes has made it easy to build clusters using off-the-shelf components.
I walk through how I built my 'Pi Dramble' (www.pidramble.com) over the years using various Raspberry Pi model B computers, and I compare a set of Raspberry Pis to a giant 'ThreadRipper' CPU, leading up to an introduction to the Turing Pi, a game-changing Raspberry Pi-based cluster board.
Sponsor me on GitHub: github.com/sponsors/geerlingguy
Sponsor me on Patreon: / geerlingguy
More info about the Pi Cluster series and links relating to topics discussed in the video: www.jeffgeerling.com/blog/202...
Contents:
00:00 - Intro
00:43 - What is a cluster?
02:08 - Two problems scaling horizontally
02:44 - My Raspberry Pi 'Dramble'
04:16 - Close Encounters of the Ansible Kind
05:10 - Why build a Raspberry Pi Cluster?
06:48 - Resource constraints help you learn
07:33 - Outtro
@elistatham6876 2 роки тому
"Since my bramble ran droople, i came up with the portmanteau, dramble" - The Sims 4
@jaredj631 2 роки тому
I literally came down here to post the exact comment. Thanks for covering my nines LOL
@stickmakerman 2 роки тому
Drupal
@watsonb2008 4 роки тому
Jeff, you are an amazing teacher and have been very instrumental in expanding the Ansible community. Thank you so much for these videos.
@otteydw 4 роки тому
Great explanation. I appreciate that you explained simple things like what a LAMP stack is, instead of assuming the audience would know.
@wkGeoff 3 роки тому
I appreciate that sentiment. I did not know what that meant until I watched this, and then researched it.
@truedreams1 3 роки тому
Is the LAMP stack still around? One of my first projects as a kid was setting up the LAMP stack, when I was done I was like: Now what do I do with it?
@univera1111 3 роки тому
@@truedreams1 have you gotten what to do with it.
@truedreams1 3 роки тому
@@univera1111 Well it's mostly useful for companies, in the late 90's I was following a guide to set up a webshop, but setting up a webshop doesn't make you a company. Like cooking food don't make you a restaurant, there's much more to it. A more modern version of the LAMP would be L for Linux (unchanged) you can run basically any application on it no license required, A for Azure or AWS (cloud) instead of Apache, I know those aren't the same. M for MongoDB instead of MySQL, and P for Python instead of PHP.
@univera1111 3 роки тому
@@truedreams1 so in a nutshell it's for web hosting of any services I can provide?
@rodneyflores1952 3 роки тому
This is everything I've wanted. Bought a ClusterHat a year ago and had no idea what do with it. Informative and encouraging, I don't feel so lost.
@silverywingsagain 3 роки тому
I've been doing devops with ansible since 2018 and I learned a lot of it from roles that you wrote. Thanks for that! Glad I found your channel.
@energyideas 4 роки тому
Lots of experience with clustering went into this well thought out video. Thanks. 'Pi Dramble' great for learning clustering and testing. Sandia National Labs uses them as a low cost cluster theory tester.
@charlesfries 4 роки тому
Incredibly straightforward and insightful vid. Thank you
@darkstatehk 4 роки тому
That Turing board should be called a Custard board; "A game changing Custard Pi."
@Crisosphinx 3 роки тому
A Cluster Compute Module... Or should I say, "Bramble Custard Compute Module".
@faultyinterface 3 роки тому
I give this comment a golden sticker
@opportunityknocks3088 3 роки тому
Damn I said this but you beat me to it lol
@jimborn3039 2 роки тому
Custard cluster
@CraniumSlows 4 роки тому
Heck ya my dude. Thanks for all your awesome contributions. This was super cool.
@santanudasgupta6170 4 роки тому
First time hearing you, instant subscribe after hearing your few point of views on systems design and architecture principles!
@sergioguzman5971 4 роки тому
Love the Close Encounter reference.
@voxelfied 4 місяці тому
Came across this old video! So much time has passed since then, and we could say that the Raspberry Pi has made a lot of progress! The Pi 5s now come with up to 8gb RAM and super fast processors. Making clusters now are definitely worth it because of the price and performance!
@SteveRHanson 4 роки тому
And powering all of this is schleem.
@koysdo 4 роки тому
Super informative video! Can't wait to see you explore the Turing motherboard
@TheSQLPro 3 роки тому
Excellent explanation. The diagrams are great especially comparing older ways of doing things versus newer (Scale up vs Scale out). Thank you.
@lawrencedoliveiro9104 4 роки тому
3:25 A touch of brass for that steampunk feel. ;)
@pjotrvrolijk3537 4 роки тому
Just started playing with my own Pi cluster. I agree that the main thing is to have a testbed in actual hardware. It won't ever be super fast, but it's a great learning experience. Looking forward to your series.
@user-oz2jn4yh2e 5 місяців тому
I made a pi zero 2 w with 300 of them and it was crazy fast
@Xenthera 4 роки тому
Since my bramble ran droople I called it dramble Lol sounds like intergalactic cable on rick and morty
@EvenTheDogAgrees 3 роки тому
It's Drupal, not droople. ;)
@egs-zs8-127 3 роки тому
Lol, this is so accurate. I had to see if I was in a real fake doors commercial.
@carterash2588 3 роки тому
@@EvenTheDogAgrees idk Im pretty sure it is DrooOple?
@pentachronic 3 роки тому
Sounds like someone dropped acid.
@efmelectronicsformakers778 3 роки тому
I dont use a dramble, I use a grapple...
@sh1k4maru 4 роки тому
Love the content. Keep it up bud!
@Astra7525 4 роки тому
I had been eyeing the Pine64 Clusterboard and Sopine A64 Compute Modules. I hope you mention why you chose the RaspberryPi and compare it to alternative solutions in the next video.
@johnmahugu 3 роки тому
thank you so very much for this, been arguing the concept with my friend for a while now now i have the proof to shut him up and get on with our super computer implementation.
@michdem100 4 роки тому
Cool think I've watched it with great interest because my uni thesis is exactly about clusters using SBC computers - I'm sure some of what you're doing will prove helpful as well as my research for you once it's done.
@acidpunk7423 2 роки тому
Thank you for covering this,really helped me alot.
@PeterRichardsandYoureNot 4 роки тому
‘What is a cluster?” Low hanging fruit....
@an1rb 3 роки тому
Rather, low hanging raspberries.
@salmamoorad1604 3 роки тому
Multiple computers connected on network to process ,
@questionablecommands9423 4 роки тому
I'm a little jealous you've got your hands on one of those boards. I pre-ordered one just a week or two ago.
@JeffGeerling 4 роки тому
I don't get to keep it long... but I'm pretty sure I'll join you on that pre-order list soon!
@IhabSolimanM 3 роки тому
Where did you pre-order it from? I can’t find the place for it
@questionablecommands9423 3 роки тому
@@IhabSolimanM turingpi.com -- it looks like the first production run is sold out, though.
@IhabSolimanM 3 роки тому
Yea, it’s been out of stock for a while now
@rajat0610 4 роки тому
i didn't understand anything. But it was amazing to watch!
@AndrewPrinster 3 роки тому
I just ordered my first two Pi's...I was lost quickly.
@hnasr 3 роки тому
Damn your good ! Loved the setup and how simple you explain things. Subbed!
@purteatin 4 роки тому
This guy is totally full of.......... interesting and fun ideas for computing!
@ramimann9742 4 роки тому
Well explained. Thanks
@vikrammaiya003 3 роки тому
Awesome. Crystal Clear explanation. Support from India. Keep posting.
@BhupinderSinghTitan 2 роки тому
Finally I learned about it. Great explanation.
@yannlallemant5369 3 роки тому
"Close Encounters of the Third Kind"... cool !
@jsaenzMusic 4 роки тому
Outstanding vid, very informative; thank you! New sub.
@theesteady1 2 роки тому
Great teaching! You have a gift of explaining...
@KienTran 4 роки тому
Wow. Super tempted to get that turingPI to play with K8
@khalifdove7955 2 роки тому
Jeff dude, you are just amazing
@Aaron-iz3hk 3 роки тому
In every one of your videos, you do one small thing that reminds me to Like the video. This is also what got me to subscribe. I like your content! It really is the little things, like your movie homage and Red Shirt Jeff, that add a great touch!
@tonya9614 3 роки тому
Looks the algorithms brought me another great channel!
@bluehornet6752 4 роки тому
Sweet! Definitely a new subscriber here. Thanks for the effort!
@Stonehands43 2 роки тому
Sounds like the tune that Gizmo plays in Gremlins.
@lahmyaj 3 роки тому
Sweet! Love these sort of videos ;)
@chiragsukhala 4 роки тому
Awesome Work!
@bci3937 3 роки тому
Man thats a great Project, i am fixed, looks great, powerful, and that tower is quiet outstanding futuristic looking technology, that i can build alone...cool
@jimbo-dev 3 роки тому
Interesting video, there definitely exists some need for this board. People buying this hardware are most often hobbyists and while it is true that threadripper is likely too expensive for ”homelab”, but used server hardware is not. For example I paid ~600€ for two 8core server cpus (but my goal was to get faster cores, not as many as possible). My current home server has dual socket motherboard with a bit older cpus and ”only” ddr3 ram. Power consumption is definitely a bit higher, but for most people used hardware is probably a better deal than a raspberry pi cluster. Raspberry pi has probably less hardware compatibility issues so it is definitely easier to set up and there’s always need for this kind of bit more experimental setups, otherwise there wouldn’t be any new ideas and innovations, so keep up your truly interesting content!
@CrazyCoupleDIY 3 роки тому
This was quite informative.
@paulalmquist5683 3 роки тому
I build a cluster many years ago using 8 retired 486 machines plus a master node. The buzzword in those days was "Beowulf Cluster". The control software i used was PVM - Parallel Virtual Machine with Linux installed on each machine. The network was an Ethernet bus using RG-58 coax cable - 10Base2. It worked but I did not really have a need for it so it went away. It was a fun learning experience. It is interesting to see what can be done with current hardware. Certainly a lot smaller package. My cluster was about 7-8 cubic feet and of course a lot slower that today's processors.
@madsonweb 4 роки тому
It is really useful for learning purposes
@joshdheda8776 4 роки тому
you got me to subscribe, well played
@sajidchouhan002 4 роки тому
I am not a software engineer and dnt no about coding n oll but as a power user, Cluster computing seems very cool, following many cluster projects it seems every project ends up with learning experience only.Really want that it gains some practicality like Powerful password crack-machine,Hosting website,Bulk photo encoder to auto-resize big DSLR images, and many more day to day work even converting(downsizing) collection of 1000s of Video clips and auto uploading to cloud to save a lot of space on cloud. If some how these above said work are possible on Cluster it will become the best time and pain saver of the world. Considering a 15 node cluster these tasks are not impossible but the problem is lack of interest of coders and developers in doing so. I really hope you will develop a cluster which will make life easy&fast. waiting for your upcoming videos.
@verykittypretty 5 місяців тому
thanks for being thorough in ur explanation as someone who didnt know the basics of servers xd!
@devdylan6152 2 роки тому
one of the cool things I think would be intersting to expand on with the zero boards is running a cluster within a single robotics project that may have a large chassis yet need IO management for motors, etc while running tensorflow over a single robotics system... a pi zero cluster would be easy to fit, be cheap and light as well... I found the reason for your rack finally XD.
@id104335409 4 роки тому
You nerd! 😄 I see you smiling as you explain how you came up with the name Dramble!
@JeffGeerling 4 роки тому
id104335409 oh it goes deeper: a drupelet is "any of the small individual drupes forming a fleshy aggregate fruit such as a blackberry or raspberry"... and the Dramble runs Drupal! So each of the Pis is like it's own little 'Drupal'et... 😉
@id104335409 4 роки тому
@@JeffGeerling ha ha, naturally.
@GordieGii 4 роки тому
I've thought it would be a great idea to put a bunch of Compute Modules on some sort of bus board for years. Nice to see I'm not the only one.
@inibapakbudi 4 роки тому
I got new things here. Hence, I like and subs. Then explore other videos for more knowledge about Pi Clustering. Very interesting.
@ATomRileyA 3 роки тому
These would be great for building a Skycoin Skyminer/node, Also great video explaining how to set the Turing Pi, just subscribed to your channel after watching the video. Keep up the great work :)
@matthewfarrell317 3 роки тому
Apart from the fact that a bramble is an entire plant (roots and canes), and not the berries, this was a very interesting random UKposts recommendation, that I will need to watch all the parts too lol.
@abba4468 3 роки тому
Excelente video! Gracias
@ConanDuke 3 роки тому
Yay, Drupal! Can't wait to build a Dramble!
@ConanDuke 3 роки тому
The Close Encounters shout-out was epic.
@redpillproductionscanada5563 3 роки тому
Ahhh, thats why I recognize you, I seen the bramble video lol. I literally bought my first pi because of your video bro lol. Dude the movie reference....
@carlosgodoy1 3 роки тому
Super video. Subbed.
@haarisblue06 3 роки тому
I like that you named it after Alan Turing, the person who created the first supercomputer, the Turing supercomputer.
@AshtonSnapp 3 роки тому
He didn’t name the board, the people who made it did
@N0RD0M 4 роки тому
Thank you for making this video. I often wondered what advantages a making a clustered PI and what software would it run.
@johnmadsen37 4 роки тому
No advantage except fun. Every other computer is superior in every way to cluster.
@gteixeira 3 роки тому
I would say to make low power servers, although I think that a high performance server with several virtual servers are probably more power efficient on a per server basis. In the end, I would say that this is more like a proof of concept that you can make at home for little money and learn in the process.
@PatrioticGestalt 3 роки тому
Looks like I have to watch this "series". My job wants to look at a Ceph clustering infrastructure. LOL
@johnp.weiksnar6861 4 роки тому
Well done! And to think, I was there at that original DrupalCon BoF! . . . #dramble
@JeffGeerling 4 роки тому
One of the OG!
@BlensonPaul 3 роки тому
great video . awesome.
@Pablo_. 2 роки тому
Your a cool guy man you teach well I subbed
@jakobfindlay4136 4 роки тому
ive been interested in this idea for years, why did it just blow up now ive seen a few youtubers doing it
@mustache2295 3 роки тому
After you make these clusters, how do you utilize them? I'm interested in doing these sorts of projects but not sure what to do with them once I make them.
@jeramykelton5304 2 роки тому
Masters of the Universe!
@real_andrii 4 роки тому
instantly subscribed
@pikaskew 3 роки тому
This is a great series, thanks for doing this! I wonder if they'll make one of these when the CM4's come out (seems like the pinout wouldn't be the same)
@JeffGeerling 3 роки тому
I have been told that they are anxiously anticipating the CM4 and plan on making a new model for it. If the preliminary rumors I’ve heard are true, they might even be able to reduce the cost a bit!
@pikaskew 3 роки тому
@@JeffGeerling Wow that sounds awesome. I've been using a RPi4 as a primary desktop for work (VDI helps a little for its shortcomings ofc) and it has been surprisingly capable. Very much looking forward to the CM4 version of the Turing, the cost/performance output potential is exciting :)
@yellowcrescent 3 роки тому
Another cheap option for home-based computing clusters if you have the space is buying used 2U Dell or HP servers from hardware reclamation companies off eBay. For example, for around $150~$200 you can buy a Dell R710 with 2x Xeon E5xxx/X5xxx or similarly-spec'd CPUs (12 to 24 total threads), 24 to 48GB of RAM, tons of SATA3 hotswap bays and PCI-e 8x expansion options, 4x 1GbE, plus remote IPMI management. Granted, these machines are huge and heavy (~70lbs each), and ideally you would want to rackmount them (the rackmount rails are ~$70 by themselves, which is a bit annoying). Great option if you want to build a Ceph or other storage cluster, transcode video, render 3D stuff, etc. I use a TP-Link T2600G-28TS as my rack switch, which is ~$150 managed L2-ish switch that supports bonding/LACP, VLANs, and other stuff. Not as fun as getting a cluster of RPis working, though :)
@cavamagie11 3 роки тому
i follow you from ansible roles ..great job!!
@georgekubrick3134 Рік тому
Damn! Look at that stack of compute modules!
@cowboybob7093 2 роки тому
When I see a picture of someone with their mouth open on a show's icon "dumb" is the only thing I think. Congratulations, you are getting a comment, that will help your score. I even let the ads play while I type this. 👍🏻👍🏼 The sound is off and I intend to close this tab as soon as I'm done with this comment. Raspberry Pi clustering sounds interesting. I think I'll do a search and see if anything catches my imagination.
@jlambert12013 3 роки тому
Nice Strat!
@x91w 3 роки тому
I'm currently playing with Kubernetes on a cloud platform like digital ocean/AWS for video and audio processing applications. Multiple rpi 4 compute looks like a potential answer as currently transfer to/from cloud is the main bottleneck for large files. Can you recommend any toolsets that take the hard work out of determining whether a solution should be cloud, local or hybrid (without actually building both and running multiple tests)?
@maldit0x 3 роки тому
Hi Jeff, fascinating stuff, i see u have a guitar, do you think you can put together with a Rpie a guitar fx pedal? how would you go about the foot pedal? could even run along a dedicated sound card? imo this would change the industry of foot pedals and fx as you can expand update and even have more functions like looping if it was programed well, what you think?
@Pibby_rojo 2 роки тому
Very good! I am user pi3, i work in for develop my web with docker, is incredible solution low cost! Thank you from Argentine!!🇦🇷🇦🇷🇦🇷
@brianbordenkircher52 4 роки тому
Is there a certain cluster board that would make sense for more input / output options? I’m looking to do visual recognition of obstical avoidance and object recognition basically being run on multiple pi’s
@andrewcool4587 3 роки тому
Might be my next project. After a retro pi!
@user-qc6yq1jh4j 8 місяців тому
can arm cpu and intel cpu be made into a cluster or do both cpus have to be the same type?
@gineesh 4 роки тому
Wow... Thats cool
@noelvarghese370 4 роки тому
Damn I have been using your ansible roles, didn't know you had a YT too
@AsbjornOlling 4 роки тому
i was just about to say the same thing. those docker roles are great.
@russellauger5067 3 роки тому
Good stuff. For QBit[mm].Coil[1]=1: QBit[mm].Coil[2]=0; QBit[mm].Coil[3]=-1; ... switch(priorState)...
@xarmandop 9 місяців тому
I'm going to start building my first Pi cluster with four RaspBerry pi 4, just not sure about the ram size, 4gb or 8gb? Which one should work enough to get started?
@davestinyworkshopotherprojects 3 роки тому
thanks for that great intro, is it possible to use a raspberry pi cluster to process videos created in openshot video editor?
@jamegumb7298 4 роки тому
I just have a 2200G apu in a 1U case with flex-atx. Second hand set of IB 40Gbit cards, and then a 1920X attached to it in another 1U case (boots from a tiny xpoint drive to make sure it is up as fast as possible when needed), both tucked neatly under a minimalist desk. 1920X stays off, can be waken up by lan, and then used whenever needed. Software is a bit iffy at times, but the 1920X can be used as a rendernode, compute node and with a bit of messing around can be used to run like I run software locally on a lot more cores. As a bonus, I made it reachable from outside and can use it from wherever I am on the globe, but network speed than becomes an issue. Still fun though. Not sure if I should sell the 1920X, then add another 2200G and make tiny stackable cases and use a 3300X as the main one. Or maybe 2×3300X and keep the 2200G as the main. Clusters are fun.
@MBOFOYH 3 роки тому
Clustering multiple touring boards sounds great.
@AkinaGod 4 роки тому
What cluster pi set up would you consider the best? This one, or the others you mention/worked on in the past?
@palakondarayuduuppu6420 4 роки тому
hey jeff, it's awesome man. do you think we can make lowcost supercomputer which can achieve 1Peta flop in low cost? And cheaper than using normal motherboard and gpu's?
@emanuel4all 3 роки тому
Hello Jeff. I just watched your yb video and I was wondering if you could help me. I am planning to build a 3d scanning rig using 50 dslr Canon and multiple raspberry pi's and make all the cameras shoot a round of photos at the exact same time and after a very short delay another round of photos with a noise pattern projection from a projector. You think you can tell me what programming language I should learn to achieve this? I have time to study and the will
@Jeroen_a 2 роки тому
4 Laptops on a table (in a square)..... this looks nice and organized :)
@Franchise9920 4 роки тому
I feel pretty uneducated on the cluster front. What do as far as home use my with these clusters? Media servers?
@bikram22tutu 3 роки тому
Hey Jeff, I recently started following your videos and chanel. Can we have a cluster built and run Linux on it using Raspberry?
@noahhastings6145 2 роки тому
Close encounters of the 3rd kind! I only saw that movie for the first time 2 weeks ago.
@ryanwakebradtelle8682 3 роки тому
Is there anything stopping you from using the type C port to supply internet?
@macornman 3 роки тому
Rasp pi has 8gb of RAM now. Thinking of making my own cluster for some aero CFD. Thanks for this vid
@tommaguire7868 2 роки тому
Thanks!
@dwaynehart1974 4 роки тому
Very nice work Jeff. How many videos will you be creating on this topic?
@JeffGeerling 4 роки тому
I'm planning at least three or four. The next video will walk through the basic hardware setup and getting an OS onto the boards.
@joshedwards6205 4 роки тому
Have you looked at all at Parallela? They have 64 cores for the price of a high-end single board computer as well as expansion boards to add additional processors.
Jeff Geerling
Переглядів 213 тис.
ASMR Detected
Переглядів 82 тис.