I'm literally making over-engineered pi cluster because of that

That’s why I built my pc. It was a fun pandemic project

But using a bunch of 10- year-old cheapo machines could be almost as fun, probably more performant, and teach you about the more widely deployed x86 ecosystem. I'd be interested to see where the line is, where a raspberry pi 4 becomes more performant per watt for server loads compared to older, low end pc cpus. Are we talking early i3, i5, or could a core 2 duo beat it?

Depending on which country you're in, the cost of a pi cluster is still a looooot cheaper than tuition fees to have a formal degree or course in the topics you'll learn :) and you get to keep the cluster too.

Exactly. Those are the top reasons. The third for myself is cost. Currently building / testing out a Kubernetes cluster and learning about the differences of between the virtual and physical hardware. Its a total blast. :D

Do you have a source of this?

Dangit! I should've done more research on it... I guess there are a few games where the community will go to great lengths to make the game exceed initial expectations.

@@JeffGeerling Crysis, Jeff, you always go with Crysis. ;)

I believe it was possible to run Doom 2 (or some other fps game from back in the day) on 3 different machines at once, one rendering your frontal view and the other 2 rendering your side views. Technically that's clustered.

@@KarlMathiasMoberg thanks!

I'd like a link to the original video please

The episode where Linus lost all his video archive hahaha

@@syrus3k ukposts.info/have/v-deo/n4Ojoo6eeKN8mZ8.html

on rpi cluster you would lose everything in case of some chip or memory card failure, and it is even harder to create a proper raid using rpi due to weak cpu and need to offload it with hardware raid (which is a pain to rescue)....thus nothing laughable there

@@s.i.m.c.a That could be true. This is why I think it's absolutely necessary to have a service always backing stuffs up to another machine that's not on the same physical setup. This is given that the data you have is very important to you. For LTT's case, I can see why it would be a problem. They edit 8k footages, and those arr huge files. For my case, all I need is my site(s), bots, and dbs to be backed up; which is only a few gbs max. So, I always prefer having that backup service running. Plus I use git for the "script" files, so I am always prepared for the doom day.

Good comment: you saved me the trouble of writing the same. Parallel code development for physics-based simulations is not easy, so it is convenient to have a small, cheap system for this purpose before submitting production jobs to a very expensive supercomputer, which also likely has a long queue.

I'm considering this for a compute node for BEM calculations in acouetic modeling, since solving for millions of frequencys to draw responses, can just be split into a set of frequencies for each CPU. Issue is they run on windows

Ok say this again, but like for people with an IQ of a 5 year old. Asking for a friend :]

@@liquidluck711 AFAIK, On-Die ECC corrects error inside the memory chips, while as standard ECC corrects errors when data is sent from the memory to the cpu. On-Die ECC is a newer technology that is being implemented on new memory (like LPDDR4 or DDR5) as protection because new memory chips have higher capacities and higher chances of data corruption, but it doesn't substitute traditional ECC technology used in servers and industrial devices

Literally what I was wondering thx.

So, it'll still have the SEC but not the DED?

@@0x8badf00d the issue isnt about SEC and DED... The issue is that the traditional server ECC covers not just the internal RAM, but the RAM bus. This ECC will detect and correct errors only with the memory module, but doesnt guarantee that protection throughout the full memory subsystem back to the CPU.

Yes, he mentions various software used for and applications of clustering. Here's a simpler answer... The Pi and similar inexpensive hardware gives people without access to a data center an affordable means of learning what all that stuff is.

@@PraktikoolSinik Good answer. It is also useful for developing parallel code, which is not easy.

What, you are sick of endless tutorials of setting up nginx on a K8s cluster? Well, have you tried the vast amount of other learning options, which are exclusively Kubernetes applications that address Kubernetes shortcomings? You didn't like reference books < 2 years old that are already full of deprecated commands and demos that no longer work? WELL YOU MUST HATE LEARNING THEN.

@@emeraldbonsai it depends on what they are about. web development probably moves the fastest

Red Shirt Jeff hearted this comment.

@@JeffGeerling is red shirt Jeff represented by your lawyer? I couldn't imagine wanting that liability on my company 😬 That being said red shift Jeff ftw! 🤙

Very much this. Cool, fun, and I'd argue not pointless if you're learning from the experience of working with the underlying tech.

Just like that server

Whonnock literally knocked

@@natsurii7092 So that's how you spell "Juaniq"

Which Micro PCs did you end up settling on? I've been dipping my toes into the idea but I haven't settled on if I want Optiplex 9020s (i5-4590S) or EliteDesk 705 G4s (PRO 2200G) since the Pros and Cons with the current market balance each other out.

Two suggestions (although I'll do more on this later, probably): My Turing Pi series from last year, and NetworkChuck's K3s video from a few weeks ago.

I've used a straight K8S cluster using Kubeadm - automated using ansible to set up. Use cases - its un!! and you can use for software like pi-hole, VPNs etc and internal wiki's, web sites etc - it's up to you what the use case is!

I think he covered all the use cases. :(

you can use rpi4 cluster as an heaters in your room, the bigger cluster - the better temperatures you could get. More seriously, to rpi cluster could be applied same workflows like for typical clusters

@Jeff Geerling Have you built a cluster of Pis using compute module IO board? I don’t want to buy the Turing Pi v1 as there is a v2 coming out but want something to play with interim

This sounds fun.

I should look into this for blender...

@@binarycat1237 Depending on your renderload for blender, something that can work very nice is bitcoin mining boards and just a crap tonne of 760- 1080 gtx video cards (because they look like their about to bottom out in price with nvidias latest news) . Keep in mind it will be a pcie 1 connection that they all have which in this application can be just fine. I'm using a Biostar TB360-BTC PRO and some hawked up server powersupplies,

My poor Vic20 and C64 were full of hardware mods and looked like porcupines with multi color quills from the wires sticking out

I will likely do another series once I can source enough parts for a new CM4-based cluster :) - just been hard getting everything this year :(

drink their milk and eat their dead bodies? if you say so.

@@isaackvasager9957 a little extra iron and copper never hurt a diet.

Poke with a shocky stick?

I haven't seen that video, but I am planning on getting my Pi-hole setup to be a bit more robust later this year. I've had that Pi lock up twice since I switched my whole home to it, and it's annoying to have to reboot the thing once every couple months!

Wow, nice! Have you done any benchmarks on the cluster to see what kind of throughput you could get for sequential access (and random / IOPS too)? I've been planning on testing Ceph again sometime this year, would be great to see how your cluster does so I can get a point of comparison!

@@JeffGeerling Thanks. Surprisingly, no I haven't done any benchmarking aside from the seat of the pants "good enough". It runs all my VMs and other services without any notable slowdown and the Client IO metric on the Ceph Manager dashboard shows up to 200MB/s which is about double what I could get from the two server machines. The drives are all 4TB WD Blacks (HDDs) connected via USB3 so not slow but nothing like SSD drives and NOTHING like the performance you have gotten with some of the PCI connected devices recently. Now, of course, I am going to go do some benchmarking. I guess the Prometheus install is going to have to wait a bit longer. LOL Edit: GB -> MB I wish it were 200GB/s

The Odroid HC2 is great for this. I have eight of them running LizardFS. I'm planning to get more.

@@JeffGeerling First round of benchmarking done. I used the commands available from ceph. "rados bench ...". My performance for any one device is network limited as it will saturate the gigabit link on any read or write. Same with two devices and almost with three. Running the benchmark on four machines ends up with approximately 300-320 MB/s throughput on writes. There is something I am missing with coordinating the read benchmarks using "rados bench..." across multiple clients so don't have good numbers for that yet. Issue sorted: Sequential reads across 4 devices tops out at 406MB/s, random reads is about the same. Makes me think it may actually be a little quicker but I don't have another physical wired device I can use for benchmarking.

@@BowenMarmot That's great to hear! I'm glad to hear the numbers are so good; was thinking it might not be able to saturate the Pi's network connection.

I didn't even touch on the bottleneck topic-but I found the same. The Pi can expose code paths that are _usually_ not a problem on fancier hardware, but are glaringly bad on Pi (e.g. disk accesses where you don't need them, high cyclomatic complexity in functions that get really slow on the Pi...). Running code on slower hardware exposes those flaws and takes a 'decent' app on fast hardware and can make it _fly_.

@@JeffGeerling One area where the compute bottlenecks really show is in database applications and cryptographic functions. I’ve had to experiment with using in memory caching mechanisms, currently playing around with Redis. I’m looking into options for offloading cryptographic functions to a USB module, but so far haven’t had much luck.

and that one PC dies and he still got 5+ pi's running while you are offline ;-)

He mentioned it in the video. Availability and up time. When one server dies, all the the other servers will still perform the task.

@@maplenerd22 and a new one can be spun up automatically as well for the one that crashed

To be fair, unless you're only actually using only 1-2 services at a time and not have an actual public website, then that i3 PC comparison is bullcrap. I mean, it cannot cope. Good luck watching a movie from plex when the backup service is running and you have several visits on the website. Though I have to agree, if you can somehow use past unused CPU cycles when needed, then an i3 running at 100% should be able to power most things effectively. Anyway, having a cluser also improves the bandwidth and IOPS that the storage and RAM effectively have per process. If there's a spike and 10 processes really do need to do stuff at the exact same time, a cluster of 4 Pi with modest RAM/storage would be better than one i3 with good (not best) RAM&storage.

Just because it "can" be done, doesn't necessarily make it the best option. Plenty of use cases where having stable functions working independently from others can easily be imagined. Gathering data from external IOT devices, assembled, posted onto an internal local website for monitoring, or for student matlab projects, redundant micro server, or each PI zero as a powerbank/fitbit clone, which can play music on you outing, and when back to recharge by imserting into cluster, heart rates, oxygen bloid lwvel etc can be uploaded for your own private health check, that bypasses the vendors of such systems that sell your data. I ran a professional CAD service some years ago, and while I mostly was the only person doing it all, save for hiring in people for some projects, I still had all the tasks spread out on many different machines. 1 was used for new software installs, testing, pushing the envelope etc, 2 other machines was "Stable" CAD workstations, with absolute minimum installed to avoid software breaking when new versions came along, then a specific machine for customer visits, to avoid crisis if theft, incompatabilities etc, some accounting only machine, with temote access for CPA, standard office (word/excel ect) machine, and finally redundant hot swappable file server with client backups, connected to remote storage ++. Run all that on 1 machine? Sure, i guess you could make it work. Would be slow and need lots of reinstalls and reboots though. And good luck if you broke the HD. While not a cluster, a cluster could been used for a set of thin clients instead, which was a technology a little above my cost range.

I'm also impatiently waiting for the Turing Pi 2, I've already bought 4x CM4s in preparation... I hope they don't just give up on it! It's been radio silence for months now

interesting

What that means? You use it as a router?

I still have an HBA from you on my test bench :D

I think we still call them that, maybe Jeff just doesn't want to confuse more people that aren't aware of terms like vertical and horizontal scaling

...but in a few scenarios they are the cheapest solution to get a stack of real processors.

You can get 5 RPi4Bs with 2GB Ram for 200€ + a few bucks for other accessories and a power supply with enough powerto run all of them together. This makes it less expensive and significantly more power efficient than any other option on the market. You get 20 Cores and 10GB Ram in total for this. Low power NAS options are less powerful and are much more expensive. And if you try to build something yourself it will get even more expensive if you want comparable performance and efficiency. So if you want to run something that can be run on a pi cluster, this is actually not a bad option

Funny fact is: I was running Linux (Linux-vserver) containers in production before VMs were popular, most people had no idea what was possible.

or IT is a hobby for us and even $3000 isn't going to be approved by the budget troll. I completely agree if you are supporting a production environment a test environment is very useful, as is a place to practice managing a cluster.

Kill two birds with one stone!

@@JeffGeerling sorry misspelt that's.. Should have been clutter.. :)