With our special offer you can get 2 years of NordPass with 1 month free for a personal account: www.nordpass.com/highyieldnordpass Or, use code highyieldnordpass at the checkout. Business accounts (must register with a biz domain) can get a free 3-month trial of NordPass at www.nordpass.com/highyieldbusiness with code highyieldbusiness in the form.

I think in a few years, AI accelerator cards will be the next video cards. A race to the top for the most powerful accelerator to be able to train and run AI locally on our own PCs, bypassing the need to pay for filtered models from large companies. Once people can run this kind of thing independently, that's when things will start getting _really_ exciting.

I'm a newer subscriber, working my way back through your recent videos. I just want to tell you that I think this is the best tech channel on UKposts right now, hands down. You are a wonderful teacher, you take frequent breaks to summarize key points, you provide ample context and visual aids, and when you do make personal guesses or offer your own opinions, it's always done in a transparent and logical manner. Thank you so much, and keep up the amazing work. I'll be here for every upload going forward.

Dev here... gotta say I love it. Thoroughly enjoyable thank you for your time!

I find the increase in cumpute performance incredible. As shown at 7:38 the GV100 from the beginning of 2018 had 125 TFLOPS of FP16 Tensor core compute performance. The current generation of Enterprise AI accelerators from NVIDIA are the NVIDIA H100. And the H100 provides up to 1979 TFLOPS of FP16 Tensor core compute Performance. And the Fp32 and FP64 Tensor Core performance has also obviously increased massively. Within ~5 years the raw compute performance of Tensor Cores has increased by around 16x. What previouslym required 10,000 GPUs could now be done with ~632.

Great video! I'd like to add one thing: in the segment starting at 16:06 where you talk about the Nvidia Hopper H100, in the context of Neural Networks the most important number to compare to the previous A100 should be the memory. As far as I know, as long as there is *some* kind of matrix multiplication acceleration, it doesn't matter much how fast it is. Memory bandwidth becomes the major bottleneck again. I looked it up and found the number of 3TB/s, which would be 50% higher than the A100 80GB-version. I wonder where the number of 4.9TB/s shown in the video at 18:50 comes from. It seems unrealistically high to me. Nvidia's marketing does not like to admit this. They like to instead compare other numbers, where they can claim some 10x or 20x or 30x improvement.

I just saw your post on the community page, I wish UKposts would have notified me when the video was posted instead of pushing the post without the video. I cannot wait to see what is talked about in the video! Edit: This was great, I’m definitely sharing it with some friends, keep up the great work!

Great video! Hopefully it will put things into perspective, that Nvidia, Intel, and AMD's world does not revolve around graphic card sales and squeezing the most performance out of Cyberpunk. Hundreds of millions of dollars are a stake in areas much more lucrative than $500 CPUs or $800 videocards. They must meet the demands of all their various customers as well as investors and stockholders too. Thanks!

Great and very informative video, sir. I remember watching your very early videos and thought it was bit meh. But, this is absolutely world class stuff and happy to listen to such a sharp and insightful mind.

Thanks for the tech background on what it's running from a hardware angle. The interesting thing is I think when some one comes up with a hunk of code that saves billions of hours of computational power. That disruptive type of thing from a software angle. It is an amazing time to live. Thanks for all your hard work, and an interesting vid!

Fantastic presentation. Appreciate your pragmatic and reasonable research, impressive work. Thank you!

Been beating my head against the Cost vs Scale issue of building my own AI compute rig for training models at home, and this gave me a better idea of what kind of hardware I'll need long-term by looking at what the current bleeding edge looks like. Thanks for doing all the research work on this one!

I have been an OpenAI beta member for 3 years now. It has only become better over the years. I wonder what it will look like in 5 years.

Good job here sir! Mind if you did a quick breakdown of language model like Bluewillow that also utilizes diffusion?

MS will for sure use its Azure based cloud system for hosting its ChatGPT, so that they can load balance the demand, and be able to scale out to more VM's and instances if needed to meet demand, and to increase resources on any individual instance if needed. That would be the best use of that set up and provide the best user experience. So basically, the hardware specifics would be whatever servers are running in the 'farms'. I doubt if they will have separate and specific hardware set aside just for ChatGPT as it would run like any other service out there.

Great video! Curious what you use to generate graphics / screenshot animations for your videos?

Logically the inference costs should scale pretty linearly to the amount of users since it's pretty much a fixed amount of computation and data transfer, or can you elaborate why the requirements would scale exponentially as you state at @15:40?

10:14 is so funny because "what hardware are you running on?" is one of the first questions I asked that bot 😀

Thank you for this! I’ve been looking around for this content. Great work!!👍🏼

Fantastic!! First of all, I cpuld be wrong, but I don't remember you having an in video sponsor before. Either way, that I awesome!! I'm glad you are getting the recognition you deserve!! You must have done a lot of work to get these numbers and configurations. Very interesting stuff! I am looking forward to AI splitting off from GPUs too. Especially with the demand for them going up as investment in AI grows. I, as I'm sure many others are as well, am kinda sick of having to pay or consider paying a lot more for a gaming GPU because the higher demand is in non gaming sectors that are saturated with capital to spend on them. Plus I'm sure they will do a much better job. The design, at least in regards to Nvidia because of it is quite annoying too. Tensor cores for example were mainly put there for AI and Mining use, the marketing of them for upscaling and the cost added for a gamer to use it is kinda ridiculous. If you have a lower end card with them where you wpuld benifiet from the upscaling, you could probably buy a card without them that wouldnt need to upscale. It seems to me that their existence is almost the cause for their need in that use case. I don't know how much of the cost of the card is just for them, but I imagine it's probably around 20-30% maybe?? IDK, just thinking "aloud". Anyway, thanks again for the hard work and please let us know when you get a Patreon account!! I would be proud to sponsor you as well!! Cheers!!

Thank you very much for bringing this kind of information. Even chatgpt is unware about the hardware is running on!

i was wondering how far off running inference is on a local machine. Or could a more focused model (one language, specific subjects/sites) run on today's hardware?

Congrats on the sponsorship!! Been watching since you had 1k subs

There are interesting approaches like LLaMA, which focus on inference efficiency over training efficiency. These could also help to bring down inference costs to a reasonable level.

Do you have any idea why they are back to Intel CPU for the NVIDIA DGX H100 hardware?

could you do a video on memory cooling..Seems most of the video card manufactures don't cool the memory. some do . why is that?

Out of curiosity and considering my current hardware, How many orders of magnitude slower, would a neural network of this magnitude, run on a simple CPU and virtual memory?

Kind of mind blowing that we can already run something very similar on a MacBook. The progress in AI is insane and it hasn't even started to self-improve, it's just humans that are that fast.

What a valuable information👌🏻 thank you for sharing, and keep up the good work

Some time around 1994, the era of search engines started. And now, the era of free access to general purpose Artifical intelligence, is becoming a reality in front of our eyes. What a time to be alive.

Great to see this video well. I loved your m2 chip video breakdowns ages ago

A super interesting video, really well explained too, thanks for all your hard work. It's always impressive how Nvidia seems to always be playing the long game with it's hardware development and as you mention I can't wait to see what Jim Keller comes up with at Tenstorrent because I can't think of a job he's had where he hasn't changed the face of computing with what he helps develop. I just wish Intel had backed him more and done whatever was needed to keep him for a little longer and we would maybe be in a very different Intel landscape right now.

I remember when I was growing up, I thought to myself...geez...it would have been great to live in the past, as there were so many undiscovered things that I could have figured out. Grown up, I have been working in AI for the past 7 years, and looking at this very point in time I can't help but think reflect on that moment...geez...there is just so much opportunity for new discovery.

We have 8 of those at the Apple data center I worked at. NVidia cheaped out on the CMOS bracket, and it’ll always crack. You’ll have to warranty work the whole board to fix it.

Thanks for the video I was looking at the images you created with the GPU specs, and I'm not sure if your numbers are correct. Just for comparison, I checked the numbers in TechpowerUp GPU DB. So if we'll look at GH100, for example, you mention 1000 TFlops (FP16) and 50 TFlops (FP32) On TechPowerUP GPU DB, an H100 (I checked both the SXM5 and PCIe versions) the numbers are totally different: 267 TFlops (FP16) and ~67 TFlops (FP32).

15:30 You say that the inference hardware must scale exponentially, but that must be hyperbole right? At the very most it's 1 to 1 and I'm sure there are creative ways to multiplex. The interesting thing to see would be transactions/sec for a single cluster instance.

Very very good video. SUBSCRIBE. Reason. You did not talk about hype. You explain tech concepts based on AI. I knew about Nvidia hardware running chatGPT but not the details. Thank you.

wondering if we could run a software that uses your gpu to give power to ai network? like we do with crypto mining

Mad love bro. Thanks for the vid

Does anyone have any references on how big the storage required for this was? Was it a zillion SSDs or was it all stored on HDDs?

Excellent excellent video! (edit to ask: at 19:25 .. "In-Network Compute" of ... *_INFINITY_* ... ? Can anyone explain that to me, please?)

And to think now anyone can run a GPT 3.5 Turbo-like AI on their local computer without the need for crazy specs is just incredible Stanford Alpaca and GPT4All are some models that achieve it

Was cool working on these servers

15:19 That's not a problem at all. Even if only 10 millions out of 100 are using OpenAI servers at every single time, that's 29000/10000000 gpus, or 0.0029 gpus per user. Probably less. So instead of them running the model , the end-users could , easily on their machines. Bloody hell , even small communities of few thousands of people could train their own AIs on their machines, soon to be a much smaller number. Few major problems with that. They lose much control on their product. They haven't figured out , yet , the details of monetizing these models, so they are restricted to running them in their own servers instead. Third major problem for Nvidia , it will be forced to return to the gaming gpus , their rightful capabilities , from which they were stripped back in 2008. This would mean no lucrative sales of the same hardware (with some tweaks ) to corporations and instead rely on massive sales of cheaper units. And last but not least , an end-user, gamer or not, will be able to acquire much more compute power with their 1000-3000 dollar purchases. Because , now a pc may be sporting the same cpus and gpus, but difference is gpus will be unlocked to their full computing potential . We are talking about many tens to hundreds of teraflops performance available now for the end-user to do useful work. Anf how will the mobile sector compare to this? Due to the fact that it runs on lower power budgets there is no way it could compete with fully-fledged pcs. Many will start forgetting buying a new smartphone especially the flagship-ones ; in fact the very thought of spending to buy sth that is an order of magnitude less compute-capable would be hugely annoying. Now, that I am thinking of it , losing control , worries them much more than anything else. And it would not only be control lost on a corporate level , but on a much much higher level. Right now , top heads at computer companies and possibly state planners must have shit their pants, because of what is seeminlgly an unavoidable surrendering of power from power centers to the citizens. To paraphrase Putin "Whoever becomes the leader in this sphere will not only forget about ruling this world, but lose the power he/she already has" All top leaders got this all thing wrong. This is not necessarily a bad thing.

I want to say thank you very much for a really good video with good information.

when thinking about huge capital costs involving in setting up AI hardware farm, Is it possible to take advantage of shared computing power of public peer network like we are doing now with blockchain nodes ?

We do run into lots of problems where general AI models isn't good, the future is to make a new submodel that can specifiacally solve specific tasks or just add weights to general models, but such a version of chatgpt would probably require 1000xbetter hardware.

I asked gpt3 half a year ago what it was running on. I kept asking more leading questions. I was able to get to a point where it said that it used specific mid range FPGA accelerators that retail between 4000-6000 dollars. The high end fpga are connected by pcie and the lower end use high speed uart. The servers used a lot of K series gpu's too.

makes me giggle a bit when the "Most Replayed" part is the part where the ads ends

Great video I was wondering about this

15:50 Now with GPT-3.5 Turbo they have decreased the cost 10 times, but likely not with new hardware, but with an improved, discretized and/or pruned model.

Does quantum coputer could be used for AI training?

From what I understood from chatgpt, it’s a strip down version of gpt3. Where its hardware requirement and model size are massively cut down. It’s a lot lighter running the model compared to gpt3. Itself even said chatgpt can even be ran on a high end pc.

Cool I was exactly wanting to know that!

I find this video funny in that I've asked ChatGPT about what kind of hardware it runs on; never got straightforward answer and it kept running me around loops lol

I wonder how fast ChatGPT would have trained on a Cerebras CS-2 system with their Wafer scale 2 architecture?

Really good video, and lots of efforts! Thanks man!

I would love to have a local and unlocked offline chatGPT

15:39 Exponentially? How? Why not linearly

How powerful would a model be that could be trained on a gaming laptop rtx 1650 ti? How about a natural language compiler, that translates English input into executable machine code like GCC?

what are the most advanced models like megatron are actually doing? anyone knows?

I can see a rise in private clouds particularly within government at least on a national level. The utilisation of the system itself may give away sensitive information to other nations or even corporations that may have competing self interests so a few of these systems may pop up in the UK, Australia, several in the US and probably Canada to start with (presuming each European nation may have one or two as well). Whosoever develops a homogenised and consistent build for such a system will be suddenly in demand with competing urgency.

For GAI we really need to be perpetually training during inference. You want this stuff to run cheaply open source it, folk have gotten Llama to run on an RPI.

Yay!! Happy day! Been looking forward to this! Thanks!

My friend is actually already training the smallest version of GPT on 2 A100s for his project in our ML course

I am interested to know the hardware requirements to run inference for a single user. What do you think?

This is the type of video im thankful to be watching

Anyone got chills when they heard the name megatron?

Great research and presentation

Would be both amazing and scary if AI neural networks run on quantum computers or other advanced computers in the future

I literally saw this on South Park yesterday and now it’s recommended to me on UKposts

Really insightful video. Really appreciate.

AMD made a smart move in acquiring Xillinix, and it mostly went unnoticed at the time. But it could pay off heavy in the long run.

Good and informative video, thanks a lot!

Nice one. Well presented.

Question: if INT8 tensor cores are useful for AI. why doesn't everyone use it? Shouldn't an 8 INT8 mul-add operation require much less transistors than a 16 bit FP mul-add? I know FP multiplications are "cheap" compared to same-size integer, but addition adds quite some overhead.

Of course the hardware and infrastructure behind this application is interesting, but what I find even more interesting is how this is done in software. How are alle these GPU's clustered? How is the workload actually divided and balanced? How do they maximize performance during training? And how in the world is the same model used by thousands of GPU's to server the inferencing by many, many users simultaneously? That's mindboggling to me, actually.

Fun fact: DGX A100 has a configuration with a quad socket EPYC 7742, 8 DGX A100 and 2TB of RAM Source: I know someone who work with one He said it works nice with Blender renders too, but the focus is tensor number crunching using PyTorch

Hardware requirements increase linearly for inference, not exponentially.

Yes it's very interesting, thankyou

Interesting conclusion, great video!

Why improves the HW requirement of inference exponentially by the number of users? That should be linear, shouldn't it?

AI progress is far more software bound than hardware bound. The deep learning algorithms are incapable of making logical reasoning and thus knowing if a proposition is true or not. That's the real breakthrough that needs to be done. Once deep learning gains this capability, we will really be confronted to superintelligence, with all the massive consequences that we are not really ready to face.

Any pricing analysis to go with this?

Curious about a gpt4 followup!

Something important to note is that OpenAI works with Microsoft and likely doesn't pay all that much for the GPUs. Microsoft OWNS the hardware, so their cost per day is just electricity, employees, and rent. The servers were already paid in full when the datacenter was built. Its up to Microsoft how much they want to charge OpenAI (who they are working with and just supplied Microsoft with Bing Chat, which made Bing really popular), so I'm guessing Microsoft gives them a huge discount or for free

Great video👍, have you considered using some kind of AI narrator tech trained on your own voice but with more clear english enhancements? Even the UKposts subtitles was getting your english wrong calling the Volta CPUs "Water GPUs"...😛

Great video, looking forward to AMD MI300.

Great talk! Thanks.

Very interesting! I was wondering about Tom Scott's statement on the curve, and I think your take on it is very accurate 👍🏻

Let's Gooo, 🔥🔥🔥

the best part is they already made very similar AIs that are open source and you can run in your own computer. which is way preferable than handing over data to openai and paying by character used, which is kinda absurd. and they used chat gpt to train them, lol

The Megatron AI has a memory bandwith of nearly 8 PB/s. Kind of insane.

I run this stuff locally, but I dont run mine based on training data. The analogy I use, and AI can know how to ride a bike, but if he has no bike, he isnt riding anywhere. A bot needs both the brains and tools the body uses.

What we need are locally ran AI models. Optimisations will have to be made for this, but it is a huge flaw that this type of high-speed interactive knowledge is in the hands of a few multi-billion global players. And we all know by now, "Open"AI is anything but open. This is what scares me most about this whole development. In the early days of the internet, everyone could run a server at home or people could get together and run a dedicated hardware in some co-location. With AI this is impossible because no one can afford this hardware requirement. If Chat-AI is the new internet, we need public access to the technology otherwise only few will be in control of such a huge power to decide what information should be available and what should be filtered or even altered.

Power efficiency comes in to play as well, will AMD be able to compete with slower hardware? Only if they can beat Nvidia in the overall value proposition.

I think it's a very simple equation: more layers and thus more params lead to better abstraction and deeper understanding. the brain would not be so huge if it wasn't necessary. we need to scale transformers up until they reach couple trillion params and for that we need stuff like H100 and whatever they announce at GTC next month. transformers are probably enough to solve language. that combined with CoT and as papers have shown, it will surpass humans

The box asks for more ram The box asks for another gpu The box asks for internet access

You don't need that much power to train ChatGPT. You can't compare the full training of a LLM (GPT-3) with an instruct finetune (ChatGPT). Remember that Stanford trained Vacuna which has performances similar to ChatGPT 3 for only $300 , by instruct finetuning the LLM Llama. And other open source chatbots have been trained on single gpu setups. So it's unlikely that OpenAI needed a full datacenter to train ChatGPT, the data collection was the hardest part here. Maybe they did, but then the training would have lasted less than one second, it seems useless to use 4000+ GPUs.

Great video thank you Been trying to ask chatgpt about its hardware but obviously you don't get the answer haha Those who think we're already reaching the top of ai right now are grossely mistaken. Those Terra flops we're talking for the new architecture will sound so ridiculously crap in a few years time, just like a top end gpu in 2010 wouldn't even be good enough for a cheap smartphone nowadays. And with focus turning to AI only for hardwares now it's just gonna improve exponentially for a while Although like the guys at openAI stated, don't expect chat gpt4 to be skynet level of AI, the results might look like minor improvements at first glance, but the cost, reliability, speed a'd accuracy of those models will improve a lot before going to the next phase with is actual artificial INTELLIGENCE. By 2030 the world won't be the same as it is now, that's granted imo, and most people are not ready for it