Yeah I was a bit confused when he said HBM3 is the first type that's commercially available.

@@unvergebeneid Made zero sense 🤷‍♂

​@@unvergebeneidHe said that it's the 1st of HBM3 that's commercially available.

So some gamers knew. Most I've seen have never heard of HBM memory

Yeah, he literraly says that in the video at 9:00

That's how I've always heard it, including in the computer hardware industry.

My Vega still performs to this day. The only bad thing is the memory capacity for newer games.

The reason we don't see it in consumer GPUs now is because of how good they are for these mega GPUs for data centers. It didn't so much "die" as we got priced out of getting access to it

There's very little in the future for HBM on the consumer side. The reason it never saw adoption except for a few AMD cards (and even then there hasn't been a new AMD consumer card with HBM since the Radeon VII in 2019), is because of cost. HBM is way more expensive than GDDR. You're taking larger silicon dies and stacking them on top of each other with much more expensive TSVs versus just taking single, smaller dies and placing them around the board. Also, the memory controllers on the GPU silicon is more expensive because you need a physically larger package to connect all the pins. As mentioned in the video, HBM gets its "High Bandwidth" by simply brute-forcing an extremely wide, but relatively slow data bus. Good for energy efficiency, but abysmal for silicon cost. For datacenter cards that actually benefit from and can afford huge amounts of VRAM *and* bandwidth, it makes sense. Consumer cards get all the bandwidth and capacity they need with cheaper GDDR and it benefits the silicon cost as well by requiring a smaller memory controller which needs significantly fewer pins.

I bet apple would use it

@@nutzeeer Apple. The company which charges $200 for 8GB of commodity DRAM which otherwise sells for ~$20? I'm sure they would be happy to sell you HBM if they could get you to pay $10k per 8GB of it...

a "dram" always makes me think of whisky

It's his thing. (Closed, WONTFIX)

There are some people who first read a word a certain way in their head and just stick with that first read even though they hear everyone else pronounce it differently. I don't know what goes through their head to reconcile the difference; maybe they hear it one way and while they think they pronounce it the same way, it comes out the original way. Maybe they know it's mis-pronounced and are trying to make fetch a thing. Maybe his first love is whisky. He should make a video on it; I'm sure it could be 15 minutes long and everyone will watch it through. Coincidentally though, in the cycling world, there is a parts manufacturer called SRAM. It's an acronym of the founders' initials and it's official pronounciation is not "s-ram", not "shram", but "sram"; as one syllable as you can pull off. I have no idea why a company would name themself or one of their products a name that flies in the face of conventional english pronounciation.

I've heard both ways over 20 years. It doesn't matter.

What about the people talking about DRAM memory ? Don’t they know the M in DRAM stands for Memory ? 🤔

How is anyone suppose to comment before this mans

@@connorrosekrans7348 - that guy probably pays that $6 per month to get early access.

@@connorrosekrans7348 Patreon, so they get early access i believe. Or it could have been a patreon-only video that has now shared.

a mf MONTH ago

How is it a month ago??

effective, efficient communication .. lack thereof too often a bottleneck

Say what you will about Hector Ruiz, but one of his most effective ideas was the Virtual Gorilla to compete with Intel. Those same relationships and the process to form partnerships for advances and innovations were something that Bryan Black used effectively and at scale to achieve this sort of positive outcomes.

I really appreciate CAMM for pushing the limits of memory on mobile devices to where soldered memory is not worth the lack of upgradeability in almost all use cases.

Yea memory bandwidth is basically all for LLM inference. To the point that the H2 (Chinese market card) manages to outperform the H100 on most inference workloads with something like 10-15% of the compute. The problem with a 3070 with 70gb of memory is that much memory would still make for a pricey card, and that much gddr6 is power hungry, especially when paired with older 8nm silicon. That would make a card like that a non-starter for the type of hw typically in-use at the edge. Edge inference is on their minds at nvidia but for now it’s on cards like like L4 which run much cooler and fit in a much smaller footprint than a 3070. That’s only 24GB of memory but that’s still enough for a lot of uses models you’d actually want at the edge right now. If your looking for something that works for you personally locally the 128GB variant of the new apple M3 chip is extremely compelling, decent gpu performance and because the memory space is unified you can actually load the full model in without having it use up 2x memory.

@@MikeGaruccio you’re right that 70gb of vram will have its own limitations but it’s not like it’s a brand new problem. Nvidia already does something similar with the Jetson Orin. It’s a severely cut down 3050 with a few accelerators and 64gb of unified memory. It has an arm chip as well. That can do 3-4 tokens per second with 70b llms. It is still expensive but there can be further cost savings if they get rid of all the robotics connectors etc. Right now, it’s about half the cost of a used m1 ultra. They can probably offer something like this for about $2k. It’s a price point a lot of people running local llms are willing to pay if we can get decent response time with 70b llms. I don’t want to play around with multi gpu setups as they’re not reliable and too inefficient. There are power savings as well if we have a dedicated card/device for inferencing. Something like this needs to be readily available for edge inferencing.

@@theworddoner the difference here is still the memory. The Orin is 64GB of lpddr5 vs. a hypothetical card with a comparable amount of gddr6. Those are completely different price points and power draws. A 3070 with that much memory would be pulling at least 350w, so it doesn’t really work for the edge (basically need passive cooling and

It is very good technology but expensive. Gamers could not afford them but AI companies could.

@@kazedcat I mean.. gamers kinda could. The radeon cards just didn't have enough performance otherwise. The 16GB of hbm was the star or the show and because of that the card is still pretty usable.

@@disadadi8958 If I remember correctly their HBM GPU was late to the market that is why the performance was not competitive. They then switch back to using GDDR in the next generation because HBM is expensive. HBM then find its niche in the embedded market until the AI boom.

@@kazedcat It more about : AI companies are in demand and willing to pay any astronomic price so why not prioritizing them in disfavor of the consumer market and other low-margin markets like embedded.

@@kazedcatI guess it depends on the model. For radeon VII certainly, it missed the mark against 2000 series RTX cards. However, the Vega 56 and 64 (released in 2017) were using 2048-bit memory bus with 8 gigabytes of HBM2. Those were significantly more competitive cards, than the 16GB HBM2 wielding Radeon VII released in 2019. The Vega 64 wasn't far off from gtx 1080 performance, albeit it was released a year later than the Nvidia counterpart, and Nvidia released the 1080ti to crush the competition. 1080ti had more vram (11GB) and it even reached the same memory bandwidth of the Vega 64's HBM2 with smaller memory bus.

They probably have IP licensing revenue but that is peanuts compared to their other businesses. Their gaming console revenue alone would dwarf their IP revenue and that console chip business is the lowest margin segment of their income. AMD has a lot of businesses.

@@kazedcat it's a little bit crazy to me that the main memory standard used in ai GPUs were idealized by AMD and not Nvidia like how did that even happened 😭??

@@lucasfernandesgrotto6279 AMD likes to bet on technology sometimes it works sometimes it doesn't. Bulldozer was failure but they are trying something new with that architecture. Chiplet on CPU works and it is now giving them advantage over Intel. Chiplet on gaming GPU is kind of not working but maybe they can make them work later. HBM was one of those bet and the technology works but the financial side was not working for them then.

@@lucasfernandesgrotto6279 "you get to do the memory, and we'll do compute."

Rocm doesn't work init?

What about ROCm?

​@@hurricanepootis562AMD's equivalent to CUDA cores

Rocom is lacking ngl

​@@n45a_rocom deez

The advantage of the traditional 6 transistor SRAM cell is power consumption. You can make a 4 transistor cell where you replace two transistors with pullups. However, those pullups (one or the other depending on the state of the cell) will permanently drain energy, where as the 6T cell will have 0 static current, other than leakage through the SiO2. It's also more complicated to manufacture since you need to fit a high value resistor above the transistors, on the same footprint, or you will effectively have negated the space saved from reducing the amount of transistors. And any fewer transistors than that, and you've effectively built a DRAM cell, with similar considerations as any other DRAM. Bigger picture: what are you going to use the imagined "R&D" device for? Economy and scalability is everything. There probably aren't many scenarios where giant SRAM makes sense over buying more compute power and/or DRAM and parallelizing the tasks. Certainly not for producing millions of phones or thousands of AI devices for a datacenter. The realization in the discussion about HBM is that sometimes it isn't the silicon that's the limiting factor, but the system design and the interconnects.

@@GameboygeniusI think if some genius did invent a better SRAM, it would immediately be applied to making bigger, faster cache on processors.

These are engineers! When they run out of letters, they just start on the greek alphabet.

precision machined heat spreader that takes the differing heights into account, or separate heat spreader(s) maybe?

You are thinking about GDDR6 and GDDR6X. Its a different type of RAM than DDR which feeds your CPU. The G stands for Graphics, it is a memory standard designed by the computer industry to be a much much higher bandwidth memory than DDR.

1) yes, but not everywhere. Sure, the google intern in SF may earn six figures, but I doubt that is going to continue for much longer, now that VC cash is no longer free. We have seen big waves of Software engineers getting fired, and I don't think this will stop. If you live anywhere else in the world the delta is probably much smaller, but it highly depends on who you work for and how well the economy is doing. 2) I cannot say. There are always improvements and constant research into how to make faster, cheaper and better chips, so I don't think the industry is at its peak. If you work for the right people you can do a lot of lateral moves in the industry to always work in an interesting and lucrative role. Unfortunately it is also an industry with boom and bust phases, so the future may look bleak, at least in the short term. I would look at what industry you personally find more interesting. Just chasing the higher paycheck is a great way to land in a job you don't care about with bad working conditions, too tired and sad to use you 10% higher income. If you just cannot decide you should look into doing an internship, not only will it look great on your resume but it will make the decision a lot easier.

semiconductors all the way, semiconductor industry just shifts to the next generation, the next generation seems to be AI specific semiconductors, after that there will be a next big new thing, it keeps on innovating, for me the software industry seems too saturated and with the rise of AI many tasks of software industry will be replaced by AI basically writing software itself, but it will all be powered by semiconductors

If you want your memory to be fast, direct connections between memory and logic using gates makes for fast access time but it requires power otherwise the memory will be wiped, which is what RAM is. If you want a drive to keep memory after the power turns off you have to use a solution like a disc which retains data but takes a long time to access. The engineering is all about tradeoffs so I’m sure there’s fast access permanent memory designs but they cost more in space and transistor count or have some other negative otherwise they’d be used instead.

@@submachinegun5737 I see, thanks.

If you want memory to be permanent, as in retaining it's state when powered off, you typically need to modify the properties of the material. In the case of flash memory, charge the floating transistor gate of a memory cell, which is done indirectly. In the case of a hard drive, changing the magnetic polarity of the grains in the material. These changes take a bit more time. For RAM you "just" need to charge up a simple capacitor (DRAM) or turn on a pair of transistors (SRAM) which means it can be much faster.