If anything more efficient AIs running on consumer hardware would increase demand lol. Look up Jevon's paradox. The steam engine increased the demand of coal by means of its efficiency, not decreased. Just gives more reason to get more hardware to take advantage of the scaling for large companies, and for consumers to buy more GPUs since they can finally take advantage of them. Coal companies benefitted from the efficiency of the steam engine causing increased demand, so too will nvidia benefit from the efficiency of quantization

great point, they would probably now shift to even bigger models while we get a handle on the smaller ones

Without understanding the paper, I do understand the quantization, and it's hard for me to believe they could go down to 1 bit without drastically losing quality, considering even 4 bit models are pretty bad compared to full bit models. Even 4 bit starts to feel like an AI has been given a lobotomy.

This reminds me of something Carl Sagan said, "Extraordinary claims require extraordinary evidence". It would be extraordinary if true, but 1 bit? I am sure it will be tested. I feel like 8 bits is the sweet spot, but IDK.

Will you be doing more videos in this sort of area on LLM/Machine Learning related research/papers etc? Thanks

Subscribed. Waiting for that update!

Hey Loved this ! Been thinking a lot about how to run these models on Local hardware, Could you also cover LLMs in a Flash - A research paper by apple addressing this issue

very interesting, thanks for sharing

Hey just wanted to complement your clarity of presentation. You’ve got the core of a good channel. Clarify the niche you want to go after so I can understand how it helps me out as a viewer and you’ll skyrocket.

I heard yesterday about some people selling their stock in Nvidia. Makes you wonder if the word of this has gotten around. Between this and Groq inference accelerator cards, who needs Nvidia?

such a nice explaination on 1 bit Large language models.

Do you have a link to your video on embedding spaces? It wasn't obvious from a scan of your channel.

I am now subscribed and I liked your video!!!

This is really cool! Do you know where the model can be installed and how to run it on ollama?

Very interesting running the bitnet examples on Pi5. The matrix outputs are 1, 0, -1, -0. Not sure what -0 means but 1, 0, -1 reminds me of those old Russian Ternary computers. I like to think of these as yes, no, maybe. I wonder if it could even speed up Stable Diffusion even more? Running AI on SBC is a game changer. I use this Pi5 as my home Desktop PC now.

Great vid. btw it HAS been done before, there is a BinaryBERT model that uses Trits on the backend

Totally agree with you. What also could get interesting, is that non-batched LLM inference is less compute and almost totally memory-bandwidth bound. And if you compare an Apple M2 Ultra with its 1024-Bit memory bus (and huge RAM) with Nvidia, it does not compare to badly on inference. However in prompt-processing,… the 4090,… is 10x faster. If compute can be reduced, a broader memory-bus (much cheaper than Nvidia‘s VRAM) will get very interesting. The reduced size is an additional benefit, because its less transfers from memory. Llama.cpp is already doing great work on SOTA quantization down to 2 Bits. I will be looking forward, if they manage to support the 1.58 bit algorithms (and reduce the math)

i am trying to wrap my head around this. And yes, the matrix math simplification method makes a lot of sense. But what i can not understand is why NVDA also would not start to use these 1-BIT LLMs also? It seems to be more of a "Software" approach, rather than baked into some firmware....so you can take a blackwell chip, USE 1-bit LLMs on it, and have amazing computational power! right?

While I'm excited no one has released a model that uses this paper yet. Hope it happens soon

Gotta love that “price is right” jingle😂

What camera do you use?

If anything we will see the push for 1T models or the like and much bigger models because we don't know what happens as models get bigger

nVidia A100 tensor cores have a binary mode that does exactly the acceleration you are talking about. However they removed it from later generations such as H100. Seems like a mistake. Also the 1bit nVidia approach is not suited for the 1.58bit ternary approach that a later paper has suggested.

Subscribed ❤

Wow! This sounds very significant. Looks like aspiring hardware manufacturers shud start designing hardware and related library with full throttle. But if they want any chance against Nvidia, then, they gotto build scalable hardwares for both end users and enterprises. Making Enterprise only things like Groq will never make any dent even in enterprise market.

good content

has no one tried analog opamp based multiplicators?

🤔 I wonder if a person couldn't get the kernels (if indeed you would even call them that with such an architecture) embedded into an FPGA for a proof of concept of how efficient dedicated hardware would be for this method. I want to say Groq has a hardcoded fixed function hardware that's insanely efficient for the process node (14nm I think, compared to Hopper...5nm maybe?), and while FPGAs aren't quite as efficient as ASICs in terms of price to performance, they're still quite a bit more powerful than GPUs for the same silicon in areas like this, from what I've seen. My intuition is that you'd probably need to network several of them together to get to any reasonable size of model, but once you did, the bandwidth would be honestly insane, and the hardware would be quite scalable.

idk, it's crazy how much technology isn't being used. like no fuel injectated auto detonating gasoline engines. Heat recirculating ice, True gear cvts. Metabolism slowing life extension. thorium nuclear reactors. Self powering heat engines inside air conditioners.

I think they used full floating-point numbers for training and then quantized the matrices.

Optimizations wont kill nvidia. OpenAI will need bigger GPUs either way because they just wanna train bigger and bigger models. Also a lot of the vendor lock-in happens with their software stack, not their hardware.

good video

the future for AI trianing is with analog ICs that is the only way to democratize them

Groq speeds with consumer hardware. One can dream right.

when I saw another video about the paper I couldn't avoid to think about the possibility of it being a joke. But who knows.

Stumbled upon this vid on my feed. I am VERY CLEARLY not informed at all on what you're talking about. Any tips to start out on the technical side of what you're talking about? I would say i have decent knowledge in tech all around, way above average

Oxen-AI has a good vid on running it and even a github repo...

This addition on fixed-point numbers thing is awesome, I have been thinking about what will happen when this is proven, will old Pentium 4s plugged into janky pirate-motherboards like bitcoin miners and graphics cards come-about? The thing about ALL the 'old' processors that are just laying around is that they take a lot of power, but A ghz is a ghz and a core is a core (when we are talking about just matrix addition). But yea all that electricity. Its cool when you start thinking about the number of radios that exist (billions and billions) in old smartphones that perhaps lower-spec internet-of-things-type ai-driven multi-computation could use. There are so many antennas/radios in laptops and phones all over the earth.

This optimization could lead to another GPU shortage (for consumer GPUs). Back then, it was due to crypto, now a bunch of AI startups and average companies can use RTX 4080 for their businesses.

There is no foundational model of this size for BitNet. The authors trained only 3B parameters model, how it will function in 70B no one knows. Those small models are so weak at those sizes. Another problem is that 72B parameters are still very weak models, looks like for anything useful you need grok size at least - 314B parameters, or maybe 250B at least, of gpt3.5 All this means you will still need a powerful gpu to run a useful model that can perform for real life situations tasks, just maybe one A100 80GB GPU will be enough and not two or eight.

ai won't be real for smb mkt for 5 years - it is going to take that long - really

No one needs to be scared at all of better AI performance... Also Llama 30b have horrible performance on my RTX4090 compared to Llama 13b.

I feel like the channel author is lacking a lot of context around NVIDIA's position right now to be able to make these statements.

China has a chip that (Acell) that is 3000x faster than an A100 and uses 500x less power