Attention in transformers, visually explained | Chapter 6, Deep Learning
Переглядів 846,531
День томуDemystifying attention, the key mechanism inside transformers and LLMs.
Instead of sponsored ad reads, these lessons are funded directly by viewers: 3b1b.co/support
Special thanks to these supporters: www.3blue1brown.com/lessons/a...
An equally valuable form of support is to simply share the videos.
Demystifying self-attention, multiple heads, and cross-attention.
Instead of sponsored ad reads, these lessons are funded directly by viewers: 3b1b.co/support
The first pass for the translated subtitles here is machine-generated, and therefore notably imperfect. To contribute edits or fixes, visit translate.3blue1brown.com/
And yes, at 22:00 (and elsewhere), "breaks" is a typo.
------------------
Here are a few other relevant resources
Build a GPT from scratch, by Andrej Karpathy
• Let's build GPT: from ...
If you want a conceptual understanding of language models from the ground up, @vcubingx just started a short series of videos on the topic:
• What does it mean for ...
If you're interested in the herculean task of interpreting what these large networks might actually be doing, the Transformer Circuits posts by Anthropic are great. In particular, it was only after reading one of these that I started thinking of the combination of the value and output matrices as being a combined low-rank map from the embedding space to itself, which, at least in my mind, made things much clearer than other sources.
transformer-circuits.pub/2021...
Site with exercises related to ML programming and GPTs
www.gptandchill.ai/codingprob...
History of language models by Brit Cruise, @ArtOfTheProblem
• ChatGPT: 30 Year Histo...
An early paper on how directions in embedding spaces have meaning:
arxiv.org/pdf/1301.3781.pdf
------------------
Timestamps:
0:00 - Recap on embeddings
1:39 - Motivating examples
4:29 - The attention pattern
11:08 - Masking
12:42 - Context size
13:10 - Values
15:44 - Counting parameters
18:21 - Cross-attention
19:19 - Multiple heads
22:16 - The output matrix
23:19 - Going deeper
24:54 - Ending
------------------
These animations are largely made using a custom Python library, manim. See the FAQ comments here:
3b1b.co/faq#manim
github.com/3b1b/manim
github.com/ManimCommunity/manim/
All code for specific videos is visible here:
github.com/3b1b/videos/
The music is by Vincent Rubinetti.
www.vincentrubinetti.com
vincerubinetti.bandcamp.com/a...
open.spotify.com/album/1dVyjw...
------------------
3blue1brown is a channel about animating math, in all senses of the word animate. If you're reading the bottom of a video description, I'm guessing you're more interested than the average viewer in lessons here. It would mean a lot to me if you chose to stay up to date on new ones, either by subscribing here on UKposts or otherwise following on whichever platform below you check most regularly.
Mailing list: 3blue1brown.substack.com
Twitter: / 3blue1brown
Instagram: / 3blue1brown
Reddit: / 3blue1brown
Facebook: / 3blue1brown
Patreon: / 3blue1brown
Website: www.3blue1brown.com
@3blue1brown 25 днів тому
A few added notes based on common comments I see. Concerning masking self-attention, several people ask about cases where it feels like later words should update the meaning of earlier words, like languages with adjectives following nouns. The model can always put the richest meaning into the last token (e.g. early nouns getting baked into later adjectives). For example, @victorlevoso8984 noted below how empirical evidence suggests the meaning of a sentence often gets baked into the embedding of the punctuation mark at its end. Keep in mind that the model doesn't have to conceptualize things the way we humans do, and in all likelihood doesn't at all, so I wouldn't over-index on the motivating example given in this video. Also, one thing I should have called out more explicitly is how I personally like to think of vectors like embeddings, keys, queries, etc. as columns, and as a convention display them this way, but other sources, including the Attention is All You Need paper, may present them organized in a row-by-row fashion. This is relevant to parsing the equation shown at 10:29, where the expression from the paper that looks like Q K^T would, by the conventions of this video, instead look like K^T Q.
@thuanphantrung3298 25 днів тому
I hope you make a video explain about State Space Model in Mamba, it quite confuse me with the derivative of x
@victorlevoso8984 25 днів тому
Ugh youtube aparently doesnt like me posting links to papers and deleted my reponse aparently so I'll try again. Source for what I'm saying is "Linear Representations of Sentiment in Large Language Models" but I haven't read it and have only heard a talk by the papers autors so take my vague explanation of it with a grain on salt. On a quick skim they mention the model moving summary information to "commas, periods and particular nouns."
@victorlevoso8984 25 днів тому
@MarcusHilarius I had a longer response that got deleted by youtube (probably cause I linked to my paper as an example) but I think this is not exactly correct in that the purpose of the qk matrices is for each attention head to approximately only "read" the information in some of the tokens. So at each position the model could have the meaning of all the previous tokens on it but it doesn't have to, and theres all kinds of interesting dynamics about wich positions attend to each position in each layer and attention head for a given input.
@tantarudragos 25 днів тому
Row-by-row is actually in some sense the standard notation in Deep learning, since most (all?) popular frameworks use it internally. I'm not actually sure why, I guess it's just an old GPU programming convention. It's not uncommon for researchers to use column notation in their papers, but personally I think it might be confusing for people new to the field, as the code will almost surely treat the vectors as rows (after all, usually the first data dimension is the batch size). It's not a very significant thing of course.
@TomassWilsonGT 24 дні тому
The most common association I've heard as to why there needs to be masking, is that otherwise the model could just cheat. Every word would have the same query, "hey, what's the word after my position x?" and then the keys would all be "I'm at x+1" or similar. The value at the x+1 embedding would just be the word itself, and it would know what it needs to write. Your model would always perfectly predict the next word. This is still interesting though. What if instead of asking the model to predict the next one word, we asked it to predict the next 10? You'd implement it by sliding the triangle in the masking matrix 10 steps (masking more). Your loss would rise dramatically, since even if the model gets the "gist" of what it should write (like if you want it to talk about "Harry Potter" somewhere) it would still be off if due to sentence structure the words "Harry Potter" weren't in the exact same place as in the original text. You'd need to change your loss calculation to somehow focus more on the "semantic correctness" of those 10 words (a challenge I'm sure), but the benefit would be that the model could generate whole phrases or sentences. I'd probably do this in some staged manner, like starting with one word loss and increasing the number of words as training goes on. I'm not sure what the benefit would be though, but might be fun to try.
@philrod1 26 днів тому
I'm a university lecturer with a PhD in AI, and I cannot compete with the quality of this work. Videos like this put the entire higher education system to shame. Fantastic! ❤️
@ArtOfTheProblem 26 днів тому
so true
@zach4505 26 днів тому
If you haven't already, check out the FAQ on his site about his animations. Its called Manim and is available for anyone. It seems that Grant is on a mission to create a Math and Code Animation Army. Sign me up! I teach high school CS without a full CS degree and I was able to use it in my classroom. You can install or just test it on a Jupyter notebook. Still requires an art to scaffold the instruction, but a good visualization can expedite the process.
@iTXS 26 днів тому
Another role of universities I think is to hold the knowledge system and methodology with the help of social systems and social interaction inside and with another society services also via social interaction. So, now the balance has been shifted, making Unis more provide holding services and UKposts/Online/GPT - educating and transferring the existing knowledge to wide mass of people, just because huge multiplication factor (dozens of millions of viewers vs thousands of students). This new balance shouldn't be broken or rejected but taken into account to increase quality of education as a phenomenon.
@philrod1 26 днів тому
@@iTXS - For sure. I, and the wider higher education sector, need to up our game.
@monad_tcp 26 днів тому
@@iTXS they are failing hard at that too, specially because their "social systems" felt prey to some ideologies that are not conductive to critical thinking. We need to create a better system starting from what we already on the internet. " provide holding services " Why would we need that ? Open Source has solved that problem for software, Science just need to use our software development tools. You don't need the bloated organizational structure of universities at all to actually generate and curate knowledge to a high degree of success, that's what we software developers did with things like Linux. You guys are just stuck in the past.
@actualBIAS 26 днів тому
Are you kidding me? ONE WEEK FOR 2 MASTERPIECES?! Thank you so much!
@user-vt4bz2vl6j 26 днів тому
Waiting here for someone to correct your spelling...
@Scubadooper 26 днів тому
@@user-vt4bz2vl6j🤣 Peace man
@CoolCat123450 26 днів тому
I understand the context so no use dwelling on his misteak.
@skmgeek 26 днів тому
@@user-vt4bz2vl6jon the contrary, i hope nobody ever corrects the mistake just to spite you >:3
@noobintheinternet 26 днів тому
@@skmgeek Humanity never learns from its mistakes.
@Steamrick 26 днів тому
I've got to say - "Attention Is All You Need" is an incredible title for a research paper.
@VinayAggarwal 26 днів тому
Indeed
@xephyr417 26 днів тому
Famously so I think. I really really like these names instead of the long boring ones. But... Also people aren't clever and the creative names would be bad, so maybe we stick to long boring ones for the most part haha
@adityapatil325 26 днів тому
I like the boring ones. That way I can easily decide which ones to read.
@ColinTimmins 26 днів тому
It’s going to be embedded into our future history lessons of when things began. =]
@JorgetePanete 26 днів тому
There are now more than 300 papers ending in "is/are All You Need"
@hailking5588 26 днів тому
As a graduating PhD student working in Natural Language Processing, I still found that video to be extremely beneficial. Awesome!
@mastershooter64 26 днів тому
Good luck for your PhD defense!
@dukeyin1111 26 днів тому
Wow, a nice competitive field! Best luck to you!!
@maxmyzer9172 25 днів тому
Good luck out there!
@Henry-fv3bc 26 днів тому
Attention existed before the 2017 paper "Attention Is All You Need". The main contribution was that attention was... all you needed for sequence processing (you didn't need recurrence). Self-attention specifically was novel though.
@AnitaSV 26 днів тому
Yeah 3 years earlier by Bahdanau et. al
@miriamkapeller6754 21 день тому
Before people debated how many self attention blocks you should stick into a model and where and if it was even worth it. The paper proved that more was better and that you're probably better off replacing linear layers and convolutional layers with more self attention blocks.
@waylonbarrett3456 14 днів тому
There have been many people working on this idea long before 2017. I, myself, was working on a model very similar to this in 2010, and I was inspired by papers and books going back much further. It seems people are not fond of appreciating multiple, independent instances of creation and discovery. It happens a lot in many domains.
@Haligonian 4 дні тому
@@waylonbarrett3456 Interesting, do you have a source for a much earlier similar model architecture?
@waylonbarrett3456 4 дні тому
@Haligonian the sources for their inspiration and basis are cited in the "attention is all you need" paper. I believe it's at the bottom in the acknowledgment section. Then, you go to those papers and look at their cited sources, repeat ad infinitum. This is the true, branching complexity of creative works. There's not often some lone genius having a singular "Eureka!" moment.
@sriramsrinivasan2769 17 днів тому
3b1b is the only content producer whose videos I start by first making coffee, then upvoting, then hitting the play button.
@pmnt_ 16 днів тому
... and then be disappointed at the end because you can't upvote a second time.
@DataRaeAI 25 днів тому
Geez Grant, I spent thousands of dollars on a very good deep learning executive certification from Carnegie Mellon, and your series here is better than their math slides. This series is really turning out great.
@jurjenbos228 21 день тому
I wish any of the online courses I did for work had this quality.
@Polyamathematics 26 днів тому
The drought of 3blue1brown content has finally been overcome!
@pyropulseIXXI 26 днів тому
Don’t jinx it!
@HariOmSharma-cw6rs 26 днів тому
It takes time to learn these concepts from scratch and make these high quality videos.
@georgesms 26 днів тому
As long as the videos keep coming well thought and well researched as they are, I don't care if they take their time! Congrats Grant and 3b1b team!
@ValidatingUsername 26 днів тому
Shalll I drop the next hint and see how quickly new content comes out?
@xephyr417 26 днів тому
WE FEAST
@QuantAI-kp8xt 24 дні тому
How I wish this video was available when the "Attention is What You Need" paper just came out. It was really hard to visualize by simply reading the paper. I read it multiple times but could not figure out what it was trying to do. Then subsequently, Jay Alammar posted a blog post called The illustrated transformer. That was a huge help for me back then. But this video raises the illustration to an entirely different level. Great job! I'm sure many undergraduates or hobbyist studying machine learning would benefit greatly.
@GPTandChill 26 днів тому
Thank you for the mention, Grant! For those who relate to the pain of wanting more practice problems for Machine Learning, I hear you. I’ve created coding problems (run against test cases in your browser!) & quizzes covering the core ML concepts. Check out the resource Grant mentioned (linked in the description) or just click on my channel!
@Nepal2015 26 днів тому
For real, GPTandChill got some good problems to practice on. Helped me out a lot
@rahilkapur2604 26 днів тому
love your videos GPTandchill very useful
@arunramuu 26 днів тому
Nice way to learn ML concepts from @gptandchill
@zaindarvesh9039 26 днів тому
I love your coding problems. They’ve actually helped me learn ML concepts better than most of my college courses.
@kavithagaddamanugu8083 26 днів тому
Love your videos. They are very detailed, organized and very helpful.
@MatheusC1729 26 днів тому
I cannot stress enough what a tour de force this is. It's probably one of the best math classes ever done anywhere in the world in all time. You're the best in the game and an inspiration for many. So so much thank you, Grant, you're doing God's work here.
@michaelthompson9862 25 днів тому
You not only put out some of the best content on youtube but also give constant shutouts to other content creators that you admire. You are a GOAT 3Blue1Brown.
@muelleer 26 днів тому
As a Master's student in Data Science and AI, I never really understood how attention worked. Thank you for making this video!
@jabirlang216 18 днів тому
A master student in DS and AI and never understood attention sounds wrong to me
@madghostek3026 9 днів тому
@@jabirlang216 He wasn't paying attention \s
@epistemicompute 4 дні тому
@@jabirlang216these programs are mostly non rigorous. I only hire MS statistics or MS CS
@rpbmpn 24 дні тому
Joining the list of commenters who have worked with transformers, built and trained them, explained them to others, and still learned really basic things from this video about what they're doing that just hadn't occurred to me. Great work.
@fakecubed 17 днів тому
Give an example.
@Just4Growers 26 днів тому
There are people … all over the world … like me … who really, really, really appreciate you. I cannot thank you enough for taking the time to share your knowledge and help others to understand this technology much more deeply. Seriously, kudos and sincerest thanks. ❤
@Otomega1 26 днів тому
Just Wow, the educational value of this video is incredible. There are so many highly relevant and original ideas to explain abstract concepts and drastically simplify comprehension. I'm so thankful that you've made this content available to everyone for free. I absolutely love it!!
@Otomega1 26 днів тому
Also I want to add, making such knowledge available to mainstream is truly a gift for humanity, and a thing that you deserve to be proud of
@glizzy2764 26 днів тому
If I could write poetry about how much I appreciate and learn from your videos, I would but I'm not a poet. Thanks to everyone who worked on these videos.
@captainrob4656 25 днів тому
oh how great the knowledge be, that grant can share so graciously. i would be lost in space and time, if you, my precious would not shine. it got a bit saccharine, but tried my best lol i do agree though, this channel is amazing.
@beavisandbutt-head5363 25 днів тому
@@captainrob4656 There is a secret tool named chat GPT. You can use it to help you with your poem creation problems 😉.
@KshitizRai1005 22 дні тому
In the vast digital sea, a beacon shines bright, Three Blue One Brown, with wisdom alight. AI, ML, and DL, in colors so clear, Guiding minds through concepts, year after year. With clarity and depth, their tales unfold, Grateful hearts cherish the knowledge they've told.
@LazGav 19 днів тому
In realms where algorithms dance and code reigns supreme, Where data whispers secrets and dreams in every scheme, There dwells a sage, a maestro of the digital domain, Whose words and visuals in tandem, ignite the brain. Through the ethers of the web, his teachings unfurl, Like constellations in the night, they guide and swirl, With each stroke of insight, he paints a vivid scene, Unraveling mysteries, illuminating what has been. Oh, if only words could craft the depth of gratitude, For the knowledge gained, the horizons pursued, Yet in this humble verse, let it be known and heard, The gratitude abounds, for the wisdom conferred. To the minds behind the screens, the unseen hands, Who weave the fabric of understanding in digital lands, We extend our heartfelt thanks, our deepest regard, For kindling the flame of knowledge, burning bright and hard. - By Chat GPT 3.5
@fakecubed 17 днів тому
Lessons truly grasped The videos open minds Thankfulness ensues
@scalex1882 25 днів тому
It is simply inexplicable how valuable these videos are to humanity, and I mean that literally. The way your videos convey the underlying ideas and intuitions of such complex technology opens the door for tomorrow's smart minds to learn even faster.
@brunojl2 17 днів тому
The volume of work, attention to detail and clarity we get from Grant is staggering. Bravo sir.
@user-ij7hn3gx5r 25 днів тому
The dissemination of scientific concepts transformed into art. Immensely enjoyable and useful for the education of millions. Thank you as many times as the number of model parameters!
@jonfe 26 днів тому
Finally someone explains it right. One simple thing but important that I would emphasize for the new people in AI is that all matrix/vectors used in this process are "discovered" by the model with the gradient descent mechanism (reward mechanism) for the especific task. In this cases, predict next word.
@miriamkapeller6754 21 день тому
This video is worth gold. Trying to understand exactly why and how self attention and cross attention works nearly brought me to tears these past years and this video explains it so well and with visualized examples.
@adriablancafort 26 днів тому
This video will become a part of history.
@MrZorroZorroZ 26 днів тому
bot
@Montywritespython 18 днів тому
@@MrZorroZorroZlmao
@annachester5790 23 дні тому
I'm a Computer Science student currently working with a Transformer for my master thesis and this video is absolute gold to me. I think this is the best explanation video I've ever seen. Holy shit, it is so clear and insightful. I'm so looking forward to the third video of the series!!!! The first one was absolutely amazing too. Thank you sooo much for this genius piece of work!!!!
@vedantranade1815 25 днів тому
You made the best attention video I have watched online. This should be a compulary part of all NLP courses right now. Thanks for releasing this banger!
@grimble4564 4 дні тому
I'm very much not a math person but I've been binging your stuff for a good 4 days now. Between the general sense of wonder, I keep being reminded of the HG Wells quote "History is a race between education and catastrophe" Idk if that ethos is something you deliberately bake into this channel but I very much see it everything you upload and I can't thank you enough. Idk if you fully appreciate the reach your efforts could have but either way, thank you for taking the time to break this stuff down while also not dumbing it down. I don't understand everything you say but you make me curious enough to figure it out and I feel genuinely empowered as a result. I'm sure I'm not just speaking for myself.
@chrisarets2231 26 днів тому
Rewatching the video is all i need apparantly :) On a serious note i've heard so much about this "attention" mechanic and always wondered how it worked so thank you for making this video!
@jafetsierra1875 25 днів тому
This is pure gold!. Never seen such a good explanation of the attention mechanism before. Thank you for this.
@SergeyNeskhodovskiy 23 дні тому
Man I admire your patience to explain this step by step and with great visuals. I can't thank you enough for making such complex topics somewhat comprehensible for people without the corresponding background. The day you started this channel deserves to be a national holiday.
@danverzhao9912 25 днів тому
Just incredible of how clear everything is explained for someone who's doing a nlp course right now, thank you so much!
@_The_Dumbass_ 26 днів тому
I will absolutely pay attention to this one
@DanPrince 26 днів тому
I work with ML for computer vision and have never really understood transformers. This is by far the most clear explanation of them that I have seen!
@Blattealkiller 26 днів тому
Thank you really, I never wanted to dive deep into how transformers work because Im not using them so much and so I had never understood these concepts as clearly as this, until then I was satisfied by the fact of just understanding the equations. But you explain it in such a clear way it's so refreshing. This is the number 1 reference series of videos on transformers that I would recommend.
@primechords 25 днів тому
Grant, your work provides immeasurable value. This video is a striking display of the towering heights mankind is able to achieve.
@monad_tcp 26 днів тому
I'm still absolutely fathomed we're now using O(n^3) algorithms like that because we have so much hardware in the last 10 to 20 years. That's the magic of LLMs, its not that they are "large", but that we have engineered enough hardware to be able to play with such things. But in the end its all just a lot of linear algebra, isn't it ?
@SianaGearz 24 дні тому
Imagine 25 years ago, you have your Pentium MMX, with, if you're lucky, an 8MB Voodoo and 32MB RAM, and 8GB disk, and you're told: in 20 years you're going to have a little supercomputer in your computer, thousands of operands wide, 8GB of RAM on there or more. You might not be entirely surprised, Moore's Law sort of checks out, after all you had a 486 just previously and a simple Trident VGA, and maybe a 1MHz Commodore 64 previously before that. And yet it is impossible to grasp this power. And that we'll all be using a dark faced chatting app that runs slower than AIM/ICQ did back then while not doing a whole lot more. But that you would also be able to run mad neural network stuff on your computer with an eerily almost human like capability in language and other complex fields.
@adiaphoros6842 26 днів тому
The video says that masking is for preventing later tokens from influencing earlier ones. How does GPT3 handle the sentence: "The cat that left the town came back this morning"? The phrase "left the town" modifies the noun "cat." Later tokens do influence earlier ones in this case.
@3blue1brown 26 днів тому
In that case, masked self-attention would have to do something a little different from how we humans might naturally think about it, for example, baking in all the meaning of a cat leaving the town into the embedding associated with "town".
@Scubadooper 26 днів тому
@@3blue1brownis it right to say that all of the meaning is embedded in the last token, so "cat ""left" "town" "came" and "back" are all embedded in "morning"?
@gaelyte2550 26 днів тому
@@Scubadooper Well, remember that the point is to have, on the last layer, the embedding on a word be the prediction of the next word ? So yes the goal is exactly to have the whole meaning of the sentence be on the last token
@verified2043 26 днів тому
@@3blue1brown Incorrect, the word "town" would become empty and meaningless once the cat leaves it, this example would make the GPUs explode.
@verified2043 26 днів тому
@@Scubadooper You can think about it like "THE MORNING", that important morning when the cat came back after leaving the town (again that sad town that the cat left) previously.
@djenning90 25 днів тому
The way these complex concepts are visualized makes the material very approachable for newcomers like myself. I appreciate this work very much!
@JonyBetancourt 20 днів тому
As director of video content for a major educational publisher, this is some of the best educational content I’ve ever seen. Your content gives me ideas of how to shape the future of undergraduate level STEM videos. A true legend and inspiration in this space- thank you for the meticulously outstanding work that you do.
@johncasti91 26 днів тому
To me the part that was more confusing when studying this for the first time was the seemingly arbitrary separation between Keys, Queries and Values. In particular, the Value matrix seemed sort of redundant if you already had the Keys, and the terminology coming from databases doesn't help much imo. This makes it much more clear, thank you!
@acasualviewer5861 25 днів тому
I think the separation exists because otherwise you'd need some super array for all possible combinations. At least that's the intuition I have about it.
@Damodharanjay 26 днів тому
This is history in the making. This video will benefit humanity for years to come! Absolutely best out there with amazing visuals
@klonopelki 10 днів тому
By far this is (with ch. 5) the best explanation of Transformer architecture that I've ever seen. I previously understood Attention intuitively but struggled to explain this during job interviews under stress. You just helped me land a job. Thank you!
@SantiagoPombo 22 дні тому
Congratulations, your transformers series is a technical education masterpiece. I’ve been building products with LLMs for the past two years and shamefully only understood them at a very superficial level. You have a gift for clarifying without oversimplifying and condescension that is rare.
@Sydra. 26 днів тому
So much value in one video. You are making the world a better place.
@Otomega1 26 днів тому
Truly
@MrFujinko 26 днів тому
You hold this opinion when someone unleash not aligned AI on the world.
@Otomega1 26 днів тому
@@MrFujinko then its the "someone" who IS the problem in the first place dah
@MrFujinko 26 днів тому
@@Otomega1 You ain't a teamplayer. This is modern world, everyone is to blame, everyone works for the system. You and I and anyone reading is guilty.
@krelly90277 25 днів тому
D.Trump can lead the resistance against misaligned AI; he has developed several novel strategies for reducing rogue AI systems into a harmless atavistic state.
@inlegivel 26 днів тому
Your work for humanity is invaluable. Aside from publishing free videos about these subjects, the effort you put into making something this complex to something reachable to all should always be commended. I have tried to understand this subject multiple times, reading the papers and everything, but I couldn't create a simple model in my head about what it was doing or what it brought to the table to be such a breakthrough in this field. Thanks a lot!
@Scubadooper 26 днів тому
Do you mean "immeasurable value"? I certainly value it
@nelkabosal 26 днів тому
Maybe you should say "is invaluable" vs "has no value". ;)
@inlegivel 26 днів тому
@@nelkabosal Thanks!
@inlegivel 26 днів тому
@@Scubadooper Thanks!
@donsanderson 16 днів тому
The Simpsons Dr. Nick: “Inflammable means FLAMMABLE??!”
@jojantti 13 днів тому
Grant is all you need. This was propably the tenth video or podcast about the subject and only now I understand the underlying motivation for each component it has.
@kazimhussain3590 25 днів тому
I have become a witness to such great genius with such a thorough understanding of these concepts. If at all there has been a list of greatest teachers, this guy needs to be up there
@nosy-cat 26 днів тому
Ooh, you've improved the patreon preview so much in just several days. Bloody well done, sir.
@3blue1brown 26 днів тому
Thanks, and my appreciation to you and others who helped give feedback on the earlier version.
@philwatson5132 25 днів тому
I’d really love a series like this on quantum computing. I don’t think any such video exists, let alone anything of 3B1B quality. Well, here’s begging. :)
@sanfor1997 8 днів тому
Through my journey in the academic world, I never understood a concept so good and familiarized myself with it compared to learning about it here. Thank you so much for your time and effort.
@matteomansi7499 25 днів тому
It's amazing how videos like this can cut your learning curve. I have attempted to understand self-attention in the past, but i found youtube videos or articles that were too conceptuals, or papers that got bogged down in mathematical notation. This is the perfectly accessible for anyone that understands ML basics and matrices and vectors. Keep up with the amazing work!
@Gumbly_ 26 днів тому
26 minutes of pure joy!
@user-zs4sm5yk9i 26 днів тому
This comment was 4 minutes after the video released!!
@user-zs4sm5yk9i 26 днів тому
YOU DIDN’T WATCH THE VIDEO!!!!!!
@sarthak_chauhan1010 26 днів тому
mtfk , this topic is so hard, can't understand what grant is saying after 6 minutes in the video!
@krelly90277 25 днів тому
There's only one other orator I like as much as 3blue1brown, just one. And that other orator who is just as pellucid, and engaging is Donald Trump. Yep, I only pay attention to 3blue1brown and Donald Trump; it's almost like they are the same person.
@denizcoskun111 24 дні тому
Your way of explaining concepts feels so interactive to me. Follows the narrative > A question appears > 3B1D answers it immediately > On to the next subject. Thanks for the superb content!
@robstamm60 25 днів тому
What an absolute masterpiece thank you so much for breaking this down - I read that paper over and over but never reached the enlightment this video presents ❤
@chandankumarmishra336 25 днів тому
Videos like this put the entire higher education system to shame. Fantastic! I can’t believe you can watch such an excellent lesson on UKposts for free!
@christiand3285 26 днів тому
I currently should be studying chemistry and instead I'm here watching your awsome videos. Thanks for presenting these topics so they are interesting
@Shrooblord 25 днів тому
These series are so very good. Thank you for making the dense field of machine learning more accessible.
@roy-ub7sy 25 днів тому
I can see how much work has gone into creating this masterpiece. Thank you for sharing this with the world.
@JuergenAschenbrenner 26 днів тому
Would there be a Nobel price for educational content this here would be a strong contender, well done!
@seblund 26 днів тому
Great video as always! Minor quibble at 9:00, I have always heard and understood “attend to” as being from the perspective of the query (the video uses the key’s perspective) so it would be “the embedding of creature attends to fluffy and blue” instead. It doesn’t really matter since the dot product is symmetric, I just haven’t heard it used colloquially that direction (maybe due to the axis that the softmax is applied on?)
@ryanbaten 26 днів тому
Skimming through to determine if this would explain it well to friends who are interested but yeah, I noticed all the matrices in this video are transposed relative to the notation. He's not wrong with what the calculations are doing but it's more confusing this way, especially when he shows softmax(qk^t/z)v as the notation. With this formulation, Q,K,V should have rows of projections so the attention matrix is properly formed. Softmax is then done over rows so when multiplying with V on the right, you're aggregating as weighted rows of V.
@Henk-ms5ml 16 днів тому
I had the same initially, but I'm not a native english speaker. I googled the meaning of attend to and it gave me: "has to do with". So if you think about it like that, fluffy and blue have to do with creature, not the other way around. Maybe that helps.
@drstrangeluv1680 26 днів тому
I woke up this morning and was feeling excited because I knew you will drop the transformers video today! That is what your videos mean to me! Thank you!!! :)
@gnfrank 22 дні тому
What a beautiful illustration of attention. I want more on this topic!! Variations, details, depth, concepts, illustrations - this is just wonderful. I could watch this video on repeat and learn something new each time.
@ethanfrink7059 26 днів тому
When I originally learned how matrix math worked in 9th grade, I originally asked the question many students do. “What’s the point?” If I ever end up teaching 9th grade math, I have a lot of cool examples to share with students that are similarly doubtful
@SianaGearz 24 дні тому
By the time i was at that point at some time in the 90s, i already tried implementing a 3D engine so i knew how useful they can be. But back then, we didn't really have SIMD. This came a little later, which made it that much more relevant. I think it's similar, in that you're reformulating things to make them as regular and as mutually independent as possible specifically in order to be able to parallelise them, but it gives you some neat advantages and expressive power beyond that as well. So many mathematical concepts could be considered optional but all of them increase your expressive power.
@X_platform 26 днів тому
Watching this video made me realize how convoluted the terms we, the research community of AI, have made deep learning to be. Planer / simpler terms are needed in research paper as paper like Attention is All You Need is read by all generation of researchers coming after us.
@AzzziRel 26 днів тому
I'm afraid this applies to all the science fields. People tend to create jargon in subjects they are working on to make their job more efficient / faster for them and others like them. So papers are shorter, more compressed. Others to be able to read it need to learn those "compressions algorithms" ;) Unfortunately it makes it harder to learn anything from scratch so we need people that "decompress" it for us and teach us ho we can do it ourselves.
@venugopal-nc3nz 26 днів тому
Researchers use tough language to keep outsider out
@AzzziRel 26 днів тому
@@venugopal-nc3nz Nah, it's very natural process that happens for everybody. More you know more you "compress" the data to make it more effitient to process. That way one sentence can have very deep meaning for others like you and you don't need to write 20 pages to describe new concept you are working on. In fact deep learning implements exactly this: very sophisticated compression algorithms ;)
@SianaGearz 24 дні тому
Computer science is a field so full of misnomers, that it's itself a misnomer. It's a field of engineering not science, and its subjects are not computers, those are the subject of computer engineering (electrical engineering), but software. It's also not software engineering per se, which is concerned with software architecture, though that can be considered a sub-field of computer science. It is more like information processing engineering, with heavy focus on algorithms and data structures. Unfortunately a certain amount of hindsight is needed to name things elegantly in a way that would be more leaning towards self explanatory, but there is resistance to renaming things, because it would break existing workflow.
@AzzziRel 24 дні тому
@@SianaGearz I would argue that computer science is part of science, at least AI part we are talking about. Of course there is a lot of engineering there but without science those engineers would have nothing to do ;)
@utilka5415 25 днів тому
mind blown. had to spend another hour laying down processing all concepts here and what they entail a really impactful explanation
@user-is5lv5kk9s 22 дні тому
These videos should be on the Trending list. One doesn’t necessarily need to have a CS/Math background to appreciate this content and the immense effort behind creating this masterpiece! Grant, the world is indebted to your contributions in revolutionising how Math can be taught! You TEACH intuition!
@randomenia 25 днів тому
Hi , i'm not good speaking English , I'm learning and I wanna say thanks you, in spanish there aren't these movie (i'm sorry i don't know say this in youtube, I don't know if this say video like spanish) well, Thanks you I am 16 years old and this help me
@jeremypianofreestyle7210 26 днів тому
It is 1:00 in Sydney right now and I’m up late watching your video from my bed. I should probably get some sleep, I have morning classes, it’s just your content is to God damned interesting. Plus, I’m a teenager. I can’t be separated from my phone accept by 16th century French style beheading. POST MORE VIDEOS! If I can’t sleep you shouldn’t get the luxury!
@HarsimranBhasin 16 днів тому
I already knew about Attention mechanism. Jumped onto this video since it was recommended in a newsletter. Would like to thank you for explaining things in concise and easy-to-follow manner with graphics. Learned many knew things and implementation details.
@Srednicki123 26 днів тому
Great, great, masterpiece...I read a lot about transformers and attentions mechanisms, but this makes everything so much clearer..
@Ed_Snaider 26 днів тому
Indeed, Tranformers are more than meets the eye
@IceMetalPunk 25 днів тому
If we put a mask on Figure 01 to make it look more human... would it be a Transformer robot in disguise? 😂
@SianaGearz 24 дні тому
Real transformers have windings. And they constantly buzz or hum faintly.
@AkariTheImmortal 26 днів тому
I've been spending a lot of time, trying to understand transformers, before watching this. I have built a very very small one myself, to better understand, but with my limited resources, it can't do much. Anyway, the Q,K,V was very confusing to me, but now I finally understand. The thing that is most difficult to wrap my head around are the insane dimensions sizes and parameter counts. Those numbers are so high and all of those multiplying together, to get 57 billion parameters for the attention block themselves, is insane and fascinating to me. And then adding the rest, that is not attention...
@fernandoruizruiz2472 26 днів тому
Hi Akari! If you have time, I would love to talk with you about your process of building the transfomer. I"m , as well, in the process of understanding all this world. What do you think about having a chat? We learn more when we teach ;)
@anuragsangem1442 24 дні тому
The amount of intellectual satisfaction this series has brought me 📈📈📈 Thanks a Ton.
@aminzqrti7672 26 днів тому
I can’t believe you can watch such an excellent lesson on UKposts for free!
@creativenametxt2960 26 днів тому
Q: at 24:27 we learn that all the layers can have different meaning behind how they manipulate with words and each one is not realted to another I get that the final output needs to be the same dimention as the original word in order to nudge it, but why are the value down matrix sizes the same (the rank limitation, the compression from 12k to 128 dimentional space) if some layers may benefit from more dimentions while others might not need as much?
@kaiduhrkop4335 25 днів тому
I found his explanation of Value Down/Up a bit unnecessarily complicated first, but I get why he explained it this way. As you said, the output has to match the input (because transformers are also "residual networks" where input and output always lie in the same vector space). So the value matrix has to be a huge symmetric matrix (if you don't use this factorization trick). Query/Key on the other hand don't have to be that huge - in fact if they are too huge then chances are high that everything is not related to everything (because the chance that two random vectors in a huge dimensional space point into the same direction is extremely low). The idea of multi-head attention is that you make query, key and value matrix all same size but split them into several blocks/heads. This way, each block of the value matrix is multiplied by a different attention pattern matrix. The idea is that you keep the number of key, query and value parameters the same while increasing the expressive power of the model (or to use the metaphors from 3blue1brown: you can now ask multiple questions in a single transformer block).
@creativenametxt2960 25 днів тому
@@kaiduhrkop4335 I see, so some matricies can be recycled and we just have more attention matrix parameters if we want to increase it
@kaiduhrkop4335 25 днів тому
I would not call it "recycling". In the previous video he showed that certain directions in the embedding space correspond to "gender", others to, say, "color". Now, very different words in a sentence are important to determine the gender of a word than its color. So you can think of one attention head is searching for words that are associated with gender and only effects the gender dimension in your embedding. Another attention head reacts on colors and only changing the color dimension. In extreme case you could have a single attention head per dimension, but then your attention mechanism would be one-dimensional and very lame. So instead you have one attention head that influences, say, 128 different dimensions of your embeddings that all are influenced by similar words (e.g. they all attent to adjectives) while another attention head is influencing other 128 dimensions and attents more to verbs. Multi-Head Attention is an in-between solution between "one attention mechanism per dimension" and "one attention mechanism for the complete embedding vector".
@PreethamS-th1fg 25 днів тому
At the end of the day it is the mathematicians who have the best understanding of any concept, be it physics or AI or just pure abstract math. Thank you Grant for your amazing contribution and taking us along this journey...
@mccleod6235 25 днів тому
Brilliant! Even when you start to understand how it works, it seems miraculous that it works as well as it does. I love the idea of adjusting the embedding vector to new positions in semantic-space depending on the learned relationships with previous words.
@elonitram 26 днів тому
I see upload => I watch
@L17_8 26 днів тому
Jesus loves you ❤️ Please repent and turn to him and receive Salvation before it is too late. The end times written about in the Bible are already happening in the world. Jesus is the son of God and he died for our sins on the cross and God raised him from the dead on the third day. Jesus is waiting for you with open arms but time is running out. Please repent and turn to him before it is too late. Accept Jesus into your heart and invite him to be Lord and saviour of your life and confess and believe that Jesus is Lord, that he died for your sins on the cross and that God raised him from the dead. Confess that you are a sinner in need of God's Grace and ask God to forgive you for all your sins through Jesus. Jesus loves you. Nothing can compare to how he loves you. When he hung on that cross, he thought of you. As they tore open his back, he thought of your prayer time with him. As the thorns dug into his head, he thought of you spending time in the word of God. As the spears went into his side, he imagined embracing you in heaven.
@godfreypigott 26 днів тому
@@L17_8 SPAM
@deanvangreunen6457 26 днів тому
Love your content ❤ best learning channel
@deanvangreunen6457 26 днів тому
I consider you a great teacher, one of the best in the world 🎉
@L17_8 26 днів тому
Jesus loves you ❤️ Please repent and turn to him and receive Salvation before it is too late. The end times written about in the Bible are already happening in the world. Jesus is the son of God and he died for our sins on the cross and God raised him from the dead on the third day. Jesus is waiting for you with open arms but time is running out. Please repent and turn to him before it is too late. Accept Jesus into your heart and invite him to be Lord and saviour of your life and confess and believe that Jesus is Lord, that he died for your sins on the cross and that God raised him from the dead. Confess that you are a sinner in need of God's Grace and ask God to forgive you for all your sins through Jesus. Jesus loves you. Nothing can compare to how he loves you. When he hung on that cross, he thought of you. As they tore open his back, he thought of your prayer time with him. As the thorns dug into his head, he thought of you spending time in the word of God. As the spears went into his side, he imagined embracing you in heaven.
@godfreypigott 26 днів тому
@@L17_8 Cease posting your spam and liking your own comments.
@Sapiens_Kim 26 днів тому
It is very helpful video series. This lecture provides the main features, their principles, and reasons for using them. The most surprising thing about this lecture is that if you watch this series hard, even ordinary high school students can easily and quickly gain relevant knowledge and submit simple reports.
@adkh2112 24 дні тому
The quality of this video is truly inspiring. Clear and concise explanations, great examples and analogies, highly visual.... I like to think of myself as a good science communicator and this is the level that I want to attain, fantastic
@glitch933 26 днів тому
Consistency 🗿
@InturnetHaetMachine 26 днів тому
Great video. Kind of saddens me that we're leaving the golden age of open sourced AI research. "OpenAI" used all the publicly shared transformative research from before (Transformers / Convolutions, Batchnorm etc.) used it, and then shut the door behind them. Well done to them for making billions. But unfortunately, this means that the next revolutionary advancement like Transformers aren't going to be published openly.
@kamillimak6096 25 днів тому
Absolutely amazing. Recently I've become a student of data science, professionally I'm responsible for RAN network strategy for telecom operator. I am completely deeply amazed how your work helps me with both of this fields! Indescribable :)
@meerhusamuddin3679 26 днів тому
Thank you so much Grant! Even before starting the video I know it's going to be a masterpiece
@trickshot7457 25 днів тому
Those who completely understood attention after this video will be feeling like God/Goddess 👼👼
@NouchBR 26 днів тому
First here😮
@ffffffffh 25 днів тому
3Blue1Brown Is All You Need
@obikwelukyrian6848 25 днів тому
I honestly did not expect the next video this fast. Thanks a lot of this video. As usual, it didn't fail to deliver
@Nape420 26 днів тому
5 views after 1 minute, bro fell off 💀
@arberstudio 26 днів тому
44k 3hrs in
@chromacat248 26 днів тому
6 likes after 3 hours, bro fell off
@Nape420 26 днів тому
@@chromacat248fr
@Nape420 26 днів тому
@@arberstudio bro wouldn't get the joke even if he got it painted on the ceiling of his bedroom
@KshitizRai1005 22 дні тому
As a beginner in ML and deep learning , it was overwhelming but I did get a pretty descent understanding second time . I am very excited to learn about Deep learning and Transformers . Thanks 3Blue1Brown
@Commonsenseisrare 2 дні тому
Fantastic video. Explaining complex concepts with powerful visuals is amazing. 1000s of articles on transformers/embedding are no match to this one video. Enjoyed it.
7 днів тому
this was litterally posted at the perfect time for me and my uni group. currently working on developing a transformer for natural language processing and having trouble with understanding it.
@marcelob.5300 25 днів тому
Speechless. Thanks Grant. I owe you.
@sergeyturushev1986 26 днів тому
I was able to follow the thought without a single pause, even not being a native English speaker. Wery well done and presented!
@amazing-qq3wi 25 днів тому
I am on the second video of this series, but I clicked this and liked to boost this on the algorithm. I'll leave this in about 30 seconds. Keep up the great work!
@iyytan2000 25 днів тому
Thanks for all the good work. The other day I saw some GNN application on structure representation and started thinking if transformers can work in a similar way. Just started researching how a transformer works and two days later 3B1B starts uploading transformer videos.
@christianschafer3724 25 днів тому
I've read about key/query before, but now it starts to make sense. Thank you!
@mhrzsafaeian7976 25 днів тому
Like a delicious coffee, like a fine wine, I sip every minute of these videos deliberately. It should not end in only 26 minute!!
@takeaswig4506 25 днів тому
Another one this quick?!?! You're one of my favorite channels on this platform, thank you for the high quality and informative videos
3Blue1Brown
Переглядів 2,1 млн
Суспільне Вінниця
Переглядів 529 тис.