Was coming to comment this, thanks

Please make video

Please!

Would you share the source code for the animations?

amen. the power of out of the box teachers is infinite.

"they were using a convolutional neural network in the transformer" No no, Transformers do not have any convolutional layers, the author of the video just chose CNN as a starting point in the process "Let's start with the solution that doesn't work well, understand why it doesn't work well and try to improve it, changing the solution completely along the way". The main architecture in natural language processing before transformers was RNN, recurrent neural network. Then in 2014 researchers improved it with attention mechanism. However, RNNs do not scale well, because they are inherently sequential, and scale is very important for accuracy. So, researchers tried to get rid of RNNs and succeded in 2017. CNNs were also tried, but, to my not-very-deep knowledge, were less succesful. Interesting that the author of the video chose CNN as a starting point.

@@Hexanitrobenzene, I suppose I’ll have to watch this video again. I’ll look for what you mentioned.

@@terjeoseberg990 A little off topic, but... Not long ago I noticed that UKposts deletes comments with links. Ok, automatic spam protection. (Still, the thing that it does this silently frustrates a lot...) But, does it also delete comments where links are separated into words with "dot" between them ? I tried to give you a resource I learned this from, but my comment got dropped two times...

...Silly me, I figured I could just give you the title you can search for: "Dive into deep learning". It's an open textbook with code included.

@@Hexanitrobenzene, The best thing to do when UKposts deletes comments is to provide a title or something so I can find it. A lot of words are banned too.

Thanks for the tip.

My one-sentence summary of why transformers matter would be "they are standard CNNs, except the words are re-ordered in a way that makes the CNN's job easier first before being fed ". Also, a single NN layer IS a linear layer; I'm not sure what you mean by saying you don't know how important the distinction between the two is.

thanks@@maxkho00

Thanks so much!

Thanks for the tip! I'm always happy to answer questions.

the first so far this year

Thanks, any title in particular you'd recommend?

Yes that's correct, the transformer just adds x without passing it through an additional linear layer. Including the additional linear layer doesn't actually change the model at all, because when the result of self attention is run through the MLP in the next layer, the first thing the MLP does is apply a linear transform to the input. Composition of 2 linear transforms is a linear transform, so we may as well save computation and just let the MLP's linear transform handle it.

Yeah it's a term I made up so you won't find it in any sources, sorry about that. Usually sources will just talk about self attention in terms of key, query and value lookups, so you can look at those to get a more detailed understanding of the transformer. The value transform is equivalent to the linear representation function I use in the pairwise convolution, the key and query attention scores are equivalent to the bi-linear form scoring function I use (with the bi-linear form weight matrix given by Q^TK). I chose to use this unusual terminology because, personally, I feel the key, query and value terminology comes out of nowhere, and I wanted to connect the transformer more directly to its predecessor (the CNN).

@algorithmicsimplicity, this is a surprising connection. Thanks a lot for the explanation.

@@algorithmicsimplicityit would be great if you could make this connection between terminology in video form. maybe next time?

I"ll be doing this, but in short, the little networks he showed connected to each pair are KQ (word pair representation) and the V is the value network., all of this can be done in the decoder only model as well. and cross attention is the same thing but you are using two separate sequences looking at each other (such as two sentences in a translation network). it's nice to know that GPT for example is decorder only, and so doesn't even need this

this is a good question and it's also a GOFAI type approach where we make the mistake thinking we can inject some human semantic idea to improve a network. but the reality is it will do this automatically without our help. For example papers back in 1986 show tiny networks automatically grouping words into nouns or verbs, it's amazing. let me know if you want more details

Thank you so much for the kind words.

Yep all my videos so far have been done in Manim.

The importance-scoring neural network is trained in exactly the same way that the representation neural network is. Roughly speaking, for every weight in the importance-scoring neural network you increase the value of that weight slightly and then re-evaluate the entire transformer on a training example. If the new output is closer to the training label, then that was a good change so the weight stays at its new value. If the new output is further away, then you reverse the change to that weight. Repeat this over and over again on billions of training examples and the importance-scoring neural network weights will end up set to values so that that the produced scores are useful.

The pair vectors are first concatenated together into one vector e.g. [00010100], and this vector is then run through the neural network which produces the output vector. The output is the result of the weights in the neural network. Initially, those weights are completely random (usually sampled from a normal distribution centred at 0), and then they are updated during training. The neural network is trained on a labelled training dataset of input and output pairs. For example, ChatGPT was trained to do next word prediction on billions of passages of text scraped from the internet. In this case, each training example is a random part of a text passage (e.g. "the cat sat on the") and the output is the next word that occurs in the text (e.g. "mat"). For every training example an update step is performed on the neural network to update all of the weights of all of the layers. The update step works as follows: 1) Evaluate the neural network on the input. 2) For every weight in every layer, increase the value of that weight by a small amount (e.g. 0.001) and then re-evaluate the entire neural network on the input. If the new output is closer to the target (e.g. the vector output is closer to the one-hot encoding of "mat") then it was good to change that weights value, so it keeps the new value. If the new output is further away from the target, then it was a bad change, so reverse it. And that's it. Just keep repeating that update step for billions of different inputs and all of the weights in all layers will eventually be set to values which allow the transformer as a whole to map inputs to outputs correctly. Also I should point out that in practice there is a faster way to do the update step which is called backprop. Backprop computes exactly the same result as the update process I described, it is just faster computationally (you only need to evaluate the model twice instead of once for every weight), but it is also more difficult to understand.

The entire model is trained end-to-end to solve the training task. What this means is you have some training dataset consisting of a bunch of input/label pairs. For each input, you run the model on that input, then you change the parameters in the model a bit, evaluate it again and check if the new output is closer to the training label, if it is you keep the changes. You do this process for every parameter in all layers and in all value and score networks, at the same time. By doing this process, the importance score generating networks will change over time so that they produce scores which cause the model's outputs to be closer to the training dataset labels. For standard training tasks, such as predicting the next word in a piece of text, it turns out that the best way for the score generating networks to influence the model's output is by generating 'correct' scores which roughly correspond to how related 2 words are, so this is what they end up learning to do.

The neural network takes as input a single pair (concatenation of both vector in the pair) and produces a single scalar output score for that pair. This same network is applied to every pair to produce all n^2 scores. The scores are then normalized across columns (exponentiate each score, then divide by sum of column). In practice, transformers don't use a neural network to produce scores but a simple bi-linear function, which is typically referred to as the "key-query dot product self attention".

Yep, everything is completely symmetric for rows/columns, since in either case you are still summing over all pairs that contain a particular word.

The "key-query" attention scoring is equivalent to the bi-linear scoring function in my explanation, where the bi-linear form matrix is given by K^TQ. The value transformation V is exactly the linear representation function in my explanation. I still have no idea why they decided to give the scoring function matrix two different names (key and query), it just confuses everyone.

Let's assume X is our input, a sentence containing N words. Each word has embedding dimension of size P. Thus X is an NxP matrix. Then according to the "Attention is All You Need" paper, we have: K = X * W_k Q = X * W_q V = X * W_v A = softmax(QK^T) Output = AV Where: W_k, W_q and W_v are PxP matrices K, Q and V are NxP matrices. A is an NxN matrix Output is an NxP matrix. I am confused about how the V matrix connects to the "pair-wise representations". In the video, you show operations being done on pairs of words (such as 13:40). However, there doesn't seem to be any pair-wise operations occurring when computing V? If there was a pair-wise operation, wouldn't the dimension of W_v need to be NxN instead of PxP? I agree that we are computing an a single "attention" scalar value for each word pair. This is why A has dimension NxN. However, it seems like V contains individual representation of the words that are "smooshed" together when we multiply by A, rather than V containing (or operating on) pair-wise representations? Again, great video! And I greatly appreciate your help!!@@algorithmicsimplicity

@@dmlqdk When you apply the linear transform V to the pair [x1, x2] the result is V1x1 + V2x2. Basically we are applying a linear transform to each input and summing them. Because, in a given column, x2 is the same for every pair, you are effectively just adding a constant value to each V1x_i. You can factor this constant value outside of the attention weights, at which point it just becomes part of the residual term. I explained this process in more detail here: www.reddit.com/r/MachineLearning/comments/17cmzcz/comment/k5t7g70/?context=3 At this point, you no longer have 'pair' representations, since each value vector is now just a linear transform applied to one word. Each column of the [NxN] grid of value vectors contains V1x_i for i in {1,...n}, i.e. all of the columns are identical. Since all of the columns are identical, instead of elementwise multiplying the matrix of attention values by the matrix of value vectors and then summing, you can instead rewrite this operation as a single matrix-vector product, which is what the AV operation is in the standard self attention. V is that column of value vectors, where each entry is just V1x_i.

This makes so much more sense now. Thank you!!@@algorithmicsimplicity

i'm working on a video now which is attempting to do this, been on it for months. one key thing here I notice where you get fuzzy, is thinking CNN's 'only do classification', and that this is different than nex word prediction. Because next word prediction is a type of classification (the output class is the next letter or word). and so you could have just a plain old fully connected network do "next word prediction" based on training it that way. please let me know what else you are thinking as it might help me with my script in my video I will open with RNN's applied to next work prediction (starts in 1986) then I will explain where they break and why we need parallel approaches why simple brute force doesn't work (too many parameters, and hard to train) why transformers helped (compressed many layers into fewer adaptive layers)

@@ArtOfTheProblem uff, I'm not sure my comment made any sense at all. I'll reply more on your latest micro-vid, which considers an Attention block as a dynamic-routing layer.

Yeah RWKV is similar to state-space models and long-convolutional models, the key thing that makes these methods work is regularizing the convolutional filter to have larger weights for nearby elements and smaller weights for far away elements (for RWKV this is achieved by multiplying vectors by an exponentially decaying value). I am hoping to make a video about this class of models eventually.

@@algorithmicsimplicity Please do! i've been looking forward to such a video about RWKV for a long time

Yep! so you don't even need to include it at all if you already have residual connections.

My videos were made using the Python package Manim (www.manim.community/)