Making a neural network from scratch is easy, what I really want to see is how to make a neural network ON scratch.

i like how numpy has become so ingrained in python that it's basically considered vanilla python at this point

I watched this video when I was studying in grade 11. Had no clue what he was talking about but I tried to understand as much as possible. Now I watch it again as a university student, it is so satisfying to understand everything now.

If you make more deep learning videos with numpy and math(without any framework) just like in this video, it would be great for begginers to learn basics!!! Do you think to keep continue??

This video is one of the best descriptions of neural networks written in only Numpy and Python I've ever seen. Thanks

00:51 Problem statement 01:18 Math explanation 11:18 Coding it up 27:43 Result's

Took a Machine Learning course in university and this is what we did the whole semester in Matlab. Tensorflow was introduced right at the end for the final project.

I'm so glad you actually went in depth with the math explanation. So often people will just explain surface layer and then "alright lets jump into the code".

My man really explained how a back propagated neural network works from scratch in 10 minutes

Just your intro alone in your motivations was so capturing. You laid out everything so clearly, including creating those row and column matrices in the early steps. Thank you.

I've never heard any of this explained before. After watching this once, I understand the mathematics behind neural networks and why the functions are used. Great job with the explanation here. Many thanks.

This was interesting, it certainly made neural networks far more approachable to me as someone who's never needed to/been inclined to try making one, but encounters them frequently by being involved in STEM. Your explanations coupled with my familiarity with numpy as opposed to dedicated libraries for neural networks really helped - thanks!

Just discovered this channel. Very cool stuff. Much respect for doing something challenging like this.

This was a really good video. I’ve never build a neural network but it was interesting seeing how the fundamentals add up to build something a little more complexed.

Excellent tutorial and example. Reveals the magic that most don't know about NNs and I love how you go about it.

I need to come back to this after learning some more preliminaries but you are a very natural teacher and good at presenting. Keep it up 👍

Most of the videos are titled “how to create a blabla” when they’re actually teaching how to use… so I really appreciate your video! This really contributes to knowledge 🥰

This was really neat. The math explanation was frustrating the first time around but really made sense after working through the code. Thanks for sharing.

You sir, are my hero. You are the first person to actually explain this properly to me. Thank you so much for that.

Another thing that would be helpful for those of us that want to copy what you did and experiment with it is to have all the code together instead of separated as it is using Kaggle - this way you can put in some comments with the code explaining the different features. Again, very good video.

You should continue making video similar to this maybe something a training course for machine learning and reinforcement learning AI. You have a real talent for explaining it in the best way possible then from what most videos I’d watched. 👍

this type of learning is honestly the best, i implemented k means clustering by myself in c (pretty easy stuff but still) , and i can never forget it now, makes me happy that i can do stuff too

Just learned basics around the neural networks and saw this video. So satisfied to all the math formulas are laid out clearly in numpy and real-world coding and training neural network with back propagation. It really helps beginners like me. Thank you so much!

Super cool! Would also recommend the series from The Coding Train about creating a neural network from scratch, going a little more into the details of math and what is a perceptron and so.

What an awesome video! Thank you for sharing this insightful walkthrough, it was really helpful in getting a better understanding of how neural nets works. Thank you!

Man this video is a masterpiece. I learned a lot and I love your thorough, calm style. Please keep doing similar content!! Best wishes

Amazing. Needed to see the low end and finally found it. Thank you for the amazing video!

After Andrew Ng's course, this is the first time I'm watching math functions, thanks buddy, it was a nice refresher for me.

Samson, Keep doing this kind of videos please!! Very intelligent and understandable video

This is pure gold, MSc in Data Science and Artificial Intelligence, no professor ever gave me the answer to "what is the code inside the libraries we use", until I found you. Thank you

Really excellent breakdown of a Neural Network, especially the math explanation in the beginning. I also want to say how much I appreciate you leaving in your first attempt at coding it and the mistakes you made. Coding is hard, and spending an hour debugging your code just because of one little number is so real. Great video

It feels like it took me months to understand programming feedforward neural networks but I finally understand it. Thanks for the video.

Thank you so much Mr. Samson!! This was so informative and enlightening

Better lecture and example for understanding and building NN than any in my math and stats MSc

Thanks for lovely video Samson. I'm a prof and love seeing this kind of content. I'll definitely share with students

This is great. Built a backprop in C thirty years ago to solve the same problem. Just for a goof. It worked well before I finished debugging. These things are awesome and now I want to take another look. Thanks for posting this.

Thank you for your time and effort, Samson, this tutorial is a treasure.

What an impressive speed run! Just nitpicking: 15:45 `rand` is for a uniform dist U(0,1) and `randn` is for the standard normal distribution N(0,1), therefore unbounded, not U(-0.5, 0.5)

Samson, we need more videos like this from you. Great content, more visuals would be nice, too 🙂

This is the first ASMR NN video that I have ever seen. Well done.

Hi! I did a recreation of your code with more hidden layers and noticed what I think is a bug in the db calculation. Changing it to db = 1 / m * np.sum(dZ, axis=1).reshape(-1, 1) was able to get me better results. I think the old db = 1 / m * np.sum(dZ) sums the entire dZ to one float. Very good video though!

In case any beginners to ML came here wondering why they are really confused, this video isn't really for beginners and he doesn't really explain that. Its "from scratch" in the sense of not using any prebuilt models in the code. Its a good explanation for people who are already familiar with neural networks, prebuilt layers, loss functions, etc. not for people starting their understanding "from scratch."

Samson Zhang is the BEST Cinematographer, editor, musician& tech geek in the WORLD

It's worth noting that softmax IS actually very similar to sigmoid. But it essentially does a sigmoid over multiple classes.

I’m always too intimidated to try some of these things. But seeing your process makes it really seem feasible. Need to brush up on my linear algebra again tho 😆

This solved a lot of doubts and brought up mu confidence levels to deep dive into AI/ML. Thanks for the explanation.

Awesome fundamental class on neural networks equations. Bravo!

Musician, filmmaker, data scientist, and etc. bro maxed out on skill trees. 😂

Samson, this was such a great walk through. Just wanted to say that if you ever made other videos recreating machine learning models from scratch, I'd 100% watch them. In any case, hope all is good and thanks for this great content :)

Thank you. I'm doing this in class right now and your explanations were super helpful!

Brilliant. Kind of the Hello World of neural nets. It shed a lot of light for me on how back propagation works.

Bro, that is exactly how I study! I found out your channel and I am so glad I did. Instantly subscribed! I see you have learnt from Andrew Ng

Great video! I did the same thing in python about a year ago, but I didn’t like relying on numpy so much. Your video gave me the motivation to write both a matrix manipulator and neural network from scratch in Java

Most tutorials I watch online about ML, you can just tell that the instructor doens't know whats happening. They've just memorized libraries and tensorflow syntax, and I don't want that to be me! This is exactly what i've been looking for! THANK YOU!!!

Love your sense of humor! Brought the video to life, thanks! You are appreciated!

Could you please do more tutorials ? This is such a great video

Hi Samson! I'm a developer and trying to learn the basics of ML. Much of the beginner stuff I see is using pre-trained models and frameworks which might be convenient to get things going. However, for me this is something completely new and I really what to understand what happens behind the scenes. Thank you for posting this! /Kevin from Sweden

I loved this video! Cool stuff. I implemented a tfidf clustering algorithm myself, very satisfying to see it all working

Really cool video Samson! Great stuff!

There is one thing I do not understand. Because the derivation and chain rule stuff, shouldn't the derivative of the softmax activation function also be included somewhere?

I actually did this exact same thing for my German a level project. Same database. :D good times

Keep doing it man, I am from Perú and the information that your are giving is the important I have heared about

Great video! It's really solid in foundation! I will definitely recommend this to those just like to use framework and library without understanding

An excellent nice video with abundant mathematical insight. It may be worth to note that instead of partial derivatives one can work with derivatives as the linear transformations they really are, and also looking at the networks in a more structured manner thus making clear how the basic ideas of BPP apply to much more general cases. Several steps are involved. 1.- More general processing units. Any continuously differentiable function of inputs and weights will do; these inputs and weights can belong, beyond Euclidean spaces, to any Hilbert space. Derivatives are linear transformations and the derivative of a neural processing unit is the direct sum of its partial derivatives with respect to the inputs and with respect to the weights; this is a linear transformation expressed as the sum of its restrictions to a pair of complementary subspaces. 2.- More general layers (any number of units). Single unit layers can create a bottleneck that renders the whole network useless. Putting together several units in a unique layer is equivalent to taking their product (as functions, in the sense of set theory). The layers are functions of the of inputs and of the weights of the totality of the units. The derivative of a layer is then the product of the derivatives of the units; this is a product of linear transformations. 3.- Networks with any number of layers. A network is the composition (as functions, and in the set theoretical sense) of its layers. By the chain rule the derivative of the network is the composition of the derivatives of the layers; this is a composition of linear transformations. 4.- Quadratic error of a function. ... --- Since this comment is becoming too long I will stop here. The point is that a very general viewpoint clarifies many aspects of BPP. If you are interested in the full story and have some familiarity with Hilbert spaces please google for papers dealing with backpropagation in Hilbert spaces. A related article with matrix formulas for backpropagation on semilinear networks is also available. For a glimpse into a completely new deep learning algorithm which is orders of magnitude more efficient, controllable and faster than BPP search in this platform for a video about deep learning without backpropagation; in its description there are links to a demo software. The new algorithm is based on the following very general and powerful result (google it): Polyhedrons and perceptrons are functionally equivalent. For the elementary conceptual basis of NNs see the article Neural Network Formalism. Daniel Crespin

Haven’t finished video yet, but this looks like the missing piece of my experience learning about neural networks at a high level…I probably lacked the linear algebra skills I have now though. Whoa! This could be incredibly exciting! I can’t wait!

I've been looking for this video for 6 years.

The yt algorithm only recommends me this now, 1 year after i've encountered a similar discontent with neural network tutorials. Still very interresting to see how someone else does it. I did give myself a bit of help by using a library called Eigen for the matrixes calculations. Very well done nice video

Hi, i found this video very helpful for beginners. Could you please tell how you came up the equations of dz,dw and db? That would be really helpful as well

This is a great way to teach ANN - congrats. However, I would like to suggest you to not worry too much about the time to finish the implementation. Double-checking all steps will avoid coding errors.

Nicely done, Samson, thanks!

Timestaps if you forgot 0:51 Problem Statement 1:18 Math Explanation 11:18 Coding It up 27:43 Results

It's a shame it isn't taught this way in courses. Excellent video!

Amazing video for beginners to gain an insight in how neural networks work. You just have to have programmed a simple neural net from scratch once to have a good basic understanding.

I know the Maths and Programming behind it and listening this guy doing all that on his own is pure respect from my side.

Helpful, thanks. Made my own from scratch in bare C++. From image to 32 to 16 to 10 outputs, using leaky ReLU. 96% accuracy on the test set. 🥳

It's a MLP, you easily computed the backpropagation step in closed form, but I wonder how those famous frameworks can compute any network's partial-derivatives tensors automatically

i have no idea what your were really saying but at the same time i do because you explained how the math is used and implemented for the code. thank you !

A great introduction to neural networks is Parallel Distributed Programming by Rumelhart and McLelland from about 1986. They do something similar and give a lot of additional background.

Very good video and explanation! Thanks 😊. I just would have liked it if you had explained the backprop a little more in depth. Like how the derivatives are calculated on each layer (chain rule etc.) But other than that one of the best nn videos

Perhaps I overcomplicated matters compared to your approach when I did this a couple of years ago, but like you, I wanted to program it "from scratch". My language of choice: java. I actually simulated "neurons" which were a class that stored its activation data value, and its connections to the next layer, so that it "looked" like a K_m,n graph, and the connection was an array which stored the biases along each "synapse" so to speak. Then when the hidden layers activated, I had each neuron simply sum the outputs from each synapse connecting to it from the previous layer, which was just the product of its activation value and its bias, then sigmoided this to get its own activation value. Note that while each neuron's activation was only in (-1,1), I let the biases be free parameters. When I programmed the backprop algo, I did the gradient descent the same as you, but effectively set that alpha parameter to one. It didn't occur to me to mess with that. Starting the network out with random parameters, then training it on randomly chosen sets of 10,000 images five or six times seemed to work pretty well. I saw 93% accuracy on the test data. And just for fun, I put the network on a discord bot so my friends could feed it images of the same size and see its guess. Two interesting results came out. The network fails on inverted colors: i.e., drawing white on black using MS paint or something wouldn't get reliable predictions. Secondly, using MS paint to give it new data did work, but at a much lower rate. Our best guess for why this happened was due to the sharpness of the lines between the number and backgrounds.

Very impressive! Great commentary/explanation as well

thank you for the knowledge Mr. Samsung

I agree with you. I also did this by scratch. It was a lot of fun! What’s the point of masters math degree if I am not going to use it lol. Nice work!

Amazing stuff! Just wondering what value does the coding timer add to the video? I mean instead of correcting your mistakes with overlapping text you could have taken a little bit of time to review your code instead of rushing it through. But again, amazing content!

Great explanation Sam!

That is very neat and captures the fundamental ideas of neural nets! great job

Hello, it's such a great tutorial. thank you very much. I think people who are over exited because of this AI-hyped should learn this basic, and see whether those people really fit in to this field 🤣🤣

Hey, the video was very informative for a beginner to machine learning. I did have a question about the code, I noticed that when I implemented the code as you did in the video I was getting an error: RuntimeWarning: invalid value encountered in true_divide in the softmax function. I looked at your kaggle notebook, and I see that in the notebook you do x_train = x_train / 255 and when I did this it fixed the error. I'm not sure what's going on here, does anyone know why this is happening and why dividing by 255 fixed the issue?

Samson, thank you for this video, it is very helpful! Just like you, I also benefit much more from detailed, equation-based explanations than from the high level, big picture overviews.

Really appreciate the example & tutorial, thank you. Can you make another video? Perhaps on adding more hidden layers?

You can actually use momentum for gradient descent. The result is slightly better (I tried on your nn and it gets 91% accuracy) // I'm a beginner at ML so your video taught me a lot. Keep up your great work you're doing man. It's really cool.

Just 1 minute in the video and I can easily tell that you're gonna own a multi-billion company within a few years. You've got the IQ, the voice, the clarity, the confidence, and the right personality. Best of luck Mr. Zhang

This was really useful to me, and incredibly well explained. Thank you.

the fact he actually shows the first overconfident of its memory programer stage is actually so real.

Everyone praises this video for being so helpful and I'm just sitting here understanding NOTHING. :D I feel so dumb! Maybe I should've stared with something even more basic having learned in a nutshell only print("hello world") so far. I will definitely go back and watch it all again in the future after I learn more. Thank you for the video, Samson. Cheers!

Now build one IN Scratch

I am going to do the same over the next two weeks , at the end I'm coming back to see any differences between our code, thanks for sharing :)

I heard your name as Samsung- got instantly hooked.

Hey, I found a flaw in your code and would be great if you answer it......The updation that you are doing for the bias' is not all needed as per your code because all the bias are changed by same factor hence it's still random( you have used a scalar to update the bias instead of a column vector)......I found the correct solution to it but getting an error. you should add the axis=1 in the sum function.

Thank you for sharing . You did a awesome explanation! A quick question, I got brunch of NaN in A2 after Softmax. Besides, can you please re-share the code link? The current link connects to 404 page 😂