In every “AI learns to walk” video I’ve seen, the AI either learns to walk in a weird, non-human way, or they use motion capture of a real person walking and simply train the AI to imitate that. I thought it was weird that nobody tried to train it to walk properly from scratch (without any external data), so I wanted to give it a shot! That’s what I said 4 months ago. It’s been really difficult, but I’ve finally managed to do it, so please watch the whole video! The final result ended up being awesome :) If you're interested in training your own AI like Albert but don't know how, there's now a really easy way to do it! Luda, an AI lab, recently built a web app that allows you to create and train your own AI using deep reinforcement learning (just like Albert) completely for free in your browser! You build your own character (called a Mel) with lego-like building blocks then watch it train in real-time on their website in just a few minutes (really). It's an awesome project, and just like my videos, makes deep reinforcement learning so much more accessible, which is why I love it so much. This section of the comment is sponsored by Luda, but these words are entirely my own, it's an amazing project that I would have been obsessed with had they released it before I built Albert. I've genuinely been looking for a sandbox/game exactly like this since I was a kid. They're still early, but they're giving my audience first access to their closed, pre-alpha build. Make sure you check out their site and create an AI agent for yourself!:D prealpha.mels.ai Now, back to Albert: NOTE: You can only see one Albert, but there are actually 200 copies of Albert and the room he’s in training behind the camera to speed up the training. If you want to learn more about how Albert actually works, you can read the rest of this very long comment I wrote explaining exactly how I trained him! (and please let the video play in the background while reading so UKposts will show Albert to more people) THE BASICS I created everything using Unity and ML-Agents. Albert is controlled entirely by an artificial brain (neural network) which has 5 layers, the first layer consists of the inputs (the information Albert is given before taking action, like his limb positions and velocities), the last layer tells him what actions to take and the middle 3 layers, called hidden layers, are where the calculations are performed to convert the inputs into actions. His brain was trained using the standard algorithm in reinforcement learning; proximal policy optimization (PPO). For each of Albert’s limbs I’ve given him (as an input) the position, velocity, angular velocity, contacts (if it’s touching the ground, wall or obstacle) and the strength applied to it. I’ve also given him the distance from each foot to the ground, direction of the closest target, the direction his body’s moving, his body’s velocity, the distance from his chest to his feet and the amount of time one foot has been in front of the other. As for his actions, we allow Albert to control each body part’s rotation and strength (with some limitations so his arm can’t bend backwards, for example). Just like the last videos, Albert was trained using reinforcement learning. For each of Albert's attempts, we calculate a score for how 'good' it was and make small, calculated adjustments to his brain to try to encourage the behaviors that led to a higher score and avoid those that led to a lower score. You can think of increasing Albert’s score as rewarding him and decreasing his score as punishing him, or you can think about it like natural selection where the best performing Alberts are most likely to reproduce. For this video there are 13 different types of rewards (ways to calculate Albert's score), we start off with only a couple and with each new room add more, always in an attempt to get him to walk. REWARD FUNCTION Room 1: We start off very simple in the first room, we reward him based on how much he moved to the target and we punish him for moving in the wrong direction. This led to Albert doing the worm towards the target, since he figured out that was the easiest way for him to move the quickest/get the highest score. Room 2: In the second room we start checking if his limbs hit the ground. If the limb that hits the ground is a foot we reward him (but only if it's in front of his other foot, more on that later), if it isn’t, we punish him. I also made it so Albert wasn’t rewarded at all unless his chest was high enough to force it to at least be partially standing. As seen in the video, this encourages him to not fall over and encourages him to use his feet to do it. We also introduced a new reward designed to encourage smoother movement; if he approaches the maximum strength allowed on a limb he's punished, and he's rewarded if he uses a strength of almost 0. This encourages him to opt for the more human-like movement of using a bit of strength from many limbs as opposed to a lot of strength from one limb. Room 3: This is where we start to polish Albert’s gait that developed in room 2 and teach him to turn. From here on we start using the chest height calculation as another direct reward where the higher his chest is the more he’s rewarded in an attempt to get him to stand up as straight as possible. These rewards so far give Albert a decent gait, however he’s still not using both of his feet (which was by far the hardest part of this project), so room 4 is designed to do exactly that. Room 4: We get Albert to take more steps from a few additional rewards. To start, we introduce a 2 second timer that resets when one foot goes in front of the other. We reward Albert whenever this timer is above 0 (the front foot has been in front for < 2 seconds), and we punish him whenever the timer goes below 0 (the front foot has been in front > 2 seconds). We add another reward proportional to the distance of his steps to encourage him to take larger steps. To smooth out the movement, we also add a punishment every frame proportional to the difference in his body’s velocity from the previous frame to the current frame, so if he’s moving at a perfectly consistent velocity he isn’t punished at all, and if he makes very quick erratic movements he’s punished a lot. If you're still reading this, you're probably really smart and want to learn more about Albert, so make sure to join my discord server I just made where we can talk more about the details of Albert's AI! discord.gg/jM2WkNuBnG :) Room 5: For the final room the only change I made to the reward function was to go back to an earlier version of a reward. Throughout the other rooms I had been tinkering with how I should reward Albert’s feet being grounded, my initial thought was to only reward the front foot for being grounded to try to get him to put more weight on his front foot when taking steps, but somewhere along the way I changed it to just rewarding Albert for any foot being grounded, and that was the version Albert trained with in rooms 3 and 4. For this final room I switched back to the old front foot grounded reward which resulted in a much nicer looking walk. Also, the video makes it seem like I never reset Albert’s brain, that isn't entirely true, I had to occasionally reset it because of something called decaying plasticity. OTHER For rooms 1 to 4 I only allowed Albert to make a decision every 5 game ticks, but for the final room I removed that constraint and let him make decisions every frame. I found if Albert makes a decision every game tick it’s too difficult for him to commit to any proper movements, he ends up just making very small movements like slightly pushing his front foot forward when he should be taking a full step. The 5 game tick decision time forces him to commit to his decision for at least 5 game ticks so he ends up being more careful when moving a limb. When I recorded him beating the final room I removed this limitation because he’s already learned to commit to his actions so allowing him to make a decision every tick just results in a smoother motion. We're looking to hire people to help make these videos! If you're a talented Unity game developer you can apply for a full time position here forms.gle/ko54z1LQmZNUT9Vp8 And if you're a talented AI developer (ML-Agents), you can apply here! forms.gle/Uou1Vwb5Q9VccaAY7 We're looking for full time employees, but part time works too, what we're really looking for are skilled and passionate people, so feel free to apply if you're interested! :D Thank you so much for watching, this video took me 4 months to make, so please, if you enjoyed it or learned something from it, share it with someone you think will also enjoy it! :)

The fact that albert learned how to do the worm before walking means this man is ahead of us

Imagine your toddler learning how to walk and you just start petling him with boxes. 😂

It's so cool once you realize that the way Albert learns to walk is the same way a lot of babies do. First they wiggle to move, then they learn to shuffle and slide, then they learn to crawl, then they start taking steps, albeit odd steps, and then they finally learn to walk and turn. It's pretty much the same way a human baby would learn. Super cool. :)

I love how 6:10 is the most replayed part of the video. It really looks exactly like when a human trips, doing those stupid desperate steps but ultimately you're past the point of saving yourself.

Up next: Albert learns to get his degree and start a family

Reminds me of how I learned to walk. The lab. The green squares. The flying red cubes. Ah the memories ❤

I like how the ultimate reward for all of his efforts is just endless falling into the void

Wow this is the most clever way to train an AI to walk that I have ever seen. You masterfully set up video game-like challenges to force it to learn specific behaviors that would overall give a better result. Plus, the editing is fantastic and you have a quirky sense of humor! Awesome work! So fun and fascinating to watch :D I look forward to continuations of Albert's learning journey!

I like how whenever Albert makes it to a new stage or checkpoint, he’s just like: “ok that’s enough for today. Let’s do a face plant.”

The little tantrums he throws after falling gets me every time

The fact Albert worked on learning to take a hit and stay upright, before continue to learn how to walk correctly, is a major genius move, on the programming and Alberts part as well. He was able to identify the importance of tasks, and prioritize their value to his job.

it's kind of insane how similar this is to a real person learning how to walk, from crawling to shuffling to skipping to slowly using your feet properly, even at 6:08 when he fell I saw myself falling like that 100 times and running until I either catch my balance or becoem one with the ground

Let's not forget how difficult it must have been for Albert to learn to walk with such a massive head.

nicely done! bipedal walking is super hard and the amount of curriculum learning and reward tuning here is super impressive!

imagine the person that thought he was going to be a human, but he was a deep, reinforcement learning AI

I like how he goes from skipping to an old grandpa walk, aging from 10 to 70 in the training process

Albert is growing. He will take over the world.

Lets all pray Albert is gonna laugh about this when he's grown up. So sweet to throw cubes at him, playing cheerfull slapstick music.

I love it when he tries to balance for the first times LOL it’s so cute 😭❤️

Great job! You’ve inspired me to start throwing big red cubes at my toddler as soon as he starts walking!

This is far more entertaining than watching an actual child learn to walk.

this is so cool but I can't stop laughing because of how cute and funny it is to see an AI learn to walk. I never know that this video exist not until yt recommended it. the way Albert wiggles it's entire body is just so funny and cute to me. this is awesome and entertaining. I'm subscribing now

The feat of balance at the end of the final room whilst being bombarded with boxes was something to behold , well played Albert well played .

I don't think I'll ever get tired of witnessing Albert learning how to live.

I like how Albert not only mastered the worm and skipping before walking, but that during the rough terrain test he corrects missing the button by just falling backwards on it, thus literally using his head to solve yet another problem.

Weak animal kingdom: takes millions of years to learn to walk Chad Albert: only learns it in a couple of months

I love the detailed explanation. Thank you for taking the time to write it.

Love it! But I'm super interested in everything that goes into making it... a tutorial series on how to build these simulation and training environments in Unity (or, even more preferred, Unreal) would be AMAZING!

I love that every time he falls over it looks like he's throwing a little tantrum. Absolutley adorable

I've had to relearn how to walk a few times due to having ankle surgery, and I must admit that Albert's progression from shuffle to walk is uncannily similar to my own progression.

One must imagine Sisyphus learning to walk

This is incredible! Great work!!

2:41 I love how it looks like Albert is throwing a tantrum after realizing he can't go through walls

I liked that confident walk Albert had in the last challenge, he sees the goal, and went for it, and nothing has stopped him.

Oh man! This is great stuff :) This "learning" is awesome to see

Wow! Video and visualization of your RL project were very nice and fun for me, knowing how difficult it can be!!. Well done 😀

The way he learns to first walk by skipping is really reminiscent of the way astronauts walk on the moon. The slightly reduced gravity makes skipping way easier and more efficient to get around, especially in those restrictive suits. Albert is so top-heavy and his limbs are shortened, so that was a great first step for him.

I love that it looks like sometimes he just throws himself on the floor and has a little tantrum.

Wow. This was fascinating! Thanks 😊

That's awesome, and that's hilarious! 😂 Well done!

I like to think of Albert jumping off the edge is him becoming self aware and realizing that life is nothing but sacrifice and failed attempts

Albert kicking his feet in frustration anytime he falls is a MOOD

I'm so proud of you Albert. Good job

This video explains very well on reinforcement learning. ❤

It’s like watching a toddler learn to walk. Incredible.

found myself rooting for him. rooted for him HARD in the last 17 seconds. “Stay upright! Reach the finish line!”

I can't wait until this kind of research is implemented in gaming. Thank you sir

Hey dude your videos are great keep up the hard work I know how difficult youtube can be when you are only just begining as i have started a channle recently and to see that you pushed through that faze at the begining is very impressive I know you probably won't see this comment but if you do see it i want to let you know i hope you keep going with these amazing videos and that albert isn't too anoying while you make and edit these videos have a great day

6:10 that stumble is impressive. Very humanlike movement, trying to correct the path, and failing, but moving very convincingly.

This is the stuff our robot overlords will play at the human trials.

Thank you for this video, ive laugh so much and this was super interesting

I feel like I've raised a child simply by watching Albert learn to walk and rooting for him

I love how he turns into a fussy baby when he falls over ❤

AI is just truly fascinating to see.

This is amazing. I can't imagine how complex this was to program.

I love how Alberts walking gives a baby first steps vibe

I think it's kind of adorable how, whenever he falls over, he starts waving all of his limbs as if he's throwing a tantrum out of frustration!

WE NEED MORE ALBERT

one must imagine Albert happy

You can almost see Albert get more pensive as he approaches the last challenge in the final section. He walks with unabashed confidence through the first two sections and slows down like he was anticipating getting pelted with a box before every step

Albert is a man of focus, commitment and sheer will

It looks like the baby is learning to walk. It's so cute.😊

Imagine if we left Albert alone and he somehow created a new form of movement that was more efficient than walking and maybe even sprinting.

LET’S GOOOOOOOO WE CAN SEE ALBERT LEARNING AGAIN (genuinely hyped to watch that video)

AI will definitely hate us

Imagine incorporating something like this into The Sims, so you can actually watch as your baby slowly learns to walk and run, and because of the reinforcement your sims develop unique walking styles based on experience and personality.

This is truly a MASTERPIECE!

7:58 The way he kicks the box and continues walking while ignoring the boxes is so great.

AM SO PROUD OF YOU ALBERT

Albert learning to skip is actually super interesting because as kids, when our bodies are a bit wonked out skipping is the most effective method of movement. And with his shape kind of being like a cheddar baby it makes total sense why skipping would be a primary method

7:37 albert looks like he's trying to sneak around smt like he was a thief

This was so interesting and funny!! Love it

Fifty years from now when Albert became a conscious humanoid, he will tell his son all of this misery just like our grandpas telling us how they used to go to the school

If you have raised a kid, it's amazing to see the similarities (and differences) between how they each learn to walk :)

i have felt every emotion while watching albert learn how to walk

Why is this wholesome I love albert

Excellent character development. Thought Albert was going to fall at 8:03 but he didn't, he went on. Albert worked hard to succeed, we should all be like Albert.

Albert's walk at the end just exudes confidence

I never thought the day would come that I would see an artificial intelligence find out walls exist and have a temper tantrum

I love how his first instinct when being told to reach the target through any means possible using his limbs... Is to do the worm.

One thing I would love to see is Albert running the entire course again from the beginning with his new skills just to see how much he improved.

Missed you, Albert 😂 Welcome back to eternal pain ❤

I was so invested in Albert's progress... madness

Bro gets the silent treatment each time he does something bad

The fact that Albert almost managed to walk like a normal person is shocking, and quite impressive to me. I have watched tons of other "AI learns to walk" videos on UKposts, but this one is by far the most accurate to walking like a normal person (rather than weird crawling or worming). If somehow you taught Albert to walk in a more rhythmic sense, (just like our hands and legs move together) then that would be just amazing. This in my opinion, is one of the greatest achievements in the field of AI/machine learning. EDIT: I appreciate the amount of likes this is getting... and also, this may not be one of THE greatest achievements for AI/ML, but it is still a very remarkable feat. AI has a LOT of potential in the future. This is just a glimpse of it.

Albert's failed attempts sped up is so ridiculously funny

Good job Albert, proud of you little buddy

E pensar que evoluímos dessa forma, tentativa e erro, até acertar. Essa é a real magia do universo, possibilidades, probabilidades e realidade.❤

We all need determination like Albert in our life.

This makes me think of a factoid I read a while back about how children skip because its more efficient for them. Interesting to see that Albert's inclined to skip as he learns to walk, too

It must have been so satisfying when you got him to actually walk and turn and everything.

Thanks for making Ai learning video without long boring explanation of how you trained AI. I just wanted to see these lil' fellas learning stuff through time, not dudes talking about code and shit. +Sub

It's like witnessing a baby growing up, this is just insane

6:09 This moment happens humans too and funniest shit ever lol.

Why is an AI learning how to walk the most entertaining thing I have seen all day?????????

This is the kind of training that leads an A. I. to being able to enter the real world inside of a robot body, problem solve on its own, and complete tasks that they were trained to do inside of the computer simulation irl. This leads to robots that can do some pretty interesting things.

As someone who doesn't know anything about coding or Ai, Albert means everything to me. I wish them luck on any future endeavors

No matter how satisfying a “satisfying” video is, it can’t be better than watching Albert take his first steps

This is a perfect example of how nature molds us and forces the human body to evolve every time to adapt and overcome obstacles and survive.