Seems like adding a facing reward would help stabilize the rotation.

you should also have a negative reward for high angular velocities, that way it has a reason to be more still

Theory on why it flies so slow: Its original training was based on hovering around one point, thus when it gets a new destination, it still assumes that it should arrive there without momentum to better stay at that spot. Then it got a little training with randomly moving spots, having momentum there is bad too, since its actually way more probable that you need to turn around than that you need to continue going. This, along with little time based punishment, results in a slower ar

give it access to the next 2 points so it can find a vector between them, also give it incentive to be faster

To combat the agent being slow and rotating you could add 2 other negative point rewards, every full rotation can deduct points which would likely reduce the spinning to a minimum and then also give it say 30 seconds to complete a course but deduct points for each second spent too, the agent might learn that the quicker it goes the less points he get deducted. I think revisiting this with these 2 additional criteria would be pretty interesting

Can we just admire how a few years ago AI struggled to play a 2D game and now this. It's really remarkable.

This is my first time viewing your work, and I'm struck both by how incredibly cool this is and your f'ing hilarious sense of humor. I'm always the last to know, I guess. Really fantastic work, fam.

The whole comment section giving Gonkee suggestions knowing full well he can't be arsed to do a follow up video lmao great vid, thanks for uploading

Another banger. Always love the way you use memes to make it funny!

I think part of the reason it has such a hard time is because it doesn't quite have the detailed control vectors that Iron Man does. If you watch the hovering and flight scenes in the first movie, you'll see he has little compressed air nozzles, jet redirectors, and control surfaces on the boots to help stabilize. He also obviously has flaps on his back, and in later iterations of the armor he has backpack-style thrusters so his COG can be below the thrust point. If the game simulates air drag then add the flaps and stuff, too, but the minimum I think you need to add are the micro thrusters, back jets, and elbow/knee joints.

You should’ve added more or less points depending on how much time they to get to the target, that’s what would fix the flight.

Really enjoyed your explanation and video format.

LOL every time I watch your videos I laugh at the editing. Excellent.

Great job, a lot of room to continue developing your algorithm, but love the initiative and results are fun to watch.

I'd love to see you tackle AI in a preexisting game. I dunno, throw half life at it and see what sticks.

Does it have the ability to throttle the jets? If not that could be the reason why it spins so much. It's the only way to stay at a constant height with constantly high uplift. (Also it stabilizes it really nicely)

I like these explanations bro. This is really decent content, thanks for putting in the effort with your videos

fr this is video is one of the best YT vids all time

Your reward function could be modified to get what you want. Add in score for time, add in penalty for excessive rotations/spinning

Finally! Really like your videos

Another thing that you could add to this would be random perturbations like throwing blocks at the agents so that they learn to recover from instability like the drone had at the end. Would you be willing to release the source files for the project and then do a compilation of different people's attempts at improving the result? I think the learning the actual Iron man style of flying might be possible but if you don't want to do all the work on that it could be fun to see what the community comes up with.

Bro you just validated a theory I've had for a long time. I'm not sure if I'm saying this right so please bare with me. All kinetic type movement is always multi layered. There is probabilistic correct-ness at every axis. Therefor it is necessary for every joint to learn to work together. You need a series of cooperating routines that all independently learn and get rewarded by a higher system. For drones it would look like a computer flying with a full flight controller managing the power at every motor with an operator that verbalizes instructions. Love your work here ( subscribed! )

Amazing, this is the kind of models I wanna make. Great video

I truly like you. Subscribed.

MLAPI is a blast, love it. This inspired me to (hopefully) do my next AI experiment soon.

After getting so many good suggestions on improving the ai, you have to make a part two now. And make it more of a challenge.

good having you back

glad your brain cells and hairs grew back :)

Love the sense of humor!

I had to go back and watch this three times. Hilarious!

Amazing work! I suggest adding another training step - fastest route. Eventually, the model will fly as intended. Good luck!

I would make the reward relative to the forward direction to each node to promote a flying posture and stop the spinning. If you added the next node as input as well it might be a bit better at handling its own momentum out of each node.

happy 100k!!!

100k!!! i rlly wish you get 100k subs very soon

your editing skills are getting better

Big W dude. Love the Ballerina Ironman result.

Man ur so fkn funny continue like this first time I saw u and not the last

Nice video. Are you rewarding the AI based on how fast it can get to the target? Because it you're rewarding it for staying in the air, and then a fixed reward when it hits the target, it learns to take longer. I'm sure you already know this, but this is a subject I'm rather interested in ¯\_(ツ)_/¯

Dude for real, you should have a premium version of you channel with the walkthrough this is the kind of content that some people like me can only dream of

Saw that you had some parameters related to velocity which I think depends on the direction. Haven't done much machine learning or 3D animation programming myself but I think you need to train it on 2 random points and optimize for speed instead of velocity and time taken to reach the destination.

Great video :D

Try giving the Agent a huge punishment for spinning around, that may help. Keep it up! :)

All hail the algorithm 😂 great video, subscribed!

thanks for making a long video :)

I love how there are so many comments from people that know how this works, but imo its fun to watch this

Let it know where the next goal point is going to be after the one it's currently at disappears and add a reward for getting to the next goal faster. That way it'll learn to keep the momentum between goals instead of learning to slow down before hitting goals so it doesn't overshoot them and get punished.

bro did this without even activating windows what a legend

what I learned from the video that Quaternions are not good to use for training, thank you :D Btw., a negative rewardr for overall spinning velocity would help to minimize the quirky movement

There are so many recomendations in the comments, please make a part two where you implement them cuz im curious af

like my boy Pontypants used to say "Epik ballerina simulator 2k", awesome btw

Anakin told the AI to try spinning because that's a good trick.

Tony: JARVIS I think there is something wrong with my suit... JARVIS: It's working fine Sir..

This is how Iron Man should fly in his next movie

Great video, man! Nice to see you being back for the technical stuff! I would recommend adding angular momentum with a negative factor to your loss function (with a certain unpunishable threshold and nonlinear activation) - that may fix the spinning and make your Iron Man fly more like in the movies P.S. It seems like that's what literally every commenter wrote, quite unoriginal 😅 So here's a more original proposition: make another video or two, improving this design instead of moving to a different topic! That would certainly be interesting and also more beneficial for you as a professional

I feel like if you added additional informally like g forces from spinning and added a penalty for spinning 2 much, plus maybe a bonus for having the right way for the drone it could improve the stability. Especially for the iron Man as it was just doing a lazy, I spin 2 win technique 😂😂. Super cool video though loved it.

Fabulous channel and style NEW SUBSCRIBER FROM JAPAN ❤

Maybe use radians instead of vectors for rotation? To make this effective you'll need three reward mechanics: facing, distance to point, and time. 1. Hover: (-score distance from pointA) 2. Hover: (-score distance from pointA) and Face (-score angle offset from direction to pointB) 3. Hover Time: (+score time on pointA) and Face (+score time [very close] to direction to pointB) 3. Race: Time (-score duration from start to pointA) and Face (+score time [very close] to direction to pointA) 4. Race: Time (-score duration from start to pointA then B then C) and Face (+score time to next destination point) 5. Add more and more points until you get to around 10 in one course, train them on that for several days. 6. Hover & Race: Distance (from next point), Face (offset from next point), Time (+score for time on point). Move a single point randomly every n seconds. Once they touch the point set the facing direction target randomly, until the point moves again. Now every time the point moves, they will get a high score for immediately facing the point, getting there as quick as possible, then staying there as long as possible while picking a new facing direction. 7. Bring it all together, and make another long course of 10 points or so, but remove all the rewards except completion time.

I think adding the wrist rotation joint in the hands and ankle joints in the feet would help stabilization a lotttt if trained enough! (As a bonus give it variable thrust... The ability to control how much thrust to output from each of the boosters independently and individually... But it will require a lotttt of training too)

I think the biggest problem of iron man is that he can't see into the future. He only has the next target in mind and the way he has been trained, he wants to be ready to go into a random direction. That means having as little momentum as possible. But in reality the targets are mostly in a line. So he should know the next two targets to properly use his momentum

it must have control over the propulsion force

Wonderful informative humor😄👍

i can see a future in which instagram- and tiktok content creators just rip off that scene of your ironman spinning around slowly through your obstacle course as a background video for their voiceover content

"no Jarvis, I'm fine."

This might be a good demonstration on how the heater element of my old baking oven works. Gets the job done, but only readjusts when falling under or climbing over certain temperature thresholds.

I love the low attention span shade and for that, a sub

you should have continued the first training session until it stopped spinning

Retrain with time taken between points and add penalty for collision with stage. The network will learn that spinning makes it hard to change velocity and will correct itself. It’ll zip around like you want it to.

WOW that is soo cool! please make a followup on this! i think you can make this thing go crazy wild! punish it for spinning so much and instead of training it to go to a random point train it to go for chains of points. so it can anticipate where the second next point will be instead of just being surprised where the next position will be! i like your hairline :)

Good rant. Totally agree.

I Have no clue how u did any of this ,but loved this video i was thinking is it possible u teach it to fly in a 2d space to stop the rotations then mirror on the other axis to try keep it facing one direction please tell me if I'm wrong

yo can i be honest man, I actually would want to see you go through all the equations and math. It might not be the content for this channel, but as someone new and interested in machine learning, it would be so valuable to have someone kinda walk a viewer through the system of approaching stuff like PPO. Eitehrway, I loved this video! I just discovered your channel and I immediately subscribed!

As others have said, part of it is the physical points your taking into account, your wanting 6 degree's of freedom which means that you need to take all 6 degree's properly into account if you do not want to use heading etc because of the discontinuous number though that could have been corrected for in programming, you need to look at it in terms of relative velocities and programming it to do it's best at having certain velocities as low as possible, that's how things like the PID's etc in an Arducopter work for navigation, you take into account your standard X,Y,Z velocities but then you also have horizontal rotation velocity, pitch rotation velocity and roll rotation velocities if you don't want these to be in radians or degree's, have them in m per second or g's that gives you the ability to ask the AI to fine tune the model, the ai also really needs to be allowed to change it's positing more, it seemed in every one of those that it ended up basically becoming a ridged body, which kinda nullifies half the point of the test. and that appears to be because each thruster is always giving out a constant velocity, it's hard to tell without seeing the full code base in unity. Great project never the less love these vids.

" sometimes you gotta learn to run before you can learn to walk" Ironman

Dude the edits man 😆

11:23 dude you're so true. Your words are similar to mine, we share the same knowledge , as great minds think alike Mr. Gonkee

Would be really cool if you made this into a series of a few videos, where you take ideas from the comments and other ideas you come up with to improve the flying. Basically, I think the "meta answer" is to think through all the properties you would want to see in a perfect flight, and then build all of those things into the reward function. Other commenters have mentioned penalties for taking too long, penalties for excessive rotations, etc. One approach would be to compute the total "energy" used by all the rotors, and make the reward function "Gets the the destination in the least amount of time using the least amount of total energy". This would probably have a side benefit of reducing the extra rotation, since that is mostly a "waste" of energy.

This iron man accurately represents how my life is going

seems like the rotation is so that it can use centrifugal force as a stabilization method

Careful with ppo, sometimes the inefficiency and slowness of the training make it achieve lesser results than other alternatives like SAC.

- You should add a sensor for the position of second next objective, the one right next to the current one, this way the model will be able to assess if it need to slow down or keep momentum after meeting the objective. - During training, instead of putting random point, make him follow the discrete points of a random spline or Bézier curve, looping around. It will emphasize keeping your momentum. You can event switch between both, curve and pure random, for a more resilient system. - Maybe add a speed reward too. - You could add a few neurone from output has input, this way you create some kind of short term memory for the model. Having some memory isn't such a bad thing isn't it ? Just imagine steering a car with no short term memory.

Really cool. How are you handling in what direction the AI points its limbs? that seems really difficult to do, and is the rigging limited to the action figure-like marching movements? That could be one reason it's struggling to maintain an exact position. It's hard to tell, but are the motors thrust dynamic? Like are you passing into the model the rigid body's acceleration so it could thrust more when falling faster? Wish the video was longer so I could learn more.

To fly like in the movies the suit would need to create some kind of uplift, that it obviously do not. So one feet need to keep pointing down to hover, but it is absolutly unstable. There for the AI uses spin to stabilize the position. It is really clever, and i think the only smart way to stay stable in the air. And it shows, that it would be almost impossible for a human to fly in such a suit. It would need some movable thruster attached to the hips, to perform some kind of VTOL movements. Very interesting video. Thank you a lot for this hard work!

You probably should added a learning phase where it learned to recover from an uncontrolled fall

Two things that would help greatly. 2. Train Iron Man to keep his eye on the ball. 1. Give his thrusters a variable throttle. He's overpowered right now, and that acceleration needs somewhere to go. That's why he's spinning so much. Spinning is a logical and efficient way to "blow off steam".

if you do something like this in the future, you could add a reward for it facing forwards, it's going to take a bit longer to train, but damm it is going to be more stable

You should add a style reward so he’s not spinning around like a neurodivergent fish and you should also add a speed reward so it’s not taking seven years to tickle the next ball

The first 10 seconds are so majestic 🤩

If that were Tony, He'd be puking in his helmet halfway through the course.

Jaaj ur making videos agoin. Your such a good youtuber

perhaps you could add a reward based on the direction it's facing, as well as the speed of accomplishment relative to the distance? Either way, nice vid, i personally couldn't dream of coding anything 😅

spin stabilization - it makes so much sense that even AI can use it

you could also include propulsion on\off, when it needs to go down it raises a hand and that dips it too far down bc of gravity and propulsion together, another thing could work is add lift force based on speed based on the aerodynamics formula, that could reward forward facing flying in a fast scenario

I wonder what force could've been added to the suit to balance the spinning momentum. Maybe two angled thrusters on the back.

You could try adding a time based reward (a 0 second score should be considered bad as well). A stability based reward could also help with training in the beginning 🤔

Scoring it based on the time taken to reach the point would theoretically encourage it to pick up the pace. Having 2 or more points, and only measuring the time at the final point might keep it from over-committing to any single point. You should have the next 2 or 3 points as inputs for the AI, so it has the opportunity to plan it's path beyond the next point. As others have said, giving more points for facing towards the path could help with spinning, although this should be lenient to factor in the body's inability to turn its head (particularly upwards). Penalizing angular velocity may be a better option, but this has it's own problems where you do want it to turn at times. I think ideally you'd want to have the head look towards an additional target, which may or may not be along the path. You could then penalize failing to do this, and use this to have additional control over the flight pattern. To make the points more lenient, you could increase the radius they detect for. To keep it from being counted early, the point doesn't count as being hit until the body starts moving away from it. Bonus points should be awarded for being close to the point when this happens, but it doesn't gain any benefit from turning around to hit a missed/overshot point.

1) add thrusters to the back 2) add at least 1-10 output for each thruster 3) add negative reward to limit spinning

What might improve things is to have it target the point two ahead from the closest point, that way it always wants to move forward instead of matching the points exactly.

@gonkee adding a facing reward would help stabilize the rotation nad also maybe give it sone control over the thrust control not fully like 0% - 100% but more like 50% to 100%

Gonkee closer to achieving enlightenment with every video

Doesn't get there on time, but gets there in style 😎