OSWorld project video? Yes, please!!!

User: What happens between the steps in these Ikea instructions? Agent: A fuckton of swearing! User: Test passed.

It's great an all, but I kinda think one of two things will end up happening: 1. An AI layer will become a standard for interoperability as part of the OSI and App Dev Stacks 2. A whole new OS will be developed that serves this very purpose. I suspect we may start with 1 and end up with 2 in the longer term.

Please make more videos testing and running this yourself🙏🏼, I'll be great

It would be helpful to have a catalog of pre-built open source AI agents that can be easily downloaded and used for specific tasks. My brain shuts off trying to follow video tutorials on programming my own AI agent from scratch.

Yes Please 👍 Need lots of OSWorld Videos ❗❗❗ We need a video tutorial watching AI, that creates a training set item for OSWorld on how to do X, by watching a video on how to do X (and fills in missing details not shown). 🤯🤯🤯🤯❗❗❗❗

Really look forward to your videos. You’ve helped me get the gist of developments as they come out and determine which technologies are useful and worth spending my time on, and which ones I am equipped to handle, for my personal use-cases. I have and will continue to recommend your channel to friends and co-workers. Seriously man when I see your name, I click. Thank you for continuing to do what you do.

This was good video showing what had to be done to make LLM’s agentic using computer OS’s. It showed me two things. The first was why self autonomous cars are so hard to set up. The auto system has to not only know what the “rules of the road” are, what the automobile’s driving characteristics are, and how to make the car do what it needs to do, but it has to be able to correctly parse at high speed what a situation that it has never encountered before is, what is the correct action to take is and pull off executing it in real time. The second is that a system that can do that well is way closer to AGI than any LLM.

Yes, tutorial please! Please elaborate more on the relationship to CrewAI-like frameworks and potential implications for the rumored YAML endpoints!

Can't wait for the tutorial. Wanted to say thanks for the videos Matthew.

I will be more than happy to see you testing it in real world examples, not complex task but just everyday tasks, like summarize a bunch of pdfs or make a research, and things like that. And also i need to say that a really appreciate your work❤

Yes please! Tutorial on this would be great. I see agents as being a driving force behind vast amounts of commercial AI adoption. Companies want greater efficiency and agents are the tools to bring this.

The most difficult part of making something like OSWorld is security. When you open your OS to computer manipulation, it's a lot easier for computers to manipulate it.

Computer! Computer? [Handed a mouse, he speaks into it] Hello, computer. The Dr. says just use the keyboard. Keyboard. How quaint.

Having OS World integrated in with AutoGen Studio would be really good! I'd like to see a detailed tutorial on that process.

always awesome and informative videos Matt, love it brother. And feel like that much smarter after watching them. Keep up the awesome work!

Not long before we have star-trek style computers, where we just say "computer ... do x, y and z for me".

Yes, def set it up that would be a good video. Keep up the cool videos :)

This is amazing. I think you need a team to help you set it up fast. We want to see a demo!

I was waiting for your own test the whole video. Git'r done son!

Great ! Pls more on this topics

Nice! I'd love to see you test it out.

Thanks Matt. The change-the-background task is like an Optimus realworld task. Using the mouse requires a collection of basic motion skills (eg move in XY, click right/left, scroll up/down, etc). Moving and activating the mouse on a screen are simple subtasks necessary to build actual realworld tasks (on the PC these basic skills and subtasks and more can be accomplished using AutoHotkey). The reactive sequence of mouse subtasks (including motions) is the equivalent of FSD navigation from location A to B in the realworld or Optimus stepping through a set of realworld subtasks to complete a realworld task. The advantage for a change-the-background task AI is the paucity of edge cases that make realworld tasks so difficult for Optimus and for FSD. All three AI systems need to evaluate the realworld changes they evoke before executing the next subtask. Optimus and FSD repeatedly face infinite realworld variations between subtasks. These variations are introduced by independent external agents (cars, animals, fallen trees, etc.) The change-the-background task AI will mostly face changes due to software upgrades and due to different starting states. Most computer issues can be resolved by deeper searches on the web. AutoHotkey can programatically solve simple issues (hiding open windows). Having an AI to navigate the process would fundamentally change the ability to execute complex computer tasks based upon simple sequences of verbal commands. Here is an example: Convert the most recent Matt Berman UKposts to mp4 and then extract unique screenshots to a Powerpoint file and the youtube transcript without timestamps to a text file. The filename for each file is MB1.

Great explanation video. Thanks a lot!

NICE is see you getting Brand deals! May your channel Keep getting more brand deals!

Yes I would like to learn more about OSworld , keep up with the great content

This is really interesting. I've been thinking for ages about how to go from vllm to action. It's a bit like us sitting in front of your computer and describing what you want to happen.

Amazing! I want to try it for Revit, a software for architecture. Actually I did try something that worked! However I truly need more knowledge, so your help is very very aprecciated! Thanks!

I would love to watch you setting up OS world on your own machine testing it out and using it to create a tutorial from it of it.

Would love to see you test this out doing a few different tasks! Also curious if this could run someone's social media etc.

Great Project, it would be interesting to have a video on RAG and programs available for the RAG (example H2OGPT)

Excited for the Humane Ai pin Agents feature coming .

Very interesting. I would love a testing video.

Yes.. please do a tutorial in osworld. Have been waiting for this for some time.

Great video!

You are a very good teacher…so keep teaching.

I enjoy your content Matt. You put me on to LM Studio, then Ollama, then Crewai. CrewAI has excellent case use, so thank you so much. Could you please do some more stuff with CrewAI (I have mine setup in the one file approach, but am not too sure how to set it up with multiple files and calling to and from (I'm not to familiar with Python, chat gpt is excellent help, but it still only goes so far)..

This video was great thanks. Could this work with IoT like devices?

A new golden age of open source is upon us ❤

Great presentation! Too bad the accuracy levels are currently so low but this seems to be a framework that can self-grade and, hopefully, self-adjust for improvement.

How close until I have a locally run agentic system that can install all future improved agentic systems and/or github projects autonomously?

Are we able to programme new action's? or create new connection? like what we can already do with macros?

Yes, test it out, go deeper, it is a very interesting project!

Thanks for the video! Yes, this seems like it will be very useful.

16:40 think of temperature as of maneuverability. the higher it is the more flexible the system, which is basically a closed loop control system at this point.

i would definitely like to see more OSWorld thank for bringing this exciting news to us

great vid!

I am already waiting for a linux where i could enter commands in natural language and the llm model gemerates a set of possible true commans and i just have to choose one or make a minor change

Any link to the presentation? Thanks!

Awesome Thank you.

Does anything change for your video with their addition of memory sharing?

can a multimodal model scroll up or down on a screen and see more than just wha tis displayed? Can they actually read the text on cmd terminal and then act on it instead of us copy and paste it the reply into an input cotext?

OSworld project videos please! This could be a series of videos? I could see this helping me do my job five times faster ! Helpdesk support tool to check and update XYZ application user account then email user letting them know we have updated their account and that they should be able to login. Complicated processes, such as opening VPN connection and checking active directory account settings, and then logging into administrative program(s) to search and open users account to check their settings. The user account Settings in active directory must match the user login settings in the application(s). Email the findings and let them know what was altered or changed, etc..

ive always wondered if you could incorporate vision for llm into something like shadowplay

hello i'v registred on massedcompute but couldn't find you preinstall system. Any tips? And may be you have a tutorial?

Very nice project. I would find interesting to see success rates in different OSes (or in case of Linux even DE/WM). Also GUI vs CLI - I can imagine on some tasks CLI would be a king, while in others it could fail miserably. Still, it could be useful to see for which use cases different OSes or GUI/CLI are better and might be worth of trying to utilize an AI for them.

would interesting to have a tutorial on this setup

Thanks Yes please install it and show us the procedure. I think it is going to be useful for a lot of people.

Please do a series on Fine Tuning open source models!

I think temp is set to 1 because if it fails and does another attempt it will have different approaches. When temp is set to lower values it might not get to a working solution because the tried method’s are not divergent enough to contain a valid solution. But i think having an llm fine tuned on datasets generated by humans in the format of OSWorld (the tree, screenshots ect…). Could improve the succes rate. If I am not mistaken this is what Rabbit R1 was doing. It’s basically teach mode but with more examples then just the one you give it.

Of course we want to see more about that ;) thx 4 all

Do you have plan for taking about DSPy?

What Python version as a component of OSWorld is required to launch in a Linux Arch Zen Kernel environment?

Update us always like this, especially the new research papers

With regard to difficulty of an AI to access the desktop, is there an exception if we are talking about just manipulating a browser window through the use of selenium?

Good! So, how do we test it?

For performing actions on desktops there is a macro recorder available and suitable.

Very similar to RPA (Robotic Process Automation) that I’ve been developing for 10 years now. Nothing new, but being able to do this with a typed or vocal prompt is what’s going to be interesting when it does get as good as a human can do (which is what RPA has been successful at doing for a long time), also understanding that RPA licences are expensive.

16:38 It depends on the specific formula used for the temperature setting, so a 1 here is by no means the maximum. The use of top-p implies there is nucleus sampling being used, which prevents the most improbable completions from even being considered. They are looking for a wider sampling to establish a baseline and setting the temperature too low would create more repetitive results (repeats across different runs and also repeating the same phrase in a single run until the context is full) and thus would be too easy dismiss as a strawman.

This is great! I'm going to wait for a Linux distro that has these agents built-in to automatically configure Wi-Fi, printers, drivers, or even VMs with Windows (for specific programs that don't work in Wine).

hopefully one day we'll get agents like the Muses in Eclipse Phase or the Alt-Me in Peter F Hamilton's Salvation trilogy

Soooo we basically arrived at Reinforcement Learning using LLMs? Thats sounds so awesome!

Actually your video, specifically the table, convinced me that agents at least in this interaction are not all that spectacular. They will likely get there but right now it’s a lot of hype.

Considering a project where a person documents daily experiences, thoughts, feelings and personal history in a diary specifically for a future AGI’s learning. Do you think such a personalised dataset could enhance an AGI’s ability to understand and interact with individuals on a deeper level? And lastly, is it feasible to expect an AGI to become a close, personal companion based on this method, or would it somehow be redundant useless data? Thank you for the answer.

This only needs to work once and you basically build the universal installer. Soon you just tell a computer "make the latest stablediffusion (or whatever) work on my computer, including all the hardware-specific optimizations that apply to my specific system. Then it just needs to bootsrap in the newest interaction AI for my OS, have a little conversation with the system, try promising settings, and if they fail, come up with others, and (importantly) update the weights of the remote installer system based on the successes and errors of this particular interaction.

So which team are u on: OS World or Open interpreter 01lite

Cool - This is moving SO fast! I think we will get AIs with the ability to recognize what is on the screen more directly, much like a self-driving car sees the world. This would become 'go click the button that does X', without screenshots. I bet that happens this year. Real world agents with AGI for a Christmas present!

In Linux there is the xserver.. I've been thinking it would be neat to plug a system into the xserver backend, and have an llm communicate with that directly... Somewhat bypasses most visual interpretation, except what is actually rendered as graphic

Nice, but does it support ollama?

Taking a screenshot and guessing is an impractical implementation, for desktop agents to truely work we would totally have to create new apis that directly alters the desktop state and best operating system to do this is Linux right now, but if max and Windows also provide them I think then it is possible for agents to make a significant impact

Please do it! Tutorial for OS world

Interesting take, but again, trying to go to general. I am curious if there is a team working on a real "AI OS". Not using screenshots and these half-solutions, but actually having predefined built in functions that control the device through code and track the progress in the same way to do the "grounding" step?

Please do a test. Thanks!

I have no idea what is going on, especially of not knowing what agents means, is it like a another word for chatgpt or something else?

I think they set the temp @ 1.0 to test how hard it will hallucinate if given more creative freedom, then added it to the presentation just to show off

Yes! Make a tutorial of it! :D

Is it affectors or effectors?

Often restricting model to output code only reduces the accuracy, especially on complex tasks. It's worth trying to allow it to print chain of thought (even better if there is a self-critical inner dialogue loop) and then output the final code piece.

Please create an OSWorld Tutorial

Feels like a talk form the future o_O

yes! do a tutorial!

OK excuse ignorance, but a lot of this sounds like Robotic Process Automation (RPA)? Maybe with a GUI and an effective chat layer (LLM)? Proceed to correct me :)

I want the digital agent in my phone to download my monthly invoice from the electric utility, merge that and other data I want recorded publicly into a decentralized graph representation that is maintained in collaboration with digital agents running in other personal devices to create a shared world model for planning collective action.

Temperature of 0.1 could lead to “I cannot click, I’m just an LLM.”

Ty!

Please make video about Romo AI- super AI tools in one place and learnex AI - world's First fully AI powered education platform. My favorite AI tools

20% success rate is super impressive a start. As soon as they iterate on that and train a proper model, it will reach 99%, leading to all office workers getting fired.

I think as this evolves it’s time for somebody to create a Linux system that would work directly with this, you need an operating system, catering directly to the agents

why not link the project in your description?

will be interesting to see ReALM geting benchmarked