The use of negative prompts if pretty clever. Sort of like using an adverserial network.

These should be called "Re-imaginings" rather than "Enhancement"

This looks more like gen fill than upscaling to be honest. Just check out the car example, the manufacturer logo chaanged completely even tough there was enough information to at least rebuild into some similar shape.

This can become a real problem when it comes to truthfulness of data though. Imagine this being applied to surveillance cameras used to identify criminals. The ai is likely to extrapolate faces to common features rather than recognizable, true ones.

I love how the triangle badge turned into a Nissian lol. AI: Close enough.

what was missing: taking a high res photo (ground truth), creating a low res version that is processed by this model (SUPIR), finally comparing SUPIR to ground truth. Otherwise very promising paper.

Agreed, the ai is making it up the best it can from reference it has so you are essentially looking a new render rather than enchanted image

Can't say this is something we'd want to use regularly from images on the net, considering how it basically does hallucinate details that aren't there.

You can't resolve information that's not actually present in the pixelated image. You *can* allow the model to make a reasonable guess and come up with something visually convincing. If there are several images in a series that the model can draw from for context, then presumably you can make the "upscaled image" more accurately represent what was originally there. But that probably requires another paper. The idea of using text prompts to help it understand what the missing information is is sort of a half-way measure to doing that.

Another great video, keep it up! ❤

I can't wait u til this becomes open to thr public. Would be so helpful

Amazing, so many applications for this upscaling

WHAT A TIME TO BE ALIVE!

It is indeed a fantastic model. Can’t wait for motion implementations.

Training models with bad samples associated with negative prompts is such a good idea, you wonder why it hasn't been done before. And that method can apply to all kind of generative AIs

Holy mother of papers! I agree and look forward to seeing appropriate use of this technology preceded by societal discussion to define what's appropriate and what is not.

New two minute paper! Thank you for keeping us informed Dr Károly Zsolnai-Fehér!

Wow! Astonishing!

4:35 Holy Mother of Papers! 🤯

Computer, enhance!

Simply amazing

Impressive!

Love to see some of the old Movies through this to see what they might look like if done today.

Looking forward to see it used on Space telescopes images and in phones. ... or those always blurry UFO images ;D ... I might be used as a pre-warp signature - "Can they see us ? Nop. We'll come back later." Seriously, great progress! I guess video is the next step.

6:12 Superstar on the left..mass

You should have mentioned that this works by using SDXL (Stable Diffusion XL). I cite the paper: Specifically, SUPIR employs StableDiffusion-XL (SDXL) [63] as a powerful generative prior, which contains 2.6 billion parameters. To effectively apply this model, we design and train a adaptor with more than 600 million parameters. The only thing holding us back from using SDXL like this ourselves to upscale to very high resolutions without server grade GPUs is the lack of ControlNet tile model for it.

I don't understand why you'd imply this AI up scaling could possibly 'guess' the number plate. It has like 4 blurry lines. The number plate doesn't even properly overlap the original image.

I considered it as hallucinations because entropy is entropy, if information is lost, it lost forever. So whatever the works that try to refill the missing details are just an attempt to guess the information but not the real thing. To make matter worse, we chose to believe it.

What a time to be alive

Could be really helpful in astronomy.

I guess enhancing photos in movies wasn't unrealistic, it was just futuristic 😀

The reference test for any "enhance" algorithm should be how well it is able to upscale old Star Trek episodes from SD to 4K.

Did the competitor models also have the whole zoomed out image available as context?

2:14 - managed to turn a Dacia Spring into a Nissan 🪄

4:37 "Holy mother of papers!"

I wonder how soon this will be available in an accessible tool like Upscayl?

Remember how in the 90s crime tv-shows claimed that this is possible? Having bad surveillance footage and then the detective/investigator character standing behind the 'computer specialist' character and the dialog goes like: "can you enhance it?" - "okay, applying extrapolation" or some bullshit haha…

Does it work with old Tape Footage? Like from old Formula 1 races from the 1970s-2000s ?

Very promising but the models are not available yet.

I’m surprised that the car brand badge went so wrong. I would have thought that car badges would have a pretty strong dataset by now.

why are we not seeing an original hd picture, turned into a low res picture, then upscaled with the a.i. so we can compare the a.i. upscale to the hd?

I look forward for when I can just drag and drop an old video file into a program, hit start, and end up with a fully upscaled video with crisp lines.

Finally the KreaAI/MaginificAI type upscaler as open source. Hopefully it can be run on regular hardware.

What is the best publicly available model for accurately upscaling video?

Temporal coherence? (could i use this technique for video upscaling? - would that weird monkey man character's face be stable over time - or would the upscaled clear version flickr all around the place?)

Fantastic! Now we can start analyzing all those blurry UFO photographs. 😂

Enhancing pixelated photos should have a disclaimer since other derivatives could be created, none of which may be identical to the original. Where 'scaling up' images will prove to be game changing is when it can be applied to pixelated video. Using all of the sequential images to depixelate a video until there is only one 'true' solution, that being the original video prior to pixelation, will probably throw a lot of people in shock!!

At one hand it is great, that ai not only restores pixels, but tries to restore surfaces and structures according to it`s knowledge. On the other hand we can see, that ai is not intellectual enough to suppose regular structures like stone pattern or black and yello parallel stripes. So perhaps this ai need help of another one like SAM to separate blocks and try to suppose their features. Anyway such models will do great help to designers, because retouche is very time consuming task.

2:05 - Forget the license plate! The logo changed shape drastically!

I love your videos!

in future they will think that CSI scenes with "zoom in... zoom in... enhance... zoom... enhance.. there! he's our guy!" were actually real and ahead of their times

Did you not do a video on "magnific" AI ? That's super interesting, with high res results.

This seems to use SDXL and LLAVA plus an algorithm to upscale and increase quality, considering how it works it could be embedded in Stable Diffusion WebUI or ComfyUI.

@1:50 Finally we get the technology that detectives have been using in movies... "ENHANCE!"

I wonder if/when Streaming services/UKposts will start using this (or if they don't want to, so they continue to make you pay for higher quality versions)

4:02 I think that more people nowadays have better access to the fast internet than to the powerful phone

I wonder if you can use it to get license plate numbers from bad dashcam footage in hit and run cases. Might have more than one frame to go off of as well.

2:05 You can clearly see that it's not the same car with the same emblem. I think originally it's a Dacia and it's made it to be a Nissan.

ENHANCE!

This is not to be used to enhance faces on security footage as the output will be a good face but at best it will be a face that's similar to the original person when downscaled.

just imagine when we get this for videos with good temporal coherence... oh boy

VR -> AR -> BR (Bent Reality). Soon to come in the next paper: NR! (No Reality)

Never in my life i would've imagined to see a Dacia Duster on Dr.Zslonei's channel 🤣

This one is going to need a few more papers.

You seemed to gloss over the fact that this method uses SDXL. It will neither run in real-time nor run on mobile devices as suggested. You also have to contend with how SDXL scales poorly to larger resolutions, restricting this method to smaller output resolutions than a GAN based SSR. Perhaps a smaller model with a similar method could work, but there's no guarantee that it would achieve these types of results. Additionally, using this method to decompress images on the web would probably be a bad idea since it will "hallucinate" detail that may change the meaning of the image, which would be amplified by the random nature of LDMs.

Yep, Blind Lake is almost here

Do you know if SUPIR will also be runnable offline? I'm kinda fond of stuff that can run offline cause I don't want all pictures to be processed on the internet, for example if it's pictures of myself or whatever but sadly a good bunch of the best stuff is often online or behind a paywall

Imagine ALL the low resolution 240p, 480p and lower movies, tv shows and videos from the past fully restored to basically 4K with this AI. That would be incredible. I would love to watch SD shows, and cartoons and anime from back in the day with this clarity or old 1940s and 50s black and white movies in 4K and colorized. The near future will be amazing.

2:24 logo changed to round shape from V shape - ai knows what's gonna be there but not accurate about what's there .

Can't wait to play 90s games with new graphics without having to do a remake.

Funny though, that is figured out the license plate, but it completely changed the brand of the car. That is a Dacia and it replaced it with something Opel/Nissan

So when we can watch the Charlie Chaplin movies in 4k?

What paper is that at 2:48?

Question - if this could run in real time (ie. 30-60fps) and you have an old video game (DOOM for example) being upscaled - are the upscalings of the textures done in a semi-random way that would give an individual frame a vastly improved pixel resolution that looks fine in isolation, but when checking frame by frame - it would appear that there was too much difference between them for a smooth video? Does that make sense? The process is guesswork, rather than a precise upscale method. Photos or frames of video look great - but the process is done in such a way that an animation would look corrupt or parts of the image would appear to wobble or jump around because the process is randomly generative, not interpolating based on a fixed idea? I guess this is so hard to describe because many of the words needed to describe what's happening don't exist because these new AI tools are creating things that have never been done before! Also, you can absolutely NOT use upscaled images in scientific applications or in a crime situation, like upscaling CCTV etc. Because it's generating NEW data that definitely didn't exist before. You can't use it to look for stars or galaxies in upscaled space photos, or find a person based on an image upscale because it's pure guesswork?

2:20 from Dacia to Nissan :D

It becames scarry, really fast

If it looks good it is good.

It is neat that it is creating a realistic output. One thing to keep in mind is that the output doesn't really seems to be really contradictory to what the input was. A lot of the details were clearly not the details in the input. Really, to demonstrate this we'd need to see the original then the pixilated input the AI is given then the output the AI created. I'm sure there would be tons of errors comparing the output directly with non pixilated input. Tl;Dr this isn't the "enhance enhance enhance" trope promised to is by TV shows.

I remember this in the CSI movies (Enhance....enhance more....enhance ) and saying ... thats not how image enhancing works. Looks like i might live long enough to see it become reality :D

Feels very much like current img2img with controlnet + segment anything regional/regional prompting

Previous technique weren't able to recognize it was person... Neither did I 😂

I see at least one practical application: Changing the looks and ages of the FBI's Most Wanted and missing people. The missing people section often shows corpses, which creeps me out. I'd rather AI imagine what they looked like alive.

It's still going to lack temporal coherence if used as is on video I suppose. But the results are amazing.

It's scary to think of all of those detective shows where they "zoom in and enhance" to catch the criminal. Now the technology actually exists, but it simply hallucinates the details it's putting in. But are police detectives really going to understand that? These are the same people who think they can tell who is lying from their facial expressions, and who think that lie detectors actually work.

The upscaled images were part of the original training data, right? Because otherwise I don't know how the AI could imagine the skiing womans face including correct reflections on the visor. So the AI "remembers" something. If these were images the AI has never been trained on, I don't think the results would be that good.

I hope this model to be integrated to the waifu2x caffe

It went from a Dacia Duster to a Nissan Dusty

What's the difference between this and something like Dall-E or Stable Diffusion? I mean, low resolution seems effectively the same thing as noise, and if it accepts text prompts, how is it different?

Nvidia already has this, theyve had supwr resolution for videos in real time for around a year now. Its more subtle than this though

This feels kind of odd, hasn't this tech been a thing with dlss, or Intel and amd's proprietary offering? What distinguishes dlss 2 (not frame generation) from this?

This kind of tech should be default in phones in 3 years. Like you don't need an app, just every time you take a picture it makes it super clean if you opt in.

Definitely gen fill. It's using a similar thing to reverse google image search where it compares the image, finds similar images of a higher resolution and uses the data to generate the photo. Amazing though. If they released it as paid software on the Mac, I'd probably buy it.

I think the first few seconds may be missing.

old kung fu movies and other ones will be way better now. I kinda wanna clean one up and give it to my dad.

This upscaling tech would not work in situations where fetching precision information is crucial. Look at the example of the license plating on the vehicle. Otherwise, it would be fantastic alternative to something like the JPEG format.

How fast are they?

Everything has it pros and cons. Now how to create/innovate ways to detect misuse of this technology? And of course the next paper will bypass such detection. And we get back to making detector for detection. And another paper will... Ah! We've come full circle

The problem is that nobody can install and test it because the installation instructions are incorrect and the version of Python 3.8 for the environment is the wrong one, as there are major conflicts, especially with the triton version. Plus, there's no explanation of the models and files we need to download. And there's still no change or answer to this question. For a week now. It's really strange that some people can't provide the right information. I really don't understand that.

Uuuuuh that eye

2:33 Unreal Engine Nanites have left the chat...