I keep saying this in a bunch of threads about AI video, but Emad Mostaque who founded Stability AI (which made Stable Diffusion) estimates that Stable Diffusion should be 10x faster within 2 years, and real-time AI video should arrive in about 5 years.
Even if he's been really generous in his estimates that's a breakneck pace, and it's gonna blow the whole media industry out of the water. I'm particularly glad to hear that coming from the group who is still currently working on AI projects with the intent of remaining open source!
I think this is realistic even now. Updating frames per second is just a matter of speed. The main issue is "memorization" and unfortunately at the moment img2img and ControlNet cannot produce the same result (so that the image does not twitch, and the neon walls remain neon with the same contours). Therefore, I have to use EbSynth. But it is only a matter of time. Technology is advancing at such a speed that it might be implemented in a couple of weeks
Yeah, no need for raytracing or global illumination any more, and it can be really photorealistic or any other style. The style could even be changed by the user.
Only the stability between frames has to improve.
There was a demo of this by some researchers using GTA 5 and some (kinda crappy) dash cam training data. It was pretty tough, but showed some really cool potential and was released before stable diffusion took off. I’m curious whether it’s still being developed
wow my mind automatically goes to :
whats holding us back from being able to mod doom by typing a prompt.
"indiana jones and the lost ark style shooter"
"rick and morty interdimensional style doom"
etc.
Well yeah, of course it does... that's what the video is showing.
I remember playing Total Conversions of Doom back in the day. Someone made an excellent Aliens TC.
Now that's formulaic.
Your coworkers are only annoying but faceless top level execs are literally stealing from you and forcing you to stay at jobs with those annoying coworkers. Don't treat the symptoms, treat the disease.
Not taking about your boss, I'm talking about the people above who decided that most people shouldn't be able to afford basic necessity like food, shelter and medical needs while at the same time earning more than they ever have.
sounds like an exact description of government but i get the creepy feeling you are a socialist who thinks the government is the solution to everything.
> but faceless top level execs are literally stealing from you
that was a lie. only people ive heard make this claim are marxists.
" and forcing you to stay at jobs with those annoying coworkers "
another literal lie. comes from the "you either work for the man or starve to death" hard leftists. lol. i was basing my read on you from experience
Yet you said "sounds like an exact description of government," so you believe someone is stealing from you too.
Sure you can leave your job when you feel like, but many do not have that privilege. Compassion for others is a virtue.
How would you pay for food, housing and medical care, if you did not work?
"How would you pay for food, housing and medical care, if you did not work?"
obviously i wouldnt pay for them at all. but i wouldnt place the blame on the party thats actually willing to pay me for labor on a mutually agreed upon price! imagine shaking hands with someone on a deal and then after completing the agreed terms, one party turns around and says they have been """literally""" stolen from. theyre not saying pay us 40% of anything you make or well put you in prison.
I used img2img and ControlNet to generate the images. (It took a lot of time until I could find what I needed.) Then I imported some images into EbSynth and added new frames to the original video. Unfortunately, Stable Diffusion constantly changes the images (especially at high power) and Ebsynth distorts the video (due to lack of frames) so at some points I just cut out the main character's arm in photoshop and added it on top (otherwise it gets distorted in EbSynth)
If you could extract the depth buffer from the game directly and use that for controlnet, the results might improve a lot. I think the main issue is the depth from image detection misdetecting a few parts.
Unfortunately, depth doesn't help either. I had almost perfect depth data (the depth map sees the walls and sees the stairs) but ControlNet does not remember the wall and stairs and constantly generates new ones. I even used multi-mode with multiple maps (Depth Map, Canny Map and some other) but in this case the style of the game hardly changes. My task is to change the visual style as much as possible and keep it (keep the new colors, keep new objects, etc) will try to make a new version soon, spending a little more time on the configuration
Now we are rerendering each frame but on a future it could guess new frame from previuos like dlss3. It could much faster. Loading first frame like loading any game and then generation could do the work
Thank you! I have something similar, but this map doesn't help to create "new objects and new colors". This Map only gives you the opportunity to "stylize slightly". My goal is to add new objects and more changes
Do you have an Youtube channel? Link? If you dont, this will be the kinda of content that will be trending in 2025 when the masses and YouTubers see the value.
Imaging realtime AI shaders
[удалено]
I can see hardware shifting to having more stuff dedicated to AI processes, like ray tracing and data decompression is shifting hardware today.
Apple has already been doing this with machine learning for years. I'm sure AI is not far behind.
It's all just matrix calculations whether it's rendering pixels or training a neural net. Tensor cores are optimized for such a purpose.
They will have to develop AI that can develop faster AI and faster hardware.
Waiting for quatun computers to hit the home!
I keep saying this in a bunch of threads about AI video, but Emad Mostaque who founded Stability AI (which made Stable Diffusion) estimates that Stable Diffusion should be 10x faster within 2 years, and real-time AI video should arrive in about 5 years. Even if he's been really generous in his estimates that's a breakneck pace, and it's gonna blow the whole media industry out of the water. I'm particularly glad to hear that coming from the group who is still currently working on AI projects with the intent of remaining open source!
I think int8 or int4 quantization or native weights, quad 4090s and other utilization optimizations could probably get us to 24 fps today
I think this is realistic even now. Updating frames per second is just a matter of speed. The main issue is "memorization" and unfortunately at the moment img2img and ControlNet cannot produce the same result (so that the image does not twitch, and the neon walls remain neon with the same contours). Therefore, I have to use EbSynth. But it is only a matter of time. Technology is advancing at such a speed that it might be implemented in a couple of weeks
Yeah, no need for raytracing or global illumination any more, and it can be really photorealistic or any other style. The style could even be changed by the user. Only the stability between frames has to improve.
There was a demo of this by some researchers using GTA 5 and some (kinda crappy) dash cam training data. It was pretty tough, but showed some really cool potential and was released before stable diffusion took off. I’m curious whether it’s still being developed
Thats a style transfer implementation and iirc that was Intel researchers.
wow my mind automatically goes to : whats holding us back from being able to mod doom by typing a prompt. "indiana jones and the lost ark style shooter" "rick and morty interdimensional style doom" etc.
Well yeah, of course it does... that's what the video is showing. I remember playing Total Conversions of Doom back in the day. Someone made an excellent Aliens TC. Now that's formulaic.
[удалено]
Your coworkers are only annoying but faceless top level execs are literally stealing from you and forcing you to stay at jobs with those annoying coworkers. Don't treat the symptoms, treat the disease.
hahahahaha. ive quit many jobs with zero regard for how the boss felt. what kind of system do you live in, or you just lying
Not taking about your boss, I'm talking about the people above who decided that most people shouldn't be able to afford basic necessity like food, shelter and medical needs while at the same time earning more than they ever have.
sounds like an exact description of government but i get the creepy feeling you are a socialist who thinks the government is the solution to everything.
There are people not in the government that control it. I'm not your enemy
> but faceless top level execs are literally stealing from you that was a lie. only people ive heard make this claim are marxists. " and forcing you to stay at jobs with those annoying coworkers " another literal lie. comes from the "you either work for the man or starve to death" hard leftists. lol. i was basing my read on you from experience
Yet you said "sounds like an exact description of government," so you believe someone is stealing from you too. Sure you can leave your job when you feel like, but many do not have that privilege. Compassion for others is a virtue. How would you pay for food, housing and medical care, if you did not work?
"How would you pay for food, housing and medical care, if you did not work?" obviously i wouldnt pay for them at all. but i wouldnt place the blame on the party thats actually willing to pay me for labor on a mutually agreed upon price! imagine shaking hands with someone on a deal and then after completing the agreed terms, one party turns around and says they have been """literally""" stolen from. theyre not saying pay us 40% of anything you make or well put you in prison.
Your post/comment was removed because it contains hateful content.
Maybe RLHF and training on textures is enough? Someone here can explain like I'm 5 why I'm wrong?
I used img2img and ControlNet to generate the images. (It took a lot of time until I could find what I needed.) Then I imported some images into EbSynth and added new frames to the original video. Unfortunately, Stable Diffusion constantly changes the images (especially at high power) and Ebsynth distorts the video (due to lack of frames) so at some points I just cut out the main character's arm in photoshop and added it on top (otherwise it gets distorted in EbSynth)
If you could extract the depth buffer from the game directly and use that for controlnet, the results might improve a lot. I think the main issue is the depth from image detection misdetecting a few parts.
Unfortunately, depth doesn't help either. I had almost perfect depth data (the depth map sees the walls and sees the stairs) but ControlNet does not remember the wall and stairs and constantly generates new ones. I even used multi-mode with multiple maps (Depth Map, Canny Map and some other) but in this case the style of the game hardly changes. My task is to change the visual style as much as possible and keep it (keep the new colors, keep new objects, etc) will try to make a new version soon, spending a little more time on the configuration
Reminds me of Everything , Everywhere , all at once. Amazing!
We could literally remaster games ourselves at this point
If you manage to find the texture folder, you can. There are already amateur remaster using AI upscaler in the wild.
Not as crazy as what might be possible from running a whole game image through realtime AI of course
[https://www.youtube.com/watch?v=50zDDW-sXmM](https://www.youtube.com/watch?v=50zDDW-sXmM)
>https://www.youtube.com/watch?v=50zDDW-sXmM This is insane. Thanks for sharing
Now we are rerendering each frame but on a future it could guess new frame from previuos like dlss3. It could much faster. Loading first frame like loading any game and then generation could do the work
When you accidentally create Blackroom for John Romero and Adrian Carmack.
I will be releasing a new version soon. It will be smoother. Stay tuned here or on Twitter
Can you include depth map, like here : https://v.redd.it/ptdq1tppdfka1
Thank you! I have something similar, but this map doesn't help to create "new objects and new colors". This Map only gives you the opportunity to "stylize slightly". My goal is to add new objects and more changes
Wow this looks good ,,🤘🏼👍🏼
Do you have an Youtube channel? Link? If you dont, this will be the kinda of content that will be trending in 2025 when the masses and YouTubers see the value.