Video Capture in Arcade

[Akascape]

I created a shader filter and used it with a video as an input channel, it is working properly but I want to save the rendered frames and make a video out of it.

Code:

import arcade
from arcade.experimental.shadertoy import Shadertoy
import cv2 

SCREEN_WIDTH = 400
SCREEN_HEIGHT = 300
SCREEN_TITLE = "ShaderToy Video"


class ShadertoyVideo(arcade.Window):

 def __init__(self, width, height, title):
 super().__init__(width, height, title, resizable=True)
 self.shadertoy = Shadertoy(
 self.get_framebuffer_size(),
 """
 void mainImage( out vec4 fragColor, in vec2 fragCoord )
 {
 // Normalized pixel coordinates (from 0 to 1)
 vec2 suv = fragCoord/iResolution.xy;

 fragColor = vec4(1.5 * sin(suv.y * iResolution.y/3. + iTime * 20.));
 fragColor = 1.- floor(abs(fragColor));
 fragColor *= vec4(sin(suv.y), 0, cos( 1. - suv.y * 2.) , 1);
 fragColor *= texture(iChannel0, suv);
 } 
 """,
 )
 # INSERT YOUR OWN VIDEO HERE
 self.video = cv2.VideoCapture("Example.mp4")
 width, height = (
 int(self.video.get(cv2.CAP_PROP_FRAME_WIDTH)),
 int(self.video.get(cv2.CAP_PROP_FRAME_HEIGHT)),
 )
 self.video_texture = self.ctx.texture((width, height), components=3)
 self.video_texture.wrap_x = self.ctx.CLAMP_TO_EDGE
 self.video_texture.wrap_y = self.ctx.CLAMP_TO_EDGE
 self.video_texture.swizzle = "BGR1"
 self.shadertoy.channel_0 = self.video_texture
 self.set_size(width, height)

 def on_draw(self):
 self.clear()
 self.shadertoy.render()

 def on_update(self, delta_time: float):
 self.shadertoy.time += delta_time
 self.next_frame()

 def on_resize(self, width: float, height: float):
 super().on_resize(width, height)
 self.shadertoy.resize(self.get_framebuffer_size())

 def next_frame(self):
 exists, frame = self.video.read()
 frame = cv2.flip(frame, 0)
 if exists:
 self.video_texture.write(frame)


if __name__ == "__main__":
 ShadertoyVideo(SCREEN_WIDTH, SCREEN_HEIGHT, SCREEN_TITLE)
 arcade.run()

Is there any way to save the new rendered frames and make a video? Please help :)

[einarf]

There is a simple capture module on moderngl-window we thought about porting into arcade. It simply pipes raw frames in stdin : https://github.com/moderngl/moderngl-window/tree/master/moderngl_window/capture

You can probably easily adapt that one to arcade.

[Akascape]

I am still confused on what to do :(, can't we just get the frames by any method and use opencv to convert it?

[einarf]

I guess you could check out the VideoWriter in opencv, but I haven't used that before.

To access the raw pixel data from the window framebuffer you can window.ctx.screen.read() to get the byte data.

[Akascape]

@einarf Ya, I know about the VideoWriter method of opencv, but we have to get the frame data of the window one by one to write the video.

[einarf]

You know how to get the frame data now. Be free to ask questions if you are stuck. We also have a discord server that might be a better place to have conversations about this. The plan was to add a simple capture module in arcade 2.7 at some point, so maybe this can help getting the work done.

[Akascape]

@einarf I have used this method cv2.imwrite("out.png", self.ctx.screen.read()) to write the image file but this is giving errors.

[einarf]

I think opencv wants a numpy array with a certain shape and data type. When asking for help you should always include the actual error.

Again: Discord is better for this if possible

[Akascape]

@einarf Discord link?

[einarf]

Front page in arcade docs.

[pushfoo]

Given this is now a unique ticket, I crossed it off of #1342.

[pvcraven]

Closing stale issue.