Can't receiving Video frames frequently and not sending/emiting frames back to client. (I ran this stup in local machine/Ubuntu laptop with setup of uWSGI and gevent)

[vasanthhr]

Dear ALL,

I tried to setup the flask_socket with uWSGI and Gevent in local machine/laptop (Tried in both Ubuntu and windows machine). But I didn't see the request and response properly. I executed this different steps and its respective result.

(Please help me to resolve , why these step1 and stpe2 is having this issue. Guide me If I am doing anything wrong here.).
(If ran as like Step1, then not able to connect via socket-connection from client and not able to receive response from client as well )
(If I ran as like Step2, then I client can connect but not able to send video frames , and client socket getting disconnected after few seconds).
(If i ran as like Step3, then server receives the frames and response the frames back to client, that means clients emits frames with very high performance/speed and server receives in the same speed, but server sends response with very slow or less performance, means response is having video lagging).

Step1:

client/browser trying to send video frames to local server(uWSGI with gevent) and this server not able to receive the socket connection , and it says "connect error timeout". The command I used to run the python app is as below.

uwsgi --http 127.0.0.1:5000 --gevent 1000 --http-websockets --master --wsgi-file server.py --callable app

Started logs:

*** Starting uWSGI 2.0.17.1 (64bit) on [Fri Jan 25 13:35:03 2019] ***
compiled with version: 7.3.0 on 25 January 2019 00:48:15
os: Linux-4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018
nodename: vasanthahr-VirtualBox
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /home/vasanthahr/code/python/modaka/live_video_process/live_video_proc_v1
detected binary path: /home/vasanthahr/code/python/modaka/env_package_m1/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 43789
your memory page size is 4096 bytes
detected max file descriptor number: 1024
- async cores set to 1000 - fd table size: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on 127.0.0.1:5000 fd 4
uwsgi socket 0 bound to TCP address 127.0.0.1:40137 (port auto-assigned) fd 3
Python version: 3.5.6 (default, Jan 23 2019, 15:21:07)  [GCC 7.3.0]
Python main interpreter initialized at 0x5597139354c0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 21036928 bytes (20543 KB) for 1000 cores
*** Operational MODE: async ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x5597139354c0 pid: 30090 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 30090)
spawned uWSGI worker 1 (pid: 30096, cores: 1000)
spawned uWSGI http 1 (pid: 30097)

server.py:

from flask import Flask, render_template, session, request
from flask_socketio import SocketIO, emit, join_room, leave_room, close_room, rooms, disconnect
import numpy as np
import cv2
import base64
from time import sleep

from streamer import Streamer



async_mode = "gevent_uwsgi"

app = Flask(__name__)

socketio = SocketIO(app, async_mode=async_mode)
streamer = Streamer() #threading inside this file


@socketio.on('video_frames', namespace='/facelandmarkpose')
def video_frames(message):
    input = message.split(",")[1] #image data, frame data
    streamer.enqueue_input(input)



def gen():
    while True:
        frame = streamer.get_frame() #pil_image_to_base64(camera.get_frame())
        yield frame



@socketio.on('video_feed', namespace='/facelandmarkpose')
def video_feed():
    for video_frame in gen():  
        emit('from_flask', {'data': video_frame.decode()})



@socketio.on('connect', namespace='/facelandmarkpose')
def test_connect():
    emit('test_connect', {'data': 'ConnectedFInal'})


if __name__ == '__main__':  
    socketio.run(app)

===============================================================

Step2:

client/browser trying to send video frames to local server, server receives the frames but its too slow to receive each frames but not emitting/sending any frames as response(Client expecting a socket.on response from 'from_flask' ). And in server.py i set the async_mode as "gevent"

I ran the code as follows

python server.py

logs(in command/bash prompt):

*** Starting uWSGI 2.0.17.1 (64bit) on [Fri Jan 25 13:26:47 2019] ***
compiled with version: 7.3.0 on 25 January 2019 00:48:15
os: Linux-4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018
nodename: vasanthahr-VirtualBox
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /home/vasanthahr/code/python/modaka/live_video_process/live_video_proc_v1
detected binary path: /home/vasanthahr/code/python/modaka/env_package_m1/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 43789
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on 127.0.0.1:5000 fd 4
uwsgi socket 0 bound to TCP address 127.0.0.1:38303 (port auto-assigned) fd 3
Python version: 3.5.6 (default, Jan 23 2019, 15:21:07)  [GCC 7.3.0]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x55c7ff804460
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145840 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 7 seconds on interpreter 0x55c7ff804460 pid: 30025 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 30025)
spawned uWSGI worker 1 (pid: 30031, cores: 1)
spawned uWSGI http 1 (pid: 30032)

I could see connection (socket.emit from client, not getting respose from "from_flask"), but its too slow in receiving frames: (After few seconds client side scoket not able to reach server side socket connection and getting reconnecting).

spawned uWSGI master process (pid: 30025)
spawned uWSGI worker 1 (pid: 30031, cores: 1)
spawned uWSGI http 1 (pid: 30032)
[pid: 30031|app: 0|req: 1/1] 127.0.0.1 () {40 vars in 697 bytes} [Fri Jan 25 13:26:55 2019] GET /socket.io/?EIO=3&transport=polling&t=MY6GxkU => generated 119 bytes in 7 msecs (HTTP/1.1 200) 4 headers in 195 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 2/2] 127.0.0.1 () {40 vars in 697 bytes} [Fri Jan 25 13:26:57 2019] GET /socket.io/?EIO=3&transport=polling&t=MY6Gynp => generated 119 bytes in 1 msecs (HTTP/1.1 200) 4 headers in 195 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 3/3] 127.0.0.1 () {44 vars in 833 bytes} [Fri Jan 25 13:26:57 2019] POST /socket.io/?EIO=3&transport=polling&t=MY6Gz1o&sid=2f38fda0a90e4b44977721595e190b51 => generated 2 bytes in 6 msecs (HTTP/1.1 200) 3 headers in 132 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 4/4] 127.0.0.1 () {40 vars in 772 bytes} [Fri Jan 25 13:26:57 2019] GET /socket.io/?EIO=3&transport=polling&t=MY6Gz1z&sid=2f38fda0a90e4b44977721595e190b51 => generated 96 bytes in 0 msecs (HTTP/1.1 200) 3 headers in 146 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 5/5] 127.0.0.1 () {50 vars in 978 bytes} [Fri Jan 25 13:26:58 2019] GET /socket.io/?EIO=3&transport=websocket&sid=2f38fda0a90e4b44977721595e190b51 => generated 14 bytes in 84584 msecs (HTTP/1.1 101) 4 headers in 179 bytes (0 switches on core 0)
[pid: 30031|app: 0|req: 6/6] 127.0.0.1 () {40 vars in 772 bytes} [Fri Jan 25 13:28:22 2019] GET /socket.io/?EIO=3&transport=polling&t=MY6GzBF&sid=2f38fda0a90e4b44977721595e190b51 => generated 11 bytes in 0 msecs (HTTP/1.1 400) 3 headers in 141 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 7/7] 127.0.0.1 () {40 vars in 772 bytes} [Fri Jan 25 13:28:22 2019] GET /socket.io/?EIO=3&transport=polling&t=MY6GzBF&sid=2f38fda0a90e4b44977721595e190b51 => generated 11 bytes in 0 msecs (HTTP/1.1 400) 3 headers in 141 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 8/8] 127.0.0.1 () {40 vars in 698 bytes} [Fri Jan 25 13:28:22 2019] GET /socket.io/?EIO=3&transport=polling&t=MY6HHoR => generated 119 bytes in 1 msecs (HTTP/1.1 200) 4 headers in 195 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 9/9] 127.0.0.1 () {44 vars in 833 bytes} [Fri Jan 25 13:28:22 2019] POST /socket.io/?EIO=3&transport=polling&t=MY6HHpf&sid=a1d6187bac0e440fadd355bada0dfeff => generated 2 bytes in 1 msecs (HTTP/1.1 200) 3 headers in 132 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 10/10] 127.0.0.1 () {40 vars in 772 bytes} [Fri Jan 25 13:28:23 2019] GET /socket.io/?EIO=3&transport=polling&t=MY6HHpq&sid=a1d6187bac0e440fadd355bada0dfeff => generated 96 bytes in 0 msecs (HTTP/1.1 200) 3 headers in 146 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 11/11] 127.0.0.1 () {44 vars in 836 bytes} [Fri Jan 25 13:28:23 2019] POST /socket.io/?EIO=3&transport=polling&t=MY6HHsH&sid=a1d6187bac0e440fadd355bada0dfeff => generated 2 bytes in 1 msecs (HTTP/1.1 200) 3 headers in 132 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 12/12] 127.0.0.1 () {40 vars in 772 bytes} [Fri Jan 25 13:28:23 2019] GET /socket.io/?EIO=3&transport=polling&t=MY6HHsb&sid=a1d6187bac0e440fadd355bada0dfeff => generated 11 bytes in 60013 msecs (HTTP/1.1 400) 3 headers in 141 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 13/13] 127.0.0.1 () {50 vars in 978 bytes} [Fri Jan 25 13:29:23 2019] GET /socket.io/?EIO=3&transport=websocket&sid=a1d6187bac0e440fadd355bada0dfeff => generated 11 bytes in 0 msecs (HTTP/1.1 400) 3 headers in 141 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 14/14] 127.0.0.1 () {44 vars in 836 bytes} [Fri Jan 25 13:29:23 2019] POST /socket.io/?EIO=3&transport=polling&t=MY6HHtU&sid=a1d6187bac0e440fadd355bada0dfeff => generated 11 bytes in 0 msecs (HTTP/1.1 400) 3 headers in 141 bytes (1 switches on core 0)
[uwsgi-http key: 127.0.0.1:5000 client_addr: 127.0.0.1 client_port: 21723] hr_instance_read(): Connection reset by peer [plugins/http/http.c line 647]
[pid: 30031|app: 0|req: 15/15] 127.0.0.1 () {44 vars in 832 bytes} [Fri Jan 25 13:29:23 2019] POST /socket.io/?EIO=3&transport=polling&t=MY6HWXD&sid=a1d6187bac0e440fadd355bada0dfeff => generated 11 bytes in 0 msecs (HTTP/1.1 400) 3 headers in 141 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 16/16] 127.0.0.1 () {40 vars in 698 bytes} [Fri Jan 25 13:29:23 2019] GET /socket.io/?EIO=3&transport=polling&t=MY6HWXQ => generated 119 bytes in 0 msecs (HTTP/1.1 200) 4 headers in 195 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 17/17] 127.0.0.1 () {44 vars in 833 bytes} [Fri Jan 25 13:29:23 2019] POST /socket.io/?EIO=3&transport=polling&t=MY6HWYY&sid=e00e1a3eab2c4c61a5a4a9814cd400bb => generated 2 bytes in 2 msecs (HTTP/1.1 200) 3 headers in 132 bytes (1 switches on core 0)
[pid: 30031|app: 0|req: 18/18] 127.0.0.1 () {40 vars in 772 bytes} [Fri Jan 25 13:29:23 2019] GET /socket.io/?EIO=3&transport=polling&t=MY6HWYm&sid=e00e1a3eab2c4c61a5a4a9814cd400bb => generated 96 bytes in 0 msecs (HTTP/1.1 200) 3 headers in 146 bytes (1 switches on core 0)

=====================================================

Step3

Just ran without gevent and wihtout UWSGI locally. (I did setup as async_mode="None").
(then server receives the frames and response the frames back to client, that means clients emits frames with very high performance/speed and server receives in the same speed, but server sends response with very slow or less performance, means response is having video lagging). (sometimes disconnects and reconnects).

Started Server :

python server.py

started logs:

WebSocket transport not available. Install eventlet or gevent and gevent-websocket for improved performance.
 * Serving Flask app "server" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [25/Jan/2019 13:49:14] "GET /socket.io/?EIO=3&transport=polling&t=MY6M3N0 HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2019 13:49:14] "POST /socket.io/?EIO=3&transport=polling&t=MY6M3O2&sid=53d50f91d3b040dfb9aecd7e30185ee6 HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2019 13:49:14] "GET /socket.io/?EIO=3&transport=polling&t=MY6M3Oy&sid=53d50f91d3b040dfb9aecd7e30185ee6 HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2019 13:49:14] "POST /socket.io/?EIO=3&transport=polling&t=MY6M3Rz&sid=53d50f91d3b040dfb9aecd7e30185ee6 HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2019 13:49:33] "POST /socket.io/?EIO=3&transport=polling&t=MY6M7-L&sid=53d50f91d3b040dfb9aecd7e30185ee6 HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2019 13:49:33] "GET /socket.io/?EIO=3&transport=polling&t=MY6M7-Y HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2019 13:49:33] "GET /socket.io/?EIO=3&transport=polling&t=MY6M6RW&sid=53d50f91d3b040dfb9aecd7e30185ee6 HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2019 13:49:36] "GET /socket.io/?EIO=3&transport=polling&t=MY6M8bm HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2019 13:49:37] "POST /socket.io/?EIO=3&transport=polling&t=MY6M8tb&sid=a3621fd56cef4055955ae01bf448541e HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2019 13:49:37] "GET /socket.io/?EIO=3&transport=polling&t=MY6M8tp&sid=a3621fd56cef4055955ae01bf448541e HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2019 13:49:39] "POST /socket.io/?EIO=3&transport=polling&t=MY6M9Nf&sid=a3621fd56cef4055955ae01bf448541e HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2019 13:49:39] "POST /socket.io/?EIO=3&transport=polling&t=MY6M9Q3&sid=a3621fd56cef4055955ae01bf448541e HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2019 13:49:39] "POST /socket.io/?EIO=3&transport=polling&t=MY6M9QZ&sid=a3621fd56cef4055955ae01bf448541e HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2019 13:49:39] "POST /socket.io/?EIO=3&transport=polling&t=MY6M9RB&sid=a3621fd56cef4055955ae01bf448541e HTTP/1.1" 200 -

[dxue2012]

It looks like you got a response here to your threading question: miguelgrinberg/Flask-SocketIO#885

Regarding the video lag you noticed in your 3rd step: while it's not desirable, I did notice it as well and didn't bother figuring out how to fix it. You'll see it in the demo site as well. That just means that it's not your fault in how you're running your code but rather likely due to a bug in my original implementation.