How do i reduce lag for video transmission

[ketanrd01]

I am writing a code for video transmission via internet.
The coding has been done in Python and libraries from pygame (tried using openCV but lag increased)

pxarray = pygame.PixelArray(img)
packet = bytearray(640*480*3)
k=0
for i in range(640) :
for j in range(480) :
packet[k] = pxarray[i,j] & 255
k=k+1
packet[k] = (pxarray[i,j] >> 8) & 255
k=k+1
packet[k] = (pxarray[i,j] >> 16) & 255
k=k+1
how is it possible to optimize this code??...

I do not intend to further reduce the resolution n my webcam captures images at 30fps.
Lag is of around 350-450msec.....

 

[Gryd3]

Determine the source of lag...
A while back I had some major lag playing games on my PC from a TV Tuner card. The video software I was using introduced a delay because of how it used it's buffer. Switching video software to something else helped.

This could be the code, the network, or the client. How are you testing this?

 

[ketanrd01]

i have a code each for client n server.
i am starting a timer, sending image to the client. the client sends it back to the server. this total time divided by 2.

now the above mentioned is a part of server to convert the 640*480 pixels into an array before transmitting....

 

[Gryd3]

i have a code each for client n server.
i am starting a timer, sending image to the client. the client sends it back to the server. this total time divided by 2.

now the above mentioned is a part of server to convert the 640*480 pixels into an array before transmitting....

Crunch some numbers. It looks like you are streaming uncompressed video.

Your server stores the entire bitmap before transmitting. Your client then waits for the entire image to be received before forwarding it back to the server for your test?

Here are some things to look at / into.
Video compression. Simple methods will not use much CPU cycles, and will save bandwidth.
256 - color 640 * 480 will require 300KB per frame for a total of 9MBps
It looks like you are storing a byte per color per pixel, and not a byte per pixel, so your requirements will triple.
Different transmission method. Instead of storing the entire image, send it as soon as you have enough to transmit a full packet.
Use UDP instead of TCP. TCP has some added features to ensure the data has delivered properly and adds extra latency and bandwidth.

 

[(*steve*)]

Different transmission method. Instead of storing the entire image, send it as soon as you have enough to transmit a full packet.
Use UDP instead of TCP. TCP has some added features to ensure the data has delivered properly and adds extra latency and bandwidth.

Exactly what I was going to say. (well almost)

Let's say you need to divide the resolution by 4, all you need to do is wait until the second pixel of the second line has arrived and you can start streaming out the averaged pixels. Your minimum latency is then a couple of scan lines plus the time required to fill a packet. Clearly, smaller packets mean lower latency.

However, another approach may be even better. If your problem is that the sound is getting out of sync, a better proposition is to tag the sound packets with a time signal which is also keyed to the video frame. Then the receiving software can buffer the sound until the video frame is present. (A similar effect can be achieved by interleaving sound and video, but at some point you still need to sync the two.

 

[Gryd3]

I still think that for use over the internet, the volume of data needs to be reduced... Reducing the color depth, or compressing the image similar to a jpg. This is harder to accomplish but will provide better results.
What was the result with OpenCV?

 

[ketanrd01]

Thanks steve and Gryd3....OpenCV: delay of 350msec and pygame: delay of around 300msec.
the problem with compressing is that i aint using a computer but a raspberry pi. Compressing a .bmp to .jpg will itself require some time. I was thinking of applying what you n steve mentioned as well as use the sony ps webcam (highest fps of around 120) n reduce the resolution a bit...

Thanks again both of you...

 

[Gryd3]

Keep us posted, and remember that your lag could be from your hardware, your code, or your network connection. Make sure the network is capable of reliably sending the volume of data you require. There are test that can be performed to determine the maximum amount of data that can be transmitted and received on a network. There will always be some delay as well if the code on the server or client is dealing with 'bulk' date of one or more complete frames. Sending pieces will help with this, but could result in something called 'page tearing' where the top half of the picture updates before the bottom half. Aside from using compression or reducing the resolution to further reduce the data requirements, I cant think of another solution.

Best of luck!

 

[ketanrd01]

I finally did it guys!!!!
Streamed a live video within a few milli seconds!!!(around 100msec)..
Thanks again for your help and support...
The things that i implemented:
1.tried sending image instead of pixel values
2.sent .jpg image(compressed) instead of .bmp and it works in RPi as well
3.since there was loss of segments in UDP, instead of sending a compressed image as it is, i divided it in parts n then sent it.

Further changes i would be implementing:
1.coding in C instead of Python
2.use a better webcam

I am grateful to Gryd3 and Steve, thanks guys....

 

[Gryd3]

Glad to hear it's getting much faster for you
100ms sounds much better than the 350 - 450 you had before

Any further speed improvements will be trickier to implement and will require a more in-depth look at the network and the hardware.

Have fun, and thanks for hopping on Electronics Point!

 

[ketanrd01]

Yep, our eyes cant differentiate delay if motion under 1/16th of a second ie 66.67msec so if can reach that number with pathetic internet speed in our country, it will be hell of an achievement...

 

[Gryd3]

Yep, our eyes cant differentiate delay if motion under 1/16th of a second ie 66.67msec so if can reach that number with pathetic internet speed in our country, it will be hell of an achievement...

Two more things I'd like to bring up
If your streaming video on the internet, you usually do not know what the total delay is. This would only apply if the viewer was controlling the stream somehow (like a robot), then a 400-500ms delay would be quite irritating. It's that or viewing the the original and stream at the same time

As you test your setup, you can always play with different jpg compression rates. Picture quality goes down as you compress it more but so does the data requirements. Even for experimentation, finding the lowest acceptable compression may allow you to transmit the entire image with udp instead of peices and still operate with a high degree of packet loss.

Again, something else you can play with. UDP vs TCP. TCP will create more bandwidth overhead and is very seldom used for streaming of any kind, but it does have a sort of guarantee for delivery. It will ensure that the packets are delivered, in order. If a packet is lost, it will request it to be resent. If your operating on a poor network that causes a lot of packet loss, this may be a requirement if you are having issues getting your stream to operate... It will slow things down though, By how much, I cant say.