Minimize delay - real time video chat application RRS feed

  • Question

  • Hi,

    I'm trying to capture video from camera (webcam), compress it using a codec, and send this data to remote server for further resend.
    I've managed to capture and encode the live source from webcam, but there is a big delay between the time that things get captured and the time they are decoded by the receiver.
    Currently this is about 5 seconds. This is far too long, since this is a video-chat application.

    I'm using DirectShow managed API wrapped by WindowsMediaLib. Compression parameters are defined in .prx file.
    In this file i can define bufferwindow - which i believe specifies length of buffer the client will use upon decoding and playing the content.

    I tried to set this to very low value (1000 [ms]) but the client (windows media player) ignores it, and plays the file with longer (5000 [ms]) delay. I also wrote a dedicated client in silverlight, where I can define explicitely the length of buffer on the client, and again, this did not give expected result. When I increased the value (BufferingTime) in the code, the delay was increased, but when I decreased it to 1000ms, the video still was played with ~5 seconds delay.

    So, I think that it is the codec itself that causes the delay. Is it possible to change the way it works, so that it would produce compressed output with smaller delay ?

    1. If it is not possible to reduce the latency with this codec (WMV9), is there any other one in the system (windows 7) that would be better in terms of latency (encoding delay) ? If yes, is it possible simply to modify prx file to select it for encoding ?

    2. Currently I am using NetworkSink to expose encoded content. Maybe it is this layer that causes this 5 seconds delay ? 

    Thanks in advance,

    Wednesday, June 6, 2012 3:54 PM

All replies

  • I too am seeing a rather large delay when using compression.  Although I am seeing more like 1.5-2 seconds of lag between reality and the display but only when a "tee" is inserted into the stream.  My application has a control feedback component to it so any latency is extremely noticeable by the user.  Any suggestions for a fix?
    Monday, September 24, 2012 6:05 PM
  • Latencies you typically see on the encoding side are (1) buffer size on the audio capture leg, which you need to make smaller to reach lower latencies, and (2) video compression codec latency, which comes from codec behavior when it keeps a few video frames buffered to be able to better control bitrate and do temporal compression.


    Saturday, September 29, 2012 8:18 PM
  • I now have several custom filters that I have implemented and in the most recent one (processing the video in the filter without audio or compression in the DS graph) there is up to a 10 second delay.  This delay seems proportional to the complexity of the filter.  In other words if I do simple brightness modification or gamma correction, the latency is only like 500ms but with convolution filters and statistical analysis of the video the latency increases and the refresh rate (frame rate) drops.

    For the application I am working on a poor refresh rate is more acceptable than a huge latency.  I have seen many forums suggest to flush the stream or to set the presentation time for the frames but haven't found any concrete examples on exactly what is needed to be done to accomplish this.  All my current attempts seem to have no affect.

    Tuesday, October 9, 2012 9:04 PM
  • Your description here suggest that your own processing is slow, and there is no much difference between rate and latency - it is just processing takes too much time per frame. Most filters process data synchronously and the overhead is basically the processing time. It is only advanced filters create internal queues, split processing between threads etc.


    Tuesday, October 9, 2012 9:29 PM
  • I am basically looking for any queues to go away and to simply process the latest video buffer collected.  I am processing frames at 4-10 fps so theoretically I should have not much more than between 100ms and 250ms latency (+change).  Certainly not between 4000 and 10,000ms latency.


    With a HD signal (1920x1080x24bit) the expected memory use should be 6220800 bytes per frame (~6MB), but when I run this filter the memory use is 245MB (roughly 40x the expected amount).  The 6,220,800 bytes amount is the value that is returned via the DecideBufferSize and 1 is the value set for the cBuffers parameter.

    This leads me to believe that there is a pipeline of frames getting buffered as they are received and as my filter gets more compute-intensive this pipeline is likely contributing to the perceived latency.

    Ideally this filter's memory consumption should be closer to 12MB for a double buffered solution.

    Also the latency seems to be very small comparatively (~1/4 second) for the first 6-10 seconds after starting the graph but noticeably increases at this point and stays this way from that point on. 

    • Edited by Kraz Wednesday, October 10, 2012 5:26 PM
    Tuesday, October 9, 2012 9:49 PM