[AG-TECH] Audio and Video Syncronization

[Will, Rodger (R.)]

I am very new to this technology, but I assumed that with vic and rat being independent programs, that sync would be a challenge.

One of our nodes employs an automatic camera/microphone system that switches vic's input to the active speaker's camera and microphone. I would assume that this effect would be akin to high-motion. 

I will watch the stats today, and pass along the information.

I agree that someone urgently needs to solve this problem in software.

-----Original Message-----
From: Markus Buchhorn [mailto:Markus.Buchhorn at anu.edu.au]
Sent: Tuesday, June 26, 2001 11:22 PM
To: ag-tech at mcs.anl.gov
Subject: Re: [AG-TECH] Audio and Video Syncronization


At 08:21 PM 26/06/2001 -0500, Marty Hoag wrote:
>However, I'm not sure how much
>effect [ntp] will have on the actual audio/video syncronization.  I've had a
>feeling that rat or vic can drift, especially if there is packet loss or other
>weirdness but others would know more about why that is.  I'd also check things
>like the rat settings to make sure everyone is at 16khz audio and look at the
>playout and decode stats for remote sites


I'd always understood/assumed (I may be wrong) that the sync problem was 
due to the apps being totally independent and sizing their own playout 
buffers (or delays) independently. Hence if jitter and/or loss get larger 
for one stream over another (packet size issues?) the playout buffer is 
"stretched" (or delay increased) and sync goes out the window. I think rat 
and vic try to keep this to a reasonably constant level, and drop the 
playout delay whenever they can - but they are doing it independently.

Audio can catch up during silence, video during "static" periods - so it 
can also be a function of the content

So Marty's suggestion to watch the playout delay stats is spot on. You need 
to watch both audio and video and how they relate to each other.

"Professional"/commercial software does audio and video in the same app, or 
tighly linked apps, and I'd understood it to be (partly) for that reason.

Has anybody looked at MBUS hooks for vic and rat? Synch is one of the areas 
that it is meant to address.
http://www.ietf.org/internet-drafts/draft-ietf-mmusic-mbus-transport-06.txt

Perhaps one dirty option for now is to force vic/rat to drop late packets 
in preference to variable buffers - set the playout delay to a constant...

Just my two bits worth. However, I can't stress enough how important it is 
to ultimately get this right - we have to put up with enough crap from the 
ISDN VC guys without giving them more ammunition :-) (not that ISDN is 
always fully synched either...). There's no reason why we couldn't get the 
synch perfect. Bundled MPEG streams would be another way we could do it, 
with a little effort...

Cheers,
         Markus

P.S. Lip synch can be lost at the *source* if the audio and video encoders 
aren't time synched to each other (and ditto the decoders at the receiving 
end). So it is worth doing ntp or similar. Tying it to a specific ntp 
server on the Net I don't believe is necessary though, a local ntp server 
should suffice.

P.P.S. I'm not sure why Rodger leaving the room should make it better 
(presuming it is not a personal reflection :-) ). If the encoding and 
decoding is on the same machine it may be that it reduces the pressure on 
the CPU on the encoding side or NIC on the machine. If they are on 
different machines it may be reflecting network problems somewhere upstream 
(sending less/smaller packets if the video is static and the audio is 
quiet). You could try watching a one-way stream, with nothing sent in the 
other direction and see how it behaves, and then turn on your transmission.

Markus Buchhorn, Faculty of Engineering and IT,          | Ph: +61 2 61258810
email: markus.buchhorn at anu.edu.au, mail: CSIT Bldg #108  |Fax: +61 2 61259805
Australian National University, Canberra 0200, Australia |Mobile: 0417 281429