Synchronization between multiple streamed audio RTP and low latency.
#1,091 opened on Nov 20, 2018
Repository metrics
- Stars
- (6,985 stars)
- PR merge metrics
- (No merged PRs in 30d)
Description
Hello, everyone I'm using JavaCV for almost 5 months I was able to implement a basic application that receives multiple RTSP streams from Android phones as shown in this image. 
My implementation creates a thread for each device and instantiate a FFmpegFrameGrabber object with the RTSP address and port to the device stream. The delay is noticeable and varies between 0.8 ~ 1.2 seconds approximately.
So I have two questions. It is possible to achieve real-time stream or reduce the delay to 0.2 seconds? I can achieve latency (between 0.2 to 0.3 seconds) when I run this command:
ffplay -fflags nobuffer -flags low_delay -framedrop \
-strict experimental -rtsp_transport tcp rtsp://<host>:<port>
For example the AudioPlayer has this code:
public AudioPlayer(String source, GrabberListener grabberListener)
{
if (grabberListener == null) return;
if (source.isEmpty()) return;
Thread playThread = new Thread(() -> {
try {
FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(source);
grabber.setOption("rtsp_transport", "tcp");
grabber.start();
playing = true;
LogHelper.e(TAG, "audioPlaying(on)");
if (grabber.getSampleRate() > 0 && grabber.getAudioChannels() > 0) {
AudioFormat audioFormat = new AudioFormat(grabber.getSampleRate(), 16, grabber.getAudioChannels(), true, true);
DataLine.Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
soundLine = (SourceDataLine) AudioSystem.getLine(info);
soundLine.open(audioFormat);
soundLine.start();
gainControl = (FloatControl) soundLine.getControl(FloatControl.Type.MASTER_GAIN);
grabberListener.onGainControl(gainControl);
}
ExecutorService executor = Executors.newSingleThreadExecutor();
while (running.get()) {
Frame frame = grabber.grab();
if (frame == null) break;
if (frame.samples != null) {
ShortBuffer channelSamplesFloatBuffer = (ShortBuffer) frame.samples[0];
channelSamplesFloatBuffer.rewind();
ByteBuffer outBuffer = ByteBuffer.allocate(channelSamplesFloatBuffer.capacity() * 2);
float[] samples = new float[channelSamplesFloatBuffer.capacity()];
float lastPeak = 0f;
for (int i = 0; i < channelSamplesFloatBuffer.capacity(); i++) {
short val = channelSamplesFloatBuffer.get(i);
outBuffer.putShort(val);
}
// Convert bytes to samples here
for (int i = 0, s = 0; i < channelSamplesFloatBuffer.capacity(); ) {
int sample = 0;
sample |= channelSamplesFloatBuffer.get(i++) & 0xFF; // (reverse these two lines
sample |= channelSamplesFloatBuffer.get(i++) << 8; // if the format is big endian)
// Normalize to range of +/-1.0f
samples[s++] = sample / 32768f;
}
float rms = 0f;
float peak = 0f;
for (float sample : samples) {
float abs = Math.abs(sample);
if (abs > peak) {
peak = abs;
}
rms += sample * sample;
}
rms = (float) Math.sqrt(rms / samples.length);
if (lastPeak > peak) {
peak = lastPeak * 0.875f;
}
lastPeak = peak;
grabberListener.onAudioSpectrum(rms, peak);
if (soundLine == null) return;
try {
executor.submit(() -> {
soundLine.write(outBuffer.array(), 0, outBuffer.capacity());
outBuffer.clear();
}).get();
} catch (InterruptedException interruptedException) {
Thread.currentThread().interrupt();
}
}
}
executor.shutdownNow();
executor.awaitTermination(1, SECONDS);
playing = false;
LogHelper.e(TAG, "audioPlaying(off)");
if (soundLine != null) {
soundLine.stop();
}
grabber.stop();
grabber.release();
grabberListener.onStop();
} catch (Exception e) {
LogHelper.e(TAG, e);
playing.set(false);
grabberListener.onError(new RuntimeException("Could not open input stream.", e));
}
});
playThread.start();
}
And my second question is It is possible to synchronize these RTP streams? probably using the NTP timestamp? However, the NTP timestamp is in the Sender Report RTCP Packet can I have access to that packet from the FFmpegFrameGrabber??