bytedeco/javacv

Synchronization between multiple streamed audio RTP and low latency.

Open

#1,091 opened on Nov 20, 2018

View on GitHub
 (9 comments) (0 reactions) (0 assignees)Java (1,583 forks)batch import
enhancementhelp wantedquestion

Repository metrics

Stars
 (6,985 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

Hello, everyone I'm using JavaCV for almost 5 months I was able to implement a basic application that receives multiple RTSP streams from Android phones as shown in this image. screenshot from 2018-11-20 18-06-49

My implementation creates a thread for each device and instantiate a FFmpegFrameGrabber object with the RTSP address and port to the device stream. The delay is noticeable and varies between 0.8 ~ 1.2 seconds approximately.

So I have two questions. It is possible to achieve real-time stream or reduce the delay to 0.2 seconds? I can achieve latency (between 0.2 to 0.3 seconds) when I run this command:

ffplay -fflags nobuffer -flags low_delay -framedrop \
 -strict experimental -rtsp_transport tcp rtsp://<host>:<port>

For example the AudioPlayer has this code:

public AudioPlayer(String source, GrabberListener grabberListener)
{
    if (grabberListener == null) return;
    if (source.isEmpty()) return;
    Thread playThread = new Thread(() -> {
        try {
            FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(source);
            grabber.setOption("rtsp_transport", "tcp");
            grabber.start();

            playing = true;
            LogHelper.e(TAG, "audioPlaying(on)");

            if (grabber.getSampleRate() > 0 && grabber.getAudioChannels() > 0) {
                AudioFormat audioFormat = new AudioFormat(grabber.getSampleRate(), 16, grabber.getAudioChannels(), true, true);

                DataLine.Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
                soundLine = (SourceDataLine) AudioSystem.getLine(info);
                soundLine.open(audioFormat);
                soundLine.start();

                gainControl = (FloatControl) soundLine.getControl(FloatControl.Type.MASTER_GAIN);
                grabberListener.onGainControl(gainControl);
            }

            ExecutorService executor = Executors.newSingleThreadExecutor();
            while (running.get()) {
                Frame frame = grabber.grab();
                if (frame == null) break;

                if (frame.samples != null) {
                    ShortBuffer channelSamplesFloatBuffer = (ShortBuffer) frame.samples[0];
                    channelSamplesFloatBuffer.rewind();

                    ByteBuffer outBuffer = ByteBuffer.allocate(channelSamplesFloatBuffer.capacity() * 2);
                    float[] samples = new float[channelSamplesFloatBuffer.capacity()];

                    float lastPeak = 0f;

                    for (int i = 0; i < channelSamplesFloatBuffer.capacity(); i++) {
                        short val = channelSamplesFloatBuffer.get(i);
                        outBuffer.putShort(val);
                    }

                    // Convert bytes to samples here
                    for (int i = 0, s = 0; i < channelSamplesFloatBuffer.capacity(); ) {
                        int sample = 0;

                        sample |= channelSamplesFloatBuffer.get(i++) & 0xFF; // (reverse these two lines
                        sample |= channelSamplesFloatBuffer.get(i++) << 8;   //  if the format is big endian)

                        // Normalize to range of +/-1.0f
                        samples[s++] = sample / 32768f;
                    }

                    float rms = 0f;
                    float peak = 0f;
                    for (float sample : samples) {
                        float abs = Math.abs(sample);
                        if (abs > peak) {
                            peak = abs;
                        }

                        rms += sample * sample;
                    }

                    rms = (float) Math.sqrt(rms / samples.length);

                    if (lastPeak > peak) {
                        peak = lastPeak * 0.875f;
                    }

                    lastPeak = peak;

                    grabberListener.onAudioSpectrum(rms, peak);

                    if (soundLine == null) return;
                    try {
                        executor.submit(() -> {
                            soundLine.write(outBuffer.array(), 0, outBuffer.capacity());
                            outBuffer.clear();
                        }).get();
                    } catch (InterruptedException interruptedException) {
                        Thread.currentThread().interrupt();
                    }
                }
            }

            executor.shutdownNow();
            executor.awaitTermination(1, SECONDS);

            playing = false;
            LogHelper.e(TAG, "audioPlaying(off)");

            if (soundLine != null) {
                soundLine.stop();
            }

            grabber.stop();
            grabber.release();
            grabberListener.onStop();
        } catch (Exception e) {
            LogHelper.e(TAG, e);
            playing.set(false);
            grabberListener.onError(new RuntimeException("Could not open input stream.", e));
        }
    });
    playThread.start();
}

And my second question is It is possible to synchronize these RTP streams? probably using the NTP timestamp? However, the NTP timestamp is in the Sender Report RTCP Packet can I have access to that packet from the FFmpegFrameGrabber??

Contributor guide