FFmpegFrameGrabber only grabs keyframes, skips frames in between

I'm running a basic grab on OSX. It's a Kotlin project, not on Android. Here's the code:

    private val converter = Java2DFrameConverter()
    private val frameGrabber = FFmpegFrameGrabber(mp4Path)
    private val frameRate: Double

    init {
        frameGrabber.start()
        frameRate = frameGrabber.frameRate
        println("Video has " + frameGrabber.lengthInFrames + " frames and has frame rate of " + frameRate)
    }

    fun grabFrame(frameNum: Int): BufferedImage? {
        try {
            frameGrabber.frameNumber = frameNum

            val frame = frameGrabber.grabImage()
            return converter.convert(frame)

        } catch (e: Exception) {
            e.printStackTrace()
        }

        return null
    }

It captures the same frame 6 or 7 times in a row, then repeats. When I debugged the grabber, it seems it only sends back keyframes. All frames had keyFrame = true.

The line where it may be messing up is 1276:

https://github.com/bytedeco/javacv/blob/fe484272b60ca131ee87c573550dc0a0afb17861/src/main/java/org/bytedeco/javacv/FFmpegFrameGrabber.java#L1276

When setting the frameNumber, it calls grabFrame(), and the value of this.pkt.stream_index() is not the same as this.video_st.index().

This means that it enters that condition, and not the next closure where it would have called this.frame.image = this.image_buf.

Because of this, the this.frame.image is null by the time it gets to actually grabbing the frame, i.e. not while incrementing the timestamp. It falls back to returning the most recent keyframe. That's my guess.

I've looked through all the other bugs and haven't found anyone having a similar situation.