This PR added the benchmark for opencv. The benchmark shows that there is gap. Hope the torchcodec team can find out where the gap is from and improve the decoding performance.

Local benchmark

|  decode 10 uniform frames  |  decode 10 random frames  |  first 1 frames  |  first 10 frames  |  first 100 frames
1 threads: --------------------------------------------------------------------------------------------------------------------------------
      OpenCV            |            22.4            |            22.6           |       6.4        |        9.3        |        18.1
      TorchCodecPublic  |            39.4            |            28.1           |       9.5        |        9.8        |        26.0

Times are in milliseconds (ms).

Thanks a lot for the benchmark @jinhohwang-meta , that's very useful! I made a few comments below.

Based on how the code is converting pts to indices (using the average fps), I think a fairer comparison against torchcodec would be to benchmark torchcodec with the approximate mode (instead of exact mode). I think it corresponds to torchcodec_core_nonbatch, although I'm not 100% sure (CC @scotts )

[NicolasHug]

frames seems unused and gets redefined below

[NicolasHug]

Can you add a comment explaining why this is needed?

[scotts]

@jinhohwang-meta, can we be more descriptive of what the meaning of these backends are? Why are we checking for specific ranges of the ABI and API?

I think we'll also want assert api_pref is not None after the loop.

[NicolasHug]

Let's raise here instead of a break. This will ensure that we are not reporting results in case cap.grab() fails.

[NicolasHug]

Nit: s/numbers/indices. We use the term indices for this.

[NicolasHug]

Same here, we should raise instead of break

[NicolasHug]

Let's assert that len(frames) == n to ensure no error went undetected

Thanks a lot for the benchmark @jinhohwang-meta , that's very useful! I made a few comments below.

Based on how the code is converting pts to indices (using the average fps), I think a fairer comparison against torchcodec would be to benchmark torchcodec with the approximate mode (instead of exact mode). I think it corresponds to torchcodec_core_nonbatch, although I'm not 100% sure (CC @scotts )

The comparison we'll probably be most interested in is this one:

python benchmarks/decoders/benchmark_decoders.py --decoders opencv,torchcodec_public:seek_mode=exact,torchcodec_public:seek_mode=approximate,torchaudio --min-run-seconds 40

That is, it compares:

1. OpenCV
2. TorchCodec public API, exact seek mode
3. TorchCodec public API, approximate seek mode
4. TorchAudio

We'll also want to use some more videos - the ones that are part of the README benchmark are good to use. We generate those on the fly here:

	resolutions = ["1920x1080"]
	encodings = ["libx264"]
	patterns = ["mandelbrot"]
	fpses = [60]
	gop_sizes = [600]
	durations = [10, 120]
	pix_fmts = ["yuv420p"]
	ffmpeg_path = "ffmpeg"
	generate_videos(
	resolutions,
	encodings,
	patterns,
	fpses,
	gop_sizes,
	durations,
	pix_fmts,
	ffmpeg_path,
	videos_dir_path,
	)

That doesn't need to be a part of this PR. In this PR, we should focus on correct usage of the OpenCV API and then we can investigate the comparative performance in follow-ups. Just making a note. :)

Thanks a lot for the benchmark @jinhohwang-meta , that's very useful! I made a few comments below.

Based on how the code is converting pts to indices (using the average fps), I think a fairer comparison against torchcodec would be to benchmark torchcodec with the approximate mode (instead of exact mode). I think it corresponds to torchcodec_core_nonbatch, although I'm not 100% sure (CC @scotts )

@NicolasHug I actually did compare with approximate as well, but it showed similar results. @scotts asked me to push this for the team to follow on to find out the gap and I am very much interested in to see why as well.