NVIDIA/DALI

Question about video loading speed in two different pipelines

Open

#4,624 opened on Jan 30, 2023

View on GitHub
 (3 comments) (0 reactions) (1 assignee)C++ (670 forks)auto 404
Videohelp wanted

Repository metrics

Stars
 (5,722 stars)
PR merge metrics
 (PR metrics pending)

Description

Hi @JanuszL and the team, hope you're doing well (CC'd @themattinthehatt)

We are using two different video reading pipelines in our lightning-pose repository.

  1. The first scans a video and has sequence_length=64 and step=64 -- and is working very fast.

  2. The second was tailored to our temporal context network, which takes in five video frames at a time and predicts just the third (middle) frame. it uses sequence_length=5, step=1 and batch_size=16.

Both are running on device="gpu". Changing the num_threads=4 argument doesn't make any difference, we do not see any meaningful cpu utilization with or without it (why? not sure, see top -i). Pipeline 1 does use 6%-8% GPU, Pipeline 2 uses approximately 0% GPU (as seen in watch -n 0.5 nvidia-smi.

I prepared a notebook and demo video for you. To reproduce

Installation

git clone https://github.com/danbider/lightning-pose.git
cd lightning-pose
python -m pip install -r requirements.txt
git checkout loss-landscapes

The reproducible notebook is here.

Question -- should CPU usage be low in general? Why is GPU usage low for pipeline 2? How could I accelerate it?

Your help is appreciated, as always

Contributor guide

Research direction
Profile pipeline 2 using DALI's built in profiling tools (e.g., `pipeline.profile()`). Check if the CPU GPU bottleneck is due to small batch size or inefficient decoder settings. Consider increasing `num threads` and `batch size` and using `enable memory stats` to monitor memory usage. Also verify that the video decoder is using GPU (e.g., nvidia decoder) and not falling back to CPU. Experiment with `prefetch queue depth` and `cpu deque size` to improve throughput.
Tech stack
python
Domain
datamachine learningperformance
Issue type
Performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonDALIGPU computing
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65