Torchvision running into `Cannot allocate memory` error

🐛 Describe the bug

Running on RTX 6000 Ada with CUDA 12.6 Works perfectly fine on another instance of Tesla L4 with CUDA 12.8 FFMPEG is installed on both instances including all dependencies for video decoder and lerobot.

Issue persists after reducing dataloader_num_workers = 2

File "/root/miniconda3/envs/gr00t/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/root/miniconda3/envs/gr00t/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/root/Isaac-GR00T/gr00t/data/dataset.py", line 488, in getitem return self.transforms(self.get_step_data(trajectory_id, base_index)) File "/root/Isaac-GR00T/gr00t/data/dataset.py", line 522, in get_step_data data[key] = self.get_data_by_modality(trajectory_id, modality, key, base_index) File "/root/Isaac-GR00T/gr00t/data/dataset.py", line 782, in get_data_by_modality return self.get_video(trajectory_id, key, base_index) File "/root/Isaac-GR00T/gr00t/data/dataset.py", line 652, in get_video return get_frames_by_timestamps( File "/root/Isaac-GR00T/gr00t/utils/video.py", line 112, in get_frames_by_timestamps for frame in reader: File "/root/miniconda3/envs/gr00t/lib/python3.10/site-packages/torchvision/io/video_reader.py", line 200, in next frame = next(self._c) File "av/container/input.pyx", line 208, in decode File "av/packet.pyx", line 80, in av.packet.Packet.decode File "av/video/stream.pyx", line 41, in av.video.stream.VideoStream.decode File "av/video/stream.pyx", line 50, in av.video.stream.VideoStream.decode File "av/codec/context.pyx", line 462, in av.codec.context.CodecContext.decode File "av/codec/context.pyx", line 238, in av.codec.context.CodecContext.open File "av/error.pyx", line 326, in av.error.err_check av.error.MemoryError: [Errno 12] Cannot allocate memory

Versions

termcolor==3.0.1 tianshou==0.5.1 tifffile==2025.3.30 timm==1.0.14 tokenizers==0.20.3 toml==0.10.2 tomli==2.2.1 torch==2.5.1 TorchCodec==0.2.1 torchvision==0.20.1 tqdm==4.67.1 transformers==4.45.2 triton==3.1.0 typeguard==4.4.2 typing-inspect==0.9.0 typing_extensions==4.12.2 tyro==0.9.17 tzdata==2025.2 urllib3==2.4.0 wandb==0.18.0 wcwidth==0.2.13 Werkzeug==3.1.3 wrapt==1.14.1 xxhash==3.5.0 yacs==0.1.8 yarl==1.19.0 zarr==2.18.3 zipp==3.21.0 zmq==0.0.0