mjviewer producing corrupted camera frames (black, switched, pixel errors) when other unrelated code is run #779
Description
System Info
Ubuntu 24.04.1
Robosuite 1.5.1
Mujoco 3.4.0
Pip list output:
Package Version
------------------------- -----------
absl-py 2.3.1
anyio 4.12.0
argon2-cffi 25.1.0
argon2-cffi-bindings 25.1.0
arrow 1.4.0
astroid 4.0.2
asttokens 3.0.1
async-lru 2.0.5
attrs 25.4.0
babel 2.17.0
backports.zstd 1.2.0
beautifulsoup4 4.14.3
bleach 6.3.0
Brotli 1.2.0
cached-property 1.5.2
certifi 2025.11.12
cffi 2.0.0
charset-normalizer 3.4.4
colorama 0.4.6
comm 0.2.3
contourpy 1.3.2
cycler 0.12.1
daqp 0.7.2
debugpy 1.8.17
decorator 5.2.1
defusedxml 0.7.1
dill 0.4.0
etils 1.13.0
evdev 1.9.2
exceptiongroup 1.3.1
executing 2.2.1
fastjsonschema 2.21.2
fonttools 4.61.0
fqdn 1.5.1
fsspec 2025.12.0
glfw 2.10.0
h11 0.16.0
h2 4.3.0
hpack 4.1.0
httpcore 1.0.9
httpx 0.28.1
hyperframe 6.1.0
idna 3.11
importlib_metadata 8.7.0
importlib_resources 6.5.2
iniconfig 2.3.0
ipykernel 7.1.0
ipython 8.37.0
ipywidgets 8.1.8
isoduration 20.11.0
isort 7.0.0
jedi 0.19.2
Jinja2 3.1.6
json5 0.12.1
jsonpointer 3.0.0
jsonschema 4.25.1
jsonschema-specifications 2025.9.1
jupyter 1.1.1
jupyter_client 8.6.3
jupyter-console 6.6.3
jupyter_core 5.9.1
jupyter-events 0.12.0
jupyter-lsp 2.3.0
jupyter_server 2.17.0
jupyter_server_terminals 0.5.3
jupyterlab 4.5.0
jupyterlab_pygments 0.3.0
jupyterlab_server 2.28.0
jupyterlab_widgets 3.0.16
kiwisolver 1.4.9
lark 1.3.1
llvmlite 0.46.0
MarkupSafe 3.0.3
matplotlib 3.10.8
matplotlib-inline 0.2.1
mccabe 0.7.0
mink 0.0.13
mistune 3.1.4
mujoco 3.4.0
munkres 1.1.4
nbclient 0.10.2
nbconvert 7.16.6
nbformat 5.10.4
nest_asyncio 1.6.0
notebook 7.5.0
notebook_shim 0.2.4
numba 0.63.0
numpy 2.2.6
opencv-python 4.12.0.88
overrides 7.7.0
packaging 25.0
pandocfilters 1.5.0
parso 0.8.5
pexpect 4.9.0
pickleshare 0.7.5
pillow 12.0.0
pip 25.3
platformdirs 4.5.1
pluggy 1.6.0
prometheus_client 0.23.1
prompt_toolkit 3.0.52
psutil 7.1.3
ptyprocess 0.7.0
pure_eval 0.2.3
pycparser 2.22
Pygments 2.19.2
pylint 4.0.4
pynput 1.8.1
PyOpenGL 3.1.10
pyparsing 3.2.5
pyrealsense2 2.56.5.9235
PySide6 6.9.3
PySocks 1.7.1
pytest 9.0.2
python-dateutil 2.9.0.post0
python-json-logger 2.0.7
python-xlib 0.33
pytz 2025.2
PyYAML 6.0.3
pyzmq 27.1.0
qpsolvers 4.8.2
referencing 0.37.0
requests 2.32.5
rfc3339_validator 0.1.4
rfc3986-validator 0.1.1
rfc3987-syntax 1.1.0
robosuite 1.5.1
rpds-py 0.30.0
scipy 1.15.3
Send2Trash 1.8.3
setuptools 80.9.0
shiboken6 6.9.3
six 1.17.0
sniffio 1.3.1
soupsieve 2.8
stack_data 0.6.3
termcolor 3.2.0
terminado 0.18.1
tinycss2 1.5.1
tomli 2.3.0
tomlkit 0.13.3
tornado 6.5.2
tqdm 4.67.1
traitlets 5.14.3
typing_extensions 4.15.0
typing_utils 0.1.0
tzdata 2025.2
unicodedata2 17.0.0
uri-template 1.3.0
urllib3 2.6.1
wcwidth 0.2.14
webcolors 25.10.0
webencodings 0.5.1
websocket-client 1.9.0
wheel 0.45.1
widgetsnbextension 4.0.15
zipp 3.23.0Information
When running with the mjviewer as renderer/viewer, images from additional non-viewer cameras get corrupted. These corruptions manifest as:
- switching of the cameras (the image for one camera is at the location where the other should be),
- general corruptions like color errors (switching rgb/bgr, colored lines in the images, pixel errors)
- black images
I believe this is due to some corruption of the frame-buffers. My best guess is that the OpenGL context is not correctly protected and some kind of corruption / overwriting from other threads occurs.
I have created a minimal example where the bug can reliably be reproduced using different methods. I know that there were similar issues in the past, but I hope that having a reliable way to reproduce the bug might help to find its source. This is probably also related to #108 and the comment in base.py: # TODO(yukez): investigate black screen of death
You can see examples of the produced buggy camera images in:
Video_Corruption.webm
and
Video_Flips.webm
Reproduction
I have created a minimal example where the bug turns up.
The required environment can be create with
conda create -n "testenv" python=3.10 ipython ipykernel pylint jupyter pip numpy glfw matplotlib -c conda-forge -y
conda activate testenv
pip install mujoco==3.4.0 robosuite==1.5.1 pyrealsense2
Then run the example script : robosuite_camera_corruption_bug_search.py
python robosuite_camera_corruption_bug_search.py
The bug disappears if either:
- pyrealsense2 is not imported (comment out import pyrealsense2 as rs)
- env.reset() is not called (comment out env.reset())
- if both of these are still commented out, the bug appears again, if a second unused environment is created
The bug also disappears if:
- renderer="mujoco" is used instead of "mjviewer"
- hard_reset=False is used
Since an unrelated and unused import also influences the bug, something really strange and annoying is going on here. Again my hypothesis is, that some framebuffers are not correctly protected and might be overwritten by some other code running.
MISC:
-
i was able to bisect this problem to first occur after commit 3257406, which changes the default viewer from mujoco to mjviewer
-
using different OpenGL backends did not influence the issue
MUJOCO_GL=osmesa PYOPENGL_PLATFORM=osmesa python robosuite_camera_corruption_bug_search.py
MUJOCO_GL=egl PYOPENGL_PLATFORM=egl python robosuite_camera_corruption_bug_search.py -
i tried it with different versions of mujoco, this does not change the behaviour
-
i tried with different versions of robosuite (also with robosuite==1.5.0 and the newest current commit) and this also did not change the issue, except for when using older versions ~1.4.0, where the renderer="mujoco" is used.
Expected behavior
The camera images for offscreen cameras should be correctly shown.