What should I use the Unity plugin for?

[Artur-Galstyan]

Intro

Hi!

I am a beginner in RL and I want to use MuJoCo for creating tutorials on my YouTube channel.

My setup

Not relevant for my question.

My question

There is the Unity plugin and I understand how to use it but it's not so clear to me what I should use it for or what the intended use case is.

Is the Unity plugin mainly there to more easily edit MJCF files?

I'm curious what you guys use it for :)

Minimal model and/or code that explain my question

No response

Confirmations

• I searched the latest documentation thoroughly before posting.
• I searched previous Issues and Discussions, I am certain this has not been raised before.

[Balint-H]

Hello!

Yes, editing scenes is one of the use cases. In general, the Unity plugin can play the role of a much needed development environment for MuJoCo simulations. It lets you:

• visually understand scenes
• manage assets
• write reusable scripted behaviours.
• take advantage of other tools (e.g. all the animation, rendering, game logic or ML tools already implemented for Unity)
• user interfacing tools (very easy to set up different control schemes across various devices, and to set up a custom UI)
• VR with MuJoCo physics (although there are other implementations of it, Unity VR is very robust and easy to set up)
• make standalone apps targeting various platforms using MuJoCo

Here's a plug for a project of mine and @joanllobera that uses Unity: https://github.com/Balint-H/modular-agents
We train physics animation agents with learning environments made as a Unity app. We extensively use the Unity animation pipeline for reference motion, and being able to visualize rewards/observations/etc easily and interactively was a huge time saver. We do sacrifice performance compared to a C++, or a parallelized JAX implementation, but it is still faster than logic heavy pure python environments (as C# can be used for the logic).

It doesn't make sense for every use case, and it is still not fully mature (we discover missing/broken features every now and then) but it can make a lot of projects less painful/benefit from "free" polish. Let me know if you have other questions about it! Do you have a potential use case in mind?