[P] I open-sourced a synth framework for creating physics-simulated humanoids in Unity with MuJoCo — train them with on-device RL and interact in VR

I’ve been building a system to create physics-based humanoid characters in Unity that can learn through reinforcement learning — and you can physically interact with them in mixed reality on Quest. Today I’m open-sourcing the three packages that make it up.
What it does:

• synth-core — Take any Daz Genesis 8 or Mixamo character, run it through an editor wizard (or one-click right-click menu), and get a fully physics-simulated humanoid with MuJoCo rigid-body dynamics, mesh-based collision geometry, configurable joints, and mass distribution. Extensible to other skeleton types via an adapter pattern.
• synth-training — On-device SAC (Soft Actor-Critic) reinforcement learning using TorchSharp. No external Python server — training runs directly in Unity on Mac (Metal/MPS), Windows, or Quest (CPU). Includes prioritized experience replay, automatic entropy tuning, crash-safe state persistence, and motion reference tooling for imitation learning.
• synth-vr — Mixed reality on Meta Quest. The Synth spawns in your physical room using MRUK. Physics-based hand tracking lets you push, pull, and interact with it using your real hands. Passthrough rendering with depth occlusion and ambient light estimation.

The workflow:

1. Import a humanoid model into Unity
2. Right-click -> Create Synth (or use the full wizard)
3. Drop the prefab in a scene, press Play — it’s physics-simulated
4. Add ContinuousLearningSkill and it starts learning
5. Build for Quest and interact with it in your room

Tech stack: Unity 6, MuJoCo (via patched Unity plugin), TorchSharp (with IL2CPP bridge for Quest), Meta XR SDK

Links:

• synth-core — Physics humanoid creation
• synth-training — On-device RL training
• synth-vr — Mixed reality interaction

All Apache-2.0 licensed.
The long-term goal is autonomous virtual beings with integrated perception, memory, and reasoning — but right now the core infrastructure for creating and training physics humanoids is solid and ready for others to build on. Contributions welcome.
Happy to answer questions about the architecture, MuJoCo integration challenges, or getting TorchSharp running on IL2CPP/Quest.