Drag simulated objects around in Meshcat

[vincekurtz]

This is a sloppy, vibe-coded (w/ Claude Opus 4.8) solution to a long-standing gripe I've had with Drake+Meshcat. Other simulators (MuJoCo, Newton, etc) allow you to interact with a scene by clicking and dragging. This is particularly handy for developing/evaluating controllers, since it allows you to "poke" the scene to gain some intuition about robustness.

This PR adds gives Drake the same ability. CTRL + left click + drag produces a virtual spring force:

bazel run //examples/meshcat_mouse_spring:run_mouse_spring_demo -- --integrator=cenic

Screencast.from.2026-06-15.11-47-46.webm

bazel run //examples/atlas:atlas_run_dynamics -- --simulator_target_realtime_rate=1.0 --simulation_time=600

Screencast.from.2026-06-15.11-46-56.webm

bazel run examples/hydroelastic/python_ball_paddle:contact_sim_demo_py -- --simulation_time=600

Screencast.from.2026-06-15.11-29-28.webm

This is primarily a proof of concept. If there's interest, I'd be happy to work with others on cleaning this up into Drake-ready code.

---

This change is 

[SeanCurtis-TRI]

I'll take a look at it. As features go it is, for all the reasons you mention, a viable and desirable feature.

I'll do a pass over on the implementation to see if there's anything egregious about the big picture before we worry about the details.

+a:@SeanCurtis-TRI

@SeanCurtis-TRI made 1 comment.
Reviewable status: LGTM missing from assignee SeanCurtis-TRI(platform), needs at least two assigned reviewers, commits need curation (https://drake.mit.edu/reviewable.html#curated-commits), missing label for release notes.

[SeanCurtis-TRI]

+(release notes: feature)

[xuchen-han]

I'll very much benefit from this feature. Thanks @vincekurtz!

@xuchen-han made 1 comment.
Reviewable status: LGTM missing from assignee SeanCurtis-TRI(platform), needs at least two assigned reviewers, commits need curation (https://drake.mit.edu/reviewable.html#curated-commits) (waiting on SeanCurtis-TRI).

[nepfaff]

This looks amazing! I was hoping for this for a long time.

For feature equivalence with Mujoco, Isaac, it would be great to also add an option for applying a rotational force (e.g., CTRL + right click + drag).

[SeanCurtis-TRI]

Generally, this is a good approach. The architecture looks generally reasonable. I've made some comments about some of the UI experience and I'm also open to the idea of applying torques as noted above.

I limited myself to big picture aspects of the code. I'll defer the detailed stuff when we're ready to move forward.

@SeanCurtis-TRI made 14 comments.
Reviewable status: 13 unresolved discussions, LGTM missing from assignee SeanCurtis-TRI(platform), needs at least two assigned reviewers, commits need curation (https://drake.mit.edu/reviewable.html#curated-commits) (waiting on vincekurtz).

---

a discussion (no related file):
Big picture item -- modifications to Meshcat require some manifestation in meshcat_manual_test.cc. So, we'll want to make sure there's some stage in which the tester has the opportunity to confirm this meshcat feature is working.

---

visualization/visualization_config.h line 84 at r1 (raw file):

  bool enable_alpha_sliders{false};

  /** Whether to enable interactively dragging bodies with the mouse in Meshcat.

BTW We might consider making this a single parameter:

std::optional<double> mouse_interaction_stiffness{std::nullopt};

---

geometry/meshcat.html at r1 (raw file):
I had codex add a tiny feature where when you hold Ctrl and mouse over a draggable target, the pointer changes to a hand (nice visual feedback for stuff being "doable"). Let me know if you'd like the patch -- but I'm sure Claude could reproduce it independently.

---

geometry/meshcat.html at r1 (raw file):
It would be good if there was some clear UI feedback indicating that dragging is possible and indication of how to do it. (Currently,we print it to the console where it isn't necessarily apparent).

Some in view indication would be much better -- it's also a clear indicator that the feature has been enabled.

---

geometry/meshcat.html line 104 at r1 (raw file):

    function drakeRaycast(event) {
      drakeDrag.raycaster.setFromCamera(drakeMouseNdc(event), viewer.camera);
      const hits = drakeDrag.raycaster.intersectObject(viewer.scene, true);

BTW Instead of passing the viewer.scene, and then filtering on which objects have the expected prefix, you could search the subtree directly:

      let root_node = viewer.scene_tree.find(["drake"]);
      const hits = drakeDrag.raycaster.intersectObject(root_node.object, true);

Then you've satisfied the prefix requirement by construction.

---

geometry/meshcat.html line 107 at r1 (raw file):

      return hits.find(
        (h) => drakeVisible(h.object) &&
               drakeObjectPath(h.object).startsWith("/drake")) || null;

nit: This prefix is generally fine.

However, users have a degree of freedom in specifying a MeshcatVisualizer where they can specify the prefix as an absolute path. In that case, it could live outside the /drake/ tree. We'll need to document somewhere that this code requires manipulable objects to be located in the /drake/ tree.

Alternatively, we could introduce another html parameter. It would default to /drake but allow the user to specify some alternate root path.

This could have added value (in conjunction with my previous note about intersecting rays with the subtree node) in allowing the user to strongly limit the scope of manipulable object by specifying the root as being somewhere deeper in the tree (i.e., just the visual elements of an arm, or just manipulands, etc.)

---

geometry/meshcat.html line 119 at r1 (raw file):

        geometry.setAttribute(
          "position", new THREE.BufferAttribute(new Float32Array(6), 3));
        const material = new THREE.LineBasicMaterial({color: 0x000000});

nit: A constant black means the line's visibility depends on what the line is being drawn on top of. Something like this will basically guarantee visibility:

@@ -139,14 +139,26 @@
     function drakeUpdateSpringLine(a, b) {
       if (!drakeDrag.line) {
         const geometry = new THREE.BufferGeometry();
         geometry.setAttribute(
           "position", new THREE.BufferAttribute(new Float32Array(6), 3));
-        const material = new THREE.LineBasicMaterial({color: 0x000000});
+        // Invert the framebuffer pixels under the line so it contrasts with
+        // both light and dark scene colors.
+        const material = new THREE.LineBasicMaterial({
+          color: 0xffffff,
+          transparent: true,
+          depthTest: false,
+          depthWrite: false,
+          blending: THREE.CustomBlending,
+          blendEquation: THREE.SubtractEquation,
+          blendSrc: THREE.OneFactor,
+          blendDst: THREE.OneFactor,
+        });
         drakeDrag.line = new THREE.Line(geometry, material);
         drakeDrag.line.frustumCulled = false;
         drakeDrag.line.raycast = () => {};  // Don't let the line be pickable.
+        drakeDrag.line.renderOrder = 1000;
         viewer.scene.add(drakeDrag.line);
       }
       const position = drakeDrag.line.geometry.attributes.position;
       position.array[0] = a.x;
       position.array[1] = a.y;

---

geometry/meshcat.html line 174 at r1 (raw file):

        if (connected) {
          conn.send(MeshCat.msgpack.encode({
            type: "mouse_drag",

nit: We should name this event with more targeted language. It's easily conceivable that we could want to do other things with mouse drags in the future. So, a name reflecting the more specific semantics of this message would carve out a cleaner niche.

That said, I'm open to the possibility that this is quite literally simply a mouse drag operation whose end points are world-space coordinates instead of screen space coordinates. The fact that some entities interpret that as a drag force on a body is orthogonal. Of course, that puts the burden on the recipient of the message to know whether the message belongs to it or another "mouse drag" recipient. Things to ponder. For example, I can imagine a "measuring" tool that would likewise interpret the mouse drag as a measurement position vector from point A to point B whose length should be reported.

I'll have to ponder this, but I'm open to either direction.

---

multibody/meshcat/meshcat_mouse_spring.cc line 171 at r1 (raw file):

    forces->push_back(force);
  } else {
    // Mouse interaction is only meaningful for T == double (Meshcat reports

nit: I don't buy this reasoning. We have no shortage of double-valued inputs into a AutoDiffXd-valued plant. It just means they have no derivatives.

Perhaps the true justification is that no one would run autodiff interactively so a force arising from interactive manipulation would make more sense. This system is nominally AutoDiffXd so that it doesn't limit a Diagram with such a system from being constructed.

---

visualization/visualization_config_functions.cc line 81 at r1 (raw file):

          internal::ConvertVisualizationConfigToMeshcatContactParams(config));
    }
    if (config.enable_mouse_interaction) {

BTW Probably say enabled and non-zero stiffness?

---

visualization/visualization_config_functions.cc line 85 at r1 (raw file):

      // Only add it if that port is still available; otherwise (e.g., the user
      // has already connected their own external forces) skip it with a warning
      // rather than failing the whole visualization setup.

BTW For TODO: we could have a spatial force combiner system that gets wired into the diagram graph to allow both force sources.

I can also imagine a world where MbP allows multiple sources and it simply applies all the forces.

---

visualization/visualization_config_functions.cc line 85 at r1 (raw file):

      // Only add it if that port is still available; otherwise (e.g., the user
      // has already connected their own external forces) skip it with a warning
      // rather than failing the whole visualization setup.

nit: This restriction should be carefully noted in the VisualizationConfig documentation (and possibly cited in the MeshcatMouseSpring docs).

---

multibody/meshcat/meshcat_mouse_spring.h line 137 at r1 (raw file):

}  // namespace drake

DRAKE_DECLARE_CLASS_TEMPLATE_INSTANTIATIONS_ON_DEFAULT_NONSYMBOLIC_SCALARS(

BTW It's not clear to me that this should be instantiated on more than double:

1. It doesn't currently support scalar conversion (although it could handle its stored MultibodyPlant in the same way that JointStiffnessController does).
2. We would presume that AutoDiffXd-valued diagram would be so slow that doing anything interactively would be fraught.

So, the only way to get a Diagram<AutoDiffXd> is to build it directly with the MeshcatMouseSpring explicitly, which is known to be a no-op. So, at that point, it might as well be left out.

The only people it would harm is people who explicitly construct diagrams based on T -- they'd have to specialize their code based on T (do I add the mouse spring or not?)

We have a number of systems that are double-only, so it would be in good company. Some of them are located in the visualization module. So, they are near siblings.

If we go double-only, that may justify moving this to the visualization module where we have a number of other double-only visualization artifacts. Although, there are good arguments to be made for keeping it here as well. So, no strong opinion here.

[vincekurtz]

Thanks Sean! What's the best way to move forward with this? Keep iterating here? Open a few smaller PRs? Something else?

@vincekurtz made 3 comments.
Reviewable status: 13 unresolved discussions, LGTM missing from assignee SeanCurtis-TRI(platform), needs at least two assigned reviewers, commits need curation (https://drake.mit.edu/reviewable.html#curated-commits) (waiting on SeanCurtis-TRI).

---

geometry/meshcat.html at r1 (raw file):

Previously, SeanCurtis-TRI (Sean Curtis) wrote…

I had codex add a tiny feature where when you hold Ctrl and mouse over a draggable target, the pointer changes to a hand (nice visual feedback for stuff being "doable"). Let me know if you'd like the patch -- but I'm sure Claude could reproduce it independently.

Nice! I'm certainly open to such a patch, although the other simulators I'm most familiar with (Newton and MuJoCo) do not have such an indication in their dragging GUIs. MuJoCo makes you double click on an object first, and then it turns slightly brighter. With Newton you just right click and drag (no torques as far as I can tell) and the only visual feedback is a line showing the spring force.

---

geometry/meshcat.html at r1 (raw file):

Previously, SeanCurtis-TRI (Sean Curtis) wrote…

It would be good if there was some clear UI feedback indicating that dragging is possible and indication of how to do it. (Currently,we print it to the console where it isn't necessarily apparent).

Some in view indication would be much better -- it's also a clear indicator that the feature has been enabled.

Relatedly, is it important to not draw the line if the forces aren't really getting applied? Currently the spring line is drawn regardless, including during playback or even when the server is disconnected.

That matches the behavior of Newton's gui at least, so maybe it isn't a big deal. But I imagine it could lead to some confusion/frustration.

[SeanCurtis-TRI]

This is a very large PR. Finding a way to decompose it into two separate PRs wuld be a good thing. I can imagine the MeshcatMouseSpring class and unit tests with just enough of a stub on Meshcat to allow reporting fake drags would be enough. Doing just that would take us from roughly 1200 lines in one PR to a 550-650 split. That seems reasonable.

At the same time, I'd leave this PR in place as a reference if someone needs to see the whole working during the review of the new LeafSystem.

@SeanCurtis-TRI made 3 comments.
Reviewable status: 13 unresolved discussions, LGTM missing from assignee SeanCurtis-TRI(platform), needs at least two assigned reviewers, commits need curation (https://drake.mit.edu/reviewable.html#curated-commits) (waiting on vincekurtz).

---

geometry/meshcat.html at r1 (raw file):

Previously, vincekurtz (Vince Kurtz) wrote…

Nice! I'm certainly open to such a patch, although the other simulators I'm most familiar with (Newton and MuJoCo) do not have such an indication in their dragging GUIs. MuJoCo makes you double click on an object first, and then it turns slightly brighter. With Newton you just right click and drag (no torques as far as I can tell) and the only visual feedback is a line showing the spring force.

I certainly wouldn't use the fact that those other domains have poor visual feedback as a reason to not have better feedback on our side. :)

---

geometry/meshcat.html at r1 (raw file):

Previously, vincekurtz (Vince Kurtz) wrote…

Relatedly, is it important to not draw the line if the forces aren't really getting applied? Currently the spring line is drawn regardless, including during playback or even when the server is disconnected.

That matches the behavior of Newton's gui at least, so maybe it isn't a big deal. But I imagine it could lead to some confusion/frustration.

I hadn't considered that one way or the other.

I think, on the GUI side, if the browser is running, it should always draw the vector. If the server is disconnected, we already have a large yellow banner indicating that. So, the behavior where we will illustrate a drag, irrespective of server state, feels appropriate.

[vincekurtz]

Sounds good - I'll push some updates here based on the comments so far, then open smaller PRs.

@vincekurtz made 6 comments and resolved 8 discussions.
Reviewable status: 5 unresolved discussions, LGTM missing from assignee SeanCurtis-TRI(platform), needs at least two assigned reviewers, commits need curation (https://drake.mit.edu/reviewable.html#curated-commits) (waiting on SeanCurtis-TRI).

---

a discussion (no related file):

Previously, SeanCurtis-TRI (Sean Curtis) wrote…

Big picture item -- modifications to Meshcat require some manifestation in meshcat_manual_test.cc. So, we'll want to make sure there's some stage in which the tester has the opportunity to confirm this meshcat feature is working.

Done.

---

geometry/meshcat.html line 119 at r1 (raw file):

Previously, SeanCurtis-TRI (Sean Curtis) wrote…

nit: A constant black means the line's visibility depends on what the line is being drawn on top of. Something like this will basically guarantee visibility:

@@ -139,14 +139,26 @@
     function drakeUpdateSpringLine(a, b) {
       if (!drakeDrag.line) {
         const geometry = new THREE.BufferGeometry();
         geometry.setAttribute(
           "position", new THREE.BufferAttribute(new Float32Array(6), 3));
-        const material = new THREE.LineBasicMaterial({color: 0x000000});
+        // Invert the framebuffer pixels under the line so it contrasts with
+        // both light and dark scene colors.
+        const material = new THREE.LineBasicMaterial({
+          color: 0xffffff,
+          transparent: true,
+          depthTest: false,
+          depthWrite: false,
+          blending: THREE.CustomBlending,
+          blendEquation: THREE.SubtractEquation,
+          blendSrc: THREE.OneFactor,
+          blendDst: THREE.OneFactor,
+        });
         drakeDrag.line = new THREE.Line(geometry, material);
         drakeDrag.line.frustumCulled = false;
         drakeDrag.line.raycast = () => {};  // Don't let the line be pickable.
+        drakeDrag.line.renderOrder = 1000;
         viewer.scene.add(drakeDrag.line);
       }
       const position = drakeDrag.line.geometry.attributes.position;
       position.array[0] = a.x;
       position.array[1] = a.y;

After playing around, I'm not convinced that this is much better. This just shows as plain white for me, and there are some times when having the line behind objects in the scene is actually quite helpful.

---

geometry/meshcat.html line 174 at r1 (raw file):

Previously, SeanCurtis-TRI (Sean Curtis) wrote…

nit: We should name this event with more targeted language. It's easily conceivable that we could want to do other things with mouse drags in the future. So, a name reflecting the more specific semantics of this message would carve out a cleaner niche.

That said, I'm open to the possibility that this is quite literally simply a mouse drag operation whose end points are world-space coordinates instead of screen space coordinates. The fact that some entities interpret that as a drag force on a body is orthogonal. Of course, that puts the burden on the recipient of the message to know whether the message belongs to it or another "mouse drag" recipient. Things to ponder. For example, I can imagine a "measuring" tool that would likewise interpret the mouse drag as a measurement position vector from point A to point B whose length should be reported.

I'll have to ponder this, but I'm open to either direction.

Maybe object_drag or body_drag, to specify that we're dragging an object, not just the mouse anywhere on the screen?

I don't have a strong opinion on this.

---

geometry/meshcat.html at r1 (raw file):

Previously, SeanCurtis-TRI (Sean Curtis) wrote…

I certainly wouldn't use the fact that those other domains have poor visual feedback as a reason to not have better feedback on our side. :)

Done.

---

geometry/meshcat.html at r1 (raw file):

Previously, SeanCurtis-TRI (Sean Curtis) wrote…

I hadn't considered that one way or the other.

I think, on the GUI side, if the browser is running, it should always draw the vector. If the server is disconnected, we already have a large yellow banner indicating that. So, the behavior where we will illustrate a drag, irrespective of server state, feels appropriate.

Done.