Reimplement AutoDiff from the ground up

[jwnimmer-tri]

Relates to #10991 and #16877.

There are two main ideas with the new implementation.

---

First, we change the storage type for AutoDiffXd's derivatives from Eigen::VectorXd to drake::internal::Partials. The AutoDiffXd::derivatives() accessors now return an Eigen xpr-like view into the storage, instead of a direct reference to it.

The Partials storage type offers several unique advantages compared to a Eigen::VectorXd:

(1.a) Sparse storage. When the derivatives only contain a single non-zero element, it is stored in sparse form inline (no heap operations). Arithmetic combinations of same-index sparse derivatives preserve the sparsity.

(1.b) Copy-on-write storage. When a non-sparse AutoDiffXd (with heap storage) needs to be copied, we increment the refcount of the derivatives instead of allocating a new copy.

(1.c) Separately-stored scale factor. With the chain rule, an extremely common operation is to scale up the derivatives by a constant factor (e.g., this is true for all unary functions). By storing a separate scaling factor, we can re-scale with a single multiply (to the scale factor) instead of broadcasting it over the entire data vector. That allows us to defer repeated scaling until the full vector needs to be read out later, or even skip it entirely if we eventually get zeroed out anyway. It also makes the copy-on-write more efficient by reducing need for writes. Only binops (+, -, *, /) or assignment might induce full copies.

---

Second, we replace Eigen::AutoDiffScalar<> template with a new drake::autodiff::AutoDiff class. The new class is no longer templated (i.e., the type of derivatives is always internal::Partials) but otherwise is very similar to the Eigen one.

(2.a) By using our own Drake-specific class, we have free reign to specialize Eigen operations on it. In particular, Eigen's support for using move-efficient value categories is abysmal. We specialize a handful of performance-critical Eigen templates to avoid extra copying. Even though copies are cheap, copy-on-write performs better when it ends up with uniquely-owned storage to mutate in place. With more profiling, we can probably find additional places to specialize in order to reduce refcount chatter.

(2.b) This opens the door in the future to specializing Eigen::Matrix<drake::autodiff::AutoDiff, ...> dense storage to store all of the derivatives as single block on the heap. Our xpr-like view objects use a runtime stride in anticipation of this.

(2.c) It's nice to no longer depend on an "unsupported" Eigen type so heavily.

---

PR train listing:

• [common] Add and use MatrixLikewise sugar typedef #17497
• [common] Centralize Eigen minimum version requirement #17498
• [common] Deprecate autodiffxd_make_coherent #17499
• [misc] Tidy up weird spellings of AutoDiffXd #17500
• [math] Avoid fixed-sized derivatives in quaternion_test #17501
• [systems] Use InitializeAutoDiff instead of manual looping #17502
• [math] Reduce implementation boilerplate for autodiff #17516
• [geometry] Add test epsilon in preparation for new AutoDiff #17532
• [primitives] Spell ternary operator compatibly with new AutoDiff #17533
• [ad] Add AutoDiff replacement class for AutoDiffXd #17621
• [ad] Add support for arithmetic operations #17622
  • [ad] Fix gradients of in-place x*x and x/x #17656
  • [ad] Add unit test for -0.0 unary negation signbit #17658
• [ad] Add support for comparison operations #17636
• [ad] Add support for trig and hyperbolic operations #17650
• [ad] Add support for power functions #17655
• [ad] Add support for nearest-integer functions #17657
• [ad] Opt-in to clang-format lint #19621
• [ad] Add Eigen trait specializations #19622
• [py autodiff] Bind constructor overload for unit derivatives vector #19623
• [ad] Add microbenchmark #19627
• [ad] Add a separate scaling factor to Partials #19628
• [solvers] Remove MobyLCPSolver autodiff support #23775
• [systems] Use canonical AutoDiffXd in simulator_test #23821
• [examples] Remove bead_on_a_wire #23822
• [math] Deprecate unused jacobian and hessian free functions #23823
• Remove unwanted autodiff_overloads.h includes #23824
• [ad] Change nearest-integer functions to return double #23825
• [ad] Remove unwanted ostream operator #23826
• [yaml] Port performance test away from second-order autodiff #23829
• [math] Port compute_numerical_gradient_test to AutoDiffXd #23830
• [ad] Fix logical sparsity in derivatives #23861
• [ad] Special case pow for non-negative exponents of a zero base #23862
• [common] Port autodiff_overloads_test away from fixed-size autodiff #23863
• Add miscellaneous details to test failures #23868
• [build] Add a flag to opt-in to the new AutoDiff implementation #23870
• [ad] Add overloads for drake-specific operations #23871
• [mpm] Fix GridData<AutoDiffXd>::operator== #23892
• [ad] Small code cleanups #23893
• [ad] Remove extra argument to unit vector constructor #23899
• [ad] Store partials in custom vector #23902
• TODO in atan2
• ... to be continued

---

This change is 

[jwnimmer-tri]

This is a rough cut for now, with some missing pieces. There are also a few bugs still showing up in tests, which indicates some of the computations might be wrong -- so possibly this is faster in some places by cheating (giving wrong answers) rather than for a good reason. However, a lot of unit tests do pass. Even with those caveats, I think the prototype shows promise and we should continue working.

---

Here's the latest Cassie results for mass matix, inverse dynamics, and forward dynamics:

The chart shows that not only is the throughput improved ~2x, the stddevs are narrower.

---

Here's the raw numbers, for now:

Case	Min Time	Relative reduction
MM master	1854 us
MM PR	883 us	52.3%
ID master	2793 us
ID PR	1001 us	64.2%
FD master	4241 us
FD PR	2478 us	41.6%

---

It won't be part of this first prototype, but having our own AutoDiff scalar also paves the way to sharing partials storage across all of the elements in a MatrixX<AutoDiff> down the road.

[jwnimmer-tri]

Here's the quick sample from /multibody/inverse_kinematics:position_constraint_experiment for IiwaPositionConstraintFixture:

Benchmark	Time	Relative reduction
EvalAutoDiffMbpDouble_mean (master)	80 us
EvalAutoDiffMbpDouble_mean (PR)	66 us	17%
EvalAutoDiffMbpAutoDiff_mean (master)	2288 us
EvalAutoDiffMbpAutoDiff_mean (PR)	809 us	64%

[jwnimmer-tri]

Reviewable status: 2 unresolved discussions, needs platform reviewer assigned, needs at least two assigned reviewers, labeled "do not merge", commits need curation (https://drake.mit.edu/reviewable.html#curated-commits)

---

common/autodiff/auto_diff.h line 20 at r1 (raw file):

in case only one partial is non-zero) it runs much faster than
Eigen::AutoDiffScalar<Eigen::VectorXd>. */
class AutoDiff {

This is the rough explanation of the implementation technique.

---

common/autodiff/internal/partials.h line 193 at r1 (raw file):

  void CheckInvariants() const;
  // Representation invariant:

This it the key piece of the representation of the derivatives.

[sherm1]

Wow! Awesome.

[jwnimmer-tri]

The functional tests are passing now, though apparently there are still some memory leaks to chase down.
(Edit: Fixed now; no meaningful change in the numbers.)

Here's the updated perf summary for the //examples/multibody/cassie_benchmark:experiment and //multibody/inverse_kinematics:position_constraint_experiment cases.

.	Cassie MM	Cassie ID	Cassie FD	IK MbpD	IK MbpAD
Master (min)	1854 us	2793 us	4241 us	79 us	2254 us
PR (min)	815 us	833 us	2373 us	68 us	658 us
Reduction	56.0%	70.2%	44.0%	13.5%	70.8%

[sherm1]

The timings look great! There are an assortment of independent tricks here and I'm wondering whether they all contribute meaningfully to the speedup. My thought is that if some aren't paying off the implementation could be simpler. Have you measured them independently? These seem independent to me:

• compact representation of unit vectors
• scale factor
• copy on write

[jwnimmer-tri]

There are an assortment of independent tricks here and I'm wondering whether they all contribute meaningfully to the speedup ...

• compact representation of unit vectors
• scale factor
• copy on write

What I'll do is land the changes one by one.

The #19628 has landed on master now with the scale factor. As the PR train continues, we'll see the separate effects of compact unit vectors and copy on write.

[github-actions]

Thank you for your contribution. This pull request has been open for 180 days without activity and so is considered stale. It will be automatically closed in 30 days unless you comment or remove the "status: stale" label.