Learning-Based Tracking and Iterative Control
Several Meanings of “Matching”
Learning-based motion matching can mean at least three different tasks. Imitation learning fits a policy or trajectory to demonstrations. Tracking control computes actions online from current state error. Iterative learning control (ILC) updates an entire feedforward command between repeated trials. They use related data but answer different questions. A policy can imitate demonstrated state-action pairs yet fail to track under disturbances; a tracking controller can follow a reference that was never generated by a human; ILC can improve repeatable performance without learning a general policy.
In forward model matching, the physical dynamics remain explicit: \(x_{t+1}=f(x_t,u_t;\theta)\), and controls or policy parameters are chosen so the forward rollout explains measured motion. In pure kinematic fitting, the system may directly assign poses without requiring dynamic feasibility. In reinforcement learning, an agent maximizes expected return through interaction or simulation; imitation objectives may be auxiliary rewards or initializers. Keep these labels distinct. A visually convincing simulated character does not establish that its controls are human, uniquely identified, or physically consistent with measured forces.
No inverse is generally unique. Many control sequences can produce similar observed trajectories, particularly with partial markers, hidden states, unknown initial conditions, and flexible dynamics. Equally, different physical parameters and control policies can compensate for one another. A learned policy therefore represents one solution under a chosen model, objective, data, and regularizer. It should not be presented as the unique recovered neural command or causal mechanism.
Open-Loop and Closed-Loop Control
An open-loop command is a time-indexed sequence \(u_t=\bar u_t\) chosen without using current tracking error. It can reproduce a nominal trajectory under the same initial state and model, but disturbances accumulate. A closed-loop controller uses observations or state estimates: \(u_t=\pi(x_t,t)\), often with feedforward plus feedback, \(u_t=\bar u_t+K_t(x_t-\bar x_t)\). Feedback can correct model mismatch and perturbations, but its apparent trajectory fit then depends on the estimator, gains, sensor delay, and disturbance assumptions.
This distinction matters for scientific inference. If the question is whether a candidate forward model can generate the observed trajectory from a plausible control, open-loop replay is a strong and interpretable test. If feedback is allowed, report the feedback law, information available to it, latency, bandwidth, and whether it uses measured future data. A controller that reads the reference at the current time is legitimate tracking; a controller that uses future motion can unrealistically correct errors and must be described as preview control. A policy trained against the full reference may encode timing in its state or reward and should not be mislabeled as an autonomous explanation.
A tracking loss can combine state and effort errors: \[ J=\sum_{t=0}^{T-1}(x_t-\bar x_t)^TQ_t(x_t-\bar x_t)+u_t^TR_tu_t+(u_t-\bar u_t)^TS_t(u_t-\bar u_t)+J_T. \] The last control term regularizes deviation from nominal feedforward. A time-varying linear feedback gain can be obtained from a Riccati recursion around a nominal trajectory. For nonlinear models, iLQR or DDP repeatedly linearizes/quadratizes and updates the nominal. These methods can provide stabilizing local corrections, but their guarantees depend on model validity, constraints, and local convergence.
Imitation Learning Objectives
Behavioral cloning minimizes action prediction error on demonstrations: \(\mathbb E_{(s,a)\sim D}[\ell(\pi(s),a)]\). It is simple but suffers covariate shift: at deployment, small errors visit states absent from the demonstrations, where the policy is unconstrained. Dataset aggregation addresses this by collecting expert labels at states induced by the current policy. In biomechanics, acquiring expert labels for perturbations may be impractical, so robust simulators or carefully justified proxy controllers are used; this introduces model dependence.
Trajectory imitation can instead match state distributions, key events, end-effector paths, or task outcomes. A pose loss may use markers and orientation on manifolds; a velocity or acceleration loss amplifies measurement noise. A reward can combine imitation with balance, effort, task success, and contact constraints. Reward design determines what behaviors are considered equivalent. A policy may exploit a weakly specified reward or simulation artifact. Include held-out motion segments, perturbation tests, and independent dynamics evaluation.
DeepMimic is relevant as an example of physics-based character skill learning from reference motion. It uses deep reinforcement learning to imitate motion while solving a task, demonstrating that a simulated character can learn robust skills that combine reference tracking and task rewards. Its relevance is conceptual and methodological: it shows a possible policy-learning architecture and training pattern. It does not establish that a learned policy uniquely recovers human intent, that its rigid-body or contact model matches a particular person’s mechanics, or that a motion-matching inverse problem has a unique solution. These distinctions are essential when citing reinforcement-learning character animation in a scientific forward-dynamics study.
ILC for Repeated Motions
ILC assumes repeated execution of a task with repeatable dynamics and updates the feedforward command using prior-trial error. For trial index \(j\), let output error be \(e_j(t)=r(t)-y_j(t)\). A basic update is \[ u_{j+1}(t)=u_j(t)+L e_j(t), \] where \(L\) is a learning operator. A filtered update might be \(u_{j+1}=Q(u_j+Le_j)\) to limit high-frequency amplification. Under a linear time-invariant trial model \(y=Gu\), a sufficient norm condition for monotone error reduction in a simple setting is \(\|I-GL\|<1\). Real systems require robust analysis for model uncertainty, delays, and nonminimum-phase behavior.
ILC is useful when each swing or robot trajectory is repeated under nearly the same conditions. It learns a trajectory-specific feedforward command, not necessarily a general state-feedback strategy. Trial-to-trial variability, changing fatigue, equipment changes, and nonrepeatable disturbances violate assumptions. If one human demonstration is available, ILC cannot be run on that individual without repeated trials; applying its logic to multiple athletes assumes meaningful alignment and a shared task. Distinguish subject-specific adaptation from population learning.
A practical ILC pipeline aligns trial phase, estimates output error with uncertainty, applies a bounded update, and checks safety before the next attempt. Phase alignment errors can make a correct command appear wrong and lead the update in a harmful direction. Time warping should be constrained and recorded; otherwise it can erase real timing differences. Use trust regions on command updates, limit actuator demand, and keep a baseline command available for rollback.
Combining Model Matching and Learning
A useful research architecture separates identification from policy adaptation. First estimate a family of plausible dynamics parameters from kinematic and force data. Next optimize feedforward controls or a policy over that uncertainty set. Finally replay candidates in an independent evaluator and compare prediction. This avoids treating the controller as a hidden correction term that masks dynamics error. Robust control can optimize worst-case or expected performance over parameter uncertainty; a posterior over models is more honest than a single best-fit model when identifiability is weak.
A differentiable model can support policy gradients or model-based planning, but differentiation through a learned surrogate can exploit errors. Use trust regions that keep simulated states near training data, uncertainty penalties, periodic real-engine rollouts, and final validation in the target dynamics. For contact-rich motion, gradients through a smooth surrogate may help initialize a candidate, while hybrid replay checks mode sequence and event timing. Report the surrogate and target model separately.
Evaluation should separate demonstration fidelity, task performance, robustness, and plausibility. Demonstration metrics include marker error and event timing. Task metrics include success and terminal constraints. Robustness metrics perturb initial conditions, dynamics, observations, and timing. Plausibility metrics may include actuator limits, contact forces, energy, and smoothness, but must have declared thresholds. No single weighted reward can replace a table of interpretable outcomes.
Proposed Adaptive Architecture
A proposed dimension-adaptive learner would maintain uncertainty over both physical model dimension and policy capacity. Use separate selection variables: \(d\) for generalized coordinates or structural modes, \(m\) for policy/control basis complexity. Candidate changes should be accepted only when they improve held-out prediction or task return beyond uncertainty and pass physical qualification. Synthetic tests should distinguish a missing state dimension from inadequate policy capacity by generating cases where each is independently necessary.
The policy should not be allowed to compensate invisibly for structural model inadequacy. Log residuals before and after feedback, policy action magnitude, and uncertainty. If feedback effort rises sharply in one phase, investigate missing coordinates, contact discrepancy, or sensor bias before increasing network size. Population generalization requires subject-level holdout; frame-level random splits leak adjacent temporal information. This entire adaptive architecture is a proposal, not an implemented upstream feature.
Repository Context and Limits
Upstream contains a MyoSuite surrogate training path under src/engines/physics_engines/myosuite/python/motion_matching/train_surrogate.py, an inverse-surrogate module, MuJoCo MJX tracking components, and a separate tracking plant. These are evidence of learning/tracking-related code pathways, not proof that all are integrated with the same calibrated forward model or qualify human motor intent. The pinned source revision for references in this book is d7c7d07add0d7407a5a3376d5fe4a97aee9112a0; claims about a specific algorithm should link to the exact immutable file and any candidate receipt.
Worked Illustrative Example
Consider an illustrative discrete system \(x_{t+1}=x_t+u_t+w_t\) with reference \(r_t\). Open-loop commands \(u_t\) chosen from a nominal disturbance-free model leave accumulated error when \(w_t\ne0\). A feedback law \(u_t=\bar u_t-K(x_t-r_t)\) corrects measured deviation but depends on valid state estimates and cannot correct unobserved disturbances immediately. Across repeated trials, ILC can update \(\bar u_t\) from the previous error. The three methods use different information and should be compared under distinct evaluation conditions; this toy example is not empirical evidence about a human swing.
Exercises and Solutions
Exercise 1. A policy reproduces all demonstration poses in simulation. Does that show it recovered the demonstrator’s unique control? Solution. No. Multiple controls, policies, and model parameters can produce similar observed poses. Additional force, perturbation, and physiological evidence may constrain the explanation but rarely guarantee uniqueness.
Exercise 2. Why can feedback tracking conceal an incorrect forward model? Solution. The controller can compensate online for prediction error. Report control effort and open-loop replay alongside closed-loop performance to reveal the correction burden.
Exercise 3. When is ILC inappropriate? Solution. When trials are not repeatable, phase alignment is unreliable, the environment changes substantially, or updates cannot be safely tested. ILC also cannot infer a general policy from one unrepeated trajectory.
Further Reading
The AffineDrift bibliography entry (Peng et al. 2018) provides the DeepMimic publication reference. Use it for the specific character-skill learning method, while keeping claims about model validity and inverse uniqueness separate. For trajectory optimization and local feedback context, see (Jacobson and Mayne 1970).
Validation Protocol for Learned Policies
A reproducible policy comparison fixes the state estimator, control frequency, action scaling, initial-state distribution, episode termination rules, reward weights, training budget, and random seeds. Report both training and evaluation environments; randomize only the quantities intended to represent uncertainty. Use multiple independent training seeds because deep reinforcement learning can vary substantially with initialization and exploration. A single successful video is a demonstration, not a performance distribution.
For model-based policy learning, compare predicted and actual state transitions on held-out states before trusting long-horizon rollouts. Report one-step error and rollout error separately: a small one-step bias may compound over time. Track the distance of evaluation states from the model’s training distribution. When using hardware or human trials, safety constraints and oversight define the feasible exploration protocol; offline replay cannot reveal all effects of interactive adaptation.
Keep reference tracking reward distinct from task reward. If a character receives both, report each term and ablate them independently. A task reward may appropriately permit deviations that improve the outcome, so a policy can be successful without reproducing the reference exactly. Conversely, a high imitation score may fail a task or violate contact constraints. Interpret the objective according to its construction and show the measurements that support each claim.
References
Citation
@online{untitled,
author = {},
url = {https://affinedrift.com/articles/forward_dynamics_matching/22_learning_tracking.html},
langid = {en}
}