Tangent Hyperplanes III: Control

Tangent Hyperplanes III: Control

tangent-space

Control design in the tangent space: linearizing around trajectories, feedback stabilization, and local reachability analysis.

From a Derivative Model to a Control Problem

Part 2 derived a linear equation for infinitesimal variations. A nonlinear plant does not become linear at each instant. The useful step is to solve a tractable local control problem, evaluate the resulting policy on the nonlinear dynamics, and check whether the proposed change improves the stated objective.

LQR solves a particular linear-quadratic problem. iLQR and differential dynamic programming (DDP) use related backward recursions inside nonlinear trajectory optimization. Their success requires more than good conditioning: derivative accuracy, feasibility, initialization, globalization and termination criteria all matter. They do not describe every nonlinear control method.

Finite-Horizon LQR and the Cost Convention

Consider the linear system $\dot\xi=A(t)\xi+B(t)\nu$ and cost

\[ \begin{aligned} J={}&\frac12\xi(t_f)^TQ_f\xi(t_f)\\ &+\frac12\int_{t_0}^{t_f} \bigl(\xi^TQ(t)\xi+\nu^TR(t)\nu\bigr)\,dt. \end{aligned} \]

Assume bounded continuous coefficients on a finite interval, symmetric $Q,Q_f\succeq0$, and uniformly positive-definite $R$. This unconstrained finite-horizon problem has a quadratic value function

\[ V(\xi,t)=\frac12\xi^TS(t)\xi. \]

Thus $D_\xi^2V=S$. If the cost is instead written without either factor of $1/2$, the value is $\xi^TS\xi$ and its Hessian is $2S$; the feedback gain is unchanged. Also, $v^TQv$ is a squared weighted seminorm when $Q\succeq0$, and the square of a norm only when $Q\succ0$.

Minimizing the Hamilton–Jacobi expression over $\nu$ gives

\[ \nu_*=-K\xi,\qquad K=R^{-1}B^TS, \]

with the backward Riccati equation

\[ \begin{aligned} -\dot S&=A^TS+SA-SBR^{-1}B^TS+Q,\\ S(t_f)&=Q_f. \end{aligned} \]

These are analytic optimality statements for the declared linear problem. Numerical integration and matrix solves introduce tolerances and conditioning errors; the result is not automatically accurate to machine precision. Tedrake’s LQR derivation develops the finite-horizon and tracking constructions. General tracking costs can include linear terms, producing an affine policy rather than this zero-centered regulator.

Why a Feedforward Update Is Essential

For nonlinear optimization, use a discrete dynamics map $x_{k+1}=F_k(x_k,u_k)$ obtained from a declared time discretization. Start with a dynamically feasible nominal rollout $(\bar x_k,\bar u_k)$ and approximate the local action value

\[ \mathcal Q_k(x,u)=\ell_k(x,u)+V_{k+1}(F_k(x,u)). \]

Expanding about the nominal pair gives linear terms $\mathcal Q_x^T\delta x+\mathcal Q_u^T\delta u$, with gradients written as columns, and quadratic blocks $\mathcal Q_{xx},\mathcal Q_{ux},\mathcal Q_{uu}$. When the local input Hessian is positive definite, minimizing that quadratic yields

\[ \begin{aligned} \delta u&=k_k+L_k\delta x,\\ k_k&=-\mathcal Q_{uu}^{-1}\mathcal Q_u,\\ L_k&=-\mathcal Q_{uu}^{-1}\mathcal Q_{ux}. \end{aligned} \]

Here $k_k$ is the feedforward correction; the negative sign is already included in $L_k$. The trial rollout uses

\[ u_k=\bar u_k+\alpha k_k+L_k(x_k-\bar x_k), \qquad 0<\alpha\le1. \]

In this equation $u_k$ is the physical trial input. A feedback-only policy $u_k=\bar u_k+L_k(x_k-\bar x_k)$ reproduces the old nominal rollout when it starts at the same state under the same deterministic model. It cannot improve a nonoptimal nominal input sequence by itself.

For a one-step illustration, let $x_1=x_0+u$, $x_0=0$, and $J=\tfrac12(x_1-1)^2+\tfrac12u^2$. At nominal $u=0$, $\mathcal Q_u=-1$, $\mathcal Q_{uu}=2$, hence $k=1/2$. The full step reduces cost from $1/2$ to $1/4$. Feedback alone leaves $u=0$.

iLQR and DDP Use Different Dynamics Derivatives

Let $A_k=D_xF_k$, $B_k=D_uF_k$, and let $v=D_xV_{k+1}$ and $P=D_x^2V_{k+1}$ be evaluated at the nominal successor. The second-order chain rule gives, for example,

\[ \begin{aligned} \mathcal Q_{uu} &=\ell_{uu}+B_k^TPB_k+\sum_i v_iD_{uu}^2F_{k,i},\\ \mathcal Q_{ux} &=\ell_{ux}+B_k^TPA_k+\sum_i v_iD_{ux}^2F_{k,i}. \end{aligned} \]

The analogous $xx$ block includes $\sum_i v_iD_{xx}^2F_{k,i}$. Full DDP retains these dynamics-Hessian contractions; iLQR drops them while retaining quadratic cost information. These are derivatives of the discrete map, not interchangeable formulas for a continuous vector field. The distinction is also described in Tedrake’s trajectory-optimization notes.

A practical iteration computes a backward update, regularizes when necessary, rolls out the nonlinear dynamics with a trial step, and compares actual objective improvement with the local prediction. Constraints require explicit handling; clipping an unconstrained policy does not preserve its optimality proof. A lower rollout cost neither proves convergence nor establishes a local minimum. Report feasibility, stationarity, step acceptance and termination tolerances.

Cost Decrease Does Not Automatically Prove Contraction

With the continuous LQR gain above and $A_{\rm cl}=A-BK$, direct substitution gives

\[ \dot S+A_{\rm cl}^TS+SA_{\rm cl}=-Q-K^TRK. \]

This identity connects cost and differential decay. To use $S$ as a uniformly contracting metric, one additionally needs uniform bounds $aI\preceq S\preceq bI$, $a>0$, and a rate $\lambda>0$ satisfying

\[ Q+K^TRK\succeq2\lambda S. \]

For nonlinear feedback, the corresponding inequality must hold for the full closed-loop Jacobian throughout the claimed region, not merely at sampled points on one reference. Part 5 explains the geometric conditions.

A counterexample is $\dot x=x+u$ with $Q=Q_f=0$ and $R=1$ on a finite horizon. The optimal policy is $u=0$, because only effort is penalized. Then $S=0$ and perturbations grow as $e^t$. Optimality for this objective supplies neither a metric nor stability.

Application to Golf and Humanoid Models

The objective must distinguish target accuracy, effort proxies, contact feasibility and robustness. A torque penalty is not a measurement of metabolic cost. Derivatives depend on the declared joint, shaft, muscle and contact states; changing those choices changes the local problem.

Dense Riccati-style operations can involve cubic state-dimension costs, but runtime also depends on input dimension, horizon, sparsity, derivative computation and solver implementation. Benchmark the actual model before promising real-time feedback. The next article, Residuals and Curvature, explains what the local dynamics omit and how that omission accumulates.

Citation

BibTeX citation: Download .bib

@online{untitled,
  author = {},
  title = {Tangent {Hyperplanes} {III:} {Control}},
  url = {https://affinedrift.com/articles/tangent-hyperplanes-series/part-3-control.html},
  langid = {en}
}

For attribution, please cite this work as:

“Tangent Hyperplanes III: Control.” n.d. https://affinedrift.com/articles/tangent-hyperplanes-series/part-3-control.html.

--- title: "Tangent Hyperplanes III: Control" description: "Control design in the tangent space: linearizing around trajectories, feedback stabilization, and local reachability analysis." aliases: - ../tangent-hyperplane-contraction/chapters/03-local-optimal-control.html - ../tangent-hyperplane-articles/Drafts_Original_Articles/Tangent_Hyperplanes_Series_Package/03_Tangent_Hyperplanes_III_Control.html categories: - tangent-space --- ## From a Derivative Model to a Control Problem [Part 2](./part-2-dynamics.html) derived a linear equation for infinitesimal variations. A nonlinear plant does not become linear at each instant. The useful step is to solve a tractable local control problem, evaluate the resulting policy on the nonlinear dynamics, and check whether the proposed change improves the stated objective. LQR solves a particular linear-quadratic problem. iLQR and differential dynamic programming (DDP) use related backward recursions inside nonlinear trajectory optimization. Their success requires more than good conditioning: derivative accuracy, feasibility, initialization, globalization and termination criteria all matter. They do not describe every nonlinear control method. ## Finite-Horizon LQR and the Cost Convention Consider the linear system $\dot\xi=A(t)\xi+B(t)\nu$ and cost $$ \begin{aligned} J={}&\frac12\xi(t_f)^TQ_f\xi(t_f)\\ &+\frac12\int_{t_0}^{t_f} \bigl(\xi^TQ(t)\xi+\nu^TR(t)\nu\bigr)\,dt. \end{aligned} $$ Assume bounded continuous coefficients on a finite interval, symmetric $Q,Q_f\succeq0$, and uniformly positive-definite $R$. This unconstrained finite-horizon problem has a quadratic value function $$ V(\xi,t)=\frac12\xi^TS(t)\xi. $$ Thus $D_\xi^2V=S$. If the cost is instead written without either factor of $1/2$, the value is $\xi^TS\xi$ and its Hessian is $2S$; the feedback gain is unchanged. Also, $v^TQv$ is a squared weighted seminorm when $Q\succeq0$, and the square of a norm only when $Q\succ0$. Minimizing the Hamilton–Jacobi expression over $\nu$ gives $$ \nu_*=-K\xi,\qquad K=R^{-1}B^TS, $$ with the backward Riccati equation $$ \begin{aligned} -\dot S&=A^TS+SA-SBR^{-1}B^TS+Q,\\ S(t_f)&=Q_f. \end{aligned} $$ These are analytic optimality statements for the declared linear problem. Numerical integration and matrix solves introduce tolerances and conditioning errors; the result is not automatically accurate to machine precision. [Tedrake's LQR derivation](https://underactuated.mit.edu/lqr.html) develops the finite-horizon and tracking constructions. General tracking costs can include linear terms, producing an affine policy rather than this zero-centered regulator. ## Why a Feedforward Update Is Essential For nonlinear optimization, use a discrete dynamics map $x_{k+1}=F_k(x_k,u_k)$ obtained from a declared time discretization. Start with a dynamically feasible nominal rollout $(\bar x_k,\bar u_k)$ and approximate the local action value $$ \mathcal Q_k(x,u)=\ell_k(x,u)+V_{k+1}(F_k(x,u)). $$ Expanding about the nominal pair gives linear terms $\mathcal Q_x^T\delta x+\mathcal Q_u^T\delta u$, with gradients written as columns, and quadratic blocks $\mathcal Q_{xx},\mathcal Q_{ux},\mathcal Q_{uu}$. When the local input Hessian is positive definite, minimizing that quadratic yields $$ \begin{aligned} \delta u&=k_k+L_k\delta x,\\ k_k&=-\mathcal Q_{uu}^{-1}\mathcal Q_u,\\ L_k&=-\mathcal Q_{uu}^{-1}\mathcal Q_{ux}. \end{aligned} $$ Here $k_k$ is the feedforward correction; the negative sign is already included in $L_k$. The trial rollout uses $$ u_k=\bar u_k+\alpha k_k+L_k(x_k-\bar x_k), \qquad 0<\alpha\le1. $$ In this equation $u_k$ is the physical trial input. A feedback-only policy $u_k=\bar u_k+L_k(x_k-\bar x_k)$ reproduces the old nominal rollout when it starts at the same state under the same deterministic model. It cannot improve a nonoptimal nominal input sequence by itself. For a one-step illustration, let $x_1=x_0+u$, $x_0=0$, and $J=\tfrac12(x_1-1)^2+\tfrac12u^2$. At nominal $u=0$, $\mathcal Q_u=-1$, $\mathcal Q_{uu}=2$, hence $k=1/2$. The full step reduces cost from $1/2$ to $1/4$. Feedback alone leaves $u=0$. ## iLQR and DDP Use Different Dynamics Derivatives Let $A_k=D_xF_k$, $B_k=D_uF_k$, and let $v=D_xV_{k+1}$ and $P=D_x^2V_{k+1}$ be evaluated at the nominal successor. The second-order chain rule gives, for example, $$ \begin{aligned} \mathcal Q_{uu} &=\ell_{uu}+B_k^TPB_k+\sum_i v_iD_{uu}^2F_{k,i},\\ \mathcal Q_{ux} &=\ell_{ux}+B_k^TPA_k+\sum_i v_iD_{ux}^2F_{k,i}. \end{aligned} $$ The analogous $xx$ block includes $\sum_i v_iD_{xx}^2F_{k,i}$. Full DDP retains these dynamics-Hessian contractions; iLQR drops them while retaining quadratic cost information. These are derivatives of the **discrete map**, not interchangeable formulas for a continuous vector field. The distinction is also described in [Tedrake's trajectory-optimization notes](https://underactuated.mit.edu/trajopt.html). A practical iteration computes a backward update, regularizes when necessary, rolls out the nonlinear dynamics with a trial step, and compares actual objective improvement with the local prediction. Constraints require explicit handling; clipping an unconstrained policy does not preserve its optimality proof. A lower rollout cost neither proves convergence nor establishes a local minimum. Report feasibility, stationarity, step acceptance and termination tolerances. ## Cost Decrease Does Not Automatically Prove Contraction With the continuous LQR gain above and $A_{\rm cl}=A-BK$, direct substitution gives $$ \dot S+A_{\rm cl}^TS+SA_{\rm cl}=-Q-K^TRK. $$ This identity connects cost and differential decay. To use $S$ as a uniformly contracting metric, one additionally needs uniform bounds $aI\preceq S\preceq bI$, $a>0$, and a rate $\lambda>0$ satisfying $$ Q+K^TRK\succeq2\lambda S. $$ For nonlinear feedback, the corresponding inequality must hold for the full closed-loop Jacobian throughout the claimed region, not merely at sampled points on one reference. [Part 5](./part-5-contraction.html) explains the geometric conditions. A counterexample is $\dot x=x+u$ with $Q=Q_f=0$ and $R=1$ on a finite horizon. The optimal policy is $u=0$, because only effort is penalized. Then $S=0$ and perturbations grow as $e^t$. Optimality for this objective supplies neither a metric nor stability. ## Application to Golf and Humanoid Models The objective must distinguish target accuracy, effort proxies, contact feasibility and robustness. A torque penalty is not a measurement of metabolic cost. Derivatives depend on the declared joint, shaft, muscle and contact states; changing those choices changes the local problem. Dense Riccati-style operations can involve cubic state-dimension costs, but runtime also depends on input dimension, horizon, sparsity, derivative computation and solver implementation. Benchmark the actual model before promising real-time feedback. The next article, [Residuals and Curvature](./part-4-residuals-curvature.html), explains what the local dynamics omit and how that omission accumulates. ## Related Articles ::: {.callout-note title="See Also"} - **[Tangent-Space Series](../../pages/tangent-hyperplanes.html)** — The series hub: full reading order, provenance, and companion material - **[Force and Mobility Ellipsoids](../force-mobility-matrices.html)** — The geometric view of the state-dependent input map that LQR and iLQR exploit - **[Degrees of Freedom and the Curse of Dimensionality](../degrees-of-freedom-and-dimensionality.html)** — Why high-dimensional mechanical systems stay tractable for these local methods :::