Machine Learning and Neural Networks
Machine Learning and Neural Networks
Scope of This Chapter
This chapter focuses on using neural networks to learn control policies and approximate inverse models in systems where analytical physics is intractable. Specifically:
What IS covered: How to use function approximation and neural networks to map observed states directly to control outputs, learning from interaction data or reward signals rather than hand-derived equations. This includes policy learning via Reinforcement Learning and supervised learning of inverse dynamics models. This approach is practical for complex, uncertain, or high-dimensional systems.
What IS NOT covered: Learning the forward dynamics model (i.e., learning the function mapping states and actions to next states). While related and equally important, forward model learning is treated separately in Volume I under dynamics learning and model-based RL. This chapter assumes you will use the recursive algorithms from Chapter 9 when you need to predict future motion from known dynamics; here, we focus on learning what to command, not what will happen.
With this scope in mind, we explore neural networks as tools for discovering control laws in complex systems.
The Limits of Analytical Control
By utilizing the Spatial Algebra, Exponential Coordinates, and Recursive Algorithms established in the preceding chapters, we can write a perfect mathematical model of a purely rigid robotic system.
But what if the system isn’t rigid? What if we are modeling the 15-DOF golf swing of an athlete with deformable muscles, unpredictable wind friction, and complex contact dynamics with the ground? The mathematical model \(M(\bm{q})\ddot{\bm{q}} = \bm{\tau} - C(\dot{\bm{q}}, \bm{q}) - G(\bm{q})\) becomes inherently flawed. We do not know the exact friction. We do not know the exact mass of the golfer’s shirt.
When analytical equations fail due to extreme nonlinear complexity or uncertainty, we pivot from deriving the control laws to learning them.
In the context of dynamics and control, neural networks function as universal function approximators—they learn mappings from data rather than from analytically derived equations.
If we do not know the exact mathematical function mapping “Current Posture” to “Correct Motor Torque,” we create a blank, highly flexible mathematical structure (a Neural Network) and slowly bend it, tweak it, and shift it until it accurately approximates the unseen, perfect function.
Neural Networks as Function Approximators
At its core, a Neural Network is a series of matrix operations passed through nonlinear functions.
Suppose we want a control policy \(\pi\) that takes our current state vector \(\state\) (position and velocity) and outputs a control torque \(\control\): \[\begin{equation} \control = \pi(\state) \end{equation}\]
A simple “feedforward” neural network approximates this function \(\pi\) using layers of “neurons.” A deep neural network translates the state vector \(\state \in \Reals^n\) through weight matrices \(W\) and bias vectors \(b\): \[\begin{align} h_1 &= \sigma(W_1 \state + b_1) \\ h_2 &= \sigma(W_2 h_1 + b_2) \\ \control &= W_3 h_2 + b_3 \end{align}\]
Where \(\sigma\) is an activation function (like ReLU or Tanh) which introduces nonlinearity, preventing the entire network from collapsing into just a single linear matrix multiplication.
Backpropagation and Gradient Descent
How does the network “learn” the correct weights (\(W_1, W_2, \dots\)) to control the robot? We define a Loss Function \(\mathcal{L}\), which mathematically penalizes the network for making mistakes. If the network commands the robot to punch a wall, the loss \(\mathcal{L}\) spikes massively.
Using basic calculus (the Chain Rule), we can evaluate the derivative of the Loss function with respect to every single weight matrix \(W\) in the network. This is called the Gradient (\(\nabla_W \mathcal{L}\)). The Gradient simply points uphill towards higher error.
We update the weights by stepping gently downhill (the opposite direction of the gradient) scaled by a Learning Rate \(\alpha\): \[\begin{equation} W_{new} = W_{old} - \alpha \nabla_W \mathcal{L} \end{equation}\] This incredibly robust, iterative calculus calculation is called Backpropagation (Backprop).
Reinforcement Learning (RL)
In Supervised Learning, a human must constantly provide the “correct” answer to calculate the gradient. But in robotic control, we often do not know the correct joint torques for a 15-DOF golf swing.
Reinforcement Learning (RL) solves this by allowing the agent to explore physics entirely on its own. The network commands a torque \(\control\) at state \(\state\), observes the new state \(\state'\), and receives a scalar Reward \(R\) (e.g., +100 for hitting the ball, -10 for falling over).
RL algorithms seek to learn the Value Function \(V(\state)\), which predicts the total sum of all future rewards the robot will receive if it starts in state \(\state\). By maximizing the Value Function, the neural network organically discovers complex behaviors—often outperforming classical control theory because it fully exploits passive geometric dynamics instead of fighting them.
Worked Example: Forward Pass in Python
Implementing a Neural Network does not require massive libraries, just standard linear algebra. Here is the mathematical Forward Pass of a policy \(\pi(\state)\) mapping a 4-dimensional state (e.g., an inverted pendulum) to a 1-dimensional control torque using .
import numpy as np
def relu(x):
# Nonlinear Activation Function
return np.maximum(0, x)
def neural_network_policy(state, W1, b1, W2, b2):
"""
Computes a forward pass mapping State -> Action
State: np.array of shape (4, 1)
"""
# Layer 1: Linear Transform + Activation
z1 = np.dot(W1, state) + b1
h1 = relu(z1)
# Layer 2: Linear Transform -> Output Torque
z2 = np.dot(W2, h1) + b2
control_torque = z2
return control_torque
# Example: A tiny 4 -> 8 -> 1 Network Architecture
state_space_dim = 4
hidden_layer_dim = 8
action_space_dim = 1
# Randomly initialized matrices (in reality, optimized by Backprop)
W1 = np.random.randn(hidden_layer_dim, state_space_dim) * 0.1
b1 = np.zeros((hidden_layer_dim, 1))
W2 = np.random.randn(action_space_dim, hidden_layer_dim) * 0.1
b2 = np.zeros((action_space_dim, 1))
# Current State [Position, Angle, Velocity, Angular Velocity]^T
current_state = np.array([[0.0], [0.1], [0.0], [0.0]])
# Ask the network for a command
commanded_u = neural_network_policy(current_state, W1, b1, W2, b2)
print("Commanded Torque from Neural Network policy:", np.round(commanded_u[0, 0], 4))This code block demonstrates exactly how Reinforcement Learning algorithms map physics. Millions of times per second, matrices multiply vectors and pass through ReLU functions to output physical torques inside a physics simulation (such as the companion platform’s MuJoCo integration).
With the mathematics of Reinforcement Learning, Spatial Algebra, Recursive Dynamics, and Configuration Manifolds securely established, you are now completely prepared to tackle the advanced control theory in Volume I and Volume II of Tangent-Space Methods.