A library for differentiable nonlinear optimization
Paper • Blog • Webpage • Tutorials • Docs
Theseus is an efficient application-agnostic library for building custom nonlinear optimization layers in PyTorch to support constructing various problems in robotics and vision as end-to-end differentiable architectures.
Differentiable nonlinear optimization provides a general scheme to encode inductive priors, as the objective function can be partly parameterized by neural models and partly with expert domain-specific differentiable models. The ability to compute gradients end-to-end is retained by differentiating through the optimizer which allows neural models to train on the final task loss, while also taking advantage of priors captured by the optimizer.
Our implementation provides an easy to use interface to build custom optimization layers and plug them into any neural architecture. Following differentiable features are currently available:
- Second-order nonlinear optimizers
- Gauss-Newton, Levenberg–Marquardt
- Linear solvers
- Dense: Cholesky, LU; Sparse: CHOLMOD, LU
- Commonly used costs, AutoDiffCostFunction
- Lie groups
- Robot kinematics
We support several features that improve computation times and memory consumption:
- Sparse linear solvers
- Batching and GPU acceleration
- Automatic vectorization
- Backward modes: Implicit, Truncated, Direct Loss Minimization (DLM), Sampling (LEO)
- We strongly recommend you install Theseus in a venv or conda environment with Python 3.7-3.9.
- Theseus requires
torch
installation. To install for your particular CPU/CUDA configuration, follow the instructions in the PyTorch website. - For GPU support, Theseus requires nvcc to compile custom CUDA operations. Make sure it matches the version used to compile pytorch with
nvcc --version
. If not, install it and ensure its location is on your system's$PATH
variable. - Theseus also requires
suitesparse
, which you can install via:sudo apt-get install libsuitesparse-dev
(Ubuntu).conda install -c conda-forge suitesparse
(Mac).
pip install theseus-ai
We currently provide wheels with our CUDA extensions compiled using CUDA 10.2 and Python 3.9. For other CUDA versions, consider installing from source or using our build script.
git clone https://github.com/facebookresearch/theseus.git && cd theseus
pip install -e .
If you are interested in contributing to Theseus, instead install
pip install -e ".[dev]"
and follow the more detailed instructions in CONTRIBUTING.
pytest theseus
By default, unit tests include tests for our CUDA extensions. You can add the option -m "not cudaext"
to skip them when installing without CUDA support.
Simple example. This example is fitting the curve Objective
with a single CostFunction
that computes the residual Objective
and the GaussNewton
optimizer are encapsulated into a TheseusLayer
. With RMSprop
and MSE loss, TheseusLayer
.
import torch
import theseus as th
x_true, y_true, v_true = read_data() # shapes (1, N), (1, N), (1, 1)
x = th.Variable(torch.randn_like(x_true), name="x")
y = th.Variable(y_true, name="y")
v = th.Vector(1, name="v") # a manifold subclass of Variable for optim_vars
def error_fn(optim_vars, aux_vars): # returns y - v * exp(x)
x, y = aux_vars
return y.tensor - optim_vars[0].tensor * torch.exp(x.tensor)
objective = th.Objective()
cost_function = th.AutoDiffCostFunction(
[v], error_fn, y_true.shape[1], aux_vars=[x, y],
cost_weight=th.ScaleCostWeight(1.0))
objective.add(cost_function)
layer = th.TheseusLayer(th.GaussNewton(objective, max_iterations=10))
phi = torch.nn.Parameter(x_true + 0.1 * torch.ones_like(x_true))
outer_optimizer = torch.optim.RMSprop([phi], lr=0.001)
for epoch in range(10):
solution, info = layer.forward(
input_tensors={"x": phi.clone(), "v": torch.ones(1, 1)},
optimizer_kwargs={"backward_mode": th.BackwardMode.IMPLICIT})
outer_loss = torch.nn.functional.mse_loss(solution["v"], v_true)
outer_loss.backward()
outer_optimizer.step()
See tutorials, and robotics and vision examples to learn about the API and usage.
If you use Theseus in your work, please cite the paper with the BibTeX below.
@article{pineda2022theseus,
title = {{Theseus: A Library for Differentiable Nonlinear Optimization}},
author = {Luis Pineda and Taosha Fan and Maurizio Monge and Shobha Venkataraman and Paloma Sodhi and Ricky Chen and Joseph Ortiz and Daniel DeTone and Austin Wang and Stuart Anderson and Jing Dong and Brandon Amos and Mustafa Mukadam},
journal = {arXiv preprint arXiv:2207.09442},
year = {2022}
}
Theseus is MIT licensed. See the LICENSE for details.
- Join the community on Github Discussions for questions and sugesstions.
- Use Github Issues for bugs and features.
- See CONTRIBUTING if interested in helping out.
Theseus is made possible by the following contributors:
Made with contrib.rocks.