Resources

Resources

This page lists other online resources related to ideas in deep declarative networks, implicit differentiation and differentiable optimization problems. Email us if you know of other resources that should be listed here.

Code:

• OptNet (Brandon Amos and J. Zico Kolter): enables the use of quadratic programs as layers of an end-to-end deep network [pdf]
• Kornia (Riba et al.): open source differentiable computer vision library for PyTorch [github]
• CVXPyLayers (Agrawal, Amos, Barrett, Boyd, Diamond and Kotler): differentiable convex optimization layers
• Differentiation of Blackbox Combinatorial Solvers (Vlastelica, Paulus, et al.): torch modules that wrap blackbox combinatorial solvers [pdf]
• Deep Equilibrium Models: code for the deep equilibrium (DEQ) model, an implicit-depth architecture proposed in the paper Deep Equilibrium Models by Shaojie Bai, J. Zico Kolter and Vladlen Koltun
• pytorCH OPtimize (Negiar and Pedregosa): a library for continuous and constrained optimization built on PyTorch
• BPnPNet (Campbell et al.): codebase for solving the blind perspective-n-point problem end-to-end with robust differentiable geometric optimization pdf
• jaxOpt (Blondel et al.): Hardware accelerated (GPU/TPU), batchable and differentiable optimizers in JAX.
• Betty (Choe et al.): An automatic differentiation library for generalized meta-learning and multilevel optimization.
• Theseus (Pineda et al.): A library for differentiable nonlinear optimization.
• DiffSort (Petersen et al.): Differentiable sorting networks.
• Difftopk (Petersen et al.): Differentiable top-k classification learning.

Online Tutorials and Lectures:

• DDN Tutorials: tutorials from the deep declarative networks github repository.
• ISAAC Lecture Notes (Stephen Gould): lecture notes and slides on differentiable optimisation and deep learning from the ISAAC 2022 summer school.
• Implicit Functions in PyTorch (Thomas Viehmann): a tutorial on implementing implicit functions as a PyTorch layer.
• CvxPyLayers: differentiable convex optimization layers.

Workshops and Tutorials:

• CVPR 2020 Workshop on Deep Declarative Networks held June 2020 in Seattle, Washington.
  • talks from the workshop are available on this YouTube playlist.
• ECCV 2020 Tutorial on DDNs and Differentiable Optimization held August 2020 in Glasgow, Scotland.
  • talks from the tutorial are available on this YouTube playlist.
• NeurIPS 2020 Tutorial: There and Back Again: A Tale of Slopes and Expectations by Marc Deisenroth and Cheng Soon Ong held December 2020, held virtually.
• NeurIPS 2020 Tutorial: Deep Implicit Layers by Zico Kolter, David Duvenaud and Matt Johnson held December 2020, held virtually.

Books:

• A. L. Dontchev and R. T. Rockafellar. Implicit Functions and Solution Mappings: A View from Variational Analysis. Springer-Verlag, 2014.
• S. G. Krantz and H. R. Parks. The Implicit Function Theorem: History, Theory, and Applications. Springer, 2013.
• D. P. Bertsekas. Nonlinear Programming. Athena Scientific, 2004.
• S. P. Boyd and L. Vandenberghe. Convex Optimization. Cambridge, 2004.