In this seminar talk, I will provide an introductory overview of Neural Differential Equations (NDEs), a relatively recent sub-field of Deep Learning that connects the fields of mathematical modeling of dynamical systems and machine learning. Our discussion will primarily be based on the initial chapters of Dr. Patrick Kidger’s PhD thesis [1] and the foundational paper by Chen et al. [2] on Neural Ordinary Differential Equations (NODEs), which lays the groundwork for understanding NDEs in general. I will begin by explaining the rationale behind incorporating neural networks into dynamical systems modeled by differential equations. We will then explore connections with several existing neural architectures that bridge deep learning and dynamical modeling. After that, we will briefly examine some neural architectures that follow the NDE framework [e.g., 3-5]. Additionally, we will investigate the Universal Approximation Theorems related to NDEs and the conditions required for their application, as well as discuss appropriate parameterization choices. Finally, I will conclude with practical considerations for implementing NDE networks.
Additional resources:
- On Neural Differential Equations (Ch. 1 & 2) - https://arxiv.org/abs/2202.02435
- Neural Ordinary Differential Equations - https://arxiv.org/abs/1806.07366
- Universal Differential Equations for Scientific Machine Learning - https://arxiv.org/abs/2001.04385
- Hamiltonian Neural Networks - https://arxiv.org/abs/1906.01563
- Lagrangian Neural Networks - https://arxiv.org/abs/2003.04630
- Augmented Neural ODEs - https://arxiv.org/abs/1904.01681