Introduction

Deep Learning and Differential equations both have a similar place in modelling non-linear functions.
Differential equations describe the system in the language of mechanism, identifying factors that represents the change in the system. The examples range describing Physical world, chemical reactions, economics, finance and so on. Over the years, analytical techniques as well as computational programs have been designed to solve differential equations.
Deep learning is a rather recent subject, where computationally built models approximate a given function. They shine in places where identifying underlying mechanism is rather difficult. They have wide applications in the tasks otherwise intractable, like image recognition, speech recognition, text translation and so on.
The success of Deep learning is attributed to libraries like TensorFlow, PyTorch, Flux etc. that can calculate the derivatives of the functions more accurately and faster. The methods they employ is Automatic Differentiation which escapes floating point errors by using dual numbers which helps addition between numbers of different orders like x + \epsilon immune to shifting. And they have differntiation of primitive functions encoded in them, and derivative of composite functions is done by applying chain-rule repeatedly. There are two main methods Forward-mode AD and Reverse Mode AD, wich differ in application of chain-rule, and one is better than other depending on the cardinality of the dimensions of domain and range. We will skip the details here but are easily accessible from these lecture notes.
The advantage of differential equation models is that they are descriptive and interpretive, and thus are safer even with noisy data or lack of data but they can be very hard to describe and solve.
The advantage of Deep learning models is that they can do very well without any prior knowledge of the system but they are hard to interpret and are prone to serious mismodelling where the data is unseen.
In the recent years, there have been many studies observing relations between these two which are very insightful and is a very active area of research, progressing both scientific computing as well as Machine learning. The aim of this report is an exploration of these ideas.
 
badge