Deep learning theory lecture notes [Telgarsky]

[mklabgr]

Deep learning theory lecture notes 
BY Matus Telgarsky

Summary

[b]Philosophy of these notes. Two key ideas determined what has been included so far.[/b]

1. I aim to provide simplified proofs over what appears in the literature, ideally reducing difficult things to something that fits in a single lecture.
   
2. I have primarily focused on a classical perspective of achieving a low test error for binary classification with IID data via standard (typically ReLU) feedforward networks.
   

[b]Organization. [/b]
Following the second point above, the classical view decomposes the test error into three parts.

1. [b]Approximation (starts in section 1): given a classification problem, there exists a deep network which achieves low error over the distribution.[/b]
   
2. [b]Optimization (starts in section 6): given a finite training set for a classification problem, there exist algorithms to find predictors with low training error and low complexity.[/b]
   
3. [b]Generalization (starts in section 11): the gap between training and testing error is small for low complexity networks.[/b]
   

BOOK PAGE