Calculus

Derivatives and Gradients

Derivatives

• Definition of derivative
• Geometric interpretation
• Rules of differentiation (sum, product, quotient, chain rule)
• Common derivatives (polynomials, exponentials, logarithms, trigonometric)
• Higher-order derivatives

Applications

• Finding maxima and minima
• Optimization problems
• Rate of change
• Tangent lines

Partial Derivatives

Multivariable Functions

• Functions of multiple variables
• Partial derivatives
• Computing partial derivatives
• Higher-order partial derivatives
• Mixed partial derivatives

Gradient

• Definition of gradient vector
• Computing gradients
• Gradient as direction of steepest ascent
• Gradient descent algorithm

Chain Rule

Single Variable Chain Rule

• Composition of functions
• Chain rule statement
• Applications

Multivariable Chain Rule

• Chain rule for multiple variables
• Total derivative
• Applications in backpropagation

Optimization Basics

Unconstrained Optimization

• Critical points
• First derivative test
• Second derivative test
• Global vs local optima

Constrained Optimization

• Lagrange multipliers
• Karush-Kuhn-Tucker (KKT) conditions
• Applications in machine learning

Gradient-Based Methods

• Gradient descent
• Stochastic gradient descent
• Momentum
• Adam optimizer

Practice Questions

Test your understanding with the questions in the interactive quiz below (when available).

Resources

• "Calculus" by Stewart
• Khan Academy Calculus: https://www.khanacademy.org/math/calculus-1
• 3Blue1Brown Essence of Calculus: https://www.3blue1brown.com/topics/calculus
• Automatic Differentiation: https://en.wikipedia.org/wiki/Automatic_differentiation