OCaml Scientific Computing Tutorials

• Prologue

  • A Brief History

  • Reductionism vs. Holism

  • Key Features

  • Contact Me

Part 1: Numerical Techniques

• 1. Introduction

  • What Is Scientific Computing

  • What is Functional Programming

  • Who Is This Book For

  • Structure of the Book

  • Installation

    • Option 1: Install from OPAM
    • Option 2: Pull from Docker Hub
    • Option 3: Pin the Dev-Repo
    • Option 4: Compile from Source
    • CBLAS/LAPACKE Dependency

  • Interacting with Owl

    • Using Toplevel
    • Using Notebook
    • Using Owl-Jupyter

  • Summary

• 2. Conventions

  • Pure vs. Impure

  • Ndarray vs. Scalar

  • Infix Operators

  • Operator Extension

  • Module Structures

  • Number and Precision

  • Polymorphic Functions

  • Module Shortcuts

  • Type Casting

• 3. Visualisation

  • Create Plots

  • Specification

  • Subplots

  • Multiple Lines

  • Legend

  • Drawing Patterns

  • Line Plot

  • Scatter Plot

  • Stairs Plot

  • Box Plot

  • Stem Plot

  • Area Plot

  • Histogram & CDF Plot

  • Log Plot

  • 3D Plot

  • Advanced Statistical Plot

  • Summary

  • References

• 4. Mathematical Functions

  • Basic Functions

    • Basic Unary Math Functions
    • Basic Binary Functions
    • Exponential and Logarithmic Functions
    • Trigonometric Functions
    • Other Math Functions

  • Special Functions

    • Airy Functions
    • Bessel Functions
    • Elliptic Functions
    • Gamma Functions
    • Beta Functions
    • Struve Functions
    • Zeta Functions
    • Error Functions
    • Integral Functions

  • Factorials

  • Interpolation and Extrapolation

  • Integration

  • Utility Functions

  • Summary

• 5. Statistical Functions

  • Random Variables

    • Discrete Random Variables
    • Continuous Random Variables
    • Descriptive Statistics

  • Special Distribution

  • Multiple Variables

  • Sampling

    • Unbiased Estimator
    • Inferring Population Parameters

  • Hypothesis Tests

    • Theory
    • Gaussian Distribution in Hypothesis Testing
    • Two-Sample Inferences
    • Other Types of Test

  • Covariance and Correlations

  • Summary

  • References

• 6. N-Dimensional Arrays

  • Ndarray Types

  • Creation Functions

  • Properties Functions

  • Map Functions

  • Fold Functions

  • Scan Functions

  • Comparison Functions

  • Vectorised Functions

  • Iteration Functions

  • Manipulation Functions

  • Serialisation

  • Tensors

  • Summary

  • References

• 7. Slicing and Broadcasting

  • Slicing

    • Basic Slicing
    • Fancy Slicing
    • Conventions in Definition
    • Extended Operators
    • Advanced Usage

  • Broadcasting

    • What Is Broadcasting?
    • Shape Constraints
    • Supported Operations

  • Slicing in NumPy and Julia

  • Internal Mechanism

  • Summary

• 8. Linear Algebra

  • Vectors and Matrices

    • Creating Matrices
    • Accessing Elements
    • Iterate, Map, Fold, and Filter
    • Math Operations

  • Gaussian Elimination

    • LU Factorisation
    • Inverse and Transpose

  • Vector Spaces

    • Rank and Basis
    • Orthogonality
    • Solving Ax = b
    • Matrix Sensitivity

  • Determinants

  • Eigenvalues and Eigenvectors

    • Solving \(Ax=\lambda~x\)
    • Complex Matrices
    • Similarity Transformation and Diagonalisation

  • Positive Definite Matrices

    • Positive Definiteness
    • Singular Value Decomposition

  • Internal: CBLAS and LAPACKE

    • Low-level Interface to CBLAS & LAPACKE

  • Sparse Matrices

  • Summary

  • References

• 9. Ordinary Differential Equations

  • What Is An ODE

    • Exact Solutions
    • Linear Systems

  • Solving An ODE Numerically

  • Owl-ODE

    • Example: Linear Oscillator System
    • Solver Structure
    • Symplectic Solvers
    • Features and Limits

  • Examples of using Owl-ODE

    • Explicit ODE
    • Two Body Problem
    • Lorenz Attractor
    • Damped Oscillation

  • Stiffness

    • Solve Non-Stiff ODEs
    • Solve Stiff ODEs

  • Summary

  • References

• 10. Signal Processing

  • Discrete Fourier Transform

  • Fast Fourier Transform

    • Examples

  • Applications of FFT

    • Find period of sunspots
    • Decipher the Tone
    • Image Processing

  • Filtering

    • Example: Smoothing
    • Gaussian Filter
    • Signal Convolution
    • FFT and Image Convolution

  • Summary

  • References

• 11. Algorithmic Differentiation

  • Chain Rule

  • Differentiation Methods

  • How Algorithmic Differentiation Works

    • Forward Mode
    • Reverse Mode
    • Forward or Reverse?

  • A Strawman AD Engine

    • Simple Forward Implementation
    • Simple Reverse Implementation
    • Unified Implementations

  • Forward and Reverse Propagation API

    • Expressing Computation
    • Example: Forward Mode
    • Example: Reverse Mode

  • High-Level APIs

    • Derivative and Gradient
    • Jacobian
    • Hessian and Laplacian
    • Other APIs

  • Internal of Algorithmic Differentiation

    • Go Beyond Simple Implementation
    • Extend AD module
    • Lazy Evaluation

  • Summary

  • References

• 12. Optimisation

  • Introduction

  • Root Finding

  • Univariate Function Optimisation

    • Use Derivatives
    • Golden Section Search

  • Multivariate Function Optimisation

    • Nelder-Mead Simplex Method
    • Gradient Descent Methods
    • Conjugate Gradient Method
    • Newton and Quasi-Newton Methods

  • Global Optimisation and Constrained Optimisation

  • Summary

  • References

• 13. Regression

  • Linear Regression

    • Problem: Where to locate a new McDonald’s restaurant?
    • Cost Function
    • Solving Problem with Gradient Descent

  • Multiple Regression

    • Feature Normalisation
    • Analytical Solution

  • Non-linear regressions

  • Regularisation

    • Ols, Ridge, Lasso, and Elastic_net

  • Logistic Regression

    • Sigmoid Function
    • Cost Function
    • Example
    • Multi-class classification

  • Support Vector Machine

    • Kernel and Non-linear Boundary
    • Example

  • Model error and selection

    • Error Metrics
    • Model Selection

  • Summary

  • References

• 14. Deep Neural Networks

  • Perceptron

  • Yet Another Regression

    • Model Representation
    • Forward Propagation
    • Back propagation

  • Feed Forward Network

    • Layers
    • Activation Functions
    • Initialisation
    • Training
    • Test

  • Neural Network Module

    • Module Structure
    • Neurons
    • Neural Graph
    • Training Parameters

  • Convolutional Neural Network

  • Recurrent Neural Network

    • Long Short Term Memory (LSTM)

  • Generative Adversarial Network

  • Summary

  • References

• 15. Natural Language Processing

  • Introduction

  • Text Corpus

    • Step-by-step Operation
    • Use the Corpus Module

  • Vector Space Models

  • Bag of Words (BOW)

  • Term Frequency–Inverse Document Frequency (TF-IDF)

  • Latent Dirichlet Allocation (LDA)

    • Models
    • Dirichlet Distribution
    • Gibbs Sampling
    • Topic Modelling Example

  • Latent Semantic Analysis (LSA)

  • Search Relevant Documents

    • Euclidean and Cosine Similarity
    • Linear Searching

  • Summary

  • References

• 16. Dataframe for Tabular Data

  • Basic Concepts

  • Create Frames

  • Manipulate Frames

  • Query Frames

  • Iterate, Map, and Filter

  • Read/Write CSV Files

  • Infer Type and Separator

  • Summary

• 17. Symbolic Representation

  • Introduction

  • Design

    • Core abstraction
    • Engines

  • ONNX Engine

    • Example 1: Basic operations
    • Example 2: Variable Initialisation
    • Example 3: Neural network

  • LaTeX Engine

  • Owl Engine

  • Summary

Part 2: System Architecture

• 18. Architecture Overview

  • Introduction

  • Architecture Overview

  • Core Implementation

    • N-dimensional Array
    • Interfaced Libraries

  • Advanced Functionality

    • Computation Graph
    • Algorithmic Differentiation
    • Regression
    • Neural Network

  • Parallel Computing

    • Actor Engine
    • GPU Computing
    • OpenMP

  • Community-Driven R&D

  • Summary

• 19. Core Optimisation

  • Background

    • Numerical Libraries
    • Optimisation of Numerical Computation

  • Interfacing to C Code

    • Ndarray Operations
    • From OCaml to C

  • Optimisation Techniques

    • Map Operations
    • Convolution Operations
    • Reduction Operations
    • Repeat Operations

  • References

• 20. Automatic Empirical Tuning

  • What is Parameter Tuning

  • Why Parameter Tuning in Owl

  • How to Tune OpenMP Parameters

  • Make a Difference

  • Summary

• 21. Computation Graph

  • Introduction

    • What is a Computation Graph?
    • From Dynamic to Static
    • Significance in Computing

  • Examples

    • Example 01: Basic CGraph
    • Example 02: CGraph with AD
    • Example 03: CGraph with DNN

  • Design Rationale

  • Optimisation of CGraph

    • Optimising memory with pebbles
    • Allocation Algorithm

  • As Intermediate Representations

  • Summary

• 22. Scripting and Zoo System

  • Introduction

  • Share Script with Zoo

    • Typical Scenario
    • Create a Script
    • Share via Gist
    • Import in Another Script
    • Select a Specific Version
    • Command Line Tool
    • More Examples

  • System Design

    • Services
    • Type Checking
    • Backend
    • Domain Specific Language
    • Service Discovery
    • Use Case

  • Summary

  • References

• 23. Compiler Backends

  • Base Library

  • Backend: JavaScript

    • Use Native OCaml
    • Use Facebook Reason

  • Backend: MirageOS

    • MirageOS and Unikernel
    • Example: Gradient Descent
    • Example: Neural Network

  • Evaluation

  • Summary

• 24. Distributed Computing

  • Actor System

    • Design
    • Actor Engines
    • Map-Reduce Engine
    • Parameter Server Engine
    • Peer-to-Peer Engine

  • Classic Synchronise Parallel

    • Bulk Synchronous Parallel
    • Asynchronous Parallel
    • Stale Synchronous Parallel

  • Probabilistic Synchronise Parallel

    • Basic idea: sampling
    • Compatibility
    • Barrier Trade-off Dimensions
    • Convergence

  • A Distributed Training Example

    • Step Progress
    • Accuracy

  • Summary

  • References

• 25. Testing Framework

  • Unit Test

  • Example

  • What Could Go Wrong

    • Corner Cases
    • Test Coverage

  • Use Functor

  • Summary

• 26. Constants and Metric System

  • What Is a Metric System

  • Four Metric Systems

  • SI Prefix

    • Example: Physics and Math constants

  • International System of Units

    • Time
    • Length
    • Area
    • Volume
    • Speed
    • Mass
    • Force
    • Energy
    • Power
    • Pressure
    • Viscosity
    • Luminance
    • Radioactivity

• 27. Internal Utility Modules

  • Dataset Module

    • MNIST
    • CIFAR-10

  • Graph Module

  • Stack and Heap Modules

  • Count-Min Sketch

  • Summary

Part 3: Case Studies

• 28. Case - Image Recognition

  • Background

    • LeNet
    • AlexNet
    • VGG
    • ResNet
    • SqueezeNet
    • Capsule Network

  • Building InceptionV3 Network

    • InceptionV1 and InceptionV2
    • Factorisation
    • Grid Size Reduction
    • InceptionV3 Architecture

  • Preparing Weights

  • Processing Image

  • Running Inference

  • Applications

  • Summary

  • References

• 29. Case - Instance Segmentation

  • Introduction

  • Mask R-CNN Network

  • Building Mask R-CNN

    • Feature Extractor
    • Proposal Generation
    • Classification

  • Run the Code

  • Summary

  • References

• 30. Case - Neural Style Transfer

  • Content and Style

    • Content Reconstruction
    • Style Recreation
    • Combining Content and Style
    • Running NST

  • Extending NST

  • Fast Style Transfer

    • Building FST Network
    • Running FST

  • Summary

  • References

• 31. Case - Recommender System

  • Introduction

  • Architecture

  • Build Topic Models

  • Index Text Corpus

    • Random Projection
    • Optimising Vector Storage
    • Optimise Data Structure
    • Optimise Index Algorithm

  • Search Articles

  • Code Implementation

  • Make It Live

  • Summary

  • References