Notes.txt

Conditional Probability:
P(a|b) = P(a ^ b) / P(b)
P(a ^ b) = P(a|b)P(b)
P(a ^ b) = P(b|a)P(a)

P(a|b)P(b) = P(b|a)P(a)
P(a|b) = P(b|a)P(a) / P(b) <= Bayes' Rule
P(b|a) = P(a|b)P(b) / P(a) <= Bayes' Rule

Bayes' Rule: 
Knowing - P(visible effect|unknown cause)
We can conclude - P(unknown cause | visible effect)

Example:
Knowing - P(medical test result | disease)
We can conclude - P(disease | medical test result)

If Probability of 'a' and 'b' are independent,
P(a ^ b) = P(b)P(a)

P(a∣b) = P(b∣a)P(a) / P(b)

Bayes rule/formula:
Prior Odds: 5:95
Likelihood Ratio: P(A|B) / P(A|C) : (80/100) / (10/100) = (80/100) * (100/10) = 80/10 = 8
Posterior Odds: 40:95

25% of the positive tweets contain the word "happy". 13% of the tweets contain the word "happy". 40% of the total #tweets are positive. What's the probability of "happy to learn NLP" being positive?
Knowing - P(visible effect|unknown cause) : P(happy|positive) = .25
P(happy) = .13
P(positive) = .4
P(unknown cause | visible effect) : P(positive|happy) = P(happy|positive) * P(positive) / P(happy) = .25 * .4 / .13 = .769 = 77%

Prior Ratio = P(pos) / P(neg)
- Important only in unbalanced dataset since the division is not 1.

Naive Bayes:
- Supervised Learning.
- Shares many similarities with Logistic Regression.
- "Naive" because it makes assumptions about:
  (1) The features being independent.
  (2) Relative frequencies in the corpora.
      - Training dataset is artifically balanced. In reality, the data are not balanced and much noisier.
- Naive Bayes inference condition rule for binary classification: P(X(i)|pos)/P(X(i)|neg) multiplied for all i.
  - Positive if > 1. Negative if < 1. Neutral if 1.
- P(pos) / P(neg) * P(X(i)|pos)/P(X(i)|neg) is the full Naive Bayes formula for binary classification.
- A simple, fast and powerful method to establish a baseline quickly.
- A probabilistic model used for classification.
- To avoid numerical underflow due to multiplication of floating point values < 1, use Log Likelihood.
  - lambda = log(P(Xi|pos)/P(Xi|neg))
  - log(P(pos)/P(neg)) + sum(log(P(Xi|pos)/P(Xi|neg)))
  - Positive if > 0. Negative if < 0. Neutral if 0.
- Example use cases:
  - Sentiment analysis
  - Author identification.
  - Information retrieval.
    - P(document[i] | query)
  - Word disambiguation.
    - P(context1|text) / P(context2|text)

Laplacian Smoothing:
- Used to avoid 0 conditional probability.
- P(X(i)|class) = (freq(X(i), class) + 1) / (Nclass + N); N = total #samples in the dataset, Nclass = frequency of all X in the class.

Joint Probability:
P(A|b) = P(A, b) / P(b) : Comma signifies ^
       = alpha * P(A, b), where alpha = 1/P(b) which is just some constant.
       = alpha * <Probability distribution of 'b' which sums up to 1>
=> Conditional probability is proportional to Joint probability

P(A|b) = P(A, b) / P(b)
=> P(A, b) = P(b)P(A|b)
P(a,b,c,d) = P(d|a,b,c) * P(c|a,b) * P(b|a) * P(a)
The probability of each successive token is the product of the probabilities of all the precideing tokens.

Inclusion-Exclusion:
P(a v b) = P(a) + P(b) - P(a^b)

Marginalization: Calculate probability of independent variable based on given/known joint probabilities.
P(a) = P(a,b) + P(a, Not(b))
P(X = xi) = Summation over all the values (j) y could take on of P(X = xi, Y = yj)

Conditioning: Calculate probability of independent variable based on given/known conditional probabilities.
P(a) = P(a|b)P(b) + P(a|Not(b))P(Not(b))
P(X = xi) = Summation over all the values (j) y could take on of P(X = xi|Y = yj)P(Y = yj)

Bayesian Network: Data structure which represents the dependencis among random variables.
- Directed graph
- Each node represents a random variable.
- Arrow from X to Y means X is a parent of Y
- Each node X has a probability distrubution P(X | Parents(X))

Inference:
- Query X: variable for which to compute distribution
- Evidence variables E: observed variables for event e
- Hidden variables Y: non-evidence and non-query variable
- Goal: Calculate P(X | e) where e could be more than 1.

Inference by Enumeration:
P(X | e) = alpha * P(X,e) = alpha * Summation over all the values (j) y could take on of P(X, e, y)
X: query variable
e: evidence
y: ranges over values of hidden variables
alpha: normalizes the result.
Cons: Time complexity grows with complexity of the network

Aproximate Inference:
- Sampling:
  - Rejection Sampling: Filter out samples which do not match event specification
  - Likelihood Weighting:
    - Start by fixing the values of evidence variables.
    - Sample the non-evidence variables using conditional probabilities in the Bayesian Network
    - Weight each sample by its likelihood: the probability of all of the evidence.

Markov Assumption: The assumption that the current state depends on only a finite fixed number of previous states.
Markov Chain: A sequence of random variables where the distribution of each variable follows the Markov assumption.
- A type of stochastic model that describes a sequence of possible events.
- The probability of each event depends on the states of previous events.
- Can be depicted as a directed graph.
- Transition probabilities and matrix.
Markov Property: The probability of the next event only depends on the current event.
Hidden Markov Model: a Markov model for a system with hidden states which generate/emit some observed events.
- Markov model with states which are not directly observable.
- Emission probabilities and matrix.
Sensor Markov Assumption: the assumption that the evidence variable depends only on corresponding state.
Use cases:
(1) Part-of-speech (POS) tagging.

Supervised Learning > Classification:
Perceptron Learning Rule: Given data point (x,y), update each weight according to: w =  w + alpha*(y - h(X)) * x
Start with random weights, learn from data and update the weights that result in better weight vector which reduces loss.

Support Vector Machines:
- Maximum Margin Separator: Boundary that maximizes the distance between any of the data points
- If the data is not linearly separable, it works from higher dimension to find the separation boundary. For example, other shapes than linear lines.

Optimization Problem Formulation:
- Local Search: 
  - Algorithms: Hill-Climbing, Simulated Annealing
- Linear Programming
  - Algorithms: 
- Constraint Satisfaction
  - Algorithms: AC3, Backtracking

Supervised Learning > Regression:
Learning a function mapping an input point to a continuous value.

Types of loss functions:
0-1 Loss: Used in discrete classification
L1 Loss: |actual - prediction| -> Used in continuous number prediction. Used when we don't care about outliers.
L2 Loss: (actual - prediction)^2 -> Penalizes worse / bigger loss more harshly. Used when we care about outliers.

Underfitting (High bias):
- J(train) AND J(cv) are high
- The model has a strong preconception of how the data  should fit into itself. For example, linear function, etc.
- More data will NOT help.
- Example: Training dataset performance: 64%, Test dataset performance: 47%

Variance = measure of how well the model can generalize to unseen data.

Overfitting (High variance):
- J(train) is low; J(cv) is high
- A model that fits too closely to a particular data set and therefore may fail to generalize to future data. High variance because if the model overfits, a tiny change in one of the training data set will end up completely different model.
- This is a side effect of minimizing loss of a model. If loss = 0, the model may only work on the specific data set.
- One way to counter this problem is add other parameters to optimization. For example, consider complexity:
- More data is likely to help.
- One of the causes is data leakage - some of test data leak into training data.
- Example: Training dataset performance: 93%, Test dataset performance: 99%
Note: Overfitting is sometimes a good thing to begin with. It means the model is learning.

High bias AND high variance:
- J(train) is high; J(cv) higher than J(train)
- Sometimes happen in NN. Less in linear regression.
- Doesn't really happen for linear models applied to 1-D.
- Underfit and overfit on parts of the data at the same time.

Balanced (Goldilocks zone):
- Example: Training dataset performance: 98%, Test dataset performance: 96%

Cost(h) = Loss(h) + w*Complexity(h) : 'w' gives weight to the complexity

Adding the term w*Complexity(h) is called "Regularization": Penalizing hypotheses that are more complex to favour simpler, more general hypotheses
To overcome overfitting:
(1) Collect more training samples / observations.
(2) Use fewer features.
Many features + insufficient data -> overfitting
(3) Regularization - Gently reduces the impact of some of the features without eliminating them outright. It encourages the model to shrink the values of the parameters without necessarily setting them to 0 which eliminates the features.
    Cost(w,b) = (sum((f_w_b(x) - y) ** 2) + lambda * sum(w[j] ** 2)) / 2m
    f_w_b <= Linear regression modal
    sum((f_w_b(x) - y) ** 2) <= Mean squared error. Fit data
    sum(w[j] ** 2) <= Regularization term. Keep w[j] small
    lambda balances both goals
    Example: f_w_b = w1x + w2x^2 + w3x^3 + w4x^4 + b
    lambda = 0: Overfitting. Wiggly curve/graph which tries to fit every single data point in the training data set.
    lambda = 10^10: f_w_b = b Underfitting

Gradient Descent:
w = w - alpha * (dJ(w) / dw)
Gradient Descent withh regularization:
w = w - alpha/m * (sum((f_w_b(x) - y) * x) + lambda * w[j])
  = w * (1 - alpha * lambda / m) - (alpha / m ) * sum((f_w_b(x) - y) * x)
  alpha: 0.01, lambda: 10, m: 100 => (1 - alpha * lambda / m) = (1 - 0.001) = 0.999
  This shows that every iteration of gradient descent, regularization reduces the value of w, thus reduces the impact of the feature.
- Each iteration/epoch of gradient descent will process the entire X(m,n). Choose a batch size so that:
  (1) Leverage the vectorization processing
  (2) Faster iterations. Large batch size will take too long per iteration/epoch.
  (3) Batch size should be power of 2 and fits in RAM.

Exponentially Weighted Averages
- V(t) = beta * V(t-1) + (1 - beta) * Theta(t)
  - Computes data point with average over 1 / (1 - beta) data points.
  - beta: 0.9 -> Averages over 10t of data
  - beta: 0.98 -> Averages over 50t of data
  - beta: 0.5 -> Averages over 2t of data
  - Iniial warm-up data will be skewed. Use bias correction to mitigate that. V(t) / (1 - beta^t).

Gradient Descent with Momentum:
- Smooth out the oscillations of gradients towards the global minimum.
- Vdw and Vdb initialized to 0, beta usually 0.9 (Average over last 10 gradients). Bias correction can be skipped/ignored.
- Vdw = beta*Vdw + (1-beta)*dW; W -= alpha*Vdw
- Vdb = beta*Vdb + (1-beta)*db; b -= alpha*Vdb
Analogy to a ball rolling down a bowl:
- beta : friction
- Vdw, Vdb : velocity
- Dw, db : acceleration.

RMSProp (Root Mean Squared Prop):
- Sdw and Sdb initialized to 0, beta usually 0.9 (Average over last 10 gradients). Bias correction can be skipped/ignored.
- Use beta2 hyperparameter notationi to distinguish it from the beta used in Momentum.
- Sdw = beta2*Sdw + (1-beta2)*dW^2; W -= alpha*dw/(sqrt(Sdw) + epsilon)
- Sdb = beta2*Sdb + (1-beta2)*db^2; b -= alpha*db/(sqrt(Sdb) + epsilon)
- Either Sdw and/or Sdb could have big oscillations and the division by the sqrt of it smooths out the W and b updates.

Adam (Adaptive Moment Estimation):
- Combines the best of both worlds - Momentum and RMSProp
- Uses bias corrections for all Vdw, Vdb, Sdw and Sdb, i.e., V / (1 - beta^t)
- W -= alpha*Vdw/sqrt(Sdw + epsilon)
- b -= alpha*Vdb/sqrt(Sdb + epsilon)

Learning Rate Decay:
- Big alpha values in early epochs and slowly decreasing with increasing epoch#
- This enables fast learning during the start of the training and better convergence to global minimum with smaller area of oscillations around it.

Strategy to debugging a learning algorithm:
(1) Get more data - Fixes high variance.
(2) Try smaller sets of features (Simplify polynomial equation by removing some of the higher order components.) - Fixes high variance.
(3) Add more features - Fixes high bias
(4) Add polynomial features - Fixes high bias
(5) Decrease lambda - Fixes high bias
(6) Increase lambda - Fixes high variance.

Cross-validation / development dataset is used to evaluate trained models to see which one is the best choice.
Test dataset is used to evaluate the unbiased performance of the final system before deployed to production.
These 2 must use the same distribution of data.

Feature Scaling: When a feature has a large range of values, the weight/parameter calculated by the model will have small range. In this case, a small change in weight will have big change in the cost value.
This causes the gradient descent to take longer time to find it's global minimum, i.e., converge slower. To oversome this, scale the input data. 2 ways:
(1) Divide by the maximum value of the data range. This will produce scaled input data of [0, 1]
(2) Mean normalization. This will produce scaled input data of [-1, 1] which centers around 0.
    - Find the average value: miu = numpy.mean(X)
    - X -= miu
(3) Variance normalization. This will produce features with variance = 1.
    - sigma^2 = numpy.sum(X ** 2) / m
    - OR: sigma = numpy.std(X)
    - X /= sigma
(4) Z-score normalization: Find the standard deviation (d) and the mean. (x - ave) / d.
    Mean: sum(data) / len(data)
    standard deviation: sum((data[i] - mean) ** 2) / len(data)
Note:
- Sigma = standard deviation
- Sigma^2 = Variance.
  - Vairance quantifies the spread of a normal distribution.
  - Covariance measures the variance between 2 dimensions.
    - 0: independent
    - -ve: lobside spread in y=-x direction
    - +ve: lobside spread in y=x direction.

Batch Normalization:
- HxW for each batch of data, x, (x - miu_x) / sigma_x. Do it for every channel.
- NN hidden units' outputs, Z[l], normalized in the same way as the input data X.
  - Z_normalized = (Z - miu) / sqrt(sigma^2 + epsilon)
  - Z~ = gamma * Z_normalized + beta
- parameter b is cancelled out by mean and "- miu" calculation and is replaced with beta[l] in Z~ = gamma[l] * Z_normalized + beta[l]. So it is OK to drop it from forward propagation.
- If gamma = sqrt(sigma^2 + epsilon) and beta = miu, then Z~[i] = Z[i]. So, in a way, this is learning an identty function.
- Gamma abd beta are learnable parameters which help to scale mean and variance out of 0 and 1 to benefit from activation functions.
- Covariate shifts:
  - Different batches of data, especially between traning/validation/test datasets could have different distributions.
  - For example, colour vs. grey-scale images.
  - Changes in the distribution of one variable affect the distribution of other related variables.
- Helps make weights in later layers more resilient to changes in earlier layers due to covariate shifts.
  - For example, training on greyscale images and predict on color pictures.
  - This decoupling effect helps stabilize the learning process to have more firm ground on the truth functions, f(Y|X).
  - Smooths the cost function and speeds up learning.
- Provides a regularization side effect since it usually works on batches of data.
  - The mean and standard deviation/variance calculated on batches of data introduce noise similar to dropout.
- For test and prediction, which do NOT have/use batches of input, use exponential weighted average of gamma and beta.
- Diff between training and testing time:
  - The batch mean and standard deviation are used for training while the running average statistics from all batches (computed over the entire training dataset) are used for testing.
  - The running values are fixed after training.
Rules of thumb: Aim for [-1, 1] input data range. These need to rescale:
(1) [-100 , 100] : Too large
(2) [-0.001, 0.001] : Too small
(3) [98.6, 105]: Too large. Example, Fahrenheight body temperature.
Notes:
Feature scaling usually isn't required for your target variable (labels).
Feature scaling is usually not required with tree-based models (e.g. Random Forest) since they can handle varying features.
Amount of data used must measure up to the #parameters to fit or train on.

Reinforcement Learning:
Given a set of rewards or penalties, learn what actions to take in the future.
Markov Decision Process:
Model for decision-making, representing states, actions, and their rewards.
Q-Learning:
Method for learning a function Q(s,a), estimate of the value (reward) of performing action 'a' in state 's'. Start with Q(s,a) = 0 for all s,a.
Q(s,a) <- Q(s,a) + w*(new estimate - old estimate): w:0 old values are more important; w:1 new values are more important.
Q(s,a) <- Q(s,a) + w*((r + future reward estimate) - Q(s,a)) 
Q(s,a) <- Q(s,a) + w*((r + w1*max(Q(s',a') for all a')) - Q(s,a)): w1 provides weights for future vs current reward

Balance between Exploration and Exploitation.

Function Approximation:
Approximating Q(s,a), often by a function combining various features, rathen than storing one value for every state-action pair.
Similar to depth-limiting approach in MiniMax. Used when it is not feasible to explore all the posible values of Q(s,a) in a bigger state space.

Unsupervised Learning:
Given input data without any additional feedback (label), learn patterns, find structure in the data.
Tasks:
(1) Clustering: Organize a set of objects into groups in such a way that similar objects tend to be in the same group. Example: Genetic research, Image segmentation, Market research, Medical imaging, Social network analysis.
    One technique is k-means clustering: Algorithm for clustering data based on repeatedly assigning points to clusters (k number of clusters) and updating those clusters' centers
(2) Anomaly Detection: Find unusual events in the data.
    Use case: Fraud detection.
(3) Dimensionality Reduction: Compress large corpuses of data to a much smaller dataset without losing key / important information.

k-means clustering:
(1) Choose a number for 'k' cluster centroids, u[k]. k < len(data)
(2) Randomly assigns 'k's values which have the same dimension as the dataset.
(3) Assign samples to the nearest centroid (k)
    for i of range(len(data)):
      c[i] = index (from 1 to k) of cluster centroids closest to x[i]
    The distance can be calculated with min(|x[i] - u[k]|), or more generally, min(|x[i] - u[k]|  ** 2 )
(4) Move the cluster centroids to the average of the points
    for i of range(k):
      u[i] = average (mean) of points assigned to cluster k
    The means can be calculated with sum(x[]) / len(x[])
There will be situations where one of the uk has zero data points assigned to it. In this case, either
(1) Remove the empty cluster: k = k - 1
(2) Reinitialize the cluster centroids and restart the algorithm.
Optimization objective:
c[i] = index of cluster (1,2,...,k) to which sample x[i] is currently assigned to.
uk = cluster centroid k
uc[i] = location of centroid k to which x[i] is currently assigned to.
cost function: J(c, uk) = sum(|x[i] - u[k]|  ** 2 ) / m
Every step along the way reduces J(c, uk).
To minimize local minimum and choose the best cluster centroids, use multiple random initializations and choose the cluster with the lowest J(c,u). i.e., run the algorithm multiple times, maybe 50 to 1000.

Locality Sensitive Hashing:
- Find neighbours in high-dimensional spaces.
- Use multiple planes and a single hash value.
- Vector perpendicular to a plane is a Normal Vector, P.
- P @ V.T:
  - +ve: in the same side as P, -ve: Opposite side of P. 0: On the plane.
  - >= 0: h = 0; -ve: h = 0. Use numpy.sign().
  - hash value = sum(2 ** i x h[i]) for all i.

Anomaly Detection:
Density Estimation: Model p(x) from data. Probability of x[i] being seen in the dataset.
p(x[i]) < epsilon : Anomaly
p(x[i]) >= epsilon: OK
p = Gaussian distribution - Needs mean and sigma (standard distribution); sigma ** 2 is variance
To model p(x) with multiple featues (N), every feature is a Gaussian distribution.
(1) Calculate the mean and sigma of every feature.
(2) p(x) = p(x1; u1,s1) * p(x2; u2,s2) * p(x3; u3,s3) * ... * p(xN; uN,sN)

Anomaly Detection vs Supervised Learning:
Anomaly Detection is mostly used when:
(1) Very small number of positive exaples (y=1). 0-20 is common. Large number of negative (y=0)
(2) Many different "types" of anomalies. Hard for any algorithm to learn from positive examples what the anomalies look like; future anomalies may look nothing like any of the anomalous samples seen so far (training set)
Examples: Security related applications like fraud, defects detection of manufacturing very complex products like the aircraft engines, monitoring complex systems like in a data center.

Supervised Learning:
(1) Large number of positive and negative examples
(2) Enough positive examples for algorithm to get a sense of what positive examples are lke; future positive samples likely to be similar to ones in training set.
Examples: Email spam classification, defects detection of mobile phone manufacturing, weather prediction, disease classification.

Anomaly Detection feature selection:
(1) Choose feature with gaussian distribution property (plt.hist(x)). Otherwise, try to transform it:
    (i) log(x + C) : Lower values of C transform better
    (ii) sqrt(x)
    (iii) x ^ 1/3

Anomaly Detection error analysis:
Most common problem: p(x) is comparable for both normal and anomalous samples. p(x) is large. In this case, the algorithm will fail to flag an anomalous test data. Solutions:
(1) Check the anomalous test data. Create new feature which helps detect the anomaly.
(2) Create new feature based on existing ones using simple equations like x1 / x2 or (x1^2) / x2, etc.

Principal Component Analysis:
(1) Commonly used for visualization. Example: Take data with many features, say 50, 1000 or even more, reduce the #features to a few so as to enable visualization / plotting.
    - Common used by data scientists to visualize data for analysis.
    - Choose feature(s) with meaningful degree of variation.
    - Create new feature(s) based on existing ones. Example: length x height
(2) Features should first be normalized to have zero mean before applying PCA.
(3) Features should first be scaled before applying PCA.
(4) It will choose an axis(the Principal Component) around the origin which results in the largest variance when projecting the data onto it (at 90 degree angle).
    - All new axes created will be perpendicular to one another (90-degree)
(5) Less common applications:
    (i) Data compression - Reduce storage and/or transmission cost
    (ii) Speed up training of a supervised learning model.
(6) Matrices have eigenvectors and eigenvalues.
    Eigenvector: Provide direction of uncorrelated features of the data.
    Eigenvalue: The variance of the dataset on those features. It tells the amount of information retained by each feature.
    Dot-product between embeddings and eigenvectors gives projection on uncorrelated features.
    
=========================================================================================================================================================================================================================================
Neural Network (multilayer perceptron):
- Scale drives DL progess.
  (1) Data - especially labelled data (X,Y).
  (2) Computation - Speeds up Idea -> Code -> Experiment iterations.
  (3) Algorithms - Algorithmic innovation which makes computaion runs faster. For example, switching from Sigmoid to ReLU enables graduent descent to run faster by eliminating the long tails of Sigmoid.
- Every neuron in every NN layer computes 2 things:
  (1) Z[l](i)
  (2) A[l](i) = g(Z)

Non-Linear Activation functions:
- To approximate complex functions / features.
(1) Step function: g(x) = 1 if x >=0, else 0
(2) Logistic Sigmoid: 
    - g(x) = e^x / (1 + e^x)
    - derivative is minuscule when z is both -ve/+ve large numbers. Not good for gradient descent.
    - Never use this in the hidden layers. Only in the output layer for binary classification.
    - Derivative = a * (1 - a)
(3) tanh:
    - (e^z - e^(-z)) / (e^z + e^(-z))
    - Similar to sigmoid graph but the output is [-1, 1].
    - Keeps the sign of the input, i.e., if z > 0, g(z) > 0; if z < 0, g(z) < 0.
    - Center at (0,0) means easier for the NN to learn.
    - derivative is minuscule when z is both -ve/+ve large numbers. Not good for gradient descent.
    - Derivative = 1 - a ** 2
(4) Rectified Linear Unit (ReLu):
    - g(x) = max(0,x)
    - derivative = 0 for z <= 0, 1 otherwise.
    - Preferred default choice for hidden layers.
    - Derivative = 1 if z > 0; 0 otherwise
(5) Leaky Rectified Linear Unit (Leaky ReLu):
    - g(x) = max(0.01 * z, x)
    - derivative = 0.01 for z <= 0, 1 otherwise.

Choices of Activation Function:
(1) Output layer: Depends on the nature of the label:
    (i) Binary classification: Sigmoid
    (ii) -/+: Linear. For example, stock price change 
    (iii) Only positive numbers: ReLU
(2) Hidden layers: Mostly use LeLU. Reasons:
    (i) Simpler calculation compared to Sigmoid. This makes NN runs faster.
    (ii) There are 2 horizontal asymtotes in Sigmoid. This results in derivatives being close to 0 and therefore make gradient descent slower to run to global minimum.
    Do NOT use linear/identity activation function, g(z) = z in hidden layers. This will result in a linear function and defeats the purpose of NN. The computaion of multiple linear funtions is itself a linear function.
    The "off" or disable feature of the ReLU activation enables models to stitch together linear segments to model complex non-linear functions.
(3) Sigmoid/Softmax [0,1] and tanh [-1, 1] are seldom used in hidden layers due to the asymptotic tails of 0 and 1 and therefore, causing vanishing gradients problems.
(4) ReLU: Derivative = 0 for z <= 0
  - Dying ReLU problem - NN nodes get stuck in on the same weights and stop learning.
Types of Layers:
(1) Dense: Every neuron in a layer computes it's output / activation using ALL inputs/activations from ALL neurons in previous layer.
(2) Convolution: Neurons only look at regions of the data. Benefits:
    (i) Faster computation.
    (ii) Need less training data - less prone to overfitting.

Dimensions:
- n[l] = #units at layer l
- W[l] = dW[l] = (n[l], n[l-i])
- b[l] = db[l] = (n[l], 1)
- A[l] = (n[l], 1)
(1) Stack the input horizontally:
m = #inputs.
Z = W @ X + b
Z: (m, n[l])
A: (m, n[l]) = g(Z)
W: (n[l], n[l-1])
X: (m, n[l-1])
b: (m, n[l])

Z = X @ W.T + b
(m, n[l]) = (m, n[l-1]) @ (n[l], n[l-1]).T + (m, n[l])
          = (m, n[l-1]) @ (n[l-1], n[l]) + (m, n[l])
          = (m, n[l]) + (m, n[l])

Deep vs Shallow NN:
- One comparison is using the Circuit theory to calculate some mathematical function like XOR.
- Deep NN requires O(ln(N)) complexity.
- 1-layer requires exponentially more hidden units to compute. O(2^N) complexity.

h(x1,x2) = g(w0 + w1x1 + w2x2 + ...)
Example:
(1) To model the OR function:
    h(x1,x2) = g(-1 + x1 + x2); w0 = -1, w1=w2=1
(2) To model the AND function:
    h(x1,x2) = g(-2 + x1 + x2); w0 = -2, w1=w2=1
Training - to calculate the parameters:
- Trade-offs between speed and accuracy
(1) Gradient Descent: Algorithm for minimizing loss when training NN
    Start with a random choice of weights, repeat:
    - Calculate the gradient based on ALL data points (training data set): direction that will lead to decreasing loss
    - Update weights according to the gradient
(2) Stochastic Gradient Descent: Same as Gradient Descent but "... based on ONE data point"
(3) Mini-Batch Gradient Descent: Same as Gradient Descent but "... based on ONE SMALL BATCH of data points"
Perceptron: Only capable of learning linearly separable decision boundary
Multilayer NN: Artificial NN with an input layer, an output layer, and at least one hidden layer. Capable of modelling more complex problems compared to perceptron.
To train multi-layer NN, we have to propagate the loss/error from the output back to the hidden layers:
Back Propagation: Algorithm for training NN with hidden layers
- Start with a random choice of weights, repeat:
  - Calculate error for output layer
  - For each layer, starting with output layer, and moving inwards towards the earliest hidden layer
    - Propagate error back one layer
- One step of back propagation will provide a derivative of the output variable with respect to the input variable at that layer.
- Using calculus chain rule, i.e., dJ/da == dJ/dv * dv/da, Backpropagation helps calculate derivative of final output, J, with respect to the input, a.

To prevent underfitting (high bias):
(1) Use a larger network.

To prevent Overfitting (high variance):
(1) Collect more data
(2) Choose lambda appropriately.
Dropout: 
- Temporarily removing units - selected at random - from a NN to prevent over-reliance on certain units.
- It spreadas the weights across all activations and not relying on any specific features and this shrinks the weights. Provide similar effects as L2 regularization.
- Prominent implementation is Inverted Dropout:
  - Layers: 3
  - keep_prob = 0.8
  - dropout = numpy.random.rand(A[l].shape[0], A[l].shape[1]) < keep_prob
  - A[l] *= dropout <- This shuts down (1 - keep_prob) % of the activations.
  - A[l] /= keep_prob <- This restores the (1 - keep_prob) values. This is the "inverted dropout".
- J is not well defined with dropouts. This affects debugging of J over epochs.
(3) Data augmentation.
  - Apply library functions on real photos
  - Use GAN.
  - Combination of both - RandAugment

Random Weights Initialization:
- If initialize to zeros, all the hidden units will be computing the same function. dW will have the same rows after every epoch and therefore same as W.
- b is ok to initialize to zeros.
- W = numpy.random.randn((m,n)) * 0.01
  - * 0.01 so as NOT to generate big numbers which will make NN slow to learn when using sigmoid or tanh activation functions.
Vanishing / Exploding Gradients:
- Deep NN very prone to this. If W is slightly < 1, it will vanish, otherwise it will explode going deeper in the the NN due to W @ X.
- To help with this, use random initialization:
  - n: #inputs of the neuron
  (1) ReLU activation:
    - W[l] = numpy.random.randn((shape)) * numpy.sqrt(2 / n[l-1]) <- "He Initialization" (He et al. 2015)
  (2) tanh activation:
    - W[l] = numpy.random.randn((shape)) * numpy.sqrt(1 / n[l-1]). <- Xavier initialization.
- Manifested in a model with low accuracy and worsening performance over time.

playground.tensorflow.org
Image Convolution: Applying a filter that adds each pixel value of an image to it's neighbours, weighted according to a kernel matrix.
                  - Extracts features from input. Output is a feature map
Convolutional NN: NN that uses convolution, usually for analyzing images (i.e., Image Convolution + Pooling in a NN)
                - Training is done to figure out what's the best filters for the input image
                - Models how human look at images
Input image -> [Convolution (calculates filters) -> Pooling (Summarize and reduces the size of inputs)] -> Flattening (Feeds into the inputs of NN)
[Convolution -> Pooling] can be applied multiple times before Flattening. Early layers of this to discover low-level/primitive features (Edges, Curves, Shapes, low-level/primitive audio waveform); later layers to discover high-level features (objects, person, face, basic units of sounds like phonemes, etc)
Note: In a convnet, the higher up a layer is, the more specialized it is. The first few layers in a convnet learn very simple and generic features, which generalize to almost all types of images. But as you go higher up, the features are increasingly specific to the dataset that the model is trained on.

Feed-forward NN: Good for classification. 1:1 mapping from input to output -Single-valued output.
Recurrent NN: Output from the NN is fed to it's input for future calculation. It maintains some states. Good for dealing with sequences of data, both input and/or output. N:N mapping from input to output - Output sequence of values.
              Example applications: Youtube to analyze videos, Google translator, etc
              One example is Long Short Memory Network
=========================================================================================================================================================================================================================================
File formats for training and evaluation.
For large data sets, load the data from data warehouse into SSD storage (Cloud or local). This will prevent underusage of resources, GPU/CPU during the traning/evaluation process.
JSONL: JSON Lines is a simple text-based format with rows. It is human readable and an ideal choice for small to medium-sized datasets.
TFRecord: Binary format and easier to read for computers, ideal for efficient training.
Parquet: Good for large and complex datasets.
=========================================================================================================================================================================================================================================
Data Augmentation:
(1) Image: replicate and distort/transform original data sets
(2) Voice: Add noisy background, audio on bad connection, etc.

Data Synthesis:
(1) Image OCR

Transfer Learning:
(1) Feature-based:
    - Use weights from pre-trained on a completely new model.
    - Example, word embeddings.
    - Need to train the new model from scratch.
(2) Fine-tuning:
    - Fine tune final layers of pre-trained model on similar tasks.
Example: A NN trained on 1 million images of diverse object types (animal, fruits, plants, people, etc). You want to come up with a NN for character OCR.
Make a copy of the NN, replace the output layer with a 10-unit layer. Then, there are 2 options to train the "new" NN:
(1) Fine Tuning. Only train output layer parameters - This works if having a small data set.
(2) Train all parameters - This works with a huge data set. The initial phase of training is called pre-training.
The NN trained on 1 million images is called "supervised pretrained model".
The "new" copy of the NN is called "fine-tuning".
This works because layers of neurons have learnt different parts of the images. This could generalize to other types of images not found in the original dataset.
Only use this when:
(1) The same input types: text, images, audio.
(2) Original task has a lot more data than the new task.
(3) Low level features from original task could be helpful to learn the new task.
Use-case exmaples: GPT-3, BERT, imageNet, Speech-recognition -> wake/trigger word detection, Image classification to radiology classification.

Multi-task Learning:
- One NN do several things at the same time. Every task helps, hopefully, all of the other tasks.
- Same NN when all the tasks share similar or same low-level/primitive features / early layers of the NN. The output layer will be y^ with shape (n,m). n: #tasks, m: #samples.
  - Loss function = sum(sum(L(y^(j)(i), y(j)(i)) for j:[i:n] )) for i: [1:m]
  -  L is the usual logistic loss: -y(j)(i) * log(y^(j)(i)) - (1 - y(j)(i)) * log(y^(j)(i))
  - Benefit: Works well for partially labelled data. The loss function sum over j with valid value of [0,1] and omit inf, -inf, or NaN.
Only use this when:
- Tasks having similar low-level/primitive features.
- Amount of data for each task is quite similar. The aggregate of N-1 tasks' data should be a lot more than the remaining 1 other task. Symmetrically, data of N-1 tasks augment training of every other tasks.
- Can train a big enough NN to do well on ALL of the tasks. This NN should perform better than having a separate NN for each of the tasks.

End-to-End Deep Learning:
- A NN that replaces a traditional multi-staged learning pipeline and removes intermediate steps.
- Only use this when there is large engough end-to-end dataset (X,Y).
- Carefully choose what types of X->Y mappings to learn depeneding on what task the data could be obtained for.
Pros:
(1) Let the data speak for itself.
(2) Reduce hand-crafted / manual components. Dont' force the ML to learn the archaic ways.
Cons:
(1) Excludes potentially useful hand-crafted / manual components. This is especially true when there is lack of end-to-end data. The NN don't have sufficient data to learn the features to map X to Y in this case.
(2) Sometimes divide-and-conquer works better. Need to weight the relative complexity vs data availability.

Skewed Datasets:
Precision = #True positives / (#True + #False positives) = #True positives / (#Total predicted positives)
Recall = #True positives / (#True positives + #False negatives) = #True positives / (#Total actual positives) - Helps detect if the learning algorithm is predicting negatives all the time because the value will be zero.
0 Precision and/or Recall is bad.
For use cases with skewed classes or a rare class, decently high precision and recall values helps reassures the usefulness of the learning algorithm.

Trade-off between Precision and Recall:
Logistic regression: 0 <= F(w,b) <= 1
Predict 1 if F(w,b) >= threshold
Predict 0 if F(w,b) < threshold
To predict 1 only if very confident, use high value of threshold. This results in high precision, low recall
To predict 1 even when in doubt, use low value of threshold. This results in low precision, high recall

Picking the threshold is not something can be done with cross-validation. It's up to the business needs / use-case.
To automatically decide on the best learning algorithm without having to manually select between Precision and Recall, choose the algorithm with the highest F1 score.
F1 score = 2 * PR / (P + R) <= Harmonic mean. A mean calculation which pays attention to the lower value.

Classification model evaluation metrics:
(1) Accuracy
(2) Area under ROC (Receiver Operating Characteristics) curve - binary classification models.
    - Comparison of a model's true-positive rate to false-positive rate at different classification thresholds.
    - The AUC metric tells you how well your model is at choosing between classes (for example, how well it is at deciding whether someone has heart disease or not). A perfect model will get an AUC score of 1.
(3) Confusion matrix
(4) Classification Report

Regression model evaluation metrics:
(1) R^2
(2) Mean absolute error (MAE)
(3) Mean squared error (MSE)

Tune model hyperparameters on Evaluation dataset.
3 ways to adjust hyperparameters:
(1) Manually
(2) Randomly with RandomSearchCV
(3) Exhaustively with GridSearchCV - Brute force, trying every single combination.
Use appropriate scale for fair random search.
- Some paramameters like beta in Exponential Weighted Average algorithm, is sensitive when it is very close to 1. Linear search will not provide fair search in the tiny range of valid values of concern.
- Use log scale. Example for learning_rate [0.0001, 1]:
  - r = -4 * numpy.random.rand() <= [-4, 0]
    alpha = 10^r <= [0.0001, 1]

2 ways to save and load models:
(1) With Python pickle module.
(2) With joblib module
=========================================================================================================================================================================================================================================
Decision Tree Learning:
- Can model non-linear associations / capture non-linear relationships in data.
- Boundaries are always vertical or horizontal
Decision 1: How to choose what feature to split on at each node?
            Goal: Maximize purity / Minimize impurity - decision which gives fastest path to leave nodes.
Decision 2: When do you stop splitting?
            (1) When a node is 100% one class
            (2) Tree depth. Deep tree could result in overfitting / high variance.
            (3) When improvement on purity score are below a threshold.
            (4) When the #samples in a node is below a threshold.
Entropy as a measure of impurity
p1 = Fraction of samples that are positive
p0 = 1 - p1
H(p1) = -p1log2(p1) - (1 - p1)log2(1 - p1)
      = -p1log2(p1) - p0log2(p0)
Note:
(1) Use log2 to make the symmetric bell curve range from 0 to 1
(2) 0log2(0) = 0 <- log(0) is negative infinity

Decision 1: Goal: Reduce entropy, H(p1) = Information gain.
Information Gain = H(p1root) - (wl * H(p1l) + wr * H(p1r))
wl = weight of left branch = #left samples / #root samples
wr = weight of right branch = #right samples / #root samples

Regression Tree: Decision Tree which predicts a number.
- Use variance of the sample labels to decide which features to split on.
- Split on feature which gives highest reduction in variance (highest information gain)
Information Gain = V(p1root) - (wl * V(p1l) + wr * V(p1r))

Decision Tree Classifier:
- Partitioning the input feature space into regions.

Any single decision tree is highly sensitive to small change in the data. Solution is to use tree ensemble to let multiple trees vote on the final prediction.
(1) Use sampling with replacement to build data for tree ensemble. This is called "bagged" decision tree (https://medium.com/data-science/understanding-sampling-with-and-without-replacement-python-7aff8f47ebe4)
(2) Random forest - Randomizing the feature selection. Select from k < n features at each node. k is usually sqrt(n). Works better for large feature set.
(3) XGBoost - Instead of picking from all examples with equal 1/m probability, make it more likely to pick misclassified samples from all the trees constructed so far.

Decision Trees vs. Neural Networks
Decision Trees and Tree Ensembles:
(1) Works well on structured data
(2) Not recommended for Unstrcutured data (video, audio, images, text)
(3) Fast to train
(4) Small decision trees maybe human interpretable.

NN:
(1) Works well on ALL types of data and the combination of them.
(2) Maybe slower to train compared to decision trees
(3) Works with transfer learning - Can use pretrained models to train the output layer with fewer data.
(4) Can string together multiple NN when building a system of multiple models working together.
=========================================================================================================================================================================================================================================
Collabortive Filtering: Recommend items to user based on ratings of other users who gave similar ratings as the user.
Content-based Filtering: Recommend items to user based on features of both the user and the items to find a good match.
  - R[i,j]=1 if user[j] has rated item[i]
  - Y[i,j] ratings of user[j] on item[i]
  User and item will have different sizes of features. Prediction can be simplified to numpy.dot(W[j], X[i]). bias doesn't affect the performance of the model much.
  For the calculation to work, need to transform the feature vectors to vectors of same dimension. Use NN to achieve this. Call it User NN and Item NN with the output layers having the same #neurons.
  These 2 NNs are trained as a single NN with ONE cost function J = sum((V[u,j] * V[m,i] - Y[i,j]) ** 2) + regularization for all R[i,j] = 1 (Use tf.keras.layers.Dot layer)
Recommendation from LARGE Catalogue - can be done with 2 steps: Retrieval and Ranking.
Retrieval:
(1) Generate last list of plausible item candidates:
    - For each item purchased by the user, find 10 most similar items. This can use overnight batch to calculate the min(|V[k] - V[i]| ** 2)
    - For most viewed 3 categories/genres, find top 10 items.
    - Top 20 items in the country/region.
(2) Combine the retrieved items into a list, removing duplicates and items already purchased.
Ranking:
(1) Use the NN to calculate predictions on the retrieved list. Only need to calculate the missing V[j] or V[i] for the final dot product. Can be done on separated NN. This should be quick.
(2) Display ranked items to the user.
To speed up the response times of the system, pre-compute vectors V[i] for ALL items that it might recommend. This can be done even before a user logs in or even before Xu or Vu vector is computed.
If 2 items have vectors V[i] and V[k] which are similar to one another, i.e., |V[k] - V[i]| is small, a conclusion could be drawn that they will be liked by similar users.
=========================================================================================================================================================================================================================================
Reinforement Learning
State action value function (Q-function)
Q(s,a) = Return if it, the agent:
(1) Start at state s
(2) Take action a (ONCE)
(3) Behave optimally after that: max(Q(s', a'))
The best possible return from state s is max(Q(s,a))
The best possible action in state s is the action which  gives max(Q(s,a))

Policy, pi(s) = a It's job is to take as input any state, s, and map it to some action that it wants the agent to take in that state.
Goal of Reinforcement learning: Find a policy, pi, which tells the agent what action (a = pi(s)) to take in every state (s) so as to maximize the return.
Bellman Equation: Q(s,a) = R(s) + gamma * max(Q(s', a'))
gamma is discount factor [0, 1]: smaller value means higher discount and makes the agent impatient, larger value makes it more patient to obtain reward at later steps. Usually close to 1. Example, 0.9, 0.99, etc.

In random/Stochastic environment, there will be multiple sequence of different rewards due to uncontrolled environment / misstep probabilities. A Policy execution could produce many different results.
In this case, choose a Policy (pi) which maximizes the average/expected sums of discounted rewards.
Bellman Equation: Q(s,a) = R(s) + gamma * E(max(Q(s', a'))); E = Average

In continuous state-space, s is a vector of numbers (some continuous numbers, some binary). Encode the actions (a) using one-hot feature vector.
Use Deep Reinforcement Learning to compute Q(s,a) at the current state (s). X = [s a]. In a state, s, use the NN to compute Q(s, a) of all possible actions to take. Then, pick the action which maximizes Q(s,a).
Use Bellman Equation to generate training datasets with Q(s,a) = X; R(s) + gamma * max(Q(s',a')) = Y

Learning Algorithm:
Input: [s, a]; Output: Q(s,a)
Initialize NN randomly as guess of Q(s,a).
Repeat {
  Take actions wuth the agent. Get (s,a,R(s),s') tuples. <= **
  Store 10,000 most recent tuples. (Replay Buffer)

  Train NN:
    Create training set of 10,000 samples using x = (s,a); y = R(s) + gamma * max(Q(s', a'))
    Train Qnew such that Qnew(s,a) ~= y (f_w_b(x) ~= y)
  Set Q = Qnew ***
}
Q will improve after every training. This is Deep Q-Learning/Q-Network and can be trained by adjusting it's weights at each iteration to minimize the mean-squared error in the Bellman equation.

To be more efficient, X = [s] and Y = Q(s, a) of all possible actions. Output layer has the number of neurons equal to #actions. With this a single inference will generate all the possible Q(s,a) with all possible actions. Then, the max(Q(s',a')) term is readily available.

Using neural networks in reinforcement learning to estimate action-value functions has proven to be highly unstable. Here is why:
Bellman equation error: Y - Q(s,a;w) = R(s) + gamma * max(Q(s', a'; w)) - Q(s,a; w). 
Notice that Y will keep changing in every iteration. Having a constantly moving target can lead to oscillations and instabilities. To avoid this, we can create a separate neural network for generating the Y targets. We call this separate neural network the target Q-Network and it will have the same architecture as the original Q.
Steps:
Every C time steps we will use the Q-target-Network to generate the Y targets and update the weights of the Q-target-Network using the weights of the 𝑄-Network. We will update the weights of the the Q-target-Network using a soft update.

To avoid instabilities, use a Target Network and Experience Replay.

** How to choose actions while still learning? 2 options:
(1) Pick the action a which maximizes Q(s,a) - Because of random initialization, the NN might get stuck in it's own misconception that some actions (a) are *always* bad and therefore, would never get a chance to learn them well.
(2) epsilon-greedy policy (e = 0.05)
    (i) Greedy/Exploitation: With probabiity 0.95, Pick the action a which maximizes Q(s,a)
    (ii) Exploration: With probability 0.05, pick an action randomly.
    Start e high (1.0) and gradually decreases it to 0.01 in the course of the training.

Mini-batch helps in both Supervised and Reinforcement Learning when datasets is large and it takes every iteration to compute sum((f_w_b(X) - y) ** 2)/2m over large datasets and therefore takes very long time to converge and computationally expansive. However, mini-batch will exhibit some unreliable behaviour ("noisy") in graident descent although eventually it will converge faster compared to usual batch learning.
*** To make the NN converge more reliably, soft-update learning will help:
(1) Prevent abrupt changes to the NN
(2) Prevent one worse learning step overwrite the existing better NN
Q(w,b) = Qnew(w_new, b_new)
W = 0.01 * Wnew + 0.99 * W
B = 0.01 * Bnew + 0.99 * B
0.01 and 0.99 adds up to 1 and are the hyperparameters which control how aggressively or gradually the training updates the NN parameters.

In machine learning, logits are the raw, unnormalized scores or outputs from a neural network's final layer before an activation function like softmax or sigmoid is applied.
Binary Classification uses Sigmoid activation function and BinaryCrossEntropy loss function.
Multi-class Classification uses Softmax activation function and CategoricalCrossEntropy loss function.

Input: 128x128 greyscale image
Dense: 256 neurons
#parameters = 128 * 128 * 256 + 256 = 4194560 parameters
================================================================================================================================================================================
Convolution (C4_W1.pdf):
Early/Shallow layers often see the finer details with simpler shapes and patterns in smaller regions of the input (low-level/primitive features) such as edges and simple textures.
Later/Deeper layers often see more complex shapes and patterns in larger regions of the input (high-level features) such as more complex textures and object classes.
In other words, early/shallow layers zoom in, later/deeper layers zoom out.

Vertical edge detection filter: [[1, 0, -1],
                                 [1, 0, -1],
                                 [1, 0, -1]]
Horizontal edge detection filter: [[1, 1, 1],
                                   [0, 0, 0],
                                   [-1, -1, -1]]
Sobel (vertical) filter: [[1, 0, -1],
                          [2, 0, -2],
                          [1, 0, -1]]
Schorr (vertical) filter: [[3,  0, -3],
                           [10, 0, -10],
                           [3,  0, -3]]
NN to learn the filter parameters:  [[w1, w2, w3],
                                     [w4, w5, w6],
                                     [w7, w8, w9]] to detect various types of edges at various orientations.

Padding:
- Usually the middle of the input images is emphasized more than the edges.
- To give equal emphasis/importance to the edges, add padding around the input image.
To prevent data source shrinking, use padding.
"Valid" convolution: 
- No padding.
- nxn * fxf => n-f+1 * n-f+1

"Same" convolution: 
- Use padding, p, so that the output size is the same as the input size.
- n+2p-f+1 * n+2p-f+1
- for output size equals to n, n+2p-f+1 = n => p = (f-1)/2

By convention, in computer vision, f is always odd number. Reasons being:
(1) It results in p being a whole number. This gives symmetric padding around the data source.
(2) It produces a central cell in the filter which facilitates identifiying the position of the filter.

Strided convolution:
- Instead of moving the filter by 1 position, move by the number of strides, s.
- Output size: floor(((n + 2p) / s) + 1) * floor(((n + 2p) / s) + 1). Use of floor denotes that the convolution is only carried out of the filter is completely inside the image including the padding, if required. None of the filter should be hanging outside of the image.

Convolution over volume:
For example, for images with 3 channels, RGB, the filter must also have the same number of channels. The output will NOT have any channel, i.e., 1 dimension less compared to the input.

Multiple filters:
Each filter works on different features and produces one output. All output will be stacked to form the final output.

f[l] = filter size
p[l] = padding
s[l] = stride
Nc[l] = filter#
Each filter: f[l] x f[l] x Nc[l-1]
Activations: a[l] -> Nh[l] x Nw[l] x Nc[l]
Weights: f[l] x f[l] x Nc[l-1] x Nc[l]
bias: Nc[l]; The bias vector is b, where each filter has its own (single) bias.

Input:  Nh[l-1] x Nw[l-1] x Nc[l-1]
Output: Nh[l] x Nw[l] x Nc[l]
Nh[l] = floor((Nh[l-1] + 2p[l] - f[l]) / s[l]) + 1
Nw[l] = floor((Nw[l-1] + 2p[l] - f[l]) / s[l]) + 1
Nc[l] = Filter#

One property of convolution network is that the number of parameters are independent of the size of the input. This reduces overfitting.

Example parameters# calculation:

Input: 256x256 RGB
First hidden layer: 64 neurons.
NO Convolution
Bias: Nc[l] = 64
This hidden layer has 256 * 256 * 3 * 64 + 64 = 12582976 parameters

Conv1D (C4_W4.pdf page 40):
Input: 128-length vectors with 1 timesteps
Filter: 5x1
Activations:N[l] = floor((Nh[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(128 - 5) + 1 = 124

Input: 14x14x3
Filter: 1 5x5x3
p = (f-1)/2 = 4/2 = 2
Activations:
Nh[l] = floor((Nh[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(14 - 5) + 1 = 10
Nw[l] = floor((Nw[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(14 - 5) + 1 = 10
10 * 10 = 100
Weights: 5x5x1 = 25
Bias: Nc[l] = 1
This hidden layer has 25 + 1 = 26 parameters

Input: 256x256 grayscale image
Filter: 128 3x3
  Each filter: 3x3x1 = 9
Activations:
Nh[l] = floor((Nh[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(256 - 3) + 1 = 254
Nw[l] = floor((Nw[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(256 - 3) + 1 = 254
254x254 = 64516
Weights: 3x3x1x128 = 1152
Bias: Nc[l] = 128
This hidden layer has 1152 + 128 = 1280 parameters

Input: 256x256 RGB
Filter: 128 7x7
  Each filter: 7x7x3 = 147
Activations:
Nh[l] = floor((Nh[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(256 - 7) + 1 = 250
Nw[l] = floor((Nw[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(256 - 7) + 1 = 250
250*250*128 = 8000000
Weights: 7x7x3x 128 = 18816
Bias: Nc[l] = 128
This hidden layer has 18816 + 128 = 18944 parameters

Example output volume calculation:
Input: 121x121x16
Filter: 32 4x4 s:3 p:0
Nh[l] = floor((Nh[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(121 - 4)/3 + 1 = 40
Nw[l] = floor((Nw[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(121 - 4)/3 + 1 = 40
Output volume = Nh[l] x Nw[l] x Nc[l] = 40 * 40 * 32 = 51200

Input: 63x63x16
Filter: 32 7x7 s:2 p:0
Nh[l] = floor((Nh[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(63 - 7)/2 + 1 = 29
Nw[l] = floor((Nw[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(63 - 7)/2 + 1 = 29
Output volume = Nh[l] x Nw[l] x Nc[l] = 29 * 29 * 32 = 26912

Input: 128x128x12
Max Pooling: filter size:4, stride: 4
Nh[l] = floor((n+2p-f) / s) + 1 = floor((128 - 4)/4) + 1 = floor(124/4) + 1 = 31 + 1 = 32
Nw[l] = floor((n+2p-f) / s) + 1 = floor((128 - 4)/4) + 1 = floor(124/4) + 1 = 31 + 1 = 32
Output volume = Nh[l] x Nw[l] x Nc[l] = 32 x 32 x 12

Input: 66x66x21
Max Pooling: filter size:3, stride: 3
Nh[l] = floor((n+2p-f) / s) + 1 = floor((66 - 3)/3) + 1 = floor(63/3) + 1 = 31+1 = 32
Nw[l] = floor((n+2p-f) / s) + 1 = floor((66 - 3)/3) + 1 = floor(63/3) + 1 = 31+1 = 32
Output volume = Nh[l] x Nw[l] x Nc[l] = 22 x 22 x 21

Input: 3D data with 64x64x64x3
Filter: 4x4x4
#filters = 16
Padding: 0
Stride: 2
Nh[l] = floor((Nh[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(64 - 4)/2 + 1 = 31
Nw[l] = floor((Nw[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(64 - 4)/2 + 1 = 31
Nd[l] = floor((Nw[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(64 - 4)/2 + 1 = 31 
Output volume = Nh[l] x Nw[l] x Nd[l] x Nc[l] = 31 x 31 x 31 x 16

2D:
Input: 14x14x3
Filter: 1 5x5x3 s:1 p:0
Nh[l] = floor((Nh[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(14 + 0 - 5)/1 + 1 = 10
Nw[l] = floor((Nw[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(14 + 0 - 5)/1 + 1 = 10
Output volume = Nh[l] x Nw[l] x Nc[l] = 10 x 10 x Nc[l]

Input: 10x10x16
Filter: 1 5x5x16 s:1 p:0
Nh[l] = floor((Nh[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(10 + 0 - 5)/1 + 1 = 6
Nw[l] = floor((Nw[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(10 + 0 - 5)/1 + 1 = 6
Output volume = Nh[l] x Nw[l] x Nc[l] = 10 x 10 x Nc[l]

1D:
Input: 14x1
Filter: 5x1 s:1 p:0
Nh[l] = floor((Nh[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(14 + 0 - 5)/1 + 1 = 10
Output volume = Nh[l] x Nw[l] x Nc[l] = 10 x Nc[l]

Input: 10x16
Filter: 5x16 s:1 p:0
Nh[l] = floor((Nh[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(10 + 0 - 5)/1 + 1 = 6
Output volume = Nh[l] x Nw[l] x Nc[l] = 6 x Nc[l]

Typically in Convolutional NN, the size of the data will reduce deeper into the NN while the filter# will increase.

Types of layers in a Convolutional NN:
(1) Convolution
    - Relatively few parameters.
(2) Pooling
    - Reduce the size of the representatio
    - Speed up computation.
    - Makes some of the features that it detects more robust.
(3) Fully Connected (Dense)

Pooling:
- Reducing the size of an input by sampling from regions in the input
- Get the most salient from the input.
- Preserve detected features.
- Reduce the size of the input.
(1) Max or Average.
(2) Hyperparameters: f and s
(3) No parameters to learn. Gradient descent does not learn or change anything of it in this layer.
(4) Padding. Usually not used with Max pooling.
(5) Reduces/Shrinks Nh[l] and Nw[l]

1x1 convolution:
- To shrink channel#, use 1x1 filter. Example: (28,28,192) -> (28,28,32) can be achieved by a convolution layer of 32 x (1,1,192) filters with ReLU activation. This 1x1 conv filter is also called the bottleneck layer. 
- When implemented within reason, this bottleneck layer helps reduce computational costs significantly by shrinking down the representation size significantly without affecting the performance of the NN.
- One of the key ideas in Inception Network.

Upsampling Techniques:
(1) Transposed Convolution:
    - Suffers from checkerboard pattern problem:
      - The produced images has pattern resembling a checkerboard.
      - Due to some pixels are more influenced than the ones around them.
(2) Upsampling + convolution.
    - Infers pixels using a predefine method
Benefits of Convolutional layers:
(1) Parameter sharing
    - Same parameters of the filter are applied in different parts of the input dataset.
(2) Sparsity of connection
    - Each output value depends only on a small number of inputs as determined by the size of the filter.
These benefits result in:
(1) Allow convolutional NN to be trained with smaller training datasets and less susceptible to overfitting.
(2) Good at capturing translation invariance. For example, input dataset shifted a few pixels but the same features are detected.

Gradients will decrease/increase exponentially at a function of #layers.
Exploding gradients: Can easily been solved by gradients clipping using some threshold.
Vanishing gradients: Gradients at the output will have very little/vanishing impact nn the weights at the early layers of the NN after back-propagating many layers back to the early layers. Much harder to solve.

Residual Network:
- In a plain NN, as the networks gets deeper, the optimization algorithm (gradient descent, etc), has a much harder time in training. The training gets worse when the network gets deeper.
- ResNet helps mitigate this vanishing and exploding gradient problem.
- Identity function is easy for the residual block to learn. For example, adding a residual block at the end of a large NN with output a[l],
  a[l+2] = g(z[l+2] + a[l]) = g(W[l+2]*a[l+1] + b[l+2] + a[l])   a[l] is the residual connection.
  When L2 regularization is applied, W[l+2] is shrunk. For the sake of argument, when W[l+2] and b[l+2] is almost zero, and with ReLU activation, a[l+2] reduces to g(a[l]) = a[l]
- So, adding a residual block does not only hurt performance, it helps the network to learn at least the identity function. Sometimes, this block learns something useful.
- For the addition of the residual connection to work, "Same" convolution is used or apply a weight matrix, Ws to the residual connection. Ws could be weights that are learnt or a fix matrix that implements zero paddings that takes a[l] and zero pads it to the same dimension as z[l+2].
- Very deep "plain" networks don't work in practice because vanishing gradients make them hard to train.
- Skip connections help address the Vanishing Gradient problem. They also make it easy for a ResNet block to learn an identity function.
- There are two main types of blocks: The identity block and the convolutional block.
- Very deep Residual Networks are built by stacking these blocks together.    

Inception Network:
- Consists of the same inception module repeated along the network.
- Name originates from Hollywood movie Inception. Meme: https://knowyourmeme.com/memes/we-need-to-go-deeper

Computaional Cost = #filter params x #filter positions x #filters
Example: 
Input: 6x6x3 (RGB)
Filter: 3x3x3 (Nc has to match)
Stride: 1
Activations:
Nh[l] = floor((Nh[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(6 - 3) + 1 = 4
Nw[l] = floor((Nw[l-1] + 2p[l] - f[l]) / s[l]) + 1 = floor(6 - 3) + 1 = 4
1 filter: (3 * 3 * 3) * (4 * 4)
5 filters: (3 * 3 * 3) * (4 * 4) * 5 = 2160

Depthwise separable convolution reduces the computaional costs by 2 steps:
(1) Depthwise convolution.
    - Apply a single filter to the input. This results in a Nh x Nw x 1 output.
(2) Pointwise convolution.
    - Apply 1x1xNc' filter to the output from (1). This produces the final Nh x Nw x Nc' output.
- It reduces the computaional costs to (1/Nc' + 1/f^2) of the conventional convolution. For example, Nc'=512, f=3 will result in ~10 times cheaper.
- This is the core of MobileNet.
MobileNet v2:
- Bottleneck layer consists of 3 convolution layers and 1 residual connection (ResNet):
(1) Expansion - Use 1x1xNc' to expand the dimension
(2) Depthwise convolution
(3) Pointwise convolution - Projects back to smaller representation.
- (1) and (2) allows the NN to learn richer functions.
- (3) helps reduces memory footprint.

MobileNetv2 Bottleneck block. Input: nxnx5, 30 expansion filters, 20 projection filters. How many parameters in the complete block?
Expansion: (1x1x5) * 30 = 150
Depthwise: nxnx30 -> 3x3x30 = 270
Pointwise: (1x1x30) * 20 = 600
Total: 1020 

EfficientNet:
- 3 ways to scale up or down depending on the device computational resource constraints:
(1) Resolution of the input. r
(2) Depth of the NN. d
(3) Make the layers wider. w

Transfer Learning:
- Usually used in computer vision applications.
(1) Freeze all layers except the final softmax activation to cater to different classification.
    - Precompute the layers activation using custom inputs and save the feature vectors for prediction.
    - Use this if having small dataset.
(2) Freeze fewer early layers, train later layers. Or replace the last few layers.
    - Use this if having larger dataset.
(3) Retrain the whole NN if having large dataset.

Object Detection:
- Classification with localization
  - Figuring out where in the picture is the object detected and put a bounding box around it.
- Detection: There could be multiple objects of different classes in the picture.
- Softmax activation output of label Y which consists of:
  (1) Object classes
  (2) Coordinate and dimensions of the bounding box.
  [Pc, bx, by, bh, bw, C1, C2, C3,...] Pc: Probability of presence of an object of any of the classes. When Pc=0, the rest of the values are don't care.
- L(y^, y) - squared errors of the elements in Y. Or can use logistic regression L for Pc, log-like feature L for the classes and squared error for the bounding box.
Sliding Windows Detection Algorithm 
(1) Define a window size, crop the input image by the window and use that as an input to the ConvNet. Repeat the process by sliding the window over the entire input image.
(2) Repeat (1) with a increasing window sizes so that objects would be detected by a window size somehow.
- Computationally expansive due to sequential calculation of the parameters.
Convolution implementation of the sliding windows.
- Leverage on shared computations of all the grid cells.
- It works by turning the last Dense / fully-connected layers into convolution layers.
- The output will have different volume sizes to capture all the areas of the input image defined by the Layer-1 window size.
- One single forward pass through the NN computes all the windows due to the parameter sharing property of convolution.
- Effectively this is running the sliding window concurrently.
YOLO (You Only Look Once):
- Helps output more accurate bounding boxes.
(1) Splits the input image into grid cells.
(2) Label each of the grid cells, Y = [Pc, bx, by, bh, bw, C1, C2, C3,...]
(3) Takes the midpoint of detected object and assign it to the grid cell which contains the midpoint. By using a fine-grained grid, this helps to avoid assigning multiple objects' midpoint assigned to the same cell.
(4) Output is grid shape x Y shape
(5) Every grid cell can only detect ONE object.
(6) This allows the NN to output bounding boxes of any aspect ratio and more precise coodinates which are otherwise dictated by the strides of the sliding-window classifier.
- Bounding box specification:
  - bx, by: [0,1]
  - bh, bw: could be > 1 since the detected object could span multiple grid cells.

Intersection over Union
- Compares the predicted bounding box with the label.
- Used to evaluate the accuracy of object localization.
- size(intersection box) / size(union box)
- "Correct" if >= threshold (e.g: 0.5)

Non-max suppression algorithm:
- Helps eliminate duplicate detection of the the same object.
- Discard all boxes with Pc <= 0.6
- For each of the remaining boxes,
  - Pick the box with the higest Pc as the prediction.
  - Discard any box with IoU >= 0.5 with the prediction box. This discards the overlapping bounding boxes.
- Use this for each object class that the NN is setup to predict.

Anchor Boxes:
- Used to detect multiple objects in the same grid cell.
- Only works for #objects = #anchors in a grid cell.
- Allows the NN to specialize in learning different shapes of objects.
- Detected object is assigned to a grid cell that contains the object's midpoint and anchor box for the grid cell with highest IoU with the object's shape. (grid cell, anchor box)
- Output is grid shape x Y shape x #anchors: Y = [Pc, bx, by, bh, bw, C1, C2, C3,...] repeated for each anchor box.
- Define anchor boxes manually or use K-means to group together types of object shapes it tends to detect and use that to select a set of anchor boxes which are most stereotypically representative of the multiple object classes the NN is set up to detect.
Training dataset anatomy:
- vertical grid cells x horizontal grid cells x #anchors x (5 + #classes). 5: (Pc, bx, by, bh, bw)

YOLO:
(1) Get the Y's for each of the grid cell.
(2) Remove the low-probability predictions.
(3) For each class, use Non-max supression to generate the final predictions.

Region Proposals:
- R-CNN - Region with CNN.
- Select just a few regions/windows that make sense to run through the CNN.
- Propose regions and then classify them one at a time - SLOW.
- Use Segmentation algorithm to figure out what could be objects in the input and run that region though the CNN.
- Fast R-CNN:
  - Propose regions and use convolution implementation of sliding windows to classify all of them at once - single pass through the CNN.
  - The clustering step to propose the regions is still quite slow.
- Faster R-CNN:
  - Use CNN to propose regions (instead of the traditional segmentation algorithm).

Semantic Segmentation & U-Net:
- Assign every single pixel to a class.
- The goal is to draw a careful outline around detected objects so as to know which pixels belong to the detected objects.
- Label every pixel with one of the intended classes. This produces a segmentation map.
- The YOLO (You Only Look Once) algorithm shrinks the Nh and Nw. This reduces the spatial information.
- U-Net replaces with final layers of the CNN to blow up the dimensions so that the final output is the full-sized image.
- Transpose Convolution (key component in U-Net):
  - Output dimension is greater than the input dimension.
  - Apply the filter on the padded output.
  - Convolve input with the filter, fill the element-wise multiplication result in the respective output cell. Ignore the pad cells. For any overlapping cell, due to stride, add the cell values.
- Use skip connections to pass early layers, hgih-resolution, low-level/primitive feature information to the later layers to reconstruct the contextual spatial information.
  - Also improve gradient flow to the encoder.
- Input: (h,w,3), Output: (h,w,#classes): This serves to classify each pixel in the input image.
======================================================================================================================================================================================================
Face Recognition:
- Face verification supervised learning:
  - 1:1 mapping
  - A 2-step authentication process:
    (i) Identiy who you are. For example, name, ID, etc.
    (ii) Check if the face matches the identity in step (i)
- Face Recognition:
  - 1:K mapping
  - Has a database of K persons.
  - Ouput ID if the image is found in the database.
One-shot learning:
- Learn from ONE example to recognize the person again.
- Problems:
(1) Small dataset not effective to train a convnet.
(2) New person requires retraining the convnet.
Solution: 
Learn a "similarity" function:
- d(img1, img2) = degree of diff between 2 images
- <= threshold : "same"
- > threshold: "different"
OR:
- cos(img1, im2) >= threshold : same
- cos(img1, im2) < threshold : different
- This is used to test the siamese NN architecture.

Siamese Network:
- The 2 networks share the same parameters/weights and have the same architecture combined at the end to output a similarity score (cosine similarity) using a threshold.
  - Same parameters/weights are used to reveal the same information about each input in order to compare them.
  - During training, only need to optimize 1 set of parameters/weights which is then used in both identical NN networks for prediction.
- The final Dense layer is used to encode the input image. For example, (128 , 1) vector, f(x).
- d(x1,x2) = |f(x1) - f(x2)| ** 2
- Example system: DeepFace
- To encode:
(1) Use triplet loss function:
  - Anchor image, postive image, negative image. These triplets are picked from the training dataset. (𝐴(𝑖),𝑃(𝑖),𝑁(𝑖)) is used here to denote the 𝑖-th training example.
  - Adjust the weights so that cosine similarity between A and P is close to 1; A and N is close to -1.
  - -cos(A,P) + cos(A,N) <= 0
    - If prediction is correct, cos(A,P) = 1 and cos(A,N) = -1. Therefore, -cos(A,P) + cos(A,N) = -1 - 1 = -2 which is < 0
    - If prediction is incorrect, cos(A,P) = -1 and cos(A,N) = 1. Therefore, -cos(A,P) + cos(A,N) = -(-1) + 1 = 2 which is > 0 (higher cost)
    - diff = s(A,N) - s(A,P)
    - L = 0 if diff <= 0 else diff
  - d(A,P) <= d(A,N)
    d(A,P) - d(A,N) <= 0
  - To prevent the NN from learning a trivial solution (0 - 0 <= 0) setting all the parameters to 0, use "<= 0 - alpha"
  - d(A,P) - d(A,N) + alpha <= 0
  - L = 0 if (diff + alpha) <= 0 else (diff + alpha)
  - L = max(-cos(A,P) + cos(A,N) + alpha, 0)
  - L(A,P,N) = max(sum((f(A) - f(P))^2) - sum((f(A) - f(N))^2) + alpha, 0); sum over the image instances. 
  - Bigger alpha means more optimization during training.
  - J = sum(L(A,P,N)) for all the triplets over the image corpera and #persons. Example, 10k pictures of 1k persons.
  - Needs multiple pictures of each person.
  - Caveat: d(A,N) is very easily large number with random pairs of images. To train the NN effectively, choose pairs which result in d(A,P) ~= d(A,N). Ideas from FaceNet.
  - Training batches:
    - Choose hard triplets with s(A,N) ~= s(A,P)
    - No duplicate within each batch.
    - A well-trained model will produce output matrix, y^, with diagonal values showing higher similarities than off-diagonal values.
    - Use the off-diagonal values to adjust the loss function, L for training a better model.
      (1) Mean negative: mean of off-diagonal values in each row, excluding the value in the diagonal.
          L1 = max(-cos(A,P) + mean_negative + alpha, 0)
      (2) Closest negative: Off-diagnoal value which is the closest to, but less than, the diagonal value in each row.
          - L2 = max(-cos(A,P) + closest_negative + alpha, 0)
          - This forces the model to learn to pull apart the examples through tougher training.
      (3) Cost = L1 + L2
- Use one-shot learning when testing the network.
(2) Use binary classification:
  - Linear Regression
  - y^ = sigmoid(w * sum(|f(x1) - f(x2)|) + b) for i=1...128
  - In DeepFace, the (f(x1) - f(x2))^2 / (f(x1) + f(x2)) is used in the summation.
Use cases:
(1) Identify duplicate questions in a Q&A system. Ex: Stackoverflow, Quora, etc.
(2) Identity difference. Ex: Where are you from? vs Where are you going?
    - The semantic meaning of a sentence cannot be determined only from similar words.
(3) Handwritten signature identification.

Nerual Style Transfer:
- Content picture (C) + Styling picture (S) = Generated Image (G)
- Let a convnet learn the features in a input picture.
- Pick a neuron in a layer, find the nine image patches that maximize the neuron's activation. Repeat for other neurons.
- Cost function J(G) = alpha * Jc(C, G) + beta * Js(S, G)
(1) Initialize G with a seed, (100, 100, 3). G = G - alpha/2G * J(G)
(2) Use Gradient Descent to minimize J(G). Update the generated image pixel with G = G - alpha/2G * J(G).
- Jc(C, G) - Use middle layer to compute content cost.
  - a(l,c), a(l,g): activation of layer l on images
  - Jc(C, G) = sum(|a(l,c) - a(l,g)| ** 2)
- Styles definition: Correlation among the activations across channels in the particular layer used to calculate Js(S, G).
  Style Matrix (also called a "Gram matrix.") to capture the style of the input image (S and G):
  - In linear algebra, the Gram matrix G of a set of vectors (v1,…,vn) is the matrix of dot products, whose entries are Gij=viT * vj=np.dot(vi,vj).
    In other words, G(gram)ij compares how similar vi is to vj: If they are highly similar, you would expect them to have a large dot product, and thus for G(gram)ij to be large. G(gram) = A @ A.T. The value G(gram)i,j measures how similar the activations of filter i are to the activations of filter j.
    The diagonal elements of G(gram)ii measure how "active" a filter is. For example, suppose filter i is detecting vertical textures in the image. Then G(gram)ii measures how common vertical textures are in the image as a whole. If G(gram)ii is large, this means that the image has a lot of vertical texture.
    By capturing the prevalence of different types of features (G(gram)ii), as well as how much different features occur together (G(gram)ij), the Style matrix Ggram measures the style of an image.
  - Compute the Style matrix by multiplying the "unrolled" filter matrix with its transpose: 
    - (Nc x (Nh x Nw)) @ (Nc x (Nh x Nw)).T = (Nc x (Nh x Nw)) @ ((Nh x Nw) x Nc) => (Nc, Nc)
  - a[l](i,j,k) = activation at (i,j,k) in lyaer l. Compute G[l] which is a matrix of Nc[l] x Nc[l].
  - G[l, S](k,k') = sum(sum(a[l](i,j,k) * a[l](i,j,k'))) for all i,j (height, width) over all k channels for style picture
  - G[l, G](k,k') = sum(sum(a[l](i,j,k) * a[l](i,j,k'))) for all i,j (height, width) over all k channels for generated picture
  - J[l](S, G) = |G[l, S](k,k') - G[l, G](k,k')| ** 2 = sum(sum(G[l,S] - G[l, G])) / (2 * Nh *Nw * Nc) for all channels in layer l.
  - For bettter styling effect, use J[l](S, G) from multiple layers. Js(S,G) = sum(lambda * J[l](S, G)) for some number of layers. This takes both low-level/primitive and high-level correlations into account.
  - J(G) = alpha * J(C, G) + beta * J(S, G). Use gradient descent to generate G which minimizes J(G).

Get better results if you "merge" style costs from several different layers.
Each layer will be given weights (λ[l]) that reflect how much each layer will contribute to the style.
How do you choose the coefficients for each layer? The deeper layers capture higher-level concepts, and the features in the deeper layers are less localized in the image relative to each other. So if you want the generated image to softly follow the style image, try choosing larger weights for deeper layers and smaller weights for the first layers. 
In contrast, if you want the generated image to strongly follow the style image, try choosing smaller weights for deeper layers and larger weights for the first layers.
What you should remember:
The style of an image can be represented using the Gram matrix of a hidden layer's activations.
You get even better results by combining this representation from multiple different layers.
This is in contrast to the content representation, where usually using just a single hidden layer is sufficient.
Minimizing the style cost will cause the image G to follow the style of the image S.
======================================================================================================================================================================================================
L2 Distance
- Euclidean distance
- A measure of the straight-line distance between two points in a multi-dimensional space.
- sqrt(sum((A - B) ** 2))
- dist = numpy.linalg.norm(a-b, ord=2)
- Biased by the size difference between a and b. Cosine similarity mitigates this.

Dot-product:
v @ w = ||v|| * ||w|| * cos(beta)
cos(beta) = (v @ w) / (||v|| * ||w||)
          = (v @ w) / (np.linalg.norm(a) * np.linalg.norm(b))

Frobenius Norm:
= ||x|| = sqrt(sum(X[i][j] ** 2)) for every element in the matrix
======================================================================================================================================================================================================
Sequence Models
Use Cases:
(1) Speech Recognition. Input: speech Output: text
(2) Music Generation. Input: a number, genre, etc. Output: Music notes. One-to-many.
(3) Sentiment Classification. Input: sentence, Output: integer count. #likes or #stars. Many-to-one.
(4) DNA Sequence Analysis. Input: sequence of A,C,G,T Output: which part of the sequence corresponds to a specific pattern, protein for instance.
(5) Machine Translation. Input: sentence in source language Output: sentence in translated language. 2 parts: Encoder reads the input; Decoder outputs the translated sentence.
(6) Video Activity Recognition. Input: sequence of video frames Output: activity description.
(7) Name Entity Recognition. Input: text Output: people / entities identified in the input.

Notation:
Both input and output are one-hot encoded vectors.

Standard NN (one-to-one) work well for sequence because of 2 problems:
(1) Inputs and outputs can be of different lengths in different samples of the dataset.
(2) Standard NN doeesn't share features learnt across different positions of text. For example, in name-entity recognition, if a name is identified in early in the sequence, the same string (faeature) should be identified as well in later parts of the sequence.
    - Same as how ConvNet help sharing features learnt in one part of an image to generalize quickly to other parts of the image, Recurrent NN plays the same role for sequence modeling.

Recurrent NN:
- Predict the next steps based on the knowledge of all prior steps.
- Uses activations up to the current RNN cell/time step to compute output y-hat.
- At each RNN cell/time step, a(t-1) and x(t) are input, a(t) and y(t) are output. Many-to-many.
- Tx = length of input; Ty = length of output
- a0 = vector of zeros.
  Wax = weights used to compute output(a) using x
  Wya = weights used to compute output(y) using a
  a(t) = g(Waa @ a(t-1) + Wax @ x(t) + ba) <= Usually tanh is used. Sometimes ReLU
  y^(t) = g(Wya @ a(t) + by) <= Sigmoid for binary classification, softmax for multi-class
- Simplify a(t) = g(Waa @ a(t-1) + Wax @ x(t) + ba) = g(Wa @ [a(t-1), x(t)] + ba)
  Example:
  - if a(t-1) is 100-element vector, Waa would be (100, 100)
  - if x(t) is 10,000-element vector, Wax would be (100, 10000)
  Wa is a matrix of [Waa | Wax] stacked horizontally with shape (100, 10100)
  [a(t-1, x(t))] is a matrix of [a(t-1) | x(t)] stacked vertically with shape (10100, 1)
- Simplify y^(t) = g(Wya @ a(t) + by) = g(Wy @ a(t) + by)
- L(t)(y^(t), y(t)) => use BinaryCrossEntropy loss function
  L(y^(t), y(t)) = sum(L(t)(y^(t), y(t))) for all time steps.
- Backpropagation through time using gradient descent to update parameters.
Language Model and Sequence Generation:
- Laungage model predicts probability of a sequence of tokens - P(y1,y2,y3,...).
- L(y^(t), y(t)) = -sum(y(t) * log(y^(t)))
  L = sum(L(y^(t), y(t))) for all time steps.
- P(y(1), y(2), y(3)) = P(y1) * P(y2 | y1) * P(y3 | y1,y2)
Sampling a sequence from a trained RNN:
- Generate randomly chosen sentence from RNN.
- y^(t): numpy.random.choice(P(word1)P(word2)P(word3)...)

Basic RNN is not very good at capturing long-range dependencies. This is similar to the vanishing gradients problem inherent in training very deep conventional NN.
Exploding gradients can also happen which is easily spotted because the parameters blow up which result in some NaN due to numerical overflows. Use gradient clipping to solve it. If gradient vectors larger than some threshold, rescale them.

Gated Recurrent Unit:
- Helps with vanishing gradients problem.
- Modification of RNN
- Helps RNN capture long-range dependencies and solve the vanishing gradient problem.
- C = memory cell
  - Multi-dimensional. Each element in the vector can be used to "remmeber" different context - singular/plural, food, present/past tense, people, etc.
- It outputs an activation a(t)
- C(t) = a(t) : [1, N] vector. Each index serves to remember different types of contexts. For example, singular/plural, food, places, etc
- C~(t) = tanh(Wc @ [C(t-1), x(t)] + bc) <= candidate replacement of current C(t) <- simplified version
- Relevance Gate [0, 1], Gr = sigmoid(Wr @ [a(t-1), x(t)] + br) : 1: Update 0: Don't update
- C~(t) = tanh(Wc @ [Gr * C(t-1), x(t)] + bc) <= candidate replacement of current C(t) <- Full version
- Update Gate [0, 1], Gu = sigmoid(Wu @ [a(t-1), x(t)] + bu) : 1: Update 0: Don't update
- C(t) = Gu * C~(t) + (1 - Gu) * C(t-1) : element-wise multiplications.
- y^(t) = g(Wya @ a(t) + by) usually softmax
Long Short Term Memory (LSTM)
- Modification of RNN
- More powerful than GRU
- C~(t) = tanh(Wc @ [a(t-1), x(t)] + bc) <= candidate replacement of current C(t) <- Full version
- Update Gate [0, 1], Gu = sigmoid(Wu @ [a(t-1), x(t)] + bu) : 1: Update 0: Don't update
- Forget Gate [0, 1], Gf = sigmoid(Wf @ [a(t-1), x(t)] + bf) : 1: Forget 0: Don't forget
- Output Gate [0, 1], Go = sigmoid(Wo @ [a(t-1), x(t)] + bo) : 1: Output 0: Don't output
- C(t) = Gu * C~(t) + Gf * C(t-1) : element-wise multiplications.
- a(t) = Go * tanh(C(t))
- y^(t) = g(Wya @ a(t) + by) usually softmax

Bidirectional RNN:
- Acyclic graph. Information in both directions and independent of one another.
- Both forward and backward passes can by RNN or GRU or LSTM.
- Need the entire sequence of data to work. For example, need to wait for speech to pause to start computing the prediction of the past speech.
- y^(t) = g(Wy * [a(t), a'(t)] + by): a'(t) is the backward pass activation value.
- LSTM BRNN is commonly used in NLP.
- Needs the complete sequence to work. For example, if used in speech recognition system, have to wait for the speaker to stop talking to start processing the sentences.

Deep RNN:
- a(t) and a'(t) are inputs to the next NN layer.
- y^(t) ony appears in the final layer of the NN, just like any conventional NN.
======================================================================================================================================================================================================
NLP & Word Embeddings
- One-hot encoding does not provide relationship information. Any product of any 2 vectors is always zero.
- Word Embeddings is a featurized representation of the words.
- Learn concepts / features of each word to draw similarity, etc.
- Orignal model trained on large corpera of texts (un-labelled), ~100B words, use transfer learning for other tasks which have smaller training dataset (~100K words).
- Example: Gender, Food, Fruit, Furniture, Animal, Noun, Verb, Country, City, etc.
- t-SNE visualization - Maps the features to a 2D visualization of the vocabulary. It shows clusters of words with relative same features.

Analogies using word feature vectors.
Analogy reasoning.
- Man is to woman as King is to ?.
e(man) - e(woman) ~= e(king) - e(?)
Find word(w) which maximizes similarity(e(w), e(king) - e(man) + e(woman))

Similarity function:
(1) Cosine similarity.
(2) Squared differences || U-V || ** 2
E: Embeddings matrix. (feature size, vocab size)
O: one-hot encoding vector of each of the word in the vocabulary. (vocab size, 1)
e(word) = E @ O -> (feature size, 1)

To build a language model, use a context of last N words.
To learn a word embedding, use any of the following:
(1) #words on the left & right
(2) Last/Previous 1 word
(3) Nearby 1 word - Skip-grams model.

Word2Vec
Skip-grams model
- Use Supervised learning to learn good word embeddings.
- Randomly choose a context word
- Randomly choose another word within some window size from the context word as the target word.
- So, set up a supervised learning problem where given a context word, predict what is the randomly chosen word within a windows size of the context word.
- The goal of this supervised learning is not to optimze the learning problem per se, but to use this learning problem to learn good word embeddings.
- The supervised learning problem is to learn to map Context (Input x), c to Target (Output y), t
- O(c) -> E -> e(c) = E @ O(c) -> softmax -> y^
- softmax: p(t|c) = exp(theta(t).T @ e(c)) / sum(exp(theta(j).T @ e(c))); theta(t) = parameter associated with output t, i.e., the chance of output t being the label.; sum over the vocabulary size.
- L(y^, y) = - sum(y * log(y^)) over the vocabulary size.
- y: (vocabulary size, 1)
- Problems: The denominator, sum(exp(theta(j).T @ e(c))), is slow.
  - Use binary tree to solve it - Hierarchical softmax. Time complexity is log(N) instead of N.
  - The tree is organized with commonly used words close to the root of the tree, less common ones close to the leafs.

Negative Sampling:
- Sample a context and target word.
- X: Context word (c), target word (t)  Y: 1/0
- Positive example (Y=1) Randomly sample a target word in a window size from the context word.
- Negative examples (Y=0) (k): pick a word randomly from the vocabulary. k = 5-20 for small datasets, 2-5 for large datasets.
- Logistic regression of P(y=1 | c, t) = sigmoid(theta(t).T @ e(c))
- Negative : Positive = 5:1
- Vocabulary size sigmoid is cheaper than softmax. In every iteration, only train on k+1 input.
- Sampling target words for negative examples based on P(wi) = (f(wi)^(3/4)) / (sum(f(wj)^(3/4))) sum over the vocabulary size. f(wi) is observed frequency of w in the training dataset.

GloVe (Global vectors for word representation)
- Not used as much as SKip-grams and Negative Sampling.

Non-contextualized word embeddings:
(1) Word2Vec
(2) GloVe

Contextualized word embeddings:
(1) ELMo
(2) BERT

Language Model:
- a[0] : zero vector
- x[t] = y[t-1]

Machine Translation can be thought of as a "conditional language model".
- Instead of start of with a vector of zeros, start off with the encoded source language.
- The encoded source language serves as a[0] to the RNN/sequence model, i.e., the Language Model above as the decoder.
- Instead of predicting the probability of any novel sentence, it predicts the probability of translated sentence conditioned on the input source language.
- P(y|x), x: source language sentence, y: translated language sentence, both are embedding vectors.
- Maximize P(y|x). Greedy search word by word is exponentially large depending on the vocabulary size.
Beam Search:
- Approximate / heuristic search algorithm.
- Beam width, B, best #choices to consider in every time step. Use B copies of the network for parallel calculation in every time step.
- B=1 reduces this to greedy search.
- First step: P(y1|x) y1 is [word1, word2, word3], length of it determined by B
- Second step: P(y1,y2|x) = P(y1|x) * p(y2|x,y1)
- Arg Max sum(P(yt|x,y1,...yt-1)) for all time steps, t1...Ty. Ty = #words in output sentence.
- P(yt|x,y1,...yt-1) will result in very small floating point number which will cause accuracy issue - numerical rounding error.
  - Use log(P(yt|x,y1,...yt-1)) for calculation. Maximizing log(P(y|x)) is maximising P(y|x).
  - Arg Max sum(log(P(yt|x,y1,...yt-1)))
- Second problem is that this algorithm will incline towards shorter-sentence predictions since fewer factors of P(|) will result in larger number compared to longer-sentence predictions.
  - Arg Max sum(log(P(yt|x,y1,...yt-1))) / (Ty ^ alpha) -> fixes the long-sentence penalty issue.
Error Analysis:
- B or RNN at fault?
- Best: y*; prediction: y^
(1) Compute P(y*|x) and P(y^|x)
(2) P(y*|x) > P(y^|x): This aligns with the result but Beam search chooses y^. So, it is at fault. Increase beam width, B.
(3) P(y*|x) <= P(y^|x): RNN computes the wrong result. So, it is at fault, the objective function.
Note: Need to consider length normalization as well (1 / Ty^alpha).

Minimum Bayes Risk (MBR)
(1) Generate several candidate translations
(2) Assign similarity to every pair using a similarity score, ROUGE for instance.
    - argmax(sum(ROUGEE, E")/N)
(3) Select the sample with the highest average similarity.

Machine Translation Evaluation:
- BLEU Score - Bilingual Evaluation, Understudy.
- Measures how similar is the MT to reference translations.
- Count the #times <foo>gram appears in any of the reference text. Limit/Clip that count to the max of times the matching <foo>gram appear in any of the reference text.
- Given a machine generated translation, automatically compute a score that measures how good the translation is.
- Used to evaluate text-generation applications where there could be multiple equally good output/generated text by the RNN to find the best match. 
  - Not for single-truth evaluation like speech recognition which is evaluated by exact match of the transription to the speech.
- Pn = sum(Count<clip>(n-gram)) / sum(Count(n-gram)) where n-gram appears in prediction sequence, y^. Count<clip> = max count the n-gram appears in any of the reference translations.
- Pn = 1 if y^ exactly the same as any of the reference translations.
- Combined BLEU score = BP * e^(1/n * sum(Pn)) for all lengths of n-grams.
- BP, brevity penalty, penalizes translations which are too short as short translations are easy match to any of the references.
- BP = 1 if MT_output_length > reference output length;
- BP = e^(1- reference_output_length/MT_output_length) otherwise.
- Vanilla BLEU score doesn't measure well when the NMT outputs mostly the common words as this will result in high BLEU score.
- Modified version of BLEU exhaust the matched word from all occurences of the word in all the references for subsequent matching. So subsequent word which was matched previously will not account for a matching count.
- One downside of BLEU score is that it does not account for semantic meaning and syntactic structure of the sentence for scoring.
- BLEU score is based on Precision (from F1 score)

ROUGE-N
- Based on Recall (from F1 score)
- max() of n-gram matches between the candidate and all of the references.
- So, combine both to get F1 score.
- F1 = 2 * (Precision * Recall) / (Precision + Recall)
     = 2 * (BELU * ROUGE-N) / (BLEU + ROUGE-N)

Naive encoder-decoder RNN is good at translating short sentences. BLEU score will drop when the source text gets longer.

Attention model helps with translating long text.
- Use Bidirectional RNN for the encoder (source sequence) and forward-only RNN for decoder (output sequence).
- a(t') = [a(t') | a_backward(t')]; t' = time step of the source/input language sequence.
- alpha(t,t') = weight / amount of attention y(t) should pay to a(t'). 
  - softmax(exp(e^(t,t')))
  - Use a NN to compute e^(t,t') with hidden state of the decode time-step, S(t-1) and a(t')
- C(t) = sum(alpha(t,t') @ a(t'))

Speech Recognition.
- Use spectogram.
CTC cost for speech recognition:
- Connectionist temporal classification.
- Usually output text is shorter than the input sequence. For example, with 100Hz, 10s of speech will end up with Tx = 1000.
- For RNN Ty to match Tx, use repeated character and collapse the repeated characters not separated by "space"/"blank".
- Example: "The quick brown fox": ttt_h_eee____ ___qqq__uuu__ -> The qu
======================================================================================================================================================================================================
Transformer Network
- Eliminate the sequential nature of RNN which processes one token / word at a time.
- Ingest the entire sequence at once and process the tokens/words in parallel.
- Attention + CNN
  - It was later found after publish of "Attention is all you need" paper that only Attention is needed. CNN is redundant.
- It's a multitask model. Trained once to perform multiple NLP tasks.
- Types of Attention:
  (1) Scaled dot-product attention
      - 2 matmul and one softmax: softmax(Q @ K.T / sqrt(K vector size)) @ V
      - softmax(Q @ K.T / sqrt(K vector size)): 
        - Weight matrix has total element count of #queries * #keys.
        - Each row is the query, each column in the row is the key corresponding to the query.
      - @ V:
        - Rows of context vector.
        - Each row is the query, each column in the row is the value corresponding to the query.
      - Much faster to compute but the alignment between source and target languages must be learnt elsewhere.
      - Typically the alignment is learnt in the input embeddings or other linear layers before the Attention layer.
  (2) Encoder-Decoder Attention:
      - Queries from one sentence; Keys and Values from another sentence.
  (3) Self attention
      - Q,K,V from the same sentence.
      - Each word in the sentence pays attention to every other words.
      - Meaning of each word within the sentence.
  (4) Masked Self Attention / Causal Attention:
      - Q,K,V from the same sentence.
      - Q does not attend to future positions.
      - Used in the decoder from the transformer model to ensure that predictions at each time step/position depends only on the known outputs.
      - softmax((Q @ K.T / sqrt(K vector size)) + MASK) @ V
        - MASK: 0 for knwon positions, -inf in other positions.
        - softmax((Q @ K.T / sqrt(K vector size)) + MASK):
          - weights with 0 in future time steps/positions
  (3) Multi-head attention
      - Each head learns a different linear transformation to represent words.
      - Linear transformation of the Q,K,V -> Scaled dot-product attention -> concat -> Linear. Every linear transformation step contains a set of learnable parameters.
      - #heads corresponds with the number of sets of Q,K,V embeddings and applying attention mechanism, i.e., the dot-product, to each set/head.
      - The different sets of Q,K,V can be obtained by linearly transforming the original Q,K,V embeddings.
Self-Attention ("head"):
- A(q,K,V) = attention-based vector representation of a word.
  - Similar to word embeddings but with context.
  - Calculate for each word in the input sequence. A(1),...A(5) in parallel.
  - q: query, K: key, V: value
  - q(i) = Wq @ x(i)
  - k(i) = Wk @ x(i)
  - v(i) = Wv @ x(i)
  - x(i) is the word 
- Q,K,V:
  - (#words, #embeddings)
  - Transform the text into it's embedding and stack them horizontally.
  - Generally, K and V are the same vectors although sometimes some transformation is done to the embeddings of K to get V.
- W(i) is the head(i) in multi-head attention. It can be thought of as a "question" to find out the context of every word. The same head has to be dot-product with Q,K,V of every word.
  - For example, W(1) = "What's happening?" W(2) = "When?", W(3) = "Who?", etc
  - head(i) = Attention(W(i) @ Q, W(i) @ K, W(i) @ V)
  - All heads computed in parallel.
- h = #heads
- Multihead(Q,K,V) = concat(head1, head2, head3,...) @ W
Note: 
(1) In self-attention, multiply x with matrices W. In case of the multi-head attention, you don't need to do this, as you already have the matrices W(i) in each head, and you would effectively do the calculation twice if you did the multiplication here also. 
    In the simplest case of multi-headed self-attention you would actually use q = k = v = x. The reason we anyway show q, k and v here as different values is that in one part of the transformer (where you calculate the attention between the input and output) the q, k, and v are not the same as they carry different information.
(2) Transformer Details:
    - Position encoding of the word in the input sequence. This is needed in transformer architecture since the computation is done in parallel and thus losing the position information in the original sequence.
    - i repeats itself once to encode 2 dimensions using the sine and cosine functions (same i is used for sine and cosine). To calculate i, use a helper index k, which is simply counting over the dimentions of the word embedding from 0 to d-1. i = k // 2.
    - As i increases, sin and consine frequency decreases.
    - P(t) is a vector of all the values of the sine and consine at time (t) for all k.
    - P(t) is added directly into X(t) so that each of the word vectors is influenced / coloured with the relative position in the sentence.
    i: 0
    k = 2i = 0
    k = 2i + 1 = 1
    i: 1
    k = 2i = 2
    k = 2i + 1 = 3
- Masked multihead attention is used during training to mask some texts from the input sequence so that the transformer architecture would learn to predict the masked words compared to the complete Y sequences available in the training dataset.
Positional Encoding:
- A feature that contains the information about the relative positions of words. The sum of the positional encoding and word embedding is ultimately what is fed into the model. 
  If you just hard code the positions in, say by adding a matrix of 1's or whole numbers to the word embedding, the semantic meaning is distorted. 
  Conversely, the values of the sine and cosine equations are small enough (between -1 and 1) that when you add the positional encoding to a word embedding, the word embedding is not significantly distorted, and is instead enriched with positional information. 
  Using a combination of these two equations helps your Transformer network attend to the relative positions of your input data.
======================================================================================================================================================================================================
ML Strategy:
Orthogonalization: What to tune to achieve a specific effect.
(1) Fit training set well on cost function
    - Bigger network
    - Different optimizers
    - NO: early stopping - This is not orthogonal as it will affect (2) below
(2) Fit dev/validation set well on cost function
    - Regularization
    - Bigger training set
    - NO: early stopping - This is not orthogonal as it will affect (1) above
(3) Fit test set well on cost function
    - Bigger dev/validation set
(4) Performs well in real world
    - Change dev/test set or cost function.
  
Use a single number evaluation metric
Classifier:
- 2 metrics: Precision and Recall.
- Sometimes different classifier models will yield different scores for these 2 metrics.
- Instead of using 2 to identify the "best" classsifier to use, use a single metric - F1 score.
Satisfying and Optimizing metrics:
- If N metrics, pick ONE to optimize, the rest could be just satisfying.
- Example: 
  (1) Optimize accuracy, Satisfiies running time.
  (2) Optimize accuracy, Satisfiies #false positive. Example in the use case of wake/trigger words, <= 1 false positives in every 24 hours.
  
Dev/test datasets:
- Dev / cross-validation datasets is used to evaluate different ideas and pick one. For example, used to tune hyperparameters and/or to evaluate the best performing model.
- Must come from the same distribution. In other words, the bull-eye's target must be the same for both datasets.
- It is ok for the training dataset to come from a slightly different distribution than the dev/test sets. For example, scrapping the dataset from the internet or purchase from third-party vendors.
- The goal of test set is to evaluate how good the final system is. So it is best to get these datasets from the field - data the system is built for.
How about size?
- Test set should be big enough to give high confidencein the overall performance of the system.
- Train-test split - The "test" is actually the dev/cross-validation dataset.
  - It's ok to not have a test dataset if ensuring "high confidence" in the final system is not critical.
- Must make sure large percentage of dev/test set to be as close to the field as possible.
- Example, 200,000 corpera of data scrapped from internet. 10,000 from the field.
  - Bad idea: Mix them, shuffle and distribute them to train/dev/test. This will result in dev/test having big proportion of data not of concern - away from the bulleye's target.
  - Recommended: Training: 200,000 + 5000 from the field. dev/test 25,000 each.
  
Error metric:
- Keep reviewing error metrics.
- Example: Error (J) = sum(Wi * {y^(i) != y(i)}) / sum(Wi) for all i in the dataset.
                      W(i): 1 if desirable input data; 10 if non-desirable input data.
- If doing well on metric + dev/test does not translate to doing well on final application, change metric and/or dev/test set.

Bayes optimal error - Best possible error. A theorical level which is impossible to achieve for any X->Y function, F(Y|X).
Progress is often quite fast until the system surpasses human-level performance. After that it would slow now / plateau. The reason is that it would be difficult to have a good estimate of Bayes error that still helps with making good decision clearly.

So long as ML is worse that humans,
- Get labelled data from humans.
- Gain insight from manual error analysis - Why / how does a person get it right?
- Better analysis of bias/variance.

For tasks that humans can do really well, human-level error is a proxy to Bayes error. Human error ~= Bayes error.

Example 1:
Human (~ Bayes): 1%
Training error: 8%
Dev error     : 10%
Focus on bias:
- Bigger model
- Train longer
- Bettern optimization algorithms: Add momentum or RMSProp or use better algo like Adam.
- Review / change NN architecture / hyperparameters search - RNN, CNN

Example 2:
Human (~ Bayes): 7.5%
Training error: 8%
Dev error     : 10%
Focus on variance:
- More training data
- Regularization - L2, Dropout, Data augmentation.
- Review / change NN architecture / hyperparameters search - RNN, CNN

High bias and high variance can happen at the same time especially in differnt parts of high-dimesional dataset.

Training error - human error = Avoidable bias
Dev error - Training error = Variance

Problem areas where ML significantly surpasses human-level performance:
- Structured data
- Lots of data - Easier for computer to find statistical patterns
Problem areas harder for ML to surpass human-level performance:
- Natural perception: Speech, image recognition

Carry out Error Analysis on dev/cross-validation dataset
- Count up #errors from various different categories.
  Example: cross-validation error is 10%. Analyze the distribution of this 10% of misclassified pictures. Let's say 100 pictures are misclassified and 5 of them are of alien categories. Spending tine and effort to work on this 5% misclassified category will only give 0.5% improvement on the dev set.
- Consider examining examples the algorithm prediction was right and wrong. Examining only the wrong examples will end up with biased estimates of the errors.

Mislabelled datasets
- DL algorithms are quite robust to random errors in the training set. If the total dataset size is big and the actual percentage of error not too high, it is ok to leave it.
- If mislabelled in dev/test sets, carry out error analysis.

Build first system quickly and dirty, then iterate, except:
- Have significant prior experience in the application area.
- Significant body of academic literature that can be drawn on for pretty much the same problem the system is built for.

If training data comes from different distribution than that of dev/test set and error differs significantly,
- Can't tell if it is variance problem.
- Maybe training data is high-resolution and/or dev dataset is more difficult to deal with.
- To help identify the root cause, 
  (1) Carve out a training-dev set from the training dataset. 
  (2) Train on the remaining training dataset.
  (3) Check variance between training and train-dev set. If big, there is confirmed a variance problem.
      Otherwise, if small error diff between training and training-dev but big error difference between training-dev and dev, then there is a data mismatch problem.
      Or maybe both "Avoidable bias" problem between training and training/dev + data mismatch between original training data and the dev/test set.
      - Training set - human error level = avoidable bias
      - Training-dev - training = variance
      - Dev - training-dev = data mismatch if > training/dev error. If smaller, it may mean that the training datset is much harder than the dev/test dataset.
      - Test - Dev = test = degree of overfitting to the Dev dataset. May need to find a bigger Dev dataset.

              General Application               Final Application
              (Source of training data corpera) (Field/target dataset)
Human level         4%                              6%
Training error      7%                              6.5%
Dev/test error     10%                              7%

Addressing data mismatch error:
- Carry out manual error analysis to understand diff between training and dev dataset. Only on dev but exclude the test to avoid overfitting.
- Make traiing dataset more similar to the dev/test set. Or collect more data similar to dev/test set.
- Generally, just find ways to close the gap between training and dev/test set. For example, using artifically data synthesis. The proportion of the synthesized data must match up to the size of the real data. Otherwise, ML will overfit to it. For example, noisy background sound, graphically synthesized images.
======================================================================================================================================================================================================
Vector Space
- Measures of "similarity" among words/documents in terms of angle and/or distance.
Co-occurrence matrix
Word by Word design:
- #times they appear together within a certain distance k.
Word by Document design:
- #times they appear together within a certain category.

N-gram probability:
- Count of N-gram / Count of the condition (N-gram prefix).
- P(Wn|W[1,N-1]) = C(W[1,N-1] Wn) / C(W[1,N-1])
- Bi-gram example: "I am happy because I am learning"
  - P(y|x) = C(x,y) / C(x)
  - P(am|I) = C(I am) / C(I) = 1
  - P(happy|I) = C(I happy) / C(I) = 0
  - p(learning|am) = C(am learning) / C(am) = 1/2
- Tri-gram example:
  - P(happy|I am) = C(I am happy) / C(I am) = 1/2
  - P(w3|w12) = C(w1 w2 w3) / C(w1 w2)
- To avoid 0 probability for missing n-grams in the vocabulary,
  (1) Smoothig:
      - Laplacian (add-1) Smoothing
        - P(Wn|W[1,N-1]) = (C(W[1,N-1] Wn) + 1) / (C(W[1,N-1]) + |V|)
      - K-Smoothing
        - P(Wn|W[1,N-1]) = (C(W[1,N-1] Wn) + K) / (C(W[1,N-1]) + K * |V|)
  (2) Backoff:
      - Use lower n-grams. Ex: If N-gram is missing, use (N-1)-gram, (N-2)-gram, down to unigram.
        - This distorts the probability, especially in small corpora.
      - Probability discounting, e.g, Katz Backoff: makes use of discounting.
      - "Stupid" backoff: lower N-gram probability multiplied by a constant - 0.4 is shown to work well.
- P(w1,w2,w3,w4,w5) = P(w1) * P(w2|w1) * P(w3|w2,w1) * P(w4|w3,w2,w1) * P(w5|w4,w3,w2,w1)
- Bad thing about n-gram is large memory footprint.
  - Large n-grams are needed to capture dependencies between distant words.

Evaluate a language model using perplexity metric.
- Tells if a set of sentences are written by humans (lower score) instead of being randomly generated text from a machine.
- PP(W) = P(s1,s2,s3,...,sm)^(-1/m) -> Inverse probability normalized by m. Lower better.
        = P(sm|s1,s2,s3,...)^(-1/m)
- W: test dataset containing m sentences s1,s2,..sm
- si: i-th sentence in the test dataset, each ending with </s>
- m: #words in test dataset including </s> but excluding <s>
- Typical good laungage model has the score of [20, 60]
- Compare different language models having the same vocabulary.
- using <UNK> to replace unknown words (out-of-vocabulary) will lower the score. Use it sparingly as the model might just choose to generate a lot of <UNK> tokens.

Word Embeddings:
- Low dimension
  - Ex: X-axis encodes +ve for positive words and -ve for negative words.
        Y-axis encodes +ve for concrete words and -ve for abstract words.
- Embed meaning
  - Semantic distance
  - Analogies
Creation:
(1) Corpora of texts
    - Words in context.
    - General-purpose. E.g.: Wikipedia
    - Specialized. E.g.: Contracts, experts' knowledge like books and papers, etc.
(2) Embedding model
    - Self-supervised ML model.
    - The training data (text corpora) are NOT labelled.
    - The data itself serves to supervise the learning process.
    - word embeddings is the by-product of the learning process.
Word2Vec (Google, 2013):
- Uses shallow NN
- 2 model architectures:
  (1) Continuous Bag of Words (CBOW)
      - Learn to predict missing word (Centre word) based on the surrounding words (Context words).
      - Shallow dense NN:
      (1) Input layer: (V, 1) with ReLU activation. V is vocabulary size.
          - W1 (N, V), b1 (N, 1)
          - W1 @ X = (N,V) @ (V,1) = (N,1) = a1
      (2) Hidden layer: (N, 1) with softmax activation. N is the size/dimension of the final word embeddings.
          - W2 (V, N), b2 (V, 1)
          - W2 @ a1 = (V, N) @ (N, 1) = (V, 1)
      (3) Output layer: (V, 1). Y is the centre word.
      - To transform the context vectors into a single vector, take the average of the one-hot vectors.
      - 3 ways to get word embeddings out of this training process:
      (1) W1: Each column maps to the input word in the vocabulary.
      (2) W2: Each row maps to the input word in the vocabulary.
      (3) Average of both: 0.5 * (W1 + W2.T)
  (2) Continuous skip-gram / Skip-gram with negative sampling (SGNS).
      - Opposite of (1), predict the surrounding words based on the input word.
Global Vectors (GloVe) (Standford, 2014)
- Does not use NN at all.
fastText (Facebook, 2016):
- Based on skip-gram model
- Takes into account structure of words by representing words using n-gram of characters
- This enables the model to support out-of-vocabulary words by generating embedding from similar sequence of characters.
Advanced word embeddings methods:
- DL, contextual embeddings
- BERT, ELMo, GPT-2

Word Embeddings Evaluation Metrics:
(1) Intrinsic:
    - Test / measure the semantic and syntatic relationships between words:
      - Semantic: meanings of the words.
      - Syntatic: grammar, plural forms, etc.
      (i) Analogies
      (ii) Clustering
      (iii) visualization
(2) Extrinsic:
    - Use real applications to test / evaluate the usefulness / effectiveness of the word embeddings.
      For example, name entity recognition, parts-of-speech tagging, etc.
    - More time-consuming and difficult to diagnose the root cause if the performance is poor.

Embedding layer in sequence model:
- Input to the layer is vector of indices of the words in the vocabulary.
- Transforms that into matrix of (V, N) where V is the vocabulary size, N is the embedding size.
- The embeddings are trainable weights.
- Usually followed with a Mean layer to reduce the dimension of parameters to learn. This reduces the dimension of the embedding from N to 1.
- Allows the NN to learn a good representation of vocabulary for the specific task at hand.

Google's seq2seq model in 2014:
- Uses Embedding -> LSTM layers for bothe encoder and decoder.
- Final encoder LSTM activation is the only input to the decoder.
- Limitation is that it relies on a fixed-length memory with fixed hidden state (LSTM's activation) size.
  - This causes a fixed amount of information being passed from the encoder to the decoder.

Attention:
- Alignment = Similarity between source and target word.
- The Q and K vectors are used to calculate the alignment scores, which measure how well they match. These alignment scores are then turned into weights used for weighted sum of the value vectors which is returned as the attention vector. 
- This process is performed using scaled dot-product attention.

Teacher Forcing:
- A technique used to train the NMT decoder by using the actual sentences from training dataset at each time step as input instead of output from previous timestep in order to improve training performance especially during the early stages of training a model from scratch.
- A fresh NMT model decoder will deviate from the target if the output of every LSTM unit is fed into subsequent units.
- The errors from the front of the LSTM units will cascade down the line to the later units and accumulate. This causes problem with training a fresh NMT model's decoder.

GLUE benchmark:
- General Language Understanding Evaluation
- A collection used to train, evaluate, analyze NL understanding systems.
- Multiple datasets with different genres and of different sizes and difficulties.
- Model agnostic.
======================================================================================================================================================================================================
Generative Adverserial Networks
- The discriminator has a much easier job as it does not need to modelthe entire space of what a generated could look like.
- Both discriminator and generator should always be at the same level of skill in order for them to learn together. If one is superior over the other, then there is little information that the other could learn from.
- The training stops when the generator has achieved an acceptable level of generating realistic image. At that point, the generator can be used alone to generate the same class of data that it was trained on.
- The noise vector z input to the Generator has the important role of making sure the images generated from the same class don't all look the same.
- Noise vector is referred to as latent which means the information from the noise vector is not seen directly on the output but they do determine how the output looks like.

Objective:
- Make the generated and real distributions look similar.

Challenges in evaluating GANs:
- Loss tells us little about their performance.
- Unlike with classifiers, where a low loss on a test set indicates superior performance, a low loss for the generator or discriminator suggests that learning has stopped.
- A discriminator cannot be used for evaluation because it overfits to the generator it is trained with.
- There is also no "perfect" discriminator that can differentiate reals from fakes - if there were, a lot of machine learning tasks would be solved.
- Very much sample dependent. Not model parameters.
2 important properties:
(1) Fidelity: Quality of images
    (i) Pixel distance - Not reliable due to shifts.
    (ii) Feature distance - High-level semantic information less susceptible to small differences, nuances, noise in the inputs.
        - Use pretrained models on ImageNet for feature extraction.
        - Extracted features = embeddings      
(2) Diversity: Variety of images

Frichet Inception Distance (FID):
- Univariate Normal FID = (mean(x) - mean(y))^2 - (sigma(x) - sigma(y))^2
- Lower value = closer distributions.

Inception Score:
- Develoepd before FID and more relevant to conditional generation.
- Use Inception model classifier as-is to classify the generated image.
- Tries to capture both fidelity and diversity.
- Higher value better.
- Downside:
  - Can be exploted by generating one realistic image of each class.
  - Only looks at generated images. No comparison to real images.

Precision & Recall:
(1) Precision:
- Relates to fidelity.
- Overlaps/Intersection between real and fake / total generated images.
(2) Recall:
- Relates to diversity.
- Overlaps/Intersection between real and fake / total real images.
(3) Models tend to be better at recall.
(4) Use truncation trick to improve precision, i.e., to weed out all the extra fakes, especially for downstream applications.

Truncation trick sampling:
- Done after the model is trained and when you are sampling beautiful outputs.
- Resamples the noise vector z from a truncated normal distribution which allows you to tune the generator's fidelity/diversity.
- The truncation value is at least 0, where 1 means there is little truncation (high diversity) and 0 means the distribution is all truncated except for the mean (high quality/fidelity).
- Sampling at test time from a normal distribution with it's tail clipped.
- Fakes are sampled using the training or prior distribution of z.
- It truncates the normal distribution of the noise vector at the tails. Tradoff between fidelity and diversity determinses the hyperparameter of how much to truncate.
- Human evaluation is still necessary for sampling. For exaple, HYPE (Human eYe Perceptual Evaluation).

Modes:
- Peaks in a distribution.

Modes Collapse:
- Happens when the generator gets stuck in local minima and generating one mode

BinaryCrossEntropy (BCE):
- Generator maximizes J; Discriminator minimizes J.
- Minimax problem.
- Vanishing gradients problem as it is easier for the discriminator to get ahead of the generator.
  - As the discriminator pulls apart/delinneates generated and real distribution, the produces gradients at the long tails of the sigmoid function which is ~0. This give little information for the generator to learn.

Earth Mover's Distance:
- Effort to make generated distribution equal to the real distribution.
- Depends on the distance and amount needs to be moved.

Wasserstein Loss (W-Loss):
- Generator:
  - Goal is to minimize the Wasserstein distance/Earth Mover's distance between generated and real data distributions.
- Discriminator is called "critics".
  - Goal is to approximate the Wasserstein distance/Earth Mover's distance, instead of categorizing samples as real/fake.
- E(c(x)) - E(c(g(x)))
  - c(x) prediction of real
  - c(g(x)) prediction of fakes - g(x) is the generator's output.
  - Discriminator maximizes this and Generator minimizes this distance.
- Not bounded to [0, 1], critics output any real number.
- Works on the condition that the critic needs to be 1-Lipschitz (1-L) continuous.
  - The norm (L2) of the gradient should be at most 1 for every point.
  - This so that the gradient does not grow more than linearly.
  - This ensures W-Loss is validly approximating Earth Mover's Distance.
1-L Enforcement:
(1) Weight Clipping
    - Forces the weights of the critic to a fixed interval.
    - This has the drawback of limiting the learning ability of the critic when it is retricted to a certain values that the weights could take on.
(2) Gradient Penalty
    - Works better compared to (1).
    - Adds a regularization term. (|gradient(c(x^)) - 1|) ** 2
    - Since checking the critic’s gradient at each possible point of the feature space is virtually impossible, you can approximate this by using interpolated images (x^).
    - x^: epsilon * x + (1 - epsilon)*g(z) <- Interpolatation.
          x: real, g(z): generated
    Generator Loss = - mean(Critic(G(z)))
    - The generator tries to maximize the scores (or equivalently minimize the negative)
    Critic Loss = -(mean(Critic(x)) - mean(Critic(G(z)))) + gp * c_lambda
    - The optimizer (ex: Adam) will minimize the loss. So, by minimizing -(torch.mean(crit_real_pred) - torch.mean(crit_fake_pred)), it effectively maximizes the distance between the real and fake distributions.
    - Batch Normalisation is not used in the critic anymore because batch norm maps a batch of inputs to a batch of outputs. In our case we want to be able to find gradients of each output w.r.t their respective inputs.
    
Spectral Normalization:
- Another method to stabilize training.
- Normalizes the weights, similar to BatchNormalization and instantiated as a layer.

Conditional Generation:
- Generate outputs from specific class.
- Supervised learning, i.e., training dataset must be labelled.
- Generator input: noise appened with one-hot vector
- Discriminator input: class channels in addtion to RGB channels.
Controllable Generation:
- Generate outputs with specific features
- Unsupervised learning.
- Manipulate the inputs from the Z-space, which is just the name for the vector space of the noise vectors.
  - Find the directions in the Z-space related to changes of the desired features on the GAN output.
  - Then move the noise vector in different directions in that Z-space.
- Challenges:
  (1) Feature correlation.
  (2) Z-space entanglement.
      - Movement in different directions has an effect on multiple features simultaneously in the ouput.
      - Not possible to control single output feature.
      - Common problem when the Z-space doesn't have enough dimensions relative to the #features - can't map them 1:1.
- Disentangled Z-space:
  - Specific indices in the noise vectors that change / control partiular features on the GAN output.
    Ex: v1=[3,1,7,9,10,...] index-0 maps to hair color, index-1 mapes to eye's color, etc.
  Encourage disentanglement:
  (1) Supervision:
      - Classes are embedded in the noise vector. Don't need one-hot vector.
  (2) Loss Function:
      - Unsupervised Learning.
      - Add regularization term to the original loss function (BCE, W-Loss, etc) to encourage the model to associate each index in the noise vector to different features on the output.

Density Estimation:
- Probability density over modeled features, i.e., likelihood of particular features to show up in samples.
- How often do certain features make up a target object.
- Useful for anomaly detection

Disadvantages of GAN:
- No density estimation.
- Inverting is not straightforward - Getting the noise vector (latent space representation) of the generated image.

StyleGAN:
(1) Progrssive Growing.
    - Make it easier for the generator to generate higher resolution images by gradually training it from lower to higher resolution images by doubling it.
    - Grow the percentage (alpha) of up-/down-sampling output to Conv layer to progressively learn more weights/parameters.
(2) Noise Mapping Network.
    - z (512,) -> 8 Dense layers -> w (512,)
    - Create a more disentangled representation of noise. Easier to learn 1:1 mapping to each individual feature to have finer-grained control over generated features.
    - w fed to various layers in the generator.
    - With multiple (Z, W) pairs injected in different layers of the DC GAN, the output will be a mix of images generated with the individual pair.
(3) Adaptive Instance Normalization (AdaIN)
    - HxW for each instance of data, x_i, (x_i - miu_x_i) / sigma_x_i. Do it for every channel.
    - w -> Dense => y_s, y_b
    - y_s_i * ((x_i - miu_x_i) / sigma_x_i) + y_b_i
    - Transfers learnt styles info from W onto generated images. It also normalizes the statistics so each block overrides the one that comes before it.
    - Stochastic noise causes small variations to output.
      - Sample noise from Normal distribution
      - Concatenate it to x before AdaIN.

Pix2Pix:
- Paired image-to-image translation (Supervised)
- Example:
  (1) Segmented image + real image -> generated image
  (2) Edges/Sketches of object + real object image -> generated image
Generator:
- Use real picture as input that maps to the target output image.
- No Z-space noise vector input. Use dropout instead.
- U-Net
  Encoder:
  - 8 blocks of:
    - Conv2D -> BatchNorm -> LeakyReLU
  Decoder:
  - 8 blocks of:
    - TransposeConv2D -> BatchNorm -> ReLU
- Loss = Adverserial Loss (BCE, W-loss, etc) + lambda * Pixel Distance Loss (L1)
  The Pixel Distance Loss (L1) makes it supervised learning.

Discriminator:
- Use real picture as input that maps to the target output image.
- Generated image concatenated with input (condition) image along the channel dimension - tf.concat([x, y], axis=1).
- No Z-space noise vector input. Use dropout instead.
- PatchGAN
- Matrix output instead of binary.
- Loss = Adverserial Loss (BCE, W-loss, etc)

Pix2PixHD and GauGAN are successors of Pix2Pix

CycleGAN:
- Unpaired image-to-image translation (Unsupervised).
- Mapping between 2 piles of image styles and find commonalities and differences between them.
- Common = content; Unique = style
- Uses Least Sqaure Loss (Mean Squared Error) as the adverserial loss component.
  - sum((prediction - truth)**2)
  - Reduces vanishing gradients problem which is observed when using BCE loss.
- Identity Loss:
  - Optional
  - Helps with color preservation.
  - Put image X through Y->X generator should produce exact same image.
Cycle consistency: 
- Translation to second pile (X->Y) and back to the first pile (Y->X) should produce the same image in the first pile and vice versa.
- Important in transferring uncommon style elements while maintaining common content.
- Cycle consistency loss = Pixel distance loss (L1) in a cycle. Take the sum of both directions.
- Single Optimizer.
- Ablation studies show that:
  (1) Without adverserial GAN loss component, outputs are not realisitc.
  (2) Without Cycle Consistency Loss component, mode collapse happens.
  (3) With only half the cycle consistency loss component, mode collapse happens too.
- Cycle consistency is a broader concept that's used outside of CycleGAN a lot too! It's helped with data augmentation and has been used on text translation too, e.g. French -> English -> French should get the same phrase back.
Generator (2x):
- Residual block: DCGAN in the bottleneck layers with skipped connections (ResNet).
- UNet + Residual block
- Least Squared Adverserial Loss (LSAL) = sum((generated - 1)**2)
- Loss = LSAL + lambda1 * Cycle Consistency Loss + lambda2 * Identity Loss
       + LSAL + lambda1 * Cycle Consistency Loss + lambda2 * Identity Loss
  - LSAL + Cycle Consistency Loss + Identity Loss in each direction.
Discriminator (2x):
- PatchGAN
- Loss = sum(generated**2) + sum((real - 1)**2)
======================================================================================================================================================================================================
Challenges on training AI on medical images:
(1) Class Imbalance
    - Use weighted loss function.
    - Re-sampling
(2) Multi-Tasks
    - Use one model to predict multiple classes to leverage feature sharing and making good use of the same dataset.
    - Use multi-label / multi-task loss function.
    - Each label of classes is a binary classification by itself. Each prediction of classes is the probability of '1'.
(3) Dataset Size
    - Transfer Learning + Data Augmentation
Patient Overlap in datasets - Data Leakage:
- The DCNN could inadvertently learn and remember specific features of patients which could result in overly optimistic result when the data of the same patient appears in the test dataset.
- Solution: Split dataset by patients so that the same data from the same patients only appear in one of training / validation / test dataset but not in multiple datasets.
Set Sampling:
- Minority class sampling:
  - Sample a certain percentage of minority class data in both the test and validation datasets.
  - The remaining data goes into the training dataset.

Accuracy:
Sensitivity = True positive rate = Recall
Specificity = True negative rate
Prevalence = P(disease)
Accuracy = Sensitivity * P(disease) + Specificity * (1 - P(disease))

PPV (Positive Predictive Value) = P(disease|+) = Precision
NPV (Negative Predictive Value) = P(normal|-)

PPV = Sensitivity * Prevalence / (Sensitivity * Prevalence + (1 - Specificity) * (1 - Prevalence))

Confidence Level:
- In repeated sampling, it produces confidence intervals that include the population accuracy in about x% of samples. The x is called the confidence level.

Confidence Interval (CI):
- The width of the interval is affected by the sample size.
- The lareger the sample size, the narrower is the interval and hence, the better estimate of wehere the population accuracy lies.
- Useful to express the range in which the population accuracy lies with a certain confidence level by using test result on a sample even when we can't run the model on a whole population.

3D U-Net Image Segmentation:
Soft Dice loss function:
- Popular loss function for segmentation models.
- Advantage: Works well with imbalanced datasets. Ex: a small fraction of the brain MRI is tumour region.
- 1 - 2 ^ sum(y^ * y) / (sum(y ** 2) + sum(Y^ ** 2))
-     2 ^ sum(y^ * y) / (sum(y ** 2) + sum(Y^ ** 2)) is overlap between truth and prediction and bigger is better.

Using natural log of feature values is common when there is reason to believe there is linear relationship between the ln of the feature and the target/label.

Prognosis:
Risk model:
- Concordant pairs: Patient/person with the worst outcome is given highest risk score by the model.
- Only consider pairs where the outcomes are different (permissible pair)
C-index = (#concordance + 0.5 * #risk_ties) / #permissible
- The C-index measures the discriminatory power of a risk score.
- Intuitively, a higher c-index indicates that the model's prediction is in agreement with the actual outcomes of a pair of patients.

Missing data:
Why?
(1) Missing completely at random.
    Example: Throw dice to decide to drop the recording of a patient's data.
(2) Missing at random. 
    Example: Make informed decision at a particular point and then throw dice to decide to drop the recording of a patient's data.
    - Ex: If a patient's age is below certain threshold, selectively do not record the blood pressure.
    - Reason for this missing data is known and maybe recorded as part of the dataset.
(3) Missing not at random.
    Example: Drop recording of a patient's data when the queue is long.
    - Reason for this missing data is not known and there is no record of why the data was not recorded at all.
How?
(1) Drop it: Affects the model performance - high bias and/or variance.
(2) Imputation:
    - Mean: Not recommended especially when the data is imbalanced with some in the very low value range and some very high range.
    - Median: Better than mean.
    - Regression: Y = mX + c

Survival Probability:
S(t) = Pr(T > t)
- Probability of survical to time t (days or months or years).

Survival Function:
S(u) <= S(v) if u >= v
Example: u = 80, v = 20 (Age)

Censoring:
(1) End-of-study Censoring
(2) Loss-to-follow-up censoring.

How to estimate probability of survival past t with censored observations?
- Use chain rule of conditional probability.
S(t) = Pr(T > t) = Pr (T >= t+1) = Pr(T >= t+1, T >= t, T >= t-1, ...)
     = Pr(T >= t+1|T >= t)Pr(T>=t|T >= t-1)...Pr(T>=1|T>=0)Pr(T>=0)
Pr(T >= t|T >= t-1) = Pr(T > t-1|T >= t-1) = 1 - Pr(T=t-1|T >= t-1)
Pr(T=t-1|T >= t-1) = #died at t-1 / #known to survive to t-1 = di / ni
S(t) = multiply(1 - Pr(T=i|T>=i)) for all i = [0, t]
     = multiply(1 - di/ni)
     = Kaplan Meier Estimate of survival function of a population.

Hazard:
What is the a patient's immediate risk of death if they make it to time t?
lambda(t) = Pr(T=t|T >= t) = risk of death if aged t
Survival function in terms of the hazard function is S(t) = exp(-sum(hazard)) = exp(-H(t))

Cox Proportional Hazards Model:
- Individual profiled hazards model.
- lambda(t,p) = lambda(t,0) * exp(linear regression) where lambda(t,0) is a baseline hazard (of a population, for instance)
- Example of linear regression: 0.08 * is_smoker + 0.01 * age.
- exp(weight) risk factor increase for unit increase in variable. For example 50->51 (age), 0->1 (non-smoker to smoker). "weight" is the weight associated with the feature in the linear regression.
2 disadvantages:
(1) exp(linear) is not able to capture non-linear characteristics of a feature. For example young is riskier than middle-aged and old is riskier than middel-aged.
(2) lambda(t,p) between in dividuals is always linearly proportional to each other.

Survival Tree:
- Risk at every point in time which can be represented as either lambda or cumulative lambda or S(t).
- Survival time.
- Nelson Aalen Estimator: sum(di / ni)
Mortality Score:
- Use cummulative lambda plots from the Survival tree.
- Compare cumulative hazard at the times where we observe deaths in the population.
- sum(cummulative lambda) at all the event times = motality score of a patient.

Harrell's C-index:
- Evaluate Survival model.
Survival data:
Permissible Pairs:
(1) If neither patient was censored.
(2) If the censored patient's T is >= uncensored patient's T.

Randomized Control Trial:
- Randomly assign subjects to one or another group for observation to reduce bias.

Absolute Risk Reduction (ARR) = difference between absolute risk of treatment group and control group.
Number needed to Treat (NNT) = 1 / ARR. It tells us #people to receive the treatment in order to benefit one of them.
p-value: 
- There is less than p% probability that we would observe a difference risk of ARR or mroe if the treatment has zero effect. 
- If the p-value is small, the result is statistically significant.
- A function of #patients in each group.

Neyman-Rubin causal model:
Yi(1) - Outcome with treatment. 0: bad outcome does NOT happen, 1: bad outcome does happen
Yi(0) - Outcome without treatment. 0: bad outcome does NOT happen, 1: bad outcome does happen
mean(Yi(1)) - mean(Yi(0)) = Average Treatment Effect.
- Can be estimated only in Randomized Control Trial data.
- negative: benefit; positive: harm; zero: No effect of treatment.

ATE = -ARR

Conditional Average Treatment Effect:
- Individual Treatment Estimate (ITE)
- Use ML model to lean the relationships between the conditional features and the outcome, Yi (have or doesn't have the disease).
- Base learners. Decision Tree, linear regression, etc.
Two-Tree method (T-learner)
- miu1(x): Prognostic model for Yi(1)
- miu0(x): Prognostic model for Yi(0)
- miu1 - miu0
- Disadvantage: Splitting the dataset to 2 - Yi(1) and Yi(0) results in less data to learn for both trees.
Single-Tree method (S-learner)
- miu(x, w): w:0 for control group, w:1 for treatment group.
- miu(x, 1) - miu(x, 0)
- Disadvantage: 
  (1) May learn to NOT use the treatment feature at all.
  (2) Might produce estimate of 0 for everyone.

To evaluate ITE:
- Use data from subjects in the opposite group with similarity in either:
  (1) Features
  (2) ITE estimate.
- For the matched pair, calculate:
  (1) Y(1) - Y(0)
  (2) Average ITE.

c-for-benefit:
Permissible:
- Different outcomes.
Concordant:
- Higher ITE estimate has higher outcome.
Tie:
- Same ITE estiame with diff outcomes.
Compare the c-for-benefit results between T-learner and S-learner:
- P(TEa > TEb | YDa > YDb), YD = Y(1) - Y(0)

Feature Importance Evaluation of a model:
(1) Drop-column method.
    - Train models with the same training dataset with a column drop for each model and compare them.
    - Check the models' performance diff with the original model. Higher diff means the dropped column/feature is more important.
(2) Permutation:
    - During evaluation using the cross-validation dataset, shuffle the column of interest.
    - Check the model's performance drop. Higher drop means the indicate higher importance of the feature/column.

Feature Importance Evaluation of a prediction on individual:
(1) Drop-column method - Same as the evaluation of a model but instead of using the model performance index, it uses the prediction value on an individual's profile data.
    - Does NOT work in the presence of correlated features / covariance. For example, [age, SBP, dBP] and the diff in predictions using this method is almost identical.
(2) Shapley Values:
    - Work well with correlated features.
    - Shapley value = importance of the particular feature.
    - Empty set: Use the expected value of the outcome in the dataset. Example, if 10% of the population actually have the event, set f of the empty set to be 0.1
    - Shapley values sum up to the risk prediction with the full model minus the base risk in the population.
    - Use SHAP library.