- AI is the broadest concept. It refers to the simulation of human intelligence in machines that are programmed to think, learn, and solve problems.
- AI aims to create systems capable of performing tasks that normally require human intelligence, such as reasoning, planning, understanding natural language, vision, decision making, etc.
- AI includes any technique or method that enables machines to mimic human cognitive functions.
- Can be rule-based systems (expert systems), symbolic logic, search algorithms, robotics, and more.
- Chess-playing programs.
- Voice assistants (Siri, Alexa).
- Autonomous vehicles.
- Spam filters.
- AI = the overall science of mimicking human intelligence.
- Encompasses all subfields including ML and DL.
- ML is a subset of AI focused specifically on the idea that machines can learn from data and improve from experience without being explicitly programmed with specific rules.
- Instead of hard-coding instructions, ML algorithms identify patterns in data to make predictions or decisions.
- You provide data (input and, optionally, labels).
- The algorithm builds a model by learning patterns from data.
- The model makes predictions on new, unseen data.
- Supervised learning: learns from labeled data (input-output pairs).
- Unsupervised learning: finds patterns in unlabeled data.
- Reinforcement learning: learns by interacting with the environment to maximize reward.
- Email spam detection.
- Fraud detection.
- Recommendation systems.
- Image classification.
- ML = algorithms that learn from data to perform tasks.
- Subset of AI focused on data-driven learning.
- DL is a subset of machine learning that uses artificial neural networks with many layers (hence “deep”) to model complex patterns in large amounts of data.
- It’s inspired by the structure and function of the brain’s neural networks.
- Uses multi-layered neural networks (deep neural networks).
- Learns hierarchical feature representations (low-level to high-level abstractions).
- Requires large datasets and high computational power (GPUs).
- Image recognition (e.g., detecting objects in photos).
- Natural language processing (e.g., language translation, chatbots).
- Speech recognition.
- Autonomous driving perception.
- DL = deep neural networks learning from large data.
- Subset of ML with more complex models and more powerful representation learning.
Artificial Intelligence (AI)
└── Machine Learning (ML)
└── Deep Learning (DL)
- DL is a specialized type of ML.
- ML is one way to achieve AI.
- AI also includes other approaches beyond ML (like rule-based systems).
| Aspect | AI | ML | DL |
|---|---|---|---|
| Definition | Broad science of making machines intelligent | Algorithms that learn from data | Neural networks with many layers learning from large data |
| Focus | Mimic human intelligence broadly | Learn patterns from data | Automatically learn hierarchical features in data |
| Techniques | Rules, logic, search, ML, DL, etc. | Regression, decision trees, SVM, neural nets | Deep neural networks, CNNs, RNNs |
| Data requirement | Can be manual rules-based or data-driven | Needs data | Needs large datasets |
| Computation | Can be simple or complex | Medium complexity | High computational power (GPUs often needed) |
| Examples | Expert systems, game AI, robotics | Spam filters, recommendation systems | Image/speech recognition, language translation |
| Interpretability | Often interpretable (rules-based) | Moderate | Often “black-box” and harder to interpret |
- AI is the big umbrella, everything related to making machines smart.
- ML is a way machines get smart by learning from data.
- DL is a powerful kind of ML that uses deep neural networks to learn very complex patterns from massive data.
- Machine Learning is a field of AI that gives computers the ability to learn patterns from data and make decisions or predictions.
- ML algorithms automatically improve their performance as they are exposed to more data.
- The model learns from labeled data — each input example has a corresponding output label.
- The goal is to learn a mapping from inputs to outputs so the model can predict the output for new, unseen inputs.
- Classification: Predict categorical labels.
- Regression: Predict continuous values.
- The model learns from unlabeled data — only input data is given, no output labels.
- The goal is to find structure, patterns, or relationships in the data.
- Clustering: Group data into clusters.
- Dimensionality Reduction: Simplify data by reducing features.
-
Definition: Predict a discrete class label from input features.
-
Output: One of several categories (e.g., spam or not spam).
-
Examples:
- Email spam detection (spam/not spam).
- Handwritten digit recognition (digits 0-9).
- Disease diagnosis (disease present/absent).
- Logistic Regression
- Support Vector Machines (SVM)
- Decision Trees and Random Forests
- Neural Networks
-
Definition: Predict a continuous output value based on input features.
-
Output: Real numbers (e.g., price, temperature).
-
Examples:
- Predicting house prices.
- Forecasting stock prices.
- Predicting temperature.
- Linear Regression
- Polynomial Regression
- Support Vector Regression (SVR)
- Neural Networks
-
Definition: Group similar data points into clusters based on features.
-
Goal: Discover natural groupings in data.
-
Examples:
- Customer segmentation for marketing.
- Grouping news articles by topic.
- Image segmentation.
- K-Means Clustering
- Hierarchical Clustering
- DBSCAN (Density-Based Spatial Clustering)
-
Definition: Reduce the number of input features while preserving important information.
-
Goal: Simplify data, visualize high-dimensional data.
-
Examples:
- Visualizing high-dimensional data in 2D or 3D.
- Preprocessing before clustering or classification.
- Principal Component Analysis (PCA)
- t-Distributed Stochastic Neighbor Embedding (t-SNE)
- Autoencoders (neural networks)
- Definition: Discover interesting relationships or patterns between variables in large datasets.
- Goal: Find rules that describe likely co-occurrences of items or events.
- Examples:
- Market basket analysis (what products are bought together).
- Web usage patterns (which pages are visited together).
- Medical symptom associations.
- Apriori Algorithm
- FP-Growth (Frequent Pattern Growth)
- ECLAT (Equivalence Class Transformation)
- Support: Frequency of item combinations
- Confidence: Probability of Y given X
- Lift: Strength of association between items
- Retail store product placement
- Recommendation systems
- Cross-selling strategies
- Website navigation design
- Disease co-occurrence analysis
| Metric | Formula | Description |
|---|---|---|
| Support | P(X ∩ Y) | How often items appear together |
| Confidence | P(Y|X) | How often Y occurs when X occurs |
| Lift | P(X ∩ Y) / (P(X) × P(Y)) | Independence measure between items |
- Uses a small amount of labeled data and a large amount of unlabeled data.
- Useful when labeling is expensive or time-consuming.
- Combines supervised and unsupervised learning techniques.
- An agent learns to make decisions by interacting with an environment.
- Learns through rewards or punishments (feedback).
- Used in robotics, game playing, autonomous driving.
| Learning Type | Data Used | Task Types | Goal | Examples |
|---|---|---|---|---|
| Supervised Learning | Labeled data | Classification, Regression | Predict outputs from inputs | Spam detection, price prediction |
| Unsupervised Learning | Unlabeled data | Clustering, Dimensionality Reduction, Association | Find patterns/groups in data | Customer segmentation, PCA |
| Semi-Supervised Learning | Small labeled + large unlabeled | Mix of supervised & unsupervised | Improve learning with limited labels | Web page classification |
| Reinforcement Learning | Environment feedback | Policy learning, decision making | Maximize cumulative reward | Game AI, robotics |
-
Traditional ML models often require manual feature engineering: domain experts need to handcraft features from raw data (e.g., edges, textures in images).
-
Deep learning uses deep neural networks with many layers that automatically learn hierarchical representations:
- Lower layers learn simple features (edges, colors).
- Higher layers learn complex concepts (faces, objects, emotions).
-
This automatic feature extraction reduces the need for manual intervention and can discover subtle patterns humans might miss.
- Modern datasets (images, audio, text, video) are large-scale and high-dimensional.
- Deep networks are capable of modeling very complex functions in these large feature spaces.
- Traditional algorithms struggle with high dimensionality or require dimensionality reduction beforehand.
- Deep learning naturally handles raw inputs without heavy preprocessing.
-
Deep learning has achieved significant breakthroughs in fields such as:
- Computer Vision: Image recognition, object detection, segmentation.
- Natural Language Processing (NLP): Translation, summarization, chatbots.
- Speech Recognition: Transcribing spoken language.
- Game Playing: Beating human champions (e.g., AlphaGo).
-
Many benchmarks and competitions show deep learning outperforms classical ML methods.
- Deep learning models can be trained end-to-end, directly from raw input data to the final output.
- This contrasts with traditional ML pipelines which may require multiple disconnected stages (feature extraction, dimensionality reduction, classification).
- End-to-end models simplify workflows and reduce error propagation.
-
Deep learning architectures can be adapted for various data types:
- Convolutional Neural Networks (CNNs) for images and spatial data.
- Recurrent Neural Networks (RNNs) and Transformers for sequential and language data.
- Autoencoders for unsupervised learning and dimensionality reduction.
-
This versatility makes deep learning applicable across many domains.
- Deep learning models tend to improve with more data.
- The availability of large datasets and powerful hardware (GPUs, TPUs) has fueled deep learning’s success.
- Many traditional ML models saturate after a certain point, while deep learning can keep improving.
- Deep networks can be more robust to variations and noise in data due to learned representations.
- Techniques like dropout, batch normalization, and data augmentation improve generalization.
| Challenge | Traditional ML | Deep Learning |
|---|---|---|
| Feature Engineering | Requires manual design | Learns features automatically |
| Large-scale data | May struggle or need simplification | Scales well with big data |
| Complex patterns | Limited expressiveness | Can model complex hierarchical features |
| Unstructured data (images, text) | Requires heavy preprocessing | Handles raw data end-to-end |
| Multimodal data | Difficult to integrate | Can integrate multiple data types |
| Reason | Explanation |
|---|---|
| Automatic Feature Learning | Learns complex features from raw data without manual design |
| High-dimensional data handling | Works well with images, audio, text, video |
| Superior performance | Leads benchmarks in many AI tasks |
| End-to-end training | Simplifies pipelines |
| Versatility | Adaptable architectures for different data types |
| Data and compute scalability | Performance improves with more data and hardware |
| Robustness | More resilient to noise and variability |
- A perceptron is a simple binary classifier that maps input features to a binary output (usually 0 or 1).
- It’s a fundamental building block of neural networks.
- Designed to separate data points into two classes using a linear decision boundary.
- Inputs:
$\mathbf{x} = [x_1, x_2, ..., x_n]$ - Weights:
$\mathbf{w} = [w_1, w_2, ..., w_n]$ - Bias:
$b$ (sometimes considered$w_0$ with input$x_0=1$ ) - Weighted sum:
$z = \mathbf{w} \cdot \mathbf{x} + b = \sum_{i=1}^n w_i x_i + b$ - Activation function (Step function):
- For each input vector
$\mathbf{x}$ , compute$z = \mathbf{w} \cdot \mathbf{x} + b$ . - Apply the step function to
$z$ to get output$y$ . - The output predicts the class label (0 or 1).
The goal of the perceptron learning algorithm is to find the weights
-
Training dataset
${(\mathbf{x}^{(1)}, y^{(1)}), (\mathbf{x}^{(2)}, y^{(2)}), ..., (\mathbf{x}^{(m)}, y^{(m)})}$ -
$\mathbf{x}^{(i)}$ is the input vector for the$i^{th}$ example. -
$y^{(i)} \in {0,1}$ is the true label.
-
- Initialize weights and bias to small random values or zeros:
-
For each training example
$(\mathbf{x}^{(i)}, y^{(i)})$ :a. Compute the predicted output
$\hat{y}$ :
b. Calculate the error:
c. Update the weights and bias if the prediction is wrong (error
Here,
- Repeat Step 2 for all training examples until:
- The algorithm converges (no errors on all examples), or
- A maximum number of iterations (epochs) is reached.
-
If the perceptron correctly classifies the example, no update occurs.
-
If the perceptron misclassifies an example, weights are updated to reduce future error:
- If predicted output is too low (predicted 0 but true label 1), increase weights on active features.
- If predicted output is too high (predicted 1 but true label 0), decrease weights on active features.
-
Bias updated similarly to shift the decision boundary.
Suppose:
-
$\mathbf{x} = [2, 3]$ , label$y=1$ - Initial weights
$\mathbf{w} = [0, 0]$ , bias$b=0$ - Learning rate
$\eta = 0.1$
Step 1: Compute prediction
Step 2: Error
No update needed.
Now suppose another input:
-
$\mathbf{x} = [1, 1]$ , label$y=0$ - Current weights
$\mathbf{w} = [0, 0]$ , bias$b=0$
Step 1: Compute prediction
Step 2: Error
Step 3: Update weights and bias:
- Convergence: The perceptron learning algorithm converges only if the data is linearly separable.
-
Learning Rate
$\eta$ : Controls step size in updates. Too large can overshoot, too small slows learning. - Extension: The perceptron can be extended to multi-layer perceptrons (MLPs) which can solve non-linear problems.
| Step | Description |
|---|---|
| Initialize weights and bias | Set weights |
| Predict | Compute weighted sum |
| Calculate error | |
| Update weights and bias |
|
| Repeat | Iterate over dataset until convergence or max iterations |
- Outputs input as-is.
- No non-linearity.
- Often used in output layers for regression problems where the target is continuous.
- Differentiable everywhere.
- Does not allow the network to model non-linear relationships if used in all layers.
- Simple, efficient.
- Allows for continuous output in regression tasks.
- Cannot capture non-linear patterns if used as the only activation.
- Output layer in regression tasks (predicting prices, temperatures, etc.).
- Binary output 0 or 1.
- Used in original perceptron to decide whether a neuron fires.
- Non-differentiable at
$x=0$ . - Produces hard binary decisions.
- Simple binary classifier.
- Intuitive threshold-based output.
- No gradient for training with gradient-based methods.
- Can only solve linearly separable problems.
- Original perceptrons and simple binary classification without gradient learning.
- Outputs -1 or +1.
- Symmetric binary output useful when labels are coded as
$\pm 1$ .
- Non-differentiable.
- Hard binary output.
- Same as threshold function but symmetric outputs.
- Binary classification with
$\pm 1$ labels.
- Outputs a smooth curve from 0 to 1.
- Converts any real-valued input into a probability-like output.
- Differentiable everywhere.
- Output range: (0, 1).
- Not zero-centered (outputs always positive).
- Gradients get very small for very positive or negative inputs (vanishing gradient problem).
- Good for probabilistic binary classification.
- Enables gradient-based optimization.
- Vanishing gradient slows training in deep networks.
- Outputs not zero-centered which can slow convergence.
- Output layers for binary classification.
- Sometimes hidden layers in shallow networks (though less common now).
- Similar shape to sigmoid but outputs between -1 and 1.
- Zero-centered output helps with optimization.
- Differentiable.
- Output range: (-1, 1).
- Steeper gradient near zero than sigmoid.
- Zero-centered outputs improve learning speed.
- Useful in hidden layers.
- Still suffers from vanishing gradients for large magnitudes.
- Computationally more expensive than ReLU.
- Hidden layers in small to medium-sized networks.
- Outputs zero for negative inputs, linear for positive.
- Introduces sparsity (many neurons inactive).
- Differentiable everywhere except at zero (works fine in practice).
- No vanishing gradient for positive inputs.
- Efficient computation.
- Mitigates vanishing gradient problem.
- Encourages sparse activations, often improving generalization.
- Can cause "dying ReLU" problem (neurons stuck outputting 0).
- Not zero-centered.
- Most common choice for hidden layers in deep networks.
For an input vector
- Converts the input vector into a probability distribution over
$K$ classes. - Each output
$y_i$ is between 0 and 1, and all outputs sum to 1. - Useful for multi-class classification tasks.
- Differentiable everywhere.
- Outputs normalized probabilities.
- Amplifies differences between inputs (the highest input gets the highest output probability).
- Ideal for multi-class classification output layers.
- Allows model to predict the most likely class.
- Works well with cross-entropy loss for training.
- Computationally expensive for very large output spaces.
- Sensitive to very large or very small input values (can be stabilized numerically).
- Output layer for multi-class classification problems (e.g., recognizing digits 0–9).
- When your output must represent a probability distribution over multiple classes.
- Use in the final output layer of a network performing multi-class classification.
- Commonly paired with cross-entropy loss for training.
| Task/Layer | Recommended Activation(s) | Why? |
|---|---|---|
| Output layer (binary classification) | Sigmoid | Outputs probability between 0 and 1. |
| Output layer (multi-class classification) | Softmax | Probability distribution over classes. |
| Output layer (regression) | Linear | Predict continuous values directly. |
| Hidden layers (shallow networks) | tanh or sigmoid | Smooth gradients, zero-centered (tanh better). |
| Hidden layers (deep networks) | ReLU (or variants like Leaky ReLU) | Fast convergence, avoids vanishing gradients. |
| Simple perceptron models | Step (threshold) or sign | Binary decisions for linearly separable problems. |
| Output layer (multi-class with probabilities) | Softmax | Normalizes outputs to sum to 1, preserves relative magnitudes. |
- Use non-linear activation functions (ReLU, tanh, sigmoid) in hidden layers to introduce non-linearity.
- Use a linear activation in the output layer if the output is continuous (regression).
- Use sigmoid in output for binary classification problems.
- For deep networks, avoid sigmoid/tanh in hidden layers because of vanishing gradient; prefer ReLU.
- Sometimes you might combine ReLU hidden layers with a sigmoid/tanh output layer, depending on task.
- For multi-class classification, the output usually uses softmax (generalization of sigmoid), but that’s beyond perceptrons.
| Activation Function | Formula/Definition | Output Range | Differentiable? | Zero-centered? | Typical Use Case |
|---|---|---|---|---|---|
| Linear | Yes | Yes | Output layer for regression problems | ||
| Threshold (Step) | {0, 1} | No | No | Classic perceptron binary classification | |
| Sign | {-1, 1} | No | Yes | Binary classification with |
|
| Sigmoid | (0, 1) | Yes | No | Binary classification output layers | |
| tanh | (-1, 1) | Yes | Yes | Hidden layers (small/medium networks) | |
| ReLU | [0, |
Yes (almost) | No | Hidden layers in deep networks | |
| Softmax | [0, 1], sum=1 | Yes | No | Output layer for multi-class classification |
- Linear: Regression output.
- Step/Sign: Classic binary classifiers, non-differentiable, simple perceptrons.
- Sigmoid: Output probabilities, smooth, but vanishing gradient.
- tanh: Zero-centered, better than sigmoid for hidden layers, still has vanishing gradient.
- ReLU: Default choice for hidden layers in deep nets, sparse activations, fast training.