Chapter 1: AutoML Fundamentals

Democratization of Machine Learning - Concepts and Components of AutoML

📖 Reading Time: 25-30 minutes 📊 Difficulty: Beginner 💻 Code Examples: 7 📝 Exercises: 5

This chapter covers the fundamentals of AutoML Fundamentals, which what is automl. You will learn differences from traditional ML workflows, Comprehend the components of AutoML, and fundamentals of Neural Architecture Search (NAS).

Learning Objectives

By reading this chapter, you will master the following:

1.1 What is AutoML

Democratization of Machine Learning

AutoML (Automated Machine Learning) is a technology that automates the machine learning model development process. It aims to enable even non-data scientists to build high-quality machine learning models.

“AutoML realizes the democratization of machine learning, allowing more people to utilize AI technology”

Purpose of AutoML

PurposeDescriptionEffect
EfficiencyAutomate manual processesReduce development time
Reduce Expertise RequirementsDeep machine learning knowledge not requiredLower barriers to entry
Performance ImprovementDiscover optimal solutions through systematic searchEliminate human bias
ReproducibilityStandardized processesImprove result reliability

Comparison with Traditional ML Workflow

```mermaid
graph TD
    subgraph "Traditional Workflow"
    A1[Data Collection] --> B1[Manual Preprocessing]
    B1 --> C1[Feature Engineering]
    C1 --> D1[Model Selection]
    D1 --> E1[Hyperparameter Tuning]
    E1 --> F1[Evaluation]
    F1 -->|Trial and Error| C1
    end

    subgraph "AutoML Workflow"
    A2[Data Collection] --> B2[Automatic Preprocessing]
    B2 --> C2[Automatic Feature Generation]
    C2 --> D2[Automatic Model Selection]
    D2 --> E2[Automatic Hyperparameter Optimization]
    E2 --> F2[Evaluation]
    end

    style A1 fill:#ffebee
    style A2 fill:#ffebee
    style B1 fill:#fff3e0
    style B2 fill:#e8f5e9
    style C1 fill:#f3e5f5
    style C2 fill:#e8f5e9
    style D1 fill:#e3f2fd
    style D2 fill:#e8f5e9
    style E1 fill:#fce4ec
    style E2 fill:#e8f5e9
```

Advantages and Disadvantages of AutoML

Advantages

Disadvantages

Example: Effects of AutoML

# Requirements:
# - Python 3.9+
# - numpy>=1.24.0, <2.0.0
# - pandas>=2.0.0, <2.2.0

"""
Example: Example: Effects of AutoML

Purpose: Demonstrate machine learning model training and evaluation
Target: Advanced
Execution time: 1-5 minutes
Dependencies: None
"""

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import time

# Data preparation
data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Traditional approach (fixed parameters)
start_time = time.time()
model_manual = RandomForestClassifier(n_estimators=100, random_state=42)
model_manual.fit(X_train, y_train)
y_pred_manual = model_manual.predict(X_test)
acc_manual = accuracy_score(y_test, y_pred_manual)
time_manual = time.time() - start_time

# AutoML-style simple implementation (grid search)
from sklearn.model_selection import GridSearchCV

start_time = time.time()
param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [None, 10, 20, 30],
    'min_samples_split': [2, 5, 10]
}
model_auto = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=3,
    n_jobs=-1
)
model_auto.fit(X_train, y_train)
y_pred_auto = model_auto.predict(X_test)
acc_auto = accuracy_score(y_test, y_pred_auto)
time_auto = time.time() - start_time

print("=== Traditional vs AutoML-style Approach ===")
print(f"\nTraditional Approach:")
print(f"  Accuracy: {acc_manual:.4f}")
print(f"  Time: {time_manual:.2f}s")

print(f"\nAutoML-style Approach:")
print(f"  Accuracy: {acc_auto:.4f}")
print(f"  Time: {time_auto:.2f}s")
print(f"  Best Parameters: {model_auto.best_params_}")

print(f"\nImprovement:")
print(f"  Accuracy Gain: {(acc_auto - acc_manual) * 100:.2f}%")

Example Output :

=== Traditional vs AutoML-style Approach ===

Traditional Approach:
  Accuracy: 0.9649
  Time: 0.15s

AutoML-style Approach:
  Accuracy: 0.9737
  Time: 12.34s
  Best Parameters: {'max_depth': 20, 'min_samples_split': 2, 'n_estimators': 100}

Improvement:
  Accuracy Gain: 0.88%

1.2 Components of AutoML

AutoML systems consist of multiple components to automate the entire machine learning pipeline.

Data Preprocessing Automation

Automates transformation from raw data to learnable formats:

Feature Engineering

Automatically generates new features:

Model Selection

Automatically selects the optimal algorithm for the task and data:

Hyperparameter Optimization

Automatically tunes model parameters:

AutoML Workflow Diagram

```mermaid
graph TD
    A[Raw Data] --> B[Data Preprocessing Automation]
    B --> C[Feature Engineering]
    C --> D[Model Selection]
    D --> E[Hyperparameter Optimization]
    E --> F[Ensemble]
    F --> G[Final Model]

    B --> B1[Missing Value Handling]
    B --> B2[Outlier Detection]
    B --> B3[Scaling]

    C --> C1[Polynomial Features]
    C --> C2[Aggregate Features]
    C --> C3[Feature Selection]

    D --> D1[Linear Models]
    D --> D2[Tree-based]
    D --> D3[Neural Networks]

    E --> E1[Grid Search]
    E --> E2[Bayesian Optimization]
    E --> E3[Evolutionary Methods]

    style A fill:#ffebee
    style B fill:#fff3e0
    style C fill:#f3e5f5
    style D fill:#e3f2fd
    style E fill:#fce4ec
    style F fill:#e8f5e9
    style G fill:#c8e6c9
```

1.3 Neural Architecture Search (NAS)

Concept of NAS

Neural Architecture Search (NAS) is a technology that automatically designs neural network architectures. Algorithms automatically search for network structures that were manually designed by humans.

NAS can be described as “a neural network that designs neural networks”

Search Space

Design elements explored by NAS:

Search Strategies

Randomly samples and evaluates architectures. Simple but inefficient.

2. Reinforcement Learning-based

A controller (RNN) generates architectures and learns using their performance as rewards.

Reward function:

$$ R = \text{Accuracy} - \lambda \cdot \text{Complexity} $$

3. Evolutionary Algorithms

Uses genetic algorithms to evolve superior architectures.

4. Gradient-based Methods (DARTS)

Relaxes the search space to be continuous and optimizes using gradient descent. Computationally efficient.

NAS Implementation Example (Simplified Version)

# Requirements:
# - Python 3.9+
# - numpy>=1.24.0, <2.0.0

"""
Example: NAS Implementation Example (Simplified Version)

Purpose: Demonstrate machine learning model training and evaluation
Target: Advanced
Execution time: 30-60 seconds
Dependencies: None
"""

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.neural_network import MLPClassifier

# Data preparation
digits = load_digits()
X_train, X_test, y_train, y_test = train_test_split(
    digits.data, digits.target, test_size=0.2, random_state=42
)

# Simple NAS: Explore architectures with random search
def random_architecture_search(n_trials=10):
    best_score = 0
    best_architecture = None

    print("=== Neural Architecture Search ===")
    for i in range(n_trials):
        # Randomly generate architecture
        n_layers = np.random.randint(1, 4)  # 1-3 layers
        hidden_layer_sizes = tuple(
            np.random.choice([32, 64, 128, 256]) for _ in range(n_layers)
        )
        activation = np.random.choice(['relu', 'tanh', 'logistic'])

        # Train and evaluate model
        model = MLPClassifier(
            hidden_layer_sizes=hidden_layer_sizes,
            activation=activation,
            max_iter=100,
            random_state=42
        )
        model.fit(X_train, y_train)
        score = model.score(X_test, y_test)

        print(f"Trial {i+1}: layers={hidden_layer_sizes}, "
              f"activation={activation}, score={score:.4f}")

        if score > best_score:
            best_score = score
            best_architecture = {
                'hidden_layer_sizes': hidden_layer_sizes,
                'activation': activation,
                'score': score
            }

    return best_architecture

# Run NAS
best_arch = random_architecture_search(n_trials=10)

print(f"\n=== Best Architecture ===")
print(f"Layer Configuration: {best_arch['hidden_layer_sizes']}")
print(f"Activation Function: {best_arch['activation']}")
print(f"Accuracy: {best_arch['score']:.4f}")

NAS Challenges

ChallengeDescriptionCountermeasure
Computational CostEvaluating thousands of architecturesEarly stopping, proxy task usage
Search Space SizeCombinatorial explosionSearch space constraints, hierarchical search
Lack of TransferabilitySearch needed for each taskTransfer learning, meta-learning utilization
OverfittingOverfitting to validation dataRegularization, use multiple datasets

1.4 Meta-Learning

Learning to Learn

Meta-Learning is a method that “learns how to learn.” It leverages experience from past tasks to efficiently learn new tasks.

“Learning the learning algorithm itself” - The essence of meta-learning

Few-shot Learning

A method for efficiently learning from a small number of samples.

N-way K-shot learning :

Transfer Learning

Transfer knowledge learned on one task to another task.

Warm-starting

Use optimal parameters from past tasks as initial values to accelerate learning on new tasks.

from sklearn.linear_model import SGDClassifier
from sklearn.datasets import make_classification

# Task 1 and Task 2 (similar tasks)
X1, y1 = make_classification(n_samples=1000, n_features=20,
                             n_informative=15, random_state=42)
X2, y2 = make_classification(n_samples=1000, n_features=20,
                             n_informative=15, random_state=43)

print("=== Warm-starting Effect Verification ===")

# Cold start (learn Task 2 from scratch)
model_cold = SGDClassifier(max_iter=100, random_state=42)
model_cold.fit(X2[:100], y2[:100])  # Learn with small data
score_cold = model_cold.score(X2[100:], y2[100:])

# Warm start (pre-train on Task 1)
model_warm = SGDClassifier(max_iter=100, random_state=42)
model_warm.fit(X1, y1)  # Learn on Task 1
model_warm.partial_fit(X2[:100], y2[:100])  # Additional learning on Task 2
score_warm = model_warm.score(X2[100:], y2[100:])

print(f"Cold Start Accuracy: {score_cold:.4f}")
print(f"Warm Start Accuracy: {score_warm:.4f}")
print(f"Improvement: {(score_warm - score_cold) * 100:.2f}%")

1.5 AutoML Evaluation

Performance Metrics

Metrics for evaluating AutoML system performance:

MetricDescriptionImportance
Prediction AccuracyModel prediction performanceMost important
Search TimeTime to find optimal modelPractically important
Computational CostRequired resources (CPU, GPU, memory)Scalability
RobustnessStability across different datasetsGenerality

Computational Cost

Quantifying AutoML computational cost:

$$ \text{Total Cost} = \sum_{i=1}^{n} C_i \times T_i $$

Reproducibility

Whether the same input produces the same results:

Interpretability

Understanding AutoML’s decision process:

1.6 Chapter Summary

What We Learned

  1. AutoML Concepts

    • Realizing democratization of machine learning
    • Efficiency and reduced expertise requirements
    • Differences and advantages over traditional methods

  2. AutoML Components

    • Data preprocessing automation
    • Feature engineering
    • Model selection and hyperparameter optimization

  3. Neural Architecture Search

    • Automatic network structure design
    • Search strategies (RL, evolutionary, gradient-based)
    • Battle with computational cost

  4. Meta-Learning

    • Learning how to learn
    • Few-shot learning, Transfer learning
    • Acceleration through warm-starting

  5. AutoML Evaluation

    • Performance metrics (accuracy, time, cost)
    • Importance of reproducibility and interpretability

AutoML Principles

PrincipleDescription
Balance Automation and TransparencyAvoid black boxes, maintain interpretability
EfficiencySearch strategies considering computational resources
GeneralityApplicable to various tasks and data
Leverage Domain KnowledgeCombination of automation and expertise
Continuous ImprovementImprove learning efficiency through meta-learning

Next Chapter

In Chapter 2, we will learn about AutoML Tools and Frameworks :

Exercises

Question 1 (Difficulty: easy)

List three main purposes of AutoML and explain each.

Answer Example

Answer :

  1. Efficiency

    • Description: Automate manual model development processes and significantly reduce development time
    • Effect: Can reduce work from weeks to hours

  2. Reduce Expertise Requirements

    • Description: Enable building high-quality models without deep machine learning expertise
    • Effect: More people can utilize AI technology (democratization)

  3. Performance Improvement

    • Description: Discover optimal combinations that humans might overlook through systematic search
    • Effect: Eliminate human bias and objectively find the best model

Question 2 (Difficulty: medium)

Explain four search strategies for Neural Architecture Search (NAS) and describe the advantages and disadvantages of each.

Answer Example

Answer :

Search StrategyDescriptionAdvantagesDisadvantages
Random SearchRandomly sample architecturesSimple implementation, easy parallelizationInefficient, unsuitable for large-scale search
Reinforcement Learning-basedRNN controller generates architecturesEfficiently explores promising regionsHigh computational cost, stability issues
Evolutionary AlgorithmsEvolve superior architectures through genetic operationsMaintains diversity, avoids local optimaSlow convergence, requires large populations
Gradient-based (DARTS)Relax search space and optimize with gradient descentComputationally efficient, fastDiscretization errors, search space constraints

Question 3 (Difficulty: medium)

Explain what “5-way 3-shot learning” means in few-shot learning and calculate the number of training samples in this setting.

Answer Example

Answer :

Meaning of “5-way 3-shot learning” :

Number of training samples :

$$ \text{Number of samples} = \text{Number of classes} \times \text{Samples per class} = 5 \times 3 = 15 $$

That is, learning 5-class classification with only 15 samples.

Concrete example :

Question 4 (Difficulty: hard)

Complete the following code to implement a simple AutoML system. Include data preprocessing, model selection, and hyperparameter optimization.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

# Data preparation
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Implement AutoML system here

Answer Example

# Requirements:
# - Python 3.9+
# - numpy>=1.24.0, <2.0.0

"""
Example: Complete the following code to implement a simple AutoML sys

Purpose: Demonstrate machine learning model training and evaluation
Target: Advanced
Execution time: 1-5 minutes
Dependencies: None
"""

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
import numpy as np

# Data preparation
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

print("=== Simple AutoML System ===\n")

# Step 1: Define model candidates and hyperparameter space
models = {
    'Logistic Regression': {
        'model': LogisticRegression(max_iter=1000),
        'params': {
            'classifier__C': [0.1, 1.0, 10.0],
            'classifier__penalty': ['l2']
        }
    },
    'Random Forest': {
        'model': RandomForestClassifier(random_state=42),
        'params': {
            'classifier__n_estimators': [50, 100, 200],
            'classifier__max_depth': [None, 10, 20],
            'classifier__min_samples_split': [2, 5]
        }
    },
    'SVM': {
        'model': SVC(),
        'params': {
            'classifier__C': [0.1, 1.0, 10.0],
            'classifier__kernel': ['rbf', 'linear']
        }
    }
}

# Step 2: Preprocessing pipeline + hyperparameter optimization for each model
best_overall_score = 0
best_overall_model = None
best_overall_name = None

for name, config in models.items():
    print(f"--- {name} ---")

    # Build pipeline (preprocessing + model)
    pipeline = Pipeline([
        ('scaler', StandardScaler()),
        ('classifier', config['model'])
    ])

    # Hyperparameter optimization with grid search
    grid_search = GridSearchCV(
        pipeline,
        param_grid=config['params'],
        cv=5,
        scoring='accuracy',
        n_jobs=-1
    )

    grid_search.fit(X_train, y_train)

    # Results
    cv_score = grid_search.best_score_
    test_score = grid_search.score(X_test, y_test)

    print(f"  Best CV Score: {cv_score:.4f}")
    print(f"  Test Score: {test_score:.4f}")
    print(f"  Best Parameters: {grid_search.best_params_}")
    print()

    # Update best model
    if cv_score > best_overall_score:
        best_overall_score = cv_score
        best_overall_model = grid_search.best_estimator_
        best_overall_name = name

# Step 3: Final results
print("=" * 50)
print(f"Best Model: {best_overall_name}")
print(f"CV Score: {best_overall_score:.4f}")
print(f"Test Score: {best_overall_model.score(X_test, y_test):.4f}")
print("=" * 50)

Example Output :

=== Simple AutoML System ===

--- Logistic Regression ---
  Best CV Score: 0.9780
  Test Score: 0.9825
  Best Parameters: {'classifier__C': 1.0, 'classifier__penalty': 'l2'}

--- Random Forest ---
  Best CV Score: 0.9648
  Test Score: 0.9649
  Best Parameters: {'classifier__max_depth': None, ...}

--- SVM ---
  Best CV Score: 0.9758
  Test Score: 0.9737
  Best Parameters: {'classifier__C': 1.0, 'classifier__kernel': 'linear'}

==================================================
Best Model: Logistic Regression
CV Score: 0.9780
Test Score: 0.9825
==================================================

Question 5 (Difficulty: hard)

Explain the tradeoff between “computational cost” and “prediction accuracy” in AutoML, and describe how to balance them practically.

Answer Example

Answer :

Essence of Tradeoff :

AspectHigh Accuracy PursuitLow Cost Pursuit
Search RangeExtensive search (thousands of models)Limited search (dozens of models)
TimeDays to weeksHours to days
ResourcesLarge-scale GPU/clusterSingle machine
Accuracy Improvement+1-2% improvementBaseline achievement

Strategies for Balance :

  1. Staged Approach

    • Phase 1: Fast search to narrow down promising model candidates (hours)
    • Phase 2: Detailed optimization on candidates (days)

  2. Early Stopping

    • Terminate search if validation accuracy doesn’t improve
    • Set computational budget limits (time/cost)

  3. Efficient Search Methods

    • Use Bayesian optimization instead of random search
    • Improve initial state with transfer learning or meta-learning

  4. Task-dependent Priorities

    • Production systems: Accuracy priority (allow high cost)
    • Prototypes: Speed priority (emphasize low cost)
    • Research: Balance both

  5. Multi-objective Optimization

    • Include computational cost in objective function

$$ \text{Objective} = \alpha \cdot \text{Accuracy} - (1-\alpha) \cdot \log(\text{Cost}) $$

Practical Recommendations :

References

  1. Hutter, F., Kotthoff, L., & Vanschoren, J. (Eds.). (2019). Automated Machine Learning: Methods, Systems, Challenges. Springer.
  2. Elsken, T., Metzen, J. H., & Hutter, F. (2019). Neural Architecture Search: A Survey. Journal of Machine Learning Research , 20(55), 1-21.
  3. Hospedales, T., Antoniou, A., Micaelli, P., & Storkey, A. (2021). Meta-Learning in Neural Networks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  4. Feurer, M., & Hutter, F. (2019). Hyperparameter Optimization. In Automated Machine Learning (pp. 3-33). Springer.
  5. He, X., Zhao, K., & Chu, X. (2021). AutoML: A survey of the state-of-the-art. Knowledge-Based Systems , 212, 106622.
Back to Series Index Next Chapter →