Machine Learning · Topic 2 of 2
Key Machine Learning Methods and Common Algorithms
Learn gradient descent, hyperparameters, grid search, k-fold cross-validation, and linear regression, logistic regression, trees, SVM, KNN and k-means.
- 5 min
- Gradient descent steps against the gradient; BGD uses all samples, SGD one, mini-batch a small group.
- Parameters are learned; hyperparameters are set before training and tuned with grid or random search.
- K-fold cross-validation trains k times, each with a different validation fold.
What it is
Key machine learning methods are the techniques that make training work: optimizing a loss with gradient descent, choosing hyperparameters, and checking models with cross-validation.
Common machine learning algorithms are the standard model families, such as linear regression, decision trees, SVM and k-means. Each fits a certain kind of task.
How it works
Gradient descent
Training means finding parameters that minimize a loss function, such as mean squared error. Gradient descent moves each parameter a small step against the gradient:
w_new = w_old - learning_rate x gradient
| Variant | Samples used per update | Character |
|---|---|---|
| Batch gradient descent (BGD) | All training samples | Stable but slow on large data |
| Stochastic gradient descent (SGD) | One sample | Fast, noisy updates |
| Mini-batch gradient descent (MBGD) | A small batch, such as 32 or 64 | The usual compromise |
The learning rate sets the step size. Too large and training oscillates or diverges; too small and it converges very slowly.
Parameters versus hyperparameters
| Kind | Who sets it | Examples |
|---|---|---|
| Parameters | Learned from data during training | Weights and bias of linear regression, split points of a tree |
| Hyperparameters | Set by the engineer before training | Learning rate, k in KNN, tree depth, number of clusters, regularization strength |
Hyperparameter search
Grid search tries every combination from fixed lists of values, which is thorough but costly. Random search samples combinations at random, which often finds good values faster when only a few hyperparameters matter.
Cross-validation
In k-fold cross-validation, the data is split into k equal folds. The model trains k times, each time validating on a different fold and training on the other k - 1, and the k scores are averaged.
With 1,000 samples and k = 5, each round trains on 800 samples and validates on 200. This gives a more reliable estimate than a single split.
Regularization
Regularization adds a penalty on large weights to the loss to reduce overfitting. L1 (Lasso) adds the sum of absolute weights and drives some weights to exactly zero. L2 (Ridge) adds the sum of squared weights and shrinks weights smoothly.
Common algorithms
| Algorithm | Type | Task | Key idea |
|---|---|---|---|
| Linear regression | Supervised | Regression | Fits y = wx + b by minimizing squared error |
| Logistic regression | Supervised | Classification | Linear score passed through the sigmoid to give a probability |
| Decision tree | Supervised | Classification or regression | Splits data on features using information gain, gain ratio or Gini index |
| Support vector machine (SVM) | Supervised | Classification (also regression) | Finds the maximum margin boundary; kernels handle nonlinear data |
| K-nearest neighbors (KNN) | Supervised | Classification or regression | Uses the majority label (or average) of the k closest samples |
| Naive Bayes | Supervised | Classification | Bayes' theorem, assuming features are independent given the class |
| Random forest | Supervised, ensemble | Classification or regression | Many trees on bootstrap samples (bagging), combined by vote or average |
| GBDT | Supervised, ensemble | Classification or regression | Trees added one by one, each fitting the errors of the previous ones (boosting) |
| K-means | Unsupervised | Clustering | Assigns points to the nearest of k centroids, then moves centroids, repeated |
Despite its name, logistic regression is a classification algorithm.
Decision tree splitting
ID3 uses information gain, C4.5 uses gain ratio, and CART uses the Gini index. The tree picks the feature that makes child nodes purest.
Trees overfit easily when they grow deep. Pruning and depth limits control this.
Ensemble learning
| Method | How models are combined | Effect |
|---|---|---|
| Bagging | Models trained in parallel on bootstrap samples, then voted or averaged | Lowers variance |
| Boosting | Models trained in sequence, each fixing the previous errors | Lowers bias |
K-means steps
K must be chosen in advance, and results depend on the starting centroids.
A worked example
Minimize the loss L(w) = w squared, starting at w = 3 with learning rate 0.1. The gradient is 2w = 6, so w becomes 3 - 0.1 x 6 = 2.4.
Next step: gradient 2 x 2.4 = 4.8, so w becomes 2.4 - 0.48 = 1.92. Each step moves toward the minimum at w = 0.
With learning rate 1.1, the first step gives 3 - 1.1 x 6 = -3.6, larger in size than the start. The updates diverge, which shows a learning rate that is too large.
How the exam asks about this
- "Which algorithm, despite its name, is used for classification?" → Logistic regression.
- "Which of these is a hyperparameter: weights, learning rate, bias?" → The learning rate.
- "Which ensemble method trains models in sequence, each correcting the previous one?" → Boosting, such as GBDT.
What to remember for the exam
- Gradient descent steps against the gradient; BGD uses all samples, SGD one, mini-batch a small group.
- Parameters are learned; hyperparameters are set before training and tuned with grid or random search.
- K-fold cross-validation trains k times, each with a different validation fold.
- L1 gives sparse weights; L2 shrinks weights smoothly.
- K-means is unsupervised clustering; KNN is supervised classification.
Do not confuse k in KNN with k in k-means. In KNN it is the number of neighbors that vote; in k-means it is the number of clusters.
Common mistakes
Thinking a bigger learning rate always trains faster. Beyond a point, steps overshoot the minimum and the loss grows instead of shrinking.
Calling random forest a boosting method. Random forest uses bagging with trees trained independently. Boosting trains trees in sequence.
Thinking KNN has a long training phase. KNN simply stores the training data. The work happens at prediction time, when distances to stored samples are computed.
Check yourself
What is the difference between SGD and batch gradient descent?
SGD updates the parameters after each single sample, which is fast but noisy. Batch gradient descent uses all samples per update, which is stable but slow.
Why does L1 regularization help with feature selection?
L1 pushes some weights to exactly zero, which removes those features from the model.
In 10-fold cross-validation, how many times is the model trained?
Ten times, each time validating on a different fold and training on the other nine.
Topic 2 of 2 in Machine Learning
Test yourself on Machine Learning
The full Huawei HCIA-AI H13-311 question bank has 600 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Huawei. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.