Principal Component Analysis in Machine Learning: A Step-by-Step Guide

TL;DR: Principal Component Analysis, or PCA, reduces a dataset with many variables into fewer uncorrelated components while preserving as much variation as possible. It is commonly used for visualization, noise reduction, compression, and faster machine learning.

Datasets with many variables can be difficult to visualize and process. Some features may also contain overlapping information. Principal Component Analysis simplifies such datasets by combining related variables into a smaller set of new variables called principal components.

PCA is widely used in machine learning, data analysis, image processing, finance, and healthcare. It is most useful when a dataset contains several numerical and correlated features.

What Is Principal Component Analysis?

Principal Component Analysis is an unsupervised dimensionality reduction technique. It transforms the original variables into a new set of uncorrelated variables called principal components.

The first principal component, PC1, captures the greatest possible variation in the data. The second component, PC2, captures the next greatest variation while remaining perpendicular to PC1. Each later component captures progressively less variance.

PCA is a feature extraction method, not a feature selection method. It creates new variables by combining the original features rather than selecting a subset of them.

Note: PC1 captures the greatest variation in the dataset, while PC2 captures the next greatest variation.

Why Is PCA Used?

PCA helps when a dataset contains too many variables or several strongly correlated features. Common uses include:

• Converting high-dimensional data into two or three dimensions for visualization
• Reduction of  redundant information
• Lowering storage and computational requirements
• Creating compact inputs for machine learning models
• Identifying clusters, patterns, and outliers
• Removing some low-variance noise

PCA does not always improve model performance. A low-variance feature may still be useful for prediction, so the transformed data should be evaluated for the task at hand.

Learn In-Demand AI Engineering SkillsWith the Trending Microsoft AI ProgramExplore Program

How Does PCA Work?

PCA finds new directions in the data that capture the greatest amount of variation. It then projects the original observations onto those directions, creating a smaller set of uncorrelated variables called principal components.

1. Center and Scale the Data

PCA is sensitive to feature scale. A variable measured in thousands may influence the result more than one measured between zero and one.

Centering subtracts the mean from each feature. When features use different units or ranges, they are also standardized using:

z = (x - μ) / σ

Here, x is the original value, μ is the feature mean, and σ is the standard deviation.

Centering is required for PCA. Scaling is usually recommended when the features are measured on different scales.

2. Calculate the Covariance Matrix

The covariance matrix shows how the features vary together. Its diagonal values represent the variance of individual features, while the other values represent covariance between feature pairs.

For centered data, the covariance matrix is calculated as:

C = (1 / (n - 1)) × Xcᵀ × Xc

Here, C is the covariance matrix, Xc is the centered data, Xcᵀ is its transpose, and n is the number of observations.

Positive covariance means two features tend to increase together. Negative covariance means they tend to move in opposite directions.

3. Find Eigenvectors and Eigenvalues

PCA calculates eigenvectors and eigenvalues from the covariance matrix.

Eigenvectors define the directions of the new component axes. Eigenvalues show how much variance is captured along each direction. The eigenvector with the largest eigenvalue defines PC1, while the next largest defines PC2.

4. Select the Principal Components

The components are ranked by the amount of variance they explain. A scree plot shows the explained variance for each component and can help identify where additional components contribute little extra information.

There is no fixed number of components to retain. Two may be enough for visualization, while a machine learning model may need more to preserve sufficient information.

5. Project the Data

The final step projects the processed data onto the selected component directions:

Z = Xprocessed × Wk

Here, Xprocessed is the centered or standardized dataset, Wk contains the selected eigenvectors, and Z is the reduced dataset.

If the original dataset has p features and k components are retained, PCA reduces each observation from p dimensions to k dimensions. The resulting values are called component scores.

Become an AI Engineering ExpertWith the Microsoft AI Engineer ProgramExplore Program

PCA Example in Python Using the Wine Dataset

Let’s take a Wine dataset, which contains 178 samples and 13 numerical features, including alcohol, magnesium, flavonoids, color intensity, hue, and proline. It also includes three wine classes.

In this example, PCA reduces the 13 features to two principal components. This makes it possible to visualize the samples on a two-dimensional scatter plot.

Step 1: Load and Standardize the Data

The Wine features use different measurement scales. Proline, for example, contains values in the hundreds, while some other features use small decimal values. Applying PCA directly could allow features with larger numerical ranges to dominate the result.

The features are therefore standardized before PCA. Standardization gives each feature a mean of zero and a standard deviation of one.

from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler

wine = load_wine()

X = wine.data
y = wine.target

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

The target variable, y, contains the wine classes. It is not used to calculate the principal components. The labels will only be used later to identify the classes in the final visualization.

Step 2: Examine the Explained Variance

Before selecting the number of components, PCA can be fitted using all 13 components.

from sklearn.decomposition import PCA
import numpy as np

pca_full = PCA()
pca_full.fit(X_scaled)

explained_variance = pca_full.explained_variance_ratio_
cumulative_variance = np.cumsum(explained_variance)

print(explained_variance)
print(cumulative_variance)

The first principal component captures approximately 36.2 percent of the dataset’s variance. The second captures about 19.2 percent. Together, PC1 and PC2 preserve roughly 55.4 percent of the total variance.

The scree plot helps show how much information each component adds. Retaining only two components results in some information loss, but it provides a compact representation that can be plotted in two dimensions.

A predictive model may require more components. For this example, two are selected because the goal is visualization.

Step 3: Reduce the Dataset to Two Components

PCA is now applied again with n_components=2.

pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)

print(X.shape)
print(X_pca.shape)

The original dataset has the shape:

(178, 13)

After PCA, its shape becomes:

(178, 2)

Each wine sample is now represented by two values, its score on PC1 and its score on PC2, instead of the original 13 feature values.

Step 4: Visualize the Transformed Data

The two principal components can be plotted on a scatter chart. The wine classes are added as labels only to help interpret the result.

import matplotlib.pyplot as plt

markers = ["o", "s", "^"]

for class_value, class_name in enumerate(wine.target_names):
    class_rows = y == class_value

    plt.scatter(
        X_pca[class_rows, 0],
        X_pca[class_rows, 1],
        label=class_name,
        marker=markers[class_value],
        alpha=0.8
    )

plt.xlabel(
    f"PC1 ({pca.explained_variance_ratio_[0] * 100:.1f}% variance)"
)
plt.ylabel(
    f"PC2 ({pca.explained_variance_ratio_[1] * 100:.1f}% variance)"
)
plt.title("Wine Samples Projected onto the First Two Principal Components")
plt.legend(title="Wine Class")
plt.show()

Each point represents one wine sample. Samples located close together have similar characteristics across the original features.

The plot also reveals visible separation between the three wine classes. However, this does not mean PCA has classified the samples. PCA is unsupervised and did not use the class labels while creating PC1 and PC2.

Step 5: Interpret the Principal Components

The scatter plot shows where each sample appears in the reduced space, but it does not explain what PC1 and PC2 represent. Component loadings help answer that question.

A loading measures how strongly an original feature contributes to a principal component.

import pandas as pd

loadings = pd.DataFrame(
    pca.components_.T,
    columns=["PC1", "PC2"],
    index=wine.feature_names
)

print(loadings)

Features with larger positive or negative loadings have a stronger influence on the component. In this example, flavonoids, total phenols, and the OD280/OD315 ratio contribute strongly to PC1. Alcohol and color intensity make larger contributions to PC2.

The direction of a loading also matters. Positive and negative loadings indicate that features vary in opposite directions along the component. A negative loading is not less important than a positive one.

The example shows the complete PCA process: standardize the variables, examine explained variance, select the components, transform the dataset, visualize the samples, and interpret the component loadings.

Build real-world AI and Machine Learning skills with our Microsoft AI Engineer Course. Designed to match current industry needs, it helps you learn practical concepts and apply them with confidence. Start your journey today and take a clear step toward a future-ready career.

How to Interpret PCA Results

PCA produces three main outputs:

• Scores: The transformed values of each observation on the principal components
• Loadings: The contribution of each original feature to a component
• Explained variance: The proportion of total variation captured by each component

Large positive or negative loadings show that a feature has a strong relationship with a component. Explained variance indicates how much information each component retains.

Visible separation between classes in a PCA plot can reveal useful structure, but it does not measure classification accuracy.

Applications of PCA

• Data Visualization: PCA can reduce datasets with many variables to two or three dimensions, making clusters and outliers easier to identify.
• Machine Learning Preprocessing: Models may train faster when they use fewer input variables. PCA can also reduce problems caused by multicollinearity.
• Image Compression: Images can be represented using fewer components while preserving much of their dominant visual structure.
• Finance and Healthcare: PCA can summarize correlated financial indicators, patient measurements, sensor readings, and genomic variables.

Stand Out With Applied Agentic AI ExpertiseWith Our PCP in Agentic AI & Multi-Agent SystemsExplore Program

Advantages and Limitations of PCA

These trade-offs mean PCA works best when dimensionality reduction is more important than preserving the meaning of every original feature.

ML Engineers work with tools like Python, TensorFlow, Docker, and AWS SageMaker to build and deploy models at scale. See the complete breakdown of skills and tools for every career level in this ML Engineer roadmap.

Conclusion

Principal Component Analysis is used to reduce the dimensionality of data sets that have many correlated variables. If used carefully, it can simplify complex datasets and make them easier to visualize, reduce computing loads, and show patterns that don't appear in the original feature space. 

For those looking to move from data analysis into newer AI applications, Simplilearn’s Applied Agentic AI program covers Python, RAG, MCP, agentic frameworks, multi-agent systems, and workflow automation. Through live training and practical projects, learners work with the technologies used to develop and manage agentic AI solutions.

FAQs

1. What is the difference between a principal component and an eigenvector?

An eigenvector defines the direction of a component. The principal component is the transformed data projected along that direction. 

2. What is the difference between PCA and factor analysis?

PCA reduces dimensions by preserving variance. Factor analysis identifies hidden factors that explain relationships between variables. 

3. Can PCA be used on categorical data?

Standard PCA is designed for numerical data. For mainly categorical datasets, Multiple Correspondence Analysis is usually more appropriate.