An Introduction to Logistic Regression in Machine Learning

TL;DR: Logistic regression is a fundamental machine learning algorithm for binary classification. It predicts probabilities using the sigmoid function, mapping the outputs to the range [0, 1].

Machine learning has revolutionized business and is helping us build sophisticated applications to solve tough problems. With supervised and unsupervised machine learning models, you can solve problems using classification, regression, and clustering algorithms.

In this article, we’ll discuss logistic regression in machine learning, a supervised machine learning algorithm, in detail.

What Is Logistic Regression?

Logistic regression in machine learning is a supervised learning algorithm used for classification, not regression, as the name suggests. It does not predict a number but rather the likelihood that it will be classified as a certain class when used as input.

It's sort of like how logistic regression doesn't give you a credit score when you input a person's age, income, and credit history. It spits out a probability, for instance, 0.82, for the person's loan to be accepted. This is then translated into a binary decision by a threshold (usually 0.5): approved or not approved.

This is different from linear regression. Logistic regression models an S-shaped relationship between inputs and probabilities of 0 and 1.

Key Concepts Behind Logistic Regression

Three basic concepts are the basis of logistic regression:

1. The Sigmoid Function

The sigmoid function maps any number to a value between 0 and 1. For a large positive input, the output approaches 1. However, when you input a large negative number, the output should approach 0. For this reason, it is useful in logistic regression, which is used to transform the model's output into a probability.

2. Log-Odds (Logit)

There are limits to the numerical value of a probability: it must be between 0 and 1, and logistic regression does not plot a straight line for probabilities. This transformation is called the logit transform, which helps the model mathematically link the input features to the final probability.

3. Maximum Likelihood Estimation

Logistic regression seeks the set of coefficient values that maximizes the likelihood of reproducing the training data. In other words, it continually fine-tunes the coefficients so that the predictions match the results as closely as possible.

Learn In-Demand AI Engineering SkillsWith the Trending Microsoft AI ProgramExplore Program

Why Is Logistic Regression Used in Machine Learning?

The reason why logistic regression became one of the best-known classification techniques isn't that it's the flashiest. It's that logistic regression is interpretable, relatively simple, and successful in countless real problems. Practitioners turn to it for these reasons:

• Probabilistic output
• Interpretable coefficients
• Low computational cost
• Feature importance

How Logistic Regression Works

Similar to linear regression, logistic regression begins by analyzing the input data and then computes a weighted score from the features. This score then passes through a sigmoid function, which maps it to a probability between 0 and 1.

The output probability indicates the likelihood that the data belong to a specific class. For instance, a value of 0.87 during training indicates an 87% probability that this data point will be classified as positive.

In training, logistic regression adjusts the coefficients to improve accuracy on labeled data. Its simplicity and effectiveness make it a popular choice for binary classification problems like spam detection, customer churn prediction, and medical diagnosis.

Types of Logistic Regression

Logistic regression can be categorized into different types depending on the nature of the dependent variable. The main types include Binary, Multinomial, and Ordinal Logistic Regression.

Type 1: Binary Logistic Regression

Binary logistic regression is the most common type, where the dependent variable has only two possible outcomes, represented as 0 and 1. It is used when the target variable is binary, such as yes/no, pass/fail, or true/false. The logistic function models the probability that an observation belongs to one of the two classes.

Type 2: Multinomial Logistic Regression

Multinomial logistic regression, also known as softmax regression, is used when the dependent variable has more than two unordered categories. Unlike binary logistic regression, it can handle multiple classes simultaneously. It models the probability that an observation belongs to each class using the softmax function, which ensures the predicted probabilities sum to 1 across all classes.

Type 3: Ordinal Logistic Regression

Ordinal logistic regression is employed when the dependent variable has more than two ordered categories, meaning the outcome has a natural ordering or hierarchy.

Examples include scales such as low, medium, and high, or Likert-scale responses ranging from strongly disagree to agree strongly. It models the cumulative probability that an observation falls into or below each category using the cumulative logistic distribution function.

ML Engineers work with tools like Python, TensorFlow, Docker, and AWS SageMaker to build and deploy models at scale. See the complete breakdown of skills and tools for every career level in this ML Engineer roadmap.

Logistic Regression in Python With Code

Python's scikit-learn makes implementing logistic regression in machine learning straightforward. Here are two examples.

Binary Classification: Breast Cancer Dataset

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=23)
model = LogisticRegression(max_iter=10000, random_state=0)
model.fit(X_train, y_train)
print(f"Accuracy: {accuracy_score(y_test, model.predict(X_test)) * 100:.2f}%")

Output Accuracy: 96.49%

It predicts the nature of a sample (malignant or benign) using 30 numeric properties. Due to the linearity of the separation of the two classes here, the accuracy is very high.

Multinomial Classification: Digits Dataset

from sklearn import datasets, linear_model, metrics
from sklearn.model_selection import train_test_split
digits = datasets.load_digits()
X_train, X_test, y_train, y_test = train_test_split(digits.data, digits.target, test_size=0.4, random_state=1)
model = linear_model.LogisticRegression(max_iter=10000, random_state=0)
model.fit(X_train, y_train)
print(f"Accuracy: {metrics.accuracy_score(y_test, model.predict(X_test)) * 100:.2f}%")

Output Accuracy: 96.66%

This model recognizes handwritten digits (0-9), which is a 10-class problem. Scikit-learn handles this automatically behind the scenes using a one-vs-rest approach.

Evaluating a Logistic Regression Model

Accuracy alone doesn't tell the full story, especially on imbalanced datasets. A proper evaluation of logistic regression in machine learning uses multiple metrics together.

Advantages and Limitations

Here are the advantages and limitations of logical regression in machine learning.

Level Up Your AI and Machine Learning CareerWith Professional Certificate in AI and MLLearn More Now

Real-World Applications of Logistic Regression

Logistic regression powers a wide range of real-world tasks across industries. Here are some of the most common.

1. Optical Character Recognition (OCR)

OCR converts handwritten or printed characters into digital text that computers can read. Because it involves identifying specific characters from a set of possible outcomes, it's a classification task. Logistic regression helps by classifying whether a character is present or absent in an image.

2. Fraud Detection

Fraud detection identifies and prevents deceptive activities in finance, insurance, and e-commerce. Logistic regression flags fraudulent transactions by classifying them as legitimate or fraudulent, using variables such as transaction value, location, time, and user information to predict the likelihood of fraud.

3. Disease Spread Prediction

Predicting disease spread can be framed as a classification problem: determining whether an individual is likely to contract a disease. Logistic regression models the relationship between population demographics, health conditions, environmental factors, medical resource availability, and the probability of transmission.

4. Illness Mortality Prediction

In healthcare, logistic regression is used to predict mortality among patients with specific illnesses. Trained on patient demographics, health status, and clinical indicators such as age, gender, and vital signs, the model estimates the probability that a patient will die from an illness.

5. Churn Prediction

Churn prediction identifies customers likely to stop using a product or service. By analyzing demographic information, usage patterns, and behavior, logistic regression assigns each customer a probability of churning.

Looking to build hands-on skills in ML? Simplilearn's Professional Certificate in AI and Machine Learning covers everything from foundational concepts to advanced applications, including deep learning, NLP, computer vision, and generative AI. Check it out today!

Key Takeaways

• Logistic regression can solve classification problems when the target variable is binary, multinomial, or ordinal, depending on its structure.
• Because it outputs probabilities rather than raw labels, logistic regression helps teams make clearer, threshold-based business decisions.
• Logistic regression is useful in real-world applications such as fraud detection, churn prediction, OCR, and healthcare risk modeling.

FAQs

1. Why is logistic regression used for classification?

Logistic regression is used for classification because it predicts the probability of outcomes falling into distinct categories. It’s ideal for determining whether an event will occur, such as predicting whether a customer will purchase a product based on their characteristics.

2. How does the sigmoid function work?

The sigmoid function maps any input to the value range between 0 and 1.

• The z values that are large and positive are plotted around 1
• The z values that are large and negative are plotted around 0

This S-curve enables logistic regression to yield valid probabilities.

3. What are the assumptions of logistic regression?

Logistic regression assumes several conditions for the results to be accurate. Observations should be independent, and the target variable should be categorical.

4. What is the difference between logistic and linear regression?

Linear regression is used to fit a straight line to data and estimate parameters using least squares. Logistic regression is a regression model used to predict a categorical outcome using a sigmoid function and estimates model parameters via maximum likelihood. The output, loss function, and interpretation differ.

5. Is logistic regression supervised learning?

Yes. Logistic regression is a supervised learning method that extracts classification patterns from given input-output pairs. Like the decision tree and support vector machine algorithms, it belongs to the classification branch of supervised learning.