Python is an object-oriented, high-level programming language with dynamic semantics that is interpreted. Its high-level built-in data structures, combined with dynamic typing and dynamic binding, make it very appealing for Rapid Application Development, as well as use as a scripting or glue language to connect existing components.
Python's simple, easy-to-learn syntax prioritizes readability, lowering the cost of programme maintenance. Python provides support for modules and packages, which promotes programme modularity and code reuse. The Python interpreter and extensive standard library are free to use and distribute in source or binary form for all major platforms.
In this article, we'll look at what a list is in Python. Since a Python list is a collection of multiple elements, including duplicates, it is sometimes necessary to make the list unique. In this section, we will look at the various methods for removing duplicates from a list in Python. Firstly, let us know what exactly is a list?
What Is a Python list?
The list is the most important data type in the Python programming language. The list is written in Python as a list of comma-separated values enclosed by square brackets. The most significant advantage of the list is that the elements within it do not have to be of the same data type and can be indexed negatively. Furthermore, all string operations, such as slicing and concatenation, are also applicable to list data types. We can also make a nested list, which is a list that contains another list.
How to Remove Duplicates From a Python list?
In Python, there are numerous methods for removing duplicates from a list. To remove duplicates from a given list, you can make use of these methods and get your work done. Let's take a look at them:
Method 1 - Naive Method
To remove duplicates from a list in Python, iterate through the elements of the list and store the first occurrence of an element in a temporary list while ignoring any other occurrences of that element.
The basic approach is implemented in the naive method by:
- Using a For-loop to traverse the list
- If the elements do not already exist in a temporary list, they are added to it
- The temporary list has been assigned to the main list
Example:
# removing duplicates from the list using naive methods
# initializing list
sam_list = [11, 13, 15, 16, 13, 15, 16, 11]
print ("The list is: " + str(sam_list))
# remove duplicates from list
result = []
for i in sam_list:
if i not in result:
result.append(i)
# printing list after removal
print ("The list after removing duplicates : " + str(result))
Output:
The list is: [11, 13, 15, 16, 13, 15, 16, 11]
The list after removing duplicates: [11, 13, 15, 16]
Method 2 - Using List Comprehension
Instead of using the For-loop to implement the Naive method of removing duplicates from a list, we can use Python's List comprehension functionality to do so in just one line of code.
Example:
# removing duplicates from the list using list comprehension
# initializing list
sam_list = [11, 13, 15, 16, 13, 15, 16, 11]
print ("The list is: " + str(sam_list))
# to remove duplicates from list
result = []
[result.append(x) for x in sam_list if x not in result]
# printing list after removal
print ("The list after removing duplicates: " + str(result))
Output:
The list is: [11, 13, 15, 16, 13, 15, 16, 11]
The list after removing duplicates: [11, 13, 15, 16]
Method 3 - Using set()
This is the most commonly used method for removing a duplicate from a Python list. This is due to the fact that duplicates are not permitted in the set data structure. However, the order of the elements is lost when using this method.
Example:
# removing duplicates from the list using set()
# initializing list
sam_list = [11, 15, 13, 16, 13, 15, 16, 11]
print ("The list is: " + str(sam_list))
# to remove duplicates from list
sam_list = list(set(sam_list))
# printing list after removal
# ordering distorted
print ("The list after removing duplicates: " + str(sam_list))
Output:
The list is: [11, 15, 13, 16, 13, 15, 16, 11]
The list after removing duplicates: [16, 11, 13, 15]
Method 4 - Using list comprehension + enumerate ()
We find the distinct elements and store them in a temporary list using the List comprehension method. When we use the List comprehension in conjunction with the enumerate() function, the programme checks for previously encountered elements and skips adding them to the temporary list. The enumerate function takes an iterable as an argument and returns it as an enumerating object (index, element), which adds a counter to each iterable element.
Example:
# removing duplicates from the list using list comprehension + enumerate()
# initializing list
sam_list = [11, 15, 13, 16, 13, 15, 16, 11]
print ("The list is: " + str(sam_list))
# to remove duplicates from list
result = [i for n, i in enumerate(sam_list) if i not in sam_list[:n]]
# printing list after removal
print ("The list after removing duplicates: " + str(result))
Output:
The list is: [11, 13, 15, 16, 13, 15, 16, 11]
The list after removing duplicates: [11, 13, 15, 16]
Method 5 - Using collections.OrderedDict.fromkeys()
This is the quickest way to accomplish the goal of removing duplicates from the Python list. This method first removes duplicates before returning a dictionary that has been converted to a list. In addition, this method works well with strings.
Example:
# removing duplicates from list using collections.OrderedDict.fromkeys()
from collections import OrderedDict
# initializing list
sam_list = [11, 15, 13, 16, 13, 15, 16, 11]
print ("The list is: " + str(sam_list))
# to remove duplicates from list
result = list(OrderedDict.fromkeys(sam_list))
# printing list after removal
print ("The list after removing duplicates: " + str(result))
Output:
The list is: [11, 15, 13, 16, 13, 15, 16, 11]
The list after removing duplicates: [11, 15, 13, 16]
Conclusion
Sets, built-in functions, and iterative methods can be used to remove duplicates from a list. If the list's elements are non-hashable, always take an iterative approach, traversing the list and extracting unique items. The Naive Method, List comprehensions, and List.count() methods are examples of iterative approaches. If the order of the elements is not critical, we can remove duplicates using the Set method and the Numpy unique() function. We can use Pandas functions, OrderedDict, reduce() function, Set + sort() method, and iterative approaches to keep the order of elements.
To learn and make a successful career as a Python programmer, enroll in Simplilearn’s PGP Full Stack Development Course. With this course you will also master data operations, conditional statements, shell scripting, Django and so much more.
