Linux offers really good text processing and editing tools. One of these tools is the uniq command. The uniq command helps you detect and delete adjacent occurrences of the same line. That means it deals with repetitions of sentences in a piece of text.
Using the uniq command in Linux
In this tutorial we will see how the uniq command works. Let’s get started.
1. Create a sample text file
We will create a sample text file with a few repeated lines.
The text for the file is given below :
The weather is good today.
The weather is good today.
The Weather Is Good Today.
Today is a sunny day.
Today is a sunny day.
Today is a sunny day.
Here the first two lines are the same. The third line is different and the remaining lines are all alike.
To create a file with this text use the cat command.
cat > file.txt
Output :
2. Using uniq to delete repeated lines from the text
To delete repeated lines from the text, use :
uniq file.txt
Output :
The weather is good today.
The Weather Is Good Today.
Today is a sunny day.
There are no repetitions in the text anymore. As you can see, the output displays line 1 and line 2 as unique lines even though the content is the same. That’s because Linux is case sensitive.
3. Get a count of the number of repetitions
To count the number of repetitions use the following line of code :
uniq -c file.txt
Output :
2 The weather is good today.
1 The Weather Is Good Today.
3 Today is a sunny day.
The output contains lines from the text with the count at the beginning.
4. Only print the repeated lines
The uniq command gives you an option to only print the lines that occur more than once. To print only the repeated lines use :
uniq -D file.txt
Output :
The weather is good today.
The weather is good today.
Today is a sunny day.
Today is a sunny day.
Today is a sunny day.
5. Only print the non-repeated lines
This is the opposite of the example above. When you use the -u flag along with the uniq command then only the lines that occur once are printed. To print only the non-repeated lines use :
uniq -u file.txt
Output :
The Weather Is Good Today.
6. How to delete repeated lines that don’t occur together?
If you want to delete multiple occurrences of a line that don’t occur together then you can sort the text first.
For example, consider the text given below :
The weather is good today
Today is a sunny day
The weather is good today
The Weather Is Good Today
Today is a sunny day
Today is a sunny day
Let’s see what happens when we run the uniq command on this text.
uniq file.txt
Output :
The weather is good today
Today is a sunny day
The weather is good today
The Weather Is Good Today
Today is a sunny day
There is no change in the text. Let’s use sorting so that same lines occur together.
We can sort the file and store the output in another file with the use of the sort command:
sort file.txt > file1.txt
After sorting the text looks like this :
The Weather Is Good Today
The weather is good today
The weather is good today
Today is a sunny day
Today is a sunny day
Today is a sunny day
Now we can use the uniq command to delete the repeated lines.
uniq file1.txt
Output :
The Weather Is Good Today
The weather is good today
Today is a sunny day
We can also count the number of occurrences for each line.
uniq -c file1.txt
Output :
1 The Weather Is Good Today
2 The weather is good today
3 Today is a sunny day
7. How to store the output to a file?
When you run the uniq command on a file, the contents of the original file are not modified. To save the output of the uniq command you can redirect it to a file. You can do that using :
uniq file.txt > [filename]
Conclusion
This tutorial was about uniq command in Linux. We learned how to use this command for deleting repeated occurrences of a line. Hope you had fun learning with us! You can learn more about the uniq command using the man command.
