Listing unique lines in a file using the UNIQ command in Linux

Home > Search > Linux commands
  by

Let's use an example where file1.txt contains the following text.

Hello World
How are you today?
Hello World
I am good today

 

The uniq command will fail to display the lines that are unique if the file is not sorted. In this example, Hello World appears on lines 1 and 3, and the uniq command fails to find the lines that are unique.

~]# uniq file1.txt
Hello World
How are you today?
Hello World
I am good today

 

The sort command with the -u (unique) and -o (overwrite) commands is an excellent way to update a file to only contain one line for each string of unique text.

~]# sort -u -o /path/to/file1.txt /path/to/file1.txt

 

Only unique lines are displayed. In another words, duplicate lines are only listed once.

~]# cat file1.txt
Hello World
How are you today?
I am good today

 


Only display lines that are unique

The -u option will only display lines that are unique. In another words, only lines that are not duplicated will be displayed.

~]# uniq -u file1.txt
How are you today?
I am good today

 

 


Only display lines that are not unique

The -d (duplicate) option will only display lines that are duplicated.

~]# uniq -d file1.txt
Hello World

 


Display every occurrence of lines that are not unique

The -D (dupliate) option displays every occurrence of duplicated lines.

~]# uniq -D file1.txt
Hello World
Hello World

 


Ignore case

Let's say you have identical lines, where one is upper case and the other is lower case.

~]# cat file1.txt
HELLO WORLD
hello world

 

These lines would be considered unique (not duplicate).

~]# uniq -u file1.txt
HELLO WORLD
hello world

 

The -i (ignore case) option can be used to consider these lines duplicates of each other.

~]# uniq -Di file1.txt
HELLO WORLD
hello world

 


Count the number of unique lines

The -c option can be used to count the number of times a string is repeated in a file.

~]# uniq file1.txt
     2  Hello World
     1  How are you today?
     1  I am good today

 



Add a Comment




We will never share your name or email with anyone. Enter your email if you would like to be notified when we respond to your comment.




Please enter in the box below so that we can be sure you are a human.




Comments