Let's use an example where file1.txt contains the following text.

Hello World
How are you today?
Hello World
I am good today

The uniq command will fail to display the lines that are unique if the file is not sorted. In this example, Hello World appears on lines 1 and 3, and the uniq command fails to find the lines that are unique.

~]# uniq file1.txt
Hello World
How are you today?
Hello World
I am good today

The sort command with the -u or --unique and -o or --output commands is an excellent way to update a file to only contain one line for each string of unique text.

~]# sort --unique --output /path/to/file1.txt /path/to/file1.txt

Only unique lines are displayed. In another words, duplicate lines are only listed once.

~]# cat file1.txt
Hello World
How are you today?
I am good today

Only display lines that are unique

The -u option will only display lines that are unique. In another words, only lines that are not duplicated will be displayed.

~]# uniq -u file1.txt
How are you today?
I am good today

Only display lines that are not unique

The -d (duplicate) option will only display lines that are duplicated.

~]# uniq -d file1.txt
Hello World

Display every occurrence of lines that are not unique

The -D (dupliate) option displays every occurrence of duplicated lines.

~]# uniq -D file1.txt
Hello World
Hello World

Ignore case

Let's say you have identical lines, where one is upper case and the other is lower case.

~]# cat file1.txt
HELLO WORLD
hello world

These lines would be considered unique (not duplicate).

~]# uniq -u file1.txt
HELLO WORLD
hello world

The -i (ignore case) option can be used to consider these lines duplicates of each other.

~]# uniq -Di file1.txt
HELLO WORLD
hello world

Count the number of unique lines

The -c option can be used to count the number of times a string is repeated in a file.

~]# uniq file1.txt
     2  Hello World
     1  How are you today?
     1  I am good today