Text files cleanup

Text files cleanup and content deduplication

Sort file lines and deduplicate content:

sort file.txt | uniq -u

Usage of AWK for deduplicate file content:

awk '!seen[$0]++' file.txt