How To Find Duplicate Files (Even if they have different names!). And some History.

One of the great things about Unix/Linux is the "tools" philosophy: rather than building large applications to do everything, Unix/Linux is built on the idea of creating small tools for specific tasks that can work together to accomplish more complicated tasks.  There's an info page buried in Linux with some great history on this philosophy.  This approach to application development still holds today.

"The Unix developers at Bell Labs were all professional programmers and trained computer scientists. They had found that while a one-size-fits-all program might appeal to a user because there's only one program to use, in practice such programs are
a. difficult to write,
b. difficult to maintain and debug, and
c. difficult to extend to meet new situations.
 Instead, they felt that programs should be specialized tools. In short, each program "should do one thing well." No more and no less. Such programs are simpler to design, write, and get right--they only do one thing."

Let's look at an example.  I'm sure people pay good money for GUI applications that do this, but it's all built into Linux for free.

Here is a way to generate a list of duplicate files with just three commands: find, md5sum, and uniq -- even if the files are in completely different locations with completely different names.
This is a perfect way to find those pesky duplicate files (i.e. mp3s) strewn about your hard drive, all with different names. ... Who knew bunzip2 and bzcat where the same thing?

jknight@shadowbox:/bin$ find /bin -type f -exec md5sum "{}" \; | uniq -D -w 32
bdf6bd945da1c84c86916ac3631143f3  /bin/bunzip2
bdf6bd945da1c84c86916ac3631143f3  /bin/bzcat
dfaba3a92070a1881dd8ec64a26069a4  /bin/bzcmp
dfaba3a92070a1881dd8ec64a26069a4  /bin/bzdiff
2b11565d85da178b3a1942a22d20c624  /bin/bzfgrep
2b11565d85da178b3a1942a22d20c624  /bin/bzgrep
e243255b6cf3b9403df53cb9cd6176e1  /bin/bzless
e243255b6cf3b9403df53cb9cd6176e1  /bin/bzmore
ce9dfb81f78f7d042b25565c0115b8e4  /bin/zcmp
ce9dfb81f78f7d042b25565c0115b8e4  /bin/zdiff
05c8810af8c45fc25ce1117f141d9203  /bin/zegrep
05c8810af8c45fc25ce1117f141d9203  /bin/zfgrep
jknight@shadowbox:/bin$