[Jason Dunn]

Digital Home Thoughts readers, I need your help. I'm looking for a tool that will, in batch mode, analyze a bunch of images (JPEG and RAW) and allow me to quickly and easily purge the duplicates. Through some incredibly bad image management and workflow decisions on my part, I've ended up with 17,092 images from my Hawaii vacation back in 2006...it's a long story. Probably 75% of those are duplicates, and I want a fast way to purge them all. I was hoping that Lightroom 2's import (which has a "Detect Duplicates" function) would help ease the pain, but it imported 17.078 images - it turns out it only detects duplicates based on file name, not on file attribute or EXIF data. I tried ACDSee Pro's duplicate detection, but it forces you to go through each duplicate set one by one, which would take me hours. I tried another tool called Easy Duplicate Finder, but it also required manual clicking on each duplicate file. I need a tool that will analyze the images, decide which is a duplicate, then let me delete all the duplicates at once. Freeware would be preferable, but I'm so desperate to finish these photos I'll pay for a good tool. Suggestions?

[Jason Dunn]

Digital Home Thoughts readers, I need your help. I'm looking for a tool that will, in batch mode, analyze a bunch of images (JPEG and RAW) and allow me to quickly and easily purge the duplicates. Through some incredibly bad image management and workflow decisions on my part, I've ended up with 17,092 images from my Hawaii vacation back in 2006...it's a long story. Probably 75% of those are duplicates, and I want a fast way to purge them all. I was hoping that Lightroom 2's import (which has a "Detect Duplicates" function) would help ease the pain, but it imported 17.078 images - it turns out it only detects duplicates based on file name, not on file attribute or EXIF data. I tried ACDSee Pro's duplicate detection, but it forces you to go through each duplicate set one by one, which would take me hours. I tried another tool called Easy Duplicate Finder, but it also required manual clicking on each duplicate file. I need a tool that will analyze the images, decide which is a duplicate, then let me delete all the duplicates at once. Freeware would be preferable, but I'm so desperate to finish these photos I'll pay for a good tool. Suggestions?

[paschott]

Haven't tried it myself, but Picasa is supposed to detect duplicates. That falls under the free category. I've run another dupe finder that works well and ignores filename, but I'll have to dig it up. I remember that it's rather intense because it actually can look at file contents. Of course, that means any resized or tweaked pictures wouldn't be caught so that may not work for you. I'd give Picasa 3 a try to see if it will work.

-Pete

[Hooch Tan]

Duplifinder supposedly does what you're asking for, but I'll admit that I've not used it myself. It does it be visual similarity, but not EXIF data. So it might not be an exact match for what you want.

http://www.codeplex.com/DupliFinder

Hope it helps though! Oh, and it's freeware.

[fulltilt]

No Clone 2007 Home Edition.
www.noclone.net

Seemed reasonable. Trying to track down another freeware one that I ended up using.

[uzziah0]

I have tried one with many features (like size, name, crc) that works good.
I think I heard about it from Gear Diary (but I'm not sure),
It is called DoubleKiller, and here is a like to their site:
http://www.bigbangenterprises.de/en/doublekiller

[Felix Torres]

I've been using Ashisoft's Duplicate finder for years; it's not photo-specific but it does have a byte-by-byte mode in addition to filename, CRC, etc...
Duplicate File Finder - Find and remove duplicate files, byte by byte/crc32

The old version came with a very reputable recommendation:
Digital Home Thoughts: Duplicate Finder 2.0

[farrunner]

For many years I have used D'peg duplicate image finder for scanning through all my pictures looking for duplicates.

See url here: http://www.somewareonthe.net/gotdupes/

[Gordo]

Jason,

I hope you will post about the software that worked the best in finding the duplicates. I assume that the following would be the ideal requirements:
Pick a directory to use as a source then search all hard drives, or selected folder locations for duplicates based on the following:

• Same file name exactly
• Same file name regardless of file extension
• same EXIF data
• same or similar image

[Jason Dunn]

Originally Posted by uzziah0

It is called DoubleKiller, and here is a like to their site:
http://www.bigbangenterprises.de/en/doublekiller

Thanks - this is the one I ended up using. It deleted about 9000 duplicate files. I found some decent tools, but it's shocking how hard many of them are to use. And some that are easy to use and powerful have completely crippled trial versions - so crippled they won't even delete one file.

Now on to processing these in Lightroom.

[John Lane]

I use Glary Utilities. It works well, but then I read your full requirements and it doesn't meet them. It is a file comparison utility.