I Need a Good Duplicate Image Finder [Archive]

View Full Version : I Need a Good Duplicate Image Finder

Jason Dunn

10-19-2008, 11:00 AM

<p>Digital Home Thoughts readers, I need your help. I'm looking for a tool that will, in batch mode, analyze a bunch of images (JPEG and RAW) and allow me to quickly and easily purge the duplicates. Through some incredibly bad image management and workflow decisions on my part, I've ended up with 17,092 images from my <a href="http://forums.thoughtsmedia.com/f303/holiday-photo-workflow-further-explorations-raw-photo-shooting-30605.html" target="_blank">Hawaii vacation back in 2006</a>...it's a long story. Probably 75% of those are duplicates, and I want a fast way to purge them all. I was hoping that Lightroom 2's import (which has a "Detect Duplicates" function) would help ease the pain, but it imported 17.078 images - it turns out it only detects duplicates based on file name, not on file attribute or EXIF data. I tried ACDSee Pro's duplicate detection, but it forces you to go through each duplicate set one by one, which would take me hours. I tried another tool called <a href="http://www.easyduplicatefinder.com/" target="_blank">Easy Duplicate Finder</a>, but it also required manual clicking on each duplicate file. I need a tool that will analyze the images, decide which is a duplicate, then let me delete all the duplicates at once. Freeware would be preferable, but I'm so desperate to finish these photos I'll pay for a good tool. Suggestions?</p>

paschott

10-19-2008, 01:05 PM

Haven't tried it myself, but Picasa is supposed to detect duplicates. That falls under the free category. I've run another dupe finder that works well and ignores filename, but I'll have to dig it up. I remember that it's rather intense because it actually can look at file contents. Of course, that means any resized or tweaked pictures wouldn't be caught so that may not work for you. I'd give Picasa 3 a try to see if it will work.

-Pete

Hooch Tan

10-19-2008, 01:11 PM

Duplifinder supposedly does what you're asking for, but I'll admit that I've not used it myself. It does it be visual similarity, but not EXIF data. So it might not be an exact match for what you want.

http://www.codeplex.com/DupliFinder

Hope it helps though! Oh, and it's freeware.

fulltilt

10-19-2008, 01:47 PM

No Clone 2007 Home Edition.
www.noclone.net

Seemed reasonable. Trying to track down another freeware one that I ended up using.

uzziah0

10-19-2008, 02:01 PM

I have tried one with many features (like size, name, crc) that works good.
I think I heard about it from Gear Diary (but I'm not sure),
It is called DoubleKiller, and here is a like to their site:
http://www.bigbangenterprises.de/en/doublekiller

Felix Torres

10-19-2008, 02:19 PM

I've been using Ashisoft's Duplicate finder for years; it's not photo-specific but it does have a byte-by-byte mode in addition to filename, CRC, etc...
Duplicate File Finder - Find and remove duplicate files, byte by byte/crc32 (http://www.ashisoft.com/)

The old version came with a very reputable recommendation:
Digital Home Thoughts: Duplicate Finder 2.0 (http://www.digitalhomethoughts.com/news/show/27533/duplicate-finder-2-0.html)

farrunner

10-19-2008, 09:27 PM

For many years I have used D'peg duplicate image finder for scanning through all my pictures looking for duplicates.

See url here: http://www.somewareonthe.net/gotdupes/

Gordo

10-20-2008, 02:23 AM

Jason,

I hope you will post about the software that worked the best in finding the duplicates. I assume that the following would be the ideal requirements:
Pick a directory to use as a source then search all hard drives, or selected folder locations for duplicates based on the following:

Same file name exactly
Same file name regardless of file extension
same EXIF data
same or similar image

Jason Dunn

10-20-2008, 04:36 AM

It is called DoubleKiller, and here is a like to their site:
http://www.bigbangenterprises.de/en/doublekiller

Thanks - this is the one I ended up using. It deleted about 9000 duplicate files. I found some decent tools, but it's shocking how hard many of them are to use. And some that are easy to use and powerful have completely crippled trial versions - so crippled they won't even delete one file. :rolleyes:

Now on to processing these in Lightroom. :eek:

John Lane

10-20-2008, 05:23 AM

I use Glary Utilities. It works well, but then I read your full requirements and it doesn't meet them. It is a file comparison utility.

Haplo

10-21-2008, 03:16 AM

Specifically for images I've always liked Dup Detector.
http://www.keronsoft.com/dupdetector.html

Cheers

AllanIsKing

11-26-2008, 05:35 PM

I use Directory Report to find duplicate files
http://www.file-utilities.com

It can find duplicates based on the same:
Name, size, CRC and/or comparing byte-by-byte

It cannot process EXIF data

On the duplicate files windows, you can select multiple duplicate files and delete them all at once

The website has a video to show you how to run the program

allancass

12-10-2008, 08:06 PM

Check Visual Similarity Duplicate Image Finder is the fastest and the most precise of all the similar tool. Also the only one that supports RAW formats:
http://www.mindgems.com/products/VS-Duplicate-Image-Finder/VSDIF-RAW-Formats.htm

Here is a link:
http://www.mindgems.com/products/VS-Duplicate-Image-Finder/VSDIF-About.htm

bolide

01-01-2009, 06:48 PM

Image Comparer (http://www.bolidesoft.com/imagecomparer.html) supports RAW image file comparison as well. And it is able to locate similar files too, not just a full duplicates.

Jason Dunn

03-28-2009, 01:18 AM

Image Comparer (http://www.bolidesoft.com/imagecomparer.html) supports RAW image file comparison as well. And it is able to locate similar files too, not just a full duplicates.

I ended up purchasing this one - I guess you're the developer? It's definitely the most nicely designed app out of all of them, and I think it will work. The freeware ones I looked at were just awful. I wish, however, that your software didn't cost me $45 - this is probably the only time I'm going to use it, so it's a high price to pay for a single use. :(

thinker

10-28-2009, 07:09 PM

Try Duplicate Checker - http://www.duplicatechecker.com
It has image preview and thumbnails bar which allows to see all duplicate images in group at once.

antony09

12-02-2009, 03:10 PM

I know good duplicate images finder. It is Clone Remover. <link rel="File-List" href="file:///C:%5CDOCUME%7E1%5Cadmin%5CLOCALS%7E1%5CTemp%5Cmsohtml1%5C01%5Cclip_filelist.xml"><style>  </style>It has the following ways to search for duplicate files: search by content, search by properties, files with a zero size.

http://www.moleskinsoft.com (http://www.moleskinsoft.com/)