Hello Andrew,
I use ZTree for such use, with Branch | Alt-F4 (Binary).
If I get two identical files, I delete one of them (choice established by judging best wanted location). If I get three identicals, I delete two. Etc. Sometime I don't delete if duplication is voluntary. This is done manually. Tedious but relatively fast.
It does exclude however any pictures that had been retouched, commented, resized, rotated, etc.
Another solution would be to de-deplicate files but not *organization* of files by hardlinking them.
https://schinagl.priv.at/nt/hardlinkshellext/linkshellextension.html
It means three "copies" but one physical files only. Works on NTFS.
Use however with good understanding of its implications.
Not that some archivers can "de-deduplicate" in similar manner by not archiving identical files, but only their references.
ex: switch -oi with rar.exe
Of course, archived pictures are not accessibles unless uncompressed, but saving the less used ones as archive on hard disk will easily take less space without having to spend hours dedpulicating them.
See below for ZTree script.
Another solution, if possible, would be to change hard disk of new device for a 1Tb one. If however it is something thightlty closed like a Surface however, no luck.
At last, I once used this software (now "old").
https://www.portablefreeware.com/index.php?id=225
https://web.archive.org/web/201401280221...ware.com/dupdetector/dupdetector.html
I found it most interesting by finding "almost-identical" pictures by comparing image pixel data
Might worth a try?
::A R*A*R [branch] | DEL | 5% | 100Kb | PLAIN
REM #ZTStart "Archiving"
COLOR 1F
:: a Add to Archive
:: -r Recurse subfolders
:: -MA5 Force archiving format v5.0+
:: -md128m Set dictionary size to 128mb
:: -v100m Create volumes 100mb
:: -m3 Set compression method to 3=Normal
:: -ms Store these files instead of archiving
:: default = 7z, ace, arj, bz2, cab, gz, jpeg, jpg, lha, lzh, mp3, rar, taz, tgz, xz, z, zip
:: -rrp5 Add data recovery record (5%)
:: -oi Save identical files as reference (v5.0+ only)
:: -htb Set hash type to Blake2 (256bits, ensure files are identicale) (v5.0+ only_
:: -tl Set archive time to newest file
:: -hp[pwd] Encrypt both file and headers
:: -t Test files after archiving
:: -k Lock archive (prevents any further modifications)
:: -df Delete files after archiving
#ZTHome\RAR\RAR.exe a -r -MA5 -md128m -v100m -m3 -ms -rr5p -oi -htb -tl -t -k -df %3.rar *.*
