I’m de-duplicating my archives. I processed 3.6 million files and then crashed:

Symlinks: 30,836
Directories: 513,860
Files: 3,665,000
Hardlinks: 2,241,681
Duplicates: 55,309

Time to start again. Duplicates that have already been processed will show up as unduplicated files on this next iteration… my script is here.

And now we’re finished:

Symlinks: 31,926
Directories: 2,071,219
Files: 11,682,777
Hardlinks: 8,034,685
Duplicates: 669,452
Rate: 3555.44 files/second