Duplicate file report
First stage of removing duplicate files
Finds files that look like copies of each other and shows you them in groups. Nothing is removed. The workbook has a Recycle? column, and a CSV of the same rows is saved beside it: put Yes beside the copies you agree to lose in either one, and Remove duplicate files acts on your marks and nothing else.
What it tells you
- Which files are duplicates of which, grouped into sets rather than listed flat.
- How much space each set is costing you, so the effort can be aimed at the sets worth clearing.
- Where every copy lives. If you turn on pre-marking (off by default), every copy but one in each group is already marked Yes for you to check.
- What was not compared, and why, rather than an unexplained gap.
How it runs
- Sign in and pick a scope. Clicking the tile asks which account to use and how to connect, signs you in if you are not already, and drops you straight on the site list. No feature page asks you to connect a second time. Sites, libraries or folders.
- Choose what counts as a duplicate. Same name and same size is the default - free, and usually right. Same name alone is quicker and says nothing about the contents. Same contents compares the content hash SharePoint already stores for each file, so usually no file is downloaded; on a sign-in that cannot read that hash, each file is downloaded and hashed here. It can also be told to ignore the name, so a renamed copy is caught.
- Save the workbook. A CSV of the same rows is written beside it. Mark up whichever you prefer.
- Approve, then remove separately. Put Yes in the Recycle? column beside the copies you accept losing, then run Remove duplicate files from Optimise Storage Tools and hand it the workbook or the CSV. Removal is a deliberate second act, on your terms.
How it works
Only document libraries are read. With subsites included, a whole site expands into its subsites, each read once; a library or folder you tick is read as it is. When a whole site is swept, page libraries, hidden libraries and the standard system libraries are left out by default. Libraries are read one at a time, every file in each, and the size used is the size of the current version.
At name level two files match when their names are the same ignoring case, extension included. Name and size adds the exact size in bytes. Contents compares a fingerprint of the file's bytes, plus the size, and the name unless you tell it to ignore names. Copy suffixes such as (2), - Copy and a leading Copy of can be ignored too. Only files whose exact size occurs more than once are fingerprinted. Where SharePoint already holds a fingerprint for every file in a size group, that is used; otherwise every file in the group is downloaded and fingerprinted in memory, so the two kinds are never compared with each other, and nothing is written to disk. On a browser-session sign-in every candidate is downloaded. Files over 2 GB are not fingerprinted.
Each set lists its copies oldest modified first. What a set costs is the total size of its copies minus the largest one, and sets are numbered after sorting by that cost, largest first. No copy is marked for removal unless you turn on pre-marking, which keeps the oldest created, the newest modified or the shallowest copy and marks every other copy Yes.
A library whose read fails or stops short contributes nothing, so a set is never shown smaller than it really is. It is listed under Not compared with the reason, alongside files with no readable size, empty files, files under your minimum size and files that could not be fingerprinted, and the run carries on. A CSV with the same columns is written beside the workbook. The removal run matches rows back by list and item, checks the unique ID and size, and leaves alone any file that has changed since the scan. The workbook goes into History when the run ends.
What you get
An Excel workbook, named like contoso-duplicate-files-20260901-142207.xlsx, kept in Report History whether or not you save it. The Group column is the grouping. Every row in one group matched the others at the level you chose, in a different place.
Summary sheet
Scope, counts, and what the groups add up to.
Duplicates sheet
Every copy found, grouped.
| Column | Example |
|---|---|
| Recycle? | |
| Group | 17 |
| Copies | 3 |
| Site URL | https://contoso.sharepoint.com/sites/projects |
| Library | Shared Documents |
| Folder path | Clients/Northwind |
| File name | Proposal v4.pdf |
| Size (bytes) | 3,118,442 |
| Matched on | Name and size |
Not compared sheet
Only when something was left out. Libraries and files left out of the comparison, each with its reason.
| Column | Example |
|---|---|
| Site URL | https://contoso.sharepoint.com/sites/hr |
| Library | Policies |
| File | /sites/hr/Policies/Handbook.docx |
| Reason | The file is smaller than the minimum size set for this run. |
What you need
- Read access to the libraries in scope for the report.
- The removal run, later and separately, needs write access. This report does not.
What it will not do
- This report removes nothing, and it cannot. Removal is a separate tile that only ever acts on the rows you marked.
- Matching on name and size cannot see inside a file, so two different files that happen to share both are reported as a set. Choose the contents level when that matters.
- Comparing contents uses SharePoint's stored hash where the sign-in can read it. Where it cannot, every file in scope is downloaded to be hashed, which on a large scope takes real time and bandwidth.
- Removal goes to the recycle bin by default, so an approval you regret is recoverable. Removal can also delete permanently, after you type DELETE to confirm - that frees the space at once and cannot be undone.