Boston's Digital Archives Are Drowning in Duplicate Images — and the Numbers Tell a Costly Story

Boston's public institutions are sitting on millions of duplicate digital images, a problem that has quietly inflated storage costs and degraded database performance across city agencies, university libraries, and cultural archives. The scale of the redundancy is only now becoming clear as municipal technology offices begin systematic audits ahead of a July 2027 deadline tied to the city's broader digital infrastructure overhaul.

The timing matters. Mayor Michelle Wu's administration has made digital equity and transparent government data a centerpiece of its second-term agenda. But city technology staff have found that duplicate image files — the same photograph or scanned document stored multiple times across different platforms — account for a disproportionate share of operational storage consumption in departments ranging from the Boston Inspectional Services Department to the Office of Arts and Culture. Before any meaningful open-data expansion can happen, the redundancy problem has to be addressed.

What the Numbers Actually Show

The problem is not trivial. Industry benchmarks from digital asset management research consistently place duplicate file rates in large institutional repositories at between 20 and 35 percent of total stored content. For a city archive the size of Boston's — which spans physical locations including the Copley Square branch of the Boston Public Library and the City Archives facility on Boylston Street in West Roxbury — that range translates into tens of terabytes of redundant data.

Cloud storage pricing for institutional-grade services currently runs roughly $23 per terabyte per month at mid-tier rates, meaning a conservative estimate of 50 terabytes in duplicate image files alone could represent more than $13,000 in annual unnecessary expenditure — just for storage, before factoring in backup, retrieval, and staff time spent managing redundant records. Multiply that across the dozens of separate databases maintained by Boston's 14 major city departments, and the aggregate waste becomes a genuine budget line item rather than a rounding error.

The Boston Public Library's Digital Commonwealth program, which aggregates digitized collections from institutions across Massachusetts and is headquartered at the BPL's central branch on Dartmouth Street, has been piloting automated deduplication tools since early 2025. The program hosts more than 1.9 million digital objects drawn from partner collections statewide, and administrators have publicly acknowledged that cross-institutional uploads regularly produce duplicate entries when the same photograph or document exists in multiple contributing archives.

Local Institutions Moving to Address the Backlog

Northeastern University's library system, which maintains a substantial digital archive of Boston neighborhood history including collections documenting Roxbury and Dorchester going back to the late 19th century, has integrated hash-based deduplication into its asset management workflow as of January 2026. The method assigns a unique fingerprint to each image file; when two files share an identical fingerprint, the system flags one for review rather than storing both. Northeastern's library technology team declined to provide specific figures for publication, but the approach is increasingly standard among research university libraries.

The MBTA, separately, has faced its own version of this problem in its infrastructure inspection database. Transit systems nationally have grappled with field inspectors uploading multiple near-identical photographs of the same track segment or station fixture — a workflow issue as much as a technical one. The MBTA's ongoing technology modernization program, which received federal infrastructure funding, includes provisions for image database cleanup, though the authority has not publicly detailed the scope of its deduplication work.

For Boston residents and city staff, the practical stakes are concrete. Redundant images slow retrieval times in public-facing portals, create version-control confusion when records are updated, and make genuine open-data initiatives harder to execute cleanly. The city's Open Data portal, accessible through Analyze Boston, currently lists more than 300 active datasets — but image-heavy records remain among the least consistently managed.

City technology offices and library administrators should expect the July 2027 compliance deadline to sharpen internal timelines considerably over the next 12 months. Institutions that have not yet run a baseline deduplication audit would be well advised to start now, before the combination of expanded digitization projects and budget scrutiny makes the backlog significantly harder to clear.