How Boston's City Archives Ended Up With Thousands of Duplicate Digital Images — and What's Finally Being Done About It

Boston's city government is sitting on an estimated 40,000 duplicate digital image files scattered across at least six municipal departments, a problem that archivists and records managers say grew quietly over roughly ten years of disconnected digitization efforts and has now become expensive enough to demand a fix.

The duplication crisis matters right now because Mayor Michelle Wu's administration has committed to expanding public access to city records under its Open and Accountable Government initiative, and duplicate images — particularly of property records, zoning maps, and building permits — are creating retrieval failures and inflating cloud storage costs that come directly out of already stretched departmental budgets.

How the Backlog Built Up

The roots of the problem stretch back to roughly 2014 and 2015, when the City of Boston began accelerating its push to get paper records into digital form. The Boston City Archives, based at 201 Rivermoor Street in West Roxbury, ran its own scanning program. Simultaneously, the Boston Inspectional Services Department, which operates out of 1010 Massachusetts Avenue, launched a separate effort to digitize building permits and code enforcement files. Neither operation used a shared file-naming protocol, meaning the same document — say, a 1987 variance application for a triple-decker in Dorchester — could be scanned twice, saved under two different filenames, and stored in two separate systems with no automated flag to catch the redundancy.

Between 2017 and 2022, at least three additional city departments, including the Office of Environment, Energy and Open Space and the Boston Planning and Development Agency, conducted their own scanning drives using vendor contracts that were procured independently. Each vendor delivered files in slightly different formats and resolutions. The BPDA, which maintains planning documents for neighborhoods from Roxbury to East Boston, used a TIFF-based workflow. ISD favored lower-resolution JPEGs to save space. The result was a patchwork that made cross-departmental deduplication technically complicated even for staff who understood the problem existed.

The Wu administration's digital services team flagged the scope of the issue in a February 2026 internal review, though the findings have not been made public in full. City records staff familiar with the review have described it in general terms at public meetings of the Boston Digital Equity Advisory Committee, where the storage cost question has come up in discussions about redeploying technology budget dollars toward resident-facing services.

The Cost and the Path Forward

Cloud storage is not free. Municipal IT departments typically pay between $0.02 and $0.05 per gigabyte per month for enterprise-tier cloud storage under state contract frameworks, and duplicated high-resolution TIFF files — which can run to 50 megabytes each — add up fast across tens of thousands of redundant records. The inefficiency also slows public records request fulfillment, a politically sensitive point for an administration that has made transparency a signature issue.

The city's Digital Services team has been piloting a deduplication workflow since April 2026, using hash-matching software to identify identical files regardless of what they've been named. The pilot started with property records from Jamaica Plain and Dorchester — two neighborhoods with heavy housing development activity and therefore high document volume — precisely because those are the files most likely to be requested by developers, attorneys, and residents navigating the permitting process.

The practical implication for anyone who relies on Boston's public records portal, PermitBOS, or the BPDA's online document library is that retrieval accuracy should improve over the next several months as deduplicated, consistently named files replace the current jumble. Residents with pending public records requests involving building or zoning documents may want to resubmit if earlier searches came back incomplete. The city's records request portal is accessible through boston.gov, and the Boston City Archives can be reached directly at its West Roxbury office for historical document inquiries predating the digital era entirely.