Boston Agencies Wrestle With Duplicate Image Problem in Public Records: What Officials and Experts Are Saying

Boston's municipal record-keeping system is sitting on a growing crisis that officials and digital archivists say has been quietly compounding for years: duplicate images embedded in public documents are bloating city databases, delaying Freedom of Information requests and burning through storage budgets that could be redirected toward services. The problem, which spans everything from building-permit filings in Roxbury to zoning documents in East Boston, has accelerated as the city digitized more paper records after 2020.

The timing matters. Mayor Michelle Wu's administration has pushed hard on housing production in Jamaica Plain and Dorchester, generating thousands of new permit applications, environmental assessments and inspection reports — each of them dense with attached photographs, site plans and scanned correspondence. Every duplicate image uploaded without deduplication protocols adds to a backlog that archivists say undermines the transparency goals that drove digitization in the first place.

What the Experts Are Saying

Digital records specialists at Northeastern University's library and information science program have flagged the issue in internal consultations with city agencies, describing it as a structural deficiency rather than a staffing failure. The core problem is the absence of a mandatory hash-checking step — a standard process that identifies identical image files before they are saved — at the point of upload across Boston's permitting and inspection platforms. Without that gate, the same photograph of a cracked foundation wall or a roof violation can be filed dozens of times across different case numbers, each instance consuming storage space and creating confusion for clerks pulling records.

The Boston Public Library's digital preservation unit on Boylston Street has dealt with a version of the same problem in its own archival collections, and staff there have developed deduplication workflows using open-source tools that the city could theoretically adopt. The process involves running image libraries through perceptual hashing algorithms — software that can recognize visually identical files even when they carry different file names or timestamps. Libraries and universities have used such tools routinely for more than a decade; municipal governments have been slower to adopt them.

Advocates for government transparency have also weighed in. The New England First Amendment Coalition, which monitors public records compliance across Massachusetts, has noted in its annual assessments that prolonged delays in producing city documents often trace back to database disorganization rather than deliberate obstruction. When a records officer has to manually sort through hundreds of duplicated attachments to pull a single case file, response times stretch well beyond the state's ten-business-day statutory deadline under Massachusetts General Laws Chapter 66.

The Cost in Dollars and Days

Cloud storage costs for municipal governments are not trivial. Enterprise-grade storage contracts for a mid-size city typically run into six figures annually, and bloated databases drive those contracts upward at renewal. Boston's Department of Innovation and Technology, headquartered at City Hall on Cambridge Street, manages the underlying infrastructure, and officials there have acknowledged in public budget presentations that storage capacity planning is an ongoing challenge — though they have not publicly attributed cost overruns specifically to duplicate images.

The MBTA's own document management systems, which intersect with city planning offices on transit-oriented development projects along the Orange Line corridor, have faced similar audit findings. A 2024 state Inspector General review of transit agency record-keeping cited file redundancy as a contributing factor in delayed document production, a finding that put the issue on the radar of municipal IT departments across Greater Boston.

Getting ahead of the problem requires a short list of concrete steps, according to records management professionals: first, an audit of existing storage to quantify the scale of duplication; second, retroactive deduplication using hash-matching tools on archives dating back to at least 2018; and third, enforcement of upload protocols that flag redundant files in real time. The City of Boston could begin with the Inspectional Services Department, which handles the highest volume of image-heavy permit documentation, as a pilot before scaling citywide. For residents filing requests or developers tracking project documents, the practical advice for now is to be specific — cite case numbers, dates and addresses precisely — to help clerks navigate databases that remain, by most expert accounts, more cluttered than they should be.