Boston Archivists and Tech Officials Warn City Agencies Over Duplicate Image Problem in Digital Records

Boston's city agencies are sitting on a growing problem inside their digital document systems: thousands of duplicate images embedded in permit applications, inspection reports, and public records that are eating storage budgets and making official archives harder to search and audit. Digital records managers across several departments have flagged the issue in recent months, and the conversation is intensifying as the city prepares to expand its online permitting portal this fall.

The timing matters. Mayor Michelle Wu's administration has pushed hard on digital transparency, and the city's updated Open Records initiative — tied to a broader modernization push that drew federal grant interest in early 2025 — depends on clean, well-organized document repositories. Experts who work with municipal systems say duplicate imagery is rarely a crisis on its own, but left unaddressed it compounds every other digital governance problem a city has.

What Experts Are Saying About the Scope

Digital archivists at Northeastern University's Library Services, which has worked with Boston-area government bodies on document digitization, have described the duplicate image problem as endemic to agencies that scanned paper records in bulk without deduplication protocols in place. The Inspectional Services Department, which handles building permits and code enforcement across neighborhoods including Jamaica Plain and South Boston, is among the offices that digitized large backlogs during the 2020-2022 period when in-person counter services were curtailed.

Independent records management consultants familiar with mid-sized American city systems say that without automated deduplication tools, a single permit file can contain the same property photograph four or five times — once from the initial application, once from each subsequent inspection upload, and again if a clerk manually re-attached a document during a system migration. Multiply that across tens of thousands of active permits and the redundancy compounds fast. One industry benchmark used by archivists holds that unmanaged municipal imaging systems typically carry between 15 and 30 percent duplicate content by file count within five years of digitization.

At the Boston Public Library's Digital Repository on Boylston Street, staff have dealt with a version of this problem in their own historical collections. Librarians there have publicly discussed, at professional conferences, the challenge of maintaining unique image identifiers across large digitized collections — a workflow discipline that city permit and inspection systems have been slower to adopt.

City Hall's Response and What Comes Next

The city's Department of Innovation and Technology, based at City Hall on Cambridge Street, has been working with department IT liaisons on a records hygiene review since at least late 2025. The scope of that review — including which agencies are prioritized and what deduplication software might be procured — has not been publicly detailed, and no formal policy document has been released as of July 4, 2026.

The practical stakes are concrete. Cloud storage costs for municipal government have risen sharply since 2022. Procurement specialists in comparable American cities have noted storage line items growing by 20 to 40 percent annually as document volumes increase, with duplicate imagery cited as a meaningful contributor. For Boston, which has been migrating inspection and permitting data to a cloud-based infrastructure as part of a multi-year IT contract, excess storage translates directly into contract costs that fall on taxpayers.

Community groups in Dorchester and Roxbury that regularly file public records requests — particularly around zoning and construction activity — say bloated and disorganized files slow down the response process. When a records request returns a document package with dozens of identical images, it takes extra staff time to sort and takes longer for requesters to parse.

Archivists and IT managers working in the municipal space recommend three concrete steps: adopting hash-based deduplication at the point of upload so duplicate files are flagged before they enter the system; auditing existing repositories with open-source tools before committing to expensive proprietary solutions; and establishing clear agency-level standards for what constitutes a unique image record. For Boston, the window to address the problem cleanly is the period before the expanded permitting portal launches. Once new data starts flowing in volume, retrofitting the archive becomes significantly harder.