Boston Archives and City Agencies Tackle a Surge in Duplicate Digital Images Clogging Public Records

Boston's city archivists confirmed this week that a spike in duplicate digital image files — copies of the same scanned documents stored multiple times across separate servers — has created a measurable bottleneck inside the city's public records infrastructure, forcing an emergency review by the Department of Innovation and Technology based at City Hall on Cambridge Street.

The problem isn't new, but it got worse fast. Over the past 18 months, the city's push to digitize paper records held at the Boston City Archives in West Roxbury and at branch offices in Roxbury and South Boston added tens of thousands of scanned images to the municipal document management system. With no automated deduplication layer in place at the time of upload, the same files were often saved two, three, or more times under different file names or folder paths. By late June 2026, the redundant files were consuming an estimated 40 percent of allocated server storage, according to an internal review circulated to department heads ahead of the July 1 fiscal year transition.

Why This Week's Review Matters

The timing is pointed. Mayor Michelle Wu's administration has tied expanded digital public access to her broader housing and transparency agenda, and the city's open-records portal — used heavily by developers, lawyers, and residents tracking permits in Jamaica Plain and Dorchester — depends on the same document management infrastructure now clogged with duplicate files. A slowdown in the portal directly delays permit lookups, a practical problem for the neighborhood-level housing production drives the administration has been running along Washington Street and Blue Hill Avenue.

The Boston Public Library's digital collections team at the Copley Square branch has been dealing with a parallel issue on its own servers, where historical photograph collections digitized under a 2023 grant program ended up with duplicate master files stored redundantly across two separate library network drives. Librarians flagged the problem to the city's IT department in May. The convergence of both institutions reporting the same structural flaw in the same month gave the Department of Innovation and Technology the cross-agency evidence it needed to escalate the issue to a citywide policy response rather than treating each case in isolation.

Deduplication software licenses for municipal-scale document systems typically run between $15,000 and $80,000 annually depending on storage volume, based on publicly available procurement data from comparable municipal contracts in cities like Chicago and Philadelphia. The city has not yet posted a formal procurement notice, but the fiscal year 2027 technology budget — approved by the Boston City Council in June — set aside $2.3 million for document management infrastructure upgrades, a figure that would comfortably cover both software licensing and the staff hours needed to audit and remove redundant files across departments.

What Comes Next for City Records Access

The Department of Innovation and Technology has until September 30, 2026, to complete an initial audit of the largest redundant file clusters — a deadline set internally as part of the FY2027 transition plan. The audit will prioritize records tied to the Inspectional Services Department, which handles building permits and code enforcement, because those files generate the highest daily query volume from residents and contractors working in neighborhoods like Jamaica Plain, Dorchester, and East Boston.

For residents and professionals who rely on the city's public records portal, the practical advice from archivists is to submit open-records requests through the city's Efile portal rather than walking in to branch offices, which frees up staff time for the deduplication work. Requests filed electronically are also being processed on a separate, cleaner server queue that is not affected by the duplicate-image backlog.

The Boston City Archives in West Roxbury is also accepting appointments for in-person document reviews while the digital system is under repair — a temporary workaround, not a long-term fix, but one that keeps the public's access to historical records intact through the summer. The archive holds materials dating to the 17th century, and its physical collections are unaffected by the server-side problem.