How Boston's City Archives Ended Up With Thousands of Duplicate Images — and What It's Doing About It

Boston's city government is sitting on an estimated tens of thousands of duplicate digital images — photos, scanned documents, and infrastructure records spread across at least a dozen separate department servers — and a quiet but consequential effort to clean up that backlog is now underway inside City Hall on Cambridge Street.

The problem did not appear overnight. It accumulated across three mayoral administrations, multiple rounds of federal digitisation funding, and a series of technology procurement decisions that prioritised speed over coordination. Understanding how Boston got here matters now because the Wu administration's broader push toward open-data government and streamlined permitting — particularly through the Inspectional Services Department's online portal — depends on a clean, non-redundant image archive to function properly.

A Paper Trail That Became a Digital Tangle

The roots of the duplication problem trace back to the early 2000s, when the City of Boston began migrating paper records into digital formats under separate contracts awarded to different vendors. The Boston Public Library's Leventhal Map & Education Center digitised its own holdings independently. The Public Works Department scanned street-condition photographs under a separate contract. The Boston Landmarks Commission maintained its own image library for historic preservation reviews. None of these systems talked to each other.

By the time the Menino administration ended in 2013, city IT staff had flagged internal redundancy as a concern, but no unified image management policy was ever enacted. The problem compounded during the Walsh years, when the rollout of Analyze Boston — the city's open data portal, launched publicly around 2016 — pulled records from multiple legacy systems simultaneously, dragging duplicate files along with legitimate content.

Jamaica Plain and Dorchester bore a disproportionate share of the mess. Both neighbourhoods became focal points for housing development reviews and zoning variance applications after 2018, generating high volumes of site photography that was uploaded by applicants, rescanned by city reviewers, and sometimes duplicated again when cases were transferred between departments. A single parcel on Blue Hill Avenue might have the same inspection photograph stored four or five times across different systems under different file names.

The Push to Deduplicate — and Why It's Complicated

The Wu administration flagged digital records hygiene as a priority as part of its 2023 technology strategic plan, and the Department of Innovation and Technology began a formal image audit in the spring of 2024. The work involves running hash-matching algorithms across archived files — a technically straightforward process, but one that requires a human review layer before any file is deleted from a government system, given public records obligations under Massachusetts General Law Chapter 66.

That legal constraint is significant. Massachusetts public records law requires that government bodies follow a formal records retention schedule before destroying any document, digital or otherwise. Image files connected to building permits, enforcement actions, or litigation holds cannot simply be bulk-deleted, even if they are exact duplicates of files stored elsewhere. The State Archives in Dorchester sets those retention schedules, and city departments must apply for disposition authority before clearing backlogs.

The practical effect is that Boston's deduplication project is moving in phases, targeting internally generated administrative photography first — the category with the clearest retention rules — before tackling more legally complex categories like enforcement case files and zoning appeal records.

For residents and businesses interacting with city systems, the cleanup has a direct downstream benefit. The Inspectional Services Department's online permit portal, accessible through Boston.gov, has been slowed at points by the overhead of bloated image databases. Contractors filing for building permits in Roxbury and East Boston have flagged document upload failures and slow retrieval times as persistent frustrations. Trimming redundant files from backend systems is expected to improve portal response times, though city officials have not publicly committed to a completion date for the full deduplication effort.

The next milestone is a progress report expected from the Department of Innovation and Technology later this summer, which will detail how many gigabytes of duplicate content have been identified and how many files have been cleared for deletion under existing retention authority. Anyone with a pending permit application or an open ISD case file who wants to confirm their records are intact can contact the city's constituent services line at 617-635-4500 or visit City Hall in person at One City Hall Square.