Boston's Fight Against Duplicate Images in City Records Puts It Ahead of London, Behind Amsterdam

Boston city archivists confirmed this spring that a cleanup initiative targeting duplicate images across municipal digital records — permit files, property assessments, zoning submissions — had flagged more than 340,000 redundant image files stored across the city's data infrastructure since the project launched in January 2026. The effort, coordinated through the Department of Innovation and Technology on City Hall Plaza, is the most aggressive deduplication push the city has undertaken since migrating its property records to cloud storage in 2022.

The timing matters. Cities from Dublin to São Paulo have spent the past two years wrestling with ballooning storage costs and degraded database performance as legacy paper records were mass-scanned during the pandemic-era digitization surge. Boston is not alone in the mess, but how aggressively it attacks the backlog is now a point of genuine competition among mid-sized cities with large institutional footprints.

What Boston Is Actually Doing

The deduplication program relies on perceptual hashing software — a technique that generates a compact fingerprint for each image file and flags near-identical copies even if file names or metadata differ. The city's IT department applied the tool first to building inspection photo archives covering Jamaica Plain and Dorchester, two neighborhoods where housing production has accelerated sharply under Mayor Michelle Wu's zoning reform agenda. Those two neighborhoods alone accounted for roughly 90,000 of the flagged files, according to city officials, because multiple inspectors had uploaded the same site photos through different departmental portals.

The Boston Public Library's Rare Books and Manuscripts department at Copley Square is running a parallel but separate effort, using open-source tools developed through a partnership with Northeastern University's library science program to deduplicate digitized historical photograph collections. Librarians there identified more than 12,000 duplicate scans of 19th-century Boston street scenes that had been ingested twice during a 2021 grant-funded digitization sprint.

The MBTA, which maintains its own extensive image archive for station inspections and accessibility compliance documentation, has not yet joined the coordinated city effort. Transit agency officials have said publicly that a procurement process for deduplication tooling is ongoing, though no contract award date has been announced.

How Boston Compares Globally

London's Government Digital Service published guidance in March 2025 recommending that borough councils adopt deduplication standards for planning application imagery, but implementation has been left to individual councils, producing uneven results. As of early 2026, fewer than a third of London's 32 boroughs had completed an initial audit, according to a summary posted by the Local Government Association.

Amsterdam is further along. The city's Stadsarchief — the municipal archive — completed a full deduplication pass of its digitized records in late 2024 and has published its methodology as open-source tooling available to other European cities. Amsterdam's project covered approximately 1.2 million image files and reduced active storage load by 31 percent, according to figures the Stadsarchief published on its public project page. Boston IT officials have cited Amsterdam's methodology in internal presentations, though Boston's dataset and institutional structure differ considerably.

Toronto completed a narrower deduplication exercise in 2023 focused solely on building permit photography, and found that redundant images had added an estimated CAD $180,000 in annual cloud storage costs before the cleanup. Boston has not published a comparable cost figure for its own project, though the Department of Innovation and Technology has said the initiative is expected to reduce storage expenditure in the current fiscal year.

Where Boston lags is in cross-departmental coordination. Amsterdam and Copenhagen both operate centralized image repositories with mandatory deduplication checks at the point of upload, meaning duplicates are caught before they enter the system. Boston still runs departmental silos — the Inspectional Services Department, the Boston Landmarks Commission, and the Office of Planning use separate upload systems. Merging those into a unified portal is listed as a Phase 2 goal, with no firm completion date attached.

For residents and developers filing permit applications on projects along Washington Street or Columbia Road, the practical implication is modest for now: faster search returns on public-facing property record lookups, which the city's Assessing Department database serves to title researchers and real estate attorneys daily. The longer-term payoff comes if Phase 2 lands on schedule — a single upload portal would eliminate the duplicate problem at the source, rather than cleaning it up after the fact.