Boston's Duplicate Image Crisis: What Happens Next and the Key Decisions Ahead

Boston's public agencies are sitting on hundreds of thousands of duplicate digital images spread across city servers, and a deadline is approaching. The city's new Digital Records Retention Policy, set to take effect October 1, 2026, will require all municipal departments to certify their image archives are deduplicated and properly catalogued — or face a freeze on new storage procurement. For a city running aging infrastructure alongside a $4.3 billion annual operating budget, the cleanup is anything but routine.

The stakes are higher than they might look. Duplicate image files are not just a storage nuisance. They slow archival retrieval, inflate cloud licensing costs, and create legal exposure when records requests produce inconsistent document sets. Boston's switch to a hybrid cloud storage model in 2024 — migrating data from the Government Center data facility to vendors including Microsoft Azure — exposed just how badly image libraries had compounded across departments. One city IT assessment, shared with the Boston City Council's Government Operations Committee earlier this year, found duplication rates in some agencies exceeded 40 percent of total stored files.

Where the Problem Is Most Acute

The Boston Planning Department, which manages permit documentation across neighborhoods from Jamaica Plain to Dorchester, has among the largest image backlogs in city government. Permit photos, site inspection images, and zoning-board presentation decks have accumulated since at least 2016 with no systematic deduplication protocol. The Inspectional Services Department on City Hall Plaza faces a similar crunch: its field inspectors upload photos from mobile devices that sync to multiple servers simultaneously, creating automatic duplicates before anyone reviews them.

Boston Public Schools, which operates 125 school buildings and generates facility documentation at scale, is also in scope. The district's IT office has been working with a third-party contractor since March to audit image stores tied to its capital improvement program, a project that has already identified more than 80,000 redundant files in preliminary scans.

The MBTA, though a state agency rather than a city department, has its own parallel reckoning. The Authority's transit modernization push — which includes camera upgrades along the Red and Orange Lines — has generated a surge in video frame captures and still images that require the same kind of archive discipline the city is now scrambling to apply.

The Decisions That Will Define the Outcome

Three choices will determine whether Boston meets its October deadline. The first is whether to use automated deduplication software or conduct manual review. Software tools are faster and cheaper upfront, but city IT staff have raised concerns that algorithm-based deletion could remove images that look identical but carry different metadata — a risk in any litigation hold. The Government Operations Committee is expected to take up this question at a hearing scheduled for late July.

The second decision involves who pays. The Mayor's Office of New Urban Mechanics, which has championed tech modernization since the Wu administration took office, has proposed using remaining American Rescue Plan Act funds to cover contractor costs, but that pool of money is under competing pressure from housing initiatives in Roxbury and East Boston. Any ARPA reallocation requires City Council approval, adding another procedural layer before work can begin in earnest.

The third and least-discussed question is what to do with legitimately unique historical images that surface during the cleanup. The City Archives on School Street has the authority to designate materials for permanent retention, but its staff of fewer than ten archivists is not sized to handle a rapid influx of flagged files from a dozen agencies simultaneously.

Department heads have until August 15 to submit preliminary compliance plans to the city's Chief Information Officer. Agencies that miss that internal deadline are unlikely to certify compliance by October 1. For residents and businesses that rely on permit records — especially in fast-developing corridors like the South End and the Seaport — a records freeze would mean slower responses to public document requests at exactly the moment construction activity in the city remains near its highest level in a decade.