How Boston's City Archives Ended Up With Thousands of Duplicate Photos — and What It's Doing About It

Boston's city government is sitting on a digital image library that has ballooned to more than 400,000 files, a significant portion of which are exact or near-exact duplicates — the accumulated residue of ad-hoc scanning drives, departmental uploads, and at least two emergency migrations of legacy servers that nobody fully documented at the time. The Office of Digital Innovation and Technology, which operates out of City Hall on Cambridge Street, confirmed this spring that a formal deduplication project is now underway, targeting municipal records spanning from roughly 2008 through 2024.

The timing matters. Mayor Michelle Wu's administration has made open-government data a stated priority, and the city's ongoing effort to modernize the MBTA corridor and expand housing production in neighborhoods like Jamaica Plain and Dorchester depends on accurate, accessible permitting and inspection photographs being retrievable on demand. When engineers or zoning attorneys pull records for a project on Centre Street or Blue Hill Avenue, they can't afford to wade through five copies of the same image stamped with different metadata.

How the Backlog Built Up

The problem didn't start with any single decision. It compounded across administrations. Under Mayor Thomas Menino, city departments largely kept their own siloed photo archives with minimal cross-departmental coordination. When Mayor Martin Walsh consolidated several municipal functions under the Department of Innovation and Technology between 2014 and 2017, staff migrated images from older systems without deduplication protocols, because there simply weren't any. Files moved; copies stayed. The process repeated itself at least once more during a server infrastructure upgrade around 2020, according to a city procurement document posted to the Boston.gov vendor portal last year.

The Boston Public Library's Digital Commonwealth project, which handles digitization of historical collections at the Copley Square branch, ran into a parallel version of this problem on its own archival holdings and developed an internal hashing workflow to flag duplicate scans. That approach — using checksums to match identical files regardless of filename — became something of a local model when city IT staff were looking for scalable options. The two institutions are not formally partnered on the current project, but city procurement filings reference the methodology.

Boston City Council approved a line item in the fiscal year 2025 budget allocating funds toward digital records modernization broadly, though the specific deduplication contract — awarded to a Massachusetts-based vendor — was not individually itemized in public budget documents reviewed by The Daily Boston. Cloud storage costs for municipal governments have risen sharply since 2021, and redundant files represent direct carrying costs, not just an organizational nuisance.

What the Fix Actually Looks Like

The current project, which began in earnest in January 2026, works in two phases. The first phase targets inspection and permitting photographs held by the Inspectional Services Department, whose records are critical for housing enforcement actions in Roxbury and East Boston. Duplicate images in that category were flagged at a rate city documents describe as unusually high, partly because field inspectors historically emailed photos to supervisors who then uploaded them independently to the central archive.

Phase two will address images held by the Parks and Recreation Department and the Boston Transportation Department, whose street-level documentation of infrastructure projects along corridors like Melnea Cass Boulevard has grown substantially as the city photographs road conditions before and after repair contracts.

For residents and researchers who use the city's public records portal, the practical effect should become visible later this year. Search returns for property addresses should shrink to manageable result sets rather than returning dozens of functionally identical images. Historic preservationists working through the Boston Landmarks Commission on cases in the South End have flagged the duplicate problem to city staff before — too many copies with mismatched dates made it hard to establish a reliable photographic timeline for buildings under review.

The deduplication work is expected to wrap up before the end of fiscal year 2026. Anyone with a pending public records request touching photographic evidence from a city department should note that some image file references in older responses may be updated or consolidated as the cleanup progresses — the city's public records office has been advised to flag that possibility to active requesters.