Boston's Digital Archives Are Full of Duplicate Images. Officials and Experts Say the Cleanup Is Overdue.

Boston's public records offices and academic institutions are sitting on digital archives bloated with duplicate photographs, mislabeled scans, and redundant image files that inflate storage costs, slow retrieval times, and compromise the integrity of everything from building permits to historical collections. The problem has been quietly growing for years, but a push from City Hall and several local universities this summer has brought it into the open.

The timing matters. Mayor Michelle Wu's administration has made digital modernization a pillar of its second-term agenda, and the city's Department of Innovation and Technology is currently midway through a two-year overhaul of the permitting and inspectional services database — a system that processes tens of thousands of property-related image attachments every year. Redundant files aren't just a storage nuisance; they can delay permits, muddy legal records, and cost taxpayers real money in server infrastructure and staff hours spent manually sorting through identical or near-identical scans.

What Officials and Experts Are Saying

Officials at the Inspectional Services Department, which handles building permits and code enforcement across neighborhoods including Jamaica Plain and Dorchester, have acknowledged internally that duplicate image submissions from contractors and property owners are a persistent drain on the system. Neither the department nor City Hall has released specific figures on how many redundant files currently sit in the database, but digital records specialists say the problem is endemic to any permitting platform that accepts open-format uploads without automated deduplication checks.

At Northeastern University's library on Huntington Avenue, archivists working with the Boston City Archives have been piloting a duplicate-image detection protocol since January 2026. The project uses perceptual hashing — a technique that assigns each image a fingerprint based on visual content rather than file name — to flag near-identical photographs before they are ingested into permanent collections. Early results from the pilot, which covers digitized records from the Boston Landmarks Commission, found that roughly one in eight uploaded image files was a duplicate or near-duplicate of something already in the repository. That ratio, if it held across the city's broader digital holdings, would represent a significant volume of wasted storage and cataloguing effort.

Researchers at MIT's Libraries, located on the Cambridge side of the Charles River, have been tracking the same issue in scientific data repositories. A 2025 study published through the MIT Libraries' data stewardship program found that image duplication rates in biomedical research archives — a category highly relevant to Boston's Longwood Medical Area — averaged around 12 percent across surveyed institutions. The study recommended mandatory deduplication review as part of any federally funded research data management plan, a standard that several Boston-area hospitals and biotech firms have not yet formally adopted.

The Practical Stakes for Boston Neighborhoods

In Jamaica Plain, where housing production has accelerated under Wu's Affordable Homes Act implementation, contractors filing renovation permits at the ISD office on Massachusetts Avenue have reported confusion when submitted photographs are rejected or duplicated across multiple permit applications for the same property. The administrative friction can add days to an already stretched permitting timeline.

Dorchester's community development corporations, including Dorchester Bay Economic Development Corporation on Bowdoin Street, maintain their own property and project image databases for grant reporting. Staff there have described the manual deduplication process as time-consuming, particularly when managing federally required documentation for HUD-funded projects with strict submission deadlines.

Digital records consultants recommend that any institution dealing with large image volumes implement automated deduplication at the point of upload rather than retrospectively. Tools capable of handling this at municipal scale are commercially available, with licensing costs for mid-size government deployments typically running between $40,000 and $120,000 annually depending on volume — a range city budget officials would need to weigh against ongoing staff and storage expenditures.

The Wu administration has not yet committed to a specific timeline for implementing deduplication tools citywide. The Department of Innovation and Technology's current database overhaul is scheduled for completion by the end of fiscal year 2027, which officials and outside experts increasingly agree is the logical moment to bake deduplication standards into the new system architecture — before the duplicate problem migrates intact into the next generation of city records.