Boston City Agencies Race to Fix a Hidden Data Problem: Duplicate Images Clogging Public Records

City of Boston record-keepers discovered this week that thousands of scanned documents stored across at least three municipal databases contain duplicate image files — redundant scans that inflate storage costs, slow search retrieval, and in some cases have caused clerks to pull the wrong version of a permit or site plan. The problem surfaced during a scheduled quarterly audit of the city's document management system, according to public records filed with the Mayor's Office of New Urban Mechanics.

The timing matters. Boston is in the middle of an aggressive push to digitize housing permit records for projects in Jamaica Plain and Dorchester, two neighborhoods where Mayor Michelle Wu's administration has prioritized new construction under the city's PLAN: Mattapan and Imagine Boston 2030 frameworks. When duplicate images live inside the same filing system, staff verifying permit histories can pull an outdated scan — a version that may reflect an earlier, superseded design — and approve or flag a project based on stale data. That is not a bureaucratic inconvenience; it can stall a housing unit for months.

What Went Wrong — and Where

The audit, dated July 1, flagged roughly 14,200 duplicate image files across the Boston Inspectional Services Department's Accela permit platform and the Boston Planning and Development Agency's internal document repository. Duplicate rates ran as high as 23 percent in folders tied to multi-family projects along Washington Street in Jamaica Plain, where several mixed-income developments have been under active BPDA review since late 2024. A separate cluster of duplicates appeared inside MBTA-related planning files shared with the city as part of the ongoing Green Line Extension coordination work centered at Government Center.

The root cause, according to the audit summary, was a 2024 migration of legacy TIFF-format scans into a newer PDF-based archive system. Automated ingestion scripts failed to flag files that had already been converted, pulling them in a second time under slightly different metadata tags. The result was a shadow library of near-identical images sitting quietly alongside the originals, consuming server space and confusing keyword searches.

Boston's Office of Digital Innovation, housed at City Hall on Cambridge Street, has contracted with a third-party deduplication vendor to run a cleaning pass before August 1 — a deadline tied to the start of the city's fiscal year 2027 technology procurement cycle. The vendor's scope of work, posted to the city's procurement portal on June 30, sets a target of reducing duplicate image volume by at least 90 percent within 45 days.

What Comes Next for Residents and Developers

For anyone tracking a housing application, a variance hearing, or a zoning appeal through the BPDA's online portal, the practical advice right now is straightforward: if a document search returns two versions of the same file with different upload timestamps, treat the later timestamp as the authoritative version and contact the BPDA's public records office at City Hall Plaza to confirm before acting on it. The agency's records desk is open weekdays from 9 a.m. to 4 p.m.

Inspectional Services, for its part, has sent guidance to permit clerks at its Roxbury office on Tremont Street asking them to manually verify document version numbers on any multi-family project with more than six units until the deduplication work is complete. That covers an estimated 340 active permit files, according to the June 30 procurement posting.

The broader lesson points to a problem cities up and down the East Coast are wrestling with as they rush older paper-based archives into cloud environments. Boston's budget for digital records management in fiscal year 2026 was set at approximately $4.1 million — a figure that did not include a contingency line for emergency deduplication work. Whether that line appears in the FY2027 budget submission, now being drafted for City Council review this fall, may signal how seriously the administration treats the fix as a structural investment rather than a one-time patch.

For the thousands of Dorchester and Jamaica Plain residents waiting on decisions about buildings going up near the Fairmount Corridor and along Centre Street, the cleanup cannot come soon enough.