New report: VDH got into water plant 6 hours later than city initially claimed

The city’s updated water crisis report, released early Monday morning, contains revisions to the timeline of events that unfolded on Jan. 6 , placing the arrival of state health officials more than 6 hours later than the city initially claimed.

The city’s preliminary report, produced by the engineering firm HNTB and released on Feb. 13 , indicated that a representative for the Virginia Department of Health was on site at Richmond’s water treatment plant by 12:30 p.m. on Jan. 6, the day that a power failure at the facility took the city’s entire water treatment system offline.

But the new report shows that the VDH could not access the plant until 6:50 p.m.

The updated report also shows VDH staffers had to reach out to the city to be briefed on the unfolding disaster inside the plant after “hearing about issues … from a third party.” That didn’t happen until 2:50 p.m., according to the revised timeline — over two hours after the city originally said VDH was aware of the situation.

The report does not notate or explain the changes, although they align with texts and emails between senior VDH officials exchanged on Jan. 6. The Richmond Times-Dispatch previously reported that those internal communications contradicted the city’s timeline for informing state health officials of the plant failure.

During a news conference on Monday afternoon, Richmond Department of Public Utilities Director Scott Morris explained that the original timeline had been “cursory” and generated based on “reviews with staff.”

After issuing the first report, HNTB reviewed “call logs, responses from VDH and (other) types of correspondence” that conflicted with information provided by staff, and then made the appropriate alterations, Morris said.

Morris appeared to place some of the blame for the differences on VDH, which he said was provided a draft copy of the report but didn’t “note … the discrepancies.”

VDH representative locked out of plant, left the scene

HNTB’s second report partially attributed the slowed VDH response to state health officials’ inability to enter the plant upon arrival.

According to the report, the VDH representative made it to the plant by 5:30 p.m. on Jan. 6, but “(could) not get in the gate.” Unable to get in touch with DPU staff inside the plant, the VDH representative left the scene.

The representative returned at 6:50 p.m. and finally was admitted to the facility.

James Reynolds, VDH’s Richmond director of field operations, arrived at the plant at 11:30 a.m. the following day, almost 18 hours after the crisis began.

The report notes that, while the delayed communication with VDH did not violate state waterworks regulations — which require localities to inform state health officials “within 24 hours of (an) incident” — it is “best practice to notify them as soon as it was evident that … impacts to the distribution system were possible.”

Richmond’s plant supplies water to Chesterfield, Hanover and Henrico counties, which experienced the impacts of the outage to varying degrees. Both Hanover and Henrico, like Richmond, had to issue boil advisories as a result of low water pressure.

According to the report, the first discussions with the localities “did not adequately convey the severity of the situation” — although Mayor Danny Avula pushed back against that finding during the Monday afternoon news conference, and defended the city’s communication with regional leaders.

“I think 9 a.m. is reasonable,” Avula said, referring to an email sent at that time to Henrico leaders. “What I’m looking at shows that there was a good-faith attempt … I don’t know how to comment outside of that.”

Report identifies decades of deferred maintenance

The sequence of events that caused the plant meltdown and left residents without drinkable water for six days is largely unchanged in the new report, but the 30-page document does contain detailed information on some of the infrastructural and operational problems identified by HNTB.

For example, in addition to the lack of training and the inadequate standard operating procedures cited in the previous report, the updated document faults city officials’ sluggish response to the crisis.

Officials failed to alert regional customers in a timely fashion, struggled to establish a clear chain of command between operations and maintenance crews and were short-staffed after a “mechanical staff member … went home early” following snow and ice clearing operations, the report says.

And “there were no electrical or (instrumentation and control staff) on site at the time of the power outage,” HNTB found. “Based on staff interviews there are occasional problems finding coverage when an operator calls in sick or otherwise cannot come in.”

During a Monday afternoon meeting of City Council’s Organizational Development Standing Committee, HNTB Vice President Robert Page told the body that the communications breakdown created “a very dangerous environment for staff” — particularly when crews restored power to the plant without warning DPU employees.

“We’re probably lucky no one was hurt,” 2nd District Councilwoman Katherine Jordan responded.

But city officials’ failings also long preceded Jan. 6, the report suggests. HNTB highlighted decades of deferred maintenance and a poor process for submitting work orders.

Project documentation provided to the engineering firm by DPU showed multiple projects that were “slated for implementation from 2001 to 2007,” but were “only recently implemented or are just now being implemented.”

“There could be many reasons for the 20-year delay for capital projects, but it indicates that there may be substantial amounts of deferred replacement that leads to running equipment failure,” the report says.

The deferred projects are not specifically identified.

The Times-Dispatch previously reported that city officials for at least 8 years put off replacing the plant’s switchgear . The switchgear is designed to transfer the plant from its main power feed to a second primary power feed in the event of a blackout. Its bus tie failed during the Jan. 6 power outage.

Compounding the maintenance problems, according to the report, is the fact that only operations supervisors — and not lower-level operations staff — can access the system to submit and check the status of work orders. That places a high “administrative load” on the supervisors.

“A majority of the work orders entered into DPU’s (system) lacked key identifying information and details, such as asset IDs, asset names, asset locations and the specific actions taken to complete the (work order),” the report says.

Overlapping work orders seeking the same or similar services indicated that “some preventive maintenance is not regularly completed within the recommended time interval.”

Flooding at plant a ‘common occurrence’

Some of the challenges faced by DPU are a result of the layout of the century-old plant, the HNTB found.

The plant’s clearwells — storage basins for water going through the treatment process — are located in the plant’s basement. The basement also hosts critical electrical components.

The combination can make operations tricky, the report says.

Valves to the pumps that feed the clearwells are “in the lowest level of the plant, and are accessed via stairs and a number of ladders from platforms in the center of the basement,” the report says. During the Jan. 6 power outage, those valves remained open rather than closing automatically.

That caused the clearwells to overflow and the basement to flood, and destroyed non-water-resistant equipment.

“Manual operation of the valves quickly is impractical because of how they are accessed, the number of valves (22), the amount of time it takes to operate (them) by hand and the potential safety risks for an operator to be in the lowest levels once flooding starts,” the report says.

Plant staff told HNTB flooding was a “common occurrence,” citing a previous flooding incident that shut down the facility for at least six hours — although the plant had power during that incident.

During flooding events, plant staff use basement sump pumps and standby pumps at ground level to drain the affected area. But “the conditions these pumps are operating in are less than ideal,” the report says.

DPU staff have to run 60 feet of pipe and hose from the basement to the standby pumps, which sometimes discharge water back into the basement — a failure that occurred during the water crisis.

Confusion over generators persists

While the updated report still indicates that the plant’s third redundancy — backup diesel generators — never came into play, it is still not clear why.

According to the report, the on-call electrician arrived at the plant at 6:30 a.m., but staff made “no attempts … to start the backup generators.” Instead, DPU staff were focused on making the manual switch to the secondary power feed that the switchgear had failed to complete.

The report characterizes that as the “prudent course of action,” arguing that, if staff had been trained and reacted correctly, it would’ve taken less time to make the manual switch to the secondary power feed than it would’ve taken to transfer the plant to the generators.

But the report also says staff were not adequately trained and did not react correctly — leaving unclear the wisdom of neglecting the backup generators altogether.

HNTB has recommended automating the backup generators.

Utility rate raises may be on the horizon

The time elapsed between reports has continued to be a source of frustration for multiple councilmembers, especially given the pressing need for information in light of the upcoming budgeting process.

Mayor Avula will propose his budget to City Council later this month, but during Monday’s committee meeting, 8th District Councilwoman Reva Trammell stressed the fact that the body cannot make decisions about billing and spending allocations without knowing more about what’s needed at the plant.

“There are so many unanswered questions,” Trammell said. “Right now, we are going through the budget, so we need to know. This is the most important thing right now that should be on all our minds.”

She then sparred with Morris over the prospect of raising utility rates.

“I know we’re also talking again about raising the water and stormwater (rates),” Trammell said. “People are in crisis, they cannot afford to keep paying. Every year, it keeps going up.”

“Now, they have to pay for something that … they had no control of,” she added.

“I would not advocate for not increasing rates,” Morris replied. “When you stop having rate increases, you start having deferred maintenance.”

But the rate increases cited by Trammell have coincided with the deferred maintenance spelled out in HNTB’s report.

“You try telling 50 people a day ‘I’m sorry,’” Trammell snapped at Morris. “They don’t want to hear you’re sorry, they want to know why, and what you’re going to do about it.”

“I’ll tell you what, you will get an earful, and you won’t be too happy,” she said.

Get email notifications on {{subject}} daily!

Email notifications are only sent once a day, and only if there are new matching items.