Months after a PG&E gas pipeline exploded in San Bruno in 2010, killing eight people, Palo Alto launched an ambitious, $3 million effort aimed at preventing a similar accident from happening here.
In July 2011, the city entered into a contract with the firm Hydromax to inspect "cross-bores" -- areas where one utility pipeline intersects with another, often unbeknownst to the property owner. In most cases, this happens when a gas line get installed through drilling, rather than trenching, and ends up crossing an existing sewer pipeline.
Under the terms of the $3.5 million contract, Hydromax was charged with using cameras to inspect the city's sewer mains and 18,028 lateral pipes to identify cross-bores and flag any major defects in mains. The city later added several provisions to the agreement, including ones calling for the firm to supplement the city's efforts to clean pipelines and requiring it to re-inspect areas where obstructions had demanded construction and repair.
But things didn't go as planned. By the time the project ended in December 2012, the firm had invoiced the city for 13,725 lateral inspections out of the 18,028 it was supposed to perform, according to a new audit of the cross-boring inspection by City Auditor Harriet Richardson. And when the auditors looked at Hydromax's inspections records, they found that the company had inspected only 10,791 laterals – roughly 60 percent of the total.
Furthermore, the audit indicated that while the company invoiced the city for 13,725 inspections, it failed to complete nearly half of them (6,625 or 48 percent) because of "unanticipated field conditions," according to the audit. The report also noted that the firm's inspection reports contain "inaccurate, incomplete and potentially duplicative information."
"The City did not have a process to validate Hydromax inspection data and allowed the contract to lapse before Hydromax attempted to inspect at least 7,237 laterals," the audit states.
The audit was prompted by Assistant City Manager Ed Shikada, who also serves as general manager of the Utilities Department, and has shed light on oversight lapses and areas of needed improvement in contract management.
Shikada requested the audit after recognizing the "magnitude of dollars expended on the contract and the inability to ultimately complete the goals originally laid out," he told the Policy and Services Committee during a June 13 discussion of the new audit.
The audit, which was conducted by Senior Performance Auditor Houman Boussina and which Richardson's office released last month, found that Hydromax fell far short of the city's goal of inspecting all of its laterals and that the city's oversight was insufficient to ensure that it had obtained accurate and complete inspection data.
The city ended up paying Hydromax $2.95 million for the work. Because the work was concluded in 2012, the statute of limitations prohibits the city from seeking any reductions to the invoiced amount at this time. Furthermore, Richardson said that the audit could not determine whether the city was overcharged because there "wasn't a clear link between the invoices and what laterals were inspected for those invoices."
Even so, city officials believe that the audit taught them a few important lessons about a complex undertaking – lessons that will help shape the next contract. Shikada noted that the contract with Hydromax required the city to pay for even an incomplete inspection, provided the contractor made a "reasonable attempt."
"That set a relatively permissive bar for the contract," Shikada said.
The City Manager's Office concurred with the recommendations of the audit, which the committee approved last month. This includes a recommendation that the Utilities Department identify the sewer pipelines that Hydromax did not fully inspect or attempt to inspect and to prioritize these pipelines in future contracts.
"To the extent possible, based on past experience, predict potential inspection challenges, such as poor pipeline conditions, that may hinder future inspections," the audit states. "Disclose these challenges in future contract solicitations."
Robert Item, an engineer with the Utilities Department who was involved in the cross-boring project, said that Hydromax relied on closed-circuit television and three levels of inspection. First, there's the field inspection. Then, someone else in an office reviews the results and inputs them using coding standards established by the National Association of Sewer Service Companies. Finally, another person reviews the tapes and uses the Geographic Information System data from the city to try to identify any potential intersections between gas and sewer lines.
According to the audit, the company had identified 26 gas cross-bores but did not rule out the possibility of cross-bores for as many as 5,900 other laterals, about 45 percent of the total.
So far, Item said, the city has not had any instances in which city employees found a cross-bore in a location that had been inspected by Hydromax.
He noted that many residents choose drilling over building trenches when installing new lines to "minimize the open cutwork," particularly in areas with extensive landscaping.
"Initially, it saves significant costs," Item told the Policy and Services Committee during a June 13 discussion of the new audit. "However, there are some drawbacks and this has been one of them: making sure there are no cross-bores."
Comments
Crescent Park
on Jul 3, 2017 at 7:55 pm
on Jul 3, 2017 at 7:55 pm
If the city is going to rely on RFPs and outside contractors to do work that its own staff of civil engineers ought be capable of doing, then there should to be better monitoring and supervision of the contracted work by the City Engineering Department.
When the cat's away...the mice will play and/or overbill. If Hydromax is still in business, the city should force them to complete the project as per the original contract agreement. Anything less is a dereliction of duty on the part of both the contractor/consultant firm and the city engineering department.
Crescent Park
on Jul 3, 2017 at 8:09 pm
on Jul 3, 2017 at 8:09 pm
As ever, the city's on top of the contracts they award.
Remember when we were outrageously over-charged by Enron? Then when the city finally got around to canceling the Enron contract, we had to pay a few million more dollars in damages for violating the contract provisions for cancellations?
But hey, no problem for the city. They'll just keep raising our utility rates some more while giving out maximum raises and outragous benefits to all.
[Portion removed.]
Downtown North
on Jul 3, 2017 at 8:27 pm
on Jul 3, 2017 at 8:27 pm
"The City did not have a process to validate Hydromax inspection data ... "
Another costly preventable fumble.
Sigh.
Embarcadero Oaks/Leland
on Jul 3, 2017 at 8:39 pm
on Jul 3, 2017 at 8:39 pm
@Curmodeon, re costly preventable fumbles, several of us were talking about the "Palo Alto Process" the other day and how long it takes to get approval for various things like remodels and how everything comes to a screeching halt when one city employee is out and the paper copy is stalled on his/her desk.
Several people pointed out the Apple donated enough software and services to Cupertino City Govt that they've been able to streamline the contracting and permitting process so that ONE single electronic copy is shared with all relevant parties. Project management proceeds, due dates are tracked, advance warning is given and lo and behold, it works.
PA can't even track complaints to a single department when the dates of the complaints and the locations of the problems are repeatedly cited.
Perhaps one of our large tech companies can DONATE software and services to make the city work instead of finding new ways to bill us?? Since Palantir, for example, sits on so many of our committees, they obviously have the free time.
Downtown North
on Jul 3, 2017 at 10:35 pm
on Jul 3, 2017 at 10:35 pm
Apple's motivation for helping streamline the Cupertino process is plain: that super structure they're building in Cupertino. I suspect our local large projects get expedited via personal red carpet attention. 27 University comes to mind.
The right software can be a big help to a competent contract monitor. It is no help whatever for somebody who cannot even read a contractor's report.
Eyes roll. Heads should roll also. Many heads.
South of Midtown
on Jul 3, 2017 at 11:04 pm
on Jul 3, 2017 at 11:04 pm
Sometimes lowest bid is not always the best. Furthermore, I believe these outside contractors should abide by law,one example, they need to stop using undocumented workers.
Another Palo Alto neighborhood
on Jul 4, 2017 at 12:04 am
on Jul 4, 2017 at 12:04 am
Can ee get these guys to audit the school district construction!
Another Palo Alto neighborhood
on Jul 4, 2017 at 8:39 am
on Jul 4, 2017 at 8:39 am
Although it is easy to get cynical about this topic and about municipal/government employees, IN THE END, city employees will do this type of work with more conviction and more conscientiously than outside contractors whose companies have a somewhat different mission than city government.
Crescent Park
on Jul 4, 2017 at 10:30 am
on Jul 4, 2017 at 10:30 am
> IN THE END, city employees will do this type of work with more conviction and more conscientiously than outside contractors whose companies have a somewhat different mission than city government.
(1) The mission of most outside contractors is to somehow land the contract, make some serious dough and then 'pull the wool' (if possible).
(2) The mission of many city and county governments is to award lucrative contracts to outside vendors/consultants because their in-house staff is either too lazy and/or incompetent to proceed with and manage the project themselves.
Hey, it's taxpayer dollars so the decision makers don't really care...so long as they get their PERS benefits, oddball holidays off and extended lunch breaks. Welcome to reality.
Downtown North
on Jul 4, 2017 at 12:29 pm
on Jul 4, 2017 at 12:29 pm
" The mission of most outside contractors is to somehow land the contract, make some serious dough and then 'pull the wool' (if possible)."
Like all sellers, contractors want the sale. It is up to the customer to verify their capability and prior performance before hiring them. It is then up to the customer to monitor their work for quality and state of completion. That latter was apparently not done here.
Like, would you pay a roofer the full contract price if they left a big part of your house unroofed? You'd at least look at the roof first, wouldn't you?
another community
on Jul 5, 2017 at 11:33 am
on Jul 5, 2017 at 11:33 am
Grandma Millie--and many more like her--are completely ill informed on the Enron deal. It's very lucky for the City (and, ultimately, Utilities ratepayers) that Enron went bankrupt since Utilities ended up paying pennies on the dollar for getting out of a contractual commitment. If Enron had not gone bankrupt, Utilities would have had to pay the contract price (which was competitively bid and Enron provided the lowest price) for the duration of the contract term.
Regarding this audit on the Hydromax contract, I hope the City finds someone to complete the work since this is an important safety program. It sounds like the work is more costly than originally anticipated. I wish PG&E would do the work of scanning for potential crossbores and fixing any found in my town!
Another Palo Alto neighborhood
on Jul 5, 2017 at 11:34 am
on Jul 5, 2017 at 11:34 am
I'm glad Ed Shikada initiated this audit. Generally, I find him to be competent and motivated city employee. We want to encourage that. Unfortunately, it appears to be too little too late this time.
I am concerned about the increase in use of contractors across the board. Generally, I find they are not adequately supervised, and the quality of their work reflects that. I notice this particularly in the Planning & Community Environment Department.
City Council, I am sure you have noticed the decline in staff report quality. There is a reason for that. Please be vigilant. Hold staff to high standards.
Registered user
Mountain View
on Jul 5, 2017 at 6:43 pm
Registered user
on Jul 5, 2017 at 6:43 pm
The City of Palo Alto got screwed by the contractor. There is no other way to describe the situation.
I note that the same #132 pipe that incinerated houses in San Bruno runs through Palo Alto and Mountain View. If a proper video inspection was done, it might be seen that all the crossbores hold #132 together!
When you pull a permit, the utilities in the area are supposed to be marked so that the work you do does not create any crossbores. Unfortunately, many people and contractors do the work and ask " forgiveness " AFTER they do the damage!
This problem is under CONTRACT LAW. That means the City Attorney has to get off it's butt and actually do something! It's also time to get a replacement if this Attorney DOES NOT DO THE JOB.
A side note: As I was tracing #132, I found the pipe actually runs in the ROW of my parent's house! When the Big One hits, I'll have to assume my parents will get incinerated when #132 breaks. They just replaced their service water main last week. They had it trenched across the front lawn. The plumbers tried to screw them too.