Hundreds of water customers in Palo Alto received erroneous bills after the city failed to accurately record the size of their water meters, according to a new audit from the office of City Auditor Harriet Richardson.

The audit, which was released last week, specifically focused on 577 errors that city officials attributed to mistakes involving water meters. The size of the water meters determines the flat monthly fee customers are charged: Those with 5/8-inch mains pay a fee of $16.77, while those with 1-inch mains are charged $34.26.

The errors involved 2.8 percent of the 20,633 meters that the utilities department had installed citywide.

Altogether, the billing discrepancies resulted in $227,900 in errors, which includes $184,000 in underbilling and $44,000 in overbilling, according to the audit.

The underbilling occurred when the city replaced customers' smaller meters with larger ones but did not register the change in its billing system, said Senior Performance Auditor Mimi Nguyen, who physically inspected the meters.

The audit, which the City Council's Policy and Services Committee discussed Tuesday night, found that the city of Palo Alto's Utilities (CPAU) "has not adequately prevented, detected, nor corrected water billing errors." It cites 240 errors that Utilities had identified and corrected in 2015 and 2016, along with 214 errors that the auditor's office identified by physically checking meter sizes and uncovering discrepancies in the city's records. The City Auditor's Office also pointed to 123 "potential errors" that it had not physically verified but that stemmed from discrepancies in purchasing records and water bills.

While the errors weren't limited to any particular section of Palo Alto, more than 100 occurred in Southgate, where the city replaced a water main in 1998 and upgraded some customers from 5/8-inch meters to 1-inch meters. At the time, the difference in water-bill fees charged was only $1.50.

In February 2014, utilities workers discovered that they had been charging 115 customers for the smaller meters even though they had the larger ones and back-billed customers a total of $45,000, according to the audit.

The audit said the city's errors -- and the growing difference between the two different meter types -- raise the question about equity.

"This raises the question of whether two neighbors in similar homes, right across the street from each other, should be paying different meter rates," the audit states. "If so, should the difference be as significant as it has become? It also raises the question of whether CPAU should have looked at the impact of that difference on residents when it first noticed the meter errors in Southgate."

The audit found that most of the errors were caused by a system that requires manual data entry at four different points in the installation and billing processes. The city's SAP system was not configured for "full system integration," according to the report, and the city lacked a monitoring process to compensate for this flaw.

In some cases, the meter shop changed a water meter to a different size but did not notify billing; in others it could not verify the meter size being installed and assumed it was correct in the inventory records.

The audit recommends that Utilities correct all the records, investigate the 123 meters that do not match their purchasing records; explore options for addressing equity issues when changing meter sizes; and ensure that its new enterprise-resource-planning system has automated controls in place to prevent discrepancies.

Utilities officials largely agreed with the audit's findings and recommendations, though General Manager Ed Shikada pushed back against the auditor's characterization of the billing problem as widespread and that the errors resulted in equity issues. He noted that the Utilities Department has recently taken steps to reduce errors, including monthly reconciliation of installation record and billing records.

"That said, it's a highly labor intensive process, so as we go through it, there's still potential for human error to be a part of what we need to look for," Shikada said.

He noted in his response to the audit that staff is reviewing policy options for addressing the issue going forward. One option is "consolidating the fixed rate for the 5/8-inch and 1-inch meters or establishing a special rate.

Shikada acknowledged that quality assurance was lacking prior to 2015 but wrote that staff has "invested tremendous effort and resources to improve data accuracy in the billing system."

"Staff has developed new business procedures and quality assurance controls for the meter lifecycle from procurement to installation, operation and retirement," Shikada wrote. "Since the 2015 meter audit, there have only been two new water meter billing errors identified in the audit -- a significant improvement that deserves recognition."

In addition to the errors on bills, the auditor's office also scrutinized the city's decision in 2013 to install 1,178 water e-meters, which rely on ultrasonic signals to gauge the amount of water passing through a meter. In a separate finding, the audit stated that there are currently "no testing standards, and the accuracy, performance and reliability of these meters are uncertain."

The audit noted that when the new meters were tested for low, medium and high flow, 83 of the 100 e-meters failed one or more of the three required flow tests. But Shikada countered that citing such a high failure rate is misleading because up until the middle of 2015, the city didn't have specific testing measures for the e-meters and had to rely on the manufacturer's.

Referring to an 83 percent failure rate is "not appropriate," Shikada wrote, given that the city's test bench has been "acknowledged as not being property configured for these ultrasonic meters." A sampling of meters, he wrote, was sent to manufacturer for testing and in all cases, the sampled meters functioned within accuracy limits, he wrote.

Even so, he agreed that the city's transition to e-meters was not executed appropriately. The audit found that the city's purchase of the water e-meters was made through what's known as the "sole source" process, which bypasses the typical competitive solicitation process. Normally, when a department wants to request sole sourcing, it has to "demonstrate that the product is necessary for the health, safety or welfare or the city or that a significant cost savings can be realized."

Normally, the purchasing division in the Administrative Services Department reviews and approves such requests, particularly if they refer to "standardized" products that the city had used in the past and that have been shown to be effective. The new e-meters had not been approved as a standardized product, the audit states, and "did not undergo the typical new product review and scrutiny."

Shikada concurred with this finding and wrote in his response that all customers with e-meters will be immediately notified of the audit and that procurement and installation of e-meters will remain suspended until new standards for testing are adopted.

Staff will also address any accuracy concerns with e-meters already installed, Shikada wrote, and any request for removal of an e-meter will be completed at no cost to the customer.

"While Palo Alto prides itself on innovation and being on the leading edge, in this case there was inconsistency between purchasing approval granted and actual purchases," Shikada wrote. "We have therefor discontinued installation of new e-meters and are developing a plan for supplemental testing and customer communications to ensure customer confidence in CPAU's meter reading and billing practices."