The City of Palo Alto's business operation had a significant security breach that left sensitive employee and customer information open to outside access, an investigation by the City Auditor's Office found.
According to the audit, which was released this week, the firm SAP failed to secure a "powerful account," allowing the auditor's office access to sensitive and confidential information for what the report called an "extended period of time." The report also found that the Administrative Services Department, which oversees the city's finances, failed to effectively manage SAP user accounts to ensure security.
"Such access could have allowed a motivated and sufficiently capable person to destroy or modify data, expose sensitive employee and customer information, or defraud the City," City Auditor Michael Edwards wrote in the report.
The SAP Enterprise Resource Planning application, which the city has been using since 2002, supports the city's accounting, finance, purchasing, human resource, and utilities functions.
The auditor's office made its finding about breached security after a January incident in which the office was able to use a "default" password to tap into an account that should've been sensitive, including employees' Social Security numbers, payroll records and credit information. The account also granted the auditor's office access to create vendors and approve invoice payments, according to the audit.
Further investigation found that the account is usually "locked," but an SAP administrator "opened" it because of a technical issue during software installation last December. After the installation was completed, the SAP employee did not secure this account, the audit found.
The report also stated that the Administrative Services Department "did not have adequate policies and procedures to secure these powerful standard accounts," the audit stated. The department has since taken steps to identify and secure these accounts.
The report also found that the department "violated two critical security principles by not properly restricting access for all user accounts." The audit recommends that the department adopt formal policies addressing user access and implement procedures "to either prohibit or control the use of all other powerful system-provided SAP profiles."
In a response to the audit, Lalo Perez, director of the Administrative Services Department, wrote that staff has "made it a top priority to rectify" the security problems and has "taken action to address many of the findings in the audit." The security breach, he wrote, was limited to a very small number of city employees. Outside users, he wrote, would have to first breach a firewall and the SAP security system to access the account.
"While it is unacceptable that sensitive information was exposed, the limited number of staff with the ability to access the information is trained to access sensitive information while upholding confidentiality standards," Perez wrote.
He noted that over the past few months, his department has been working with the City Auditor's Office in developing a system to monitor the SAP system -- a process that he said "has significantly improved the security of the SAP system."
Comments
Adobe-Meadow
on Oct 14, 2011 at 10:23 am
on Oct 14, 2011 at 10:23 am
So was our private information hacked or not?
Charleston Meadows
on Oct 14, 2011 at 11:37 am
on Oct 14, 2011 at 11:37 am
Minor internal affair -- hardly newsworthy. Problem identified, planned fix in place... done.
Now if the article was about spending controls, and reducing City expenditures per resident consistent with the progress made by other neighboring cities, THAT would be news.
Respectfully,
Tim Gray
Charleston Gardens
on Oct 14, 2011 at 11:41 am
on Oct 14, 2011 at 11:41 am
how can something like that happen with an award winning finance department at the most forward-thinking city along the peninsula?
If the city was not aware of the problem, how would they know if the personal/sensitive information leaked? Could employees who used to work in that area (or HR/Utilities) still have access and take the info internally and sell it?????
Barron Park
on Oct 14, 2011 at 11:45 am
on Oct 14, 2011 at 11:45 am
The article effectively indicates that there wasn't a release of information, much less a hacking, that is known at this time. If there was a release/hacking known, I am certain that it would have been in the auditor's report.
The report appears to be about serious procedural error, a "backdoor" to the data and to payment authorizations that was mistakenly left "open" after the ERP implementation. A grave error, without doubt. But (as best can be told from the article), not resulting in data release nor hacking.
We can't tell from this if there is a systems audit trail that could indicate if an data release or payment authorization wrongly occurred. I am sure that was asked and a review of the auditor's report would likely make this clear.
Another Palo Alto neighborhood
on Oct 14, 2011 at 12:39 pm
on Oct 14, 2011 at 12:39 pm
> If there was a release/hacking known, I am certain that it
> would have been in the auditor's report.
That depends if there were any record of the illegal access to the system, and then evidence of that illegal access made public in some way, such as posting the data on a web-site, or sending it to the media.
The whole area of computer security is poorly understood, and it’s almost impossible to make any cogent statements about the security of a given computer system (or software package) without having a lot of access to the operating system’s, or application package’s, source code. This access would then need a full code review, and then a lot of testing, to determine just how secure the system is to illegal penetration attempts.
The problem becomes difficult to analyze when a default password is not changed, and someone attempting illegal entry tries the known default passwords. If the accesses are logged, then the question as to how much information is logged becomes central to being able to detect an illegal access. Then there is the issue of data access. Security-oriented operating systems, and software, might log all access to data. Historically, security has never been particularly important to software developers, and too often seen as more of a hindrance, than a necessity. So, without a lot of special knowledge about both the Operating System (OS) and the application software (SAP, in this case), claiming that no one had gained access to the system, or no information had been removed, would be hard to know.
Every organization needs to encrypt all of its crucial data, and to add penetration detection software. The City of Palo Alto is long overdue for these sorts of audits, and probably has few people on staff that knows what to do to make these systems more secure.
Midtown
on Oct 14, 2011 at 5:51 pm
on Oct 14, 2011 at 5:51 pm
How do these guys keep their jobs?
This is the same IT department that threw away $250,000 on a “new & improved” city website. It also spent $8.8 million for online utilities billing, which “included such pesky glitches as confusing computer-screen displays and bills that don't add up.” Council was then asked for an additional $223,725 to resolve a list of "post-implementation issues."
Web Link
Midtown
on Oct 14, 2011 at 8:57 pm
on Oct 14, 2011 at 8:57 pm
OK, let’s assume it is unlikely any critical information was truly compromised. I’m a retired IT exec and have consistently found Palo Alto IT staff to be remarkably incompetent.
I offer the following analogy: A police office addresses a school class for a community services presentation. In the process, he chooses to display his firearm and accidentally discharges a shot into the wall. Well, you can say no one was hurt, but the act itself was felony stupid. So is the conduct of the Palo Alto IT department… felony stupid.
Whoever failed to secure the account and his manager should be fired immediately
another community
on Oct 14, 2011 at 10:30 pm
on Oct 14, 2011 at 10:30 pm
Palo Alto IT has suffered 4 changes in leadership in the past several years, and is about to suffer another soon.
When these changes happen, all the competent people move on to greener pastures, and only the dregs remain. As it is now, Palo Alto IT is staffed mostly with people who couldn't find IT jobs during the Dot Com boom.
The next change of guard will cause another such purge, but the tech sector is again doing well in the valley. Given that, plus public hostility towards government workers, an insufferable work environment within the City, low wages vs private industry, and that once hired in government, you are forever "branded" and will never be employed in the private sector again.... Given all of that, don't expect any improvements.
Midtown
on Oct 15, 2011 at 5:00 pm
on Oct 15, 2011 at 5:00 pm
"Palo Alto IT has suffered 4 changes in leadership in the past several years, and is about to suffer another soon."
What changes? Glenn Loo has been Chief Information Officer since at least 2002.
another community
on Oct 15, 2011 at 5:13 pm
on Oct 15, 2011 at 5:13 pm
Longer term and higher up.
Neff -> Harrison -> Yeats -> Perez -> ?
another community
on Oct 15, 2011 at 8:50 pm
on Oct 15, 2011 at 8:50 pm
The fault rests with SAP, not the City. Staff reductions imposed by Council and upper management have thinned the ranks so dramatically that "backchecking" the work of a vendor is improbable if not impossible. The IT staff that's left is perfectly competent to "backcheck", but it doesn't have the time. This is just the tip of the iceberg. More instances of these lapses will occur because there are too many unfilled positions.
Another Palo Alto neighborhood
on Oct 15, 2011 at 8:52 pm
on Oct 15, 2011 at 8:52 pm
Dianah Neff was the IT manager, but not Administrative Services Manager, if memory serves; whereas the other three were Administrative Services Managers with responsibility for IT.
However, the general point about people at the top not really understanding the details of Information Technology stands.
Neff, who did not seem to provide much in the way of leadership when she was here, actually has moved onto a number of high visibility municipal technology leadership roles:
Web Link
another community
on Oct 15, 2011 at 9:05 pm
on Oct 15, 2011 at 9:05 pm
@Wilson
Neff was a direct report to the City Manager, as were the others.
You are right about Neff, she went on to become a prominent figure in the public IT sector.
But, as usual, Palo Alto had no clue as to what they had when she was here.
Another Palo Alto neighborhood
on Oct 16, 2011 at 8:28 am
on Oct 16, 2011 at 8:28 am
> The fault rests with SAP, not the City.
And so if you get into a collision with your car, the fault lies with the manufacturer?
Pleeezzzzeee...
Responsibility for any organization lies at the top, and should be delegated down the chain-of-command to appropriate management levels. Unfortunately, local governments have never been particularly well managed, and so this idea of "responsibility" does not seem to be well defined in most city-level governments. That's most certainly true in Palo Alto.
While computer security has never been all that well understood, there are consultants that have emerged, over the years, that can be engaged to do period audits of most vendors’ systems. These consultants can do the work of auditing a system, or organization’s, security, or they can develop a security model, leaving the local staff to do the work every periodically. Creating a checklist of security “weak points” that requires staff to check various passwords to insure that they are not “defaults”, or that they are long enough to make penetration difficult, is not that difficult.
But.. someone has to recognize that this must be done, and that these periodic audits, and follow-ups, are done falls to the top level of management. In the private sector, firing people for failing to do the work is an option. In the public sector, it seems that “employee rights” trumps the obligation of the local government agency to protect data—so it’s hard to believe that anyone involved in any kind of incompetence, short of actually breaking state law, would ever “get the ax” for failures like this one. To make matters worse, what oversight that exists—via the City Council—does not seem to have much power over the City Manager, other than to dismiss at will. Moreover, Council’s are not chosen for their technology skills. They seem to be chosen by special interests groups to make certain that the City money is spent on them. So, having the Council even understand that “computer security” is important is something that may, or may not, happen.
By way of example, some computer systems keep remind people to change their passwords every so often (like ninety days, or so). The login code then badgers people to change their password once that time has expired, until they do. Site Administrators might even visit those who have not changed their passwords to remind them that they are not complying with the efforts of the organization to keep the computer system secure. When people resign, or are terminated, from such organizations, it’s not uncommon to require people to change their passwords immediately. This is a lot of work on everyone’s part to insure the integrity of the intellectual property stored on an organization’s computers. Unfortunately, we don’t seem to be seeing that sort of commitment on the part of the Palo Alto City Government management
another community
on Oct 16, 2011 at 2:38 pm
on Oct 16, 2011 at 2:38 pm
@PA-Needs-Security-Audits
1. A collision may indeed be the fault of the manufacturer.
2. Couldn't agree more with the need for security. Measures used to be in place. But the City eliminated so many positions that the work can't get done anymore. The coach can't win with an empty bench.
Midtown
on Oct 16, 2011 at 2:51 pm
on Oct 16, 2011 at 2:51 pm
Retired Staffer: Please be specific and tell us just how many people have left the IT department in the last year.
I don't think the coach has an empty bench. I think the coach and those sitting on the bench are responsible for the problems.
Another Palo Alto neighborhood
on Oct 16, 2011 at 8:02 pm
on Oct 16, 2011 at 8:02 pm
> A collision may indeed be the fault of the manufacturer.
Yes, that's true .. but when this happens, it happens to a lot of people about the same time .. and all of the country's ambulance chasing lawyers show up and make a lot of money out of the failure of the manufacturer. How many similar situations have occurred with other SAP customers?
According the data released by the City, it's Chief Information Officer made over 150,000 for FY 2010. That's a lot of money, even for an "underpaid" City employee. In the private sector, people would actually put in more than the nominal 40 hours a week.
Given that the City of Palo Alto only requires its employees to put in nine days out of every ten, and there is little evidence that all the employees actually show up when they are supposed to, and they stay until they are supposed to, and that they actually work as hard as people in the private sector do, or face the same consequences if they fail to get their projects done on time .. so most of us residents/taxpayers sort of wonder what the Chief Information Officer does, and why he isn't responsible for information security, like his counterpart in the private sector would likely be?
Why couldn't the CIO put in a few extra hours at night, or on the weekends, to do some of the work of the "missing bench"? Is it that hard to expect a little "extra" from the CIO for that kind of money?
Adobe-Meadow
on Feb 4, 2013 at 9:00 am
on Feb 4, 2013 at 9:00 am
All this means is that the External SAP audit revealed that the delivered user ID for super user administration had been left with a default password that is built into SAP for installation and administration purposes. This happens often in companies if the id deleted. This special id has to be locked, and authorizations removed to secure this issue. There is also a system parameter that can be set to protect from this vulnerablitiy but often administrators forget to set it back after a change. When this id is available you have ALL access to the companies business processes and are god in the enterprise. I can't believe this is being treated like a breach. I have seen this id available with the default password in many places, but rarely in a production environment. It is often a problem in a non-production environment. The non-production environment might be a copy of production which would then make it just as bad of a breach when it comes to private data. There are other safeguards in place however such as email alerts to various IT management staff if this super ID is ever used. Security this standard default admin user id and password is the first thing on any security administrator checklist so it really does look bad when an external auditor finds the default super user wide open for anyone who knows the well publicized default password.