Audit finds 'breach' in city's business operation Crimes & Incidents, posted by Editor, Palo Alto Online, on Oct 14, 2011 at 10:23 am
The City of Palo Alto's business operation had a significant security breach that left sensitive employee and customer information open to outside access, an investigation by the City Auditor's Office found.
Read the full story here Web Link posted Friday, October 14, 2011, 9:53 AM
Posted by who knows, a resident of the Charleston Gardens neighborhood, on Oct 14, 2011 at 11:41 am
how can something like that happen with an award winning finance department at the most forward-thinking city along the peninsula?
If the city was not aware of the problem, how would they know if the personal/sensitive information leaked? Could employees who used to work in that area (or HR/Utilities) still have access and take the info internally and sell it?????
Posted by Barron Park, a resident of the Barron Park neighborhood, on Oct 14, 2011 at 11:45 am
The article effectively indicates that there wasn't a release of information, much less a hacking, that is known at this time. If there was a release/hacking known, I am certain that it would have been in the auditor's report.
The report appears to be about serious procedural error, a "backdoor" to the data and to payment authorizations that was mistakenly left "open" after the ERP implementation. A grave error, without doubt. But (as best can be told from the article), not resulting in data release nor hacking.
We can't tell from this if there is a systems audit trail that could indicate if an data release or payment authorization wrongly occurred. I am sure that was asked and a review of the auditor's report would likely make this clear.
Posted by PA-Needs-Security-Audits, a resident of Another Palo Alto neighborhood, on Oct 14, 2011 at 12:39 pm
> If there was a release/hacking known, I am certain that it
> would have been in the auditor's report.
That depends if there were any record of the illegal access to the system, and then evidence of that illegal access made public in some way, such as posting the data on a web-site, or sending it to the media.
The whole area of computer security is poorly understood, and it’s almost impossible to make any cogent statements about the security of a given computer system (or software package) without having a lot of access to the operating system’s, or application package’s, source code. This access would then need a full code review, and then a lot of testing, to determine just how secure the system is to illegal penetration attempts.
The problem becomes difficult to analyze when a default password is not changed, and someone attempting illegal entry tries the known default passwords. If the accesses are logged, then the question as to how much information is logged becomes central to being able to detect an illegal access. Then there is the issue of data access. Security-oriented operating systems, and software, might log all access to data. Historically, security has never been particularly important to software developers, and too often seen as more of a hindrance, than a necessity. So, without a lot of special knowledge about both the Operating System (OS) and the application software (SAP, in this case), claiming that no one had gained access to the system, or no information had been removed, would be hard to know.
Every organization needs to encrypt all of its crucial data, and to add penetration detection software. The City of Palo Alto is long overdue for these sorts of audits, and probably has few people on staff that knows what to do to make these systems more secure.
Posted by pat, a resident of the Midtown neighborhood, on Oct 14, 2011 at 5:51 pm
How do these guys keep their jobs?
This is the same IT department that threw away $250,000 on a “new & improved” city website. It also spent $8.8 million for online utilities billing, which “included such pesky glitches as confusing computer-screen displays and bills that don't add up.” Council was then asked for an additional $223,725 to resolve a list of "post-implementation issues."
Posted by Terry, a resident of the Midtown neighborhood, on Oct 14, 2011 at 8:57 pm
OK, let’s assume it is unlikely any critical information was truly compromised. I’m a retired IT exec and have consistently found Palo Alto IT staff to be remarkably incompetent.
I offer the following analogy: A police office addresses a school class for a community services presentation. In the process, he chooses to display his firearm and accidentally discharges a shot into the wall. Well, you can say no one was hurt, but the act itself was felony stupid. So is the conduct of the Palo Alto IT department… felony stupid.
Whoever failed to secure the account and his manager should be fired immediately
Posted by Outside Observer, a resident of another community, on Oct 14, 2011 at 10:30 pm
Palo Alto IT has suffered 4 changes in leadership in the past several years, and is about to suffer another soon.
When these changes happen, all the competent people move on to greener pastures, and only the dregs remain. As it is now, Palo Alto IT is staffed mostly with people who couldn't find IT jobs during the Dot Com boom.
The next change of guard will cause another such purge, but the tech sector is again doing well in the valley. Given that, plus public hostility towards government workers, an insufferable work environment within the City, low wages vs private industry, and that once hired in government, you are forever "branded" and will never be employed in the private sector again.... Given all of that, don't expect any improvements.
Posted by Retired Staffer, a resident of another community, on Oct 15, 2011 at 8:50 pm
The fault rests with SAP, not the City. Staff reductions imposed by Council and upper management have thinned the ranks so dramatically that "backchecking" the work of a vendor is improbable if not impossible. The IT staff that's left is perfectly competent to "backcheck", but it doesn't have the time. This is just the tip of the iceberg. More instances of these lapses will occur because there are too many unfilled positions.
Posted by PA-Needs-Security-Audits, a resident of Another Palo Alto neighborhood, on Oct 16, 2011 at 8:28 am
> The fault rests with SAP, not the City.
And so if you get into a collision with your car, the fault lies with the manufacturer?
Responsibility for any organization lies at the top, and should be delegated down the chain-of-command to appropriate management levels. Unfortunately, local governments have never been particularly well managed, and so this idea of "responsibility" does not seem to be well defined in most city-level governments. That's most certainly true in Palo Alto.
While computer security has never been all that well understood, there are consultants that have emerged, over the years, that can be engaged to do period audits of most vendors’ systems. These consultants can do the work of auditing a system, or organization’s, security, or they can develop a security model, leaving the local staff to do the work every periodically. Creating a checklist of security “weak points” that requires staff to check various passwords to insure that they are not “defaults”, or that they are long enough to make penetration difficult, is not that difficult.
But.. someone has to recognize that this must be done, and that these periodic audits, and follow-ups, are done falls to the top level of management. In the private sector, firing people for failing to do the work is an option. In the public sector, it seems that “employee rights” trumps the obligation of the local government agency to protect data—so it’s hard to believe that anyone involved in any kind of incompetence, short of actually breaking state law, would ever “get the ax” for failures like this one. To make matters worse, what oversight that exists—via the City Council—does not seem to have much power over the City Manager, other than to dismiss at will. Moreover, Council’s are not chosen for their technology skills. They seem to be chosen by special interests groups to make certain that the City money is spent on them. So, having the Council even understand that “computer security” is important is something that may, or may not, happen.
By way of example, some computer systems keep remind people to change their passwords every so often (like ninety days, or so). The login code then badgers people to change their password once that time has expired, until they do. Site Administrators might even visit those who have not changed their passwords to remind them that they are not complying with the efforts of the organization to keep the computer system secure. When people resign, or are terminated, from such organizations, it’s not uncommon to require people to change their passwords immediately. This is a lot of work on everyone’s part to insure the integrity of the intellectual property stored on an organization’s computers. Unfortunately, we don’t seem to be seeing that sort of commitment on the part of the Palo Alto City Government management
Posted by Retired Staffer, a resident of another community, on Oct 16, 2011 at 2:38 pm
1. A collision may indeed be the fault of the manufacturer.
2. Couldn't agree more with the need for security. Measures used to be in place. But the City eliminated so many positions that the work can't get done anymore. The coach can't win with an empty bench.
Posted by PA-Needs-Security-Audits, a resident of Another Palo Alto neighborhood, on Oct 16, 2011 at 8:02 pm
> A collision may indeed be the fault of the manufacturer.
Yes, that's true .. but when this happens, it happens to a lot of people about the same time .. and all of the country's ambulance chasing lawyers show up and make a lot of money out of the failure of the manufacturer. How many similar situations have occurred with other SAP customers?
According the data released by the City, it's Chief Information Officer made over 150,000 for FY 2010. That's a lot of money, even for an "underpaid" City employee. In the private sector, people would actually put in more than the nominal 40 hours a week.
Given that the City of Palo Alto only requires its employees to put in nine days out of every ten, and there is little evidence that all the employees actually show up when they are supposed to, and they stay until they are supposed to, and that they actually work as hard as people in the private sector do, or face the same consequences if they fail to get their projects done on time .. so most of us residents/taxpayers sort of wonder what the Chief Information Officer does, and why he isn't responsible for information security, like his counterpart in the private sector would likely be?
Why couldn't the CIO put in a few extra hours at night, or on the weekends, to do some of the work of the "missing bench"? Is it that hard to expect a little "extra" from the CIO for that kind of money?
Posted by JustSayin, a resident of the Adobe-Meadows neighborhood, on Feb 4, 2013 at 9:00 am
All this means is that the External SAP audit revealed that the delivered user ID for super user administration had been left with a default password that is built into SAP for installation and administration purposes. This happens often in companies if the id deleted. This special id has to be locked, and authorizations removed to secure this issue. There is also a system parameter that can be set to protect from this vulnerablitiy but often administrators forget to set it back after a change. When this id is available you have ALL access to the companies business processes and are god in the enterprise. I can't believe this is being treated like a breach. I have seen this id available with the default password in many places, but rarely in a production environment. It is often a problem in a non-production environment. The non-production environment might be a copy of production which would then make it just as bad of a breach when it comes to private data. There are other safeguards in place however such as email alerts to various IT management staff if this super ID is ever used. Security this standard default admin user id and password is the first thing on any security administrator checklist so it really does look bad when an external auditor finds the default super user wide open for anyone who knows the well publicized default password.