Month: March 2017

Catastrophic Data Storage Failure in Australia – Could You Be Next?

By Scram Software

“. .this is the first time this problem has been encountered anywhere in the world,” said acting CIO Steve Hamilton of the Australian Tax office (ATO). Except that maybe it has happened before. It’s a reasonable assumption that he was merely repeating a line that was fed to him by somebody he believed. I wonder who that could be?

The ATO transferred their data storage capability from end-of-life EMC/HPE equipment to a new HPE 3PAR SAN (Storage Area Network) ‘as-a-service’ model in Nov 2015. It failed “catastrophically” just over a year later. The ATO lost 1 Petabyte of data because the automatic failover to the second SAN did not come online. Corrupted storage blocks on the main SAN had been faithfully copied to the second SAN.

The ATO does have another backup source, so the data loss is not total or permanent. It knocked out the operation of a large portion of a nation state’s government department for two days. Undoubtedly there will be financial repercussions for HPE and it’s another major blow to the Australian government’s reputation for technical capability in a short period of time after the recent census disaster.

Who are HPE and what is 3PAR?

Hewlett Packard Enterprise invites large enterprises to outsource: “We deliver high-quality, high-value products, consulting, and support services in a single package. That’s one of our principal differentiators.” HPE and Dell fought a bidding war to acquire storage systems supplier 3PAR in 2010. HPE won and paid $2.35 billion. 3PAR SAN systems utilize solid state flash storage that takes advantage of virtualization and cloud resources to promise faster processing speeds.

What went wrong?

The backup design seemingly allowed undetected corrupted data storage blocks to be duplicated to the second SAN, which may indicate a lack of data integrity checking. The root cause analysis of high profile incidents like this is rarely made public. The culprit could be a defective firmware upgrade or simple human error. We will probably never know. The symptom has surfaced previously with 3PAR SAN solutions, like this incident two months earlier. Anecdotal evidence seems to indicate other similar occurrences but IT failures at regular commercial enterprises rarely make it into the headlines.

What could have been done to mitigate the extent of the impact?

From what we know, the design revolved around a single data domain. That appears to represent a single point of failure no matter how much redundancy is built in.

Who is the usual victim of incidents like this?

Large enterprises up to government level are key clients to vendors such as HPE. Outsourcing deals of this nature are big budget projects. The ATO installation was part of a $92 million (AUD 1.29 billion) IT investment, to put it in perspective. Finger pointing inevitably occurs but the client typically puts its faith in the perceived capability and reliability of the vendor for technical design and support of a fit-for-purpose delivery.

How can this scenario be avoided?

It all boils down to the robustness of the design and whether or not the client is willing to spend sufficient budget for the safest possible option. That is not to criticize the ATO. The optimum solution could involve multiple vendors and come in at a cost that any financial controller would blanch at. As always, the delivered solution is a compromise between suitability and affordability. The ATO incident generated interesting technical debate on forums such as Whirlpool that sheds some light on SAN design and similar incidents.

Whenever an organization outsources an operational function, it places its reputation in the hands of the supplier. The bigger the organization, the harder the fall if things go wrong. And it does not get much bigger than a government’s reputation and the consequent slap-in-the face to the politicians in charge. The media exposure and feeding frenzy guarantees a major hit to the supplier also. For the client, the old tried and trusted avenues of due diligence and assigning qualified experts to perform rigorous scrutiny of the vendor’s proposed solution remain the best defense.

Are the Chinese Reading your Texts Right This Minute?

By Scram Software

About 700 million Android phones were sending the contents to a server address in China every 72 hours. The owners of the phones had no idea it was happening. The data known to be extracted included:

  • Location information (where you were, at all times)
  • Call logs (who you spoke with)
  • Text messages (including deleted messages)
  • Contact lists

The backdoor was in the firmware in a chip of a component part that was supplied to a large number of Asian phone manufacturers and at least one American one. It was only discovered when a security analyst bought a $50 infected phone for testing and noticed an unusually high level of network traffic when he powered it up.

Was this a one-off occurrence?

Roughly 5 to 6 new Android phones are released by manufacturers on the worldwide market every day on average. The greatest growth surge is in the Asian market where there is a proliferation of extremely cheap devices. Intense price competition means that manufacturing costs must be kept low, which invites supply of the cheapest components whose cost may be secretly subsidized by an interested party. The company involved, Shanghai AdUps Technology, supplies software to phone component manufacturers that can also remotely install apps on a smartphone and update them on demand.

Just how private is my phone’s data?

Forget any notion that a simple passcode keeps your phone data confidential. There are companies, many of them, who specialize in developing and selling equipment that can crack most any phone’s security and suck out the data contents in seconds. Those companies sell that hardware to government security agencies like the FBI and NSA both domestic and foreign, police forces, corporate clients, and most anybody who can pay the price. They may present a veneer of ethics by checking the credentials of potential clients but that is a flimsy defense against allowing the equipment to fall into the “wrong hands”. That is in addition to the shadier activities of software and component suppliers like the China company.

Why would they want my information?

A state actor would have zero interest in the phone data of the average person – only of individuals with access to facilities, organizations or activities (including criminal) of interest to them. The biggest usage by far is in the field of Big Data that is used for marketing or product development purposes by manufacturers and software system vendors. That is the purpose that the Chinese company said was behind their theft of phone data. But don’t for one moment think that a government agency would not suborn that data if the need arose.

How can it be justified?

All nation states have their own specific security interests and prioritized list of targets. They can use the excuse of national security, counter-terrorism, policing criminal activity, or any other rationale they choose. History is littered with examples of government agencies acting outside the law in most any country you choose to name. The Internet enables state actors to easily access any other state, attempt to hack into its agencies, government departments, defense contractors, banks, and even the general population as evidenced by the China mass phone hack.

Why is it so important?

Some people naturally think “what the heck – I don’t care because nobody would be interested in my boring old texts etc.” But that misses the point. Privacy is as vital a concept as freedom of speech or the right to vote. Just because you don’t bother to vote does not mean you don’t care about that freedom or right. Erosion of personal rights is a characteristic of despotic states. Any weakening of those rights and freedoms is a move in that direction.