Chapter 6

The conduct of the 2016 census

6.1        The reference date for the 2016 census was 9 August. On this date, millions of Australian households paused to complete their census forms. Unfortunately, for many people, the evening was one of frustration as the website and phone systems put in place failed.

6.2        This chapter begins with a summary timeline of key events between 9 and 11 August 2016, before going on to discuss the failure of the eCensus website, problems with the telephone service, and the subsequent actions and conduct of census field officers.

Failure of the eCensus

6.3        This section provides a timeline of the events surrounding the failure of the eCensus website, and then considers the causes and response to this event.

9 August

6.4        During the morning of 9 August, the ABS experienced two Distributed Denial of Service (DDoS) attacks resulting in very short outages.[1]  During the second attack the 'Island Australia' protocol—IBM's chosen DDoS defence—was enabled. This measure successfully stopped the attack and 'the ABS was of the understanding that the online form would now be protected from any further attacks'.[2] A third attack in the afternoon was repelled without any loss of service.

6.5        At approximately 7.00 pm, the eCensus website had already successfully processed over 1.8 million household forms and was running at 7 167 forms per minute, with demand growing in accordance with ABS modelling.[3] Commencing at 7.28 pm a fourth DDoS attack occurred resulting in the website being unavailable from 7.33 pm.[4]

6.6        At the time of the attack, the ABS and IBM observed a spike in outbound traffic in the IBM monitoring systems, prompting concerns that the system may have been compromised and experiencing data leakage.[5] IBM reported:

[The] 7.27 pm DDoS attack also caused one of the mechanisms used by IBM to monitor the performance of the eCensus site to miscarry. As a result, some IBM employees who were observing the monitor mistakenly formed the view that there was a risk that data was being exfiltrated from the website and that the risk needed to be further investigated. Out of an abundance of caution, IBM shut down access to the site and assessed the situation.[6]

6.7        IBM attempted to reboot its system at 7.43 pm but was unable to restore services.[7] The committee heard that IBM had incorrectly configured at least one of the two routers it had in place.[8] When the routers were restarted the link to the Telstra network did not load its previous configuration; this left only the third party provider, Nextgen's link in place, and that link was consumed by DDoS traffic.[9]

6.8        At the request of the ABS, IBM enabled 'overload' controls at 8.09 pm which prevented Australian households from commencing new census forms.[10]

6.9        The Minister for Small Business, the Hon. Michael McCormack MP, was provided an initial briefing by the Australian Statistician at 8.26 pm.[11] At 8:38pm the ABS published a message through social media channels informing people that the census website was experiencing an outage, and at 8.50 pm, the ABS requested Dentsu Mitchell—a consultancy providing advertising for the census—cease all social and digital advertising.[12] The ABS was informed that advertising would cease by approximately 9. 40pm.[13]

6.10      IBM was able to successfully restore the online census system at 10.26 pm. At this point the ABS stated that it would have been possible to make the eCensus available to the public again. However, the ABS elected to keep the eCensus system closed until: it had understood the unanticipated spike in outbound traffic earlier in the evening; was sure it could defend against future DDoS attacks; and was sure that the infrastructure was robust including the routers which had experienced issues.[14]

6.11      At 10.59 pm, the ABS posted updates to social media stating that the eCensus service would be unavailable for the remainder of the evening, and that an update would be posted in the morning.

10 August

6.12      It was reported to the committee that in the early hours of the morning, 'the ABS and IBM conclusively determined that both the outage was caused by an overseas-based DDoS attack and that no data was lost'.[15]

6.13      The ABS published a statement on 10 August explaining what had occurred the previous day:

The 2016 online Census form was subject to four Denial of Service attacks yesterday of varying nature and severity.

The first three caused minor disruption but more than 2 [million] forms were successfully submitted and safely stored.

After the fourth attack, just after 7.30 pm, the ABS took the precaution of closing down the system to ensure the integrity of the data.[16]

11 August

6.14      On the morning of 11 August, the Australian Privacy Commissioner announced that he was 'satisfied that personal information was not inappropriately accessed, lost or mishandled'.[17]

6.15      At 1.16 pm, the Australian Signals Directorate (ASD) notified the ABS that IBM 'had taken all steps that could reasonably be taken in the time available to mitigate denial of service attacks similar to those that occurred on 9 August'.[18]

6.16      At 2.29 pm on 11 August, the system was reopened to the public.

What was the cause of the outage

6.17      The cause of the suspension of the eCensus website on 9 August was the ABS taking the deliberate decision to prevent households from logging onto the website. This decision was precipitated by a DDoS attack that was not adequately protected against, and was of such a small size that it should have easily been handled effectively. As explained by the Special Adviser to the Prime Minister on Cyber Security, Mr Alastair MacGibbon:

There were indeed attacks, they should have been expected, they were expected, protection against them was contracted for—and these were definitely small attacks and they should not have degraded the ABS system. But it was not the denial-of-service attacks that actually eventually took the ABS e-census website offline. That was a decision made by the ABS, and there were two other critical components in there. The denial-of-service attacks that degraded the system, the attempts by IBM in turning the routers off to re-communicate with their data centre—finding that they had misconfigured the router at the Telstra end of the link—and then, in a sense, the final straw that broke the camel's back was the misinterpretation of data on a load-monitoring system, which was interpreted at first as possibly the exfiltration of data—or, an actual hack, as opposed to an attack.[19]

6.18      The influx of traffic to the IBM routers appear to have caused a malfunction within the internal monitoring of the IBM system:

During the fourth attack, a system monitoring dashboard, which included a graph of inbound and outbound [Internet Service Provider] traffic to the eCensus site, showed what appeared to be a spike in outbound traffic. This caused some IBM employees to, it is now accepted mistakenly, form the view that there was a risk of data egress from the eCensus site. In fact there was no data egress and the spike was a 'false positive'.[20]

6.19      Due to fear that a recorded spike in outbound data traffic from the eCensus system represented a potential data leak, the ABS decided to temporarily close access to the website.[21]

What is a DDoS

6.20      IBM provided the committee with an explanation of how DDoS attacks operate:

A denial of service attack is a malicious attempt to make a system unavailable to its intended audience by overloading servers with requests to render it unavailable or causing it to shut down.

A denial of service attack is typically accomplished by flooding the targeted machine or resource with superfluous requests in an attempt to overload systems and prevent some or all legitimate requests from being fulfilled.

A 'distributed' denial of service or DDoS attack occurs where the attack source has multiple unique IP addresses or 'nodes'. It is typically achieved by using a 'botnet', being a group of internet-connected devices on which malware has been installed so as to enable the devices to be controlled from a remote location without the knowledge of the devices’ owners. A DDoS attack may be regarded as analogous to a group of people crowding the entry door to a business and not allowing legitimate customers to enter, thus disrupting the business’ normal operations.

The effects of denial of service attacks include slower network performance (opening files or accessing websites), or the unavailability of a particular website, or an inability to access any website. Denial of service attacks are therefore directed towards the performance or availability of a website.[22]

6.21      DDoS attacks are neither new nor unusual. The committee heard that it is commonplace for government websites to regularly experience denial of service attacks which are routinely managed without interruption to services.[23]

6.22      It is unclear if the eCensus website experienced further DDoS attack attempts following restoration of the system. IBM informed the committee that there were further DDoS attacks which were successfully defended.[24] In contrast, the ABS claimed to be unaware of any further attacks.[25] The perpetrators of the DDoS attack remain unknown.[26]

What plans were put in place to prevent DDoS

6.23      The ABS was aware of the threats posed by a prospective DDoS and had contracted for measures to be put in place to mitigate them. The committee heard that DDoS events are routine occurrences, and are also routinely managed without incident:

Denial-of-service attacks, or distributed denial of service attacks, are eminently predictable and should be expected. In fact, the Australian Bureau of Statistics did call for denial-of-service protection in its tender process with IBM, and IBM responded to say that they would put in place denial-of-service protection. So yes, it is expected, and it should be dealt with.

...

There were indeed attacks, they should have been expected, they were expected, protection against them was contracted for—and these were definitely small attacks and they should not have degraded the ABS system.[27]

6.24      The risks of DDoS attacks were included in the risk management plan for the 2016 census, with the ABS reporting:

In the final risk management plan (July 2016), one of the risks was 'Loss of system availability through a Distributed Denial of Service Attack'. This risk had pre-mitigated exposure rating of 'high' and a residual exposure of 'medium'. Under the plan, IBM was responsible for mitigating this risk, with ISP measures of Island Australia (geoblocking international traffic) a key measure.[28]

6.25      IBM states that it met its requirement to provide DDoS protection through the 'Island Australia' policy.[29] IBM explained how this mitigation strategy would work:

The protocol was an ISP-based DDoS attack mitigation strategy which required the ISPs who provide access to the eCensus site to block or divert all international traffic to the site at the direction of IBM. Such blocking is known as 'geo-blocking'. The protocol was to be deployed in the event that a DDoS attack occurred.

The Island Australia protocol, while not a form of protection that is appropriate for websites with users who are widely distributed, was well-adapted to the 2016 eCensus because it took advantage of the fact that the Census form was required to be completed (during the Census period) only by persons who, on the Census Day, were physically present in Australia. Accordingly, with some exceptions, legitimate traffic to the eCensus site could be expected to be domestic to Australia.[30]

6.26      IBM tested the Island Australia protocol on 5 August 2016 and reported to the ABS that it had 'worked exactly as expected'.[31] IBM reported that the success of this test gave them a high level of confidence that appropriate DDoS mitigation measures were in place:

IBM considers that, following the testing on 5 August 2016, it had every reason to think that Island Australia would provide effective protection against DDoS attacks if needed.[32]

6.27      While preparing for the eCensus, IBM 'considered other possible options for the defence of DDoS attacks'.[33] IBM examined other products offered for DDoS protection yet concluded that these services would not be suitable for the 2016 eCensus 'because of the unique traffic profile it was expected to generate'.[34]

6.28      In July 2016, the ABS and IBM received a briefing from ASD on cyber threats and incident response support. The ABS recalls:

The potential for DDoS attacks was discussed, as were general mitigations for a range of threats. ABS does not believe that any new areas of concern were raised, nor were there any suggestions of potential mitigations or additional preparations that were not pursued.[35]

The failure of Island Australia

6.29      IBM anticipated, and assured the ABS, that Island Australia would prevent international traffic from reaching the eCensus website, thereby ensuring that international traffic could not be used for a DDoS attack.

6.30      The internet is a network of networks, and in order to allow public access to a webpage, a company like IBM must pair with internet service providers (ISPs) to link the IBM network to the rest of the internet. Public access to the eCensus site was provided via two ISP links, one provided by Telstra Limited (Telstra) and the other by Nextgen Networks Pty Ltd (Nextgen).[36] If a serious DDoS attack occurred during the census collection period, IBM would direct Nextgen and Telstra to put in place Island Australia.[37] These ISPs would prevent the malicious traffic from reaching the IBM network processing the census data.

6.31      When a household attempted to fill in the eCensus the message needs to move from their home to the final destination; which in this case is IBM. In order for this to be possible, there needs to be a path along the networks of the internet from an origin and the destination. Suppose Alice wants to complete her eCensus. Alice's personal computer (A) would speak to her router (B), which would send a message to a router (C) operated by her internet service provider. That router (C) would try and pass the message to a router (D) owned by—in this case—either Telstra or Nextgen which have a direct link to the destination—IBM. If no such router can be found, it (C) would pass the message onto another network's router (E) which would attempt to pass the message onto a router (D) on the desired network. Once the message finds the correct network, the message will be delivered to IBM.

6.32      Under the Island Australia protocol, if any of the routers belonging to Telstra or Nextgen received a message that was addressed to the eCensus and sent from an international router (ISP), that message would not be processed.

6.33      Commencing at 7.28 pm, a DDoS attack began eroding the capacity of the system. The size of the attack was estimated to be around 1.5 gigabits per second.[38] The attack reportedly had the effect of commencing new sessions which quickly exhausted the memory capacity of the IBM's router facing the Nextgen link.[39] Mr MacGibbon noted that this attack was a '...weapon but a small one and one that we should have had protections against, absolutely' and '[it] certainly should not have caused the damage that it did'.[40]

6.34      IBM alleges in its submission that Nextgen failed to properly implement the Island Australia protocol which allowed traffic to flood the IBM system:

IBM was informed–later that day after the attack had passed–that a Singapore link operated by one of Nextgen's upstream suppliers (Vocus Communications or Vocus) had not been closed off and this was the route through which the attack traffic had entered the Nextgen link to the eCensus site.[41]

6.35      IBM contends that had Nextgen properly implemented Island Australia the eCensus website may not have become unavailable:

Had Nextgen (and through it Vocus) properly implemented Island Australia, it would have been effective to prevent this DDoS attack and the effects it had on the eCensus site. As a result, the eCensus site would not have become unavailable to the public during the peak period on 9 August 2016.[42]

6.36      Nextgen disputed this allegation, noting that they had implemented the same settings as IBM had successfully tested on 5 August 2016, and that their analysis demonstrated that both the Telstra and Nextgen links showed DDoS traffic during the fourth attack.[43]

6.37      Vocus Communications[44] (Vocus) informed the committee that it recorded a peak traffic load through the Singapore link of 563Mbps. Vocus' submission states that this 'is not considered significant in the industry' and 'not of a size to cause the census website to become unresponsive, had appropriate network security measures been implemented by IBM'.[45]

6.38      As previously discussed, IBM had two routers in place to facilitate Australians filling in their eCensus forms. Each router alone had sufficient capacity to transfer all of the anticipated legitimate census traffic. Representatives of the ABS, in a foray into biology, explained:

At this point we knew we could operate with one router. We knew the system was designed to have sufficient capacity...I would say it is a bit like functioning on one kidney, but you do not really want to when you have two.[46]

6.39      Even though IBM reports that all of the DDoS traffic was coming through the Nextgen link, both routers appeared to be experiencing an unusually heavy load.

6.40      This heavy load adversely affected IBM's internal system monitoring the flows of data in and out of the network, eventually resulting in erroneous telemetry from automated monitoring systems that would lead to the eCensus website being disabled. As IBM explains:

IBM’s investigations have revealed that the false positive reading occurred because the system was programmed to measure and report the traffic volume of the eCensus site at 60 second intervals. Once the fourth DDoS attack was underway, the information was being reported for varying intervals but the dashboard was treating the information as though it had applied the standard 60 second interval. This resulted in an incorrect graphic creating the impression that there had been a spike of outbound traffic that could be data egress.[47]

6.41      This spike was what led IBM and the ABS to fear census data was being exfiltrated from the system. It has been suggested that the elevated data flow was IBM's computer system reporting performance information and logs to an offshore data centre.[48]

6.42      The DDoS attack had the effect of consuming the memory of the routers meaning they were unable to properly process requests. In order to restore the system to functionality, IBM reset the routers. When they did this, the router on the Telstra link did not restore its settings, meaning that the only link to the eCensus website was the Nextgen link which was subject to the DDoS attack.[49] Mr MacGibbon explained the problem on IBM's Telstra router:

I think the biggest issue was that the router that was at the Telstra link was incorrectly coded by IBM, so when it was turned off the coding fell out, for want of a better description, from that router and it made it a 'dumb unit'. The time it then took to get it back up to scratch is where the confusion happened.[50]

6.43      IBM reports that it took one hour and twenty minutes to restore the Telstra link. Once the Telstra link was restored IBM reports closing the Nextgen link. IBM states that at this point there was an immediate drop in attack traffic and a resumption of normal application behaviour.[51] IBM reports that it was ready to restore the eCensus at 10.32 pm, approximately three hours after the site had become unresponsive.[52]

6.44      In summary, a DDoS attack adversely affected the operation of the routers' reporting system. When this reporting system displayed several minutes' of data at once—rather than the typical 60 second increment—it ignited fears that the system had been compromised. Further, IBM were unable to effectively reset their router to the Telstra network due to configuration errors. These two events, precipitated by the DDoS, led the ABS to take the decision to take the eCensus website offline. The combination of the three events ultimately led to the eCensus being unavailable. Mr MacGibbon observed:

Anyone of those three things not existing—if there were no denial-of-service attacks or if they were properly mitigated—we would not have had the other two problems. If we had had the unmitigated denial-of-service attacks but the router had functioned properly, we would not have had the third problem, and we would not have had the third problem if the people that were monitoring the system properly interpreted the system, which was functioning oddly given the nature of the stresses that it was under, based on the first two points. So any one of those in isolation was not really sufficient to lead us to this committee today, some months later. It was the three combined that led us here.[53]

6.45      The natural question this raises is was the preparation and protection by IBM and ABS sufficient?

Were the protections put in place by IBM sufficient?

6.46      The events of the evening of 9 August would imply that mistakes were made in the preparation and execution of the DDoS defence. The IBM submission argues that Island Australia was approved by the ABS as a means of defending against DDoS attacks.[54] It appears that IBM and the ABS were in agreement that any botnet[55] in Australia was of insufficient size to cause serious damage to the eCensus website, and therefore geoblocking would be sufficient.[56]

6.47      However, Island Australia geoblocking was not the only option available to IBM in meeting its contractual obligations in ensuring resilience in the face of DDoS attacks. This report has already canvassed why Island Australia was thought by IBM to be sufficient.

6.48      It was put to the committee that Nextgen had raised concerns with IBM that the Internet Protocol (IP) ranges that IBM planned to block were not exhaustive, and that Island Australia may not adequately protect the eCensus.[57] 

6.49      The committee heard that Nextgen offered IBM DDoS protection which was rejected by IBM.[58] IBM claimed that the Nextgen solution was not fit for purpose:

We looked at and considered the DDoS protection that was being offered by Nextgen and, on the basis of the information that they provided, there were three distinct attributes that we felt rendered it unsuitable. The first concern was that Nextgen advised us that it required a four-week training period to learn the particular traffic patterns that would be coming into the site. We did not have a four-week lead-in period during which to learn the traffic patterns. And the traffic pattern in the weeks leading up to census night is not at all representative of what we experience on the night.

The second concern, which we discussed with Nextgen in advance of the RFT response being submitted in 2014, was the ability of their solution to deal with that very high peak we experience on census night and whether it in fact might be interpreted as a DDoS attack, and that they share that concern with us. The third was one related to the specifics of the application and the way we were doing load balancing to distribute the load across multiple back-end process streams. There was a concern early on in the process that the Nextgen protection approach might interfere with that load-balancing mechanism. So for those three reasons we felt that the geoblocking was a very well adapted solution for the particular characteristics of the traffic, and it is one which we had experience of in 2011, with both Telstra and Optus, that they could very effectively and easily implement, so we chose that as our preferred strategy.[59]

6.50      Evidence received by the committee indicates that the Nextgen solution was adopted after the resumption of the eCensus, casting doubt on explanations of why it would have been unsuitable before 9 August made by IBM.[60] IBM claims that the adoption of the Nextgen DDoS products following the resumption of the eCensus was in response to a changed threat environment where it had become public knowledge how the eCensus was being defended.[61]

6.51      The choice of geoblocking as the sole means of DDoS protection is interesting given a number of technical considerations related to the eCensus:

There were some technical problems in that some Australians, with Australian-based ISPs, will also route in from overseas, just by the nature of the ways in which those ISPs operate. In fact, the password reset facility that IBM used actually relied on traffic coming in from overseas to give Australians that password. So there was a fundamental failure in the logic of an Island Australia. I could see it as part of a series of protections adding some value, but to rely solely on it clearly was a failure.[62]

6.52      Given that the eCensus system itself relied on communications from outside Australia, the proposal to protect the application by blocking this traffic appears curious.

6.53      It was pointed out to the committee that there was sufficient redundancy built into the system by having two ISP links. Had the Telstra router been properly configured such that when it was restarted it worked properly, the problems experienced on 9 August would likely have been avoided.[63]

6.54      IBM admitted that, in hindsight, further testing of the router would have uncovered the configuration error before it become an issue on census night:

But we tested that router failure by simulating it, which is relatively easy to do in a repeatable fashion. If we had our time again, we would probably do a hard, powered-off powered-on test of that router. That would have discovered earlier that we had that reboot and configuration loading problem.[64]

6.55      There appeared to be some confusion on census night regarding what had actually happened with the router, with the ABS being initially under the impression that the router on IBM's Telstra link had suffered from a hardware failure.[65]

6.56      Mr MacGibbon highlighted that the greatest failure on the part of IBM was that they did not check that Island Australia had been properly implemented by its subcontractors.[66] IBM informed the committee that, with hindsight, they would have sought greater certainty that their geoblocking protocols had been correctly implemented by their contractors.[67]

6.57      IBM claims that they assumed that all of their contractors had the ability to implement their instructions:

We as the prime contractor dealt with both Telstra and Nextgen as our ISPs and expected them, as large internet service providers, to be able to implement those instructions correctly.[68]

6.58      The committee is not in a position to determine the relative truth of where any fault lies between IBM and its contractors. Mr MacGibbon explained the situation between the parties nicely:

The Commonwealth, in this instance, was the customer. The customer went to a builder to build a house. There were blueprints put before them as to what that house would look like, and the Commonwealth, as the customer, paid money to the builder to build the house. The builder is now in dispute with a plumber and a bricklayer about what was or was not done in relation to the deficient house. IBM, of course, being the builder and Vocus and Nextgen, when you look at their responses, being the plumber and the bricklayer...I cannot determine who is right and who is wrong. What I will say is that as the customer the Commonwealth was not well served.[69]

6.59      Extending on this example, the taxpayer has bought a house off the plan with a sinking foundation and cracks in the walls: they can feel rightly aggrieved.

Did the ABS have too much trust in IBM?

6.60      The ABS had contracted IBM to provide DDoS prevention measures, and IBM assured the ABS that this was done.[70] The ABS can rightly say that they expected the eCensus to be secure and stable in the face of threats:

Senator HUME: Were you surprised by the extent of the disruption caused by the DDoS events considering on a relative basis they did not seem to be particularly large?

Mr Kalisch: We were certainly surprised that the system was vulnerable.

Senator HUME: And you were assured, I assume, beforehand; otherwise, you were assured that it was invulnerable?

Mr Kalisch: We were assured that that system was robust and was ready to go to a range of different attacks and mechanisms, not just DDoS.[71]

6.61      It was suggested to the committee that the ABS showed too much trust in IBM and was not sufficiently proactive in ensuring that IBM was meeting their contractual obligations:

In many respects, while I will say to you that this was a failure to deliver on the contractual obligations that IBM had, there was a failure on the part of the ABS to sufficiently check that the contract had been delivered. That could have been achieved through more thorough assessments of the work done for them by IBM and their subcontractors.[72]

6.62      Mr MacGibbon suggested further that the ABS could have been more proactive in overseeing the implementation of the eCensus project to ensure that all contractual undertakings were being fulfilled:

If I understand your question correctly, if they had engaged IBM, could they have verified that they were building the system that they were contracted to do? Yes, they could have. They could have had more third-party testing done. They may have asked more questions of IBM to provide proof that they were delivering the services they were contracted to do.[73]

6.63      The committee heard that the close relationship between the ABS and IBM could be interpreted as 'vendor lock-in', and that such relationships risk complacency in project management:

Mr MacGibbon: ...In relation to whether IBM was the natural choice—and I did hear the questions from the committee earlier—I do believe there was a degree of vendor lock in—that they were a trusted partner that had established a relationship over many years and were seen as the natural choice by the ABS to deliver upon the project. Whether that is right or wrong is really for others to decide. But certainly I came to the conclusion that there was a degree of vendor lock in there.

CHAIR: There are risks associated with that type of closeness of relationship.

Mr MacGibbon: I would not be here in front of you today if those risks were not real.[74]

6.64      This view is supported by the findings of the CapDA report—commissioned by the ABS to consider options in delivering the eCensus system—which reported that some prospective solution vendors believed that the ABS would not move away from IBM:

Whilst the market scan revealed there are some potentially capable suppliers interested in bidding for this work and thereby generating competition, there were also substantial reservations expressed as to whether ABS would genuinely consider alternatives to IBM.[75]

6.65      One of the reasons cited by the ABS for partnering with IBM was their previous experience working with ABS systems and on the Censuses. As explained to the committee, the ABS assumed that IBM were familiar with their requirements:

There was considerable clarity as to our requirements and their contractual obligations to meet those. We were building on our experience with two prior censuses, so they [IBM] had an excellent understanding of what we needed. So I do not think that there was any lack of understanding leading up to the census.[76]

6.66      This assumed familiarity may have contributed to a level of complacency in project management on the part of the ABS, and in the priority which IBM gave the project. The ABS could have been more proactive in ensuring DDoS protection was in place. Whereas the ABS contracted third parties to undertake load testing and code reviews, IBM was left to test their own DDoS prevention solution. It was suggested that an external party might have uncovered the hole in Island Australia that was not revealed through IBM's internal testing.[77] It has not been explained to the committee why IBM's testing showed Island Australia to be effective on 5 August, but was later to fail on 9 August.

6.67      The committee heard that the ABS did not undertake an Information Security Registered Assessors Program (IRAP) assessment. An IRAP assessment assesses the implementation, appropriateness and effectiveness of an information security system's security controls.[78] As explained by Alastair MacGibbon:

The IRAP process is a process designed for third parties that are certified by or accredited by the ASD to come in and look at the architecture of systems—the schematics, for want of a better description—to ensure that they meet the information security requirements of the Commonwealth depending upon the level of security and protection needed in those systems. I would describe it as a compliance assurance process that should not be relied upon for all of IT security but is a practice that is necessary for ensuring that systems at least meet certain standards.[79]

6.68      The committee was further informed that it is not possible to know whether an IRAP assessment would have uncovered the flaws that allowed the DDoS attack to affect the eCensus.[80]

6.69      The relevant consideration here is whether the ABS should have had taken a more proactive oversight role in relation to the eCensus, and also whether they had the capacity to do so. CapDA's report that recommended the ABS partner with IBM was, in part, premised on the fact that ABS lacked the internal capacity to deliver the eCensus without outside assistance:

CapDA's report, as you are aware, noted that we did not have sufficient in-house capacity to run that application ourselves and that we would need to look at procuring an external partner to host that application.[81]

6.70      CapDA's report highlights the professionalism and dedication of the staff at the ABS, but in the end recommends that the ABS did not have the internal capacity to develop and deploy an eCensus.[82] If they did not have the ability to develop a solution themselves, it stands to reason that they would only have a limited capacity to question and challenge a contractor employed to develop such a solution.

Why did it take so long for the eCensus to resume

6.71      The eCensus website was offline for over 40 hours; from around 8.00 pm on 9 August to approximately 2.30 pm on 11 August.[83] The committee heard that IBM was ready to resume collecting eCensus forms on the night of 9 August, but that the ABS wanted to ensure that the security of the eCensus application before proceeding further:

As the protection of personal information was paramount for all concerned on census day, a cautious approach was taken by IBM, the ABS and ASD to ensure and verify that data security was never compromised before the site was restored. IBM was ready to restore the eCensus site after three hours, around 10.30 pm, but its closure was extended by 40 hours, following a direction given to IBM by the ABS.[84]

6.72      The committee heard that the ABS wanted to ensure that the eCensus website was no longer vulnerable to further DDoS attacks, that additional backups were in place, and that no data had been compromised before reopening the eCensus website. As the Australian Statistician explained:

There were three particular aspects that we wanted to be satisfied about. One was about the security of the data which, as we say, was something that we were assured of after three am the next morning. The second aspect was really related to the nature of the router and that there was a working contingency, and that was something that was still the subject of some discussion on the following morning and through the following day. The third aspect was that—certainly against the backdrop of the system being vulnerable to a DDoS event on the night before—I wanted to be as sure as we could be that it would not be vulnerable to a second DDoS event. And so that was where there was further engagement with ASD—and Telstra certainly assisted in that process as well—making sure that there were a number of additional protections put in place.[85]

6.73      Mr MacGibbon expressed his support for ABS' approach in delaying the resumption of the eCensus until satisfied that all faults had been rectified, observing:

There would only be one thing worse than the site being taken down that evening after those four denial of service attacks, and then the confusion around the router and the outbound traffic, and that was to get the site back up and have it knocked back down again.[86]

6.74      Once the ABS was satisfied that the system was secure and robust, the Australian Statistician ordered that the eCensus website be reopened; the eCensus was back online at approximately 2.30 pm on 11 August.

Was any personal data at risk?

6.75      The eCensus website was shut down due to fears that personal information might be compromised. As discussed above, at around 7.30 pm there was an observed elevation of outgoing data from the eCensus website. Fears that this traffic representing personal data led to the website being shut down.

6.76      The system developed by IBM for the 2016 eCensus employed a range of measures to prevent any loss of data.[87] IBM explained to the committee that:

In terms of the primary security objective here of protecting respondent data, we had encryption mechanisms in place to ensure that the data was fully encrypted while it was in transit—in flight from the respondent to the census site—and that it was encrypted while at rest and stored within the backend of databases. IBM does not have the keys to be able to decrypt that data, so we have not and have never been at any point able to see any of the respondent data that is stored on our systems. So encryption is the primary mechanism that ensures the integrity and protection of the data from external inspection.[88]

6.77      These keys are in the possession of the ABS and are necessary to extract any meaningful data from the census forms.[89]

6.78      The ABS reported to the committee that they were confident that eCensus data had been secure at all times:

I have to say I was confident all along that there was no breach, because of the nature of the security architecture. We had end-to-end encryption; we had had the architecture well reviewed. So the problem was that we could not explain this—as it turned out—false positive alert, and IBM could not explain to us what it was. We felt it was extremely important that we be able to assure the Australian public that our faith in our security was well placed.[90]

6.79      The Australian Privacy Commissioner and Acting Australian Information Commissioner 'received the necessary assurances [he] required to be satisfied that the personal information being collected as part of the 2016 census was secure'.[91] The Special Advisor to the Prime Minister on Cyber Security was also clear that no data was at risk:

There has been no dispute with any party that no data was lost from the census, and there is agreement that there was encryption from end to end and that the ABS held the keys to decrypt.[92]

6.80      Evidence to this inquiry from parties involved with either the events or the investigation on 9 August 2016 are all in agreement that no personal data was incorrectly accessed or released.[93]

Committee View

6.81      It goes without saying that the eCensus website should have had the capacity to withstand what was a relatively minor attack. IBM designed the system so that even if one link was disabled, the second should have been sufficient to carry legitimate traffic and continue processing census forms.

6.82      Criticisms made with the benefit of hindsight must necessarily be tempered, but there appears to have been significant and obvious oversights in the preparation of the eCensus. IBM's failure to have tested a router restart, or have a backup synchronised and in place, appears to have been significant contributing factors to the failure of the eCensus on 9 August. Further, the appropriateness of Island Australia must also be questioned given that some components of the eCensus—such as password resets—required access to international servers. Although it is impossible to say with certainty and hindsight what would have been had the ABS made different decisions, allowing IBM to undertake their own testing and the failure to complete an IRAP assessment appear to be significant oversights in project management.

6.83      The ABS' primary consideration during the period under discussion was to complete an accurate census of Australia. As discussed in preceding chapters, community trust is a central ingredient of a reliable census, and the ABS has repeatedly said that information security is one of its primary objectives. In light of this—and the information available to the ABS through the IBM monitoring system— the decision taken by the ABS on the evening of 9 August to close access to the eCensus website appears to be justifiable, understandable, and entirely correct.

6.84      A narrow focus on the events of August risks treating the symptoms and ignoring the disease. Questions regarding the validity of the ABS' actions should be focused on the years and months before the 2016 census when the decisions were made that would manifest themselves on 9 August 2016. The confirmation that the census would proceed, the delayed development of an eCensus solution, the use of a limited tender and the erosion of internal capacity to adequately oversee the development of the eCensus are all serious concerns that may contributed to the events of 9 August 2016.

6.85      The committee expected that the 2016 census would be subject to the two-pass ICT Investment Approval Process (IIAP) outlined by the Department of Finance.[94] The census project had lifetime ICT costs over $10 million once IBM, UXC Saltbush and Revolution IT contracts were taken into account. Further, as recognised by the ABS, the 2016 census was a high risk project.[95]

6.86      The ABS told the committee that the Department of Finance determined in October 2012 that the 2016 census was not required to complete to the IIAP.[96] In answers to questions on notice, the Department of Finance noted that the project did not meet the criteria for inclusion in the IIAP.[97] The committee notes that the ICT Review document recommending tendering for the eCensus was not completed until May 2014. The committee considers that the IIAP is in need of review to ensure that:

  1. projects such as the 2016 census fall within scope;
  2. the Department of Finance re-assesses projects at a later date if required; and
  3. the splitting of contracts is not a mechanism to skirt whole of life cost limits included in the IIAP.

6.87      The committee makes no suggestion that the Department of Finance acted inappropriately or that the ABS split contracts to minimise value-based scrutiny.

6.88      The committee also notes that the responsible Minister has not taken responsibility for the outcomes of the 2016 census. The committee calls on the current Minister, on behalf of the government, to take responsibility for the shortfalls in oversight that have been revealed through this inquiry. While many parties have not lived up to their responsibilities in delivering the 2016 census, the primary responsibility lies with the government.

Recommendation 4

6.89      The committee recommends that the Australian Government commit the necessary funding for the 2021 census in the 2017–18 Budget.

Recommendation 5

6.90      The committee recommends that the ABS conduct open tendering processes for future census solutions requiring the participation of the private sector.

Recommendation 6

6.91      The committee recommends that the ABS give greater attention to intellectual property provisions in contracts that include licensing and royalty arrangements.

Recommendation 7

6.92      The committee recommends that the 2021 eCensus application be subject to an Information Security Registered Assessors Program Assessment.

Recommendation 8

6.93      The committee recommends that the ABS take a more proactive role in validating the resilience of the eCensus application for the 2021 census.

Recommendation 9

6.94      The committee recommends that the Department of Finance review its ICT Investment Approval Process to ensure that projects such as the 2016 Census are covered by the cabinet two-pass process.

Recommendation 10

6.95      The committee recommends that the Australian Government provide portfolio stability for the ABS.

Recommendation 11

6.96      The committee recommends responsible ministers seek six-monthly briefings on the progress of census preparations. These briefings should cover issues including, but not limited to, cyber security, system redundancy, procurement processes and the capacity of the ABS to manage risks associated with the census.

Census Inquiry Service

6.97      In addition to the eCensus website suffering a prolonged outage, the telephone-based Census Inquiry Service (CIS) also buckled, but this time under the demands of the community. As the ABS explained:

Due to a range of factors including public concerns regarding fines that had been unprompted by the ABS, faster than expected postage of approach letters and a general high awareness of the Census, the CIS experienced unprecedented demand that greatly exceeded ABS forecasts. The unavailability of the online Census on Census night significantly exacerbated the number of calls. This led to significant ‘call blocking’ and inconvenience for Australians both in the lead-up to and on Census night. We apologise for this, and the ABS will take account of this experience in planning future Censuses.[98]

6.98      The ABS reported that the demand forecast for the 2016 census was 1.6 million calls, compared with 1.04 million calls received for the 2011 census.[99] The committee heard that by 8 September 2016 there had been 3.2 million attempts to call the CIS, of which 1.1 million had been answered. In addition there were 1.6 million calls to the automated inquiry service, of which 0.9 million were handled electronically.[100]

6.99      The ABS outlined their strategy for dealing with excess demand should it eventuate:

It was decided that the strategy for managing excess calls would be to politely request that callers call back later if the queues are at capacity, rather than provide callers with long wait times. This strategy is known as call blocking.[101]

6.100         The ABS submission concludes: 'The use of call blocking meant that 90.8 [per cent] of callers that got through to the CIS had their calls answered within 5 minutes'.[102] For the approximately one-in-three people who managed to make it onto the phone queue, the wait was relatively short.

6.101         Some submissions reported that people were unable to request a paper form in a timely manner due to excess demand on the CIS, meaning that their paper forms arrived well after 9 August.[103] Other submitters reported that their request for paper forms was not processed correctly or in a timely manner.[104] It was pointed out to the committee that households that do not have a telephone or internet connection would not have had any way to complete their census or obtain a paper form in 2016: a challenge that was previously circumvented by the physical delivery of paper forms to every household.[105] These households were likely followed up by Field Officers at a later date.

6.102         The problems affecting the telephone service were of particular concern to vision impaired Australians who were required to contact the telephone system in order to access the 12-digit code for the census, clarify information about the accessibility of the online forms, or request the census in an alternate form.[106] The Science Party also expressed concern that the unavailability of both the eCensus and CIS may adversely affect the response rate to the census.[107]

6.103         It was put to the committee that having to request a paper form via a telephone service added an additional unnecessary step that households had to complete in order to undertake the census.[108] ID Consulting, an Australian based demographic consultancy, argued that:

There is a large segment of Australia’s population who would prefer a paper Census form (particularly older residents), but the phone lines were jammed and these households were unable to get through. In any case asking a household to ring up to enable them to respond to the Census in their preferred way creates another level of impediment to responding and reduces the response rate.[109]

6.104         The committee heard that community perceptions that the census had to be completed on the night of 9 August or face a fine added to general frustration and distress.[110] Vision Australia, for example, told the committee:

Vision Australia spoke to people who were deeply distressed by the lack of information available; their anxiety was exacerbated by the threat of fines for not completing the Census on time. The abrupt message left on the service that people should call back later only added to this confusion.[111]

6.105         Vision Australia recommended that future Censuses should feature a dedicated, separate telephone service for people who require alternate formats or assistance to complete the census.[112]

Recommendation 12

6.106         The committee recommends that the ABS consider establishing a dedicated telephone assistance line for people who require special assistance in completing the census.

Development, delivery and collection of census forms and materials

6.107         In June 2014, the ABS signed a Memorandum of Understanding with Australia Post to support the 2016 census.[113] The ABS explained that Australia Post's mail service was used to deliver and return required materials from the majority of households.[114] Households that were going to respond online received a log-in code via the post, and households that requested a paper census form would have delivered and returned that information via the post.

6.108         In preparing for the 2016 census, the ABS attempted to anticipate local needs by tailoring delivery to area needs:

In some areas of Australia, where the postal service was likely to be unsuitable or insufficient address information was known, Census Field Officers delivered materials to each dwelling, enabling residents to either complete their form online or mail back a paper form. In other areas where a high proportion of residents were expected to need to complete the Census form on paper, all households were delivered paper forms in addition to login numbers.[115]

6.109         Despite the relative surety of the mail system in Australia, some submissions claimed that residents at certain addresses were not provided with census information. One such example related to the committee states:

We know of one elderly person in our community, whose address was apparently not listed by the ABS, who did not receive the letter at all, and had to go to extraordinary lengths to obtain the census questionnaire form. The person has lived at that address for 48 years and reports having had no difficulties in receiving previous census forms and completing them.[116]

6.110         It was put to the committee that the correspondence from the ABS with the log-in codes for the eCensus had the appearance of junk-mail and may have been discarded by some residents.[117] From the perspective it of survey design, ID Consulting observed:

Lack of personal contact with a collector on delivery means households are less engaged with the Census and the mail-out logins were often ignored or thrown in the bin as junk mail.[118]

6.111         The ABS submission highlights the lengths they went to in order to support the community to complete their census forms:

In partnership with CSIRO, ABS developed and tested the Census instruction letter, reminder letter and their envelopes to best support the Australian public undertaking the required actions – completing the Census online or requesting a paper form. Forty-nine variants were tested through random control trials, in order to select the best approach letters to households and reminder letters where completed forms had not been submitted.[119]

The role of field officers

6.112         One of the largest logistical elements of the census is recruiting the large number of Census Field Officers (CFOs) required to assist households complete the census. Approximately 38 000 temporary staff were recruited for the 2016 census.[120]

6.113         The evidence received by the committee amply highlighted some of the challenges in conducting such a large project that incorporates new elements such as the eCensus and form mail-outs. Some submitters felt that the conduct of CFOs was inappropriately persistent and aggressive. It was reported by one CFO that:

Field Officers were required to adhere to a strict timeline for visiting dwellings, with five visits scheduled over three weeks from 26 August to 16 September. Given the context of 'I've got until 23 September', this was seen as tantamount to harassment by many householders, who resisted what they perceived as unreasonable pressure to comply.[121]

6.114         At least one CFO reported to the committee cases where the households were visited multiple times because of insufficient information sharing.[122]

6.115         Others felt that the ABS—as represented by CFOs—were insufficiently proactive in following up on households.[123] The committee was told that CFOs in many mail-out areas did not commence their visits until two weeks after census night, by which time the census was no longer 'front of mind' for many people.[124] This appears to be a consequence of the move to a digital first census which reduced the role for CFO:

The use and approach of reminder letters were planned to allow half of all Australians to respond to the Census before household visits were required. Household visits were planned to provide support to any households that required it, deliver additional materials and remind households to complete the Census.[125]

6.116         Additional concerns raised in submissions included issues such as insufficient training, an over-reliance on CFOs using personal resources to ensure the success of the census, and that the information flows between the ABS and CFOs were insufficient.[126] As the face of the census in the community, CFOs were the ones who were left to motivate people to participate in the census following the events of 9 August.[127]

Navigation: Previous Page | Contents | Next Page

Top