Chapter 2The Optus network outage
2.1This chapter outlines the major events before, during, and after the Optus network outage on 8 November 2023. It provides a detailed, chronological list of relevant events, correspondence, public statements and announcements.
2.2The chapter also summarises the Australian Government's response including the reviews that were announced following the outage.
Timeline
2.3Below is the committee's understanding of the timeline of key events related to the Optus network outage. It draws on public sources and evidence from submissions and witnesses. All times are in Australian Eastern Daylight Time.
8 November 2023
4.05am: unplanned nationwide outage of all internet, cellular, and fixed-line services using Optus network in Australia;
4.37am: Optus network team 'fully engaged and dealing with the [outage]'; begins taking media enquiries;
4.38am: Optus issues nationwide alert to frontline teams;
6.33am: Optus issues first statement on the outage;
6.47am: Optus provides first updates via social media;
7.00am: officials from the Department of Home Affairs reach out to Optus to offer assistance;
7.02am: Optus informs the Australian Communications and Media Authority (ACMA) of the effect of the outage on Triple Zero calls;
7.40am: Minister for Communications, the Hon Michelle Rowland MP makes a first statement on ABC Radio National and publicly urges Optus to update Australians;
7.45am–8.30am: Optus holds internal crisis meeting;
8.32am: the CEO of Optus, Ms Kelly Bayer Rosmarin, calls the Minister for Communications to provide an update on the outage;
10.20am: Optus 'confident' cyberattack not the cause of outage;
10.30am: Optus begins hard reboot of network;
10.38am: Optus consumer internet services start to come back online;
10.40am: Optus CEO gives first media interview of the day;
11.00am: Communications Minister holds press conference;
11.32am: Optus updates Telecommunications Industry Ombudsman (TIO);
12.00am: over half of Optus network sites restored;
1.30pm: Optus announces gradual restoration of some services;
2.00pm: National Coordination Mechanism (NCM) meeting held, including Optus; services restored for 'vast majority' of customers; 98 per cent of Radio Access Network sites restored;
4.00pm: services restored for 99 per cent of customers; Optus declares the outage over; second National Coordination Mechanism meeting held;
4.35pm: Optus reports restoration of most services; slow and intermittent access continues throughout the night for some customers;
5.15pm: Optus CEO completes media interviews confirming restoration of services.
9 November 2023
Optus announces it will provide customers on eligible postpaid mobile SIM and mobile data plans with at least 200GB of extra data.
17 November 2023
Optus CEO, and the Managing Director, Networks, Mr Lambo Kanagaratnam, appear at a public hearing to discuss the outage.
20 November 2023
Optus CEO, Ms Kelly Bayer Rosmarin, announces her resignation; Chief Financial Officer, Mr Michael Venter, appointed interim CEO.
The impact of the outage
2.4An estimated 10.2 million Optus subscribers were affected by the outage, with its network handling up to 51.3 million mobile calls and 51 million SMS messages on a typical day. Also affected were subscribers to Mobile Virtual Network Operators (MVNOs) that rely on the Optus network, such as Dodo, Amaysim, and Moose.
2.5The immediate impact of the outage was wide-ranging, affecting major banks, transport operators, healthcare providers, government departments, and the Triple Zero emergency services. Optus initially advised that during the outage, 229 calls to Triple Zero by Optus customers were unable to be connected.
2.6However, on 23 January 2024, Optus advised that information it had provided to the Senate about 229 calls not reaching Triple Zero was not accurate, and the figure was at least 2697 calls.
2.7The Australian Communications Consumer Action Network (ACCAN) submitted that, in its view:
… the most significant risk of harm arising from the Outage for consumers was the breakdown of contact with emergency services … This had immediate and serious consequences for the safety and wellbeing of Australians across the country.
2.8State Emergency Services organisations (SES) were similarly impacted. TheNew South Wales (NSW) SES reported that its operations were compromised as some volunteer units and operational support staff were uncontactable or unable to access vital IT services, exemplifying the 'broad ranging consequences of such network failures on the operational activities of an [emergency services organisation] and the safety of the community'.
2.9ACCAN stated that the outage caused 'significant disruption across Australia'; specifically, '[i]t inconvenienced customers, closed businesses, and disrupted access to essential services'.
2.10The Australian Chamber of Commerce and Industry (ACCI) submitted that the outage:
… had a significant impact on businesses across Australia, but the impacts were more keenly felt by small businesses who are so dependent on day-to-day trade and operations.
2.11An estimated 400000 small businesses were affected by the outage. ACCIexplained that many small businesses have become cashless and are therefore heavily reliant on phone and internet connectivity to operate and take payments. Optus told the committee that around 50 per cent of payment terminals were unable to switch to an alternative provider and could not take payments during the outage, but was unable to confirm the number of small businesses impacted. As a result, ACCI submitted that many small businesses had faced a 'significant loss of trade' of as much as a whole day’s trade as a result of the outage.
2.12ACCI expressed disappointment at what it suggested was a failure by Optus to acknowledge the extent of the impact of the outage on small businesses, and submitted that the lack of information from Optus compounded the direct impact of the outage, leading to closures, loss of trade, and other financial challenges for small business. See Chapter 3 for a discussion of the adequacy of Optus' public communications response during the outage.
2.13The outage also affected hospitals and access to emergency services across the country. Optus confirmed that landlines were unable to make triple zero calls. Healthcare provider and private hospital owner, Ramsay Health Care, reported that phone services to its 70 hospitals and clinics were impacted. The Royal Adelaide Hospital was similarly compromised, which had a knock-on effect to the Broken Hill Base Hospital as patients were unable to be referred to their normal retrieval centre.
2.14Banking services, including some ATMs using the Optus network, were impacted by the outage. The Commonwealth Bank, ANZ, and Westpac all reported issues with their call centres, processing EFTPOS transactions, and some online banking services. ANZ reported that following the outage, scammers had commenced activities to try to obtain individuals’ banking details.
2.15Metro Trains Melbourne experienced an hour-long shutdown in the early morning due to drivers not being able to communicate with the control centre, causing hundreds of trains to be cancelled throughout the day.
2.16Ridesharing companies, such as Uber and 13Cabs vehicles, were unable to take bookings or use their mobile applications for directions.
2.17Australian Government departments, such as the Australian Tax Office and the Department of Home Affairs, were also affected as well as up to nine NSW Government agencies.
2.18Additionally, the Department of Home Affairs noted that the Optus outage highlighted the extent to which Australia's critical infrastructure sectors are integrated, including that disruptions in one sector may flow to other sectors. The outage highlighted issues with the resiliency of Australia's telecommunications networks, discussed further in Chapter 4.
The cause of the outage
2.19Optus initially reported that the outage had been caused by a 'technical network fault' and was the result of a 'deep network' problem.
2.20Early reporting suggested that the network outage may have been the result of an accidentally misconfigured border gateway protocol (BGP), a method for networks to share addresses with one another to enable clients to reach their intended destination on the internet. Some speculation suggested a cyberattack could have flooded Optus’ BGP routers to bring down the network. However, by 10.20am on the day of the outage, Optus reported that it was 'pretty confident' that it was not a cyberattack, and Ms Bayer Rosmarin made public statements to that effect.
2.21On 13 November 2023, Optus provided the following details on the cause of the outage:
At around 4.05am Wednesday morning, the Optus network received changes to routing information from an international peering network following a routine software upgrade…These routing information changes propagated through multiple layers in our network and exceeded preset safety levels on key routers which could not handle these. This resulted in those routers disconnecting from the Optus IP Core network to protect themselves.
2.22On 15 November 2023, an Optus spokesperson reportedly told the Australian Broadcasting Corporation (ABC) that the Singtel internet exchange, STiX was responsible for the outage when changes to routing information were sent to the Optus network, triggering the crash.
2.23On 16 November 2023, Singtel clarified in a media statement that while it had conducted a 'routine software upgrade' on STiX, the upgrade was not responsible for the outage:
We are aware that Optus experienced a network outage after the upgrade when a significant increase in addresses being propagated through their network triggered preset failsafes. However, the upgrade was not the root cause.
2.24On 17 November 2023, Optus provided further details on the outage in evidence to the committee, in which it described how a scheduled update from STiX (the'trigger event') caused Optus traffic to be routed through a set of improperly-configured Cisco branded routers (the 'root cause') that reached a pre-configured failsafe and independently shut down the network:
… key Optus routers disconnected from the network in response to a change in routing information that resulted in the shift from an alternative peering router during a scheduled upgrade on the Singtel international peering network.
2.25Ms Bayer Rosmarin insisted that Optus’ explanation of the outage accorded with Singtel's media release of 16 November 2023:
… our statement on the root cause that we put out was agreed and approved with SingTel. The statement that they put out is not a contradiction but rather a clarification … They were trying to clarify the way that that statement was interpreted.
2.26In response to Optus’ explanation of the outage, Cisco released the following statement:
We can confirm that Cisco routers performed as configured and we continue to advise the customer and provide relevant support.
2.27Mr Lambo Kanagaratnam, Optus’ Managing Director of Networks, further explained how the update affected the Optus’ network:
… there were almost 90 routers across the country that were impacted by the single issue that we had where a preset safety limit on these routers was reached or exceeded. What then happened was they all shut down, and effectively disconnected all these levels of redundancy, and effectively created the outage that we had …
The outage was as a result of our defences for the change in routing information not working as they should have.
2.28Ms Bayer Rosmarin stated that Optus’ network should have sufficient redundancy to manage traffic being redirected from one link in the event of an upgrade, but explained:
When one of those links was being upgraded and was therefore not available, what was coming through that link needed to be diverted through another link, which happened to be configured differently. It then propagated through our network in a way that triggered these failsafes in each of the different routers.
2.29Mr Kanagaratnam stated that around 100 devices across 14 locations had to be rebooted to restore connectivity.
2.30Ms Bayer Rosmarin further noted:
Even though we'd done very detailed reviews into our architecture, processes and risks to make sure we could be as resilient as possible, we did not have an articulated clear risk that each one of our 90 routers would independently shut itself down at the same time, using a failsafe on those Cisco routers that our team was not aware of.
2.31See Chapter 3 for further discussion of Optus’ response to the outage and see Chapter 4 for a further discussion of Optus’ level of preparedness and resilience.
The cause of the failure of Triple Zero calls
2.32The reason for the inability of some of Optus’ mobile customers to access Triple Zero was initially, and for a number of months after the outage, unknown. Despite significant speculation and reporting on the cause, Optus made no mention of it in its submission.
2.33Asked why shared roaming arrangements failed to connect emergency calls from Optus’ mobile services during the outage, Ms Bayer Rosmarin told the committee that investigations were ongoing and that it was 'too early to tell where the issue actually occurred'.
2.34Ms Bayer Rosmarin told the committee that while Optus takes accountability for the role it had in the inability of customers to access Triple Zero, investigations were ongoing, and it was too soon to determine what caused the issue. Shefurther noted that the Triple Zero system is designed to be able to 'pick up the traffic when we have an outage like this', and stated that the system was 'very complex', involving carriers, device manufacturers, as well as Triple Zero operators.
2.35Telstra confirmed to the committee that, throughout the outage on 8 November 2023, the Triple Zero network operated normally and accepted camp-on requests (a feature whereby customers who cannot access their carrier's network may make a Triple Zero call on another network), submitting that: ‘[a]ll calls successfully delivered to the Triple Zero Emergency Call Person Answer Point were answered'. Telstra also noted that it had received calls from Optus customers during the outage.
Australian Government response to the outage
2.36The Australian Government has a role in preparing for, responding to, and recovering from national crises. Whole-of-government arrangements are set out in the Australian Government Crisis Management Framework (AGCMF), which 'provides ministers and senior officials with guidance on their respective roles and responsibilities' in responding to national emergencies. The AGCMF is designed to be applicable to any crisis, with departments and agencies assigned different responsibilities under the AGCMF, depending on the nature of the emergency.
2.37The relevant Australian Government departments and agencies in this case were the Department of Home Affairs, the ACMA, and the Department of Infrastructure, Transport, Regional Development, Communications and the Arts (DITRDCA). All three provided evidence to the committee outlining their role and involvement in responding to the outage, discussed below. While the focus of each department and agency differed, all advised the committee that first contact with Optus was made at around 7.00am on the morning of the outage.
2.38The Department of Home Affairs, the lead agency for critical infrastructure policy and the management of national cyber incidents, submitted that it maintained an active presence throughout the day as it was unable to 'categorically rule out a cyberattack or other malicious action' as being the cause of the outage.
2.39The Department of Home Affairs also advised that the NCM, a function of the AGCMF, was convened twice during the outage. At the first meeting, Optus briefed Commonwealth, state and territory officials on its understanding of the outage, including some technical details. The second NCM meeting included other telecommunications carriers, who were 'invited to discuss any second and third order consequences or impacts of the outage, including to emergency call service availability'.
2.40DITRDCA submitted that it 'actively engaged with a range of stakeholders to support the government response to the outage' including liaising with key government stakeholders, Optus, and co-chairing the NCM. DITRDCA also 'closely monitored the situation to understand the impact of the outage on essential services'. Specifically, it had ongoing communications throughout the day with Telstra and the ACMA about the inability of some Optus customers to access Triple Zero.
2.41As the agency responsible for oversight of the Emergency Call Service, the ACMA submitted that it worked with Optus and Telstra to 'understand how the issues were impacting Optus’ customers and their ability to make emergency calls' and that this formed the basis of its communications with stakeholders during the outage.
Reviews into the Optus network outage
2.42A number of public investigations were announced following the outage and are discussed in detail below.
Department of Infrastructure, Transport, Regional Development, Communications and the Arts Review
2.43On 9 November 2023, the Australian Government announced that DITRDCA would undertake a post-incident review 'to identify lessons to be learned' from the Optus outage.
2.44The review was published in March 2024 with a focus on:
how Triple Zero functioned during the outage;
the role of government in managing and responding to national service outages;
the adequacy of requirements for customer communication during national service outages; and
the adequacy of customer complaints and the compensation process following the outage.
2.45Crucially, the review identified a potential cause of the Triple Zero connectivity issues during the outage and reported that:
At a meeting on 22 January, Optus provided information to the Review that the Optus network automatically wilted its 4G and 5G towers, but some elements of its 3G network and towers remained live, though unable to carry calls because of the failure of the core network. Optus suggests that it was the presence of these live towers that resulted in some Optus mobiles not seeking out alternative networks to carry calls to Triple Zero, hence customers in range of those towers did not have their calls passed through to Triple Zero.
Expert advice to the Review supports this conclusion. It is apparent that a combination of tower malfunction, core failure and the way mobile devices operate caused some calls to fail to access Triple Zero.
2.46The review noted that the above conclusion was based on information provided by Optus and other industry participants, along with subsequent discussions with submitters to the review. The ongoing investigation by the ACMA (see below) is expected to examine the technical details of why some calls failed to connect to Triple Zero.
2.47The review made 18 recommendations:
mandatory requirements to more clearly and explicitly articulate exactly what is expected of network operators to ensure calls are delivered to Triple Zero;
establish a Triple Zero custodian;
carriers to conduct six-monthly end-to-end testing of the Triple Zero system;
carriers be required to share real-time network information on outages with relevant emergency services organisations and other appropriate entities;
providers be required to report to the ACMA and the Department of Infrastructure, Transport, Regional Development, Communications and the Arts within a mandated timeframe on the causes of a major outage, steps taken to resolve it, the impact on Triple Zero and a clear plan with a timeline to address issues identified and avoid similar outages in the future;
the Department of Infrastructure, Transport, Regional Development, Communications and the Arts to review the contract under which Triple Zero is delivered;
the ACMA to develop a standard or determination requiring carriers to communicate specific information to customers during and about outages;
providers to implement an industry wide standardised approach to the form of resolutions available to consumers affected by a crisis or large-scale outage;
Australian Government to follow up work currently underway on roaming during natural disasters with work on temporary roaming during outages caused by other events;
Australian Government to require the establishment of mutual assistance arrangements between telecommunications service providers during outages;
network operators be required to establish the ability to remotely access and activate network management tools in the event of a core network outage;
Australian governments to review arrangements for maintaining their operations during outages; and
the Department of Infrastructure, Transport, Regional Development, Communications and the Arts and the ACMA to establish a review of all legislation and regulation relating to Triple Zero.
2.48On 30 April 2024, Minister Rowland announced that the Australian Government would implement all 18 of the recommendations. It agreed:
to establish a Triple Zero custodian after further investigation and consultation;
in principle, to conducting work on temporary roaming during outages caused by events other than natural disasters, noting 'industry concerns about the feasibility of implementing these arrangements'; and
to all other recommendations.
ACMA Review
2.49On 13 November 2023, the ACMA opened an investigation into Optus’ compliance with the Telecommunications (Emergency Call Service) Determination 2019. The ACMA advised that the investigation would focus on the obligation for carriers to provide assistance to one another in relation to Triple Zero calls, access by customers to emergency call services, welfare checks on customers unable to place emergency calls during the outage, and related matters.
2.50The ACMA submitted that it has a range of enforcement powers, including issuing a formal warning, issuing remedial direction, issuing infringement notices, accepting a court-enforceable undertaking, and commencing civil penalty proceedings in the Federal Court. It also undertook to make the investigation public once concluded. At the time of writing, the ACMA has not yet published the findings of its review.
Other reviews
2.51The Department of Home Affairs submitted that it was considering whether the Optus outage may have resulted from a lack of compliance by Optus with the Security of Critical Infrastructure Act 2018 (SOCI Act) and the Telecommunications Act 1997—which would fall under the department’s area of responsibility as the regulator for telecommunications security. It advised the committee that it had not commenced a formal investigation into the Optus outage at the time of its submission to this inquiry, but stated that 'a future investigation cannot be ruled out'.
2.52It also submitted that it may consider undertaking an assessment under the SOCI Act of Optus’ critical infrastructure assets to determine whether they pose risks to national security.