Data collection about customer satisfaction
and network performance
The picture painted in evidence to this committee and in the broader
public discussion of the NBN is that the project is beset with issues relating
to poor customer experiences. It is difficult to ascertain however, just how
representative this is of the overall experience for customers. Some submitters
and witnesses argued that notwithstanding the issues experienced by a
significant number of customers, the overall satisfaction level of people
connected to the NBN is high.
There are several sources of information that were brought to the
committee's attention that provide some level of data on these issues, namely:
data collected by nbn; complaints data from the TIO; and research data from the
Customer satisfaction data captured by nbn
Mr Bill Morrow, CEO of nbn, stated in May 2017 that the overall rate of
satisfaction for customers remains positive, with over 85 per cent of end users
surveyed by nbn stating that their NBN service meets or exceeds their
Mr Morrow stated:
[T]he vast majority of people using the NBN are satisfied but
they are also a silent majority. Nobody calls their local MP or talkback radio
to say, 'Everything is wonderful' or, 'Everything is working fine.'
...The people who call are having problems and we acknowledge
there are too many. We are working hard on this but remember this is a small
percentage of a very large number. Across all of our technologies we average 21
faults for every 10,000 active premises.
Representatives from nbn explained to the committee that nbn surveys
customer satisfaction by asking its end users to rate their satisfaction with
their NBN service on an 11 point scale from zero to ten; where response scores
of eight to 10 represent customers being satisfied or extremely satisfied,
scores of zero to four represent dissatisfied customers, and scores of five to
seven represent customers who feel neutral about their service.
An overall 'net promoter score' for customers is calculated by
subtracting the number of dissatisfied customers from the number of satisfied
customers to provide a rough idea as to whether the overall response is
positive or negative.
On the issue of nbn's customer satisfaction data and net promoter
score, ACCAN submitted that this information should be published on a regular
nbn surveys all customers on their experience of switching
and using the network. They previously released this 'Net Promoter Score'
information on a technology basis, but do not release it regularly. If nbn
named areas, technologies or RSPs which scored below 7 on the Net Promoter
Score it would provide visibility about those that are not having a good
experience connecting and using services.
Monthly reports to the Department
of Communications and the Arts
At the public hearing in Canberra in June, the committee questioned officers
from the Department of Communications and the Arts (Department) about the
information that nbn provides to the Department. However, the Department does
receive monthly reports from nbn on the 'consumer experience':
The sorts of metrics [the Department] receive relate to
customer satisfaction and how that changes over time ... [the Department] receive
periodic reports from the company on other aspects of performance. For example,
you have had a range of evidence that goes to the performance experienced on
the Sky Muster service. It is well documented that that service has provided an
inadequate quality of service, particularly towards the end of 2016 and the
early parts of this year. So in relation to that, the government sought a
number of briefings from [nbn] to understand the nature of those issues and
what steps the company was taking to improve service quality. So through those briefings,
there was information provided about the number of faults, the connection time
frames and other factors.
On notice, the Department provided the following summary of the
information that it receives from nbn and the frequency of reporting:
NBN Co Limited (nbn) is required to maintain a high degree of
transparency and engage closely with Shareholder Ministers and Departments,
including by providing monthly progress reports, publishing online weekly
progress reporting of network deployment and active services, and delivering
quarterly management briefings to the public, including detailed financial and operational
information. Additionally, nbn is also required to publish information that
details how it has addressed the Government's objectives in the Statement of
Expectations, in its Corporate Plan and Annual Report.
During hearings the committee was informed that the Department did not
receive information about faults within the NBN on a regular basis. This
evidence was at odds with the Department's response to questions on notice
where it advised:
As part of a monthly reporting regime, the Department
receives data from nbn on network fault restoration, service fault restoration,
connection performance such as right first time activations, and activities
undertaken in accordance with service level agreement.
nbn also regularly provides the Department with information
on consumer satisfaction which includes data collected regarding connection
processes, the network usage experience, and issue resolution.
Mr Andrew Madsen, Assistant Secretary of the Broadband Implementation
Branch of the Department, was not able to confirm whether this reporting by nbn
was required, instead stating:
It has been the practice of the company to provide those
I think it would be the expectation of the shareholder
ministers that the company [nbn] provides those regular reports.
The committee was advised that the source of the information in the
reports that the Department received is 'the company itself, from its own
tracking of its performance and reporting from its delivery partners'.
The committee continued this line of questioning with officers from the
Department at the public hearing in Sydney. The committee requested a copy of
the most recent monthly report and was informed that it was a
The committee also asked Mr Bill Morrow, Chief Executive Officer, nbn, if he
was able to provide that report to the committee. Mr Morrow deferred to the government
to decide whether to make the information available.
In terms of the information received from RSPs, the Department does not
receive formal reports from RSPs, however:
...[the Department] talk to them about their experience of
interacting with NBN, the work they are doing in the rollout their services as
the NBN is deployed and the products they are developing.
Complaints data collected by the TIO
In addition to information collected and reported to the Department by
nbn, the committee also sought information from the Telecommunications Industry
Ombudsman (TIO) about the complaints information collected by the TIO. In
particular, the committee explored in some detail what information is collected
by the TIO, and whether the TIO should collect more information from
complainants in order to provide a better statistical snapshot of NBN
complaints and inform the TIO's ability to raise systemic issues that arise
with government and the industry.
Information collected by the TIO
during complaints resolution process
The TIO noted that it categorises complaint issues relating to NBN
services in broad first-tier categories (including 'connections', 'faults',
'customer service', and 'complaint handling'), and then in second tier
categories. The TIO noted that for complaints about services delivered over the
NBN in 2016, the highest ranked first tier issues were 'Connections' and
The TIO emphasised in its submission that its primary purpose in
collecting information from customers is on ensuring there is sufficient
information recorded to facilitate resolution of the complaint.
The TIO stated:
The primary role of the TIO is to facilitate the resolution
of complaints. In accordance with Treasury's Benchmarks and Key Practices for
Industry-based Customer Dispute Resolution, the TIO must be efficient,
accessible to all Australian consumers, and the dispute resolution processes
must be easy to use.
This requires engaging with consumers in a way that makes
lodging a dispute easy and shows we are listening. The TIO must not put up
barriers or require technical information that the consumer does not have or is
unable to provide, for any purpose other than resolving their dispute.
Ms Teresa Corbin of ACCAN suggested that the complaints data collected
by the TIO should be viewed cautiously in terms of using them to draw
conclusions about the overall level of issues being experienced on the NBN:
I think that the issue with the TIO statistics is that it is
very difficult to gauge from them whether in fact there are issues with NBN. ...I
do not think that it is adequate to say that the number of complaints only a
small percentage of connections at the moment, because the truth of the matter
is we have not reached the peak time for connections. They are likely to
increase over the next six months, and then into next year will be the major
time when we are connecting a lot of new people. Any one complaint could be a
reflection of many other complaints; it is just that nobody spoke up or nobody
knew about the TIO. We need to be quite careful with those complaint statistics
and be aware that they are really just an indicative thing. They are not going
to be comprehensive.
When asked whether conclusions could be drawn about the performance of
particular RSPs based on the number of complaints against each RSP lodged with
the TIO, the current Ombudsman, Ms Judi Jones, commented that simply
looking at the raw complaints numbers for each RSP would not paint an accurate
picture, without also understanding the number of NBN customers each RSP has
Ms Jones noted that the TIO does not have access to the data about the
number of connections for each RSP that would allow for meaningful analysis of
complaints statistics on an RSP basis.
Data on the number of missed
The Ombudsman also confirmed that the way the TIO records complaints
means that the data also could not be used to calculate matters such as the
total number of missed appointments by NBN technicians, as one customer's case
would be logged as a single complaint relating to the connection of a service,
even if the case involved multiple instances of missed appointments with NBN
technicians or RSP representatives.
When questioned whether the TIO should collect information so as to
enable visibility of the number of missed or rescheduled appointments, Ms Jones
argued that any data provided by the TIO on this issue would be incomplete, and
that nbn would have access to the full number of missed appointments across the
The TIO expanded on this issue in response to a question on notice:
Out of a total of 7,948 connection issues reported in the
2016 financial year about services delivered over the NBN, 1,066 involved
missed appointments... [T]he TIO does not quantitatively report on the number of
visits nor the number of missed appointments. This is because the TIO does not
award punitive damages based on the number of missed appointments. Instead, the
TIO's focus is on facilitating the connection to be established, in order to
reduce the ongoing consumer detriment.
Collecting information about the
underlying causes of complaints and responsible parties
A question raised throughout the committee's inquiry was whether the TIO
should collect more information relating to the underlying causes of complaints
made to the TIO, including identifying where the issue has originated and who
is responsible for addressing it.
Ms Kathleen Silleri, Assistant Secretary Consumer Safeguards Branch at
the Department, stated that the TIO would ideally collect more information
about the causes of complaints:
[W]hat we would like the TIO to do is to provide a very
accurate picture of what exactly is occurring in the industry. If that means we
need to determine exactly who is to blame for an issue that is being
experienced by a consumer, and then incentivise that not to occur by either
being very public about it or reporting exactly where the problems are, that
would be a good thing.
Ms Silleri noted ongoing discussions between the Department and the TIO
on this issue, and explained the importance of getting clearer information on
We are talking with the TIO about how we determine what is
actually occurring in a situation where a consumer has lost service, has been
unable to get a service or has issues with that service. The first issue that
is encountered, generally, from our perspective...is the way that they record the
initial contact with the consumer, and it becomes a route that is set from the
initial contact. The threshold question then almost determines what the nature
of the complaint is. There is also an issue that follows on from that. If the TIO
is not understanding exactly what is occurring when a consumer is experiencing
a fault, nobody is ever going to get to the bottom of it and be able to solve
it for a consumer.
Ms Silleri elaborated on this point further at a later public hearing in
We think the fundamental issue is that, when somebody rings
with a complaint, they are ringing to advise what isn't working. From that, the
TIO complaints officer goes down a path set by various keywords. It would be
useful to think about what is most illustrative of the actual problem in
identifying what those keywords would be. For example, I think the TIO at the
moment asks whether it's a problem with your landline, mobile or internet. Your
internet problem could be caused by your landline or mobile, so that doesn't
take you anywhere. It would be very useful to understand whether it's a problem
with an existing connection or a new connection and whether or not that problem
has occurred as a result of transition to a new network. Those are the sorts of
things we would've encouraged the TIO to consider.
On the issue of identifying who is ultimately responsible for the
problems being raised in a complaint, the current Ombudsman, Ms Judi Jones,
outlined the complexity in some cases of the TIO trying to attribute
responsibility for a complaint to a single party:
[I]n the vast number of complaints we do not find out what
the problem was. It is not just the distinction between what the RSP is
contributing and what NBN Co is contributing. There may be things related to
the wholesaler or aggregator or things on the consumer's premises.
Ms Jones cited the example of a specific case to highlight the
difficulties involved in classifying complex complaints:
A consumer had an appointment to connect to the NBN, and an
NBN Co technician arrived. When they arrived they found there were problems
with the consumer's internal wiring and so could not connect them. The consumer
went away and fixed the wiring problems, and later in that same month an NBN
technician attended again. That time the technician identified there were
problems with aerial cabling and he did not have the necessary equipment to fix
that, so he had to go away. After three more appointments with NBN Co
technicians, the work was unable to be completed. There was another technician
that missed an appointment, without any information to the consumer about why.
So six scheduled appointments and the consumer was still not connected to the
national broadband network. In that there is a combination of problems: a
problem on the consumer's premises and a problem with the appointment-keeping
or work to be done by NBN Co. It is not always easy, even when we do look at
those complaints, to say that it was a single fault, that it was the fault of
either the RSP or NBN Co.
Ms Jones reiterated in evidence to the committee that because most
complaints to the TIO are resolved simply by referral back to the RSP, the TIO
does not generally know the root cause of these complaints.
Ms Jones explained that in the majority of cases, while customers do describe
the problem that has occurred to the TIO, there will not necessarily be an
explanation or understanding in this initial phase of what has caused the
When questioned whether RSPs should be required after the fact to advise
the TIO what the cause of the problem was and how it was resolved, Ms Jones
contended that this would add a significant cost to the TIO's operations:
[When] you are looking at 112,000 complaints [in the 2015-16
financial year], to have every retail service provider ring us and get us to
record the outcome would be an extraordinary burden on cost. At the end of the
day—let's be real—consumers pay the cost even though the members fund the
scheme. That would be an enormous impost, I think.
Collecting information by
The committee discussed with the TIO the issue of whether it could
record NBN complaint data by technology type, to enable analysis about any
specific issues affecting customers on the different NBN technologies.
The TIO stated in its submission that it does not routinely record the
technology type for the consumer's connection to the NBN. It stated there are a
number of reasons for this practice, arguing as follows:
consumers do not generally know the technology type;
where the complaint is being made by a referral agency (e.g.
financial counsellor), the referral agency is unlikely to know the technology
requiring a consumer to identify the technology type before
accepting the complaint would introduce an unreasonable barrier to making a
complaint to the TIO – and not align with the TIO's requirement to provide an
it is not necessary to know the technology type to effectively
refer complaints to the member for resolution – the vast majority of complaints
(90 per cent) are resolved by referral back to the member, without the TIO
providing the technology type;
identifying and recording the technology type would take
additional resources, without a clear dispute resolution benefit; and
it is currently not possible to automate the collection of this
Representatives from the ACMA expressed the view that the TIO should
ideally collect information on the technology type underlying complaints made
to the TIO, but acknowledged that the TIO's primary purpose was to resolve
issues rather than analyse their causes, and that consumers often were not even
aware of the technology type in place at their premises.
Ms Silleri from the Department informed the committee that it would be
supportive of the TIO capturing complaints by technology type.
Publication of data collected by
Representatives from the ACMA informed the committee that it is provided
with raw complaints statistics each month by the TIO on a confidential basis,
to allow it to get a sense of emerging issues in a timely manner.
Ms Jones noted at a public hearing in March 2017 that the TIO only
currently publically reports complaints statistics annually, but stated that
the TIO is looking at potentially publishing data six-monthly, depending on the
availability of rollout data from nbn:
I think it is always important with complaints about services
delivered over the National Broadband Network that we do it in context to the
rollout. We are just reviewing our reporting across the organisation, trying to
look at the effort that is required for the reporting. It might sound like it
is just a matter of pushing a button, but there is a lot of checking and
quality assurance and giving people guidance on how to interpret the data as
well—so effort and value. We expect to have finished that work by the end of
March and then we will go back into more regular reporting. With services
delivered over the NBN you really need the number of premises connected, and
that data is only available publicly from NBN Co in six-monthly batches. We
certainly would not be reporting about services delivered over the NBN more
frequently than six monthly, unless the data becomes available more frequently.
Ability of the TIO to raise
systemic issues relating to the NBN rollout
Another question discussed in relation to the TIO's data collection was
whether it is sufficient to enable the TIO to identity systemic issues arising
in the NBN rollout and raise these issues with government and industry.
The TIO informed the committee that in 2016, seven systemic issues were
finalised that related to the actions of RSPs and the NBN rollout, and stated
that as at 28 April 2017, there were three possible systemic issues relating to
the NBN currently under consideration by the TIO.
The TIO stated that it does already identify NBN-related issues and work
with industry stakeholders to address them. It submitted:
The TIO monitors complaint trends and becomes involved in a
range of formal and informal discussions, including on issues about the NBN.
The TIO facilitates meetings and discussions between retail service providers
and nbn to highlight issues and to exchange information. Members are encouraged
to take prompt action to resolve wider issues across the industry.
The TIO also engages with consumer organisations (including
ACCAN), regulators (the ACMA and ACCC), representatives of the Department of
Communications and the Arts, retail service providers and nbn to support the
resolution of issues. The TIO does this by sharing insights to highlight the
issues consumers identify and experience in the rollout of the NBN.
The TIO stated that examples of issues it had raised through these
processes include: technician appointment systems; infrastructure difficulties;
sales and marketing practices; early termination fees; and the NBN interim
satellite service and SkyMuster.
Research on the customer experience by the ACMA
The ACMA's 2016 research paper Migrating to the NBN—The experience of
Australian consumers included survey data in relation to the satisfaction
of NBN users in FTTP premises. Nine hundred residents and 304 businesses were
surveyed about their experience migrating to the NBN, and further qualitative
research was also undertaken for the study. In relation to overall consumer
satisfaction with the connection process, the ACMA's findings were as follows:
Most consumers were satisfied (rating of five or above out of
10) with the whole process of connecting to the NBN; however, one in five
residents and more than a third of businesses were dissatisfied with the
process (rating of four or below out of 10). A key positive factor influencing
consumers' satisfaction ratings was the absence of any service disruptions.
In relation to customers' experience once connected to the NBN, the
findings were summarised as follows:
It was evident from the research that, for most consumers,
migrating to the NBN met their expectations of having access to a faster, more
reliable internet service. For connected consumers, satisfaction with internet
speeds was higher than for those not connected. Nearly half of residents (48
per cent) and two-thirds of businesses (65 per cent) expected the NBN to be
faster than their current service. Residents were reasonably satisfied with
their fixed-internet and landline phone services since connecting to the NBN,
with 33 per cent reporting that their internet was more reliable now than
before and 51 per cent reporting that their internet service was comparable to
their service prior to connecting. Satisfaction levels for fixed-internet
services were similar for businesses but slightly lower for landline phone
services. A quarter of businesses reported that their internet was more
reliable now than before.
The findings suggest, however, that some consumers continued
to experience concerns with service reliability, with around one in five
reporting that their fixed-internet and landline phone services were less
reliable now than before connecting to the NBN.
At the public hearing in Sydney, Ms Jennifer McNeill, General Manager,
Content Consumer and Citizen Division, ACMA, noted that the authority was
moving to a new phase of evidence gathering in relation to the NBN:
We will have a new consumer-focused research piece in the
field in a few months time. Again, that will be looking at the consumer
experience of Australians across a range of technologies as the network's
rolled out and, also, smaller and medium-sized businesses. That's the
consumer-facing piece. We're also moving to collect more granular information
from companies involved in the NBN supply chain so that we can get a better
grasp of the extent, the scope and the nature of problems that people are
encountering as the network rolls out. Then, we can reflect on that
information, together with industry and others involved in government, to see
what might be done to make the experience as positive a one as it can be for
In answer to a question on notice from the 23 June hearing in Canberra,
the Department provided some further information on how the research by ACMA
would be carried out:
....[the] Australian Communications and Media Authority (ACMA)
will conduct research and collect data from businesses across the NBN supply
chain using its powers under the Telecommunications Act 1997. Twenty-one
industry participants including retailers, wholesale providers and nbn will
receive notices seeking a range of data on issues such as fault handling,
connection timeframes, and appointment keeping.
In the committee's view there is clearly significant data being
collected in relation to the customer experience in relation to the rollout of
the NBN. However, the committee considers that the information made publically
available relating to customer satisfaction on the NBN network does not enable
adequate analysis and evaluation of the overall customer experience.
nbn's customer satisfaction metrics provide a much greater sample size
than any other available data set. While a headline figure on end user
satisfaction ratings is generally included in nbn's Annual Reports and
Corporate Plan, this level of disclosure does not allow for any rigorous
analysis of the data. The committee considers that the regular publication of
nbn's end user satisfaction metrics would provide much needed transparency
about the overall level of satisfaction in the NBN, and would provide context
for the broader public discussion around the success of the rollout from a
The committee recommends that nbn publish prominently on its website,
monthly information relating to its end user satisfaction metrics, including:
its overall net promoter score as measured each month;
the overall net promoter score for each technology type as
measured each month;
relevant disaggregated information about end user satisfaction
metrics in relation to each RSP; and
any relevant disaggregated information about end user
satisfaction metrics in specific geographic areas, such as:
data broken down by state and territory; and
data relating to each fixed-line area in the rollout footprint,
as areas are designated Ready for Service.
Data collection by the TIO
The committee considers that there is considerable scope for the TIO's
data collection activities to be enhanced in order to enable its statistics to
become a much richer source of information in evaluating the performance of the
NBN. The committee acknowledges that it is not possible in every instance for
the TIO to collect information such as the NBN technology type at the
complainant's residence, however it is important that such data be collected
The committee notes that both the Department and the ACMA expressed the
view that the TIO could collect data in a more robust fashion to provide a
greater level of information on the NBN. The committee awaits to see how these
issues are dealt with by the current independent review of the TIO, and the
committee will provide further monitoring of this issue in its future work.
The committee recommends that the scope, function, and operation of the
Telecommunications Industry Ombudsman (TIO) be expanded so that, among other
improvements determined through the current review process, the TIO should keep
data according to technology type, and should record and report multiple issues
as separate items, especially where nbn and an RSP are both involved.
The committee welcomes the additional research work that will be
undertaken by the ACMA, examining the NBN consumer experience across all stages
and technology types. This will provide valuable, in-depth qualitative data on
Provision of information to the
The NBN rollout is the largest public infrastructure project in
Australia which touches every premises in the country. The significant
conjecture about the speed, efficiency and effectiveness of the rollout
underscores the importance of transparency and disclosure.
The committee notes the refusal of the Department to provide the
committee with the monthly reports provided to the Department by nbn, despite
the committee's offer to accept this material in confidence. The committee
believes that the information contained in nbn's monthly reports would greatly
assist the committee in its inquiry and strongly encourages the government to
reconsider the publication of this information.
The committee recommends that the Department of Communications and the
Arts publish the data it receives from nbn as part of its monthly reporting
regime, including data relating to:
network fault restoration;
service fault restoration;
connection performance, such as right first time activations; and
activities undertaken in accordance with service level agreement.
Navigation: Previous Page | Contents | Next Page