Chapter 2 Audit Report No.41 2011–12 National Partnership Agreement on
Literacy and Numeracy
The National Partnership Agreement on Literacy and Numeracy
(LNNP) was one of three ‘Smarter Schools’ National Partnerships announced in
the 2008–09 Budget. Commencing in 2009, the
LNNP was one of the first National Partnerships operating under the Intergovernmental
Agreement on Federal Financial Relations (IGA FFR), and one of the
first to include reward payments to states and territories.
The LNNP was envisaged as a tool to:
… galvanise the collective resources and energy of the
Australian Government and the state, territory and non-government education
systems, to put in place the infrastructure and practices that will deliver
sustained improvement in literacy and numeracy outcomes for all students,
especially those who are falling behind.
The LNNP also aimed to ‘accelerate progress towards the ambitious literacy
and numeracy target’ set by the Council of Australian Governments (COAG) to
‘halve the gap for Indigenous students in reading, writing and numeracy within
The LNNP committed a total of $540 million of Commonwealth Government
funding over four years, comprising the following:
- $150 million in
‘facilitation payments’ to states and territories to support literacy and
numeracy reform activities over the first two years of the partnership (2009
and 2010). These payments were to be contingent on equivalent ‘co-investments’
from states, including existing or
- $350 million in
‘reward payments’ to states based on the achievement of agreed literacy and
numeracy targets over the last two years of the partnership (2011 and 2012). Targets
were to be ‘ambitious’ and to ‘aim for accelerated improvement’ for schools and
students involved in the partnership, but would reflect the ‘different starting
points in each state’.
- $40 million for
research initiatives targeted at improving teaching capacity in literacy and
In addition to the research initiatives, an agreed outcome of the LNNP
was the Commonwealth’s management of a database of effective strategies and
approaches known as a ‘framework for effective practice’ or ‘Evidence Base’.
The Evidence Base was expected to encourage sharing of information and
effective practice, leading to better informed and evidence-based decisions on
literacy and numeracy. The Evidence Base was expected to be available by early
The Department of Education, Employment and Workplace Relations (DEEWR)
coordinated the implementation of the LNNP. This included negotiating bilateral
agreements and implementation plans with state government agencies. As part of
this process, DEEWR also negotiated the reform targets that would be the basis
for reward payments to states.
Approximately 1050 government and non-government schools received LNNP
assistance, accounting for around 13 per cent of Australia’s student population
and around 14 per cent of Indigenous and low-performing students.
In May 2012, the government committed to a $243 million extension to the
LNNP in the form of a new agreement that will expire in December 2013.
At the time of writing, details of the extended partnership and its
implementation plans were not publicly available.
The ANAO audit
Audit objective and scope
The objective of the Australian National Audit Office (ANAO)’s audit was
to assess the effectiveness of DEEWR’s administration of the LNNP. The three
high-level audit criteria used to form a conclusion examined the extent to
- established sound
administrative and payment arrangements consistent with government policy,
including through its negotiation of bilateral agreements, implementation plans
and reform targets;
- properly managed
administrative and payment arrangements; and
- effectively monitored
and reported on delivery and outcomes.
The audit report also included analysis of changes in National
Assessment Program – Literacy and Numeracy (NAPLAN) test results for
participating schools; presented case studies of literacy and numeracy
initiatives; and examined national trends in literacy and numeracy performance
since the commencement of the LNNP.
In performing the audit, the ANAO consulted with education authorities
in four states and other relevant stakeholders, including the Australian
Council for Educational Research; the Australian Curriculum, Assessment and
Reporting Authority; the COAG Reform Council; the Commonwealth Treasury; and
several schools that received LNNP funding. The audit covered the LNNP’s
operation from when it was signed in 2009 until March 2012.
Overall audit conclusion
The ANAO reported that, through the partnership, education authorities
and schools had implemented a range of initiatives in the agreed reform areas,
with positive impacts on schools, teachers and students.
Despite these positive impacts, ANAO analysis of NAPLAN data found no
statistically significant improvement in any state on the average results of
schools receiving LNNP funding when compared to schools that did not receive
funding. This lack of progress
was at least partially attributed to the short amount of time between the
commencement of LNNP activities and NAPLAN testing, with the ANAO suggesting it
may take several years until a ‘reliable assessment of the impact of the LNNP’
could be made.
The ANAO concluded that ‘overall, the effectiveness of DEEWR’s
administration of the LNNP has been mixed’. This was in the context of National
Partnerships being a new form of program delivery and the LNNP being one of the
first to include reward payments to states.
The report noted that while DEEWR worked collaboratively with state
agencies, the department ‘did not apply a structured approach to negotiating
key implementation arrangements’, including the number of participating
schools, performance indicators, and reform targets. This resulted in
‘significant variability at a state level in the coverage of the LNNP and
performance indicators used, and reward targets were not necessarily
Additionally, the ANAO found that DEEWR did not progress the Evidence
Base of effective literacy and numeracy strategies as promptly as envisaged
under the partnership agreement, limiting the guidance available to education
authorities and schools implementing LNNP programs.
The ANAO made two recommendations aimed at strengthening program and
payment design for future National Partnerships, and better assessing the
partnership’s impact on literacy and numeracy outcomes for participating
schools following the conclusion of the LNNP.
Table 2.1 ANAO recommendations, Audit Report No.41 2011–12
To inform program and
payment design that provides strong performance incentives for future National
Partnerships, the ANAO recommends that DEEWR review the approach taken in establishing
reform targets for the LNNP as the basis of reward payments, to draw on opportunities
DEEWR Response: Agreed.
To assess the impact of the LNNP and different literacy
and numeracy strategies, the ANAO recommends that DEEWR analyse the literacy and
numeracy outcomes of participating schools at an appropriate stage following
the conclusion of the National Partnership, as part of the national evaluation
of the Smarter Schools National Partnerships.
DEEWR Response: Agreed
The audit report also highlighted a range of broader lessons for the
establishment of National Partnership payment arrangements by responsible
agencies, including that National Partnerships should be designed to:
… allow sufficient time for initiatives funded using
facilitation payments to significantly influence performance results, prior to
measuring achievement against reform targets and making reward payments;
… provide the best opportunity for achievement of intended
outcomes, such as by linking accessible reward funding with the coverage of
planned initiatives and targeted levels of improvement.
The Committee’s review
The Committee held a public hearing on Wednesday 12 September 2012 with
representatives of the following organisations:
- The Australian
National Audit Office
- The Department of
Education, Employment and Workplace Relations.
The Committee took evidence on the following issues:
strengths and improvements over time
- Setting of reform targets
- Reward payments
- Co-investments by
- Impact of the LNNP
- Evidence Base
- Ongoing evaluation
- New LNNP agreement.
Administrative strengths and improvements over time
At the public hearing, DEEWR emphasised the strengths that the audit
found in elements of the department’s implementation of the LNNP, including
sound relationship management mechanisms and formal multilateral governance
DEEWR attributed these strengths to the National Partnerships
Implementation Working Group, which was established by federal and state education
ministers as an oversight body to steer the implementation of partnership agreements.
DEEWR also highlighted the audit report’s finding that the mixed
effectiveness of its administration was in part due to DEEWR not having access
to formal guidance when developing the LNNP framework. This was in the context
of the LNNP being one of the ‘earliest’ National Partnerships and one of the
first to include reward payments.
DEEWR pointed to improvements that were made during the LNNP’s
implementation period to the process for assessing reward targets, and noted
that lessons from the LNNP had helped inform guidance from central agencies.
The department indicated that it accepted the ANAO’s recommendations and
has applied the lessons learned from the audit in its subsequent negotiation of
a new literacy and numeracy partnership, and in developing the next phase of
its evaluation of the LNNP. The Auditor-General
agreed that the department had ‘responded positively to the report’.
Setting of reform targets
As noted above, the LNNP agreement stipulated that the reform targets
negotiated with states would be ‘ambitious’ and would ‘aim for accelerated
improvement’ in literacy and numeracy outcomes, while reflecting ‘the different
starting points’ of each state. The initial targets were
set by states based on 2008 NAPLAN data for reading and numeracy, supplemented
by a range of local measures. Targets were submitted
by DEEWR to the Australian Council for Educational Research (ACER) for
independent advice on their level of ambition.
The audit found that ACER had used its ‘professional judgement’ to
assess the ambition of the initial reform targets, and that DEEWR ‘could have
applied a more rigorous approach to assess ambition’.
At the public hearing, the ANAO explained that it was looking for more
structure or methodology in how ambition was assessed,
… it is a balancing act for the department. On the one hand,
with the national partnership you are trying to provide flexibility to the
states to develop the right implementation arrangements. On the other hand, in
this national partnership we are looking for ambitious, accelerated
improvements … We simply raise the point in terms of factors like the number of
participating schools and students and the levels of improvement, we think the
scale could have been tipped a little more towards the ambition side in
providing that flexibility.
The COAG Reform Council (CRC) had responsibility for independently
assessing whether reform targets had been met before payments were made to
states. The CRC’s performance
report for 2010 noted wide variations in state reward frameworks and the level
of ambition of targets, and made a range of recommendations for improving the performance
When questioned about the negative findings on the level of ambition of
the 2010 reform targets, DEEWR pointed out the difficulty that it faced in
setting targets based on a ‘very scant’ evidence base. The initial targets were
made against only one year of NAPLAN data (for 2008), meaning no existing
trends were visible. DEEWR explained that this was likely to have led to a
‘slightly conservative bias’ on the part of states.
However, during the setting of targets for 2011, three years of NAPLAN data
were available, meaning the department could apply a more structured approach
that was ‘far more transparent and robust’.
The ANAO agreed with the department’s position that during the setting
of targets for 2011 the process had been improved.
Timing of payments
The audit report noted the limited time allowed under the LNNP agreement
for states to demonstrate improvements in literacy and numeracy prior to reward
payments being made. The final bilateral agreement was signed in February 2010,
and in some cases state-level planning prior to implementation was only
completed at the end of 2010. The first LNNP reward payments were based on the
results of the May 2010 NAPLAN tests. The ANAO commented that any
improvements measured were therefore ‘unlikely to be significantly influenced
by LNNP activities’.
At the hearing, the Auditor-General acknowledged that in administering
the LNNP the department was acting within the particular framework established
by government, but emphasised that more time needed to be allowed for measures
to be implemented before performance was assessed.
Of the $350 million available in reward funding to states, the Committee
heard that $64 million was withheld from the final payments in June 2012 due to
targets not being met.
The Auditor-General noted that it was ‘encouraging’ that where targets
were not met, funds were being withheld, commenting that ‘this is the system
working as intended’.
At the request of the Committee, DEEWR provided a state by state
breakdown of the reward payments that were withheld. The data revealed that the
largest amount of unspent reward funding was for New South Wales, which
received only 26.67 per cent of its allocated funds. The best performing
jurisdictions were Victoria and the Australian Capital Territory, which each
received more than 90 per cent of the allocated reward funding.
There was also wide variability between jurisdictions in both the performance
against and the weighting allocated to Indigenous reading and writing targets.
For example, information provided by DEEWR showed that Queensland met or
exceeded two of its four Indigenous targets, and made significant progress on
the other two. These Indigenous targets had been weighted to 31.5 per cent of
the total performance targets. On the other hand, New South Wales did not make
progress on any of its Indigenous targets, but these targets only accounted for
10 per cent of the total.
The Committee noted the May 2012 media announcement that the majority of
the LNNP’s unallocated Commonwealth reward funding would be re-directed to the ‘Focus Schools’ program to improve Indigenous educational outcomes.
Quality of state data
A criticism in the audit report was that neither DEEWR nor the CRC had
access to the data or methodological information required to verify the
accuracy of performance results provided by states, risking inaccuracies that
‘may lead to corresponding inaccuracies in the allocation of reward funding’. Despite
having access to a more detailed NAPLAN dataset than that which had been
provided to DEEWR, the ANAO was unable to verify the states’ performance data
for either 2010 or 2011, as only limited information was available on the
methodologies used to calculate the reported results.
The report emphasised the importance of verifying or assuring the
accuracy of results to ensure that public money is spent appropriately, and suggested
that for future National Partnerships, ‘administering agencies would benefit
from working with states to coordinate preparation of performance results and
to consider related assurance processes’.
The ANAO explained at the hearing that DEEWR had only received
aggregated data from the states, and reiterated that better quality assurance
mechanisms were needed:
… a department like DEEWR needs to be better positioned once
it receives the performance data to be able to interrogate it and check that it
is accurate. That requires the school level data and a clear understanding of
the methodologies used to calculate the performance results.
The Committee was informed that individual states had negotiated their
own reporting methodology within their implementation plans.
DEEWR was asked about whether there was scope for the
Commonwealth to increase its visibility over the spending of reward funding by
states to ensure appropriate targeting. The department advised that its
agreements with state education authorities stipulated that funds must be
directed to educational outcomes, but that it had no further visibility or
control over spending. One of the ‘end-points’ of the LNNP, however, was to develop an
evidence base on improving literacy and numeracy (see below) which would be
informed by the LNNP funded activities of state education authorities.
Co-investments by states
As noted earlier, the LNNP agreement stipulated that states were
required to match the Commonwealth’s facilitation payments. The agreement specified
that the first facilitation payments would be triggered by the negotiation of
bilateral agreements and implementation plans, and that bilateral agreements
would include ‘the monitoring and reporting arrangements’ to track state
co-investments. Facilitation and reward
payments would only be made once the Commonwealth Treasurer received advice
that states had ‘complied in full with their earlier funding (co-investment)
The ANAO reported that co-investment obligations by states had not been
monitored by DEEWR. The audit found that the first facilitation payments had
been approved by the Minister for School Education prior to the signing of
bilateral agreements, ‘so that the implementation of the LNNP was not further
delayed’. In the bilateral agreements,
once signed, all states agreed to match or exceed the Commonwealth’s payments, however,
only four states agreed to include co-investment information in their annual
reports to DEEWR. No states actually reported this information.
The audit also found that DEEWR’s Chief Financial Officer had made
certifications to Treasury for facilitation and reward payments to be made that
did not make reference to co-investments. Treasury had access to data
on co-investments for the relevant financial years through its role in
acquitting co-investment reports on behalf of the Standing Council on Federal
Financial Relations, however, it had not obtained agreement to share its data
with relevant Commonwealth agencies.
When questioned, DEEWR referred to the matter as a ‘gap in the process’,
and explained to the Committee:
We asked Treasury for the reports, because these reports are
provided to Treasury, not to DEEWR. Treasury asked permission to provide them
to DEEWR. That was not given. Therefore we did not have it in place.
DEEWR further informed the Committee that subsequent to the LNNP, there
has been agreement that co-investment requirements would no longer be included
in future national partnerships, as such requirements are ‘input controls’ (and
therefore at odds with an IGA FFR principle).
Impact of the LNNP
It an opening statement to the Committee, DEEWR highlighted the LNNP’s
overall positive impacts:
DEEWR is particularly proud of the fact that the literacy and
numeracy national partnership is making a real and positive difference in our
schools, particularly for the lowest-achieving students. It is pleasing to see
that Indigenous students have shown the biggest gains in reading and numeracy
over the past four years. LNNP reforms have contributed to a changed culture in
schools through a focus on quality teaching, leadership, transparency and
The CRC’s performance report for 2011, released in May 2012, found that
schools participating in the LNNP generally improved their results in reading
In contrast, the ANAO’s analysis of NAPLAN results found that when
comparing LNNP schools to non-LNNP schools, the partnership was ‘yet to make a
statistically significant improvement’ in any state. This finding was in the
context that it may still have been ‘too early for such impacts to be clearly
evident’ (as noted in the above discussion on the timing of reward payments).
Asked about these findings, DEEWR explained that the ANAO’s analysis was
‘testing around a mean’, whereas the department was also looking at ‘other
measures’ in which there had been significant improvements.
The department explained that the LNNP was intended to focus on students that
were falling behind, and that NAPLAN results have shown a reduction in the number
of students below the national minimum standard—to ‘decrease the tail’.
In particular, DEEWR noted that the proportion of Indigenous Year 3 students
at or below the minimum standard for reading had decreased from 53 per cent in
2008 to 46.1 per cent in 2011.
Additionally, DEEWR noted strong support for the program amongst school
communities and principals, and told the Committee that when assessing the
success of the LNNP more outcomes than just NAPLAN results need to be
The report acknowledges that
significant achievement has been made against the other key outcomes of the
LNNP: increased collaboration between schools and systems in achieving literacy
and numeracy reform, improved classroom practice in literacy and numeracy, and
positive impact on school leadership, teacher practice and student engagement.
DEEWR informed the Committee that the literacy and numeracy Evidence
Base, a key output of the LNNP agreement, had been launched in June 2012
in the form of a website known as the Teach, Learn, Share database.
DEEWR explained that the database was being used to ‘share the success of the
LNNP’ by presenting a ‘body of evidence’ developed by state and territory
education authorities over the four years of the partnership.
The audit report noted that DEEWR had begun a national evaluation of the
three Smarter Schools National Partnerships, with the first phase having been
completed. The ANAO suggested that ‘given the complexities in measuring the effectiveness
of reform activities, it may take several years until a reliable assessment of
the LNNP approach can be made’. The report recommended
that DEEWR analyse the literacy and numeracy outcomes of participating schools
‘at an appropriate stage’ following the LNNP’s conclusion.
At the public hearing, the ANAO further explained that although there were
some ‘positive signs’, it was ‘still too early to tell at this stage the
DEEWR agreed with the recommendation, and undertook to continue
monitoring the LNNP’s impact with further analysis ‘for many years’.
The department said it would use the findings of the audit report to inform the
second phase of the LNNP’s evaluation, which was currently being scoped.
New LNNP agreement
DEEWR informed the Committee that negotiations were currently underway
for the new agreement, commencing in the 2013 school year. The department
planned to have implementation agreements in place for the new agreement by the
end of 2012.
DEEWR indicated that under the new agreement it was ‘looking’ to require
jurisdictions to identify ‘what strategies have been working, evidence of why
they have been working and for what cohorts, and for the funding to be provided
to continue and expand that particular practice within the state or territory’.
The Committee welcomes the audit report into the LNNP and strongly supports
the Auditor-General’s findings and recommendations.
While the audit report has highlighted a range of concerns, the
Committee notes that the LNNP appears to be having a positive impact on
literacy and numeracy outcomes and has provided a robust evidence base which
education authorities can draw on for future initiatives. Moreover, DEEWR has
made clear improvements to the LNNP’s implementation over time, and has
responded positively to the audit report’s findings.
The Committee agrees with the Auditor-General’s comments about the need
for longer term evaluation of the LNNP’s impacts. Such evaluation will help
ensure future initiatives are better planned and public money is spent in the
most effective way possible. The Committee is interested to hear more detail
from the department on its plan for implementing the Auditor-General’s recommendation
for evaluating the LNNP’s impact ‘at an appropriate stage following its
conclusion’, and therefore recommends:
That the Department of Education, Employment and Workplace
Relations report to the Committee within six months on the progress of its
implementation of the Auditor-General’s Recommendation 2 regarding the
development of a longer term evaluation strategy for the National Partnership
Agreement on Literacy and Numeracy.
It is concerning, however, that the initial performance targets set
under the LNNP framework were not as ambitious as they could have been. The
targets were not aimed at a level that would ‘accelerate improvements’ to the
extent foreshadowed in the language of the LNNP agreement. It is also
concerning that implementation plans were finalised, and performance targets
set, in some instances, very shortly before the first round of reward payments
were being evaluated. There may have been value in the LNNP specifying in
clearer terms what ‘ambitious’ means, perhaps to the extent of including
specific performance targets in the agreement at the outset.
These findings appear to strengthen the case for National Partnership
agreements and implementation plans to be negotiated in tandem, as the
Committee has previously suggested, and as has now been
included in advice to Commonwealth agencies issued by the Treasury.
At the same time, the Committee was reassured to hear that where
performance targets have not been met, reward funds have been withheld—as was intended
under the framework.
Another key concern of the Committee is that DEEWR did not fulfil its obligation
under the LNNP to monitor the agreed co-investment of literacy and numeracy
funds by states. DEEWR apparently provided certifications for payments to be
made to states despite having been unsuccessful in obtaining the required data
The Committee considers that the LNNP agreement was clear in its
stipulation that DEEWR would confirm that co-investments had been made prior to
payments being recommended to Treasury, and that state implementation plans
would specify the mechanisms by which the relevant data would be provided. That
only half of the implementation plans included the required details, and that, of
those, none were adhered to in practice, suggests an unwarranted disregard for
the terms and intent of the LNNP agreement.
Further, the inability of DEEWR, and the ANAO, to verify the accuracy of
performance results provided by states suggests that departments need to pay
more attention to setting up appropriate performance data monitoring and assurance
mechanisms early in the development of National Partnership agreements. As was
suggested in JCPAA Report 427—Inquiry into National Funding Agreements—outside
expertise, either from the Auditor-General or from other sources, could be
engaged to assist departments in developing performance data monitoring and
assurance frameworks, central to which would be consistent agreement on what
data assurance means in the National Partnership context.
The above findings suggest a general need for DEEWR and the Minister for
School Education to be more active in ensuring that the terms of its National
Partnership agreements are abided by during their implementation. This may require:
better mechanisms to ensure the adequacy of performance targets; the introduction
of assurance measures for performance data; improving Commonwealth visibility
of state co-investment data; and potentially negotiating more visibility over
how the funds provided to states are being spent.
The findings echo previous JCPAA comments on the need for implementation
plans to better reflect the expectations of National Partnerships and to have
good quality performance indicators.
Towards this end, the Committee recommends:
That, in order to help ensure the expectations of future
National Partnerships are met, the Department of Education, Employment and
Workplace Relations work with states and territories to develop
implementation plans and reform targets at the same time as any new National
Partnership agreements are developed, in accordance with advice from the
That where state and territory co-investment obligations are
included in the terms of current National Partnership agreements, the
Department of Education, Employment and Workplace Relations take
responsibility for monitoring these investments and assessing them before
payments are made, even if this requires negotiating more visibility of state
and territory data.
That when negotiating National Partnership agreements, the
Department of Education, Employment and Workplace Relations seek external
advice, potentially from the Auditor-General, on the monitoring and assurance
mechanisms that should be incorporated to enable verification of performance
data provided by states and territories.
Finally, the Committee strongly supports the Auditor-General’s comments
that in future National Partnership agreements more time must be allowed for
programs to have an impact before the criteria for reward payments are
assessed. In the case of the LNNP, more time being allowed between facilitation
and reward payments could have enabled more ambitious targets to be set by
states, and perhaps led to more meaningful and clearly measureable outcomes.
That the Department of Education, Employment and Workplace
Relations and the Minister for School Education ensure that sufficient time is
allowed in future National Partnership agreements for facilitation payments
to take effect before measuring the performance of states and territories
against reform targets and making reward payments.