Government senators' additional and dissenting comments
Government senators are pleased to see that the committee
majority recognises and supports the importance of national testing for
literacy and numeracy and will provide some additional commentary on this
below. Result analysis and reporting are the subject of ongoing improvements
which should address the committee majority's recommendations regarding
reporting refinements. However, government senators do not agree with the
proposal to expand NAPLAN testing and this is addressed below. Government
senators also do not agree with all the revisions proposed in the committee
majority report for the My School website, particularly the focus on
value-added measures and provide reasons as well as additional commentary later
in this report.
Government senators acknowledge the central importance of
literacy and numeracy as the foundations for further learning. As indicated by
the Australian Council for Educational Research (ACER):
Educational research studies have underscored the fundamental
importance of literacy and numeracy, not only to educational success, but also
to successful transitions into employment and adult life. Levels of literacy
are correlated with a wide range of outcomes, including lifetime earnings,
employment prospects, levels of health and involvement in crime.[1]
Benefits of NAPLAN testing
The government's education revolution is driving a renewed
focus on the foundation skills of literacy and numeracy to lift student
achievement across the country. National Assessment Program – Literacy and
Numeracy (NAPLAN) testing is intended to:
...identify whether all students have the literacy and
numeracy skills and knowledge that provide the critical foundation for other
learning and for their productive and rewarding participation in the community.[2]
Dr Peter Hill, Chief Executive Officer, Australian Curriculum,
Assessment and Reporting Authority (ACARA) emphasised to the committee that
NAPLAN is intended to provide a snapshot of student performance to better
inform stakeholders for the purpose of improvement.[3]
Uses of the data
Government senators support NAPLAN testing and reporting as
an important tool to provide valuable information for many uses which include:
- enabling a sophisticated analysis of growth in attainment over
time;
-
showing student results by sex, location, parental background and
Indigenous status, enabling the community to clearly see those student groups
which need more support to improve their educational outcomes;
-
tracking progress towards Council of Australian Governments
(COAG) targets;[4]
- giving the community, teachers and parents better information
about how schools are performing;
- helping schools to identify strengths and weaknesses in teaching
programs;
- allowing governments to target additional resources to schools
which may be struggling;
-
complementing class and school based assessment to provide a
comprehensive picture of student achievement that helps determine areas of priority
to improve literacy and numeracy outcomes;
- allowing teachers to better identify students who require additional
support; and
- enabling parents to use individual results to discuss student
achievement and progress with teachers.
ACER confirmed that
the tests provide a new level of information for teachers and schools,
education systems and governments as well as parents. It explained how the data
can be used:
At the school level, the tests can be used to identify
areas of relative strength and weakness and to assist in planning teaching
interventions and special support. A number of Australian states provide
software to support schools to explore and make best use of the diagnostic
information available through NAPLAN. At a national and education system level,
NAPLAN provides diagnostic information to inform educational policies and
planning.[5]
Mrs Sharyn Lidster,
Acting General Manager, Strategic Policy and Performance, Department of
Education, Tasmania, explained how important the NAPLAN data is for school
education systems:
It provides us with performance information, for the first
time when we had the national test, that enabled each of the states and
territories to actually look at our results relative to other states and
territories across Australia. That is very important because prior to 2008 we
had state-wide testing but each of the states had different tests, so it made
it very difficult to see how our students were performing across the nation in
relative terms.[6]
Mrs Lidster indicated
how schools are supported to make best use of the data:
The NAPLAN tests are used extensively to support schools. Our
jurisdiction, and others, provides the information back to schools. A lot of
analysis is done that supports the schools. Workshops are run to help the
schools to interpret the results and use them effectively to support their
teachers. We also conduct workshops to assist senior people within schools to
interpret the information and, where appropriate, we provide additional support
for schools in relation to teacher development. Also, funding is provided to
support the additional programs to improve the outcomes for students, where
they are identified as performing below where we would expect them to be.[7]
Mrs Lidster emphasised
the usefulness of the data for schools and teachers:
There is a large amount of evidence that shows us that the
quality of our teachers makes a big difference to the outcomes of our students.
The use of NAPLAN, or state-wide and national testing, especially when it is
provided back to schools, is valuable to help teachers look at how effective
their delivery of the curriculum has been. It also helps school leaders to sit
down and have critical conversations with their teachers to say, for example:
‘These results are particularly fantastic. What is it that you are doing in
your classroom? Let’s share that practice across our school.’ It also helps our
district support staff to look at the school’s results and to work with the
school either to bring in additional professional learning where it is needed
or to share some particularly gifted teachers’ talents with other schools. So I
think it is critical, and it will lead to improvement in Australia if we are
using those results for that purpose.[8]
The ACT Council of Parents
and Citizens Associations supported the continuation of national testing and pointed
out the importance of the data for parents:
...The council believes that parents should receive as much
detailed information as possible, as this encourages them to speak to their
child’s teacher and get involved in their education. Research consistently
shows that the best outcomes are achieved when teachers and parents work
closely to give students the best education possible.[9]
While impressing on the committee that NAPLAN is only one
source of information about student achievement, the President of the
Australian Primary Principals Association (APPA), Ms Leonie Trimper, told the
committee that her organisation 'supports the use of NAPLAN to provide schools
and systems with quality information about the performance of Australian
primary school students in literacy and numeracy'.[10]
Government senators view
Government senators support the increased transparency and
rich information source now available for governments, educational authorities,
schools, principals, teachers and parents as a result of the publication of
NAPLAN test results.
International research and best
practice
The former Minister for Education noted that the NAPLAN
measurement scales are unique and represent world best practice in the
measurement of student progress.[11]
Professor Geoff Masters, Chief Executive Officer, ACER emphasised that NAPLAN
is firmly grounded in 20 years experience through state literacy and numeracy
testing programs, adding:
NAPLAN is also pretty firmly grounded in international best
practice in tests of this kind...Part of the reason that the Australian Council
for Educational Research are managing the PISA [Programme for International
Student Assessment] tests for the OECD out of Melbourne is that we do have
international expertise in the analysis and reporting of data, and we are
applying that to the NAPLAN tests.[12]
He explained that the
test development process used is accepted as international best practice, as
well as the statistical methods and processes used to analyse the data and
report the results.[13]
Dr Ben Jensen, Director, School Education Program, Grattan
Institute, stated that NAPLAN testing and to a lesser extent the My School
website are important advances in the development of the school education
system, noting that:
...the international research clearly shows that high
performing education systems are more likely to have national assessments and
utilise them in some form to foster school improvement, school accountability
and school choice.[14]
Using NAPLAN data to address
disadvantage
NAPLAN results have showed that there are still some
students who have not attained the literacy and numeracy skills expected of
students in their year level. The federal government has entered into National
Partnerships with the states and territories to address disadvantage, support
teacher quality, and improve literacy and numeracy.
As noted in the 2010-11 Budget, funding of $2.6 billion over
five years from 2008-09 has been provided for the three Smarter Schools
National Partnerships. There is $540.0 million for the Literacy and Numeracy
National Partnership which seeks to accelerate improvement in student literacy
and numeracy outcomes for those students most in need of support; the $1.5
billion low Socio Economic Status School Communities National Partnership to support
the learning needs and wellbeing of Indigenous students to improve engagement
and attendance; and the $550.0 million Teacher Quality National Partnership which
will support teachers, including those in schools with predominantly Indigenous
students.[15]
Through the NAPLAN assessment and My School website,
the government has identified an additional 110 struggling schools that would
have missed out on a share of the $2.6 million Smarter Schools National
Partnerships and will now share in $11 million in extra funding to ensure
students improve literacy and numeracy.[16]
The increased transparency and attention on support for low
performing students was supported by Mr Gary Banks, Chairman of the
Productivity Commission, who stated:
While there are low performing students across the
socio-economic spectrum, justifying some attention to raising performance per
se, there is evidence that low SES kids underperform relative to their
potential across the board, justifying the targeting of this group as a whole.
Hence COAG’s attention to schools in lower SES areas would seem warranted,
especially for those performing poorly relative to other schools with similar
student populations. (This can now be revealed through NAPLAN data — with ‘My
School’ transparency upping the ante for action — again illustrating the
perversity of opposition to reporting by those professing to hold the interests
of students paramount.)[17]
Dr Ben Jensen also supported the greater emphasis on
low-performing students:
If we look at the very low-performing students, these are the
students who are most in need of help and assistance and who unfortunately are
a greater cost to society later on and who miss out on opportunities later in
life. I actually see the value in NAPLAN and My School identifying those
students who are below minimum levels of literacy and numeracy, because they
will struggle the most later in life.[18]
Dr Jensen provided the following statistics to support the
attention on students falling behind:
We have a real problem in our country of letting students
slip further and further behind. If you look at the NAPLAN data, in year 3
roughly eight per cent of students perform at or below minimum levels of
writing literacy. By year 9, it is over 30 per cent—nearly a third of students
are performing at or below minimum levels of writing literacy. That is an
incredible percentage of students. Not only do we fail to address and help the
students who are performing poorly at a young age, we actually let others fall
to that level as well. So I think there is a good case to be made that, as soon
as a student falls to those levels—particularly the younger they are, because
if we get them young we can help them much more efficiently, much more
effectively—we should institute specific programs to help them. This could be
special assistance or simply identifying to the school principal, ‘You have
these students who really need your help.’ I think that can work effectively;
it has in other countries—very high-performing countries which do not have the
same problems of very low performance that we have.[19]
Examples of the strategies used in other countries to assist
students starting to fall behind were provided by Professor Geoff Masters who expressed
the view that:
...there is a general lesson there about the importance of
identifying and catching children who are beginning to slip behind in their
learning and getting them back onto a trajectory.[20]
NAPLAN was supported as a major step in the education system
by Professor Peter Ridd at James Cook University in Queensland, who reported
that he has noticed a decline in the standards of students on entry into university
in the areas of maths and science. Professor Ridd cited a review undertaken by
ACER in Queensland which found:
In summary, there appears to have been a decline in the
relative performance of Queensland students in maths and science over a period
of decades. In the period 1964 to 1995, the absolute decline in lower secondary
maths achievement appears to have been greater than in any other State, and to
have been the equivalent of about two years of schooling.[21]
Professor Ridd emphasised that regular and consistent
monitoring and assessment is essential to address the decline in standards over
the years and welcomed the development of NAPLAN which, when used as intended,
is a 'successful and useful tool'.[22]
In practical terms Mrs Sharyn Lidster from the Tasmanian
Department of Education reported on how providing extra assistance to struggling
schools occurs:
All of the measures are used as part of schools providing an
annual school improvement report where they identify where they need to
improve. That is done in conjunction with the regional general managers. Then
the support is provided where schools determine that they may need to put in
extra professional development. For example, if their reading results for a
particular group of students were not as expected, additional support and
additional programs will be provided to the school so that they can improve
those outcomes for the students. So they do not submit; it is part of the
system, part of the conversations and part of the budget process.[23]
NAPLAN is just one of a range of
measures
NAPLAN is one indicator of student performance. It is not an
indicator of school performance on its own. As noted by Ms Trimper, APPA,
'NAPLAN is only one piece of the educational jigsaw; it is only one piece of
the story about a school.'[24]
This view was supported by Mrs Sharyn Lidster, Department of
Education Tasmania, who emphasised that NAPLAN is just one of a suite of
measures used to look at performance.[25]
Professor Masters emphasised
the need to protect NAPLAN as a good measure of student performance and to make
clear the original purpose of providing information to schools, parents and
systems. He added that it will be important to develop better measures of
teacher, school and system performance.[26]
Margins of error are recognised
Professor Geoff Masters
acknowledged the current limitations of NAPLAN tests:
Of course, they are point-in-time tests, so they are limited
in that sense. They only assess part of what is important in schools. There is
inevitably a degree of imprecision, measurement error, around the estimates
that they provide.[27]
ACARA has acknowledged the measurement errors inherent in any
tests and explained how this is being addressed:
...NAPLAN are short tests, so the error band for an
individual is wider. We can improve that in two ways: increasing the testing
time, which is not very popular; or we can have a test where the items are more
focused around the abilities of the individual student and then your testing
becomes more efficient and the error becomes much smaller. That is one of the
things that ACARA wants to look at for the future along with of course aligning
the test with the curriculum...We are also keen on the notion of seeing if we
can improve the efficiency of the testing to reduce margins of error. That is
something that we want to look at.[28]
ACARA also noted that the first version of the My School website
recognised the margins of error inherent with any testing, and the next version
will display this even more clearly.[29]
Timing
In response to concerns about the timing of the NAPLAN test, Dr
Peter Hill, CEO of ACARA provided the following explanation:
NAPLAN will never work as a diagnostic test. A diagnostic
test should be administered by the teacher so that they can take action the
next day. No national survey can return all of that the next day very easily.
There are ways in which we can provide resources—for example, in computer
adaptive tests that can provide feedback instantly. Perhaps in the future those
sorts of resources can be made available. There is a lot of interest in doing
that, but that is not a part of NAPLAN at this stage.[30]
Government senators view
Government senators were concerned to see signs during the
inquiry that that the original purpose of NAPLAN has either not been well
communicated[31]
or its importance exaggerated by some. It is essential to keep the test in
perspective. Its purpose is to provide a snapshot of student performance in
order to focus on improvement. It is not a diagnostic assessment which looks at
the reasons why students are not performing and which requires immediate
feedback.[32]
The limitations of NAPLAN as with other testing are acknowledged. It is a point
in time test with margins of error and it should be seen as just one
information source within broader contextual information about a school.
Government senators welcome the new National Partnership
Agreements with governments and schools which will allow resources to be better
targeted to disadvantaged schools and students which need the most support.
Government senators note that distribution of the $2.6 billion committed
through these partnerships to improve the quality of teaching, improve literacy
and numeracy and provide additional assistance to disadvantaged schools[33]
relies on NAPLAN tests revealing where needs are the greatest.
In response to the committee majority recommendation to use below-average
NAPLAN test results to trigger the provision of assistance to schools and
students, government senators note this is already occurring. The focus of
NAPLAN testing is to provide assistance to schools and students which are
identified as requiring it. As reported to the committee by Mrs Sharyn Lidster from
the Tasmanian Department of Education, mechanisms are already in place to
facilitate this. The committee also heard that teachers use the results to
evaluate their teaching programs and identify students requiring additional
support. As for more immediate feedback, this is not the purpose of NAPLAN and
government senators note that consideration of resources to provide this more
immediate feedback to teachers is underway (see below).
Learning from overseas experiences
Ms Trimper, APPA, advised that research from the US and Britain
shows that when assessment programs become high stakes there are unintended
negative effects on the quality of teaching and learning. She emphasised that APPA
wants to ensure that NAPLAN does not go down the same path.[34]
Dr Peter Hill, Chief Executive Officer, ACARA responded to
comparisons made by some with the UK and US experiences:
...Those two countries got into accountability in terms of
having really-high-stakes assessment well ahead of others, often with quite
severe consequences for poor performance. I was in the USA when President Bush
introduced No Child Left Behind and if you failed to meet satisfactory progress
targets then there was a consequence: you were on watch the next year; in the
second year there was one set of consequences and in the third year another set
of consequences. This, of course, led to a lot of contention and debate. Now we
are seeing some of the research into whether it really did make a difference
and so on...
...Australia has not made the mistake of the UK and the USA
of having those extreme high-stakes consequences. We are in a position where we
can have a good look at the data and say, ‘Let’s look at other schools that are
doing a good job. What can I learn here?’ within an environment where you are
not expecting to be closed down next year for poor performance, and to really
look at how you can improve.
The answer I would have to those critics is that we are in a
different context to the research that you are reading. We are in a situation
where I think we are having a more intelligent approach to accountability and a
more intelligent approach to the use of that data, particularly in not
comparing schools against others where it is an illegitimate comparison but
having comparisons with schools that have similar students.[35]
Dr Hill stressed to the committee that Australia has learnt
from the UK and US experiences and is not going down the same path.[36]
Responding to the concerns about NAPLAN being a high stakes test, Dr Hill added:
I think the education community is coming to grips with a new
world in which transparency is at a level that we have never seen before and
accountability is of a kind that we have never seen before. We are all coming
to grips with what it means to have intelligent accountability in all of this.
I do understand the feelings of many people who have seen what has happened in
the USA and the UK, which was not, in my mind, very intelligent in terms of
accountability...[37]
ACER also noted the experiences overseas which are often cited
to argue against standardised testing. It argued that this occurs without
acknowledgement of the differences in the testing regimes or in the educational
circumstances in other countries. For example Finland is often referred to:
...without acknowledging the very different features of the
Finnish education system, including the fact that Finnish teachers are
significantly more highly qualified than Australian teachers and Finnish
schools are much more alike in composition and outcomes than Australian
schools.[38]
Government senators view
Government senators stress that NAPLAN is not the same high
stakes test that occurs overseas where penalties are applied for poor
performance. Government senators highlight the evidence from Dr Peter Hill, CEO
ACARA, who emphasised that Australia has not made the same mistake as the UK
and USA which have negative consequences for their testing. Government senators
emphasise the intention here is to identify where support is required for
students and schools and ensure they receive it.
Improvements to NAPLAN testing
Government senators note that as NAPLAN is now in its third
year it is possible to report on the growth of student achievement which adds
an important element to the usefulness of the tests. Government senators
recognise that NAPLAN tests and the ways in which results are analysed and
reported are the subject of ongoing improvements, and ACER listed a number of
areas under consideration.[39]
Dr Hill also told the committee that resources to provide
more immediate feedback for teachers are under consideration:
There are ways in which we can provide resources—for example,
in computer adaptive tests that can provide feedback instantly. Perhaps in the
future those sorts of resources can be made available. There is a lot of
interest in doing that, but that is not a part of NAPLAN at this stage.[40]
Government senators view
Government senators note the committee majority
recommendation to expand NAPLAN testing. Government senators disagree with this
proposal as increased large‑scale cohort testing is not the next step
that needs to be taken. The next step is providing teachers with better
diagnostic tools to address the needs of individual students. Government
senators note that the government has committed to developing a national online
assessment and learning bank for students, parents and teachers to provide a
sophisticated diagnostic assessment of each student’s strengths and learning
needs.[41]
Allegations of cheating have been dealt with seriously
Allegations of cheating on the NAPLAN tests[42]
have been thoroughly investigated[43]
and actions taken to stop any recurrence of security breaches.
In October 2010 at Senate
estimates hearings, Dr Peter Hill, Chief Executive Officer, ACARA, provided an
update on action underway:
...We have two that are under investigation in terms of
security breaches. We have 10 under investigation that involve cheating and
four that involve breaches of the protocols of administration. They are of
varying degrees of seriousness. Obviously, the ones that involve cheating are
the ones that are of concern to us, and there are 10 of those under
investigation, but there have only been two that have been confirmed at this
stage, so we are talking altogether about 12 allegations that have either been
confirmed or are still under investigation.
...These are matters which are being done by the test
administration authorities in each state or territory. Because they can have
very serious consequences for the individuals concerned, they do tend to take
rather a long time to finalise, so we cannot put an end date on them.[44]
He added that where a child's score may be affected, the
jurisdictions notify parents and may need to withdraw the results. He assured
the committee that the effect of this on the averages for statistically similar
schools or for all schools would be negligible.[45]
ACARA outlined plans to enhance the security of test
administration which include strengthening protocols and embarking on a
'multi-level' communication strategy for 2011 which will develop greater
understanding of the required protocols to manage test materials.[46]
As this action is already underway, government senators do not see the need for
the committee majority's recommendation for additional work in this area.
My School website
NAPLAN testing is complemented by the My School website,
which provides detailed information about almost 10,000 schools. People can locate
statistical and contextual information about schools in their community and
compare them with statistically similar schools across the country.
Background
The website has been developed by the Australian Curriculum
Assessment and Reporting Agency (ACARA), an independent authority that is
responsible, among other things, for publishing nationally comparable data and
contextual information on schools. Education ministers have agreed the Principles
for Reporting on Schooling in Australia which include:
- the protection of individual student privacy;
-
not publishing comparative data without contextual information;
and
- the publication of error margins, caveats and explanatory notes
to ensure accurate information.[47]
The design of the website was endorsed by the Ministerial
Council for Education, Early Childhood Development and Youth Affairs (MCEECDYA)
in September 2009 and was launched on 28 January 2010. It provides profiles of
almost 10,000 schools, contextual information and NAPLAN results that can be
compared with results from statistically similar schools.[48]
ACARA reported that as at 25 June 2010 the website had received 2,445,308
visitors and 3,368,847 visits.[49]
The importance of context
Cognisant of the concerns raised about the website, Dr Peter
Hill, ACARA, cautioned that the information on the website about each school
needs to be understood within a broader context of all the other information
available about the school, including factors such as student characteristics,
percentage of Indigenous students and location.[50]
In response to concerns that the first version of the website may have been
made available too early, he stated that he believed the publication of the
data improved the quality of the data:
I believe that until we published we were not going to get
the data. Since we published there has been a scramble to get extra data. So
things have moved along very rapidly. My view is that if we had waited for that
to be in place we would still be waiting in five years time.[51]
Professor Geoff Masters, CEO of
ACER, also emphasised the important distinction between measures of student
performance and measures of school, teacher or system performance:
What we have done in Australia to date is that we have worked
hard on getting very good measures of student performance. They can be, as I
said, point-in-time measures. We can look at trends over time because of the
way we calibrate the tests statistically. We can look at trends in a system or
even in a school over time. We can look at the growth that is occurring—for
example, between year 3 and year 5 or between year 5 and year 7. But all of
these, whether they are point-in-time measures of status, growth measures or
measures of trends, are measures of student performance.[52]
Professor Masters added:
I think we have been pretty careful with the My School
website to say that what we are reporting on My School is measures of student
performance. We are not trying to report a measure of a school’s performance.
It is true that an attempt is made to make the information perhaps more useful
by comparing schools that have similar student intakes, but our focus with My
School is still very much on the reporting of student achievement, not trying
to construct measures of a school’s performance to get a number for a school.
Professor Masters also emphasised the need to understand the
information in context:
I think the thing we have to guard against is giving it all
the value, giving it a priority above everything else. I would be arguing for
reporting NAPLAN results, as we are, but also providing rich information about
other aspects of the school’s circumstances as well as the students’
performance.[53]
ACARA also responded to
concerns about My School using the NAPLAN data, which assesses student
performance, to provide information on schools:
It is quite acceptable to get an overall measurement of the performance
of students in the school by averaging the performance of the students within
it. That is done in every international survey of educational achievement. We
compare the performance of countries—for example, in PISA—by doing the same
thing. We aggregate up from the student level to a national level and indeed to
an international level. So aggregating the responses of individuals within a
unit is quite acceptable in measurement terms and that is what is done in, for
example, educational research. All educational research is based on aggregating
performance of individuals within the unit.[54]
Facilitating conversations
The My School website provides valuable information to
facilitate conversations at many levels. Dr Ben Jensen told the committee that
he supports NAPLAN and the My School website as:
...this website has highlighted both the positive and
negative issues within schools, and it is great that it has encouraged these
conversations to take place – not just engaging parents and families with the school
but facilitating discussions internally within schools.[55]
Mrs Sharyn Lidster, Department of Education, Tasmania also
reported the website information is leading to increased conversations between
parents and teachers:
It certainly has increased the level of discussion. Of
course, our media have also jumped in and made some simplistic summaries of
that information, but I think that the rich conversations that are happening at
a local level between parents and the schools are where a lot of the positive
outcomes are coming from in publicly reporting the information.[56]
Mrs Lidster added:
I can only speak anecdotally in relation to parents’
engagement with their local schools. Anecdotally, there have been a lot more
conversations around that. I can say that, when you provide a whole suite of
information, there is probably no school that is hitting the top marks in every
area. That is the nature of any endeavour, especially something as complex as a
school and providing an education for students. There will be areas of
strengths and areas of weaknesses. That is where it is valuable to share it
publicly with parents, so that they can have that discussion with the school
around how they can improve it. Because it is a partnership, things like
attendance, for example, are not in the total control of the school. They can
encourage attendance but in actual fact parents have a direct responsibility in
relation to attendance. There are great conversations happening in relation to
it.[57]
Increased transparency
While advocating for including
a value-added score on the website,[58]
Dr Ben Jensen acknowledged the greater transparency provided by the My
School website:
My School has been a dramatic increase in transparency in
school education. That has been undertaken in an industry or a sector that has
not had a lot of transparency—particularly when you consider that this year we
are going to put financial information on it as well. That is a huge increase
from where we have been. I think whenever you have that sort of change you are
going to get this sort of emphasis.[59]
Dr Jensen spoke in more detail
about increased transparency leading to greater student improvement:
...Overall what you are getting is a gradual decline over the
years and an inability within a system to effectively help those students at or
below minimum levels. As I said before, clearly the students at year 9 who are
at or below minimum levels of writing literacy are still improving in absolute
terms but it is the rate of improvement—this is where I greatly support the
transparency of My School. We are going to have some of the greatest
improvements by shining a light on the fact that we have this issue because
this data is not well known. We are only beginning to develop effective
programs to address these issues. That is why I would advocate that we need
greater systems that immediately flow to the students who are performing at
these levels. The sorts of equity programs which I would advocate have been
successful in countries only focus on equity and not on absolute height. These
are the programs that very high-performing countries have.[60]
Government senators view
Government senators note that the intention of COAG was for
school performance data to be published in the context of broader information
about a school's students, teachers and resources.[61]
Therefore, along with the NAPLAN results, each school has a self-written
profile page where it describes the mission and values of the school, the
number of students, number of teacher and non-teaching staff, the attendance
rate and any additional programs and support in place for students. Government
senators emphasise that the data on the website should be considered along with
an understanding of the context of the school and the composition of the
student body.
Government senators agree that apart from the uses at the
system and school level, the information provided by NAPLAN via the My
School website is acting as a useful tool for parents to engage directly in
conversations with teachers and ask questions about what is happening in their
school.
Further development of the My
School website
It was always intended
that the My School website would be developed in stages and that
subsequent versions would be improved by the additional of information as it
became available.[62]
To this end, ACARA advised that it was asked by education ministers to
investigate proposals for enhancing the website including:
- school financial data;
- nationally comparable senior secondary information;
- satisfaction with schooling;
-
student population indicators;
-
growth data on literacy and numeracy achievement;
- teaching staff and levels of expertise;
- using student-level data to compute the Index of Community
Socio-Educational Advantage (ICSEA)
- other enhancements to ICSEA
-
reporting of results; and
- action to minimise misuse of My School data.[63]
Making more
information available was advocated by ACER, which stated that the website 'has
the potential to become a valuable single source of comparable, public
information about every Australian school'.[64]
Government senators agree with advice from ACER that '[m]any of
the current concerns about My School are likely to be addressed by
broadening the range of information provided and increasing levels of user
choice'.[65]
Government senators note that a working party made up of
educational experts including literacy and numeracy specialists, principal
organisations and representatives from the Australian Education Union and
Independent Education Union of Australia was formed. It is providing advice on the
use of student performance data and other indicators of school effectiveness.[66]
Recommendations from the working party were considered by MCEECDYA on 15
October 2010. Ms Trimper, APPA, who is part of the working party, confirmed
that the recommendations developed by the working party will go a long way to
addressing the concerns that have been raised about the website.[67]
Government senators note that the substantial amount of work
undertaken by the working party and the ability of My School version 2
to address concerns raised about the website has been given limited acknowledgment
in the committee majority report. Government senators support the view
expressed by Dr Peter Hill, CEO, ACARA, who told the committee that he believed
the publication of the data has improved the quality of the data.
Making more information available
Following the meeting on 15 October 2010 with state and
territory education ministers, the Minister for School Education, the Hon Peter
Garrett MP, reported that ministers supported most of the enhancements proposed
by the working party and that school finances and richer community detail will
now be reported. He stated:
This is a really big reform for us and it's important that we
get the website right. We've asked for some further work to be done on how the
site will look and details about how it will function. But certainly we're very
pleased that we're starting to see additional information coming to MySchool.
Additional information that will enable parents, teachers, the communities and
others to get a good sense of the educational progress that is being made by
their students and by their schools right across the country.[68]
The minister added that the second iteration of My School
will provide a leading information source for parents, teachers and school
communities. It will also provide better information for governments to
consider policies to best support education.[69]
My School version 2 will address many of the issues
raised in submissions. Some specific examples are outlined below.
Measures to address the misuse of
data
Government senators note that to address the misuse of data, for
example, using it to create 'league tables', on 15 October 2010 MCEECDYA endorsed
implementation of the following measures to protect the integrity of data on
the site and the collection of direct student data:
- a 'click wrap'[70]
requiring users to indicate their agreement up-front to terms and conditions of
use of My School data;
- a tool to deter automatic scraping of data from the website.[71]
These measures will be implemented prior to the release of
version 2 of the website.[72]
Government senators view
Government senators emphasise that the My School
website is not a league table and support mechanisms to prevent and deter the
practice of using My School data to create such tables. My School
allows people to go to the website and view the NAPLAN results in an
appropriate context.
ICSEA changes
APPA President Ms Leonie Trimper emphasised the concerns that
the ICSEA values caused for their members. She suggested that to legitimately
compare schools you need accurate details of the children at the school itself
rather than details from a census data collection district.[73]
The use of Census Collection District (CCD) data for ICSEA
values was considered by MCEECDYA on 15 October 2010. ACARA acknowledged that
the use of this census data resulted in anomalous ICSEA values in a small
number of cases. At the 21 October 2010 Estimates hearing Dr Peter Hill,
Chief Executive Officer, ACARA, told the committee that ministers have agreed
to move from census-based data to a model where information is obtained from
parents:
...We have looked at it and now we have a new formula, which
will mean that the ICSEA value of at least 70 per cent of schools, the
socio-economic status component, will be based on direct student measures.
...It improves the predictor of validity by seven per cent.
It means that we will not get the anomalies we have had in the past where the
census collection district does not properly characterise the nature of the
students who live in them.
...The only instances where we will revert to census
collection data is where we do not have the individual level data and that
relates particularly to schools in very remote areas of the Northern Territory
or in cases where the number of students for whom we have this direct data is
too small to get a reliable fix.[74]
Dr Hill added that the new formula will be used when the
revised website is released in December 2010.[75]
The use of individual student information on parental education and occupation over
census data was supported in evidence to the committee.[76]
Enhanced reporting
The provision of more information, particularly qualitative
and contextual information, was supported by witnesses.[77]
Government senators note that MCEECDYA has agreed to expand the contextual
information about schools by publishing the percentage of students with a
language background other than English. Principals will also be able to include
more information about their school. In addition, information on student
absences, withdrawals and exclusions from NAPLAN testing will be more
prominent. The website will show the growth in learning for students who took
the test in 2008 and 2010 and were in the same school both years.[78]
Schools will also be able to provide a commentary on their results. Comments
will be collected and reported in the first half of 2011. Further enhancements
are also planned for My School version 3, such as nationally comparable
senior secondary information and teaching levels of expertise.[79]
Government senators point
out that after this year's NAPLAN testing we are now in a position to see
progress of the same children two years later, for example seeing how the grade
3 students tested in 2008 are now doing in grade 5 in 2010.
Enhanced search facilities
APPA called for a more interactive website where more
detailed comparisons are possible.[80]
Government senators note and support the recommendation from ACER to develop
other ways of comparing student results across schools which would allow users
to choose their own comparisons.[81]
Government senators note that for version 3 of the website,
MCEECDYA has agreed that ACARA will investigate enhancing search facilities and
filters which would allow users to refine their searches for like and
statistically similar schools.[82]
Need for better communication
Mrs Sharyn Lidster, Department of Education Tasmania told
the committee that a lot of education is needed to support schools in making
best use of the data and being able to talk about their performance with their
communities.[83]
ACARA endorsed this view, and Dr Hill said that improving communication of the
information would be a focus for the next version of the website.
There is always a big communication issue with presenting any
form of complex information. We are conscious this time around—because we have
a little bit more time than we did last year—that we need to put a lot of work
into communicating ahead of the release about what the website will look like
and the sorts of caveats that were in the first version. What we said in the
first version is: if you see one result that might not be to your liking, that
in itself may not be significant. Look at the previous year and then, if you
have three years, look again. Certainly, if you have an indication that there
is a problem three years in a row, you should take it seriously. One poor
result does not necessarily mean you have a problem. It could be an aberration.
That particularly applies to small schools. We would say that, particularly for
small schools, you have to be very careful in interpreting these roles. Indeed,
the errors in measurement around those estimates will be larger for those
schools. The report on the current website and on the new one makes that very
clear.[84]
Government senators note the inherent difficulty in presenting
complex information and the intention by ACARA to put more work into better
communicating the limitations of the data on the website.
Availability of financial data
One of the enhancements agreed by state and federal
education ministers is requiring schools to list financial information. This will
include 2009 recurrent income, disaggregated by source of funding (Commonwealth
government; state/territory government; school initiated fees, charges and
voluntary contributions; and other sources such as parent or third party
initiated). It will also include 2009 capital expenditure broken down by source
of funding.[85]
The MCEECDYA Communiqué from the 15 October meeting noted that school financial
information will be reported from 2008 to 2010 when the new version of the
website is available in December 2010.[86]
This issue has received some media reporting, with the WA
Primary Principals Association President Stephen Breen expressing concern that
published financial data may present a misleading picture of how much money is
spent on a student. He argued that public schools would see their costs
inflated because they would also have to include the cost of education department
head office bureaucrats. He added that he would like to see the data include
school assets such as real estate.[87]
The WA Director of the Association of Independent Schools, Valerie Gould, was
concerned that publishing such financial data would lead to 'simplistic'
comparisons such as linking school wealth to test scores. She added that it
would be difficult to compare schools across difference sectors as costs and
expenses are not treated in the same way:
For example, independent schools had to account for
depreciation and insurance that State governments covered for public schools.[88]
These concerns were echoed in Victoria, where it was
reported that the disclosure of private income will be misleading as:
They must cover costs that individual government schools
don't, such as salaries, cleaning, maintenance, capital works, depreciation and
loan interest.[89]
The Chief Executive of Independent Schools Victoria argued
that schools would not be able to put their income and expenditure in context.[90]
The Executive Director of the Association of Independent Schools, NSW, Mr Geoff
Newcombe, stated that he did not believe private schools should have to report
income from all sources:
I don't think there is a lot of relevance in disclosing a
school's assets...It would be almost impossible to compare. We are looking at
the recurrent costs of education children.[91]
The Australian Education Union argued in favour of
disclosing all financial information such as trusts, foundations, bequests,
share and property portfolios to provide an accurate account of the financial
resources of schools and increase transparency.[92]
Dr Ben Jensen also supported the publication of financial data:
I think that when we introduce financial data onto My School
we are going to start a lot of conversations about whether our resources are
being effectively spent to help students. That is a conversation that is long
overdue in school education. I think that parents will ask—and I am not just
talking about parents with children in independent schools—‘I am spending X for
this; am I really getting value for money?’ But I also think we will get
conversations such as ‘My state government is spending this per student; why
are they performing at a lower level than students in other jurisdictions where
the governments spend less?’—those sorts of conversations.[93]
The office of the School Education Minister Peter Garrett responded
to the concerns raised by stating that representatives from independent and
catholic schools have been involved in working out how to collect financial
data so that it could be compared.[94]
Government senators view
Government senators note the financial data to be captured on
My School version 2 is a good start but will not capture accumulated
surpluses, assets, trusts or foundations. In the interests of providing more
information, government senators believe that there should be full disclosure
of financial assets including assets, foundations and investments, otherwise
true comparisons will not be possible. There are limited obligations on private
schools in return for public funding. Possibilities could include being obliged
to open facilities to government schools and offer scholarships for
disadvantaged and Indigenous students.
If non-government schools continue to expect a share in
federal funding then full financial disclosure in the interests of the tax
payer and the better allocation of resources must be required. If
non-government schools do not wish to comply with full financial disclosure,
then public funding should not be provided.
Recommendation 1
Government senators recommend that in the interests of
transparency, accountability and facilitating meaningful comparisons, the My
School website capture full disclosure of financial assets. Those schools
who do not agree to this requirement should not receive public funding.
Problems with value-added measures
Government senators note the Coalition recommendation to
revise the My School website by publishing a value-added measurement of
school performance rather than the raw performance data results.[95]
The committee received an explanation of value-added
measures from Dr Ben Jensen[96]
and Professor Geoff Masters.[97]
Professor Masters noted that the term value-added can mean many different
things. It could mean the growth occurring or an attempt to take into account
the background of the students, which he understood to be the aim of Dr Jensen.
He described several problems with this approach:
One is the attempt to interpret residuals that I described
before, where the residual is not just the influence of the school,
necessarily; it could be all sorts of other influences. Another issue I have is
that in that process you potentially lose the performance of the students
themselves, so what becomes important is how much better or worse the school
did than you predicted in your regression analysis, and you may end up saying
well this school performed as well as expected, but in an absolute sense the
literacy and numeracy levels could be unacceptably low, but it is as well as
expected given their socioeconomic background. There is a bit of a risk in this
approach of not recognising the absolute levels of achievement in the school
and thinking everything is fine—I guess I have a philosophical problem with
approaches that end up drawing conclusions like ‘students in this school are
doing as well as expected given their backgrounds’. I can understand why people
say that, but there is a fine line between explaining and making excuses.
Sometimes I worry about that line.[98]
ACER pointed out that the
current comparisons used are superior to the approaches used in other countries
and outlined the problems with the school comparisons used in England which are
advocated by the relevant Grattan Institute report:[99]
...school comparisons in England are based on 'contextualised
value added' (CVA) measures which are constructed by first predicting how
students in a school will perform from their backgrounds and then calculating
the difference between the predicted and actual performances of students in
each school. This method has the apparent advantage of providing a single
number for each school enabling it to be compared with every other school in
the country. These CVA measures are residuals that are assumed to reflect the
contribution of the school, but in fact also reflect any other influences not
taken into account in making the 'prediction'.[100]
Importantly,
Government senators recognise that this method also has the disadvantage of
obscuring students' actual levels of performance. As noted by ACER, when using
such a system, 'students in a school can be judged to be performing as well as
expected even when their absolute achievement levels are unacceptably low'.[101]
Government senators also note the critical point made by ACER that, unlike the
system in use in England, My School does not compare every school with
every other school but encourages comparisons only between schools with similar
student intakes.[102]
Conclusion
Rather than more
convoluted analysis, clearly the solution to the issues raised with the website
is providing more information, not less. Government senators welcome the calls
for more contextual information to be provided by schools on the My School
website and note that this was always the intention over time. The launch of
the My School website in January 2010 was an important step forward in
reporting the measurement of student performance and progress as well as
increasing transparency and accountability. My School version 2 will be
the next step in addressing these calls for more information to be provided. Government
senators note that the next version is to be available in December 2010.
My School places the school at the centre of
reporting, with its own home page containing a range of rich information and
context about the school. In contrast, reporting overseas often places the
school as a line in a table. Government senators note and support the intention
to continue building on and improving the My School website to make more
information available and improve and update the information that is there.
My School has shown there are schools with student
populations of similar social backgrounds which are doing very different
things, with many achieving unexpectedly good results. It has also highlighted
schools which need additional assistance, which enables government to allocate
additional resources to struggling schools.
Government senators note the success of the working party announced
to address stakeholder concerns in relation to the My School website.
Government senators emphasise that the debate on NAPLAN
assessment and the My School website is just one part of the broader
education reform agenda being addressed in partnership with the states and
territories. These broader issues include the development of the national
curriculum, the school funding review underway, providing additional assistance
to disadvantaged schools and improving the quality of teaching.[103]
Senator Gavin Marshall Senator
Catryna Bilyk
Deputy Chair
Navigation: Previous Page | Contents | Next Page