Chapter 3
My School website
In education, good
decision making is facilitated by access to relevant, reliable and timely
information. Dependable information is required at all levels of educational
decision making to identify areas of deficiency and special need, to monitor
progress towards goals, to evaluate the effectiveness of special interventions
and initiatives and to make decisions in the best interests of individual
learners.FF[1]FF
Background
3.1
In 2008, the Council of Australian Governments (COAG) agreed that
greater transparency and accountability for school performance was essential 'to
ensure that every Australian child receives the highest quality education and
opportunity to achieve through participation in employment and society'.FF[2]FF It also agreed that high‑quality
reporting is important for students, parents, carers and the community and
should include:
- streamlined and consistent reports on national progress,
including an annual national report on the outcomes of schooling in Australia;
- national reporting on performance of individual schools to inform
parents and carers and for evaluation by governments of school performance; and
- provision by schools of plain language student reports to parents
and carers and an annual report made publicly available to their school
community on the school’s achievements and other contextual information.FF[3]
3.2
To provide the public with information on each school, COAG agreed that the
Australian Curriculum, Assessment and Reporting Authority (ACARA) would:
...be supplied with
the information necessary to enable it to publish relevant, nationally
comparable information on all schools to support accountability, school
evaluation, collaborative policy development and resource allocation.FF[4]
3.3
ACARA would then provide information on each school and this would
include:
...data on each
school’s performance, including national testing results and school attainment
rates, the indicators relevant to the needs of the student population and the
school’s capacity including the numbers and qualifications of its teaching
staff and its resources. The publication of this information will allow
comparison of like schools (that is, schools with similar student populations
across the nation) and comparison of a school with other schools in their local
community.FF[5]
3.4
The committee majority notes that COAG intended school performance data
to be published in the context of broader information about a school's students,
teachers and resources.FF[6]
3.5
In August 2008 the Ministerial Council on Education, Employment,
Training and Youth Affairs (MCEETYA) agreed that an Expert Working Group (EWG)
convened by the Australian Government would provide a report to ministers on
relevant measures to guide school evaluation, accountability and resource
allocation. In late 2008, EWG commissioned the Australian Council for
Educational Research (ACER) to provide advice on national schools data
collection and reporting for school evaluation and resource allocation. The
report recommended the use of NAPLAN results as the basis for the comparative
performance of schools.FF[7]FF It noted that few
countries have developed measurement scales along which gain and growth can be
measured for all students. The report concluded that NAPLAN is an effective way
to assess whether a school is making a difference in a student's numeracy and
literacy skills by measuring improvement across the years:
The NAPLAN
measurement scales enable status, gain and growth to be measured across Years
3, 5, 7 and 9 and in this sense represent world's best practice in the
measurement of student progress.FF[8]
3.6
Regarding public reporting, the Australian Council of Educational
Research (ACER) report recommended that:
For the purpose of
providing public information about schools, a common national website should be
used to provide parents/caregivers and the public with access to rich
information about individual schools.FF[9]
3.7
The ACER report noted that it would be important that characteristics
that are known to be correlated with student outcomes are taken into
consideration:
Research consistently shows a correlation between students'
socio-economic backgrounds and their level of school attainment. For this
reason, the socioeconomic backgrounds of a school's student intake also must be
taken into consideration in any evaluation of the school's performance...
The socio-economic
backgrounds of students in a school can be measured either at the level of the
school, eg, using data from the ABS census collection districts for the home
addresses of the students attending the school) or by aggregating information
about the SES backgrounds of individual students in the school. FF[10]
3.8
The ACER report was considered by MCEETYA in April 2009, and the
ministerial council agreed that, from 2009, ACARA would be responsible for
publishing relevant, nationally comparable information on '...a common national
website [which would]...provide parents/caregivers and the public with access
to rich information about individual schools'.FF[11]FF This would include
publication of the 2008 NAPLAN data and associated contextual information. The
information would enable comparison of each school with other schools serving
similar student populations around the nation and with the best-performing
school in each cohort of ‘like schools’. MCEETYA noted that this information is
intended to support accountability, school evaluation, collaborative policy
development and resource allocation.FF[12]
Use of ICSEA values
3.9
As noted by ACARA, the best way to compare the performance of schools in
the NAPLAN tests would be to find groups of schools with students of similar
abilities on commencing school. However, no such measures of starting abilities
are available nationally.FF[13]FF The data used instead is
Australian Bureau of Statistics (ABS) census data which is also used to
allocate funds to non-government schools and to identify enrolling students from similar social backgrounds.FF[14]
3.10
Noting that ACER had emphasised the clear links between a student's socio‑economic
background and educational outcomes, ACARA decided to use socio‑economic
indices as a starting point to build a comparative tool for student
populations. ACARA noted that the ABS produces four indices of socio-economic
status, the Socio-Economic Indicators for Areas (SEIFA). Although the indices
correlate positively with student achievement they were not designed to predict
educational attainment. Therefore a new index was developed, the Index of
Community Socio-Educational Advantage (ICSEA):FF[15]
ICSEA uses the SEIFA
variables and school data to create an index that best predicts schools'
average performance on NAPLAN tests. The variables that make up ICSEA include
socio-economic characteristics of the small areas where students live (in this
case an ABS census collection district (CCD)), as well as an index of
remoteness, and the proportion of Indigenous students enrolled at the school.FF[16]
3.11
The steps taken to calculate an ICSEA value for each school are detailed
in the ACARA submission.FF[17]FF ICSEA places schools on
a numerical scale, for example:
- a school in a regional town with a student population drawn
largely from relatively disadvantaged households may have an ICSEA value of
about 850;
- a school in a metropolitan area which draws its students from
relatively advantaged households may have an ICSEA value of about 1150; and
- a school in a remote Indigenous community may have an ICSEA value
of about 540.FF[18]
3.12
ACARA noted that some schools will not have an ICSEA value because of
the nature of the student population—for example, a school solely for children
with intellectual disabilities. ACARA also noted:
In a small
proportion of cases, ICSEA may provide an inappropriate measure of the
socio-educational level of the school. This can occur in instances where there
is a mismatch between students' actual levels of socio-educational advantage
and that of the CCD values associated with their addresses. An example would be
remote schools where the ICSEA values are inflated where a mining community is
located in an otherwise disadvantaged remote community.FF[19]
3.13
Statistically Similar School Groups (SSSGs) were checked with state and
territory departments and non-government sector authorities prior to the My
School website going live. As a result of this checking, the ICSEA values
of around 650 out of approximately 10,000 schools were revised before the
website was launched.FF[20]FF These changes were
reviewed by an expert panel. The use of the ICSEA index on the website was
endorsed by MCEECDYA in September 2009.FF[21]
Publishing contextual
information
3.14
ACARA is required to publish contextual information on schools on the My School
website. Education ministers agreed the Principles for Reporting on
Schooling in Australia which include:
- the protection of individual student privacy;
-
not publishing comparative data without contextual information;
and
- the publication of error margins, caveats and explanatory notes
to ensure accurate information.FF[22]
Website launch
3.15
The design of the My School website was endorsed by MCEECDYA in
September 2009, and the website was launched on 28 January 2010. It provides
profiles of almost 10,000 schools, contextual information and NAPLAN results
that can be compared with results from statistically similar schools.FF[23]FF Detailed information on
website content is available from the ACARA submission.FF[24]FF ACARA reported that as
at 25 June 2010 the website had received 2,445,308 visitors and 3,368,847
visits.FF[25]
Further development of the My
School website
3.16
The committee majority notes the intention to develop the My School
website in stages.FF[26]FF ACARA advised that it
was asked by education ministers in a series of meetings in 2009-10 to
investigate proposals for enhancing the website by:
- showing school financial data;
-
including nationally comparable
senior secondary information;
- including satisfaction with
schooling;
- showing student population
indicators;
- including growth data on literacy
and numeracy achievement;
- listing teaching staff and levels
of expertise;
- using student-level data to
compute ICSEA;
-
making other enhancements to ICSEA;
-
providing increased contextual
information; and
- taking action to minimise misuse
of My School data.FF[27]
3.17
However, the committee majority believes the response so far is
inadequate to deal with the level of concern in the community and raised during
the inquiry, as discussed below.
Issues raised during the
inquiry regarding the My School website
3.18
The committee majority notes the large number of issues raised regarding
the My School website.FF[28]FF Many of these focused on
misuse of data obtained from the website.FF[29]FF Many also questioned the
reliability of school comparisons based on ICSEA values.FF[30]FF Other submissions stated
that My School might be placing excessive emphasis on 'point in time
measures of student achievement', thereby undermining what they saw as the
intended diagnostic aims of NAPLAN.FF[31]FF
3.19
Some, such as the South Australian branch of the Australian Education
Union, raised serious questions about the My School website's role as
the foundation of the government's 'transparency agenda', explaining that the
government already had direct access to a wide range of information on school
performance and socio-economic status from existing sources.FF[32]
3.20
Teachers also expressed concern, one stating that NAPLAN tests '...are
taking on a life of their own, all because of the unethical way they are being
used'.FF[33]FF These concerns are
fundamental and call into question whether the website in its current form is
presenting useful information.
3.21
The Australian Education Union (AEU), whilst reiterating its in principle
support for effective assessment, stated:
Our concerns stem
from the fact that we believe that the policy mix advanced by the government
has the potential for perverse consequences, perverse consequences which
undermine our capacity to deliver sound educational programs for children... [W]e
are concerned by a series of policy statements and announcements of government
that allocate or attach to NAPLAN something for which it was never designed and
something for which it cannot be used—that is, school performance measurement.
That remains our central point of contention and concern, because stemming from
that is the misuse of that student data for purposes never intended. Therein
begin the perverse consequences.FF[34]FF
The AEU also expressed
its belief that problems have emerged as a result of the creation of the My
School website being '...driven by political imperatives and political
timetables'.FF[35]
3.22
The section below describes in more detail some of the issues raised in
submissions and then outlines the committee majority's suggestions for improving
the presentation and usefulness of the data presented on My School.
Using ICSEA values to group 'like schools'
3.23
A large number of submissions expressed deep dissatisfaction with the
use of ICSEA values to inform school comparisons on the My School website.FF[36]FF The issues focused on
the validity of groupings of statistically similar schools on the My School website.
3.24
The committee majority notes that the ICSEA value is currently
calculated by looking at the community a school is located in, not data
pertaining to the actual socio-economic status of a school's enrolled students.FF[37]FF This means that schools
located in the same community are considered to be at the same level of
advantage or disadvantage, irrespective of the socio-economic backgrounds or
other factors specific to their students. Given that private and selective
schools attract and can pick students who may not always live near the school,
whereas public schools tend to enrol students from their surrounding
communities, the ICSEA-based method of measuring socio-economic status has
understandably attracted substantial criticism.
3.25
The AEU argued that the way in which the ICSEA measure is calculated is
the crux of the problem, saying that the measure does not take into account the
fact that '...some higher income families live in lower SES regions and vice
versa'.FF[38]FF If higher income
families live in areas surrounding public schools, their presence would impact
on census data collected in the area, which in turn '...causes an over estimate
of the SES of government schools and an underestimate of non-government
schools...[and impact on] their placement in the so called statistically
similar schools'.FF[39]FF
3.26
The AEU submitted a number of examples of anomalies in 'like school'
comparisons, including:
- The Kings' School, a wealthy private school in Parramatta, and
Gundaroo Public School, in a small town a short distance from Canberra;
- Blacktown Boys High, Western Sydney, and Alice Springs School of
the Air;
- Terrigal High School, with a student population of 1300, and
Cameron Downs Public School, with only six students in the Queensland outback;
and
-
Haileybury College, a private school in Melbourne, and Cleveland
Street Intensive English High School, a NSW public school which focuses on
teaching students who are from a non-English speaking background.FF[40]
3.27
The Department of Education Tasmania explained the importance of
applying appropriate safeguards around the presentation of test results and
argued that:
The perceived
weakness of the current ICSEA measure is that it is more an index of community
socio-economic status, than an accurate measure of the socio-economic status of
the students who attend a particular school.FF[41]
3.28
Other submissions raised similar concerns about ICSEA being
'...systematically and substantially biased,' and argued that such area-based
indexes, although useful in other contexts, can be '...vulnerable to the
ecological fallacy'.FF[42]FF The ecological fallacy
occurs when individual-level relationships are inferred from aggregate-level
ones.FF[43]
3.29
Dr Mark Drummond, a researcher with a focus on mathematics and
statistics, submitted that only around three per cent of the data used to
calculate ICSEA scores is
...meaningful and
valid data based on the actual families of the actual students at the actual
schools. The other 97 per cent or so of data is meaningless "noise",
based on families and households with no substantive connection at all to the
schools whose ICSEA scores are being determined.FF[44]
3.30
Alternatives to area-based measures of advantage were outlined in
submissions. These included using individual-level measuresFF[45]FF and conducting data
matching between government agencies such as the Taxation Office and
Centrelink, as already occurs for parents who receive benefits.FF[46]FF Such measures would go
some of the way towards remedying statistical problems currently affecting
'like school' comparisons.FF[47]
3.31
Over the course of a series of meetings in the year to June 2010, education
ministers asked ACARA to investigate making use of student-level socio-economic
status (SES). This information is currently available from some states and
territories but not all. ACARA is considering either obtaining family-level
information in all jurisdictions or using the data for those jurisdictions
where it is available.FF[48]FF Education ministers have
also asked ACARA to look into:
- obtaining updated and comprehensive home address data for all
students to improve the accuracy of ICSEA in cases where CCD data is used;
- including within the ICSEA formula a variable to take account of
the effect of language background other than English;
- improving the process for quality assuring ICSEA values for
individual schools and, for those for which CCD data is used, identifying
instances where the initial estimates is inappropriate.FF[49]
3.32
The committee majority notes that education ministers have how endorsed
the proposals outlined above and the new model outlined below:FF[50]FF
An analysis undertaken by ACARA compared the current ICSEA
formula with a new formula based on student level-measures of parent education
and occupation status, as well as considerations for LBOTE and the proportion
of Indigenous students. The result of this modelling indicates that the new
formula will improve ICSEA’s ability to predict school NAPLAN performance, in
addition to having greater face validity. Analyses indicate that there is a 7%
increase in the explanatory power of ICSEA when direct student-level indicators
of parent education and occupation are used.
Under the methodology endorsed by Ministers, data on parent
occupation and education collected directly by schools from parents will be
used (where available) in preference to census data. This will ensure that the
ICSEA value assigned to a school closely reflects the socioeconomic backgrounds
of the students (SES) actually enrolled in that school. ACARA is currently
collecting updated direct parent data from jurisdictions and sectors. At the
same time, ACARA is collecting updated student address data to enhance the
quality and completeness of the indirect parent data (census data).
The new model utilises ‘direct parent data’; however, to
obtain an accurate indication of the backgrounds of students in each school, it
is necessary that a certain percentage of data in each school be available.
Where the available direct parent data do not meet this threshold, or where
updated student address data are unavailable, the school’s ICSEA calculation
will revert to the current calculation based on 2007 CCD information.
Once updated student
information is collected for 2010, the ICSEA will be recalculated using the
recommended approaches. All data included in the revised ICSEA will be tested
prior to broader distribution and will involve extensive consultation with the
ICSEA Expert Panel.FF[51]
Committee majority view
3.33
The committee majority notes the importance of accurately measuring the relative
level of school advantage to ensure that 'like school' comparisons are much
more reliable. The committee majority is alarmed by the evidence it has
received outlining severe shortcomings of present methods of calculating ICSEA
and is disappointed that ACARA launched My School without first
anticipating and addressing at least some of these shortcomings.
3.34
The committee majority notes the approved enhancements outlined above,
but retains some serious concerns about the enhancements' capacity to
adequately address the issues raised. Of particular concern is the fact that
student-level data will only be used 'where available'. The committee majority
would prefer to see a more tangible commitment to replacing the current ICSEA
calculation method for all schools.
3.35
In addition, the committee majority is concerned that—given the fact
that some schools will in future have their ICSEA values calculated on the
basis of student-level data and others on the basis of community-level data as
is currently the case—there is potential for further inconsistency and
distortion if NAPLAN test results for schools from the two groups are ever
compared.
Recommendation 8
3.36
The committee majority recommends that ACARA prioritises the
improvement of the method used to develop like school comparisons and commits
to the introduction of a method based on student-level SES data for all
schools prior to the reporting of 2011 NAPLAN test results.
League tables
3.37
In the absence of proper contextual information and transparent,
professional interpretation of data, there is a concerning potential for
schools to be crudely and unreliably ranked on the basis of raw NAPLAN data. This
was one of the most frequent concerns raised in submissions—that is, that the
publication of raw school performance data on the My School website
could result in league tables of questionable accuracy being published.FF[52]FF
3.38
League tables are assembled in order to rank schools according to
student performance in NAPLAN tests, and were created by a number of media
publications across the country '...within days of the student data going
online.'FF[53]FF
3.39
Private companies also sought to profit from disseminating NAPLAN
results.
In February [2010] a
private company Australia School Ranking established a website from
which it was selling for $97 a 854 page report containing the rankings and
league tables of all kinds of all Australian schools. To its credit, the threat
of legal action by ACARA forced the company to withdraw its report from sale.FF[54]
3.40
More recently another website, Better Education Australia, was
established. The website is funded through advertising and operated to
...[provide] informative
and comparative NAPLAN results and information including private school
scholarships and selective schools to parents wanting to make choices about
schooling for their children. We also provide free service to the community in
Australia and abroad by answering questions via emails. The website is just a
hobby, and not for commercial purposes.FF[55]FF
3.41
The AEU believes that there is no substantive difference between Better
Education Australia, which offers league tables free of charge, and websites
which charge for their reports.FF[56]
3.42
In its submission the AEU also drew attention to what it claimed was
inaccurate information published by media outlets such as the Sydney Morning
Herald, which produced its own state performance rankings by averaging
school mean scores for literacy and numeracy.FF[57]
3.43
The AEU further highlighted the unreliability of using league tables to
rank schools by highlighting the dramatic effects of shifting student cohorts
on particular school results, such as that of Mount Blowhard Primary School,
where top rankings for some Year 3 students '...were over 100 points higher in
2009 than 2008 due to the changing cohort of students'.FF[58]FF
3.44
A submission from the South Australian branch of the AEU pointed to
repeated assurances from former education minister the Hon. Julia Gillard MP,
who categorically defended My School and maintained that it was not
possible to compile league tables using information from the website. According
to the South Australian AEU branch, given the ease with which league tables
were compiled almost immediately following the publication of results on My
School, this '...is either stunning technical incompetence,
political naivety or deliberate misinformation – or perhaps a mixture of all
three'.FF[59]
3.45
The Australian Primary Principals Association pointed to league tables
being a consequence of publicly reporting test results without adequately
taking into consideration factors beyond a school's control.FF[60]FF The NSW Primary
Principals' Association argued that results available on the My School
website 'should not exist in a form that allows comparisons between schools'
because league tables are 'educationally indefensible'.FF[61]FF
3.46
The Department of Education in Tasmania advised that:
...the 'use by third
parties of NAPLAN results to create simplistic league tables often maligns the
excellent work of schools in enriching the lives of their students and working
towards improved outcomes, and raises the stakes of the tests from their
original purpose to one of 'high stakes'.FF[62]
3.47
It is evident that these concerns are present in the community from the
following examples provided to the committee:
League tables are
only harmful to schools and students by labelling them.FF[63]
The regrettable
outcome [of NAPLAN] is the excessive importance given to it by the media and
selective interest groups including the teachers' unions.FF[64]
3.48
Submissions also highlighted the argument over whether student
performance data belongs only to teachers, parents and students and as such
should not be released publicly in light of the potential for unintended harm.
The AEU emphasised the distinction between parents' rights to information
relating to their own children and that of the broader community, arguing that
parents do not have any inherent right to information relating to the progress
of other people's children.FF[65]FF
3.49
A range of submissions offered suggestions for minimising the potential
for harm, including by removing raw school averages from the My School
page. The ACT Council of Parents and Citizens Association would like to see raw
averages replaced with student results in bands, thereby making it more
difficult for the media or anyone else to devise simplistic league tables.FF[66]FF The AEU among others
argued for the introduction of legislation which would prohibit the publishing
of league tables.FF[67]FF The NSW Primary
Principals Association proposed a number of ways in which adverse effects could
be minimised, including the incorporation of an 'Acceptable Use' page which
would require users to agree to conditions of use before accessing My School
data.FF[68]
3.50
ACARA reiterated its firm position against league tables at a hearing
held in Canberra:
One thing that I
think every educator would agree with is that we do not want league tables. A
league table, to my mind, is where you rank schools without regard to the
nature of the students within the school. My School explicitly does not do
that, and all of us in Australia are very much against having league tables.FF[69]
3.51
The committee majority notes that ACARA has taken action to minimise
misuse of data on the My School website and will implement changes prior
to the release of the new version of the website in December 2010. Enhancements
will include:
- a 'click-wrap' requiring users to indicate their agreement
up-front to terms and conditions of use of My School data;
- a tool to deter automatic scraping of data from the website.FF[70]
3.52
However, there are limits to the action ACARA can take in response to
the misuse of data, particularly where newspapers, which are covered by their
own legislation and not ACARA's copyright clause, are concerned. Federal
legislation would have to be enacted, and enforced, in order to prosecute
newspapers which printed school league tables using data from the My School website.FF[71]
Committee majority view
3.53
The committee majority makes a clear distinction between the My
School website, which provides contextual information (to be enhanced as a
priority, as per Recommendation 12 later in this chapter), and the publication
of crudely designed school rankings by the media and other third parties.
3.54
The committee majority supports submissions calling for more rigorous
protocols on reporting, accessing and using student data in order to prevent
the media and other third parties from publishing crude league tables.
Effect of publishing comparative data and
league tables on school, teacher and student morale
3.55
A range of views were expressed indicating that judging and comparing
teachers on the basis of NAPLAN results was damaging school morale, sometimes
even having a divisive influence among teaching staff.FF[72]FF A November 2009 letter
from peak parent, principal and union organisations to the then Minister for
Education, the Hon. Julia Gillard MP, stated that:
There is
considerable evidence that the inappropriate use of data to compare schools can
have serious negative impacts, both on the testing itself and on the very
schools and children it was intended to help. Allowing student data to be
inappropriately or mischievously used for the creation and publication of
league tables could exacerbate the difficulties of the communities concerned,
narrow the school curriculum and risk the testing process itself becoming
corrupted.FF[73]
3.56
The Queensland Council for Parents and Citizens' Associations cited reports
from parents of children being physically sick before tests, and being
pressured to do well in order to avoid making '...their teacher look bad.'FF[74]FF Moggill State School
Queensland Teachers Union argued that teachers now had their reputations at stake
and had been given an incentive to teach strong performers and gifted students,
who are often clustered in classes, instead of being judged on the performance
of lower achievers.FF[75]FF The Australian College
of Educators added that teachers in the most disadvantaged schools are already
'...more likely to be our less experienced teachers [who] need clear standards
and support, not more pressure'.FF[76]
3.57
A significant number of teachers reported feeling frustrated and
demoralised. One of these, Marianne Scholem, pointed out that '...not every
school can be at the top of the pile...teachers like myself will become
disillusioned and add to the burnout statistics.'FF[77]FF Some also pointed to
overseas experience which suggests '...that league tables lead to a climate of
trepidation, incrimination and blame in schools'.FF[78]
3.58
Others were displeased with what they saw as forced competition at the
expense of teacher and school collaboration, calling My School '...a
veneer of action' and questioning '...how telling people that they work at the
worst school will lead to them improving the school'.FF[79]
3.59
The Australian College of Educators advised that:
...[educational]
excellence occurs and improvement agendas are most effective when respectful,
relational cultures are built through schools in partnerships with parents,
caregivers and their communities.FF[80]FF
The College also
expressed its view that 'school against school data presentation', in its
current form on the My School website, does not promote the
'...development of networks, communities of practice, and communities of
interest,' and suggested that MCEECDYA and ACARA might explore renaming the
website from My School to OurSchools.FF[81]
3.60
Other submissions drew the committee's attention to the importance of
morale at more disadvantaged schools which do not perform well when ranked. One
teacher recalled her own schooling in Fairfield, Sydney—by no means an affluent
suburb—where dedicated teachers imbued her with educational aspirations despite
the circumstances.FF[82]FF
3.61
A parent of a child attending a school with below average NAPLAN scores
commented on what she saw as 'labelling' students as low achievers, expressing
a fear that her daughter and others like her would simply accept the label and
stop trying to do better.FF[83]FF Another submission
lamented the harm done to parent, student and teacher self-esteem in low
socio-economic communities which work hard and do their best to improve
outcomes.FF[84]
3.62
Other submissions echoed this, such as that from Christine Turner on
behalf of Chatswood Hills State School, who noted that some schools only appear
to be underperforming when compared to others because they cater to particular
groups of students or operate under particular circumstances of disadvantage.
They are nonetheless proud of their record and afraid that their '...hard
earned reputation is at risk'.FF[85]FF
3.63
The ACT Council of Parents and Citizens Associations referred the
committee to a comment from a parent who had looked up her child's school on
the My School website:
‘All it did was
leave me with a bad taste in my mouth,’ because given where she lives within
the school boundaries her child could not go to another school and her child’s
school was being compared against private schools that she could not afford. So
she felt that the school was being stigmatised with this bad reputation and she
could not do anything about that. When she went to the website it left her with
this bad taste.FF[86]
The
Council cited the above as an example of how the current approach to publishing
NAPLAN results:
...punishes,
humiliates and demoralizes students, teachers and schools who have been singled
out by the crude and at times inaccurate comparisons made between apparently
"similar" schools as well as from the creation of simplistic league
tables by the media and other organisations.FF[87]FF
Committee majority view
3.64
The committee majority notes the potential for harm to be caused by
simplistic and crude league tables constructed using information from the My
School website and takes very seriously any reports of adverse effects,
however small in number. Until more is done to protect My School information
from misuse the website and league tables will be inexorably linked by
association, as will responsibility for any resulting harm or distress.
3.65
The committee majority endorses recent initiatives from ACARA and
MCEECDYA regarding the responsible use of My School data but remains
concerned about the harmful impact of irresponsible and unchecked conduct by
third parties, such as the media, which are still able to misuse NAPLAN data
available on the government's My School website.
Recommendation 9
3.66
The committee majority recommends that ACARA and MCEECDYA examine and
publicly report on ways to mitigate the harm caused by simplistic and often
distorted information published in newspaper league tables.
Turning NAPLAN into a 'high stakes' test
3.67
Professor Robin Alexander from the United Kingdom, who has written
extensively on education policy and was the director of the Cambridge Primary
Review of English Primary Education, recently gave a number of lectures in
Australia in which he captured the advent of high stakes testing thus:
Of all the so-called
'levers' of systemic reform, tests seem to be the instrument of choice in
policy-makers' efforts to do the two things which they believe they must always
be seen to do: raise educational standards and call teachers and schools to
account. This means that tests are high stakes not just for children and
teachers but also for politicians, and that they may be as much about political
capital as educational progress.FF[88]
3.68
Other submissions also conveyed the sense that the reputational
consequences of publicly reporting and comparing school NAPLAN results have
increased the stakes for schools to do well to unacceptable levels.FF[89]FF For example, the
Association of Heads of Independent Schools of Australia (AHISA) argued:
The use of NAPLAN
test results for the purposes of comparative measurement of school performance
and distribution of some federal grants to the states and territories and to
identified 'disadvantaged' schools has served to morph NAPLAN diagnostic tests
into 'high stakes' tests, where money and/or reputation rides on their outcome.
Media reporting and exploitation of the data through the creation of 'league
tables' have exacerbated this misuse of NAPLAN.FF[90]
3.69
The Australian Primary Principals Association echoed this concern about
turning NAPLAN into a high stakes test, citing two potential factors which
contribute to this outcome: the My School website and '...the $350
million of reward money that has been offered for an improvement in NAPLAN
scores':FF[91]
We think it is fantastic that there is money for schools in
need. That is a big tick. But when reward money is used to threaten principals
or set targets—you must improve by five per cent or 10 per cent before you can
get the reward money—I think that has a perverse effect of what the reward
money is intended to do. When I moved around the country talking to principals,
I think what alarmed me most was a discussion saying yes we have been given
targets that we have to reach; we now are going to choose the group of children
in our school that we will put most our energy into because they are just below
the national benchmark; we are not going to focus on those children that
probably, with all the resources in the world, will struggle to get to the
benchmark and improve our scores. This was a real ethical dilemma for
principals that I sat with in three states. Some were saying you can’t do that,
and others were saying they had been told they had to improve; this is a
business proposition and that is how we have to look at it.
They are the sorts
of things that we should be aware were happening. They only happened this year.
That is what we have to keep in perspective—it was not happening before that
[the advent of My School]. We do not have a problem with students doing
the NAPLAN test; we have always supported the NAPLAN test. The test itself is
not the problem.FF[92]
3.70
The creation of this 'high stakes' environment has a number of potential
negative consequences including 'teaching to the test' and narrowing the
curriculum, as outlined below.
Teaching to the test and narrowing the curriculum
3.71
Teaching to the test involves repeated practice of test format as opposed
to merely instruction on test content, often at the expense of other, possibly
more educationally valuable, curriculum content.FF[93]FF Many teachers and
schools made submissions to the inquiry which expressed a sense that they were
being forced to teach to the test, that is, to rearrange their teaching plans
to focus on NAPLAN tests.FF[94]FF
3.72
This extended to sacrificing 'desirable pedagogies' such as
inquiry-based learning due to time constraints and instead applying a
teacher-directed style more suited to extracting better NAPLAN results.FF[95]FF
3.73
The Junee Teachers Association stated that intensive preparation for a
single point-in-time 'snapshot' of student achievement levels could actually
compromise the diagnostic value of NAPLAN.FF[96]FF
3.74
The AEU argued that the high reputational stakes attached to NAPLAN have
forced excessive emphasis to be placed on the tests, which '...has had a
profound effect on schools, curriculum, teaching and students,'FF[97]FF and means that the
'...primacy of the educational needs of students is subjugated to the
requirements of schools to achieve in testing regimes.'FF[98]FF The AEU attributes some
of the pressure on teachers to teach to the test to competition-induced
pressure between jurisdictions to perform in NAPLAN tests, citing as an example
Victoria, where the state education department set out a ten week 'delivery
strategy' ahead of NAPLAN 2010:
Principals were
directed to appoint a NAPLAN coordinator, to "facilitate a sample testing
benchmarking process which may require further resourcing", to
"provide additional assistance to students identified as capable of making
a significant improvement" and to "privilege the testing as an event
of significance". Teachers were directed to, "explicitly teach for
NAPLAN by including the genre of NAPLAN, commonly used terms and a daily NAPLAN
item in the program of instruction".FF[99]FF
3.75
Submissions highlighted that the curriculum can be narrowed or distorted
if teachers feel they must focus on a particular aspect of teaching, such as
teaching for a specific test. Helen Stearman, a teacher, argued that NAPLAN is
having precisely this effect on teaching practices around the country,
threatening to become '...the de facto curriculum.'FF[100]FF This concern that
schools are increasingly sacrificing the broader curriculum in pursuit of
better NAPLAN results was echoed by Lutheran Education Australia, among others.FF[101]
3.76
The inquiry received submissions suggesting ways to prevent the
curriculum being narrowed by NAPLAN. David Andrich, Chapple Professor of Education
at the University of Western Australia, offered that:
...not every student
in a particular year needs to sit a test in which every student responds to
exactly the same items. The technology exists in test construction,
administration, analysis and interpretation that the results of students and of
schools can be placed on the same metric even if all students in the same year
do not respond to exactly the same items. This is the same technology that
currently permits the results of students from different grades, who do not
respond to exactly the same items, to be placed on the same scale.FF[102]
3.77
Teachers also reminded the committee that it is important for NAPLAN
tests and the wider curriculum to work together rather than compete for
attention.FF[103]
3.78
This is echoed by international academics such as Professor Robin
Alexander, from Cambridge University, who said in a recent keynote address at
the University of Melbourne:
Over-concentration
on the practice of basic skills in literacy and numeracy unrelated to a context
in which they are needed means that those skills are insufficiently extended
and applied.FF[104]
3.79
Others question whether teaching to the test is an entirely negative
proposition. Dr Ben Jensen stated that teaching to the test ultimately results
in an increased focus on literacy and numeracy, which is not necessarily
negative.FF[105]
3.80
Addressing a similar point, Mr Angelo Gavrielatos, federal president of
the AEU, explained:
Please do not
misunderstand what I am saying as an argument against mastering, and proficiency
in, literacy and numeracy. They are the foundation blocks, the building blocks,
of learning. There is a significant difference between teaching to ensure
children can attain the best literacy and numeracy skills and teaching to the
test. Schools are being directed by bureaucrats to teach to the test, to teach
NAPLAN, to teach ‘the genre of NAPLAN’. With all due respect, NAPLAN is not a
genre; it is a test.FF[106]
Committee
majority view
3.81
The committee majority believes that these examples of teacher
dissatisfaction and concern show the importance of increasing teacher
engagement in education policy development and rollout. The committee majority
also believes that more must be done to encourage a complementary relationship
between the tests and the wider curriculum.
3.82
The committee majority notes that ACARA has expressed a desire to look
at ways in which to align NAPLAN tests with the wider curriculum in the future.FF[107]FF However, given community
scepticism of the national curriculum currently being developed by ACARA, the
committee majority cannot support this intention until concerns about the
curriculum have been adequately addressed.FF[108]FF
Recommendation 10
3.83
The committee majority recommends that ACARA identify, analyse and
report publicly on possible means of strengthening the relationship between
NAPLAN tests and the wider curriculum. The committee majority reserves its
support for any alignment between the tests and the new national curriculum
until the quality of, and community support for, the curriculum become clearer.
Do parents really have a choice
about where to send their children to school?
3.84
One of the key arguments underpinning the publication of NAPLAN results
on My School is that the website provides parents with information and
choice when assessing which school their children will attend. A number of
submissions questioned this assertion.
3.85
The Tasmanian Education Department considers the choice argument to be
misleading for the following reasons:
- It implies that schools are entirely responsible for a student's
results, without taking into account the personal, political and social context
in which they operate;
- Most parents do not have choice. In many parts of Australia there
is no other school and in other areas many parents so not have the resources to
move their child to another school;
- Where there is choice, parents choose schools for a variety of
reasons, not only test results. They choose because of the needs of their
children, the school culture, the social or extra-curricular programs the
school has, because their child has friends who attend that school, because it
is close to where they live or merely because they went there themselves;
- If the argument is accepted that parents will choose schools
based on the public reporting of NAPLAN result, the parents who take this
option could be changing schools each year as results change due to the
different cohorts being tests.FF[109]FF
3.86
The Australian College of Educators, too, emphasised that parents with
children in disadvantaged schools rely on the government to provide a sound
standard of education for their children and cannot 'vote with their feet'.FF[110]FF Their submission argued
that for schools serving economically and socially marginalised parents and
children '...the logic of parent power and school choice, as a response to
NAPLAN comparative information, does not apply'.FF[111]
3.87
Other submissions, such as that from Junee Teachers Association,
informed the committee that parents were in fact withdrawing their children
from particular schools in order to send them to better performing schools
nearby, resulting in
...dramatic effects
on the staffing, resourcing, and programming of schools...[including]...a
significant narrowing of curriculum offerings, particularly in the senior
school.FF[112]
3.88
This was echoed in by other submissions which also reported that
talented students were leaving because their school had been 'branded a
failure'.FF[113]
3.89
Mrs Sharyn Lidster, Acting General Manager, Strategic Policy and Performance,
Department of Education, Tasmania, informed the committee the department had
done a statistical analysis of student movement across schools in response to
principals' concerns. The findings indicated that no significant movement has
occurred; in fact, the '...movement variation is what we saw from year to year
before we did public reporting, so it has not impacted'.FF[114]
3.90
The Australian Primary Principals Association (APPA) confirmed that some
parents have decided to change the school their children attend as a result of
NAPLAN reports, but that this only appeared to be happening in 'tiny numbers'.FF[115]FF
Inadequate contextual information
3.91
Submissions questioned the value of the comparisons between schools
available on the My School website for reasons beyond those to do with
ICSEA values and often in connection with a perceived lack of adequate
contextual information about schools. For example, the NSW Teachers Federation
argued that '[t]he prominence given to quantitative data presented in colourful
graphics projects an air of scientific authenticity that is essentially
populist.'FF[116]FF The submission quotes a
NSW principal who says that the coloured comparison graphs used by My School
will always get more attention than any other information on the website.FF[117]FF
3.92
A substantial number of submissions lamented the lack, or at least lack
of prominence, of contextual information on the My School website.FF[118]FF Information on parents'
views presented to the committee by organisations such as the ACT Council of
Parents and Citizens Associations indicates that parents want more contextual
information about the schools available to their children.FF[119]FF
3.93
The committee majority notes that, as stated by the Australian Council
for Educational Research, My School '...is best viewed as the first
version of the website,' which will be '...refined and further developed into
the future'.FF[120]
3.94
ACARA has notified the committee of its plans to increase reporting of
school contextual information by publishing the percentages of enrolled
students who come from a language background other than English, as well as by
expanding the text field available for principals to describe school profiles.
The latter are in the process of being collected from schools.FF[121]
3.95
The AEU expressed the following hopes for the new version of the My
School website:
Our view would be
that in the next iteration of the My School website, which will emerge, there
needs to be a very serious and honest communication strategy in order to put
front and centre what the data is about and what the limitation is of the data
and contextualise it so that people understand that this is not the be-all and
end-all of schools. We cannot lead with information about schools with the use
of NAPLAN data. There is a lot more to a school and the need to establish
context of a school before you even start discussing what is data—which
ostensibly is a snapshot of student skills in the area of literacy and
numeracy, only one at a particular time and only one narrow slither, important
as it is—of the educational wellbeing of a child.FF[122]
The improvement that
is required with the federal system of the My School website is that at the
very least any information presented on that website should lead with
contextual information about that school so that it can present a picture of
itself which deals with, on a daily basis, its successes, its challenges and
the like. Any subsequent information should be just that, subsequent
information, and it should not be privileged, given the fact that it is not the
be all and end all; in fact it is a very small part of a school’s life.FF[123]
Committee
majority view
3.96
Providing parents with information to make empowered decisions about
which school their child attends is a strong argument in favour of reporting
NAPLAN results on the My School website. The committee majority
acknowledges, however, the evidence received that not all parents have options
about where to send their children to school.
3.97
The committee majority also notes the evidence received suggesting that
too much can be mistakenly inferred about a school on the basis of student
performance data. Therefore, the committee majority does not support publishing
raw test results devoid of context or acknowledgement of the fact that schools
are not solely responsible for student performance.
3.98
The committee majority notes and supports ACARA's plans to include
information on the percentages of students from a language background other
than English, but believes this only goes some of the way toward providing
adequate contextual information about schools.
Recommendation 11
3.99
The committee majority recommends that ACARA and MCEECDYA move to
include more contextual information about schools on the My School
website, reflecting the complex range of factors that affect schools, and
acknowledge to users of the website their awareness of the limitations of
comparisons based on raw performance data due to extrinsic factors. The
committee majority further recommends that ACARA commit to ensuring this
contextual information is available ahead of the reporting of 2011 NAPLAN
results.
Can we learn from the international
experience?
3.100
Whilst standardised testing is used by a number of countries, student
performance in international tests suggests that the existence of such testing does
not lead directly to improved performance. Neither does the international
experience with standardised testing necessarily translate directly into the
Australian context. As stated by Dr Peter Hill of ACARA:
...[T]hat history of
high-stakes testing leading to high-stakes consequences for the staff within
schools has, of course, scared other people in other jurisdictions, and there
has been a thought around that Australia is simply going down this path. What I
would say is that we are not. Educators come and visit us from the UK and tell
us all the bad news about what happened there. We know what happened there, we
have learnt from that and we are not going down that path.FF[124]
3.101
Certain countries which outperform most others on international
assessments of literacy and numeracy without relying on national standardised
testing are frequently cited as arguments against standardised testing. On
this, the committee received considerable information on Finland in particular.FF[125]FF
3.102
Finland achieves indisputably impressive results, consistently coming at
or near the top of international rankings, with very narrow achievement gaps
between the highest and lowest performers within individual schools and between
schools, as well as between students from different socio-economic backgrounds
and regions of the country.FF[126]
3.103
However, the fact that Finland does not base policy decisions on
national testing in the same way as Australia does not of itself explain
Finland's success. There are a number of important additional factors at play
which this report can only touch on. In Finland every teacher:
...[has] a masters
degree and is an expert in how to assess, test and act on the results of
student assessment, about a seventh of all professional teachers in the school
are people who are trained to ensure that no student will fall behind by more
than 48 hours. That is the kind of strategy that is going to lift the
performance of students and close the gap between our high and low-performing
students.FF[127]
3.104
The Finnish education system enjoys what Professor Brian Caldwell of
Educational Transformations calls a 'cultural advantage' which means that the
system is not directly comparable to Australia's:
It is not
necessarily salaries, because when you adjust for purchasing power of currency our
teaching salaries are probably a little above those of Finland. There is a
cultural advantage that Finland enjoys, and that is that for many decades
teaching has been a very highly valued profession. The initiatives of the
Finnish government, in saying that the level of professional knowledge and
skill that you now require if you are really going to make a difference for
each and every child requires five years of university preparation—and then
making a master’s degree a requirement for beginning teachers—lifted the status
of the profession quite significantly and, as Geoff Masters points out, and as
the McKinsey report also points out, education is one of the top three
preferences for those entering university.FF[128]
3.105
The English and American school systems do administer standardised tests
and use the data to compare schools. Neither country performs well in
international literacy and numeracy assessments.FF[129]FF Both countries'
approaches to the publication of comparative reporting of test results, which
focus on punitive measures for schools which do not meet targets, are
controversial and the consequences of their respective policies disputed.FF[130]FF
3.106
Dr Peter Hill of ACARA elaborated on these international examples:
I do understand the feelings of many people who have seen
what has happened in the USA and the UK, which was not, in my mind, very
intelligent in terms of accountability. To be honest, it happened a long time
ago in the UK and more recently in the USA.
In the UK they certainly published lists of schools in terms
of raw percentages meeting a standard, and they ranked the schools. The schools
at the top, of course, were the schools that had all the smartest kids. They
were independent schools, and typically girls’ independent schools. The schools
at the bottom tended to be the schools in boroughs like the inner London
borough of Hackney, where I have done quite a bit of work recently. Of course,
the demographic composition there means that the schools are really struggling
to do the right thing with their students. So schools were unfairly compared.
The other thing that happened was in the USA following the
passing in January 2000 of the No Child Left Behind legislation, which
incidentally was supported by both the Republicans and the Democrats; there was
complete agreement on it. It meant that each state within the USA enacted its
testing program but that there would be serious consequences for schools that
did not meet annual targets. These serious consequences meant that you were
pretty well placed on notice after one year. After two years there was some
small action, but after three years the action got very serious, even down to
taking money from the school so that it could be given out to parents to get
private tutors. In other words, the school was deemed not to be able to deliver
this, so the money was taken from them. Indeed, the school could be closed down
or staff removed and so on. In other words, there were very high-stakes consequences,
often unfairly, because the schools that were the subject of this found it very
hard to improve the students for reasons which were partly outside their
control.
So that history of
high-stakes testing leading to high-stakes consequences for the staff within
schools has, of course, scared other people in other jurisdictions, and there
has been a thought around that Australia is simply going down this path. What I
would say is that we are not. Educators come and visit us from the UK and tell
us all the bad news about what happened there. We know what happened there, we
have learnt from that and we are not going down that path.FF[131]
3.107
The committee majority notes that, despite similarities to the
Australian system, the context and outcomes are quite different. The US and
English education systems both pursue a policy of withdrawing resources from
underperforming schools. Conversely, in Australia, the focus of the NAPLAN
testing regime is improvement, and poor performance is identified in order to
better direct additional resources.
3.108
The Australian College of Educators supported making school performance
information available to the public and pointed to '...international literature
on transparency and accountability for governments, [which] provides strong evidence
that public access to this sort of information is a potentially powerful force
that can contribute to keeping governments focussed on their policy promises.'FF[132]FF The College of Educators
also pointed out the problem with applying this to the Australian context,
where '...the way in which the information has been organised for public
consumption is not focussed around "keeping the government honest"'.FF[133]
What are the alternatives?
3.109
The committee majority notes that some submissions to the inquiry
support replacing current school performance data on the My School
website with information on the value a school adds to student performance.FF[134]FF
3.110
Value added to student performance by schools can be calculated by
comparing the progress made by individual students between one test and the
next, controlling for extrinsic factors such as student background information.
Once the background factors are controlled for, what is left is a measure of
the contribution a school has made to individual student progress,FF[135]FF which is calculated by:
...using a
statistical model that compares the progress made by each student with the same
initial level of attainment, controlling for background factors.FF[136]FF
3.111
Dr Ben Jensen informed the committee that value-added measures of school
performance are preferable because they are more accurate than using raw NAPLAN
test scores or similar 'contextualised attainment models' which attempt to
control for background factors in less precise ways:
Australian and
international research consistently shows that value added measures provide
more accurate measures of school performance than the use of raw test scores or
what have previously been termed contextualised attainment models. These models
attempt to control for the socioeconomic background of students, usually
through some form of multivariant modelling. These models are similar to the
methods used to measure school performance on the My School website that use
raw NAPLAN scores and then attempt to control for the socioeconomic background
of the school... research consistently shows that this method produces less
accurate measures of school performance than value added modelling.FF[137]
Even with data far
more comprehensive than anything available in Australia, and modelling far more
complex than that utilised for the My School website, the predictive power, the
accuracy, of these types of models were roughly half that of a simple value
added model. These findings are echoed around the world. Value added measures
of school performance are not 100 per cent accurate—no estimate of school
performance ever will be—but they are a substantial improvement on current
measures and widely considered to be the most accurate measures available,
particularly for schools serving disadvantaged communities. That is why they
have been supported by stakeholders such as unions and school associations in
numerous countries.FF[138]
3.112
Dr Jensen elaborated on what appears to be a widespread misunderstanding
of what value-added measures of school performance are, explaining that they
are not '...simple measures of student progress' but are instead relative,
because:
All students will
progress. Even a student who in year 3 performs at an average level but in year
5 performs at a below average level will have progressed in absolute terms, but
we should be interested in the contributions schools make to that progression
relative to the progress made by students at each initial level of attainment.FF[139]FF
3.113
An additional benefit of using value-added measures of school performance
is that their introduction would remove the need to use the ICSEA index and to
group schools into 'like schools', because both would be contained in the
estimation used to calculate the value added.FF[140]FF This would address one
of the most contentious aspects of the My School website.
3.114
The committee majority notes that Dr Jensen advocates including both raw
test result data and value-added measures on the My School website.FF[141]
Committee majority view
3.115
Given the large number of concerns expressed in submissions about the My
School website, the committee majority believes it is necessary to increase
the accuracy of student performance measurements on the website in order to
provide better quality information on how schools and students are progressing.
3.116
The committee majority has considerable reservations about the My
School website in its current form, and believes that shutting down the
website to prevent further harm may be necessary unless steps are taken to
improve the quality of the information presented. However, the committee
majority does not believe that taking the My School website down is the
best way forward and instead sees substantial potential for improvement in methods
of publishing school performance data on the website by incorporating value-added
measures as outlined above.
3.117
The committee majority emphasises the importance of following up low
NAPLAN test results, or low school value-added scores in future, with immediate
intervention aimed at assisting individual schools and students. The committee
majority does not support putting punitive measures in place for schools with
low value-added scores, as is the case in England.FF[142]
Recommendation 12
3.118
The committee majority recommends that ACARA and MCEECDYA comprehensively
revise the type of information available on the My School website to
shift the focus from raw school performance data to value-added measurement of
school performance.
Conclusion
3.119
On the weight of evidence received outlining numerous community concerns
about the My School website, the committee majority cannot support the
website in its current form.
3.120
The problems outlined in this chapter are wide-ranging and a cause for
serious concern. They point to a substantial and justified lack of confidence
in the website among the parent, teacher and wider community. The potential for
this lack of confidence in My School to engulf the entire national
literacy and numeracy assessment system, thereby compromising the benefits of
NAPLAN tests themselves, is considerable.
3.121
For this reason and in the interests of progressing the central aim of
national literacy and numeracy testing—that is, improving educational outcomes
for students—the committee majority concludes that the best way forward is to
instigate, as a matter of priority, a comprehensive revision of the My
School website.
3.122
The committee majority is firmly of the view that a more measured and
evidence-based approach must be taken. Instead of holding schools and teachers
accountable on the basis of opaque and often distorted comparisons between raw
student test results, the government must instead turn its attention to
developing an optimally accurate, reliable and verifiable measure of school
performance. Teachers and schools must then be allowed the autonomy and given
the necessary support to design and implement frameworks aimed at improving
student performance across the board, but in particular where students are
performing below national benchmarks.
3.123
The committee majority concludes that the focus of the My School website
must shift to include information on the value added by schools to student
performance, taking into consideration background and extrinsic factors in a
much more reliable fashion than is currently being achieved by the use of ICSEA
values as a basis for comparison. The committee majority cannot support any
future version of the website which fails to do this.
Senator Chris Back
Chair
Navigation: Previous Page | Contents | Next Page