Government senators' additional and dissenting comments

Government senators are pleased to see that the committee majority recognises and supports the importance of national testing for literacy and numeracy and will provide some additional commentary on this below. Result analysis and reporting are the subject of ongoing improvements which should address the committee majority's recommendations regarding reporting refinements. However, government senators do not agree with the proposal to expand NAPLAN testing and this is addressed below. Government senators also do not agree with all the revisions proposed in the committee majority report for the My School website, particularly the focus on value-added measures and provide reasons as well as additional commentary later in this report.

Government senators acknowledge the central importance of literacy and numeracy as the foundations for further learning. As indicated by the Australian Council for Educational Research (ACER):

Educational research studies have underscored the fundamental importance of literacy and numeracy, not only to educational success, but also to successful transitions into employment and adult life. Levels of literacy are correlated with a wide range of outcomes, including lifetime earnings, employment prospects, levels of health and involvement in crime.[1]

Benefits of NAPLAN testing

The government's education revolution is driving a renewed focus on the foundation skills of literacy and numeracy to lift student achievement across the country. National Assessment Program – Literacy and Numeracy (NAPLAN) testing is intended to:

...identify whether all students have the literacy and numeracy skills and knowledge that provide the critical foundation for other learning and for their productive and rewarding participation in the community.[2]

Dr Peter Hill, Chief Executive Officer, Australian Curriculum, Assessment and Reporting Authority (ACARA) emphasised to the committee that NAPLAN is intended to provide a snapshot of student performance to better inform stakeholders for the purpose of improvement.[3]

Uses of the data

Government senators support NAPLAN testing and reporting as an important tool to provide valuable information for many uses which include:

enabling a sophisticated analysis of growth in attainment over time;
showing student results by sex, location, parental background and Indigenous status, enabling the community to clearly see those student groups which need more support to improve their educational outcomes;
tracking progress towards Council of Australian Governments (COAG) targets;[4]
giving the community, teachers and parents better information about how schools are performing;
helping schools to identify strengths and weaknesses in teaching programs;
allowing governments to target additional resources to schools which may be struggling;
complementing class and school based assessment to provide a comprehensive picture of student achievement that helps determine areas of priority to improve literacy and numeracy outcomes;
allowing teachers to better identify students who require additional support; and
enabling parents to use individual results to discuss student achievement and progress with teachers.

ACER confirmed that the tests provide a new level of information for teachers and schools, education systems and governments as well as parents. It explained how the data can be used:

At the school level, the tests can be used to identify areas of relative strength and weakness and to assist in planning teaching interventions and special support. A number of Australian states provide software to support schools to explore and make best use of the diagnostic information available through NAPLAN. At a national and education system level, NAPLAN provides diagnostic information to inform educational policies and planning.[5]

Mrs Sharyn Lidster, Acting General Manager, Strategic Policy and Performance, Department of Education, Tasmania, explained how important the NAPLAN data is for school education systems:

It provides us with performance information, for the first time when we had the national test, that enabled each of the states and territories to actually look at our results relative to other states and territories across Australia. That is very important because prior to 2008 we had state-wide testing but each of the states had different tests, so it made it very difficult to see how our students were performing across the nation in relative terms.[6]

Mrs Lidster indicated how schools are supported to make best use of the data:

The NAPLAN tests are used extensively to support schools. Our jurisdiction, and others, provides the information back to schools. A lot of analysis is done that supports the schools. Workshops are run to help the schools to interpret the results and use them effectively to support their teachers. We also conduct workshops to assist senior people within schools to interpret the information and, where appropriate, we provide additional support for schools in relation to teacher development. Also, funding is provided to support the additional programs to improve the outcomes for students, where they are identified as performing below where we would expect them to be.[7]

Mrs Lidster emphasised the usefulness of the data for schools and teachers:

There is a large amount of evidence that shows us that the quality of our teachers makes a big difference to the outcomes of our students. The use of NAPLAN, or state-wide and national testing, especially when it is provided back to schools, is valuable to help teachers look at how effective their delivery of the curriculum has been. It also helps school leaders to sit down and have critical conversations with their teachers to say, for example: ‘These results are particularly fantastic. What is it that you are doing in your classroom? Let’s share that practice across our school.’ It also helps our district support staff to look at the school’s results and to work with the school either to bring in additional professional learning where it is needed or to share some particularly gifted teachers’ talents with other schools. So I think it is critical, and it will lead to improvement in Australia if we are using those results for that purpose.[8]

The ACT Council of Parents and Citizens Associations supported the continuation of national testing and pointed out the importance of the data for parents:

...The council believes that parents should receive as much detailed information as possible, as this encourages them to speak to their child’s teacher and get involved in their education. Research consistently shows that the best outcomes are achieved when teachers and parents work closely to give students the best education possible.[9]

While impressing on the committee that NAPLAN is only one source of information about student achievement, the President of the Australian Primary Principals Association (APPA), Ms Leonie Trimper, told the committee that her organisation 'supports the use of NAPLAN to provide schools and systems with quality information about the performance of Australian primary school students in literacy and numeracy'.[10]

Government senators view

Government senators support the increased transparency and rich information source now available for governments, educational authorities, schools, principals, teachers and parents as a result of the publication of NAPLAN test results.

International research and best practice

The former Minister for Education noted that the NAPLAN measurement scales are unique and represent world best practice in the measurement of student progress.[11] Professor Geoff Masters, Chief Executive Officer, ACER emphasised that NAPLAN is firmly grounded in 20 years experience through state literacy and numeracy testing programs, adding:

NAPLAN is also pretty firmly grounded in international best practice in tests of this kind...Part of the reason that the Australian Council for Educational Research are managing the PISA [Programme for International Student Assessment] tests for the OECD out of Melbourne is that we do have international expertise in the analysis and reporting of data, and we are applying that to the NAPLAN tests.[12]

He explained that the test development process used is accepted as international best practice, as well as the statistical methods and processes used to analyse the data and report the results.[13]

Dr Ben Jensen, Director, School Education Program, Grattan Institute, stated that NAPLAN testing and to a lesser extent the My School website are important advances in the development of the school education system, noting that:

...the international research clearly shows that high performing education systems are more likely to have national assessments and utilise them in some form to foster school improvement, school accountability and school choice.[14]

Using NAPLAN data to address disadvantage

NAPLAN results have showed that there are still some students who have not attained the literacy and numeracy skills expected of students in their year level. The federal government has entered into National Partnerships with the states and territories to address disadvantage, support teacher quality, and improve literacy and numeracy.

As noted in the 2010-11 Budget, funding of $2.6 billion over five years from 2008-09 has been provided for the three Smarter Schools National Partnerships. There is $540.0 million for the Literacy and Numeracy National Partnership which seeks to accelerate improvement in student literacy and numeracy outcomes for those students most in need of support; the $1.5 billion low Socio Economic Status School Communities National Partnership to support the learning needs and wellbeing of Indigenous students to improve engagement and attendance; and the $550.0 million Teacher Quality National Partnership which will support teachers, including those in schools with predominantly Indigenous students.[15]

Through the NAPLAN assessment and My School website, the government has identified an additional 110 struggling schools that would have missed out on a share of the $2.6 million Smarter Schools National Partnerships and will now share in $11 million in extra funding to ensure students improve literacy and numeracy.[16]

The increased transparency and attention on support for low performing students was supported by Mr Gary Banks, Chairman of the Productivity Commission, who stated:

While there are low performing students across the socio-economic spectrum, justifying some attention to raising performance per se, there is evidence that low SES kids underperform relative to their potential across the board, justifying the targeting of this group as a whole. Hence COAG’s attention to schools in lower SES areas would seem warranted, especially for those performing poorly relative to other schools with similar student populations. (This can now be revealed through NAPLAN data — with ‘My School’ transparency upping the ante for action — again illustrating the perversity of opposition to reporting by those professing to hold the interests of students paramount.)[17]

Dr Ben Jensen also supported the greater emphasis on low-performing students:

If we look at the very low-performing students, these are the students who are most in need of help and assistance and who unfortunately are a greater cost to society later on and who miss out on opportunities later in life. I actually see the value in NAPLAN and My School identifying those students who are below minimum levels of literacy and numeracy, because they will struggle the most later in life.[18]

Dr Jensen provided the following statistics to support the attention on students falling behind:

We have a real problem in our country of letting students slip further and further behind. If you look at the NAPLAN data, in year 3 roughly eight per cent of students perform at or below minimum levels of writing literacy. By year 9, it is over 30 per cent—nearly a third of students are performing at or below minimum levels of writing literacy. That is an incredible percentage of students. Not only do we fail to address and help the students who are performing poorly at a young age, we actually let others fall to that level as well. So I think there is a good case to be made that, as soon as a student falls to those levels—particularly the younger they are, because if we get them young we can help them much more efficiently, much more effectively—we should institute specific programs to help them. This could be special assistance or simply identifying to the school principal, ‘You have these students who really need your help.’ I think that can work effectively; it has in other countries—very high-performing countries which do not have the same problems of very low performance that we have.[19]

Examples of the strategies used in other countries to assist students starting to fall behind were provided by Professor Geoff Masters who expressed the view that:

...there is a general lesson there about the importance of identifying and catching children who are beginning to slip behind in their learning and getting them back onto a trajectory.[20]

NAPLAN was supported as a major step in the education system by Professor Peter Ridd at James Cook University in Queensland, who reported that he has noticed a decline in the standards of students on entry into university in the areas of maths and science. Professor Ridd cited a review undertaken by ACER in Queensland which found:

In summary, there appears to have been a decline in the relative performance of Queensland students in maths and science over a period of decades. In the period 1964 to 1995, the absolute decline in lower secondary maths achievement appears to have been greater than in any other State, and to have been the equivalent of about two years of schooling.[21]

Professor Ridd emphasised that regular and consistent monitoring and assessment is essential to address the decline in standards over the years and welcomed the development of NAPLAN which, when used as intended, is a 'successful and useful tool'.[22]

In practical terms Mrs Sharyn Lidster from the Tasmanian Department of Education reported on how providing extra assistance to struggling schools occurs:

All of the measures are used as part of schools providing an annual school improvement report where they identify where they need to improve. That is done in conjunction with the regional general managers. Then the support is provided where schools determine that they may need to put in extra professional development. For example, if their reading results for a particular group of students were not as expected, additional support and additional programs will be provided to the school so that they can improve those outcomes for the students. So they do not submit; it is part of the system, part of the conversations and part of the budget process.[23]

NAPLAN is just one of a range of measures

NAPLAN is one indicator of student performance. It is not an indicator of school performance on its own. As noted by Ms Trimper, APPA, 'NAPLAN is only one piece of the educational jigsaw; it is only one piece of the story about a school.'[24]

This view was supported by Mrs Sharyn Lidster, Department of Education Tasmania, who emphasised that NAPLAN is just one of a suite of measures used to look at performance.[25]

Professor Masters emphasised the need to protect NAPLAN as a good measure of student performance and to make clear the original purpose of providing information to schools, parents and systems. He added that it will be important to develop better measures of teacher, school and system performance.[26]

Margins of error are recognised

Professor Geoff Masters acknowledged the current limitations of NAPLAN tests:

Of course, they are point-in-time tests, so they are limited in that sense. They only assess part of what is important in schools. There is inevitably a degree of imprecision, measurement error, around the estimates that they provide.[27]

ACARA has acknowledged the measurement errors inherent in any tests and explained how this is being addressed:

...NAPLAN are short tests, so the error band for an individual is wider. We can improve that in two ways: increasing the testing time, which is not very popular; or we can have a test where the items are more focused around the abilities of the individual student and then your testing becomes more efficient and the error becomes much smaller. That is one of the things that ACARA wants to look at for the future along with of course aligning the test with the curriculum...We are also keen on the notion of seeing if we can improve the efficiency of the testing to reduce margins of error. That is something that we want to look at.[28]

ACARA also noted that the first version of the My School website recognised the margins of error inherent with any testing, and the next version will display this even more clearly.[29]

Timing

In response to concerns about the timing of the NAPLAN test, Dr Peter Hill, CEO of ACARA provided the following explanation:

NAPLAN will never work as a diagnostic test. A diagnostic test should be administered by the teacher so that they can take action the next day. No national survey can return all of that the next day very easily. There are ways in which we can provide resources—for example, in computer adaptive tests that can provide feedback instantly. Perhaps in the future those sorts of resources can be made available. There is a lot of interest in doing that, but that is not a part of NAPLAN at this stage.[30]

Government senators view

Government senators were concerned to see signs during the inquiry that that the original purpose of NAPLAN has either not been well communicated[31] or its importance exaggerated by some. It is essential to keep the test in perspective. Its purpose is to provide a snapshot of student performance in order to focus on improvement. It is not a diagnostic assessment which looks at the reasons why students are not performing and which requires immediate feedback.[32] The limitations of NAPLAN as with other testing are acknowledged. It is a point in time test with margins of error and it should be seen as just one information source within broader contextual information about a school.

Government senators welcome the new National Partnership Agreements with governments and schools which will allow resources to be better targeted to disadvantaged schools and students which need the most support. Government senators note that distribution of the $2.6 billion committed through these partnerships to improve the quality of teaching, improve literacy and numeracy and provide additional assistance to disadvantaged schools [33] relies on NAPLAN tests revealing where needs are the greatest.

In response to the committee majority recommendation to use below-average NAPLAN test results to trigger the provision of assistance to schools and students, government senators note this is already occurring. The focus of NAPLAN testing is to provide assistance to schools and students which are identified as requiring it. As reported to the committee by Mrs Sharyn Lidster from the Tasmanian Department of Education, mechanisms are already in place to facilitate this. The committee also heard that teachers use the results to evaluate their teaching programs and identify students requiring additional support. As for more immediate feedback, this is not the purpose of NAPLAN and government senators note that consideration of resources to provide this more immediate feedback to teachers is underway (see below).

Learning from overseas experiences

Ms Trimper, APPA, advised that research from the US and Britain shows that when assessment programs become high stakes there are unintended negative effects on the quality of teaching and learning. She emphasised that APPA wants to ensure that NAPLAN does not go down the same path.[34]

Dr Peter Hill, Chief Executive Officer, ACARA responded to comparisons made by some with the UK and US experiences:

...Those two countries got into accountability in terms of having really-high-stakes assessment well ahead of others, often with quite severe consequences for poor performance. I was in the USA when President Bush introduced No Child Left Behind and if you failed to meet satisfactory progress targets then there was a consequence: you were on watch the next year; in the second year there was one set of consequences and in the third year another set of consequences. This, of course, led to a lot of contention and debate. Now we are seeing some of the research into whether it really did make a difference and so on...

...Australia has not made the mistake of the UK and the USA of having those extreme high-stakes consequences. We are in a position where we can have a good look at the data and say, ‘Let’s look at other schools that are doing a good job. What can I learn here?’ within an environment where you are not expecting to be closed down next year for poor performance, and to really look at how you can improve.

The answer I would have to those critics is that we are in a different context to the research that you are reading. We are in a situation where I think we are having a more intelligent approach to accountability and a more intelligent approach to the use of that data, particularly in not comparing schools against others where it is an illegitimate comparison but having comparisons with schools that have similar students.[35]

Dr Hill stressed to the committee that Australia has learnt from the UK and US experiences and is not going down the same path.[36] Responding to the concerns about NAPLAN being a high stakes test, Dr Hill added:

I think the education community is coming to grips with a new world in which transparency is at a level that we have never seen before and accountability is of a kind that we have never seen before. We are all coming to grips with what it means to have intelligent accountability in all of this. I do understand the feelings of many people who have seen what has happened in the USA and the UK, which was not, in my mind, very intelligent in terms of accountability...[37]

ACER also noted the experiences overseas which are often cited to argue against standardised testing. It argued that this occurs without acknowledgement of the differences in the testing regimes or in the educational circumstances in other countries. For example Finland is often referred to:

...without acknowledging the very different features of the Finnish education system, including the fact that Finnish teachers are significantly more highly qualified than Australian teachers and Finnish schools are much more alike in composition and outcomes than Australian schools.[38]

Government senators view

Government senators stress that NAPLAN is not the same high stakes test that occurs overseas where penalties are applied for poor performance. Government senators highlight the evidence from Dr Peter Hill, CEO ACARA, who emphasised that Australia has not made the same mistake as the UK and USA which have negative consequences for their testing. Government senators emphasise the intention here is to identify where support is required for students and schools and ensure they receive it.

Improvements to NAPLAN testing

Government senators note that as NAPLAN is now in its third year it is possible to report on the growth of student achievement which adds an important element to the usefulness of the tests. Government senators recognise that NAPLAN tests and the ways in which results are analysed and reported are the subject of ongoing improvements, and ACER listed a number of areas under consideration.[39]

Dr Hill also told the committee that resources to provide more immediate feedback for teachers are under consideration:

There are ways in which we can provide resources—for example, in computer adaptive tests that can provide feedback instantly. Perhaps in the future those sorts of resources can be made available. There is a lot of interest in doing that, but that is not a part of NAPLAN at this stage.[40]

Government senators view

Government senators note the committee majority recommendation to expand NAPLAN testing. Government senators disagree with this proposal as increased large‑scale cohort testing is not the next step that needs to be taken. The next step is providing teachers with better diagnostic tools to address the needs of individual students. Government senators note that the government has committed to developing a national online assessment and learning bank for students, parents and teachers to provide a sophisticated diagnostic assessment of each student’s strengths and learning needs.[41]

Allegations of cheating have been dealt with seriously

Allegations of cheating on the NAPLAN tests [42] have been thoroughly investigated [43] and actions taken to stop any recurrence of security breaches.

In October 2010 at Senate estimates hearings, Dr Peter Hill, Chief Executive Officer, ACARA, provided an update on action underway:

...We have two that are under investigation in terms of security breaches. We have 10 under investigation that involve cheating and four that involve breaches of the protocols of administration. They are of varying degrees of seriousness. Obviously, the ones that involve cheating are the ones that are of concern to us, and there are 10 of those under investigation, but there have only been two that have been confirmed at this stage, so we are talking altogether about 12 allegations that have either been confirmed or are still under investigation.

...These are matters which are being done by the test administration authorities in each state or territory. Because they can have very serious consequences for the individuals concerned, they do tend to take rather a long time to finalise, so we cannot put an end date on them.[44]

He added that where a child's score may be affected, the jurisdictions notify parents and may need to withdraw the results. He assured the committee that the effect of this on the averages for statistically similar schools or for all schools would be negligible.[45]

ACARA outlined plans to enhance the security of test administration which include strengthening protocols and embarking on a 'multi-level' communication strategy for 2011 which will develop greater understanding of the required protocols to manage test materials.[46] As this action is already underway, government senators do not see the need for the committee majority's recommendation for additional work in this area.

My School website

NAPLAN testing is complemented by the My School website, which provides detailed information about almost 10,000 schools. People can locate statistical and contextual information about schools in their community and compare them with statistically similar schools across the country.

Background

The website has been developed by the Australian Curriculum Assessment and Reporting Agency (ACARA), an independent authority that is responsible, among other things, for publishing nationally comparable data and contextual information on schools. Education ministers have agreed the Principles for Reporting on Schooling in Australia which include:

the protection of individual student privacy;
not publishing comparative data without contextual information; and
the publication of error margins, caveats and explanatory notes to ensure accurate information.[47]

The design of the website was endorsed by the Ministerial Council for Education, Early Childhood Development and Youth Affairs (MCEECDYA) in September 2009 and was launched on 28 January 2010. It provides profiles of almost 10,000 schools, contextual information and NAPLAN results that can be compared with results from statistically similar schools.[48] ACARA reported that as at 25 June 2010 the website had received 2,445,308 visitors and 3,368,847 visits.[49]

The importance of context

Cognisant of the concerns raised about the website, Dr Peter Hill, ACARA, cautioned that the information on the website about each school needs to be understood within a broader context of all the other information available about the school, including factors such as student characteristics, percentage of Indigenous students and location.[50] In response to concerns that the first version of the website may have been made available too early, he stated that he believed the publication of the data improved the quality of the data:

I believe that until we published we were not going to get the data. Since we published there has been a scramble to get extra data. So things have moved along very rapidly. My view is that if we had waited for that to be in place we would still be waiting in five years time.[51]

Professor Geoff Masters, CEO of ACER, also emphasised the important distinction between measures of student performance and measures of school, teacher or system performance:

What we have done in Australia to date is that we have worked hard on getting very good measures of student performance. They can be, as I said, point-in-time measures. We can look at trends over time because of the way we calibrate the tests statistically. We can look at trends in a system or even in a school over time. We can look at the growth that is occurring—for example, between year 3 and year 5 or between year 5 and year 7. But all of these, whether they are point-in-time measures of status, growth measures or measures of trends, are measures of student performance.[52]

Professor Masters added:

I think we have been pretty careful with the My School website to say that what we are reporting on My School is measures of student performance. We are not trying to report a measure of a school’s performance. It is true that an attempt is made to make the information perhaps more useful by comparing schools that have similar student intakes, but our focus with My School is still very much on the reporting of student achievement, not trying to construct measures of a school’s performance to get a number for a school.

Professor Masters also emphasised the need to understand the information in context:

I think the thing we have to guard against is giving it all the value, giving it a priority above everything else. I would be arguing for reporting NAPLAN results, as we are, but also providing rich information about other aspects of the school’s circumstances as well as the students’ performance.[53]

ACARA also responded to concerns about My School using the NAPLAN data, which assesses student performance, to provide information on schools:

It is quite acceptable to get an overall measurement of the performance of students in the school by averaging the performance of the students within it. That is done in every international survey of educational achievement. We compare the performance of countries—for example, in PISA—by doing the same thing. We aggregate up from the student level to a national level and indeed to an international level. So aggregating the responses of individuals within a unit is quite acceptable in measurement terms and that is what is done in, for example, educational research. All educational research is based on aggregating performance of individuals within the unit.[54]

Facilitating conversations

The My School website provides valuable information to facilitate conversations at many levels. Dr Ben Jensen told the committee that he supports NAPLAN and the My School website as:

...this website has highlighted both the positive and negative issues within schools, and it is great that it has encouraged these conversations to take place – not just engaging parents and families with the school but facilitating discussions internally within schools.[55]

Mrs Sharyn Lidster, Department of Education, Tasmania also reported the website information is leading to increased conversations between parents and teachers:

It certainly has increased the level of discussion. Of course, our media have also jumped in and made some simplistic summaries of that information, but I think that the rich conversations that are happening at a local level between parents and the schools are where a lot of the positive outcomes are coming from in publicly reporting the information.[56]

Mrs Lidster added:

I can only speak anecdotally in relation to parents’ engagement with their local schools. Anecdotally, there have been a lot more conversations around that. I can say that, when you provide a whole suite of information, there is probably no school that is hitting the top marks in every area. That is the nature of any endeavour, especially something as complex as a school and providing an education for students. There will be areas of strengths and areas of weaknesses. That is where it is valuable to share it publicly with parents, so that they can have that discussion with the school around how they can improve it. Because it is a partnership, things like attendance, for example, are not in the total control of the school. They can encourage attendance but in actual fact parents have a direct responsibility in relation to attendance. There are great conversations happening in relation to it.[57]

Increased transparency

While advocating for including a value-added score on the website,[58] Dr Ben Jensen acknowledged the greater transparency provided by the My School website:

My School has been a dramatic increase in transparency in school education. That has been undertaken in an industry or a sector that has not had a lot of transparency—particularly when you consider that this year we are going to put financial information on it as well. That is a huge increase from where we have been. I think whenever you have that sort of change you are going to get this sort of emphasis.[59]

Dr Jensen spoke in more detail about increased transparency leading to greater student improvement:

...Overall what you are getting is a gradual decline over the years and an inability within a system to effectively help those students at or below minimum levels. As I said before, clearly the students at year 9 who are at or below minimum levels of writing literacy are still improving in absolute terms but it is the rate of improvement—this is where I greatly support the transparency of My School. We are going to have some of the greatest improvements by shining a light on the fact that we have this issue because this data is not well known. We are only beginning to develop effective programs to address these issues. That is why I would advocate that we need greater systems that immediately flow to the students who are performing at these levels. The sorts of equity programs which I would advocate have been successful in countries only focus on equity and not on absolute height. These are the programs that very high-performing countries have.[60]

Government senators view

Government senators note that the intention of COAG was for school performance data to be published in the context of broader information about a school's students, teachers and resources.[61] Therefore, along with the NAPLAN results, each school has a self-written profile page where it describes the mission and values of the school, the number of students, number of teacher and non-teaching staff, the attendance rate and any additional programs and support in place for students. Government senators emphasise that the data on the website should be considered along with an understanding of the context of the school and the composition of the student body.

Government senators agree that apart from the uses at the system and school level, the information provided by NAPLAN via the My School website is acting as a useful tool for parents to engage directly in conversations with teachers and ask questions about what is happening in their school.

Further development of the My School website

It was always intended that the My School website would be developed in stages and that subsequent versions would be improved by the additional of information as it became available.[62] To this end, ACARA advised that it was asked by education ministers to investigate proposals for enhancing the website including:

school financial data;
nationally comparable senior secondary information;
satisfaction with schooling;
student population indicators;
growth data on literacy and numeracy achievement;
teaching staff and levels of expertise;
using student-level data to compute the Index of Community Socio-Educational Advantage (ICSEA)
other enhancements to ICSEA
reporting of results; and
action to minimise misuse of My School data.[63]

Making more information available was advocated by ACER, which stated that the website 'has the potential to become a valuable single source of comparable, public information about every Australian school'.[64]

Government senators agree with advice from ACER that '[m]any of the current concerns about My School are likely to be addressed by broadening the range of information provided and increasing levels of user choice'.[65]

Government senators note that a working party made up of educational experts including literacy and numeracy specialists, principal organisations and representatives from the Australian Education Union and Independent Education Union of Australia was formed. It is providing advice on the use of student performance data and other indicators of school effectiveness.[66] Recommendations from the working party were considered by MCEECDYA on 15 October 2010. Ms Trimper, APPA, who is part of the working party, confirmed that the recommendations developed by the working party will go a long way to addressing the concerns that have been raised about the website.[67]

Government senators note that the substantial amount of work undertaken by the working party and the ability of My School version 2 to address concerns raised about the website has been given limited acknowledgment in the committee majority report. Government senators support the view expressed by Dr Peter Hill, CEO, ACARA, who told the committee that he believed the publication of the data has improved the quality of the data.

Making more information available

Following the meeting on 15 October 2010 with state and territory education ministers, the Minister for School Education, the Hon Peter Garrett MP, reported that ministers supported most of the enhancements proposed by the working party and that school finances and richer community detail will now be reported. He stated:

This is a really big reform for us and it's important that we get the website right. We've asked for some further work to be done on how the site will look and details about how it will function. But certainly we're very pleased that we're starting to see additional information coming to MySchool. Additional information that will enable parents, teachers, the communities and others to get a good sense of the educational progress that is being made by their students and by their schools right across the country.[68]

The minister added that the second iteration of My School will provide a leading information source for parents, teachers and school communities. It will also provide better information for governments to consider policies to best support education.[69]

My School version 2 will address many of the issues raised in submissions. Some specific examples are outlined below.

Measures to address the misuse of data

Government senators note that to address the misuse of data, for example, using it to create 'league tables', on 15 October 2010 MCEECDYA endorsed implementation of the following measures to protect the integrity of data on the site and the collection of direct student data:

a 'click wrap'[70] requiring users to indicate their agreement up-front to terms and conditions of use of My School data;
a tool to deter automatic scraping of data from the website.[71]

These measures will be implemented prior to the release of version 2 of the website.[72]

Government senators view

Government senators emphasise that the My School website is not a league table and support mechanisms to prevent and deter the practice of using My School data to create such tables. My School allows people to go to the website and view the NAPLAN results in an appropriate context.

ICSEA changes

APPA President Ms Leonie Trimper emphasised the concerns that the ICSEA values caused for their members. She suggested that to legitimately compare schools you need accurate details of the children at the school itself rather than details from a census data collection district.[73]

The use of Census Collection District (CCD) data for ICSEA values was considered by MCEECDYA on 15 October 2010. ACARA acknowledged that the use of this census data resulted in anomalous ICSEA values in a small number of cases. At the 21 October 2010 Estimates hearing Dr Peter Hill, Chief Executive Officer, ACARA, told the committee that ministers have agreed to move from census-based data to a model where information is obtained from parents:

...We have looked at it and now we have a new formula, which will mean that the ICSEA value of at least 70 per cent of schools, the socio-economic status component, will be based on direct student measures.

...It improves the predictor of validity by seven per cent. It means that we will not get the anomalies we have had in the past where the census collection district does not properly characterise the nature of the students who live in them.

...The only instances where we will revert to census collection data is where we do not have the individual level data and that relates particularly to schools in very remote areas of the Northern Territory or in cases where the number of students for whom we have this direct data is too small to get a reliable fix.[74]

Dr Hill added that the new formula will be used when the revised website is released in December 2010.[75] The use of individual student information on parental education and occupation over census data was supported in evidence to the committee.[76]

Enhanced reporting

The provision of more information, particularly qualitative and contextual information, was supported by witnesses.[77] Government senators note that MCEECDYA has agreed to expand the contextual information about schools by publishing the percentage of students with a language background other than English. Principals will also be able to include more information about their school. In addition, information on student absences, withdrawals and exclusions from NAPLAN testing will be more prominent. The website will show the growth in learning for students who took the test in 2008 and 2010 and were in the same school both years.[78] Schools will also be able to provide a commentary on their results. Comments will be collected and reported in the first half of 2011. Further enhancements are also planned for My School version 3, such as nationally comparable senior secondary information and teaching levels of expertise.[79]

Government senators point out that after this year's NAPLAN testing we are now in a position to see progress of the same children two years later, for example seeing how the grade 3 students tested in 2008 are now doing in grade 5 in 2010.

Enhanced search facilities

APPA called for a more interactive website where more detailed comparisons are possible.[80] Government senators note and support the recommendation from ACER to develop other ways of comparing student results across schools which would allow users to choose their own comparisons.[81]

Government senators note that for version 3 of the website, MCEECDYA has agreed that ACARA will investigate enhancing search facilities and filters which would allow users to refine their searches for like and statistically similar schools.[82]

Need for better communication

Mrs Sharyn Lidster, Department of Education Tasmania told the committee that a lot of education is needed to support schools in making best use of the data and being able to talk about their performance with their communities.[83] ACARA endorsed this view, and Dr Hill said that improving communication of the information would be a focus for the next version of the website.

There is always a big communication issue with presenting any form of complex information. We are conscious this time around—because we have a little bit more time than we did last year—that we need to put a lot of work into communicating ahead of the release about what the website will look like and the sorts of caveats that were in the first version. What we said in the first version is: if you see one result that might not be to your liking, that in itself may not be significant. Look at the previous year and then, if you have three years, look again. Certainly, if you have an indication that there is a problem three years in a row, you should take it seriously. One poor result does not necessarily mean you have a problem. It could be an aberration. That particularly applies to small schools. We would say that, particularly for small schools, you have to be very careful in interpreting these roles. Indeed, the errors in measurement around those estimates will be larger for those schools. The report on the current website and on the new one makes that very clear.[84]

Government senators note the inherent difficulty in presenting complex information and the intention by ACARA to put more work into better communicating the limitations of the data on the website.

Availability of financial data

One of the enhancements agreed by state and federal education ministers is requiring schools to list financial information. This will include 2009 recurrent income, disaggregated by source of funding (Commonwealth government; state/territory government; school initiated fees, charges and voluntary contributions; and other sources such as parent or third party initiated). It will also include 2009 capital expenditure broken down by source of funding.[85] The MCEECDYA Communiqué from the 15 October meeting noted that school financial information will be reported from 2008 to 2010 when the new version of the website is available in December 2010.[86]

This issue has received some media reporting, with the WA Primary Principals Association President Stephen Breen expressing concern that published financial data may present a misleading picture of how much money is spent on a student. He argued that public schools would see their costs inflated because they would also have to include the cost of education department head office bureaucrats. He added that he would like to see the data include school assets such as real estate.[87] The WA Director of the Association of Independent Schools, Valerie Gould, was concerned that publishing such financial data would lead to 'simplistic' comparisons such as linking school wealth to test scores. She added that it would be difficult to compare schools across difference sectors as costs and expenses are not treated in the same way:

For example, independent schools had to account for depreciation and insurance that State governments covered for public schools.[88]

These concerns were echoed in Victoria, where it was reported that the disclosure of private income will be misleading as:

They must cover costs that individual government schools don't, such as salaries, cleaning, maintenance, capital works, depreciation and loan interest.[89]

The Chief Executive of Independent Schools Victoria argued that schools would not be able to put their income and expenditure in context.[90] The Executive Director of the Association of Independent Schools, NSW, Mr Geoff Newcombe, stated that he did not believe private schools should have to report income from all sources:

I don't think there is a lot of relevance in disclosing a school's assets...It would be almost impossible to compare. We are looking at the recurrent costs of education children.[91]

The Australian Education Union argued in favour of disclosing all financial information such as trusts, foundations, bequests, share and property portfolios to provide an accurate account of the financial resources of schools and increase transparency.[92] Dr Ben Jensen also supported the publication of financial data:

I think that when we introduce financial data onto My School we are going to start a lot of conversations about whether our resources are being effectively spent to help students. That is a conversation that is long overdue in school education. I think that parents will ask—and I am not just talking about parents with children in independent schools—‘I am spending X for this; am I really getting value for money?’ But I also think we will get conversations such as ‘My state government is spending this per student; why are they performing at a lower level than students in other jurisdictions where the governments spend less?’—those sorts of conversations.[93]

The office of the School Education Minister Peter Garrett responded to the concerns raised by stating that representatives from independent and catholic schools have been involved in working out how to collect financial data so that it could be compared.[94]

Government senators view

Government senators note the financial data to be captured on My School version 2 is a good start but will not capture accumulated surpluses, assets, trusts or foundations. In the interests of providing more information, government senators believe that there should be full disclosure of financial assets including assets, foundations and investments, otherwise true comparisons will not be possible. There are limited obligations on private schools in return for public funding. Possibilities could include being obliged to open facilities to government schools and offer scholarships for disadvantaged and Indigenous students.

If non-government schools continue to expect a share in federal funding then full financial disclosure in the interests of the tax payer and the better allocation of resources must be required. If non-government schools do not wish to comply with full financial disclosure, then public funding should not be provided.

Recommendation 1

Government senators recommend that in the interests of transparency, accountability and facilitating meaningful comparisons, the My School website capture full disclosure of financial assets. Those schools who do not agree to this requirement should not receive public funding.

Problems with value-added measures

Government senators note the Coalition recommendation to revise the My School website by publishing a value-added measurement of school performance rather than the raw performance data results.[95]

The committee received an explanation of value-added measures from Dr Ben Jensen [96] and Professor Geoff Masters.[97] Professor Masters noted that the term value-added can mean many different things. It could mean the growth occurring or an attempt to take into account the background of the students, which he understood to be the aim of Dr Jensen. He described several problems with this approach:

One is the attempt to interpret residuals that I described before, where the residual is not just the influence of the school, necessarily; it could be all sorts of other influences. Another issue I have is that in that process you potentially lose the performance of the students themselves, so what becomes important is how much better or worse the school did than you predicted in your regression analysis, and you may end up saying well this school performed as well as expected, but in an absolute sense the literacy and numeracy levels could be unacceptably low, but it is as well as expected given their socioeconomic background. There is a bit of a risk in this approach of not recognising the absolute levels of achievement in the school and thinking everything is fine—I guess I have a philosophical problem with approaches that end up drawing conclusions like ‘students in this school are doing as well as expected given their backgrounds’. I can understand why people say that, but there is a fine line between explaining and making excuses. Sometimes I worry about that line.[98]

ACER pointed out that the current comparisons used are superior to the approaches used in other countries and outlined the problems with the school comparisons used in England which are advocated by the relevant Grattan Institute report:[99]

...school comparisons in England are based on 'contextualised value added' (CVA) measures which are constructed by first predicting how students in a school will perform from their backgrounds and then calculating the difference between the predicted and actual performances of students in each school. This method has the apparent advantage of providing a single number for each school enabling it to be compared with every other school in the country. These CVA measures are residuals that are assumed to reflect the contribution of the school, but in fact also reflect any other influences not taken into account in making the 'prediction'.[100]

Importantly, Government senators recognise that this method also has the disadvantage of obscuring students' actual levels of performance. As noted by ACER, when using such a system, 'students in a school can be judged to be performing as well as expected even when their absolute achievement levels are unacceptably low'.[101] Government senators also note the critical point made by ACER that, unlike the system in use in England, My School does not compare every school with every other school but encourages comparisons only between schools with similar student intakes.[102]

Conclusion

Rather than more convoluted analysis, clearly the solution to the issues raised with the website is providing more information, not less. Government senators welcome the calls for more contextual information to be provided by schools on the My School website and note that this was always the intention over time. The launch of the My School website in January 2010 was an important step forward in reporting the measurement of student performance and progress as well as increasing transparency and accountability. My School version 2 will be the next step in addressing these calls for more information to be provided. Government senators note that the next version is to be available in December 2010.

My School places the school at the centre of reporting, with its own home page containing a range of rich information and context about the school. In contrast, reporting overseas often places the school as a line in a table. Government senators note and support the intention to continue building on and improving the My School website to make more information available and improve and update the information that is there.

My School has shown there are schools with student populations of similar social backgrounds which are doing very different things, with many achieving unexpectedly good results. It has also highlighted schools which need additional assistance, which enables government to allocate additional resources to struggling schools.

Government senators note the success of the working party announced to address stakeholder concerns in relation to the My School website.

Government senators emphasise that the debate on NAPLAN assessment and the My School website is just one part of the broader education reform agenda being addressed in partnership with the states and territories. These broader issues include the development of the national curriculum, the school funding review underway, providing additional assistance to disadvantaged schools and improving the quality of teaching.[103]

Senator Gavin Marshall Senator Catryna Bilyk

Deputy Chair

Navigation: Previous Page | Contents | Next Page