Chapter 3

Risk Estimation Matrix

Background

The Import Risk Analysis process

3.1 As indicated in Chapter 1, the committee has, over recent years, undertaken a number of inquiries into biosecurity and quarantine arrangements in relation to the importation (or proposed importation) of specific plant or animal products. In particular, the committee has undertaken a number of inquiries into the Import Risk Analyses (IRAs) for various plant products – including bananas from the Philippines and apples from New Zealand.

3.2 These inquiries gave stakeholders the opportunity to raise concerns in relation to IRAs conducted for specific products. Industry stakeholders also outlined a number of concerns about the IRA process more generally.

3.3 DA described the IRA process as 'an important part of Australia's biosecurity policies'[1] and argued that the IRA process:

...enables the Australian Government to formally consider the risks that could be associated with proposals to import new products into Australia. If the risks are found to exceed Australia's appropriate level of protection (ALOP), risk management measures are proposed to reduce the risks to an acceptable level. But, if it is not possible to reduce the risks to an acceptable level, then no trade will be allowed.[2]

3.4 The committee notes however, that as far back as 1996, concerns were being expressed regarding the way in which import risk analysis was being undertaken. The Nairn review noted for example, that:

A great deal of concern was expressed to the Review Committee on the way risk analysis is conducted on applications to import animals, plants or their products into Australia. There is a lack of confidence in the process used for such analysis, and the recommendations contained in Chapter 7 are designed to rectify the problems that were brought to the Review Committee's attention. Industry and the general public need a greater opportunity for having their views considered and the process should be conducted in a way that is transparent, scientifically based and with a mechanism for appeal on process. All this needs to be done in the context of Australia's international obligations.[3]

Appropriate Level of Protection (ALOP)

3.5 The committee acknowledges that successive Australian Governments have maintained a conservative, but not a zero-risk approach, to the management of biosecurity risks. DA argue that this approach:

...is expressed in terms of Australia's appropriate level of protection (ALOP), which reflects community expectations through government policy and is currently described as providing a high level of protection aimed at reducing risk to a very low level, but not to zero.[4]

3.6 The committee is aware however, that stakeholders have expressed reservations about Australia's current definition of ALOP, the current import process arrangements, the way DA Biosecurity calculates risk and the formal IRA process itself.

Current inquiries – Risk Estimation Matrix

3.7 The committee notes that over many years, industry stakeholder groups (for example, fruit and vegetable growers) have raised concerns about the way DA Biosecurity calculates risk based on the Risk Estimation Matrix (REM).

3.8 These concerns were again raised during the committee's inquiry into the importation of pineapple, and they were also raised during both the ginger and the potato inquiries. In discharging its responsibilities in relation to the pineapple reference, the committee was provided with conflicting evidence in relation to the way in which DA Biosecurity calculates risk based on the REM.[5]

3.9 Tropical Pines Pty Ltd for example argued that:

The risk matrices and method of assessing risk, used by DAFF Biosecurity are heavily biased toward achieving overall risk assessments of low or very low. This is a concern for all risk assessments undertaken by DAFF Biosecurity and not just the risk assessment for the import of Dickey sp.[6] Working within the constraints of these risk matrices, Tropical Pines and Biosecurity Queensland have concluded the overall risk is either moderate or high, rather than low.[7]

3.10 Mr Glenn Taniguchi, Research Associate, College of Tropical Agriculture and Human Resources, University of Hawaii also stated that:

The risk rating developed by Biosecurity Australia [now DA Biosecurity] does not clearly state how rankings were determined. The nomenclature of raking probability of likelihoods is heavily skewed and biased towards the low probability.[8]

Engagement of consultant

3.11 From the early stages of the committee's pineapple inquiry it was evident that DA Biosecurity's method of calculating risk and the REM itself would be a central issue for the inquiry and to the committee's deliberations.

3.12 The committee was also aware that in order to give appropriate consideration to stakeholders' concerns about DA Biosecurity's calculation of risk, it would be necessary to review the way in which DA Biosecurity use the REM when preparing IRAs for various products.

3.13 In conducting its inquiry, the committee was aware that the design and operation of the REM is a highly technical and specialised area. As a result, the committee made a decision to seek independent advice in relation to the matrix used by DA Biosecurity as part of the IRA processes.[9]

3.14 The process of identifying a consultant best able to provide appropriate advice and comment on the matrix took several months. During its consideration of a number of consultancy proposals, the committee was mindful that it was not only seeking specialist advice, but advice that was both informed and independent.

3.15 In contacting a number of major organisations and individual risk analysis experts within Australia, the committee became aware that many potential consultants had been employed by DA Biosecurity in the past or had current links to DA through various committees, working groups, research institutions and panels. In general, these potential consultants declined to apply for the consultancy on the basis of a perceived or actual conflict of interest. Conversely, the committee made a decision that it was inappropriate to engage the services of those professionals who had previously, or were currently, providing advice to industry or other stakeholder groups. Overall the committee was surprised by the extensive links that all risk analysis experts approached had with either DA Biosecurity or with various industry bodies.

3.16 The committee decide to look outside Australia to identify a suitably independent consultant, eventually engaging Mr Chris Peace, Principal Consultant, Risk Management Ltd, Wellington, New Zealand.

3.17 Mr Peace's report – Advice on the risk estimation matrix used by DAFF Biosecurity as part of the Import Risk Analysis process (the Peace report) – was provided to the committee on 10 January 2013, and published on the committee's website on 6 February 2013. The report was also provided to DA on 7 February 2013, with a request for DA Biosecurity's preliminary views prior to a final hearing on 12 March 2013. A copy of the Peace report is included at Appendix 8.

The Peace report

Terms of reference

3.18 The agreed terms of reference for the review undertaken by Mr Peace were as follows:

Conduct a literature review covering:
- earlier DAFF Biosecurity IRA documents published on the DAFF website or elsewhere;
- any comparable Risk Estimation Matrices developed or used elsewhere;
- published academic literature critiquing the design and use of matrices.
Critique the DAFF Biosecurity Risk Estimation Matrix from an informed position.
Develop and test alternative approaches to quantitative or semi‑quantitative risk analysis, some using alternative matrices.
Suggest risk analysis techniques that would enable DAFF Biosecurity to report more effectively on the nature of the risk.
Report to the committee by an agreed date and attend a teleconference meeting/hearing at an agreed time.[10]

3.19 The review of the REM included a comparison of the matrix with guidance in the joint Australia/New Zealand Standards Handbook – HB 436 Risk Management Guidelines: a companion to AS/SNZ 4360:2004 (SA/SNZ, 2004), and draft joint handbook HB 89 Risk Management – Guidelines on risk assessment techniques (SA/SNZ, 2001). The author's approach placed the DA Biosecurity REM in the overall context of international treaties, codes, agreements and standards. The review also provided comment and guidance on the use of consequences/likelihood matrices used for risk analyses such as the DA Biosecurity REM.[11]

Issues

Description of REM in IRA Handbook

3.20 In reviewing the REM, Mr Peace noted that the IRA Handbook, 'does not mention, let alone describe the use of, the DAFF risk estimation matrix'.[12] Mr Peace also stated that:

If the matrix is to be seen as a valid risk technique, capable of withstanding legal scrutiny, its development and application ought to be the subject of a detailed description.[13]

Unreliability of qualitative descriptors

3.21 Mr Peace noted that the Fijian ginger IRA[14] and the New Zealand apple IRA[15] include probability ranges that were not given in other reports. It was also noted that the New Zealand apple IRA gave midpoints of the ranges which were not included in other reports. The indicative probability ranges are shown in Table 3.1 below.[16]

Table 3.1—Nomenclature for qualitative likelihoods – Fijian ginger IRA

Column 1	Column 2	Column 3	Column 4
Likelihood	Descriptive definition	Indicative probability (P) range	Midpoint (if uniform distribution used)
High	The event would be very likely to occur	0.7 < P ≤ 1	0.85
Moderate	The event would occur with an even probability	0.3 < P ≤ 0.7	0.5
Low	The event would be unlikely to occur	0.05 < P ≤ 0.3	0.175
Very low	The event would be very unlikely to occur	0.001 < P ≤ 0.05	0.026
Extremely low	The event would be extremely unlikely to occur	0.000001 < P ≤ 0.001	0.0005
Negligible	The event would almost certainly not occur	0 ≤ P ≤ 0.000001	0.0000005

3.22 Mr Peace noted that qualitative likelihood descriptors – and definitions, without reference to numeric probabilities – are prone to wide interpretation. He used the following example from former CIA officer Sherman Kent who wrote:

A few days after the estimate appeared, I was in informal conversation with the Policy Planning Staff's chairman. We spoke of Yugoslavia and the estimate. Suddenly he said, "By the way, what did you people mean by the expression 'serious possibility'? What kind of odds did you have in mind?" I told him that my personal estimate was on the dark side, namely, that the odds were around 65 to 35 in favour of an attack. He was somewhat jolted by this; he and his colleagues had read 'serious possibility' to mean odds very considerably lower. Understandably troubled by this want of communication, I began asking my own colleagues on the Board of National Estimates what odds they had in mind when they agreed to that wording. It was another jolt to find that each Board member had had somewhat different odds in mind and the low man was thinking of about 20 to 80, the high of 80 to 20. The rest ranged in between.[17]

3.23 Mr Peace noted that such variations in interpretation have led to a body of research on judgement. Such research indicates that there are large differences in the way in which people understand risk-descriptors such as those used in Table 3.1, and that individual interpretation may lead to confusion and errors in communication. Mr Peace cited specific research undertaken in relation to interpretations of likelihood terms used by the Intergovernmental Panel on Climate Change (IPCC) to communicate 'uncertainty'. The terms use a set of probabilities accompanied by global interpretational guidelines. Mr Peace noted that researchers found that respondents' judgements deviated significantly from the IPCC guidelines, even when the respondents had access to these guidelines.[18]

3.24 In this regard Mr Peace stated that:

From this research and our experience we find it likely that DAFF risk analysts may place their own interpretations on the words used in table 2.1 of the Malaysian pineapple report (DAFF 2012b) and other DAFF/Biosecurity reports. In making this statement we are aware the word likely is, itself, open to interpretation. We therefore suggest there is an 80% probability of idiosyncratic interpretation of the DAFF nomenclature for qualitative likelihoods. This probability might be revised following research within DAFF.[19]

Combination of qualitative likelihood terms

3.25 Mr Peace noted that Table 2.2 in the Malaysian pineapple IRA (and other DA Biosecurity reports) sets out rules for combining descriptive likelihoods. Mr Peace observed that:

No rationale or source for these rules is given, making the rules opaque and difficult to comment on. They appear to be the result of combining probabilities and so may be based on logic. If this is the case, DAFF officials should be able to explain it.[20]

3.26 However, Mr Peace also noted that the need for Table 2.2 'only exists if a risk analyst needs to estimate the qualitative likelihood of three events giving rise to the likelihood of a specified consequence'.[21] Mr Peace concluded that:

This is not good risk analysis practice and is not necessary if establishment of a pest is seen as an event or change in specific circumstances while entry, import and distribution are causes of establishment.[22]

Use of the REM in practice

3.27 The Peace report examined the use of the REM in practice and noted that applying 'the rules for combining qualitative likelihoods can give some apparently strange results'. The report noted for example that:

Combining two qualitative low likelihoods gives a very low likelihood. However, low has a maximum indicative probability of 0.3 in the Malaysian pineapples report and 0.3 x 0.3 = 0.09. The resulting 0.09 is within the low range of indicative probabilities: should a risk analyst determine the probability is low (based on the indicative probabilities) or very low (based on the rules for combining qualitative likelihoods)?[23]

3.28 The Peace report argued that the distinction outlined above is significant, particularly as very low is Australia's ALOP. A low risk would require mitigation measures whereas a very low risk would be acceptable. He further argued that this issue has the potential 'to lead to litigation following refusal to allow entry of a low risk commodity when a slightly different analysis might have shown it to be a very low risk commodity'.[24]

3.29 The Peace report also raised a further problem of interpretation faced by DA Biosecurity risk assessors:

...0.3 is the top of the low range and bottom of the moderate range. If a risk analyst determined the probability of an event was 0.3 should they name it low or moderate?[25]

3.30 Mr Peace further argued that:

Matrices are too often poorly designed and incorrectly interpreted. If they are to be used, they must be simple, based on relevant data, used following a clear understanding of the nature of a risk, and with their limitations understood by risk assessors and decision-makers.[26]

Consequence scales – geographical impacts

3.31 The Peace report observed that the methodology in the REM describes the assessment of consequences. It is also noted that four levels of consequence are considered for levels of Australian community defined as:

Local: an aggregate of households or enterprises (a rural community, a town or a local government area).
District: a geographically or geopolitically associated collection of aggregates (generally a recognised section of a state or territory, such as 'Far North Queensland').
Regional: a geographically or geopolitically associated collection of districts in a geographic area (generally a state or territory, although there may be exceptions with larger states such as Western Australia).
National: Australia wide (Australian mainland states and territories and Tasmania).[27]

3.32 Mr Peace suggested that whilst the four levels of consequence are reasonable, they may apply to any size of community:

For example, a small community might be a major contributor to the regional or national economy. As shown, such a contribution may be understated. The reverse might be true with a pest having trivial national impacts felt catastrophically at a local level.[28]

3.33 Mr Peace suggested that this problem could be overcome by 'developing consequence scales based on, for example, national GDP, percentage of national crop at risk, or viable planting area at risk'.[29]

Does the REM overstate or understate the level of risk?

3.34 The Peace report argued that it is possible that the rules for combining qualitative likelihoods, either overstate or understate the level of risk in some cases. It also suggested that 'the rules are opaque with no source cited and therefore leave in doubt their reliability'.[30] Mr Peace also observed that:

Two of the reports provide indicative probability ranges. These would be most helpful if their sources were cited; we are again left with doubt about the provenance and reliability of the indicative probabilities. Furthermore, our calculations [the author refers to Tables 3 and 4 of his report] suggest that some indicative probability range combinations may give results that breach the DAFF rules for combining qualitative likelihoods.

Overall, combining the likelihoods and/or their indicative probabilities may either overstate or understate the level of import risk.[31]

Risk perception

3.35 Mr Peace argued that if the design of the REM is to be improved, risk analysts need to know and understand the perception of risk, both in DA Biosecurity and external stakeholders, including the committee.

3.36 Mr Peace noted that risk perception is defined in the ISO Risk Management Vocabulary as 'the stakeholder's view on a risk' and 'reflects the stakeholder's needs, issues, knowledge, belief and values'.[32]

3.37 The author argued that:

Risk perceptions of external stakeholders may be intuitive feelings, based on media reports (Slovic, 2000). Some stakeholders may believe that levels of risk are increasing whereas the reverse may be the case. DAFF risk analysts need to understand the risk perceptions of external stakeholders as distinct from their professional perception of risk.

In Australia, public perceptions of biosecurity risks may be shaded by, for example, environmental damage caused by the release of wild rabbits in the 1800s and the harm caused by cane toads. Or there may be a proposal to import from overseas an exotic species or a species already in Australia that can carry some disease or pest (for example, the recent change to allow imports of European rabbits that might carry epizootic rabbit enteropathy).[33]

3.38 Mr Peace stressed the importance of risk perception and suggested that such 'risk perceptions should be incorporated into risk criteria used to analyse the consequences of a given import risk'.[34]

Conclusions

3.39 The Peace report concluded that the existing Australian biosecurity REM does not meet best practice because:

it combines likelihoods with events and consequences;
it is opaque in describing how to combine likelihoods;
probability and likelihood seem to be confused even though they are distinct concepts;
sources for the indicative probabilities used in recent reports are not given; and
the labels on the consequence and likelihood scales and risk level cells are very similar.[35]

Recommendations

3.40 Based on its analysis and conclusions, the Peace report made a number of recommendations, including that:

The DA Biosecurity REM be redesigned as a simple consequence/likelihood matrix to overcome the deficiencies identified in the report.
The Senate Rural and Regional Affairs and Transport Committee encourage DA to develop the use of fault tree, event tree and bow-tie analyses and other techniques to help understand and show the nature of import risks. This should be done in combination with a redesigned consequence/likelihood matrix to help determine the level of risk.
In order to aid transparency in import risk analysis and decision-making, DA revise the IRA Handbook to include full details of techniques available to DA risk analysts and any underlying data or research validating those techniques.
A revised IRA Handbook include a copy of the Peace report's draft Import risk analysis effectiveness checklist. (The checklist was developed to be an assurance tool demonstrating each import risk analysis meets the World Trade Organization criterion of an 'objective and defensible' import risk analysis. This might be combined with the DA IRA template that now seems to be in use).[36]

Eminent Scientists Group's comments on the Peace report

3.41 On 8 February 2013, the Rural and Regional Affairs and Transport Legislation Committee (the Legislation committee) conducted a hearing in relation to its inquiry into the Biosecurity Bill 2012. During the hearing, Dr John Radcliffe, Chair of the Eminent Scientists Group (ESG) was invited to review Mr Peace's report.

3.42 In reviewing the Peace report, the ESG noted that whilst Mr Peace 'appears to recognise the quality of scientific rigour provided by DAFF Biosecurity in undertaking Import Risk Analyses' the report also discusses changes that could be made to current analytical practices.[37]

Language

3.43 The ESG noted Mr Peace's comments in relation to the variations in meanings and definitions between treaties, agreements and standards and acknowledged that much of the debate revolving around the use of IRAs is of an 'etymological' nature. The ESG suggest that:

The Committee may wish to explore whether more consistent and better understood terms could be identified for use in Import Risk Analyses (IRAs) by DAFF Biosecurity to minimise what Peace refers to as 'idiosyncratic interpretation'...[38]

3.44 The ESG also argued, however, that these 'issues are largely a matter of risk communication and may not materially alter the scientific outcome of the analysis.'[39]

Risk

3.45 The ESG noted Mr Peace's comments regarding risk being the likelihood of the consequences of an event, but argued that this statement may not be 'tacitly correct in that it discounts the likelihood of an event occurring in the first place'.[40] The ESG also stated that:

We are not convinced that DAFF's definition "risk being the likelihood of an event occurring" is wrong. An event may well occur ("a food product passes undetected through the barrier"), but it may or may not prove to have quarantine consequence ("the importer ate most of it and destroyed the remainder").[41]

Quantitative risk estimates and scientific uncertainty

3.46 The ESG supported Mr Peace's view in relation to quantitative risk assessment, in that the numerical element is only one part of what is essentially a judgement exercise. It further argued that in the context of analysing biosecurity risks from a proposed import:

...it should be recalled that the analysis has to resolve matters of scientific uncertainty in terms of the potential biological impact of a new species on agricultural practices or the natural environment, if any.[42]

3.47 The ESG further argued that:

When identifying the risks (hazards) that could eventuate from the introduction of new biological products at the border, the 'level of risk', the probability of occurrence, and the consequences will rarely have any prior measured estimations available in the Australian environment being addressed for the purposes of establishing an Appropriate Level of Protection for Australia (ALOP) that is defendable internationally.[43]

3.48 The ESG acknowledged the importance of those doing the estimations being free of any conflict of interest (in terms of benefitting from the judgements to be made). At the same time, the ESG argued that the existing process is designed for that purpose.[44]

Use of matrices and alternative instruments

3.49 In reviewing Mr Peace's theoretical discussion about qualitative risk matrices, the ESG acknowledged its members did not feel qualified to comment specifically on the mathematical issues involved. The ESG did however suggest that the Australian Centre of Excellence for Risk Analysis (ACERA) has been established specifically for this purpose and:

...receives funding from DAFF to research methodology for biosecurity analysis and could be asked more explicitly by DAFF to provide advice, including any effect of qualitative versus quantitative risk analysis on the consequences and methodology of sampling and on the forms of and use of matrices.[45]

3.50 The ESG suggested that the committee may also wish to seek advice from the ACERA on these issues, and noted that it had previously indicated its support for having a suitable independent party (such as ACERA) review the range of models used in the IRA process by Australia's major trading parties.[46] ACERA's views on the Peace report are outlined below.

Risk analysis checklist and revision of IRA Handbook

3.51 The ESG noted Mr Peace's suggestion in relation to a risk analysis checklist and indicated that it 'would support this as a constructive suggestion'. The ESG also acknowledged that the IRA Handbook will need to be revised as a result of the current review of the biosecurity legislation, and indicated its support for Mr Peace's suggestion that the IRA Handbook include full details of techniques available to DA Biosecurity risk analysts and any underlying data or research validating those techniques.[47]

DA's response to the Peace report

3.52 On 7 February 2013 the committee requested that DA Biosecurity review and provide an initial response to the issues raised by the Peace report. DA Biosecurity's response, dated 8 March 2013, is provided at Appendix 10.

3.53 The committee also received a detailed response from the (then) Department of Agriculture (DA) Secretary, Mr Andrew Metcalfe AO, on 22 May 2013 which is provided at Appendix 11.

SPS Agreement and International Standards for Phytosanitary Measures (ISPMs)

3.54 In responding to the Peace report, DA Biosecurity noted that it has been the policy of successive Australian governments that risk assessments used to establish phytosanitary measures be consistent with the SPS Agreement. DA also noted that this position is consistent with Australia's obligations as a signatory to the WTO.[48]

3.55 DA also indicated that in conducting import risk analyses for plant products, 'specific guidance is provided by the internationally agreed International Standards for Phytosanitary Measures (ISPMs) which are developed under the International Plant Protection Convention (IPPC)'.[49] It was argued that:

Other methods may be informative, but it is the SPS Agreement and the ISPMs that establish the basis for import risk analysis. In the case of animals and animal products, guidance similar to the IPPC is provided by the Office International des Epizooties (OIE).[50]

3.56 DA noted that although the Peace report referred to the IPPC and some of its training materials, it did not reference ISPM 2: Framework for pest risk analysis or ISPM 11: Pest risk analysis for quarantine pests, including analysis of environmental risk and living modified organisms, 'which provide substantial guidance on pest risk analysis for quarantine pests of plants'.[51]

3.57 DA also noted Mr Peace's observation that chapter two of the Terrestrial Animal Health Code, published by the OIE, does not define hazard, risk, risk analysis and risk assessment – and submitted that 'each of these terms is clearly defined in ISPM 5: Glossary of phytosanitary terms'.[52]

Development of REM

3.58 DA Biosecurity's response stated that the matrix-based approach to combining the likelihood of entry, establishment and spread of a pest or disease, with the consequences if that were to occur, first appeared in the draft IRA report for non‑domestic Felidae in February 2001.[53] It was noted that current form and labelling of the matrix subsequently appeared in the August 2001 Issues Paper for the generic import risk analysis for fresh pineapple fruit.

3.59 DA argued that the current methodology for assessing biosecurity risks:

...was the subject of substantial discussion between the Commonwealth Government and the states and territories. These discussions led to the formal endorsement of the current methodology through the Primary Industries Ministerial Council in 2002.[54]

Consequence and likelihood

3.60 In responding to Mr Peace's suggestion that the REM is not a true consequence/likelihood approach, DA argued that its risk analysis process aligns with the relevant international standard for plant pest risk analysis, 'in this case ISPM 11, by considering the probability of introduction and spread and the potential economic consequences'.[55] DA further argued that:

These are then combined using a matrix based approach as demonstrated in the IPPC training materials, and as endorsed in the annex to ISO:31010 as being 'strongly applicable' for assessing level of risk. DAFF's matrix combines the probability (or likelihood) of entry, establishment and spread with the consequences if that sequence of events were to occur. This results in an assessment of unrestricted risk. If the unrestricted risk is 'low' or greater, specific risk management measures are necessary.[56]

Description of REM in IRA Handbook

3.61 DA Biosecurity noted Mr Peace's comments regarding the REM not being included in the IRA Handbook and the recommendation that the IRA Handbook be revised to include details of techniques available to DA risk analysts and a description of the REM. In responding to this recommendation, DA Biosecurity argued that Mr Peace 'does not have a correct understanding of the purpose of the IRA Handbook':

...The IRA Handbook describes the administrative process for conducting import risk analyses, regulated steps under the Quarantine Regulations 2000, and relevant background information on domestic and international policies.[57]

3.62 DA Biosecurity further noted that:

To inform stakeholders and readers of import risk analysis reports, each report contains a detailed description of the methodology being employed and worked examples for combining likelihoods.[58]

3.63 However the Secretary of the Department of Agriculture later told the committee that the IRA Handbook is being revised:

I am preparing to withdraw the Handbook and make more up-to-date and comprehensive information available about the department’s role in managing imports into Australia, including the IRA process.[59]

Communication of risk

3.64 At the 12 March 2013 hearing, the committee questioned departmental officers about DA's response to the Peace report. In particular, the committee asked whether the report had raised any issues the department may be prepared to 'take on board'.[60]

3.65 In response, DA officers indicated that whilst they were already undertaking a review of the import risk analysis processes, there were a number of things in Mr Peace's report the department could look at and possibly develop further, particularly in relation to communicating risk:

I think the greatest value that Mr Peace's report provides is, again, identifying where we know one of the things we can work on more is risk communication as opposed to the risk method that we undertake. Certainly the suggestions that Mr Peace provides gives us some further food for thought about how we might better involve and engage stakeholders and make our risk assessments more open for stakeholders to be able to analyse and look at themselves.[61]

3.66 DA also indicated that its work with ACERA had resulted in the evaluation of, and improvement in, policies and operational programs. DA also noted that it is currently developing and seeking stakeholder feedback on new biosecurity regulations and guidelines, including the consideration of issues such as:

addressing regional differences in biosecurity status;
independent scientific review of import risk analyses;
the rights of appeal on the outcome of an import risk analysis; and
the application of Australia's appropriate level of protection.[62]

3.67 In concluding its comments DA argued that 'there is no substantive evidence that DAFF's risk analysis processes have not been effective in protecting Australia's favourable pest and disease status'.[63]

ACERA's comments on the Peace report

3.68 In correspondence to the committee dated 24 May 2013, the Secretary of DA indicated that he had asked the Australian Centre of Excellence for Risk Analysis (ACERA) to review Mr Peace's report.

3.69 In providing advice to DA on the Peace report, ACERA acknowledged that whilst some of the issues raised by Mr Peace were valid, the organisation felt that others were problematic. ACERA noted, for example, that the comment made by Mr Peace at a public hearing, that for 'Australia's risk analyses, using the qualitative matrices, if the overall assessment is 'negligible', there may be as much as a 10-15% likelihood of the risk being higher than 'negligible''[64] was:

...made, based on a qualitative interpretation of a qualitative risk analysis. While there is almost certainly at least some small chance that the risk is higher than negligible, there is no justification, or any conceivable rational basis on which one could quantify this chance.[65]

3.70 However, ACERA did acknowledge Mr Peace's criticism of qualitative assessments that 'do not provide clear guidelines regarding the meaning of indicative probability distributions'[66]:

In particular, in his testimony to the committee, he highlighted the difficulties of aggregating qualitative risk assessments without these. We agree that this raises a problem with the transparency of the overall assessment. We note that in other IRAs these indicative qualitative intervals are provided.[67]

3.71 ACERA's response noted Mr Peace's advocacy for the use of a bow-tie analysis in combination with a revised qualitative consequence/likelihood matrix and quantified fault tree and event tree analyses. ACERA agreed with Mr Peace's assertion that this type of analysis provides a way of visualising the causal process and can be an aid to understanding. It was also agreed that this kind of analysis can be qualitative or quantitative. However, ACERA argued that this type of approach does have a number of weaknesses:

Because this method uses fault 'trees' rather than 'graphs', it cannot represent all kinds of conditional dependencies. (This means that the simple arithmetic operations on probabilities may make incorrect independence assumptions) which may result in incorrect overall probabilities.
An event tree typically 'fans out' to numerous possibilities, meaning that they are inherently limited for modelling the impact of many factors or intervention actions.
The bow-tie analysis is based around a single event (the introduction of a single pest) and does not appear to provide any obvious way to aggregate. This means that scalability seems to be problematic for the bow-tie approach.[68]

3.72 ACERA's response concluded by noting that:

To our knowledge, there are no case studies demonstrating how bow-tie analysis (combining fault-tree and event-tree analysis) can be used for IRAs (particularly in combination with Australia's qualitative matrices as proposed by Peace). It is not advocated for use in IRAs in the peer-reviewed research literature.[69]

Meeting between Mr Peace and Department of Agriculture officials

3.73 On 3 July 2013, officers from the Department of Agriculture met with Mr Peace to discuss his review of the department's import risk assessment methods.

3.74 In addition to officers from the department, Professor Mark Burgman from the Centre for Excellence for Biosecurity Risk Analysis (CEBRA)[70] was also present at the meeting.[71] Meeting participants agreed that the points of difference related to the REM, and Mr Peace emphasised his view that 'the matrix was "opaque" for stakeholders outside DAFF who would find difficulties understanding how probabilities and consequences were combined'. The group also discussed the need for some alternative risk analysis technique that was transparent and the documentation and publication of the import risk analysis process.[72]

3.75 At the conclusion of the meeting, the group had reached agreement that:

the department would consider alternative options to better represent the import risk analysis process and outcomes; and
the import risk analysis process and techniques used will be documented and published on the DAFF website so stakeholders can download, read and better understand the process.[73]

Committee comment

3.76 As indicated previously in this report, the committee was aware from the early stages of its inquiry into the Malaysian pineapple IRA, that DA Biosecurity's method of calculating risk and the REM itself would be issues of central importance to the committee's inquiry.

3.77 The committee was also aware that, if it were to give appropriate consideration to the issues of concern to stakeholders, it would be necessary to have a clearer understanding of the REM and the way in which DA Biosecurity use it when preparing IRAs for various commodities.

3.78 The committee believes that the review undertaken by Mr Chris Peace has been both informed and independent. The Peace report has provided the committee with valuable information, in relation to processes for assessing or analysing risks, risk matrix literature, alternative risk techniques and the 'language' of risk.

3.79 The Peace report has also provided the committee with a more thorough understanding of DA Biosecurity's REM, the way in which DA Biosecurity use the REM to calculate risk, and the ways in which both of these could be improved.

3.80 The committee acknowledges the comments provided by the ESG, DA Biosecurity and ACREA (recently renamed CEBRA) in response to the Peace report. The committee notes that successive Australian governments have maintained a policy of risk assessments used to establish phytosanitary measures being consistent with the SPS Agreement. The committee also notes that this position is consistent with Australia's obligations as a signatory to the WTO.

3.81 The committee is aware that the current form and labelling of the matrix, and the current methodology for assessing biosecurity risks, has been developed over a considerable period of time. The committee acknowledges that these issues have been the subject of considerable high level discussion, and subsequent agreement between federal and state governments.

3.82 The committee notes that DA Biosecurity has not refuted many of the issues raised by the Peace report, including:

the unreliability of qualitative descriptors;
the combination in the DA Biosecurity REM of the terms likelihoods with events and consequences;
the opacity of the DA Biosecurity REM in describing the method for combining likelihoods;
the lack of sources for the indicative probabilities used in recent reports;
the apparent confusion of the terms probability and likelihood;
labels on the consequence and likelihood scales and risk level cells are very similar – which leads to confusion;
limitations in the consequence scales' use of geographical impacts;
difficulties associated with combining qualitative likelihoods; and
the need for risk perceptions to be incorporated into risk criteria.

Recommendation 3

3.83 The committee recommends that the Department of Agriculture give thorough consideration to the Peace report, as well as the underlying themes of all other recommendations contained in this report, in developing the new biosecurity regulations and guidelines.

3.84 In addition, the committee notes the Peace recommendation that a revision of the IRA Handbook should include full details of techniques available to DA risk analysts and any underlying data or research validating those techniques. The committee notes DA Biosecurity's response that some of this information is provided in individual IRA reports. The committee believes, however, that the REM used by DA Biosecurity to calculate risk has been identified as central to the IRA process, and therefore supports Mr Peace's recommendation to include these additional details in the IRA Handbook.

Recommendation 4

3.85 The committee recommends that the IRA Handbook should be amended to include full details of techniques available to Department of Agriculture risk analysts and any underlying data or research validating those techniques.

3.86 The committee is also of the view that the IRA Handbook should include an IRA effectiveness checklist similar to that recommended by Mr Peace (and included at Appendix 12) – a proposal the ESG indicated it would be support. In this regard, the committee notes that the DA Secretary has indicated that the department is preparing to withdraw the Handbook and make more up-to-date and comprehensive information available about the department’s role in managing imports into Australia, including the IRA process.

Recommendation 5

3.87 The committee recommends that the IRA Handbook should include an IRA effectiveness checklist similar to that recommended by Mr Peace.

3.88 The committee notes Mr Peace's comments in relation to 'perception of risk' and the suggestion that those undertaking risk analysis should know and understand how risk is perceived – particularly by external stakeholders. The committee agrees that if the IRA process or the design of the REM are to be improved, stakeholders' needs, issues, knowledge, beliefs and values – their risk perceptions – need to be taken into consideration. This is a proposal for which DA has also indicated support.

3.89 As indicated earlier in this report, the committee has over a number of years undertaken a number of inquiries in relation to the importation (or proposed importation) of specific plant or animal products. The committee's reports on these issues have always stressed the importance of stakeholders having their views taken into consideration and being able to fully participate in the import risk analysis process.

3.90 The committee supports Mr Peace's recommendation that stakeholder's risk perceptions should be incorporated into risk criteria used to analyse the consequences of a given import risk.

Recommendation 6

3.91 The committee recommends that stakeholders' risk perceptions should be incorporated into risk criteria used to analyse the consequences of a given import risk.

Recommendation 7

3.92 The committee recommends that the Department of Agriculture consider ways to improve the way it communicates risk (and the risk assessment process) to stakeholders.

3.93 The committee notes views expressed by Mr Peace in relation to the issue of geographic impacts and the assessment of consequences. The committee agrees with Mr Peace's view that the definitions used for the four levels of consequence – local, district, regional and national – are reasonable. However, the committee also agrees that they could nonetheless easily apply to any size of community.

3.94 The committee notes, for example, that it is often the case that a small community contributes significantly more to the regional or national economy than a large community. It is also true that the impact of specific pests can have a minor impact nationally, but prove devastating at a local level.

Recommendation 8

3.95 The committee recommends that the Department of Agriculture reconsiders the operation of geographic impacts in the IRA process, and give consideration to developing consequence scales based on, for example, national GDP, percentage of national crop at risk, or viable planting area at risk.

Navigation: Previous Page | Contents | Next Page