Chapter 3
Risk Estimation Matrix
Background
The Import Risk Analysis process
3.1
As indicated in Chapter 1, the committee has, over recent years,
undertaken a number of inquiries into biosecurity and quarantine arrangements
in relation to the importation (or proposed importation) of specific plant or
animal products. In particular, the committee has undertaken a number of inquiries
into the Import Risk Analyses (IRAs) for various plant products – including
bananas from the Philippines and apples from New Zealand.
3.2
These inquiries gave stakeholders the opportunity to raise concerns in
relation to IRAs conducted for specific products. Industry stakeholders also
outlined a number of concerns about the IRA process more generally.
3.3
DA described the IRA process as 'an important part of Australia's
biosecurity policies'[1]
and argued that the IRA process:
...enables the Australian Government to formally consider the
risks that could be associated with proposals to import new products into
Australia. If the risks are found to exceed Australia's appropriate level of
protection (ALOP), risk management measures are proposed to reduce the risks to
an acceptable level. But, if it is not possible to reduce the risks to an
acceptable level, then no trade will be allowed.[2]
3.4
The committee notes however, that as far back as 1996, concerns were
being expressed regarding the way in which import risk analysis was being
undertaken. The Nairn review noted for example, that:
A great deal of concern was expressed to the Review Committee
on the way risk analysis is conducted on applications to import animals, plants
or their products into Australia. There is a lack of confidence in the process
used for such analysis, and the recommendations contained in Chapter 7 are
designed to rectify the problems that were brought to the Review Committee's
attention. Industry and the general public need a greater opportunity for
having their views considered and the process should be conducted in a way that
is transparent, scientifically based and with a mechanism for appeal on
process. All this needs to be done in the context of Australia's international
obligations.[3]
Appropriate Level of Protection
(ALOP)
3.5
The committee acknowledges that successive Australian Governments have
maintained a conservative, but not a zero-risk approach, to the management of
biosecurity risks. DA argue that this approach:
...is expressed in terms of Australia's appropriate level of
protection (ALOP), which reflects community expectations through government
policy and is currently described as providing a high level of protection aimed
at reducing risk to a very low level, but not to zero.[4]
3.6
The committee is aware however, that stakeholders have expressed
reservations about Australia's current definition of ALOP, the current import
process arrangements, the way DA Biosecurity calculates risk and the formal IRA
process itself.
Current inquiries – Risk Estimation
Matrix
3.7
The committee notes that over many years, industry stakeholder groups
(for example, fruit and vegetable growers) have raised concerns about the way
DA Biosecurity calculates risk based on the Risk Estimation Matrix (REM).
3.8
These concerns were again raised during the committee's inquiry into the
importation of pineapple, and they were also raised during both the ginger and the
potato inquiries. In discharging its responsibilities in relation to the
pineapple reference, the committee was provided with conflicting evidence in relation
to the way in which DA Biosecurity calculates risk based on the REM.[5]
3.9
Tropical Pines Pty Ltd for example argued that:
The risk matrices and method of assessing risk, used by DAFF
Biosecurity are heavily biased toward achieving overall risk assessments of low
or very low. This is a concern for all risk assessments undertaken by DAFF
Biosecurity and not just the risk assessment for the import of Dickey sp.[6]
Working within the constraints of these risk matrices, Tropical Pines and
Biosecurity Queensland have concluded the overall risk is either moderate or
high, rather than low.[7]
3.10
Mr Glenn Taniguchi, Research Associate, College of Tropical Agriculture
and Human Resources, University of Hawaii also stated that:
The risk rating developed by Biosecurity Australia [now DA
Biosecurity] does not clearly state how rankings were determined. The
nomenclature of raking probability of likelihoods is heavily skewed and biased
towards the low probability.[8]
Engagement of consultant
3.11
From the early stages of the committee's pineapple inquiry it was
evident that DA Biosecurity's method of calculating risk and the REM itself would
be a central issue for the inquiry and to the committee's deliberations.
3.12
The committee was also aware that in order to give appropriate
consideration to stakeholders' concerns about DA Biosecurity's calculation of
risk, it would be necessary to review the way in which DA Biosecurity use the
REM when preparing IRAs for various products.
3.13
In conducting its inquiry, the committee was aware that the design and
operation of the REM is a highly technical and specialised area. As a result,
the committee made a decision to seek independent advice in relation to the
matrix used by DA Biosecurity as part of the IRA processes.[9]
3.14
The process of identifying a consultant best able to provide appropriate
advice and comment on the matrix took several months. During its consideration
of a number of consultancy proposals, the committee was mindful that it was not
only seeking specialist advice, but advice that was both informed and
independent.
3.15
In contacting a number of major organisations and individual risk
analysis experts within Australia, the committee became aware that many
potential consultants had been employed by DA Biosecurity in the past or had
current links to DA through various committees, working groups, research
institutions and panels. In general, these potential consultants declined to
apply for the consultancy on the basis of a perceived or actual conflict of
interest. Conversely, the committee made a decision that it was inappropriate
to engage the services of those professionals who had previously, or were currently,
providing advice to industry or other stakeholder groups. Overall the committee
was surprised by the extensive links that all risk analysis experts approached
had with either DA Biosecurity or with various industry bodies.
3.16
The committee decide to look outside Australia to identify a suitably
independent consultant, eventually engaging Mr Chris Peace, Principal
Consultant, Risk Management Ltd, Wellington, New Zealand.
3.17
Mr Peace's report – Advice on the risk estimation matrix used by DAFF
Biosecurity as part of the Import Risk Analysis process (the Peace report) –
was provided to the committee on 10 January 2013, and published on the
committee's website on 6 February 2013. The report was also provided to DA on 7
February 2013, with a request for DA Biosecurity's preliminary views prior to a
final hearing on 12 March 2013. A copy of the Peace report is
included at Appendix 8.
The Peace report
Terms of reference
3.18
The agreed terms of reference for the review undertaken by Mr Peace were
as follows:
-
Conduct a literature review covering:
-
earlier DAFF Biosecurity IRA documents published on the DAFF
website or elsewhere;
-
any comparable Risk Estimation Matrices developed or used
elsewhere;
-
published academic literature critiquing the design and use of
matrices.
-
Critique the DAFF Biosecurity Risk Estimation Matrix from an
informed position.
-
Develop and test alternative approaches to quantitative or semi‑quantitative
risk analysis, some using alternative matrices.
-
Suggest risk analysis techniques that would enable DAFF
Biosecurity to report more effectively on the nature of the risk.
-
Report to the committee by an agreed date and attend a
teleconference meeting/hearing at an agreed time.[10]
3.19
The review of the REM included a comparison of the matrix with guidance
in the joint Australia/New Zealand Standards Handbook – HB 436 Risk
Management Guidelines: a companion to AS/SNZ 4360:2004 (SA/SNZ, 2004), and
draft joint handbook HB 89 Risk Management – Guidelines on risk assessment
techniques (SA/SNZ, 2001). The author's approach placed the DA Biosecurity
REM in the overall context of international treaties, codes, agreements and
standards. The review also provided comment and guidance on the use of
consequences/likelihood matrices used for risk analyses such as the DA
Biosecurity REM.[11]
Issues
Description of REM in IRA Handbook
3.20
In reviewing the REM, Mr Peace noted that the IRA Handbook, 'does not
mention, let alone describe the use of, the DAFF risk estimation matrix'.[12]
Mr Peace also stated that:
If the matrix is to be seen as a valid risk technique,
capable of withstanding legal scrutiny, its development and application ought
to be the subject of a detailed description.[13]
Unreliability of qualitative
descriptors
3.21
Mr Peace noted that the Fijian ginger IRA[14]
and the New Zealand apple IRA[15]
include probability ranges that were not given in other reports. It was also
noted that the New Zealand apple IRA gave midpoints of the ranges which were
not included in other reports. The indicative probability ranges are shown in
Table 3.1 below.[16]
Table 3.1—Nomenclature for qualitative likelihoods –
Fijian ginger IRA
Column 1
|
Column 2
|
Column 3
|
Column 4
|
Likelihood
|
Descriptive definition
|
Indicative probability (P) range
|
Midpoint (if uniform distribution
used)
|
High
|
The event
would be very likely to occur
|
0.7 < P ≤ 1
|
0.85
|
Moderate
|
The event
would occur with an even probability
|
0.3 < P ≤ 0.7
|
0.5
|
Low
|
The event
would be unlikely to occur
|
0.05 < P ≤ 0.3
|
0.175
|
Very low
|
The event would
be very unlikely to occur
|
0.001 < P ≤ 0.05
|
0.026
|
Extremely
low
|
The event
would be extremely unlikely to occur
|
0.000001 < P ≤ 0.001
|
0.0005
|
Negligible
|
The event
would almost certainly not occur
|
0 ≤ P ≤ 0.000001
|
0.0000005
|
3.22
Mr Peace noted that qualitative likelihood descriptors – and definitions,
without reference to numeric probabilities – are prone to wide interpretation.
He used the following example from former CIA officer Sherman Kent who wrote:
A few days after the estimate appeared, I was in informal
conversation with the Policy Planning Staff's chairman. We spoke of Yugoslavia
and the estimate. Suddenly he said, "By the way, what did you people mean
by the expression 'serious possibility'? What kind of odds did you have in
mind?" I told him that my personal estimate was on the dark side, namely,
that the odds were around 65 to 35 in favour of an attack. He was somewhat
jolted by this; he and his colleagues had read 'serious possibility' to mean
odds very considerably lower. Understandably troubled by this want of
communication, I began asking my own colleagues on the Board of National
Estimates what odds they had in mind when they agreed to that wording. It was
another jolt to find that each Board member had had somewhat different odds in
mind and the low man was thinking of about 20 to 80, the high of 80 to 20. The
rest ranged in between.[17]
3.23
Mr Peace noted that such variations in interpretation have led to a body
of research on judgement. Such research indicates that there are large differences
in the way in which people understand risk-descriptors such as those used in
Table 3.1, and that individual interpretation may lead to confusion and errors
in communication. Mr Peace cited specific research undertaken in relation
to interpretations of likelihood terms used by the Intergovernmental Panel on
Climate Change (IPCC) to communicate 'uncertainty'. The terms use a set of
probabilities accompanied by global interpretational guidelines. Mr Peace noted
that researchers found that respondents' judgements deviated significantly from
the IPCC guidelines, even when the respondents had access to these guidelines.[18]
3.24
In this regard Mr Peace stated that:
From this research and our experience we find it likely that
DAFF risk analysts may place their own interpretations on the words used in
table 2.1 of the Malaysian pineapple report (DAFF 2012b) and other
DAFF/Biosecurity reports. In making this statement we are aware the word likely
is, itself, open to interpretation. We therefore suggest there is an 80%
probability of idiosyncratic interpretation of the DAFF nomenclature for
qualitative likelihoods. This probability might be revised following research
within DAFF.[19]
Combination of qualitative
likelihood terms
3.25
Mr Peace noted that Table 2.2 in the Malaysian pineapple IRA (and other
DA Biosecurity reports) sets out rules for combining descriptive likelihoods. Mr
Peace observed that:
No rationale or source for these rules is given, making the
rules opaque and difficult to comment on. They appear to be the result of
combining probabilities and so may be based on logic. If this is the case, DAFF
officials should be able to explain it.[20]
3.26
However, Mr Peace also noted that the need for Table 2.2 'only exists if
a risk analyst needs to estimate the qualitative likelihood of three events
giving rise to the likelihood of a specified consequence'.[21]
Mr Peace concluded that:
This is not good risk analysis practice and is not necessary
if establishment of a pest is seen as an event or change in specific
circumstances while entry, import and distribution are causes of establishment.[22]
Use of the REM in practice
3.27
The Peace report examined the use of the REM in practice and noted that
applying 'the rules for combining qualitative likelihoods can give some
apparently strange results'. The report noted for example that:
Combining two qualitative low likelihoods gives a very
low likelihood. However, low has a maximum indicative probability of
0.3 in the Malaysian pineapples report and 0.3 x 0.3 = 0.09. The resulting 0.09
is within the low range of indicative probabilities: should a risk
analyst determine the probability is low (based on the indicative
probabilities) or very low (based on the rules for combining qualitative
likelihoods)?[23]
3.28
The Peace report argued that the distinction outlined above is
significant, particularly as very low is Australia's ALOP. A low
risk would require mitigation measures whereas a very low risk would be
acceptable. He further argued that this issue has the potential 'to lead to
litigation following refusal to allow entry of a low risk commodity when
a slightly different analysis might have shown it to be a very low risk
commodity'.[24]
3.29
The Peace report also raised a further problem of interpretation faced
by DA Biosecurity risk assessors:
...0.3 is the top of the low range and bottom of the moderate
range. If a risk analyst determined the probability of an event was 0.3
should they name it low or moderate?[25]
3.30
Mr Peace further argued that:
Matrices are too often poorly designed and incorrectly
interpreted. If they are to be used, they must be simple, based on relevant
data, used following a clear understanding of the nature of a risk, and with
their limitations understood by risk assessors and decision-makers.[26]
Consequence scales – geographical
impacts
3.31
The Peace report observed that the methodology in the REM describes the
assessment of consequences. It is also noted that four levels of consequence
are considered for levels of Australian community defined as:
-
Local: an aggregate of households or enterprises (a rural
community, a town or a local government area).
-
District: a geographically or geopolitically associated
collection of aggregates (generally a recognised section of a state or
territory, such as 'Far North Queensland').
-
Regional: a geographically or geopolitically associated
collection of districts in a geographic area (generally a state or territory,
although there may be exceptions with larger states such as Western Australia).
-
National: Australia wide (Australian mainland states and
territories and Tasmania).[27]
3.32
Mr Peace suggested that whilst the four levels of consequence are
reasonable, they may apply to any size of community:
For example, a small community might be a major contributor
to the regional or national economy. As shown, such a contribution may be
understated. The reverse might be true with a pest having trivial national
impacts felt catastrophically at a local level.[28]
3.33
Mr Peace suggested that this problem could be overcome by 'developing
consequence scales based on, for example, national GDP, percentage of national
crop at risk, or viable planting area at risk'.[29]
Does the REM overstate or
understate the level of risk?
3.34
The Peace report argued that it is possible that the rules for combining
qualitative likelihoods, either overstate or understate the level of risk in
some cases. It also suggested that 'the rules are opaque with no source cited
and therefore leave in doubt their reliability'.[30]
Mr Peace also observed that:
Two of the reports provide indicative probability ranges.
These would be most helpful if their sources were cited; we are again left with
doubt about the provenance and reliability of the indicative probabilities.
Furthermore, our calculations [the author refers to Tables 3 and 4 of his
report] suggest that some indicative probability range combinations may give
results that breach the DAFF rules for combining qualitative likelihoods.
Overall, combining the likelihoods and/or their indicative
probabilities may either overstate or understate the level of import risk.[31]
Risk perception
3.35
Mr Peace argued that if the design of the REM is to be improved, risk
analysts need to know and understand the perception of risk, both in DA
Biosecurity and external stakeholders, including the committee.
3.36
Mr Peace noted that risk perception is defined in the ISO Risk
Management Vocabulary as 'the stakeholder's view on a risk' and 'reflects
the stakeholder's needs, issues, knowledge, belief and values'.[32]
3.37
The author argued that:
Risk perceptions of external stakeholders may be intuitive
feelings, based on media reports (Slovic, 2000). Some stakeholders may believe
that levels of risk are increasing whereas the reverse may be the case. DAFF
risk analysts need to understand the risk perceptions of external stakeholders
as distinct from their professional perception of risk.
In Australia, public perceptions of biosecurity risks may be
shaded by, for example, environmental damage caused by the release of wild
rabbits in the 1800s and the harm caused by cane toads. Or there may be a
proposal to import from overseas an exotic species or a species already in
Australia that can carry some disease or pest (for example, the recent change
to allow imports of European rabbits that might carry epizootic rabbit
enteropathy).[33]
3.38
Mr Peace stressed the importance of risk perception and suggested that
such 'risk perceptions should be incorporated into risk criteria used to
analyse the consequences of a given import risk'.[34]
Conclusions
3.39
The Peace report concluded that the existing Australian biosecurity REM
does not meet best practice because:
-
it combines likelihoods with events and consequences;
-
it is opaque in describing how to combine likelihoods;
-
probability and likelihood seem to be confused even though they
are distinct concepts;
-
sources for the indicative probabilities used in recent reports
are not given; and
-
the labels on the consequence and likelihood scales and risk
level cells are very similar.[35]
Recommendations
3.40
Based on its analysis and conclusions, the Peace report made a number of
recommendations, including that:
-
The DA Biosecurity REM be redesigned as a simple
consequence/likelihood matrix to overcome the deficiencies identified in the
report.
-
The Senate Rural and Regional Affairs and Transport Committee
encourage DA to develop the use of fault tree, event tree and bow-tie analyses
and other techniques to help understand and show the nature of import risks.
This should be done in combination with a redesigned consequence/likelihood
matrix to help determine the level of risk.
-
In order to aid transparency in import risk analysis and
decision-making, DA revise the IRA Handbook to include full details of
techniques available to DA risk analysts and any underlying data or research
validating those techniques.
-
A revised IRA Handbook include a copy of the Peace report's draft
Import risk analysis effectiveness checklist. (The checklist was
developed to be an assurance tool demonstrating each import risk analysis meets
the World Trade Organization criterion of an 'objective and defensible' import
risk analysis. This might be combined with the DA IRA template that now seems
to be in use).[36]
Eminent Scientists Group's comments on the Peace report
3.41
On 8 February 2013, the Rural and Regional Affairs and Transport
Legislation Committee (the Legislation committee) conducted a hearing in
relation to its inquiry into the Biosecurity Bill 2012. During the hearing, Dr
John Radcliffe, Chair of the Eminent Scientists Group (ESG) was invited to
review Mr Peace's report.
3.42
In reviewing the Peace report, the ESG noted that whilst Mr Peace 'appears
to recognise the quality of scientific rigour provided by DAFF Biosecurity in undertaking
Import Risk Analyses' the report also discusses changes that could be made to
current analytical practices.[37]
Language
3.43
The ESG noted Mr Peace's comments in relation to the variations in
meanings and definitions between treaties, agreements and standards and
acknowledged that much of the debate revolving around the use of IRAs is of an
'etymological' nature. The ESG suggest that:
The Committee may wish to explore whether more consistent and
better understood terms could be identified for use in Import Risk Analyses
(IRAs) by DAFF Biosecurity to minimise what Peace refers to as 'idiosyncratic
interpretation'...[38]
3.44
The ESG also argued, however, that these 'issues are largely a matter of
risk communication and may not materially alter the scientific outcome of the
analysis.'[39]
Risk
3.45
The ESG noted Mr Peace's comments regarding risk being the likelihood of
the consequences of an event, but argued that this statement may not be
'tacitly correct in that it discounts the likelihood of an event occurring in
the first place'.[40]
The ESG also stated that:
We are not convinced that DAFF's definition "risk being
the likelihood of an event occurring" is wrong. An event may well occur
("a food product passes undetected through the barrier"), but it may
or may not prove to have quarantine consequence ("the importer ate most of
it and destroyed the remainder").[41]
Quantitative risk estimates and
scientific uncertainty
3.46
The ESG supported Mr Peace's view in relation to quantitative risk
assessment, in that the numerical element is only one part of what is
essentially a judgement exercise. It further argued that in the context of
analysing biosecurity risks from a proposed import:
...it should be recalled that the analysis has to resolve
matters of scientific uncertainty in terms of the potential biological impact
of a new species on agricultural practices or the natural environment, if any.[42]
3.47
The ESG further argued that:
When identifying the risks (hazards) that could eventuate
from the introduction of new biological products at the border, the 'level of
risk', the probability of occurrence, and the consequences will rarely have any
prior measured estimations available in the Australian environment being
addressed for the purposes of establishing an Appropriate Level of Protection
for Australia (ALOP) that is defendable internationally.[43]
3.48
The ESG acknowledged the importance of those doing the estimations being
free of any conflict of interest (in terms of benefitting from the judgements
to be made). At the same time, the ESG argued that the existing process is
designed for that purpose.[44]
Use of matrices and alternative
instruments
3.49
In reviewing Mr Peace's theoretical discussion about qualitative risk
matrices, the ESG acknowledged its members did not feel qualified to comment
specifically on the mathematical issues involved. The ESG did however suggest
that the Australian Centre of Excellence for Risk Analysis (ACERA) has been
established specifically for this purpose and:
...receives funding from DAFF to research methodology for
biosecurity analysis and could be asked more explicitly by DAFF to provide
advice, including any effect of qualitative versus quantitative risk analysis
on the consequences and methodology of sampling and on the forms of and use of
matrices.[45]
3.50
The ESG suggested that the committee may also wish to seek advice from
the ACERA on these issues, and noted that it had previously indicated its
support for having a suitable independent party (such as ACERA) review the
range of models used in the IRA process by Australia's major trading parties.[46]
ACERA's views on the Peace report are outlined below.
Risk analysis checklist and
revision of IRA Handbook
3.51
The ESG noted Mr Peace's suggestion in relation to a risk analysis
checklist and indicated that it 'would support this as a constructive
suggestion'. The ESG also acknowledged that the IRA Handbook will need to be
revised as a result of the current review of the biosecurity legislation, and
indicated its support for Mr Peace's suggestion that the IRA Handbook include
full details of techniques available to DA Biosecurity risk analysts and any
underlying data or research validating those techniques.[47]
DA's response to the Peace report
3.52
On 7 February 2013 the committee requested that DA Biosecurity review
and provide an initial response to the issues raised by the Peace report. DA Biosecurity's
response, dated 8 March 2013, is provided at Appendix 10.
3.53
The committee also received a detailed response from the (then) Department
of Agriculture (DA) Secretary, Mr Andrew Metcalfe AO, on 22 May 2013 which
is provided at Appendix 11.
SPS Agreement and International
Standards for Phytosanitary Measures (ISPMs)
3.54
In responding to the Peace report, DA Biosecurity noted that it has been
the policy of successive Australian governments that risk assessments used to
establish phytosanitary measures be consistent with the SPS Agreement. DA also
noted that this position is consistent with Australia's obligations as a
signatory to the WTO.[48]
3.55
DA also indicated that in conducting import risk analyses for plant
products, 'specific guidance is provided by the internationally agreed
International Standards for Phytosanitary Measures (ISPMs) which are developed
under the International Plant Protection Convention (IPPC)'.[49]
It was argued that:
Other methods may be informative, but it is the SPS Agreement
and the ISPMs that establish the basis for import risk analysis. In the case of
animals and animal products, guidance similar to the IPPC is provided by the
Office International des Epizooties (OIE).[50]
3.56
DA noted that although the Peace report referred to the IPPC and some of
its training materials, it did not reference ISPM 2: Framework for pest risk
analysis or ISPM 11: Pest risk analysis for quarantine pests, including
analysis of environmental risk and living modified organisms, 'which
provide substantial guidance on pest risk analysis for quarantine pests of
plants'.[51]
3.57
DA also noted Mr Peace's observation that chapter two of the Terrestrial
Animal Health Code, published by the OIE, does not define hazard, risk, risk
analysis and risk assessment – and submitted that 'each of these terms is
clearly defined in ISPM 5: Glossary of phytosanitary terms'.[52]
Development of REM
3.58
DA Biosecurity's response stated that the matrix-based approach to
combining the likelihood of entry, establishment and spread of a pest or
disease, with the consequences if that were to occur, first appeared in the
draft IRA report for non‑domestic Felidae in February 2001.[53]
It was noted that current form and labelling of the matrix subsequently
appeared in the August 2001 Issues Paper for the generic import risk analysis
for fresh pineapple fruit.
3.59
DA argued that the current methodology for assessing biosecurity risks:
...was the subject of substantial discussion between the
Commonwealth Government and the states and territories. These discussions led
to the formal endorsement of the current methodology through the Primary
Industries Ministerial Council in 2002.[54]
Consequence and likelihood
3.60
In responding to Mr Peace's suggestion that the REM is not a true consequence/likelihood
approach, DA argued that its risk analysis process aligns with the relevant
international standard for plant pest risk analysis, 'in this case ISPM 11, by
considering the probability of introduction and spread and the potential
economic consequences'.[55]
DA further argued that:
These are then combined using a matrix based approach as
demonstrated in the IPPC training materials, and as endorsed in the annex to
ISO:31010 as being 'strongly applicable' for assessing level of risk. DAFF's
matrix combines the probability (or likelihood) of entry, establishment and
spread with the consequences if that sequence of events were to occur. This
results in an assessment of unrestricted risk. If the unrestricted risk is
'low' or greater, specific risk management measures are necessary.[56]
Description of REM in IRA Handbook
3.61
DA Biosecurity noted Mr Peace's comments regarding the REM not being
included in the IRA Handbook and the recommendation that the IRA Handbook be
revised to include details of techniques available to DA risk analysts and a
description of the REM. In responding to this recommendation, DA Biosecurity
argued that Mr Peace 'does not have a correct understanding of the purpose of
the IRA Handbook':
...The IRA Handbook describes the administrative process for
conducting import risk analyses, regulated steps under the Quarantine
Regulations 2000, and relevant background information on domestic and
international policies.[57]
3.62
DA Biosecurity further noted that:
To inform stakeholders and readers of import risk analysis
reports, each report contains a detailed description of the methodology being
employed and worked examples for combining likelihoods.[58]
3.63
However the Secretary of the Department of Agriculture later told the
committee that the IRA Handbook is being revised:
I am preparing to withdraw the Handbook and make more
up-to-date and comprehensive information available about the department’s role
in managing imports into Australia, including the IRA process.[59]
Communication of risk
3.64
At the 12 March 2013 hearing, the committee questioned departmental
officers about DA's response to the Peace report. In particular, the committee asked
whether the report had raised any issues the department may be prepared to
'take on board'.[60]
3.65
In response, DA officers indicated that whilst they were already
undertaking a review of the import risk analysis processes, there were a number
of things in Mr Peace's report the department could look at and possibly
develop further, particularly in relation to communicating risk:
I think the greatest value that Mr Peace's report provides
is, again, identifying where we know one of the things we can work on more is
risk communication as opposed to the risk method that we undertake. Certainly
the suggestions that Mr Peace provides gives us some further food for thought
about how we might better involve and engage stakeholders and make our risk
assessments more open for stakeholders to be able to analyse and look at
themselves.[61]
3.66
DA also indicated that its work with ACERA had resulted in the
evaluation of, and improvement in, policies and operational programs. DA also
noted that it is currently developing and seeking stakeholder feedback on new
biosecurity regulations and guidelines, including the consideration of issues
such as:
-
addressing regional differences in biosecurity status;
-
independent scientific review of import risk analyses;
-
the rights of appeal on the outcome of an import risk analysis;
and
-
the application of Australia's appropriate level of protection.[62]
3.67
In concluding its comments DA argued that 'there is no substantive
evidence that DAFF's risk analysis processes have not been effective in
protecting Australia's favourable pest and disease status'.[63]
ACERA's comments on the Peace report
3.68
In correspondence to the committee dated 24 May 2013, the Secretary of
DA indicated that he had asked the Australian Centre of Excellence for Risk
Analysis (ACERA) to review Mr Peace's report.
3.69
In providing advice to DA on the Peace report, ACERA acknowledged that
whilst some of the issues raised by Mr Peace were valid, the organisation felt
that others were problematic. ACERA noted, for example, that the comment made
by Mr Peace at a public hearing, that for 'Australia's risk analyses, using the
qualitative matrices, if the overall assessment is 'negligible', there may be
as much as a 10-15% likelihood of the risk being higher than 'negligible''[64]
was:
...made, based on a qualitative interpretation of a qualitative
risk analysis. While there is almost certainly at least some small chance that
the risk is higher than negligible, there is no justification, or any
conceivable rational basis on which one could quantify this chance.[65]
3.70
However, ACERA did acknowledge Mr Peace's criticism of qualitative
assessments that 'do not provide clear guidelines regarding the meaning of
indicative probability distributions'[66]:
In particular, in his testimony to the committee, he
highlighted the difficulties of aggregating qualitative risk assessments
without these. We agree that this raises a problem with the transparency of the
overall assessment. We note that in other IRAs these indicative qualitative
intervals are provided.[67]
3.71
ACERA's response noted Mr Peace's advocacy for the use of a bow-tie
analysis in combination with a revised qualitative consequence/likelihood
matrix and quantified fault tree and event tree analyses. ACERA agreed with Mr
Peace's assertion that this type of analysis provides a way of visualising the
causal process and can be an aid to understanding. It was also agreed that this
kind of analysis can be qualitative or quantitative. However, ACERA argued that
this type of approach does have a number of weaknesses:
-
Because this method uses fault 'trees' rather than 'graphs', it
cannot represent all kinds of conditional dependencies. (This means that the
simple arithmetic operations on probabilities may make incorrect independence
assumptions) which may result in incorrect overall probabilities.
-
An event tree typically 'fans out' to numerous possibilities,
meaning that they are inherently limited for modelling the impact of many
factors or intervention actions.
-
The bow-tie analysis is based around a single event (the
introduction of a single pest) and does not appear to provide any obvious way
to aggregate. This means that scalability seems to be problematic for the
bow-tie approach.[68]
3.72
ACERA's response concluded by noting that:
To our knowledge, there are no case studies demonstrating how
bow-tie analysis (combining fault-tree and event-tree analysis) can be used for
IRAs (particularly in combination with Australia's qualitative matrices as
proposed by Peace). It is not advocated for use in IRAs in the peer-reviewed
research literature.[69]
Meeting between Mr Peace and Department of Agriculture officials
3.73
On 3 July 2013, officers from the Department of Agriculture met with Mr
Peace to discuss his review of the department's import risk assessment methods.
3.74
In addition to officers from the department, Professor Mark Burgman from
the Centre for Excellence for Biosecurity Risk Analysis (CEBRA)[70]
was also present at the meeting.[71]
Meeting participants agreed that the points of difference related to the REM,
and Mr Peace emphasised his view that 'the matrix was "opaque" for
stakeholders outside DAFF who would find difficulties understanding how
probabilities and consequences were combined'. The group also discussed the
need for some alternative risk analysis technique that was transparent and the
documentation and publication of the import risk analysis process.[72]
3.75
At the conclusion of the meeting, the group had reached agreement that:
-
the department would consider alternative options to better
represent the import risk analysis process and outcomes; and
-
the import risk analysis process and techniques used will be
documented and published on the DAFF website so stakeholders can download, read
and better understand the process.[73]
Committee comment
3.76
As indicated previously in this report, the committee was aware from the
early stages of its inquiry into the Malaysian pineapple IRA, that DA
Biosecurity's method of calculating risk and the REM itself would be issues of
central importance to the committee's inquiry.
3.77
The committee was also aware that, if it were to give appropriate
consideration to the issues of concern to stakeholders, it would be necessary
to have a clearer understanding of the REM and the way in which DA Biosecurity
use it when preparing IRAs for various commodities.
3.78
The committee believes that the review undertaken by Mr Chris Peace
has been both informed and independent. The Peace report has provided the
committee with valuable information, in relation to processes for assessing or
analysing risks, risk matrix literature, alternative risk techniques and the
'language' of risk.
3.79
The Peace report has also provided the committee with a more thorough
understanding of DA Biosecurity's REM, the way in which DA Biosecurity use the
REM to calculate risk, and the ways in which both of these could be improved.
3.80
The committee acknowledges the comments provided by the ESG, DA Biosecurity
and ACREA (recently renamed CEBRA) in response to the Peace report. The
committee notes that successive Australian governments have maintained a policy
of risk assessments used to establish phytosanitary measures being consistent
with the SPS Agreement. The committee also notes that this position is
consistent with Australia's obligations as a signatory to the WTO.
3.81
The committee is aware that the current form and labelling of the
matrix, and the current methodology for assessing biosecurity risks, has been
developed over a considerable period of time. The committee acknowledges that
these issues have been the subject of considerable high level discussion, and
subsequent agreement between federal and state governments.
3.82
The committee notes that DA Biosecurity has not refuted many of the
issues raised by the Peace report, including:
-
the unreliability of qualitative descriptors;
-
the combination in the DA Biosecurity REM of the terms likelihoods
with events and consequences;
-
the opacity of the DA Biosecurity REM in describing the method
for combining likelihoods;
-
the lack of sources for the indicative probabilities used in
recent reports;
-
the apparent confusion of the terms probability and likelihood;
-
labels on the consequence and likelihood scales and
risk level cells are very similar – which leads to confusion;
-
limitations in the consequence scales' use of geographical
impacts;
-
difficulties associated with combining qualitative likelihoods;
and
-
the need for risk perceptions to be incorporated into risk
criteria.
Recommendation 3
3.83
The committee recommends that the Department of Agriculture give
thorough consideration to the Peace report, as well as the underlying themes of
all other recommendations contained in this report, in developing the new
biosecurity regulations and guidelines.
3.84
In addition, the committee notes the Peace recommendation that a
revision of the IRA Handbook should include full details of techniques
available to DA risk analysts and any underlying data or research validating
those techniques. The committee notes DA Biosecurity's response that some of
this information is provided in individual IRA reports. The committee believes,
however, that the REM used by DA Biosecurity to calculate risk has been
identified as central to the IRA process, and therefore supports Mr Peace's
recommendation to include these additional details in the IRA Handbook.
Recommendation 4
3.85
The committee recommends that the IRA Handbook should be amended to
include full details of techniques available to Department of Agriculture risk
analysts and any underlying data or research validating those techniques.
3.86
The committee is also of the view that the IRA Handbook should include
an IRA effectiveness checklist similar to that recommended by Mr Peace (and
included at Appendix 12) – a proposal the ESG indicated it would be support. In
this regard, the committee notes that the DA Secretary has indicated that the
department is preparing to withdraw the Handbook and make more up-to-date and
comprehensive information available about the department’s role in managing
imports into Australia, including the IRA process.
Recommendation 5
3.87
The committee recommends that the IRA Handbook should include an IRA
effectiveness checklist similar to that recommended by Mr Peace.
3.88
The committee notes Mr Peace's comments in relation to 'perception of
risk' and the suggestion that those undertaking risk analysis should know and
understand how risk is perceived – particularly by external stakeholders. The
committee agrees that if the IRA process or the design of the REM are to be
improved, stakeholders' needs, issues, knowledge, beliefs and values – their
risk perceptions – need to be taken into consideration. This is a proposal for
which DA has also indicated support.
3.89
As indicated earlier in this report, the committee has over a number of
years undertaken a number of inquiries in relation to the importation (or
proposed importation) of specific plant or animal products. The committee's
reports on these issues have always stressed the importance of stakeholders
having their views taken into consideration and being able to fully participate
in the import risk analysis process.
3.90
The committee supports Mr Peace's recommendation that stakeholder's risk
perceptions should be incorporated into risk criteria used to analyse the
consequences of a given import risk.
Recommendation 6
3.91
The committee recommends that stakeholders' risk perceptions should be
incorporated into risk criteria used to analyse the consequences of a given
import risk.
Recommendation 7
3.92
The committee recommends that the Department of Agriculture consider
ways to improve the way it communicates risk (and the risk assessment process)
to stakeholders.
3.93
The committee notes views expressed by Mr Peace in relation to the issue
of geographic impacts and the assessment of consequences. The committee agrees
with Mr Peace's view that the definitions used for the four levels of
consequence – local, district, regional and national – are reasonable. However,
the committee also agrees that they could nonetheless easily apply to any size
of community.
3.94
The committee notes, for example, that it is often the case that a small
community contributes significantly more to the regional or national economy
than a large community. It is also true that the impact of specific pests can
have a minor impact nationally, but prove devastating at a local level.
Recommendation 8
3.95
The committee recommends that the Department of Agriculture reconsiders
the operation of geographic impacts in the IRA process, and give consideration
to developing consequence scales based on, for example, national GDP,
percentage of national crop at risk, or viable planting area at risk.
Navigation: Previous Page | Contents | Next Page