Offical Economic Forecasts-How Good are They?
David Richardson
Economics, Commerce and Industrial Relations Group
26 June 2001
Contents
Major
Issues
Introduction
Economic Growth: Actual versus Forecast
Are the Forecasts Getting Better?
Unemployment and Inflation: Actual versus
Forecast
Endnotes
List of Tables
Table 1: Economic Growth: Forecasts versus
actual
Table 2: Economic Growth: Forecasting errors by sub
period
Table 3: Unemployment and Inflation: Forecasts
and Outcomes
Table 4: Unemployment and Inflation: Forecasting
errors by sub period
Major
Issues
Recently there has been a good deal of interest
in economic forecasting in Australia, chiefly as a result of the
failure of official forecasts to pick the contraction in the
December 2000 quarter. The purpose of the present paper is to
examine just how well the forecasts have performed over time.
A series of forecasts going back to the 1978
Budget are examined and an assessment is made based on some summary
measures, chiefly, the average forecasting error and the mean
absolute error. Overall the paper finds a small bias towards
underestimating economic growth, overestimating unemployment but
impressive accuracy with inflation.
There remains the important question of whether
the forecasts have been improving over time, as one might expect
with improved computing facilities, better information and so on.
To test this the sample was split into two sub samples and the
forecast errors examined separately for each. For economic growth
the answer seems to be that the average forecast error has
increased but the mean absolute error has fallen. For a dart player
that would be like saying that the average has moved somewhat to
the right but the cluster, or the dispersion of the darts, is
getting smaller. For economic growth the 'cluster' is almost a
whole percentage point away from the bullseye if the effect of
missing the contractions is ignored.
The paper finds that the forecasts are just
slightly better than naïve forecasts. Two naïve forecasts
were examined. The first is a forecast which just says that next
year will be the same as the present. The second just says every
year will experience economic growth at 3.5 per cent. The latter
performs almost as well as the official forecasts, it has a lower
average forecast error but a slightly higher mean absolute error.
However, the difference in performance between the official
forecasts and the 3.5 per cent rule is very small.
For unemployment the pattern of errors tends to
follow those for growth but in the opposite direction. That is to
be expected since growth would be one of the major determinants of
employment in any economic modelling. The impressive lack of bias
in inflation forecasting for the whole sample disappears when the
sample is broken in two. The paper finds a forecast error that goes
from an early underestimate to a later overestimate. The mean
absolute error increases by the second sub sample. That is like the
dart player starting off early in the evening with a bias to the
left of the dart board and a small cluster, and ending the evening
being biased to the right with a larger cluster.
Introduction
Australia was surprised when the December 2000
national accounts were released showing that the economy had shrunk
by 0.6 per cent (seasonally adjusted) in that quarter. Few
observers expected that result, although a modest slowdown was
expected as a result of the post-GST contraction in building and
construction. In the middle of that quarter, on 15 November 2000,
Treasury released the Mid-Year Economic and Fiscal Outlook
which expressed optimism and actually increased the forecast rate
of economic growth for 2000-01. The Secretary to the Treasury, Dr
Ken Henry, was later to say:
There has been much reporting of, and some
adverse commentary on, the publication in last November's
Mid-Year Economic and Fiscal Outlook (MYEFO) of a 4 per
cent GDP growth forecast for 2000-01-an upward revision to the 33/4
per cent forecast that was published in last year's
budget.(1)
The Governor of the Reserve Bank of Australia,
Mr Ian Macfarlane, in evidence to the House of Representatives
Committee on Economics, Finance and Public Administration made the
following point about economic forecasting:
Economic forecasting is a very imperfect art-I
would not use the word 'science.' It, by and large, has not
improved in 30 years. I have been through all the attempts to
improve it-all the large econometric models, the small econometric
models, the leading indicators, all the surveys of expectations-and
basically it is about the same as it always was.
The purpose of this paper is to examine the
accuracy of the official economic forecasts. Official forecasts in
Australia are the responsibility of the Joint Economic Forecasting
Group (JEFG). The JEFG is composed of the Department of Treasury as
the chair, the Department of the Prime Minister and Cabinet, the
Department of Finance and Administration (DOFA), the Australian
Bureau of Statistics and the Reserve Bank of Australia. The JEFG
makes forecasts twice a year, at budget time and for the MYEFO. The
Charter of Budget Honesty(2) requires the public release
of a pre-election economic and fiscal outlook report within ten
days of the issue of the writ for a general election. Those
forecasts are to provide the economic parameters to be used by
Treasury and DOFA to cost election commitments.
Economic Growth: Actual versus
Forecast
Before examining the record some quick
definitions are in order. The main concepts used here are
-
- For a series of forecasts, the average forecasting
error (AFE) is the actual value less the forecast value. So if
the AFE is plus something then the forecasts can be said to be
biased towards overestimating the outcome. If the AFE is
negative then the forecasts are biased towards underestimating the
outcome. If there is no bias at all the AFE will be zero. However,
the forecasts can be unbiased but still very inaccurate,
-
- the mean absolute error (MAE) averages all the errors
but ignores negative signs. For example, a plus one is treated the
same as a minus one on the grounds that both errors are out by
one.
The following Table presents Budget growth
estimates and forecasts going back to the 1978 Budget.
Unfortunately we do not yet have final figures for 2000-01.
However, from 1994-95 there are essentially two forecasts. For the
1994-95 Budget, the delivery by the Treasurer was brought forward
from August to May. That meant that, for example, in the 1993
Budget the Treasurer already knew the growth figure for 1992-93.
However, the following year the Treasurer would have known the GDP
figure for only the first two quarters of 1993-94. Hence, beginning
with the 1994-95, Budget, growth estimates were also given for the
current year, then being 1993-94. That means we can also examine
the accuracy of estimates for the current year. The Budget Papers
have consistently referred to estimates as applying to the present
year and forecasts for beyond that. (These are both distinct from
the projections that are also contained in the Budget
Papers. Projections apply beyond the forecast period and are based
on the assumption that growth, for example, will behave as it has
done on average over the recent past.)
Table 1: Economic Growth: Forecasts versus
actual
Budget Estimate for Current
Year
|
|
|
Actual
|
Budget Forecast for next year
|
Actual
|
Errors
|
|
|
|
|
|
|
Current
|
Next Year
|
1978 budget
|
|
|
4
|
5.6
|
|
1.6
|
1979 budget
|
|
|
2.25
|
2.5
|
|
0.25
|
1980 budget
|
|
|
3
|
3.3
|
|
0.3
|
1981 budget
|
|
|
3.5
|
3.1
|
|
-0.4
|
1982 budget
|
|
|
3
|
-2.5
|
|
-5.5
|
1983 budget
|
|
|
3
|
5.6
|
|
2.6
|
1984 budget
|
|
|
4.5
|
5.1
|
|
0.6
|
1985 budget
|
|
|
4.5
|
4.3
|
|
-0.2
|
1986 budget
|
|
|
2.25
|
2.6
|
|
0.35
|
1987 budget
|
|
|
2.75
|
5.3
|
|
2.55
|
1988 budget
|
|
|
3.5
|
4.2
|
|
0.7
|
1989 budget
|
|
|
2.75
|
3.6
|
|
0.85
|
1990 budget
|
|
|
2
|
-0.2
|
|
-2.2
|
1991 budget
|
|
|
1.5
|
0.4
|
|
-1.1
|
1992 budget
|
|
|
3
|
3.6
|
|
0.6
|
1993 budget
|
|
|
2.75
|
4.1
|
|
1.35
|
1994 budget
|
4
|
4.1
|
4.5
|
4.5
|
0.1
|
0
|
1995 budget
|
4.75
|
4.5
|
3.75
|
4.4
|
-0.25
|
0.65
|
1996 budget
|
4.1
|
4.4
|
3.5
|
3.6
|
0.3
|
0.1
|
1997 budget
|
3.25
|
3.6
|
3.75
|
4.8
|
0.35
|
1.05
|
1998 budget
|
3.75
|
4.8
|
3
|
5.4
|
1.05
|
2.4
|
1999 budget
|
4.25
|
5.4
|
3
|
4.4
|
1.15
|
1.4
|
2000 budget
|
4.25
|
4.4
|
3.75
|
na
|
0.15
|
|
2001 budget
|
2
|
na
|
3.5
|
na
|
|
|
Average forecasting errors
|
|
|
|
|
0.41
|
0.36
|
Mean absolute errors
|
|
|
|
|
0.48
|
1.22
|
Source: Budget Paper No 1, various years, Australian
Bureau of Statistics, National Income, Expenditure and Product,
March quarter 2001, cat no 5206.0, 6 June 2001; Reserve Bank
of Australia web site http://www.rba.gov.au/Statistics/Bulletin/G09hist.xls.
The figures in Table 1 have been used to
construct the following graph which shows how the official
forecasts compare with the actual outcomes as well as the forecast
error for each year.


The final column in the above Table indicates
that the average forecast was 0.36 of a percentage point below the
eventual outcome. So, on average the official forecasts understated
GDP growth by 0.36 per cent. However, the mean absolute error was
1.22 per cent. That tells us that on average the outcome is the
forecast plus or minus 1.22 per cent. The biggest forecast error
was 5.5 percentage points in 1982. In that year the forecast was
for 3 per cent growth but the outcome was a 2.5 per cent
contraction. In fact, neither of the two years of negative growth
(1982 and 1990) were forecast. In more recent years, following the
Asian financial crisis, economic growth was consistently
underestimated. Whenever economic growth was above average at 5 per
cent or more the official forecasts seriously underestimated
growth. In our sample, economic growth has been 5 per cent or more
on five occasions but there has not been one case where the
forecast has been 5 per cent or more.
It is useful to compare the official forecasts
with a know-nothing rule or a naïve forecast which just makes
the hypothesis that next year's growth will be the same as this
year's growth. The average forecasting error under that rule is
minus 0.057. So it would just under-predict growth by 0.06 per cent
compared with the official over-prediction of 0.36 per cent. The
mean absolute error is 1.65 per cent which is a bit worse than 1.22
per cent, being the average forecast error in the official
estimates.
Another rule could be to assume that economic
growth will be 3.5 per cent every year. That is the figure the
Budget Papers use to make their longer term projections beyond the
forecast period. Using that rule gives an average forecast error of
0.08 per cent and a mean absolute error of 1.42 per cent, not
significantly different from the mean absolute error in the
official estimates.
The figures in the second to last column of
Table 1 show errors made in estimating the current year's growth.
Those estimates are made in May when there is only around 5 or 6
weeks of the financial year left. Nevertheless, substantial errors
were made, especially for 1998-99 and 1999-2000. As it happens the
estimates for the current year are downward biased by more than the
downward bias for the next year forecasts. On average the forecasts
were better for next year than the current year on these
figures.
There is an important practical consequence of
any downward bias in the forecasts. If the official growth
estimates were biased downward by 0.41 per cent for the current
year and 0.36 per cent for next year, then next year's GDP would be
underestimated by 0.77 per cent. That could well translate into
changes in the budget balance of well over $1 billion. Assuming a
0.77 per cent increase in GDP translates into a 0.77 per cent
increase in the wages bill and a reduction in unemployment by 0.77
percentage points from a forecast 7 per cent, then, on the basis of
the rules of thumb given in the Budget Papers, the budget balance
would improve by approximately $1.7 billion.(3) Those
figures illustrate the sensitivity of Budget figuring to the
economic forecasts.
Unlike the weather forecast, the official
economic forecasts fail to anticipate extreme values as earlier
discussed. However, to some extent that may well be deliberate with
economic forecasts. While governments may try to present honest
forecasts they will also want to avoid publishing estimates that
may be self-fulfilling. We have seen how the negative GDP growth
figure for December adversely affected consumer and business
confidence when it was published in March 2001. Yet by that time
the objective conditions in the economy had vastly improved as was
shown by the 1.1 per cent growth, or an annualised 4.5 per cent
growth in the March 2001 quarter. News about our history gave rise
to a pessimism that was later seen to be unwarranted. Given the
effect news has on economic confidence and behaviour, it is
possible that governments in the past have chosen not to publish
forecasts of negative growth and instead chosen to publish
estimates of slow positive growth instead.
Before leaving this section there is a technical
point that needs to be made. Forecasts should include all
information currently available. But a perfect forecast on the
basis of present information cannot incorporate new developments
and shocks that occur during the forecast period. That means that
the actual outcomes will display more variability than 'perfect'
forecasts. This insight has been used in an assessment of OECD
forecasts to examine their efficiency.(4) A series of
forecasts will be efficient if the variation in the forecasts is
below the variation of the outcomes. This is confirmed for
Australia's official forecasts with the standard deviation of the
actual outcomes being 1.41 while the standard deviation of the
forecasts is lower at 1.09.
Are the
Forecasts Getting Better?
This section examines whether or not the
forecasts are getting better over time. Mr Ian Macfarlane's
comments above to the effect that forecasting has not improved over
the last 30 years can be examined by splitting the above Table into
two time periods. Those examined are the 1978 to 1988 budgets and
the 1989 budget to the latest for which there is data. When that is
done we obtain the results given in Table 2. Because of the
distorting influence of the two downturns, the sub periods are
estimated with and without the years 1982 and 1990.
Table 2: Economic
Growth: Forecasting errors by sub period.
|
1978 to 1988 budgets
|
1978 to 1988 without 1982
contraction
|
1989 to 1999 budgets
|
1989 to 1999 without 1990
contraction
|
Average forecast error
|
0.26
|
0.84
|
0.46
|
0.95
|
Mean absolute error
|
1.37
|
0.96
|
1.06
|
0.73
|
The raw results in Table 2 suggest that
theaverage forecast error has worsened from the first half to the
second half of the sample. This result is not statistically
significant.(5) The results suggest that if we ignore
the failure to predict the contractions then the average error
jumps to almost one percentage point by the second subset of the
sample. This bias is in fact worse in the second half of the
sample, suggesting that the official forecasts are more biased in
more recent years. The mean absolute error appears to improve from
the first to the second half and falls substantially if the failure
to forecast the downturns are ignored. Once again the difference
between the samples is not statistically significant.
Unemployment
and Inflation: Actual versus Forecast
Other important variables that the Budget Papers
attempt to forecast include unemployment and inflation. Those
forecasts are intrinsically important but are also important for
deriving government expenses (previously 'outlays') and
revenues.
Table 3 presents budget-time forecasts for
unemployment and inflation since the 1978-79 Budget. Some of the
early years presented problems because estimates were not always
given as a precise numerical forecast. In constructing Table 3 some
of the actual numerical targets had to be inferred from the
relevant Budget papers. For example, the discussion in the 1978-79
Budget Papers predicted unemployment would be 'unchanged' when the
latest figures then published showed unemployment to be 6.8 per
cent in the relevant quarter. In following years unemployment was
forecast to be 'broadly unchanged (1979),' 'little if any decline
(1980),' 'slight fall (1981),' 'marked increase (1982),' 'higher
(1983),' 'decline slightly (1984),' '7.5 to 8' at the end of the
year (1985), and 'some increase from 7 (1986).' In each case, where
there was no specific number mentioned, but the outcome was
consistent with the anticipated movement, the Table credits the
forecasts with getting the outcome exactly right. Since 1986 the
Budget Papers have provided tables with detailed numerical
forecasts rather than the more impressionistic discussions. The
situation was better with inflation. However, in 1980 the forecast
avoided a numerical target and said 'about the same or slightly
faster at 10. 'The 1984 Budget forecast a through-year CPI increase
because of the complications of the Medicare effect, whereby the
introduction of Medicare was expected to distort year on year
figures.
Table
3: Unemployment and Inflation: Forecasts and
Outcomes
|
Unemployment rate-year
average (%)
|
Inflation-year average
(% increase in CPI)
|
|
Budget forecast for next year
|
Actual
|
Error
|
Budget forecast for next year
|
Actual
|
Error
|
1978 budget
|
6.8
|
6.3
|
-0.5
|
6.0
|
8.1
|
2.1
|
1979 budget
|
6.6
|
6.2
|
-0.4
|
10.0
|
10.2
|
0.2
|
1980 budget
|
6.2
|
5.9
|
-0.3
|
10.0
|
9.3
|
-0.7
|
1981 budget
|
6.0
|
6.2
|
0.2
|
10.75
|
10.4
|
-0.35
|
1982 budget
|
9.0
|
9.0
|
0
|
10.75
|
11.5
|
0.75
|
1983 budget
|
9.6
|
9.6
|
0
|
7.5
|
6.9
|
-0.6
|
1984 budget
|
8.6
|
8.6
|
0
|
5.25
|
6.6
|
1.35
|
1985 budget
|
7.75
|
7.5
|
-0.25
|
8.0
|
8.4
|
0.4
|
1986 budget
|
8.0
|
8.1
|
0.1
|
8.0
|
9.3
|
1.3
|
1987 budget
|
8.25
|
7.5
|
-0.75
|
7.0
|
7.3
|
0.3
|
1988 budget
|
6.25
|
6.4
|
0.15
|
7.5
|
7.3
|
-0.2
|
1989 budget
|
7.25
|
5.9
|
-1.35
|
5.5
|
8.0
|
2.5
|
1990 budget
|
7.25
|
8.1
|
0.85
|
6.5
|
5.3
|
-1.2
|
1991 budget
|
10.5
|
10.0
|
-0.5
|
3.0
|
1.9
|
-1.1
|
1992 budget
|
10.5
|
10.7
|
0.2
|
3.0
|
1.0
|
-2.0
|
1993 budget
|
10.75
|
10.2
|
-0.55
|
3.5
|
1.8
|
-1.7
|
1994 budget
|
9.75
|
8.7
|
-1.05
|
2.25
|
3.2
|
0.95
|
1995 budget
|
8.25
|
8.1
|
-0.15
|
4.0
|
4.2
|
0.2
|
1996 budget
|
8.5
|
8.3
|
-0.2
|
2.0
|
1.3
|
-0.7
|
1997 budget
|
8.25
|
8.0
|
-0.25
|
1.0
|
0
|
-1.0
|
1998 Budget
|
8.0
|
7.4
|
-0.6
|
2.5
|
1.3
|
-1.2
|
1999 budget
|
7.5
|
6.6
|
-0.9
|
2.0
|
2.4
|
0.4
|
2000 budget
|
6.5
|
|
|
5.75
|
|
|
2001 budget
|
7.0
|
|
|
2.0
|
|
|
Average forecasting errors
|
|
|
-0.28
|
|
|
-0.01
|
Mean absolute error
|
|
|
0.42
|
|
|
|
Source: Budget Paper No 1, various
years, Reserve Bank of Australia web site
http://www.rba.gov.au/Statistics/Bulletin/G09hist.xls.
Before commenting on the inflation and
unemployment forecasts is should be pointed out that labour market
forecasts will depend heavily on the forecasts for economic growth.
Errors in the latter will mean errors in the former. Table 3 shows
the influence on unemployment estimates of the earlier discussed
errors in growth forecasts. The average error of 0.28 means that
the forecasts have overestimated unemployment, consistent with
underestimating growth. The mean absolute error seems reasonably
low, however, indicating fairly accurate forecasts for
unemployment.
Inflation forecasts are very interesting. The
negligible average forecast error is very impressive. This suggests
a record of unbiased estimates of inflation. The mean absolute
error is also reasonable given the rather volatile nature of
inflation over the last 20-odd years. However, it is also
interesting to split the sample in two to examine whether or not
forecasting has improved or otherwise changed. This is done in the
following table.
Table 4:
Unemployment and Inflation: Forecasting errors by sub
period.
|
1978 to 1988 budgets
|
1989 to 1999 budgets
|
unemployment
|
|
|
average forecast error
|
-0.16
|
-0.41
|
mean absolute error
|
0.24
|
0.60
|
inflation
|
|
|
average forecast error
|
0.41
|
-0.44
|
mean absolute error
|
0.75
|
1.18
|
It has already been noted that unemployment
errors tend to reflect growth errors. The average forecasting error
for unemployment increased somewhat between the two periods. That
lower early error may only reflect the manner in which the data was
constructed for the earlier period as described above. However, the
inflation figures will not depend so much on growth forecasts. The
impressive unbiased forecasts for the 22 year period are not so
impressive when the sample is split in two. Now Table 4 indicates
that in the first half there was a systematic bias towards
underestimating inflation while in the second half there was a
systematic bias towards overestimating inflation. The latter seems
to have reflected a period of overestimating inflation following
the 1990 downturn, as well as some overestimating inflation around
the time of the Asian financial crisis and the aftermath. The mean
average error has also increased somewhat between the two
sub-samples suggesting that the official forecasts are not as good
as they used to be. Once again, however, the differences in the
Table are not statistically significant.
Endnotes
-
- Dr K. Henry, 'On economists, the economy and fiscal policy,'
Address to the Australian business economists, Sydney, 29 May 2001.
- See Charter of Budget Honesty Act 1998.
- The 'rules of thumb' for estimating the effects of various
parameter changes are given in Table B1 of Budget Strategy and
Outlook 2001-02, Budget Paper no 1, 2001, p. 2-19.
- V. Koutsogeorgopoulou, ,
A Post-Mortem on Economic Outlook Projections (PDF format)
OECD Economics Department Working Papers No. 274, 2000.
- No formal statistical test was carried out because that was
unnecessary. None of the differences reported in the Table even
approached the value of the standard deviation for the sample as a
whole or the subsets of the sample used