Current Issues Brief 17 2000-01

Offical Economic Forecasts-How Good are They?

David Richardson
Economics, Commerce and Industrial Relations Group
26 June 2001

Contents

Major Issues

Introduction

Economic Growth: Actual versus Forecast

Are the Forecasts Getting Better?

Unemployment and Inflation: Actual versus Forecast

Endnotes

List of Tables

Table 1: Economic Growth: Forecasts versus actual
Table 2: Economic Growth: Forecasting errors by sub period
Table 3: Unemployment and Inflation: Forecasts and Outcomes
Table 4: Unemployment and Inflation: Forecasting errors by sub period

 

Major Issues

Recently there has been a good deal of interest in economic forecasting in Australia, chiefly as a result of the failure of official forecasts to pick the contraction in the December 2000 quarter. The purpose of the present paper is to examine just how well the forecasts have performed over time.

A series of forecasts going back to the 1978 Budget are examined and an assessment is made based on some summary measures, chiefly, the average forecasting error and the mean absolute error. Overall the paper finds a small bias towards underestimating economic growth, overestimating unemployment but impressive accuracy with inflation.

There remains the important question of whether the forecasts have been improving over time, as one might expect with improved computing facilities, better information and so on. To test this the sample was split into two sub samples and the forecast errors examined separately for each. For economic growth the answer seems to be that the average forecast error has increased but the mean absolute error has fallen. For a dart player that would be like saying that the average has moved somewhat to the right but the cluster, or the dispersion of the darts, is getting smaller. For economic growth the 'cluster' is almost a whole percentage point away from the bullseye if the effect of missing the contractions is ignored.

The paper finds that the forecasts are just slightly better than naïve forecasts. Two naïve forecasts were examined. The first is a forecast which just says that next year will be the same as the present. The second just says every year will experience economic growth at 3.5 per cent. The latter performs almost as well as the official forecasts, it has a lower average forecast error but a slightly higher mean absolute error. However, the difference in performance between the official forecasts and the 3.5 per cent rule is very small.

For unemployment the pattern of errors tends to follow those for growth but in the opposite direction. That is to be expected since growth would be one of the major determinants of employment in any economic modelling. The impressive lack of bias in inflation forecasting for the whole sample disappears when the sample is broken in two. The paper finds a forecast error that goes from an early underestimate to a later overestimate. The mean absolute error increases by the second sub sample. That is like the dart player starting off early in the evening with a bias to the left of the dart board and a small cluster, and ending the evening being biased to the right with a larger cluster.

Introduction

Australia was surprised when the December 2000 national accounts were released showing that the economy had shrunk by 0.6 per cent (seasonally adjusted) in that quarter. Few observers expected that result, although a modest slowdown was expected as a result of the post-GST contraction in building and construction. In the middle of that quarter, on 15 November 2000, Treasury released the Mid-Year Economic and Fiscal Outlook which expressed optimism and actually increased the forecast rate of economic growth for 2000-01. The Secretary to the Treasury, Dr Ken Henry, was later to say:

The Governor of the Reserve Bank of Australia, Mr Ian Macfarlane, in evidence to the House of Representatives Committee on Economics, Finance and Public Administration made the following point about economic forecasting:

The purpose of this paper is to examine the accuracy of the official economic forecasts. Official forecasts in Australia are the responsibility of the Joint Economic Forecasting Group (JEFG). The JEFG is composed of the Department of Treasury as the chair, the Department of the Prime Minister and Cabinet, the Department of Finance and Administration (DOFA), the Australian Bureau of Statistics and the Reserve Bank of Australia. The JEFG makes forecasts twice a year, at budget time and for the MYEFO. The Charter of Budget Honesty⁽²⁾ requires the public release of a pre-election economic and fiscal outlook report within ten days of the issue of the writ for a general election. Those forecasts are to provide the economic parameters to be used by Treasury and DOFA to cost election commitments.

Economic Growth: Actual versus Forecast

Before examining the record some quick definitions are in order. The main concepts used here are

• For a series of forecasts, the average forecasting error (AFE) is the actual value less the forecast value. So if the AFE is plus something then the forecasts can be said to be biased towards overestimating the outcome. If the AFE is negative then the forecasts are biased towards underestimating the outcome. If there is no bias at all the AFE will be zero. However, the forecasts can be unbiased but still very inaccurate,
• the mean absolute error (MAE) averages all the errors but ignores negative signs. For example, a plus one is treated the same as a minus one on the grounds that both errors are out by one.

The following Table presents Budget growth estimates and forecasts going back to the 1978 Budget. Unfortunately we do not yet have final figures for 2000-01. However, from 1994-95 there are essentially two forecasts. For the 1994-95 Budget, the delivery by the Treasurer was brought forward from August to May. That meant that, for example, in the 1993 Budget the Treasurer already knew the growth figure for 1992-93. However, the following year the Treasurer would have known the GDP figure for only the first two quarters of 1993-94. Hence, beginning with the 1994-95, Budget, growth estimates were also given for the current year, then being 1993-94. That means we can also examine the accuracy of estimates for the current year. The Budget Papers have consistently referred to estimates as applying to the present year and forecasts for beyond that. (These are both distinct from the projections that are also contained in the Budget Papers. Projections apply beyond the forecast period and are based on the assumption that growth, for example, will behave as it has done on average over the recent past.)

Table 1: Economic Growth: Forecasts versus actual

Budget Estimate for Current Year
		Actual	Budget Forecast for next year	Actual	Errors
					Current	Next Year
1978 budget			4	5.6		1.6
1979 budget			2.25	2.5		0.25
1980 budget			3	3.3		0.3
1981 budget			3.5	3.1		-0.4
1982 budget			3	-2.5		-5.5
1983 budget			3	5.6		2.6
1984 budget			4.5	5.1		0.6
1985 budget			4.5	4.3		-0.2
1986 budget			2.25	2.6		0.35
1987 budget			2.75	5.3		2.55
1988 budget			3.5	4.2		0.7
1989 budget			2.75	3.6		0.85
1990 budget			2	-0.2		-2.2
1991 budget			1.5	0.4		-1.1
1992 budget			3	3.6		0.6
1993 budget			2.75	4.1		1.35
1994 budget	4	4.1	4.5	4.5	0.1	0
1995 budget	4.75	4.5	3.75	4.4	-0.25	0.65
1996 budget	4.1	4.4	3.5	3.6	0.3	0.1
1997 budget	3.25	3.6	3.75	4.8	0.35	1.05
1998 budget	3.75	4.8	3	5.4	1.05	2.4
1999 budget	4.25	5.4	3	4.4	1.15	1.4
2000 budget	4.25	4.4	3.75	na	0.15
2001 budget	2	na	3.5	na
Average forecasting errors					0.41	0.36
Mean absolute errors					0.48	1.22

Source: Budget Paper No 1, various years, Australian Bureau of Statistics, National Income, Expenditure and Product, March quarter 2001, cat no 5206.0, 6 June 2001; Reserve Bank of Australia web site http://www.rba.gov.au/Statistics/Bulletin/G09hist.xls.

The figures in Table 1 have been used to construct the following graph which shows how the official forecasts compare with the actual outcomes as well as the forecast error for each year.

 

 

The final column in the above Table indicates that the average forecast was 0.36 of a percentage point below the eventual outcome. So, on average the official forecasts understated GDP growth by 0.36 per cent. However, the mean absolute error was 1.22 per cent. That tells us that on average the outcome is the forecast plus or minus 1.22 per cent. The biggest forecast error was 5.5 percentage points in 1982. In that year the forecast was for 3 per cent growth but the outcome was a 2.5 per cent contraction. In fact, neither of the two years of negative growth (1982 and 1990) were forecast. In more recent years, following the Asian financial crisis, economic growth was consistently underestimated. Whenever economic growth was above average at 5 per cent or more the official forecasts seriously underestimated growth. In our sample, economic growth has been 5 per cent or more on five occasions but there has not been one case where the forecast has been 5 per cent or more.

It is useful to compare the official forecasts with a know-nothing rule or a naïve forecast which just makes the hypothesis that next year's growth will be the same as this year's growth. The average forecasting error under that rule is minus 0.057. So it would just under-predict growth by 0.06 per cent compared with the official over-prediction of 0.36 per cent. The mean absolute error is 1.65 per cent which is a bit worse than 1.22 per cent, being the average forecast error in the official estimates.

Another rule could be to assume that economic growth will be 3.5 per cent every year. That is the figure the Budget Papers use to make their longer term projections beyond the forecast period. Using that rule gives an average forecast error of 0.08 per cent and a mean absolute error of 1.42 per cent, not significantly different from the mean absolute error in the official estimates.

The figures in the second to last column of Table 1 show errors made in estimating the current year's growth. Those estimates are made in May when there is only around 5 or 6 weeks of the financial year left. Nevertheless, substantial errors were made, especially for 1998-99 and 1999-2000. As it happens the estimates for the current year are downward biased by more than the downward bias for the next year forecasts. On average the forecasts were better for next year than the current year on these figures.

There is an important practical consequence of any downward bias in the forecasts. If the official growth estimates were biased downward by 0.41 per cent for the current year and 0.36 per cent for next year, then next year's GDP would be underestimated by 0.77 per cent. That could well translate into changes in the budget balance of well over $1 billion. Assuming a 0.77 per cent increase in GDP translates into a 0.77 per cent increase in the wages bill and a reduction in unemployment by 0.77 percentage points from a forecast 7 per cent, then, on the basis of the rules of thumb given in the Budget Papers, the budget balance would improve by approximately $1.7 billion.⁽³⁾ Those figures illustrate the sensitivity of Budget figuring to the economic forecasts.

Unlike the weather forecast, the official economic forecasts fail to anticipate extreme values as earlier discussed. However, to some extent that may well be deliberate with economic forecasts. While governments may try to present honest forecasts they will also want to avoid publishing estimates that may be self-fulfilling. We have seen how the negative GDP growth figure for December adversely affected consumer and business confidence when it was published in March 2001. Yet by that time the objective conditions in the economy had vastly improved as was shown by the 1.1 per cent growth, or an annualised 4.5 per cent growth in the March 2001 quarter. News about our history gave rise to a pessimism that was later seen to be unwarranted. Given the effect news has on economic confidence and behaviour, it is possible that governments in the past have chosen not to publish forecasts of negative growth and instead chosen to publish estimates of slow positive growth instead.

Before leaving this section there is a technical point that needs to be made. Forecasts should include all information currently available. But a perfect forecast on the basis of present information cannot incorporate new developments and shocks that occur during the forecast period. That means that the actual outcomes will display more variability than 'perfect' forecasts. This insight has been used in an assessment of OECD forecasts to examine their efficiency.⁽⁴⁾ A series of forecasts will be efficient if the variation in the forecasts is below the variation of the outcomes. This is confirmed for Australia's official forecasts with the standard deviation of the actual outcomes being 1.41 while the standard deviation of the forecasts is lower at 1.09.

Are the Forecasts Getting Better?

This section examines whether or not the forecasts are getting better over time. Mr Ian Macfarlane's comments above to the effect that forecasting has not improved over the last 30 years can be examined by splitting the above Table into two time periods. Those examined are the 1978 to 1988 budgets and the 1989 budget to the latest for which there is data. When that is done we obtain the results given in Table 2. Because of the distorting influence of the two downturns, the sub periods are estimated with and without the years 1982 and 1990.

Table 2: Economic Growth: Forecasting errors by sub period.

	1978 to 1988 budgets	1978 to 1988 without 1982 contraction	1989 to 1999 budgets	1989 to 1999 without 1990 contraction
Average forecast error	0.26	0.84	0.46	0.95
Mean absolute error	1.37	0.96	1.06	0.73

The raw results in Table 2 suggest that theaverage forecast error has worsened from the first half to the second half of the sample. This result is not statistically significant.⁽⁵⁾ The results suggest that if we ignore the failure to predict the contractions then the average error jumps to almost one percentage point by the second subset of the sample. This bias is in fact worse in the second half of the sample, suggesting that the official forecasts are more biased in more recent years. The mean absolute error appears to improve from the first to the second half and falls substantially if the failure to forecast the downturns are ignored. Once again the difference between the samples is not statistically significant.

Unemployment and Inflation: Actual versus Forecast

Other important variables that the Budget Papers attempt to forecast include unemployment and inflation. Those forecasts are intrinsically important but are also important for deriving government expenses (previously 'outlays') and revenues.

Table 3 presents budget-time forecasts for unemployment and inflation since the 1978-79 Budget. Some of the early years presented problems because estimates were not always given as a precise numerical forecast. In constructing Table 3 some of the actual numerical targets had to be inferred from the relevant Budget papers. For example, the discussion in the 1978-79 Budget Papers predicted unemployment would be 'unchanged' when the latest figures then published showed unemployment to be 6.8 per cent in the relevant quarter. In following years unemployment was forecast to be 'broadly unchanged (1979),' 'little if any decline (1980),' 'slight fall (1981),' 'marked increase (1982),' 'higher (1983),' 'decline slightly (1984),' '7.5 to 8' at the end of the year (1985), and 'some increase from 7 (1986).' In each case, where there was no specific number mentioned, but the outcome was consistent with the anticipated movement, the Table credits the forecasts with getting the outcome exactly right. Since 1986 the Budget Papers have provided tables with detailed numerical forecasts rather than the more impressionistic discussions. The situation was better with inflation. However, in 1980 the forecast avoided a numerical target and said 'about the same or slightly faster at 10. 'The 1984 Budget forecast a through-year CPI increase because of the complications of the Medicare effect, whereby the introduction of Medicare was expected to distort year on year figures.

Table 3: Unemployment and Inflation: Forecasts and Outcomes

	Unemployment rate-year average (%)			Inflation-year average (% increase in CPI)
	Budget forecast for next year	Actual	Error	Budget forecast for next year	Actual	Error
1978 budget	6.8	6.3	-0.5	6.0	8.1	2.1
1979 budget	6.6	6.2	-0.4	10.0	10.2	0.2
1980 budget	6.2	5.9	-0.3	10.0	9.3	-0.7
1981 budget	6.0	6.2	0.2	10.75	10.4	-0.35
1982 budget	9.0	9.0	0	10.75	11.5	0.75
1983 budget	9.6	9.6	0	7.5	6.9	-0.6
1984 budget	8.6	8.6	0	5.25	6.6	1.35
1985 budget	7.75	7.5	-0.25	8.0	8.4	0.4
1986 budget	8.0	8.1	0.1	8.0	9.3	1.3
1987 budget	8.25	7.5	-0.75	7.0	7.3	0.3
1988 budget	6.25	6.4	0.15	7.5	7.3	-0.2
1989 budget	7.25	5.9	-1.35	5.5	8.0	2.5
1990 budget	7.25	8.1	0.85	6.5	5.3	-1.2
1991 budget	10.5	10.0	-0.5	3.0	1.9	-1.1
1992 budget	10.5	10.7	0.2	3.0	1.0	-2.0
1993 budget	10.75	10.2	-0.55	3.5	1.8	-1.7
1994 budget	9.75	8.7	-1.05	2.25	3.2	0.95
1995 budget	8.25	8.1	-0.15	4.0	4.2	0.2
1996 budget	8.5	8.3	-0.2	2.0	1.3	-0.7
1997 budget	8.25	8.0	-0.25	1.0	0	-1.0
1998 Budget	8.0	7.4	-0.6	2.5	1.3	-1.2
1999 budget	7.5	6.6	-0.9	2.0	2.4	0.4
2000 budget	6.5			5.75
2001 budget	7.0			2.0
Average forecasting errors			-0.28			-0.01
Mean absolute error			0.42

Source: Budget Paper No 1, various years, Reserve Bank of Australia web site
http://www.rba.gov.au/Statistics/Bulletin/G09hist.xls.

Before commenting on the inflation and unemployment forecasts is should be pointed out that labour market forecasts will depend heavily on the forecasts for economic growth. Errors in the latter will mean errors in the former. Table 3 shows the influence on unemployment estimates of the earlier discussed errors in growth forecasts. The average error of 0.28 means that the forecasts have overestimated unemployment, consistent with underestimating growth. The mean absolute error seems reasonably low, however, indicating fairly accurate forecasts for unemployment.

Inflation forecasts are very interesting. The negligible average forecast error is very impressive. This suggests a record of unbiased estimates of inflation. The mean absolute error is also reasonable given the rather volatile nature of inflation over the last 20-odd years. However, it is also interesting to split the sample in two to examine whether or not forecasting has improved or otherwise changed. This is done in the following table.

Table 4: Unemployment and Inflation: Forecasting errors by sub period.

	1978 to 1988 budgets	1989 to 1999 budgets
unemployment
average forecast error	-0.16	-0.41
mean absolute error	0.24	0.60
inflation
average forecast error	0.41	-0.44
mean absolute error	0.75	1.18

It has already been noted that unemployment errors tend to reflect growth errors. The average forecasting error for unemployment increased somewhat between the two periods. That lower early error may only reflect the manner in which the data was constructed for the earlier period as described above. However, the inflation figures will not depend so much on growth forecasts. The impressive unbiased forecasts for the 22 year period are not so impressive when the sample is split in two. Now Table 4 indicates that in the first half there was a systematic bias towards underestimating inflation while in the second half there was a systematic bias towards overestimating inflation. The latter seems to have reflected a period of overestimating inflation following the 1990 downturn, as well as some overestimating inflation around the time of the Asian financial crisis and the aftermath. The mean average error has also increased somewhat between the two sub-samples suggesting that the official forecasts are not as good as they used to be. Once again, however, the differences in the Table are not statistically significant.

Endnotes

2. Dr K. Henry, 'On economists, the economy and fiscal policy,' Address to the Australian business economists, Sydney, 29 May 2001.

3. See Charter of Budget Honesty Act 1998.

4. The 'rules of thumb' for estimating the effects of various parameter changes are given in Table B1 of Budget Strategy and Outlook 2001-02, Budget Paper no 1, 2001, p. 2-19.

5. V. Koutsogeorgopoulou, , A Post-Mortem on Economic Outlook Projections (PDF format) OECD Economics Department Working Papers No. 274, 2000.

6. No formal statistical test was carried out because that was unnecessary. None of the differences reported in the Table even approached the value of the standard deviation for the sample as a whole or the subsets of the sample used

---