How do we Measure the Success of Legislative Budget Offices?

Legislative Budget Offices (LBOs) are independent institutions staffed with economists that provide rigorous and non-partisan analysis to legislatures on matters pertaining to the budget. Their appeal lies in their ability to provide robust economic analysis that avoids taking political sides, which is meant to help create a more evidence-based dialogue in politics.

So far, there seems to be a general agreement that, in the countries where they have been instituted, LBOs create some form of good in the budget process. The problem is that measuring this good is complex.

Even as academic investigation into LBOs is growing, the actual measurement of their success has not been a primary focus. Rather, the current body of research has looked at their performance in more intangible ways. These intangible contributions have had a significant impact, but they don’t address the more concrete questions around- how successful are the LBOs?

This question is of great importance since the LBO institutional model has now proliferated across more than 60 countries. As their presence, influence, and scope expands, it becomes critically important for more integrated and more specific approaches to measuring their success – ideally in ways that go beyond the intangible or unmeasurable.

For starters, at least three disciplines take an interest in LBOs: economics, political science, and governance. First, from the perspective of economic theory, LBOs are meant to introduce a more long-term attitude towards the budget process, which is a response to dynamic inconsistency that arises from the desire to consume more in the present and save less for future periods, which leads in turn to a problem known as the permanent deficit. The analysis of LBOs is meant to provide more stability and consistency to the fiscal policy decisions over longer time horizons.

Second, from the standpoint of political theory, LBOs are meant to level the playing field in the budget process between the legislative arm and the executive arm. In many countries, including robust democracies, the dominant executive branch often has the upper-hand in the budget process. Through their analysis, LBOs help to strengthen the legislature’s hand in the budget process, and this is thought to lead to more democratic outcomes both in budget-making and in budget oversight.

Third, from the standpoint of governance theory, LBOs are considered an institution that bolsters the level of transparency and accountability in the budget process, which for the purposes of developing countries, is meant to fuel economic growth as the structural barriers created by poor institutions are lessened. The proliferation of LBOs among developing countries has in large part been driven by such thinking.

These three theoretical approaches notwithstanding, it is important to note two things. First, that there have been very few interdisciplinary approaches to understanding the LBO, and greater research is required to address this knowledge deficit. Second, these approaches are grounded in rigorous theory, but theory nonetheless – the actual assessment of LBO effectiveness remains something of a mystery in the scant literature.

Let’s look at some ways that the performance of LBOs could be measured, and why these approaches might not work due to serious limitations.

1. Counterfactuals: The ideal method of assessing LBO effectiveness would involve a counterfactual approach. This would look at the impact of an LBO in terms of changes in budget balance or fiscal stability, and then compare this with a counterfactual outcome such as what the fiscal balance would have been if the LBO had not been around.  The inherent problem with this approach is that the counterfactual cannot actually be observed. There is room for an abstract discussion of how things may have been different, but this does not lend itself to concrete measurements. The counter-factual is the least practicable of all measurement methods.

2. Enhanced transparency: A more practical approach would be to measure the enhanced transparency of the budget process. The International Budget Partnership’s (IBP) Open Budget Index is one particularly noteworthy resource in this regard, given its comprehensive and peer-reviewed methodology. Measuring the change in the IBP score assigned to a country over time can indicate whether there are improvements in the transparency of the budget process over time, and the IBP’s index has several questions specifically about the existence of an LBO. However, a more careful look at the Open Budget Index survey instrument reiterates our initial problem: Simply asking “do you have an LBO?” and answering “yes, we do”, does not speak to the underlying inquiry: “does your LBO do a good job?” To this point, there is a deeper question to ask: does better transparency lead to better fiscal outcomes? If that were true, the country with the best LBO, the United States, could have the most manageable and least worrisome budget balance. But this the truth is in fact quite the opposite, the US is veering towards an untenable fiscal position, even as its budget office, the CBO, has spent the last 40 years trying to avail politicians and the public about this situation.

3. The public and the media: In a vibrant democratic environment, the voice of the public should be seen as the ultimate arbiter of an effective public institution. However, public perceptions are not guided by a fully-informed set of inputs, either due to insufficient knowledge about the workings of an LBO, or a general disinterest in LBOs due to the seeming remoteness of the budget process itself. An active media can play a role in publicizing the work of an LBO, as it has in Canada. To this point, surveys can be conducted that ask the public of their perception of the LBO’s contribution. Press coverage can also be gauged in this manner.  Yet, it is difficult to measure the impact of the LBO in truly concrete terms if the assessment is perceptual.

4. The opposition: An important part of an LBO’s work is to arm the opposition with budgetary acumen to challenge the governing party’s budget dominance. When measuring this, an LBO can ask opposition party MPs to fill a survey instrument indicating their level of satisfaction with the analysis of the LBO. This can entail questions of response times, response details, and the like. The PBO of Australia conducts regular feedback surveys from stakeholders. While nonetheless important, the opposition’s opinion of the LBO does not cover the full extent of what an LBO can accomplish, and it may be politically-motivated and dependent on opposition and government switching positions in the legislature. It is also a highly perceptual measure.

5. Sovereign ratings: The creditworthiness of a nation depends on several factors including the current deficit/debt levels, their rate of expansion, and the ability to cover obligations coming due at various maturities. In theory, the “impact” of an LBO can be in part inferred from looking at how the sovereign rating of a country changes after its installation. If the post-LBO sovereign rating is higher, it could insinuate a positive influence on the fiscal position due to the LBO. In reality, this becomes problematic because of false inferences since many countries have instituted LBOs at times when sovereign ratings are in a state of downgrade, or with sovereign ratings unchanged, thereby implying that the LBO had no impact, or worse, a negative impact on the sovereign rating.

6. Cross-country comparisons: To create cross-country comparisons between LBOs would seem like an excellent manner of gauging their effectiveness, but in practice this is impossible due to the immense heterogeneity of LBOs, which diverge so much in size, mandate, scope, purpose, financial resources, staffing, access to information, legal recourse, leadership style, and budgeting culture. Consider this comparison: the CBO of the United States, which has the second largest number of economists of any federal agency, second only to the Federal Reserve. By contrast, the LBOs of some developing countries constitute only a few junior budget officers with limited resources and ramshackle work conditions. Countries such as Canada, Australia, and EU member states fall somewhere in between the two extremes.

These approaches and their limitations are summarized here:

Approach	Tool	Objective	Limitation(s)
Counterfactuals	Fiscal metrics*	To measure the specific impacts of the LBOs	Counterfactual is unmeasurable
Enhanced transparency	Open Budget Index (e.g.)	To measure improvements in budget process	Recognizes LBOs, but does not assess their success
Public and media	Survey instrument	Responses to the work of the LBO	Insufficient information; complexity; perceptual
Opposition party	Survey instrument	Responses to the work of the LBO	Perceptual; susceptible to politically-motivated feedback
Sovereign ratings	Change in rating	To measure LBO impact on credit worthiness	False inferences: either downgrades occur or rating unchanged
Cross-country comparisons	Fiscal metrics*	To measure LBO impact between countries	Excessive country contrasts limit value

*Fiscal Metrics refer to the fiscal position, budget balance, the size and growth of the deficit, and so forth

As result, none of the aforementioned approaches is comprehensive and each suffers from substantial limitations. Yet LBOs are becoming ever more important in the budget process, and it is generally agreed that they provide some form of good. Further research and creative approaches are thus necessary for us to really measure success of LBOs.