Image by Long Zheng CC 2.0 via Flickr

It has been previously noted that there are inherent challenges in measuring the effectiveness of Legislative Budget Offices (LBO). Although it is clear that they do provide “some form of good” for the budget process, through their role in enhancing budget transparency and fiscal accountability, as well as their assistance to legislatures in budget oversight, all of this is difficult to measure or substantiate in a concrete manner.

A previous article in the Austaxpolicy blog proposed the use of survey instruments to solicit feedback from important stakeholders, such as legislators in both government and the opposition. Feedback from parliamentary political parties on the LBO’s work would, in theory, provide an excellent manner of assessing its usefulness. In particular, it would provide insights into areas of improvement required by the institution.

However, there are limitations on the use of survey instruments in an LBO context, as a recent discussion paper has identified.

First, survey instruments are an inherently subjective measure. This is important to note because “doing a good job” will be very much a function of the expectations that legislators hold. Some legislators may set low expectations and be content with a passive “costings role”. Others may have higher expectations, particularly when they are in the opposition, expecting a more “activist” budget office that follows an open-publishing approach with strong ties to the media.

Second, a survey instrument can be vulnerable to politicization, as some legislators may use the survey instrument to provide negative feedback as a form of reprisal. As a result, the survey instrument may not be an adequate reflection of the work that an LBO does. This is because the work of an LBO challenges the legislators in the fiscal process to a certain degree.

Third, the survey may turn out to be an extremely costly enterprise, as the case of Australia’s PBO stakeholder survey shows. In 2015, the PBO had a survey instrument sent out to more than 3,000 stakeholders, but only got 63 responses in return. Worse yet for the taxpayer, the PBO sought an external consulting firm to disseminate the survey, which brought a pricetag of roughly $23,000. In a Senate Estimates hearing, ACT Labor Senator Katy Gallagher expressed her amazement at the exorbitant cost of sending a survey to so many people, saying that “It’s money for jam, really: 63 people responding for $23,000 dollars?” In response, Parliamentary Budget Officer Phil Bowen admitted that, “[in] hindsight we sent that to a lot of staff who we probably should never have.”

Fourth, survey instruments raise questions about data security and privacy. In a Senate Estimates hearing in October 2015, Labor senators told Parliamentary Budget Officer Phil Bowen that they had serious concerns about the effectiveness of the layers of security in the survey, since the PBO sent the email addresses of all 3,000 respondents to an external consulting firm without receiving consent to do so. To this point, Senator Penny Wong,  raised the question of why the consent of those involved had not been sought. Senator Gallagher added, “I don’t know if that’s usual, to provide a distribution list to a private company of people’s work emails.” In response, the PBO said that if the committee wanted a “perfectly honest answer, I did not consider that as an issue,” in part because the external firm had signed a confidentiality agreement and, “all email addresses in the stakeholder contact list have been deleted from its files”. According to the PBO, the email addresses were provided by the Department of Parliamentary Services.

Fifth, large survey instruments require significant resources in terms of both staff and technology when conducted in-house. The PBO of Australia delegates the survey procedure to external parties because it doesn’t have the capacity to conduct the survey on its own. This speaks to a point identified by academics, that PBOs are organisations universally short on resources given the large scope of budget work that they must do. Australia’s PBO is one of the best resourced institutions of its kind in the world, and yet it too has limitations that would make it prohibitive for the execution and ex-post analysis of a survey instrument.

Concerns around the use of the survey instrument are summarized below:

Nonetheless, while survey instruments face some core limitations, they can still be considered part of a larger system of assessment of the work that an LBO performs. So long as the caveats around the survey instrument are recognised, it can form part of a more comprehensive assessment of LBOs, and future research can help to explore other components of such a comprehensive assessment.

Leave a comment

Your email address will not be published. Required fields are marked *