The promises of quantitative text analysis in interest group research: A reply to Bunea and Ibenskas

Abstract

Quantitative text analysis constitutes a promising new method that allows for measuring the policy positions and the lobbying success of interest groups by analyzing their submissions to legislative consultations (Klüver, 2009). The use of quantitative text analysis allowed me to present a novel and unique research design which was the largest in scope at the time and resulted in important new insights regarding the determinants of lobbying success (Klüver, 2009, 2011, 2013). In their recent article, Bunea and Ibenskas (2015) however question the usefulness of quantitative text analysis for studying interest groups and discuss several issues which in their view constitute important disadvantages of the technique. In this article I carefully discuss each of their arguments and show that none of their objections actually prevents scholars from successfully using quantitative text analysis to study interest groups in the European Union and beyond.

Keywords

Influence interest groups lobbying quantitative text analysis

Introduction

Quantitative text analysis constitutes a promising new method that allows political scientists to systematically extract political information from texts. Due to the increasing digital availability of texts, scholars can employ quantitative text analysis for studying political phenomena in an entirely new fashion by making use of massive amounts of texts available on the world wide web and in political archives. Political Scientists have successfully used automated text analysis to analyze party manifestos, speeches, press releases, newspaper articles, legislative proposals and party congress motions (e.g. Ceron, 2014; Grimmer, 2010; Laver et al., 2003; Slapin and Proksch, 2008). In interest group research, I have introduced quantitative text analysis as a new methodological approach for measuring the policy positions and the lobbying success of interest groups by analyzing their submissions to legislative consultations (Klüver, 2009). The use of quantitative text analysis allowed me to present a novel and unique research design which was the largest in scope at the time and resulted in important new insights regarding the determinants of lobbying success (e.g. Klüver, 2009, 2011, 2013). This research project has inspired others to follow my example by using qualitative content analysis to also study interest group submissions to legislative consultations (see e.g. Bunea, 2013). In their recent article, Bunea and Ibenskas (2015) however question the usefulness of quantitative text analysis for studying interest groups and discuss several issues which in their view constitute important disadvantages of the technique. In this article I will carefully discuss each of their arguments and show that none of their objections actually prevents scholars from successfully using quantitative text analysis to study interest groups in the European Union and beyond.

Words as data

First, Bunea and Ibenskas (2015) criticize quantitative text analysis for treating words as the units of analysis. The so-called ‘Bag of Words’ text analysis techniques such as Wordfish (Slapin and Proksch, 2008), Wordscores (Laver et al., 2003) or topic models (e.g. Grimmer, 2010) all have in common that they use individual words as the unit of analysis while ignoring the context in which words occur. Obviously, this is a simplifying assumption as language is inherently contextual. However, even though the ‘Bag of Words’ approaches make this naive and unrealistic assumption, it does not mean that the results obtained by these techniques are not valid. The validity of the obtained estimates by contrast crucially depends on whether the underlying assumptions reflect the data generating process of the analyzed texts and whether careful validation checks are performed.

Both Wordfish and Wordscores rely on the assumption that ideology dominates the language that political actors use in their documents. Laver and Garry (2000) gave a very intuitive example of the assumption that word usage is affected by ideology. While all parties use the word ‘taxes’, its relative frequency varies extensively. Liberal and conservative parties which fight for more market liberalization and less state intervention extensively use this word to call for a reduction of taxes. Social democratic and socialist parties by contrast use the word ‘taxes’ significantly less often since they typically plan to increase taxes to fund higher levels of social welfare expenditure. Rather than making this explicit by talking about the increase of taxes, these parties instead talk about the benefits of redistribution. As this example illustrates, the relative frequency of words can tell us a quite a lot about policy positions.

The language employed by interest groups in their position papers is similarly dominated by their ideological stance on the issue. Even though the nature of the texts is much more circumscribed and technical than election manifestos as interest groups draft position papers to influence specific legislative proposals, the vocabulary in these position papers is similarly affected by their policy position on the issue. The word weights and word fixed effects produced by Wordfish nicely illustrate this. Wordfish discriminates between policy positions of texts drawing on words weights. A high word weight indicates that a word occurs very often in some, but not in other texts. By contrast, word fixed effects account for the fact that some words such as articles or conjunctions generally occur very often in documents. Thus, words with high fixed effects should not indicate any policy stances, while words with high word weights should be indicative of underlying ideological differences.

In order to inspect how well individual words discriminate between interest groups’ policy positions, Figure 1 plots the estimated word fixed effects against the word weights for the CO₂ emissions debate analyzed by Klüver (2009). The so-called ‘Eiffel Tower of Words’ discussed by Slapin and Proksch (2008) clearly emerges. Words with high word fixed effects have very low word weights such as the word stems ‘vehicle’, ‘car’ and ‘CO₂’ which are used very frequently by all interest groups lobbying in this policy debate.¹ Stems with high political connotation, by contrast, have very low word fixed effects, but high positive or negative word weights. Interest groups fighting for stricter emission standards frequently use word stems such as ‘zero-emiss’, ‘natur’ or ‘warm’ which have very high negative word weights. Interest groups opposing stricter emission targets by contrast frequently use word stems such as ‘tire’ and ‘resist’ and ‘disadvantage’ that have very high positive word weights.

Table 1 indicates the word stems with the highest word weights and the highest word fixed effects. Environmental non-governmental organizations (NGOs) and producers of electric cars for instance use the terms ‘zero-emiss’, ‘electr’ and ‘hybrid’ to state that electric and hybrid cars are much more environmentally friendly than traditional cars. Environmental groups use the term ‘warm’ to emphasize the negative implications of CO₂ emissions for global warming. By contrast, traditional car manufacturers for instance used the terms ‘disadvantag’ and ‘tire’ to discuss the negative consequences of stricter emission standards for engines and tires for the competitiveness of the European car industry. In addition, these groups used the term ‘differenti’ to argue for flexible reduction targets. For example, the German Automobile Manufacturer Association (VDA) calls for a ‘differentiated approach that does justice to different vehicle classes’.

Table 1.

Top ten word weights and word fixed effects.

Word weights: Pro	Word weights: Anti	Word fixed effects
zero-emiss	tire	vehicl
batteri	resist	Car
urban	roll	Co2
electr	polit	emiss
warm	pressur	by
immedi	evalu	Not
clean	segment	fuel
hybrid	basic	target
so-cal	disadvantag	eu
perfect	differenti	manufactur

Source: Klüver (2013).

The inspection of the word fixed effects and the word weights estimated by Wordfish indicate that a word-based analysis clearly captures positional differences between interest groups. Words with high word fixed effects are general terms describing the substance of the policy debate that need to be used by all interest groups to discuss the proposal. By contrast, terms with high word weights exhibit a clear ideological orientation and therefore allow for discriminating between different policy positions.

Validation

Bunea and Ibenskas (2015) furthermore question the validity of the Wordfish estimates. Quantitative text analysis can open up entirely new ways of studying interest groups as it allows for an automated investigation of massive amounts of lobbying documents. However, it is crucial that scholars validate the output they obtain using quantitative text analysis. I therefore call for a cross-validation of interest group position estimates with multiple sources to assess convergent validity (Grimmer and Stewart, 2013). I propose three major convergent validity checks. First, scholars can employ manual hand-coding to cross-validate the output of computerized techniques by relying on human coders to code the content of interest group position papers. Although hand-coding is associated with a high degree of validity, it suffers from two major disadvantages. It is very time-consuming and costly so that such a check can typically only be conducted for one or just a few policy debates. In addition, it suffers from severe reliability problems in terms of unitization (Däubler et al., 2012) and coding (Mikhaylov et al., 2012). Second, interest group scholars can cross-validate the output of quantitative text analysis with estimates obtained by other quantitative text analysis approaches. Such a cross-method comparison can shed light on how robust the results are to the choice of the precise quantitative text analysis method. However, comparing different quantitative text analysis approaches cannot help us much to assess the validity of the estimates as all fully automated techniques suffer from the same validation requirement. Third, one can use external substantive data to cross-validate the estimates obtained by quantitative text analysis which is the strongest convergent validity test that can be performed.

While Bunea and Ibenskas (2015) only rely on manual hand-coding and a cross-validation with Wordfish, they fail to validate their results with an external data source. One may therefore wonder on what ground they infer that their hand-coding analysis is superior to the hand-coded content analysis presented by Klüver (2009). This is particularly the case since Bunea and Ibenskas’s (2015) method is not based on a previously well-established coding technique, such as the coding procedure developed by the Comparative Manifesto Project. In contrast, I have applied all three validation techniques to demonstrate the validity of the Wordfish position estimates in my previous work (Klüver, 2009, 2013; Klüver and Mahoney, 2015). In addition to comparing the Wordfish estimates with a carefully conducted systematic hand-coding analysis that followed the procedure of the Comparative Manifesto Project and a cross-method comparison with Wordscores and T-Lab, I have tested the validity of the Wordfish output with survey data about cooperation partners and opponents of interest groups. In a specifically designed web survey among all interest groups that lobbied the European Commission in 56 selected policy debates, I asked interest groups to name one major opponent and the five most important cooperation partners in a given policy debate (see Table 2).²

Table 2.

External validity test.

Wordfish Estimation	Frequency	Per cent
Correct	276	79.54
Incorrect	56	16.14
Ambiguous	15	4.32
Total	347	100.00

Source: Klüver (2013).

The information about cooperation partners and opponents allows for checking whether Wordfish has correctly estimated the policy positions of interest groups. If the estimation is correct, the cooperation partners should be located on the same side of the initial policy position of the European Commission whereas the opponents should be located on the opposing side.³ If the majority of the cooperation partners and opponents that interest groups indicated in the survey are located on the correct side, the Wordfish estimation was coded as being correct. If the same number of interest groups is located on the correct and the incorrect side of the European Commission, the Wordfish estimation was coded as being ambiguous. If the majority of actors are located on the wrong side of the European Commission, the Wordfish estimation was coded as being incorrect. Out of 347 cases in which opponents and cooperation partners were reported, 80 per cent were estimated correctly and only 16 per cent were coded incorrectly (see Table 2). Hence, the external validity check strongly supports the validity of the Wordfish measurement.

Multidimensionality

It is furthermore argued that quantitative text analysis techniques are not suitable for analyzing interest group positions as they could not capture the seemingly multidimensional nature of policy debates. However, Bunea and Ibenskas (2015) overemphasize the importance of multiple dimensions in public policy debates and overlook recent advances in quantitative text analysis that allow for a multidimensional analysis of interest group positions.

First, they argue that quantitative text analysis techniques such as Wordfish or Wordscores are not useful for studying interest group positions as they only measure policy positions on one single policy dimension at a time. However, the importance of multidimensionality for interest group research is exaggerated. It is true that policy proposals contain different policy issues that are regulated by the same legislative proposal (see e.g. the hand-coding codebook developed by Klüver (2009)). However, even though a proposal may regulate multiple issues, this does not mean that interest groups actually compete on all of these issues. The decisive question is therefore not whether a proposal contains multiple issues, but whether a proposal truly triggers a multidimensional structure of conflict. For instance, even though parties take positions on a multitude of different policy issues in the EU, the policy space can be characterized by one major left–right policy dimension which structures political conflict (e.g. Gabel and Hix, 2002; Hix et al., 2005). Given that the vast majority of policy debates in the EU does not raise any attention or only lead to the mobilization of just a handful of interest groups, it is highly questionable that interest group lobbying predominantly takes place in a multidimensional context.

In one of the most comprehensive studies of interest group lobbying to date, Baumgartner et al. (2009) have analyzed 98 policy debates in the United States. They have found that the vast majority of these debates solely boil down to a unidimensional structure of conflict. One lobbying coalition typically seeks to change the status quo while the opposing lobbying coalition seeks to preserve the status quo. Similarly, Bunea and Ibenskas (2015) find themselves that the first environmental regulation dimension identified by Klüver (2009, 2013) accounts for as much as 80 per cent of the variance in the CO₂ emissions debate. It is therefore questionable whether conducting multidimensional analyses of lobbying success really brings much added value to the study of interest groups.

Second, while Bunea and Ibenskas (2015) suggest that no multidimensional quantitative text analysis has been conducted in the scholarship on EU lobbying, a new quantitative text analysis approach has already been introduced that allows for measuring interest group positions in a multidimensional policy space (Boräng et al., 2014; Klüver and Mahoney, 2015). This text analysis approach combines a cluster analysis with a correspondence analysis implemented in the software package T-Lab to study interest group positions, frames and lobbying success in a multidimensional context (for further details, see Boräng et al., 2014; Klüver and Mahoney, 2015).

Figure 2 illustrates the policy position estimates obtained by the cluster and correspondence analysis for the CO₂ emissions debate. Similar to the findings of Bunea and Ibenskas (2015), the T-Lab analysis arrives at two major dimensions along which interest groups can be aligned. The first dimension corresponds to the environmental regulation dimension identified by Klüver (2009) along which traditional automobile producers are fighting environmental groups and so-called alternative industry groups which produce electric cars or biodiesel. The second dimension indicates a conflict about the regulation of advertisement of cars with high levels of CO₂ emissions which the press groups AAUK and FAEP strongly oppose. Table 3 shows the relationship between the T-Lab estimates presented by Klüver and Mahoney (2015) with the Wordfish and Hand-Coding estimates obtained by Klüver (2009). The T-Lab estimates strongly correlate with the hand-coding (r = 0.76) and the Wordfish (r = 0.74) estimates on the environmental regulation dimension while the correlation with the second advertising regulation dimension is only moderate.

Table 3.

Comparison of text analysis estimates.

		T-LAB
Environmental regulation dimension	Hand-Coding	0.76***
	Wordfish	0.74***
Advertising regulation dimension	Hand-Coding	0.34*
	Wordfish	0.05

Note: *** $p \leq 0.01$ , ** $p \leq 0.05$ , * $p \leq 0.10$ ; coefficients represent Pearson correlation coefficients (absolute values).

Source: Klüver (2009), Klüver and Mahoney (2015).

Hence, the results of the multidimensional T-Lab analysis largely corresponds with the results presented by Bunea and Ibenskas (2015). First, both analyses show that the environmental regulation dimension dominates the debate. Second, both studies show that Wordfish correctly captures this dimension. Third, Bunea and Ibenskas (2015) and Klüver and Mahoney (2015) demonstrate that the second dimension only accounts for little additional variation regarding the conflict between press groups on the one hand and the remaining interest groups on the other hand. However, while the measurement approach by Bunea and Ibenskas (2015) requires manual hand-coding of all texts, Klüver and Mahoney (2015) suggest a fully computerized text analysis technique that avoids the severe reliability problems of hand-coding and is considerably more time and cost-efficient.

Languages and the selection of texts

Finally, Bunea and Ibenskas (2015) criticize Wordfish for its language sensitivity. While it is possible to conduct a Wordfish analysis in any language, it is necessary that texts examined in the same Wordfish analysis are all written in the same language. I therefore have decided to drop five interest group submissions not authored in English from the analysis of the CO₂ emissions debate (Klüver, 2009, 2013). The decision to exclude non-English submissions was based on my limited resources as a PhD student conducting a project analyzing 56 policy debates and more than 3200 interest groups and national authorities at the time. However, the language sensitivity of Wordfish does not prevent scholars to apply this technique in studies of interest group lobbying more generally.

First, the problem of interest group position papers written in different languages is not as important as Bunea and Ibenskas (2015) state. The European Union is a very special case since it is a multilevel polity that is composed of 28 different national political systems in which 24 official languages are used. Interest groups from the different member states can therefore submit position papers in different languages. However, the vast majority of political systems around the world have only one official language so that interest group submissions should typically be authored in the same language. If one looks beyond the EU case, the language sensitivity of Wordfish is therefore not a problem when studying interest group lobbying within different countries.

Second, the language sensitivity can moreover be overcome by document translation. To study interest group lobbying in the EU or in comparative designs across countries, scholars can first translate interest group texts into the same language before applying Wordfish. For instance, Proksch and Slapin (2010) applied Wordfish to over 50,000 speeches given in the European Parliament (EP) to measure the ideal points of parties in the EP. As Members of the European Parliament (MEPs) have the right to communicate in their national language(s), each speech must be translated into each of the 24 official languages. Proksch and Slapin (2010) analyzed all of the speeches in the English, the French and the German translations provided on the EP website. They show that the Wordfish estimation is highly robust to the choice of language as the estimates correlate at 0.86 or higher.

Finally, Bunea and Ibenskas (2015) criticize the selection of interest groups in my previous work. For practical purposes, I limited the analysis in the illustrative case study (Klüver, 2009) to interest groups that classify as membership organizations. Solely focusing on membership organizations is a well-established procedure in the interest group literature as a wide variety of interest groups such as business associations, trade unions and NGOs that are amongst others organized as national associations or European umbrella associations are captured by this definition (see e.g. Baroni et al., 2014). However, in later publications originating from the broader research project, the Wordfish analysis has already been extended to cover the submissions of all types of membership organizations, firms and even national authorities (e.g. ministries, local authorities, public institutions) (e.g. Klüver, 2011, 2013). Bunea and Ibenskas (2015) claim that the Wordfish estimates would systematically be flawed when interest groups with different organizational forms would be analyzed due to different data generating processes. However, the Wordfish estimates for the extended analysis of membership organizations, firms and national authorities presented in Klüver (2013) correlate at 0.67 with estimates obtained through manual hand-coding. This clearly indicates that Wordfish is perfectly capable of estimating the policy positions of different types of interest groups.

Conclusion

In this article, I have shown that the arguments brought forward by Bunea and Ibenskas (2015) against the use of Wordfish and quantitative text analysis more generally do not hold. Neither the language sensitivity nor the reliance on individual words as units of analysis prevents researchers from obtaining valid position estimates for interest groups using Wordfish or related techniques. The validation discussion has moreover shown that the results obtained by Wordfish have been internally and externally validated by a number of sources which go far beyond the validation provided by Bunea and Ibenskas (2015). Finally, while Bunea and Ibenskas (2015) are right that Wordfish cannot estimate policy positions in a multidimensional policy space, it is first questionable whether a multidimensional analysis actually brings much added value as most policy debates are characterized by a unidimensional structure of conflict. Second, recent advances in quantitative text analysis already allow for a multidimensional analysis of interest group positions without having to rely on time-consuming and costly hand-coding that additionally suffers from severe reliability problems. As long as interest group scholars pay careful attention to the underlying assumptions of the quantitative text analysis techniques and heavily rely on internal and external validation techniques, quantitative text analysis paves the way for a promising new research agenda in the study of interest groups.

Footnotes

Acknowledgements

The author thanks Anne Rasmussen, Berthold Rittberger, Gerald Schneider and the anonymous reviewer for valuable comments and suggestions.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Notes

References

Baroni

Carroll

Chalmers

(2014) Defining and classifying interest groups. Interest Groups and Advocacy 3(2): 141–159.

Baumgartner

Berry

Hojnacki

(2009) Lobbying and Policy Change: Who Wins, Who Loses, and Why, Chicago, IL: University of Chicago Press.

Boräng

Eising

Klüver

(2014) Identifying frames: A comparison of research methods. Interest Groups and Advocacy 3(2): 188–201.

Bunea

(2013) Issues, preferences and ties: determinants of EU interest groups’ preference attainment in the environmental policy area. Journal of European Public Policy 20(4): 552–570.

Bunea A and Ibenskas R (2015) Quantitative text analysis and the study of EU lobbying and interest groups. European Union Politics 16(3), 429–455.

Ceron

(2014) Gamson rule not for all: Patterns of portfolio allocation among Italian party factions. European Journal of Political Research 53(1): 180–199.

Däubler

Benoit

Mikhaylov

(2012) Natural sentences as valid units for coded political texts. British Journal of Political Science 42(4): 937–951.

Dür

(2008) Measuring interest group influence in the EU: A note on methodology. European Union Politics 9(4): 559–576.

Gabel

Hix

(2002) Defining the EU political space: An empirical study of the European elections manifestos, 1979–1999. Comparative Political Studies 35(8): 934–964.

10.

Grimmer

(2010) A Bayesian hierarchical topic model for political texts: Measuring expressed agendas in senate press releases. Political Analysis 18(1): 1–35.

11.

Grimmer

Stewart

(2013) Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis 21(3): 267–297.

12.

Hix

Noury

Roland

(2005) Power to the parties: Cohesion and competition in the European parliament, 1979–2001. British Journal of Political Science 35(2): 209–234.

13.

Klüver

(2009) Measuring interest group influence using quantitative text analysis. European Union Politics 10(4): 535–549.

14.

Klüver

(2011) The contextual nature of lobbying: Explaining lobbying success in the European Union. European Union Politics 12(4): 483–506.

15.

Klüver

(2013) Lobbying in the European Union: Interest groups, lobbying coalitions and policy change, Oxford: Oxford University Press.

16.

Klüver H and Mahoney C (2015) Measuring interest group framing strategies in public policy debates. Journal of Public Policy. Epub ahead of print: 5 January 2015. DOI: 10.1017/S0143814X14000294.

17.

Laver

Benoit

Garry

(2003) Extracting policy positions from political texts using word as data. American Political Science Review 97(2): 311–331.

18.

Laver

Garry

(2000) Estimating policy positions from political texts. American Journal of Political Science 44(3): 619–634.

19.

Mikhaylov

Laver

Benoit

(2012) Coder reliability and misclassification in the human coding of party manifestos. Political Analysis 20(1): 78–91.

20.

Proksch

Slapin

(2010) Position taking in European Parliament speeches. British Journal of Political Science 40(3): 587–611.

21.

Slapin

Proksch

(2008) A scaling model for estimating time series policy positions from texts. American Journal of Political Science 52(8): 705–722.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.24 MB

0.00 MB