Argument Mining for Organizational Research: A Computer-Aided Analysis of Organizational Talk

Step 1: Data Collection and Preprocessing

In order to use argument mining effectively, researchers must collect digital sources that are likely to contain a considerable amount of argumentative text units. This data can, for instance, reflect the internal perspectives of the organization or provide insights into the organization–environment relationship based on public communication. Data and documents from websites can be acquired, for example, via web scraping (Haans & Mertens, 2024). Precompiled, carefully curated research corpora such as the ParLawSpeech dataset contain machine-readable full texts of bills, laws, and plenary speeches from eight European parliaments (Schwalbach et al., 2025) that can provide valuable contexts from governmental and administrative processes. Social media conversations from specialized platforms like Reddit for the digital avant-garde, X (formerly known as Twitter) for public opinion leaders, or LinkedIn for trends among business professionals can provide further rich and diverse sources for argument mining. Product reviews from online shops or company reviews from job portals would be examples of sources already studied intensively in organization research.

For NLP applications on longer documents, some preprocessing is usually required to segment them into smaller, semantically coherent sub-units. For argument mining, either paragraphs or sentences with sliding windows of up to two context sentences would be preferred (Rieger et al., 2024, p. 81). The recommended input of segmented data to a BERT-like classifier for Task A would then look like “<target sentence>[SEP]<context before>[SEP]<context after>” where [SEP] represents the special separator token which transformer models use to distinguish between different parts of the input (Rieger et al., 2024). Presenting the sentences in this order ensures that the model pays attention to the first sentence as the primary classification target and that context sentences are recognized as additional information only. In case the input sequence exceeds the maximum length (typically 512 tokens for BERT-like models), the order ensures that the least important part gets truncated first. Sentence splitting can be performed with libraries such as NLTK (Bird et al., 2009) and spaCy (Montani et al., 2023) for Python, or udpipe for R (Wijffels, 2017). Paragraph splitting can be achieved with document-specific regular expressions. Depending on the input documents, further preprocessing steps to reduce noise in the data might be necessary, for example, to de-hyphenate words or remove header/footer text extracted from PDF documents. Step 1 results in a representation of the entire collection as short texts, for example, micro-posts from social media, paragraphs from documents, or sentences with surrounding context that serve as the analysis unit for argument mining.

Step 2: Label Candidate Sampling for Target Data Adaptation

To ensure the validity of the entire process, the argument mining workflow needs manually labeled data from the target dataset that serves as the ground truth for training and validation of the machine classifiers. The problem is that argumentative text units can be relatively rare in many real-world data sets. Random samples usually contain a significant share of nonargumentative units, making it difficult to annotate large enough sets of pro- and contra-arguments for supervised machine learning. To circumvent this problem, we suggest employing zero-/few-shot learning as an effective transfer learning method (cf. Laurer et al., 2024) for a stratified sampling of label candidates. The basic idea is to train a candidate model on suitable support data for the respective task, stance detection, or aspect sequence tagging. Alternatively, researchers can rely on prompting a generative LLM such as Llama or GPT-5 in a zero-shot manner, that is, without any labeled examples, or a few-shot manner by attaching a small number of labeled examples to the prompt (Farjam et al., 2025). Predictions from such an initially trained/prompted model on texts from the target dataset are then used as intermediate labels that inform a stratified, balanced sampling of arguments referring to certain stances and/or aspects.

BERT-like models can be pretrained as candidate models on suitable argument mining datasets previously published by NLP researchers. As these publicly available datasets usually vary significantly in their language features from the target data, models trained on such support datasets do not generalize easily to applications in new domain contexts. Therefore, they should be used only as an entry point for adaptation to the target data. If some manually coded data is available from previous iterations of training data annotation (cf. Step 3), the procedure to train a candidate model for sampling further argument candidates can be altered to a setup where, in addition to the external dataset, the annotated target data is fed into the training process, too.

Fine-tuning of a PLM can be performed as multitask learning by training on two or more datasets in parallel or as intermediate fine-tuning, first on the support dataset and then on the actual target data. According to Weller et al. (2022), multitask learning is preferred when the target data is smaller than the support data. For the stance detection task, we recommend using the UKP Sentential Argument Mining corpus (UKP SAM) (Stab et al., 2018) with ca. 25,000 argumentative English sentences from web pages about eight controversial topics, the X-stance dataset (Vamvas & Sennrich, 2020) with 67,000 multilingual comments on more than 150 political issues from a Swiss election candidate website, or the multilingual XArgMining dataset with around 72,000 English-language sentences and their corresponding machine translations (Toledo-Ronen, et al. 2020) as support datasets. If multiple issues are contained in the support data, the input to the classifier (cf. Step 1) should be prepended as follows: “<issue>[SEP]<input>.” This way, the model can learn issue-specific associations between semantic features and stance labels while profiting from a large and diverse set of ways to express argumentative stances.

For the aspect sequence tagging task, we recommend the argument aspect corpus (AAC) by Ruckdeschel and Wiedemann (2022) as a support dataset providing ca. 5,000 argumentative sentences from four topics labeled with issue-specific aspect categories. As only four topics are covered, new studies will likely require the alteration of the value categories of the support dataset. To be able to find candidates for new value categories, the model target output can be altered from sequence tagging to a natural language inference (NLI) task (Laurer et al., 2024). For NLI, a BERT-like model is given two texts: a premise and a hypothesis. The model's job is to determine the relationship between them, classifying the hypothesis as either true (entailment), false (contradiction), or undetermined (neutral), given the premise. Pretrained NLI models such as the one published by Laurer et al. can be used for zero-shot labeling when presented with the input text as premise and the potential class label formulated as a hypothesis, for example, “<input>[SEP]This argument refers to the value of safety to support its stance on nuclear energy.” This way, PLMs can provide higher-than-chance accurate predictions for semantically expressive labels without any training data solely based on their internal knowledge representations. Again, predicted intermediate labels can be used to sample candidates for each aspect evenly from the target data. For a description of fine-tuning a transformer model with selected support data and/or manually labeled examples, see Step 4.

Step 3: Manual Annotation

To adapt candidate models to the target data, we must further train them on manually annotated samples. Training data must be labeled for both the stance detection and aspect sequence tagging tasks. First, an initial codebook needs to be developed that comprehensively describes how to detect stances and the intended value categories (cf. Barberá et al., 2021). Based on this codebook, human coders closely review and label the candidates for stance and aspect categories sampled during the previous step. After a labeling pretest, the codebook can be revised to better fit the data if deemed necessary, including removing or adding categories to the aspect label set and adding representative examples from the target data. Once the codebook is fixed, the complete candidate sample can be manually annotated. Stances are coded on entire text sequences. Value references are coded as token spans. We recommend using annotation tools that particularly facilitate the task of sequence tagging and multiuser management, such as Label Studio (Tkachenko et al., 2025) or doccano (Nakayama et al., 2018). These open source tools provide intuitive user interfaces and can be configured to perform complex annotation tasks. At least two coders should code each argumentative unit in the first rounds to track inter-rater reliability (IRR). After each round, the IRR, such as Krippendorff's alpha or Cohen's kappa, should be determined to check whether the coders have developed a sufficient mutual understanding of the categories. For IRR calculation of Task B, we recommend using a relaxed metric that already considers overlapping or the presence of spans with a certain category in an augment unit as agreement instead of requiring perfect matches of coded spans. Disagreements and unclear cases can be marked during the process and discussed after each round to reach a consensus decision and further clarify the category definitions. While a mutual understanding of argument stance is relatively straightforward to establish, shaping a mutual understanding of more abstract aspect references can become quite cumbersome in the face of empirical data. Further, it is challenging to reliably annotate the beginnings and ends of text spans for aspect references. One rule of thumb is that the shortest possible substring that allows us to recognize the aspect should be coded.

For the first round(s), it can be expected that the IRR is too low and, therefore, the training data is too noisy and incoherent for a valid and reliable model. To achieve acceptable results, several rounds of codebook refinement and coder training can be necessary. For this, Step 2 can be repeated by extending the support training dataset with manually annotated target data where most annotators agree on the label decision. Barberá et al. (2021) state that annotating more data with experienced coders is preferred over coding fewer samples with multiple coders. Consequently, once the IRR reaches a satisfactory level (e.g., substantial agreement in terms of Cohen's κ > 0.6), the remaining label candidates can be coded by one coder per sample. The final ground truth datasets for the classification/ tagging task can be compiled from annotations with majority agreement from multiple coders during the first rounds, together with the single-coded examples from the last round. How many text units are necessary for a well-performing classifier depends mainly on the complexity of the categories and the heterogeneity of the target dataset. As a rule of thumb, we observed a saturating classification performance at about 100 examples per category in our transfer learning setups with PLMs and support datasets.

Step 4: Stance Detection and Aspect Sequence Tagging

The manually annotated data serves as input for training a text classification model for stance detection and a sequence tagging model for aspect text spans. Transformer-based models require almost no text preprocessing or feature extraction, which simplifies their application drastically compared to previous classifiers like SVM (cf. Hickman et al., 2022). For social media data, we recommend removing URLs. Genre-typical text characteristics such as @-mentions of fellow users and hashtags for flagging thematic content can be retained if they carry meaningful semantics for the classification task.

After preprocessing, the labeled datasets should be split into a train, dev, and test set of common sizes, such as 70%, 10%, and 20%. The dev set can be used to determine optimal hyperparameters for fine-tuning the PLM. ¹ The selection of the best PLMs for a specific project mainly depends on the language of the target data and the computing capacities available to the researcher. The DeBERTaV3 model by Microsoft (He et al., 2023) or ModernBERT (Warner et al., 2025) would be a good choice for English-only datasets. For cross-language transfer, as in our example case (cf. Section 4), the multilingual version mDeBERTaV3 (Warner et al., 2025), the XLM-RoBERTa model (Conneau et al., 2020), or the multilingual version of ModernBERT would be suitable. Although one can expect moderate losses of classification accuracy due to cross-language transfer (cf. Lauscher et al., 2020), their overall performance would still be good enough to act as a prefilter for annotation candidates.

For training the final models for Task A and B, we recommend the multi-task learning paradigm, which simultaneously trains on the public argument mining support data and the newly annotated target data, as this latter dataset would generally be the smaller one. In this setup, the parameter weights of a single transformer base model are shared across multiple tasks. The base embedding model feeds multiple classifier heads placed as parallel final layers on top of it. The simultaneous training of different tasks not only feeds additional knowledge to the classifier but also serves as a regularization process to effectively prevent overfitting of the model to the target dataset. When predicting new data, one simply extracts the labels for the target task and discards the labels of the support tasks. Depending on the dataset sizes, it can be advised to use task-specific loss weights to put more emphasis on the target task during the training process. Standard metrics such as recall, precision, and the F1-score, or Matthew's correlation, should be used to evaluate the final classifier performance. F1-scores way below 70% usually require further optimization either through annotation of more training data and/or optimization of hyperparameters. Otherwise, there is an increased risk of spurious results in downstream tasks due to too much noise in the argument mining results. For the aspect sequence tagging task, Ruckdeschel and Wiedemann (2022) advised altering the evaluation scheme from the token to the sentence level since achieving a substantial overlap between ground truth spans and predicted spans is unnecessarily difficult. More important than an exact matching of the beginning and end of a text span is whether the automatic process correctly predicts the presence of an aspect category in an argument.

Step 5: Final Prediction and Analysis

The trained models for stance detection and aspect sequence tagging can be used in a final step to predict stances and aspects for the entire target dataset, resulting in a tabular data structure similar to the example presented in Table 2. This table can be augmented by the actual text spans for each identified aspect reference. Before further processing the argument mining results, they should undergo a careful validation process, as usually advised in computer-aided text analysis on big data sources (e.g., Sen et al., 2021; Tonidandel et al., 2018). Thankfully, checking for construct validity, content validity, discriminant, and external validity (Short et al., 2010) is a straightforward process due to the opportunity to closely read samples of texts classified as either pro- or con-arguments, to look at frequent text spans for distinct value references, and to research the backgrounds of the users or organizations they originate from.

Once researchers are confident that the results from the process are sufficiently valid and complete, any aggregation for descriptive statistics or statistical modeling of more complex relationships in the data can be performed. In longitudinal designs, such aggregations allow researchers to identify shifts in the distribution of predicted stance and aspect labels across time periods. Additional arbitrary variables can be added to enrich the dataset, such as grouping selected authors to organizations or entire societal fields. For individual organizations, for instance, one can observe the development of stances and aspect references over time. For characterizing organization–environment relationships, it can be interesting to contrast this measurement with approaches that seek to explain which specific aspects can predict stances in the entire data population and how this changes over time. With network analytic approaches, communities and central actors that push pro- and contra-stances, their connectivity and degree of polarization could be determined (cf. Haunss & Hollway, 2023). Depending on the data source, interaction patterns of organizations with stakeholders and other audiences can be investigated by looking at references to previous arguments in dialogical conversations.

Eventually, quantitative argument analysis should be accompanied by a close reading of data samples, for example, from different label combinations, time spans, or actor groups, to check for face validity. Going further into mixed-method research, the approach allows sampling representative example data points for standpoints, salient aspects, time periods, and arbitrary author groups based on the quantitative results. These subsets can be exported to specific data formats, such as REFI-QDA, for further qualitative processing in QDA software (Evers et al., 2020).

Surveying Stances in the Organizational Environment

The resurrection of the nuclear energy discourse in German media and politics in 2022 was accompanied and, as our data shows in Figure 3, already preceded by intense social media activities and a significantly increased polarization on the issue. As a first approach to the data, we conduct a descriptive statistical analysis of the pro- and contra-arguments to track the majority opinions and their shifts over time. The resulting temporal aggregates of predicted stance labels make shifts in majority positions directly visible in the data, which then can be related to external events or broader developments.

OPEN IN VIEWER

Figure 3.

Longitudinal development of the nuclear energy debate.

This basic analysis already reveals highly intriguing developments that characterize the course of the debate. The tweet frequency graph in Figure 3 (gray area chart) highlights points of intensified debate in 2011, the year of the Fukushima nuclear disaster in Japan. Phasing out nuclear power was one of the primary goals of Germany's Green Party from the outset when it was founded in the early 1980s. After years of political struggle, even liberals and conservatives were ready to abandon nuclear power in light of the reactor catastrophe (cf. Haunss & Hollway, 2023). In the following years, shorter peaks in debate intensity can often be linked to events such as safety issues of single reactors and national or global discussions on future energy production in the context of climate change. The overall frequency of the debate has slowly but steadily increased since 2018, followed by a sharp surge in tweet activity from December 2020 onward. We hypothesize that much of the initial rise can be linked to growing protest movements against fossil fuel use in Germany's energy mix, such as “Ende Gelände,” which opposed new lignite mines, and “Fridays for Future,” which called for immediate climate action and organized its first global student strike in March 2019 (Marquardt, 2020). These protests sparked a broader societal debate about the future of Germany's energy production. The drastic increase in tweet activity from 2021 onward may also be partly influenced by the COVID-19 pandemic, which likely affected overall social media usage and, indirectly, specific topic trends.

When we contrast the frequency development with the development of pro-, contra-, and non-argumentative statements about nuclear energy, findings become even more intriguing. Over the study period, the analysis reveals two major shifts in the majority stances. The first occurred in 2011, when a slight pro-nuclear majority gave way to a consistent contra-majority, coinciding with the Fukushima disaster. This balance shifts again after 2017, as pro-nuclear arguments rise significantly, mainly at the expense of neutral statements, and form a clear pro-majority by mid-2018. Notably, the surge in activity from 2021 onward does not alter this distribution of argument stances. These patterns suggest an early polarization of the Twitter debate, likely driven by global climate policy discussions and related social controversies. Looking at the distribution of highly active accounts over time (not shown here) reveals that many new actors, primarily pro-nuclear and politically motivated, have entered the debate during the last years of the study period.

These findings illustrate the vital contribution that systematic observation of the environment can make for organizations, especially when operating in a specific issue area, as they must remain responsive to changing conditions (Brunsson, 1986; Luyckx & Janssens, 2016; Nyberg et al., 2020). The Green Party, for instance, which has long championed the phase-out of nuclear energy, could have benefited from an early warning that by 2018, they were no longer in line with the majority of social media on this issue.

Shifting Patterns of Value-Based Justifications

To advance organizational research, it is necessary to move beyond analyzing stance distributions and examine the underlying normative structures that give rise to argumentative positions. For organizations, it is essential not only to track shifts in public opinion on issues relevant to them but also to understand whether the underlying values they embody are changing and whether such changes could threaten the organization itself by eroding stakeholder support and societal legitimacy (Brunsson, 1986; Inglehart, 2015; Schwartz, 1992; Welzel, 2009). To address this, we examine the distribution of social values in relation to argumentative stances. Specifically, we ask to what extent references to specific values explain argumentative stances and how these relations change over time, potentially indicating broader shifts in societal value attribution.

We conduct a binary logistic regression analysis to examine the influence of referenced social values on argumentative stance. To capture temporal dynamics, we divide the dataset into three phases, informed by the descriptive statistical trends: the years 2011–2016, characterized by a clear contra-majority; 2017–2020, marked by contestation between pro- and contra-positions; and 2021–2022, during which a pro-stance stabilizes as the new hegemonic position in the Twitter discourse. As illustrated in Figure 4, the analysis reveals that roughly half of all argument aspects are strongly associated with either stance. Reactor safety and nuclear weapons emerge as key concerns for anti-nuclear arguments, while technological innovation and climate action are the strongest predictors of pro-nuclear positions. The dominance of specific values is linked to organizational pressure to conform and respond appropriately to environmental expectations. In 2011, for example, political organizations could not simply position themselves against reactor safety. Conversely, since 2017, it has become increasingly impossible to ignore the issue of climate protection in the debate on nuclear energy.

OPEN IN VIEWER

Figure 4.

Logistic regression of argument stances and justifying values.

More insights emerge from significant coefficient shifts across the different phases. For example, references to sustainability initially supported pro-nuclear positions in the early debate but later transformed into a clear counter-argument. To some extent, this shift can be explained by the dispute about changes in the EU taxonomy that 2022 declared nuclear energy as a sustainable power source (cf. Egres & Sarlós, 2024). In Germany, this was heavily criticized by the government and became a heated debate on Twitter. Energy autonomy and independence from other countries remained a consistent component of pro-nuclear arguments throughout the entire period. A qualitative review suggests that recent contra-arguments emphasize the geopolitical risks of uranium imports, particularly from unreliable partners such as Russia. In contrast, health-related concerns, previously central to contra-arguments, have shifted to the point where they now support pro-nuclear stances, as nuclear power generation is no longer considered harmful due to technological advances.

Organizations need to observe how their values align with those of their environment and if the latter's values are suspected to change. The Green Party, for instance, justifies its anti-nuclear stance mainly with references to safety issues, high costs, and nuclear waste (cf. Figure 5), premises that are increasingly contested by pro-nuclear users and whose prevalence declined over the study period. In these cases, organizations must determine whether these shifts are strategically relevant, requiring adjustments in their communication with stakeholders or their environment, or even informing their internal decision-making processes.

OPEN IN VIEWER

Figure 5.

Sankey diagram of the stances and values of the Green Party and its commentators.

(Dis-)Alignment of Organizational Talk and Environmental Interaction

Before making strategic adjustments, organizations may first engage in dialogue with their environment to better assess the stability of prevailing stances and values. Social media provides a valuable window into such organization–environment interactions. Twitter/X accounts can be categorized based on their association with organizational entities to compare talk from organizations to stances and justifications in their environment.

We examine the argumentative interaction patterns of the Green Party with other users on the platform as an example. Figure 5 displays a subset of the discourse involving all Green Party accounts (left part), that is, party organizations and parliamentarians on the federal and national level, and users replying to their posts (right part) as a Sankey diagram. As expected, the Greens post a large majority of arguments against nuclear energy. More surprisingly, however, is the large share of pro-arguments by users counter-arguing the party's position. Vice versa, only a few pro-arguments uttered by Green Party accounts are attacked with counter-arguments and supported with significantly more pro-arguments instead. Focusing on the stances, most user posts in the discussion reject the critical stance against nuclear energy pushed by the Greens. A further look at the value references reveals informative differences between the two groups. While references to reactor safety and costs mainly justify the party's position, pro-leaning users often respond with references to climate change, fossil fuels, and reliability in their counter-arguments. The fact that climate change is mainly neglected in the Greens’ argumentation against nuclear energy, while it is, at the same time, highlighted by pro-nuclear users, is a salient and surprising pattern, considering climate protection and the transition away from fossil fuels being core values for the party. The patterns we find suggest a possible gap in the party's public communication, which eventually serves as an open gateway for the opinion shift during the study period. We argue that this partially explains why the Green Party and the anti-nuclear movement could not defend their hegemony on Twitter regarding the nuclear energy issue as soon as the climate issue began to take over the public agenda. Moreover, it complements findings from previous studies that determined a shift in the general news media coverage of nuclear energy from framing it as a security risk to a means of achieving climate security (Bohn et al., 2026).