From newspapers to classrooms: Adapting an annotation scheme and automatic classifiers to mixed-quality argumentative school essays

3.1. Summary of the scheme

The basis of our work is an annotation scheme that was formulated for the AM task in a survey by Peldszus and Stede. ¹⁸ Its main inspiration is the work of Freeman, ⁷ which views argumentation as a dialectical exchange: a monological text can be understood as an interaction between the author (proponent) and an imagined opponent whose arguments are being taken into consideration (and then, usually, are being refuted). The diagram technique uses a set of building blocks that can be combined to yield representations of arguments unfolding in texts. Figure 3 shows the (common) cases of applying the support relation, and Figure 4 shows different kinds of attacks that the scheme accounts for. Every numbered node represents an ADU from a text, that is, a statement that functions as a claim or premise in an argument. Circle nodes indicate that the unit represents the viewpoint of the proponent, while box nodes denote the viewpoint of the opponent. The two types of arrows are the same as in the diagram in Figure 2 (support vs. attack).

OPEN IN VIEWER

Figure 3.

“Building blocks” of argument structure (1): cases of support. ¹⁸

OPEN IN VIEWER

Figure 4.

“Building blocks” of argument structure (2): cases of attack. ¹⁸

The target structure is specified formally as a tree: there is a single “central claim” for a text, and each premise is related to exactly one other unit (i.e., a divergent argument is ruled out), so that no circles can arise. The rationale was twofold: in empirical application, mentioned below, divergent argument did not seem to be required; and automatic AM can be simplified, since, for example, dependency parsing techniques can be applied (which in fact became a major trend in the field in the early 2020s; see, e.g., Ye and Teufel ¹⁹ ).

Following the publication of the scheme, it was subsequently formulated as a guideline for annotators (codebook), which is publicly available. ²⁰ Also, an annotation tool called “GraPat” was made available (https://github.com/discourse-lab/GraPat), which allows users to conveniently build the trees interactively and stores them in an XML format. ²¹

When annotators apply the scheme to the analysis of a particular text, they essentially form a representation of their own understanding of the argumentative “plan” that the author realized in the text. Specifically, the scheme has been tested on “pro and contra” style newspaper editorials in the Potsdam Commentary Corpus, ²² and informally in student exercises targeting fundraising letters written by various non-governmental organizations. The larger-scale application was in the “argumentative microtext corpus,” a bilingual (German/English) collection of short texts written by students (part 1, Peldszus and Stede ⁵ ) and by crowdworkers (part 2, Skeppstedt et al. ⁶ ). In total, there are 290 texts labeled with argument structure trees. The first part of the corpus, in addition, has annotations on enthymemes and discourse structure, and there are versions in other languages (Italian, Russian, and Persian) (All information about the microtext corpus is available here: https://angcl.ling.uni-potsdam.de/resources/argmicro.html).

For illustration, Figure 5 shows a GraPat screenshot of a tree from part 2 of the corpus. (Notice that GraPat adds an artificial “label” node (c1–c4) to every edge of the tree, and it visualizes rebut/undercut arrows with a square arrowhead instead of a circle one.)

OPEN IN VIEWER

Figure 5.

A sample argument tree from part 2 of the “microtext” corpus. ⁶ Nodes represent the corresponding textual argumentative discourse units (ADUs). The various elements are explained in the section “The original annotation scheme.”

5.1. Arguing in student essays: Phenomena

For our first phase of qualitative analysis and subsequent annotation, we considered essays of length between 200 and 800, and from these took a random sample of 50 essays. In these texts, we identified the following phenomena that render students’ argumentation different from what we studied and annotated previously. Some of them are indications of a lower text quality, while others (e.g., implicit claims) are common in other kinds of argumentative text, too; they are mentioned here only because they were almost completely absent in the corpora that we have so far applied our scheme to (see above).

Meta-level descriptions.

–

Students explain their writing strategy: “I will now turn to the arguments of the other side.”

–

Students describe their reasoning and judgment: “I think the arguments against fast food are more convincing.”

Inconclusive outcome. Students state they are not ready to take a position: “I think that both sides have good arguments. Everybody should decide themselves.”

Unclear outcome. Students mention arguments, but in the end, the reader cannot discern whether their position is pro or contra.

Repetition. –

Students mention their main claim more than once (e.g., repeat it at the end of the text).

–

Students state the same premise more than once (paraphrases).

Implicit claims. Students provide arguments (premises) but do not explicitly state the implied claim: “Fast food is inexpensive. And the quality is usually acceptable.” ⇒ “Fast food is good.”

Arguments for multiple claims. Students use a statement to support more than one claim (which violates tree structure constraints).

5.2. Revised annotation scheme

The phenomena listed above have several consequences for the redesign of our annotation scheme:

Non-argumentative material in essays should also be classified for its purpose, so that essay quality can be assessed, and detailed feedback can be produced.

5.3. Annotation procedure

Our new annotation guidelines divide the process into four separate steps, beginning with the ascription of an overall argumentative “constellation” to the text. The other fundamental change is the introduction of a step of assigning fine-grained AC types to the individual units.

5.3.1. Step 1: Determine the constellation

The annotator reads the entire text, determines the position that the author takes on the prompt question, and on this basis selects which of the four possible argumentative constellations is present in the text. This then affects the annotation of AC types, which we address further in Step 3 below. The set of argumentative constellations is as follows (see also Figure 7):

One-sided: This is the “traditional” constellation as described above in the section “original annotation scheme.” The author takes a position that is spelled out in a “central thesis” (cth). There are supporting pro arguments for cth, possibly with occasional counter-arguments (con) given.

Deliberative: The author takes a position, but the opposing view is also explicitly formulated (e.g., “One could also think that…”) and supported by a constellation of arguments. Then, we annotate two theses as “thesis 1” and “thesis 2” (th1 and th2), which have their own argument structures. The author’s position is later reiterated (either explicitly or implicitly) and labeled as cth. The units th1 and th2 are connected to it by a support and an attack relation, respectively.

Undecided: This is similar to “deliberative,” but the author in the end does not take a clear position. Thus, the cth represents neither a pro nor a con voice, but an “open” one. (e.g., “Everybody should make up their own mind.”) The text provides separate argument constellations for the pro and con voice, whose theses are th1 and th2, respectively. The (linearly) first-mentioned position is considered th1 and (arbitrarily) labeled as pro; the other one is th2, labeled as con. If the author explicitly states that they are not taking a decision on the matter, this is labeled as the cth and becomes the root of the overall single tree; th1 and th2 are in support relation to cth.

Unclear: The annotator cannot discern what position is taken, and the text is too disorganized for a th1/th2 analysis; none of the above-mentioned constellations can be identified. In this case, the annotator builds a set of local trees with thesis root nodes, which are all (arbitrarily) labeled as pro. A local argumentation may also switch sides, so that con arguments can arise. Importantly, there is no overall coherence to be achieved by the pro/con labeling (because the text does not warrant it) (Texts with this constellation are extremely rare in the corpus, and for the sake of noise reduction, we will ignore them in the later stage of automatically building an argument structure for the text.).

OPEN IN VIEWER

Figure 7.

Four argumentative constellations in our corpus of school essays.

5.4. IAA study: AC types

To validate the applicability of the annotation scheme to our essay corpus, we conducted an IAA study. For now, we restrict this to the (central) step of AC type recognition, which will also be the focus of our computational experiments in the next section.

Two trained annotators labeled 30 texts that were randomly chosen from our initial set of 50. In order to focus the IAA calculation firmly on type identification, the segmentation and explicitation of implicit theses (if applicable) were beforehand agreed upon by the two annotators. The total number of segments for the study is 788, and Table 2 shows the AC-type distribution in the two annotations. We calculated the overall Cohen’s κ as 0.79, and performed two ablation tests: (i) leaving out the inserted explicitations of theses reduces the κ by a mere 0.004 points. (ii) Additionally, leaving out the most conceptually simple agreements on the AC types pro, con, and concl—that is, reducing the task to its hardest part—yields a κ of 0.47.

We separately calculated the agreement of the AC-type cth that is, whether for each essay both annotators decided on the same segment as the most fundamental annotation decision. This is the case for 76% of the essays.

After the IAA calculation, the two annotators resolved the mismatches in their annotations in an adjudication step and produced a “gold standard” set, which was subsequently extended by another 20 texts (labeled by one annotator). The result is a first subcorpus of 50 essays with gold annotations, which enabled us to evaluate the automatic approach, to be described next.

In recent years, a growing body of literature has looked into different ways of applying LLMs to AM or its sub-tasks ³⁰ : conduct a systematic evaluation of multiple LLMs in both AM and argument generation tasks. Among others, the AM tasks include binary claim and evidence detection as well as multiclass detection of different types of evidence. They focus on zero-shot and few-shot prompting approaches and observe that, given as few as 10 labeled examples in few-shot prompting, small-to-medium-sized LLMs (up to 20 billion parameters) can approach the performance of supervised models trained on 500 samples.

Ajjour and Wachsmuth ³¹ explore using different LLMs and few-shot prompting to classify argumentative stance, that is, to recognize the polarity of an argument with respect to a thesis. Notably, they particularly zoom in on strategies for selecting the best example data for few-shot prompting. They find that using data instances with the same stance as the target instance to serve as few-shot examples is consistently more effective.

As an alternative to prompting, LLMs can also be fine-tuned on labeled data ³² : they also successfully target canonical AM sub-tasks such as argument type classification and argument relation extraction on established datasets. Their approach is based on finetuning different LLMs, which is made feasible despite the models’ sizes by parameter-efficient finetuning strategies such as QLoRA. ³³ A comparable approach is taken by Kawarada et al. ³⁴ They frame AM as a task of Translation between Augmented Natural Language, ³⁵ where the model is instructed to “translate” unannotated input texts into texts containing argument annotations. They achieve strong results by efficiently finetuning small LLMs.

The experiments detailed below describe our first step toward performing computational AM on our novel dataset. At present, we focus on using LLMs in zero-shot and few-shot settings. We leave approaches based on finetuning LLMs to future work.

6.2. Data pre-processing

Our current type inventory, detailed above, consists of 10 labels in total. This includes types which are strictly speaking not considered argumentative, such as bg-world or struct. Since they do not play a role in our present core task of recognizing argument relations (recall that we reserve feedback production for future work), their classification is currently of secondary importance in our overall scheme. We therefore map the labels bg-author, bg-world, struct, concl, and other to a single label n-a, for “non-argumentative.” This yields a resulting inventory of six AC labels. Figure 8 shows the distribution of these six labels among the 50 gold-labeled essays.

OPEN IN VIEWER

Figure 8.

Distribution of the six argument component (AC) labels among the 50 gold-labeled essays.

Unsurprisingly, the labels denoting claims (th1, th2, and cth) are significantly fewer than those denoting arguments (pro and con), as a single claim can be supported by multiple arguments. The abundance of the label n-a can mostly be attributed to personal anecdotes and background information, which students tend to include in their essays.

For our experiments, we have chosen the following three popular LLM models of different sizes:

Instruction fine-tuned LLama 3.1 with 8 billion parameters (https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), ³⁶ referred to as “Llama8” hereafter.

Both Llama models are instruction fine-tuned models, which means that in addition to their extensive pre-training for general language modeling, they have been further fine-tuned on human-generated prompt-response pairs for various tasks to allow the model to optimize human-like output behavior. Differences between the two models mainly relate to size (8 billion vs. 70 billion parameters). Qwen32 is trained to perform well in reasoning and instruction-following tasks, using an explicit thinking mode to support logical reasoning. In terms of model size, at 32 billion parameters, it is approximately halfway between the two Llama models.

These models are attractive since they are open-source and easy to access. In particular, they can be accessed through the Groq API platform (https://groq.com/). This obviates the need for high-capacity local hardware, which could be a limiting factor if the models were to be hosted and run locally. Moreover, all three models are multilingual and offer reliable support for German. In our experiments, all models are given exactly the same prompt texts.

6.4. Prompt engineering

As discussed above, the human annotator approaches the analysis of the essay by first determining the overall argumentative constellation. In particular, the functional meaning of various AC-type labels can be affected by whether the author of an essay argues in favor of one side of the discussion or whether she presents both sides equally without taking a stance. Hereafter, we refer to the first case as a decided constellation, which covers the argumentative constellations “one-sided” and “deliberative” as defined above; we refer to the second case as an undecided one, covering the constellations “undecided” and “unclear.” To recall, in a decided essay, the label cth is given to a segment that describes the author’s central stance on the discussion topic, while th1 and th2 are assigned to segments that describe positions which concord with or oppose the author’s stance, respectively. By contrast, in an undecided essay, cth is used for segments which explicitly express the author’s unwillingness to take a clear position, while th1 and th2 are, respectively, given to segments which express the first and second named position.

Literature has established role-playing as a prominent prompt engineering method to encourage human-like behavior from LLMs.^28,29 Therefore, in the system prompt, we ask the model to assume the role of a teacher of German for 9th-grade school classes before explaining the context of the essay writing task. Following the recommended procedure for human annotators, we ask the model to first perform the binary prediction of the overall constellation of the essay (decided vs. undecided). We then present the six AC-type labels and provide a concise definition for each label, making references to either the overall constellation where needed (see above). Finally, the model is shown the segmented target essay and prompted to assign one out of the six possible labels to each segment, following a predefined output format. An example of the full-length, German-language prompt text is given in the Appendix.

In the zero-shot prompt scenario, the prompt text described above is all that we feed to the model. In the few-shot scenario, we randomly select two gold-labeled essays, one with a decided and one with an undecided constellation. Hereafter, these two essays are referred to as “demo essays.” We feed the model each demo essay paired with the desired output text to form the basis for ICL.

6.5. Evaluation method

We use regex to parse the LLM’s output text and to extract the predicted AC-type labels. In our scheme, the prediction of the overall constellation is first and foremost an intermediate step toward classification of AC-type labels rather than a goal of its own. Nonetheless, since our gold essays contain annotations for constellations, which can easily be mapped to the binary labels that the models are instructed to output (i.e., decided and undecided), we have also extracted them for evaluation.

Models have not always provided valid AC-type output for the target essay. We consider the output invalid if (a) it uses labels other than the ones from our inventory or if (b) it produces more or fewer labels than the number of segments in the essay. Invalid outputs are excluded from evaluation.

The essay-level binary constellation predictions are assessed in terms of accuracy. The segment-level AC-type labels are evaluated against our gold labels using standard metrics for multiclass classification, including accuracy and macro-averaged F1. We perform two sets of evaluation for AC-type prediction:

In the “standard” setting, all pairs of predicted and gold labels are used for the calculation of the metrics.

The motivation for the “no-concl” setting is as follows: according to our annotation scheme, both the conclusion and the cth may express the author’s core position. It is a convention within our annotation scheme that we label such a segment as concl rather than cth if it occurs toward the end of the essay and if it is a repetition of the cth. We therefore believe that punishing the model for confusing the labels n-a (to which concl is mapped) and cth on these segments unfairly underrates the model’s performance. Hence, the metrics in the “no-concl” setting show the evaluation of model performance without these segments.

6.6. Results and Discussion

The models’ performance in terms of our chosen classification metrics is summarized in Table 6. For AC-type prediction, we show accuracy and macro-averaged precision, recall, and F1, for each of the three models, two prompt conditions (zero-shot vs. few-shot), and two evaluation settings (“standard” and “no-concl”). For the preliminary step of essay-level constellation prediction, we report accuracy for each model-prompt pair.

Most of the model-prompt combinations have produced valid and extractable constellation predictions for all or all but one test essays, so that hardly any have had to be excluded when we calculate the prediction accuracy. The only exception is Llama8 in the zero-shot scenario, where the output for 29 out of 50 essays does not use the designated labels that the model has been instructed to use and is therefore considered invalid. Overall, although the prediction of the argumentative constellation is a binary task, the accuracy values of 0.83 at most show that it is nevertheless challenging.

With respect to our main task, AC-type classification, several observations are worth discussing: first, the smaller Llama model, Llama8, struggles heavily with the task. Not only has it produced large amounts of invalid output that need exclusion from evaluation, but the metrics based on the remaining valid output show a poor performance in terms of averaged F1. This holds for both prompt conditions. Although the few-shot scenario does result in improved performance, with invalid output for 20 out of 50 essays and an F1 score of slightly above 0.5, it is nonetheless subpar. This suggests that the complexity of the task poses a great challenge to smaller LLMs such as Llama8.

Second, all models have produced invalid output under both prompt conditions, even the overall best-performing Llama70 model in the few-shot scenario. After examining the invalid LLM output texts, we find that they are due to different reasons: for Llama8, all instances of invalid output are due to the model generating made-up labels that are not in our inventory, that is, not the labels which the model has been instructed to use. Examples include Pro_Argument_1 or Con_These. For Llama70, in contrast, all invalid output is due to the model outputting multiple sets of analysis for a single input essay and discussing them as different labeling options. Our regex-based output parser has extracted AC-type labels from all analyses, which results in significantly more extracted labels than there are segments in the input essay and therefore marks the output as invalid. To better deal with this behavior, one could either prompt the model to output only a single, final set of analysis or one could instruct the model to use specific separator tokens between the different sets of labeling so that they can be automatically recognized as such. We leave this to future work.

Finally, Qwen32 has proved to be the best model in the zero-shot prompt condition, which indicates that our task benefits from models with a focus on reasoning. Despite being half the size of Llama70, it also comes close to Llama70 in the few-shot scenario. This echoes ³⁰ finding that model size does not necessarily determine performance success in complex tasks. It should be noted, however, that upon closer examination, we find that Qwen32 struggles particularly with the recognition of the cth, which we discuss below.

To gain more insight into model performance on individual AC types, Figure 9 shows the label-specific F1 scores for the best-performing Llama70 model in either evaluation setting.

OPEN IN VIEWER

Figure 9.

By class F1 scores of the best-performing Llama70 model in either evaluation setting.

While the model has given particularly strong results on the recognition of pro and contra arguments, its performance on the cth and th1 is comparatively low. Given that in the overall scheme of our AM approach, the cth will be the root of the analysis, this is unfortunate and requires improvement in further studies. We do observe, however, a significantly higher F1 score in the “no-concl” setting. More precisely, the model’s precision score for cth is 0.554 in the “standard” and 0.795 in the “no-concl” setting. As shown by the prediction-normalized confusion matrices (For a given class c , the prediction-normalized confusion matrix shows the distribution of true labels among all of the model’s predictions for c . Therefore, in Figure 10, all values within a column add up to 1.) in Figure 10, in the “standard” evaluation setting, more than one-third of the model’s predictions for the label cth are true members of the class n-a. In the “no-concl” setting, when segments originally labeled as concl in the gold data are excluded from evaluation, this value falls to only 0.05. This indicates that much of the confusion between the classes cth and n-a indeed stems from the model predicting cth for segments that are originally labeled as concl. We plan to address this issue in future work by explicitly instructing models to pay more attention to a segment’s positions within an essay.

OPEN IN VIEWER

Figure 10.

Prediction-normalized confusion matrices (values within a column add up to 1) for the best-performing Llama70 model in the “standard” (left) versus the “no-concl” (right) setting.

Finally, Figure 11 shows the performance of all models specifically on the cth class. While Qwen32 comes close to Llama70 in the few-shot scenario in terms of the macro-average F1 measure (see Table 6), its performance on cth is inferior to Llama70 by approximately 10 points. Conversely, while in the zero-shot scenario Qwen32 clearly beats both Llama models on macro-average F1, its advantage over Llama70 specifically on cth is only less than three points. This suggests that, despite its advantages, Qwen32 is less suitable for our overall AM scheme, given the importance of the correct recognition of the cth.

OPEN IN VIEWER

Figure 11.

F1 scores by all models for the class cth in either evaluation setting.