See the forest by looking at the trees: Physical education teachers’ interpersonal style profiles and students’ engagement

Abstract

On the basis of self-determination theory, we aimed to identify students’ perceptions of interpersonal teaching style profiles (i.e. within-teacher combinations of six dimensions of need-supportive and need-thwarting behaviours of autonomy, competence and relatedness) and to examine, through a cross-sectional design, the possible associations between these teaching profiles and students’ behavioural and emotional engagement. Participants were 2065 students (n_teachers = 38) of Physical Education (PE) (M_age = 11.96 ± 1.95; range = 10–16 years; 1042 girls) nested in 98 classrooms from elementary (n = 915) and secondary (n = 1150) Spanish schools. Students filled out questionnaires in a paper and pencil format during the last semester of the school year. A four-cluster solution was found to be the most suitable: (a) a high-low group (i.e. high in perceived need-support and low in need-thwarting), (b) a low-high group (i.e. low in perceived need-support and relatively high in need-thwarting), (c) a low-low group (i.e. low in both need-support and need-thwarting) and (d) a mixed group (i.e. low in autonomy support, high in autonomy-thwarting and relatively modest-to-high in competence and relatedness support and thwarting). A multivariate analysis of variance (MANOVA) indicated significant differences in students’ behavioural and emotional engagement as a function of cluster membership. Post hoc comparisons showed that the high-low group reported the highest levels of engagement and the low-high group the lowest ones. Furthermore, the mixed group scored the second highest level of engagement – higher than the low-low and the low-high group. These results suggest that teachers who fail to support students’ needs for autonomy, competence and relatedness – either by using or not using need-thwarting instructional practices – might undermine students’ engagement.

Keywords

Basic psychological needs positive outcomes self-determination theory teaching styles

Introduction

How could we keep students engaged and actively involved in Physical Education (PE) classes? Research has shown that the interpersonal style adopted by PE teachers is closely associated with students’ active engagement (Curran and Standage, 2017; Leo et al., 2020; Van den Berghe et al., 2016). Based on self-determination theory (SDT; Ryan and Deci, 2000), these studies suggest that PE teachers with a predominantly positive interpersonal style – a style through which they support students’ basic psychological needs – may enhance students’ behavioural and emotional engagement. In contrast, instructors who adopt a need-thwarting interpersonal style may undermine students’ engagement during PE lessons. But to what extent do PE teachers exclusively endorse a need-supportive or need-thwarting style? Given the different repertoires that need-supportive and need-thwarting styles include, PE teachers may use both styles to different degrees in their everyday teaching practice (Van den Berghe et al., 2013). Therefore, we need to further examine the configurations of need-supportive and need-thwarting styles that PE teachers use and how such instructional profiles may explain differences in students’ engagement (Haerens et al., 2018). In this study, based on students’ perceptions, we aimed to address the profile combinations of interpersonal teaching styles by inspecting not only autonomy support and need-thwarting, but also competence and relatedness need-support and autonomy, competence and relatedness-thwarting. By examining each of these six dimensions (i.e. the trees), we aimed to uncover the various profiles (i.e. the forest) to determine the degree to which these profiles could explain students’ differences in behavioural and emotional engagement during PE lessons.

Need-supportive and need-thwarting teaching styles

According to SDT (Ryan and Deci, 2017), a need-supportive teaching style refers to the degree to which PE teachers try to satisfy their students’ need for autonomy (which reflects students’ desire to feel themselves the agent of their actions), competence (which represents students’ propensity to feel effective) and relatedness (which corresponds to their inclination to form close and meaningful social bonds with others). An autonomy supportive teaching style is defined as the use of strategies that encourage students’ initiative-taking. This refers to instructors who provide choices, seek students’ input, show empathy and help their students feel themselves to be agents of their thoughts, feelings and behaviours (e.g. Mitchell et al., 2015; Reeve, 2009, 2015). Accordingly, a competence-supportive teaching style is characterized by providing structure. Competence-supportive PE teachers offer their students suitably challenging activities; they show confidence in their students’ capacity to effectively engage in the activity, they provide clear expectations, scaffold activities adjusted to their students’ skill levels, offer the students help to achieve their goals and give constructive feedback and sincere praise for each effortful attempt (e.g. Jang et al., 2010). Finally, a relatedness-supportive teaching style concerns strategies that are defined by interpersonal involvement (Haerens et al., 2015). Relatedness-supportive PE teachers show affection towards their students, dedicate time and energy to them and help them feel socially connected (Van den Berghe et al., 2015).

However, sometimes PE teachers may also exert more psychological pressure on their students to enforce participation (Van den Berghe et al., 2015), thereby endorsing a more need-thwarting interpersonal style. Specifically, teachers with an autonomy-thwarting teaching style tend to use directive and intimidating behaviours, adopt a position of authority when their students fail to display the desired attributes or behaviours and employ excessive personal control during the supervision of tasks by coercing and pressuring their students to execute a skill in a certain way (e.g. Assor et al., 2004; Reeve, 2009). A more subtle approach has shown that autonomy-thwarting can be expressed either overtly (through punishments, pressuring rewards and using explicitly controlling language like ‘you should’; see Reeve, 2009), or in more covert forms (through attention withdrawal or creating feelings of guilt or shame; see De Meyer et al., 2014). PE teachers with a competence-thwarting teaching style are characterized by confusing, unclear instructions and expectations, resorting to public critical feedback, normative and externally referenced comparisons and activities that prevent students from setting individualized and attainable goals that stimulate personal self-improvement and foster progress (Filak and Sheldon, 2008; Leo et al., 2020). Finally, PE teachers who thwart students’ need for relatedness tend to behave in an unfriendly, rejecting and even hostile manner towards their students; they remain emotionally distant from them and, through their behaviours, they establish a cold learning environment in their classes (De Meyer et al., 2014).

However, it should be noted that the above-mentioned need-supportive and need-thwarting styles describe prototypical behaviours. It is very unlikely for PE teachers to fully endorse a certain instructional style (e.g. autonomy support) without employing elements of the other interpersonal styles during everyday practice. For instance, students can perceive a teacher who provides more structure (i.e. competence support), but remains emotionally distant from their students (i.e. relatedness-thwarting) while providing feedback and guidance but in a controlling way (i.e. autonomy-thwarting; De Muynck et al., 2020). The fact that distinct styles can be combined by teachers to different degrees will be ultimately reflected in how students rate their teachers when they are asked to provide summary accounts of their PE teachers’ typical behaviour. Therefore, it is important to consider which interpersonal teaching style profiles endorsed in their instructional repertoire in PE are perceived by students, an issue which is typically addressed through a person-centred approach.

Interpersonal teaching style profiles in PE

A person-centred perspective classifies groups of individuals according to similar characteristics – in this case, the degree to which PE students believe that their teachers endorse autonomy, competence and relatedness supportive and thwarting interpersonal teaching styles. This approach allows us to analyse how different styles may be combined within people (Magnusson, 1988).

Additionally, a person-centred perspective can help us shed light on how interpersonal teaching style profiles could explain PE students’ higher behavioural and emotional engagement. To date, only limited research in the sports domain (Matosic and Cox, 2014; Reynders et al., 2020), academic settings (Amoura et al., 2015; study 1), and PE contexts (Haerens et al., 2018) has undertaken a person-centred approach to assess which autonomy-supportive and thwarting interpersonal teaching style profiles can explain differences in students’ motivation-related outcomes. In their study with PE students, Haerens et al. (2018) identified four different autonomy-supportive and thwarting profiles of PE teachers, according to their students’ ratings: a profile that was predominantly perceived to be high in autonomy support and low in autonomy-thwarting (high-low), a profile that was low in autonomy support and high in autonomy-thwarting (low-high), a third one that was relatively high both in autonomy support and thwarting (high-high) and a fourth profile that was characterized by relatively low autonomy support and low autonomy-thwarting (low-low).

Despite their similarities, sport and PE contexts have unique characteristics; in sports, the youths participate voluntarily, whereas PE classes are obligatory in most educational systems. Hence, PE teachers and sports coaches must quite often adjust their interpersonal styles accordingly. It should be noted that similar profiles have been reported for coaches (e.g. Matosic and Cox, 2014) and also for university teachers (Amoura et al., 2015). Further, these person-centred studies showed that participants who perceived their instructors to have high autonomy-supportive and low autonomy-thwarting profiles reported the highest need satisfaction and autonomous motivation. Similar profiles with coaches from various sports have been reported by Reynders et al. (2020), who investigated, in separate cluster analyses, two dimensions of need-supportive coaching style (i.e. autonomy support and structure) and two kinds of need-thwarting coaching styles: demanding (where discipline is imposed through assertive, commanding and threatening language) and domineering (where compliance is sought through the coach's authority; see Aelterman et al., 2019). Specifically, they distinguished between autonomy support versus demanding, autonomy support versus domineering, structure versus demanding and structure versus domineering. Across all four cluster analyses, four profiles (i.e. high-low, low-high, high-high and low-low) emerged when athletes rated their coaches’ (a) autonomy-supportive versus domineering coaching style (i.e. being harsh and intrusive in the personal domain), (b) structure versus demanding style (i.e. seeking compliance using controlling commands), (c) autonomy-supportive versus demanding style and (d) competence-supportive versus domineering style. Interestingly, in their study, Reynders et al. (2020) found that athletes reported higher engagement not only when they perceived their coaches to be high in autonomy-supportive style and low in demanding but also when they were high in both dimensions. A similar pattern was found when structure and demanding style were analysed together. Participants reported similar levels of engagement when either structure was high and demanding style was low or when both were high. Is this because autonomy-supportive and thwarting styles (or competence-supportive and thwarting styles) were considered separately from the other need-supportive and need-thwarting styles or is it because it is equally effective if instructors are perceived to use both need-supportive and need-thwarting styles? We aimed to examine this issue by jointly incorporating into our profile analysis not only the degree to which PE teachers are perceived to be autonomy and competence support and thwarting but also the degree to which they are believed to foster or undermine the need for relatedness.

The present study

Building on previous studies that have considered some, but not all three, aspects of students’ perceptions of need-supportive and thwarting interpersonal styles (e.g. Haerens et al., 2018; Leo et al., 2020; Van den Berghe et al., 2016), the present study aimed to replicate, through a person-centred approach, the four-profile solution reported in the few previous studies. We did so by considering all three need-supportive and thwarting (e.g. autonomy, competence and relatedness) interpersonal styles as perceived by the students. To reliably test the presence of this four-profile solution, we recruited a fairly large sample of adolescent PE students that included more than 2000 individuals. Further, based on previous studies that found gender and grade/age as potential moderators of perceived interpersonal style (Leo et al., 2020; Sánchez-Oliva et al., 2020), we implemented a gender- and school-level specific profile analysis to determine whether cluster distributions varied across boys and girls and elementary and secondary students. Additionally, we aimed to test to what extent students who perceived their PE teachers to have high need-supportive and low need-thwarting teaching styles reported more engagement than students who perceived their respective PE teachers to be low in need-supportive and high in need-thwarting teaching styles.

We focused on students’ engagement and, in particular, on two aspects: behavioural and emotional engagement. Student engagement can be defined as ‘the quality of a student's connection or involvement with the endeavour of schooling and hence, with the people, activities, goals, values and place that compose it’ (Skinner et al., 2009; 494). Specifically, behavioural engagement represents attention, concentration, effort and persistence; that is, academic behaviour and class participation, whereas emotional engagement reflects enthusiasm, interest and enjoyment; that is energized emotional states (Christenson et al., 2012; Leo et al., 2020; Skinner et al., 2009). Two reasons led to the evaluation of these variables: first, because they are closely related to teachers’ instructional behaviours (e.g. Assor et al., 2002; Christenson et al., 2012; Mouratidis et al., 2008; Skinner and Belmont, 1993; Van den Berghe et al., 2016). Second, because engagement is considered one of the most reliable indices of students’ well-being, quality of motivation and academic success (Christenson et al., 2012; Ryan and Deci, 2020; Skinner et al., 2009). Indeed, observational studies (e.g. Jang et al., 2010; Van den Berghe et al., 2016) have shown that need-supportive teaching was positively related to engagement and negatively to disengagement; in contrast, need-thwarting interpersonal teaching styles (i.e. controlling, cold and chaotic teaching) have been associated with more disengagement. Also, cross-sectional studies based on students’ reports have shown a positive association between perceived need-supportive teaching and engagement and a negative one between perceived need-thwarting styles and engagement (e.g. Leo et al., 2020). Therefore, it seems that the students of need-supportive PE teachers are more likely to become or remain behaviourally and emotionally engaged in a PE lesson. Students of such PE teachers are more likely to listen to their PE teacher, exert more effort and persevere, or respond positively to their teacher's requests (Furrer and Skinner, 2003; Reeve et al., 2004; Van den Berghe et al., 2016).

Based on prior empirical findings (e.g. Amoura et al., 2015; Haerens et al., 2018; Matosic and Cox, 2014; Reynders et al., 2020), we formulated the following hypotheses. First, we expected to find at least four instructional profiles as the most parsimonious solution. Specifically, we expected to find a group of students who would perceive their PE teachers as: (i) high in the three need-supportive and low in the three need-thwarting styles (high-low group); (ii) low in the three need-supportive and high in the three need-thwarting styles (low-high group); (iii) high both in need-supportive and need-thwarting styles (high-high group); and (iv) low both in the three need-supportive and need-thwarting styles (low-low group) (Hypothesis 1).

Second, we expected that students of PE teachers with a high-low profile would report higher engagement than their counterparts in the other three groups (Hypothesis 2). We were especially interested in comparing the high-low with the high-high group (providing that it emerged), given that some prior research found no differences between these two groups (see Matosic and Cox, 2014). Therefore, it is vital, both theoretically and practically, to determine whether students who perceived their PE teachers to be especially need-supportive and not need-thwarting eventually report more engagement than students who perceived their PE teachers to use both need-supportive and need-thwarting interpersonal teaching styles (again, providing that such clusters emerged).

Method

Participants

The initial sample consisted of 2087 adolescent students nested in 98 PE classrooms (range of class size: 8–31 students per class), from 17 schools, located in south-western Spain. Specifically, 405 students were enrolled in 5th grade, 532 in 6th grade, 250 in 7th grade, 170 in 8th grade, 271 in 9th grade, 267 in 10th grade and 192 in 11th grade. However, 22 questionnaires (1.05%) were excluded because they were incorrectly or not fully completed. Finally, 2065 students with a mean age of 11.96 years (SD = 1.95; range = 10–16 years; 1023 boys and 1042 girls) from elementary (5th and 6th grades; n = 915) and secondary (7th, 8th, 9th, 10th and 11th grades; n = 1150) schools participated. Also, a total of 38 PE teachers aged between 27 and 59 years (M = 44.92; SD = 8.98; 22 men and 16 women) participated in the study. Teachers had an average of 15.90 years of teaching experience (SD = 11.06, range = 3–34) and they were involved in a range of 1–4 classes.

Instruments

Perceived interpersonal teaching styles. Perceived teaching behaviour was measured using the Spanish instrument, namely the Teaching Interpersonal Style Questionnaire in Physical Education (TISQ-PE), developed by Leo et al. (2021). This questionnaire begins with the heading: ‘In Physical Education lessons, my teacher…’, followed by 24 items, corresponding to six factors. Specifically, four items measure perceived autonomy support (e.g. ‘…tries to give us choice when performing the activities’), four items assess competence support (e.g. ‘…favors learning and content improvement’) and four items tap into relatedness support (e.g. ‘…encourages good relations between classmates at all times’). Accordingly, a set of four items measured perceived autonomy-thwarting (e.g. ‘…requires me to do things in a certain way’), competence-thwarting (e.g. ‘…sets up situations that make me feel incapable’) and relatedness-thwarting (e.g. ‘…shows rejection towards me, on certain occasions’) teaching styles. Although need-thwarting could be delivered either in an externally controlling or internally controlling way (or in both ways; see De Meyer et al., 2014; Van den Berghe et al., 2015), the present scale does not differentiate between these two subtypes. In fact, behaviours such as ‘required me to do things in a certain way’ could be made salient both in an internal and an external controlling manner. Responses were rated on a 5-point scale ranging from strongly disagree (1) to strongly agree (5). Confirmatory factorial analysis (CFA) was performed to verify that the model fit was appropriate for the six-factor structure: chi square (χ²) = 1037.22, degrees of freedom (df) = 237, p < .001, comparative fit index (CFI) = .945, Tucker–Lewis index (TLI) = .935, root mean square error of approximation (RMSEA) = .040, 95% CI [.038, .043], standardized root mean square residual (SRMR) = .039.¹ Moreover, this scale showed acceptable levels of internal consistency in all the dimensions (α = .76 – .82, ω = .76 – .82).

Engagement. The Spanish version of the Engagement part of the Engagement versus Disaffection with Learning Scale (EVDLS; Skinner et al., 2009) was used to assess students’ perception of engagement in PE lessons. This scale contains a total of 10 items divided into two factors: behavioural engagement (five items, e.g. ‘I try hard to do well in class’) and emotional engagement (five items, e.g. ‘When we work on something in class, I feel interested’). Each item offers a 5-point response option, ranging from strongly disagree (1) to strongly agree (5). The CFA showed an adequate fit of the model: χ² = 300.73, df = 34, p < .001, CFI = .943, TLI = .925, RMSEA = .062, 95% CI [.055, .068], SRMR = .039. In addition, the factors of behavioural engagement (α = .82, ω = .82) and emotional engagement (α = .85, ω = .85) showed adequate values of internal consistency (Nunnally and Bernstein, 1994).

Procedure

Approval to conduct the study was received by the Ethics Committee (239/2019) of the researchers’ university. All participants were treated according to the American Psychological Association ethical guidelines regarding consent, confidentiality and anonymity of responses. To conduct data collection similarly in all the schools, a precise protocol was developed. First, the school principals and the PE teachers of the schools were approached and informed about the purpose of the study. Second, upon agreeing to participate in the study, a written consent form for the participants’ parents or legal guardians was handed to the students, who had to return it signed for the study to begin.

Data collection took place in the Spring semester of the 2019 school year. The questionnaires were completed individually during a PE lesson, typically within 10–12 minutes. Beforehand, the principal investigator explained the purpose of the study to the students, underlining that there were no right or wrong answers and that their responses would remain confidential and anonymous. The process of answering the survey was supervised by the research assistants, who provided any necessary clarifications to questions raised by the participants; the research assistants tried to achieve a quiet environment to help students concentrate on the survey without any distractions.

Data analysis

All the analyses were performed with the 0.9.0.1 version of JASP software (JASP Team, 2018). First, descriptive statistics (means and SD), and reliability correlations (omega coefficient and Pearson coefficient) were calculated. Cluster analysis was used to generate perceived teaching profiles. Before implementing cluster analysis, perceived teaching variables were transformed into standardized scores (z-scores). Furthermore, as cluster analysis is sensitive to the existence of outliers, values of more than three SD above or below the mean were deleted as univariate outliers (n = 87), whereas Mahalanobis distance was used to delete multivariate outliers (n = 3; χ²(6) = 5.99, p = .001). The cluster analysis was divided into two steps. In the first step, a hierarchical cluster analysis was carried out using Ward's method based on squared Euclidean distances (Steinley and Brusco, 2011). In the second step, the non-hierarchical k-means cluster analysis was used to obtain the final cluster solution, using the extracted initial cluster centres from the first step as non-random starting points. To examine the stability of the cluster solutions, the sample was randomly split into halves, and the previously described two-step process was implemented. Cohen's kappa (K) coefficient was used to measure the agreement between the original and the new cluster solution. Furthermore, we tested the extent to which the identified clusters differed in the two types of engagement, through Multivariate Analysis of Variance (MANOVA) followed by Analysis of Variance (ANOVA) both for behavioural and emotional engagement, and post hoc Tukey's tests with Bonferroni correction. Finally, we used the cross-tabulation χ² as a supplementary analysis to determine whether each cluster was proportionally composed of students of both genders and the two educational levels (i.e. elementary and secondary school).

Results

Preliminary analysis

Descriptive statistics, reliability and correlational analyses are shown in Table 1. Concerning gender differences, girls showed significantly higher scores in perceived competence support, whereas boys displayed higher scores in perceived autonomy-, competence- and relatedness-thwarting and emotional engagement. Perceived supportive strategies were negatively associated with thwarting strategies. As expected, behavioural and emotional engagement were positively associated with all three perceived need-supportive interpersonal teaching styles and negatively with the respective three need-thwarting interpersonal teaching styles. Lastly, all variables showed adequate reliability scores, with Cronbach’s alpha and omega coefficients ranging between .76 and .85.

Table 1.

Descriptive statistics, reliability and correlational analysis.

	M	SD	1	2	3	4	5	6	7	8	9	10
1. Gender	0.50	0.50	-
2. Educational level	0.56	0.49	.02	-
3. Autonomy support	3.33	1.03	.10**	.10**	.80/.80
4. Competence support	4.12	0.80	.06**	.13**	.56**	.77/.77
5. Relatedness support	4.18	0.83	.11**	.08**	.56**	.71**	.82/.82
6. Autonomy thwarting	2.22	0.96	.01	−.09**	−.40**	−.32**	−.34**	.76/.76
7. Competence thwarting	1.65	0.78	−.04	−.15**	−.33**	−.44**	−.43**	.54**	.78/.78
8. Relatedness thwarting	1.56	0.76	−.01	−.16**	−.28**	−.42**	−.44**	.54**	.71**	.80/.80
9. Behavioural engagement	4.26	0.73	.00	−.19**	.27**	.39**	.35**	−.17**	−.24**	-.27**	.82/.82
10. Emotional engagement	4.28	0.78	.05*	−.23**	.40**	.49**	.46**	−.26**	−.36**	−.34**	.65**	.85/.85

Note: *p < .05, **p < .01. Cronbach’s alpha (first) and omega (second) values are on the diagonal in italics. Gender and education level were dummy-coded (0 = girls; 1 = boys; and 0 = elementary school students; 1 = secondary school students).

Cluster analysis

In support of our expectations, a four-cluster solution was found to be the most optimal, given the Bayesian Information Criterion (BIC), solution parsimony and equal sizes of clusters. As shown in Figure 1, the increase of BIC relatively flattened at the four-cluster solution and therefore was considered the choice for the present analyses. Three out of the four clusters that emerged were in line with our hypotheses (see Figure 2 for the mean z-scores and absolute scores). Specifically, the largest cluster, which included almost one-third of all participants, consisted of students who perceived their PE teachers to be high in need-supportive and low in need-thwarting interpersonal styles. We, therefore, termed that group as ‘high-low’ (or, ‘need-supportive group’). The second cluster contained 337 students who rated their PE teachers to be relatively low (in terms of z-scores) in need-supportive and relatively high (in terms of z-scores) in need-thwarting interpersonal styles. Therefore, this group was labelled as ‘low-high’ or ‘need-thwarting’ group. The third cluster comprised 388 students relatively low (in terms of z-scores) both in need-supportive and need-thwarting instructional assessment; this group was named ‘low-low’. Finally, contrary to our expectations, the fourth group, which also included 388 students, did not rate their PE teachers relatively high either in need-supportive or need-thwarting styles. Interestingly, that group of students tended to rate their PE teachers in a more differentiated way; these students perceived their PE teachers to be relatively low (in terms of z-scores) in autonomy support, relatively high (in terms of z-scores) both in competence and relatedness support but also in autonomy-thwarting and relatively low (again, in terms of z-scores) both in competence- and relatedness-thwarting. Inspection of the means for that group led us to call it a ‘mixed-bag’ group.

Figure 1.

Bayesian Information Criterion (BIC) as a function of number of clusters.

Figure 2.

Standardized and absolute score means for the six dimensions per cluster for the four-cluster solution.

A MANOVA indicated, as expected, statistically significant differences in the linear combination of the six dimensions that were used to form the clusters (i.e. autonomy support and thwarting, competence support and thwarting, relatedness support and thwarting) as a function of group membership (Wilk's Λ = .07, F [18, 5561] = 352.01, p < .001, multivariate η² = .52). After Bonferroni correction, all six follow-up ANOVAs were statistically significant (Table 2) and the same was true for almost all the post hoc comparisons. The two exceptions concerned the non-significant differences in autonomy support between the low-high and the low-low groups and in autonomy-thwarting between the high-low and low-low groups. Taken together, these findings provided strong evidence that the four clusters were well separated from each other and, thus, that the PE students in the four groups did differentially perceive the way their PE teachers used need-supportive and need-thwarting interpersonal teaching styles in their everyday teaching.

Table 2.

Students’ mean scores of perceived need-supportive and need-thwarting interpersonal teaching styles as well as their behavioural and emotional engagement means (SD in parentheses) and z-scores.

	High-low		Low-high		Low-low		Mixed		F-value	η²
	(n = 862; 43.7%)		(n = 337; 17.1%)		(n = 388; 19.6%)		(n = 388; 19.6%)		F-value	η²
Gender
Girls (n = 992)	470 (47.4%)		149 (15.0%)		168 (16.9%)		205 (20.7%)		-	-
Boys (n = 983)	392 (39.9%)		188 (19.1%)		220 (22.4%)		183 (18.6%)		-	-
Educational level
Elementary (n = 880)	459 (52.2%)		96 (10.9%)		143 (16.2%)		182 (20.7%)		-	-
Secondary (n = 1095)	403 (36.8%)		241 (22.0%)		245 (22.4%)		206 (18.8%)		-	-
Dimensions	M (SD)	z-scores	M (SD)	z-scores	M (SD)	z-scores	M (SD)	z-scores
Autonomy support	4.11_a (0.12)	0.750	2.55_b (0.90)	−0.722	2.58_b (0.72)	−0.733	3.01_c (0.95)	−0.254	583.67*	.47
Competence support	4.68_a (0.36)	0.706	3.22_b (0.68)	−1.073	3.46_c (0.65)	−0.903	4.31_d (0.50)	0.273	903.50*	.58
Relatedness support	4.77_a (0.34)	0.711	3.19_b (0.74)	−1.133	3.55_c (0.69)	−0.857	4.37_d (0.54)	0.270	907.25*	.58
Autonomy thwarting	1.59_a (0.50)	−0.693	3.23_b (0.78)	1.073	1.87_a (0.60)	−0.289	3.12_c (0.73)	0.791	897.66*	.58
Competence thwarting	1.19_a (0.34)	−0.618	2.91_b (0.67)	1.633	1.42_c (0.44)	−0.286	1.83_d (0.61)	0.187	1042.56*	.61
Relatedness thwarting	1.14_a (0.30)	−0.569	2.81_b (0.69)	1.674	1.3°_c (0.41)	−0.302	1.64_d (0.57)	0.069	1067.68*	.62
Engagement measures
Behavioural engagement	4.51_a (0.54)	-	3.81_b (0.85)	-	4.01_c (0.83)	-	4.35_d (0.62)	-	108.37*	.14
Emotional engagement	4.63_a (0.53)	-	3.63_b (0.85)	-	4.0_c (0.83)	-	4.35_d (0.67)	-	198.92*	.23

Note: *p < .001. Cell means with different subscripts across rows are statistically and significantly different at p < .001 (two-tailed).

Group differences in behavioural and emotional engagement

The MANOVA through which we examined whether the linear combination of behavioural and emotional engagement differed as a function of cluster membership was statistically significant: Wilk's Λ = .76, F(6, 3940) = 100.00, p < .001, multivariate η² = .12. Follow-up ANOVAs both for behavioural and emotional engagement were also statistically significant (Table 2). Post hoc Tukey's test with Bonferroni correction showed, in support of Hypothesis 2, that students in the high-low group reported higher levels of both behavioural and emotional engagement than their counterparts in the other groups. Also, as expected, the low-high group reported the lowest levels of engagement (either behavioural or emotional). Notably, the students in that low-high group reported lower levels than either the low-low or the mixed groups. Of interest is also the difference that was found between the low-low and the mixed group, where the students of the latter group appeared to score the second highest level of engagement after the high-low group.

Supplementary analyses. We further examined to what extent boys versus girls, as well as elementary versus secondary school students, were over- or under-represented across the four clusters. Regarding gender, the χ² was statistically significant, χ²(3) = 19.75, p < .001. Inspection of the relative frequency distribution of boys and girls across the four groups (see Table 2, upper panel) suggested that, compared to girls, boys were more under-represented in the high-low group and more over-represented in the low-high and the low-low groups. Specifically, odds ratio post hoc analysis revealed that the odds for a boy to be in the low-high group or the low-low group were, respectively, 1.52 and 1.57 times higher than being in the high-low group, and 1.41 and 1.47 times higher than being in the mixed group.

Concerning elementary versus secondary school students, the χ² was also statistically significant, χ² (3) = 71.77, p < .001. Inspection of the relative frequencies (see Table 2, upper panel) suggested that elementary school students were under-represented in the low-high group and low-low group. Post hoc tests showed that, compared to secondary school students, the odds for an elementary school student to be assigned to the low-high or the low-low group were, respectively, 2.86 and 1.95 times lower than the odds to be assigned to the high-low group and 2.2 and 1.5 times lower than to be assigned to the mixed group.²

Discussion

In our study, we relied on students’ perceptions to identify a set of different profiles of interpersonal styles of PE teachers that support and thwart autonomy, competence and relatedness. Further, we examined whether the emerging profiles could explain differences in students’ behavioural and emotional engagement. Our profile analyses extracted four markedly different groups: (1) a need-supportive group, (2) a need-thwarting one, (3) a low need-supportive and a low need-thwarting group and finally, (4) a mixed-bag group characterized by relatively high autonomy-thwarting and relatively high levels of competence and relatedness support. Group comparisons amongst the four profiles indicated that students in the need-supportive group reported the highest levels of behavioural and emotional engagement in PE lessons.

Teaching style profiles

Partly in line with our first hypothesis, we found four different interpersonal teaching style profiles, as seen through the eyes of the students. First, as expected, we identified an interpersonal teaching style profile, characterized by relatively high levels of autonomy, competence and relatedness support and relatively low levels of autonomy-, competence- and relatedness-thwarting. Also, we found a profile characterized by relatively low levels of autonomy, competence and relatedness support and relatively high levels of autonomy-, competence- and relatedness-thwarting. These two profiles are consistent with previous studies focusing on perceived need-supportive interpersonal teaching styles in educational contexts. Amoura et al. (2015) and Haerens et al. (2018) also identified a group high in autonomy support and low in autonomy-thwarting (i.e. high-low), and another group that was high in autonomy-thwarting and low in autonomy support (i.e. low-high). In a similar vein, Reynders et al. (2020) also found these types of profiles when focusing on the need for competence (i.e. high competence support-low thwarting and low competence support-high thwarting) in the sport setting. Our findings extend these previous reports, as they suggest that there are PE teachers who concurrently support and do not thwart their students’ needs and PE teachers who concurrently do not support and thwart their students’ needs.

Second, we obtained an interpersonal style profile from students who perceived their PE teachers as offering relatively low autonomy, competence, and relatedness support but also abstaining from need-thwarting practices. Similar results were also reported by Amoura et al. (2015) and Haerens et al. (2018), who found an interpersonal style profile of PE teachers that was characterized by low levels of autonomy support and thwarting. A similar low-low profile of coaches was also reported by Reynders et al. (2020) when competence support and thwarting was considered. Taken together, our findings and previous ones imply that PE teachers who are perceived to neither support nor thwart their students’ psychological needs are necessarily indifferent to their students and the learning process. Such perceived indifference cannot but predict poorer outcomes, as studies using cluster analyses have shown (Amoura et al., 2015; Haerens et al., 2018).

Third, despite our expectations of finding a high-high profile as a fourth group, our fourth cluster was characterized by relatively low autonomy support and high autonomy-thwarting, combined with relatively modest-to-high levels of competence and relatedness support and modest levels of competence- and relatedness-thwarting. Although previous studies (e.g. Amoura et al., 2015; Haerens et al., 2018; Matosic and Cox, 2014) identified a high-high group (but not considering all three needs jointly), our analyses indicated a more perplexing picture when all three needs are taken into account. In our study, autonomy seems to make a difference, as we found a group of students who perceived their PE teachers as providing support for competence and relatedness but not for autonomy. Are our findings due to chance? Or is there indeed a special combination of low autonomy support and high competence and relatedness support? There is some indirect evidence that authority figures like parents or coaches may combine in various degrees aspects of need-supportive and need-thwarting interpersonal styles (such as affection and psychological control – see Aunola and Nurmi, 2005; or competence support combined with autonomy-thwarting – see De Muynck et al., 2017). As previous work in PE has not systematically assessed the interpersonal teaching styles that address all three needs jointly (Vasconcellos et al., 2019), more research is needed to examine whether there are profiles of teachers who combine different portions of autonomy-, competence- and relatedness-supportive (and thwarting) styles in their teaching repertoire.

Finally, regarding the number of teachers identified in each profile, the high-low profile was clearly the most represented (43.7%), followed by the other three profiles with similar percentages amongst them (low-low 19.6% and mixed 19.6%), although the low-high profile was the least represented (17.1%). Amoura et al. (2015) and Haerens et al. (2018) also found the least number of teachers in the low-high profile but, although Amoura et al. (2015) also found the high-low profile as the most represented, the percentage was lower (33.8%), with a much lower trend in Haerens el al.’s (2018) study (25.6%). Perhaps the presence in our study of elementary school students, who seem to have teachers with more need-supportive interpersonal styles, determines these differences in this profile, as the previous studies were carried out only with secondary (Haerens el al., 2018) and university students (Amoura et al., 2015).

Consequences of each teaching style profile

In support of our second hypothesis, our results showed that students who perceived their teachers as being relatively high in autonomy, competence and relatedness support, and low in autonomy-, competence- and relatedness-thwarting styles reported the highest levels of behavioural and emotional engagement. Consistent with previous reports (e.g. Haerens et al., 2018), this finding suggests that students benefit the most when they perceive their PE teachers as being need-supportive and concurrently abstaining from need-thwarting practices. Unfortunately, we failed to find a high-high group and, therefore, could not directly test some intriguing findings reported by Matosic and Cox (2014) and Amoura et al. (2015), who found nonsignificant differences between the high-low and the high-high autonomy support group. However, given that our study is based on a relatively large sample, it makes us confident to argue that high need-support combined with low need-thwarting is the safest route that PE teachers can use if they want to optimize their students’ engagement. Surely, PE teachers ought to think twice before using thwarting interpersonal styles or avoiding the use of need-supportive behaviours, given that our group comparisons showed that students in the low-high and low-low groups reported less engagement compared to their counterparts in the high-low group.

Moreover, our group comparisons show that students in the low-high and the low-low group reported less behavioural and emotional engagement than the mixed group. Given that this mixed group was made up of students who perceived their PE teachers to be relatively high in competence and relatedness support but not in autonomy support, this finding implies that autonomy-thwarting, combined with the presence of competence and relatedness support, might not seriously undermine students’ engagement in PE classes, although their engagement would still be less optimal compared to the high-low group. Previously, Matosic and Cox (2014) suggested that thwarting could be adaptive when paired with some supportive behaviours. Accordingly, some special characteristics of PE compared to other subjects can make students feel somewhat involved even if the need for autonomy is not fully supported. There are instances where students are simply used to receiving more directive teaching in PE class (Cothran et al., 2005; Cothran and Kulinna, 2020; Syrmpas et al., 2017) as some PE teachers have difficulty in sharing their authority with their students (Su and Reeve, 2011; Van den Berghe et al., 2013). Future research needs to examine whether indeed some teachers combine autonomy-thwarting with competence and relatedness support and whether such an instructional profile is not as detrimental as the need-thwarting (i.e. low-high) or indifferent (i.e. low-low) interpersonal teaching styles.

Finally, consistent with prior research in the educational setting (Amoura et al., 2015; Haerens et al., 2018), students in the low support and low thwarting group reported more behavioural and emotional engagement than their counterparts in the low support and high thwarting group. This means that, compared to the worst-case scenario where a PE teacher is thought to endorse a clearly need-thwarting style, students are more engaged in PE lessons when their teacher is considered to be withdrawn – that is being neither supportive nor thwarting. Again, the importance of not thwarting autonomy, competence and relatedness during PE lessons is highlighted in these results.

Gender and grade-level differences

Although not of primary importance, our analyses showed some gender and grade-level differences in the composition of the four emerging clusters. First, boys were more represented in the low-high and low-low groups and less represented in the high-low group compared to girls. This suggests that boys perceive more need-thwarting styles in their teachers during PE classes, and girls feel a more need-supportive style in their learning processes. Second, our results show that proportionally fewer elementary than secondary school students were assigned to the low-high and the low-low groups. This indicates that, in the early educational stages, teachers establish greater support during learning, whereas in the following stages, teachers try to carry out more directive classes. Furthermore, given the lack of differences in the relative amount of elementary and secondary students within the mixed profile (i.e. characterized by high autonomy-thwarting and high competence and relatedness support), these results seem to show that the most important difference in the interpersonal teaching style perception concerns autonomy-thwarting. However, future research is needed to continue to analyse in depth the interactions between the six dimensions of need-supportive and need-thwarting teaching styles by gender and educational grade levels.

Limitations and future research

The present study extends the body of work related to teaching behaviours by adopting a person-centred approach and generating and associating different profiles of interpersonal teaching style with students’ engagement in a large sample of elementary and secondary school students. However, a number of limitations warrant discussion. First, no causal inferences can be drawn, given the cross-sectional research design. Follow-up experimental studies need to manipulate PE teachers’ instructional styles that address all three needs to test whether such differentiation in instructional behaviours indeed leads to higher engagement. Second, findings are based on students’ self-reports and, therefore, they are suspect of mono-method bias. Future research needs to examine interpersonal teaching styles using multiple informants – namely, not only students but also the PE teachers themselves and observers. Third, our assessment of competence support did not include the clarifying expectations dimension, as assessed in some other widely validated scales (e.g. Aelterman et al., 2019; Belmont et al., 1988). Fourth, the generalization of our results cannot be guaranteed, as this study focused exclusively on PE as a subject of the lessons in a specific country (i.e. Spain). Future research could usefully replicate the present study findings in a more heterogeneous study sample in other subjects and contexts. Finally, this study only measures positive consequences, and only two aspects of students’ engagement; therefore, it is recommended for future studies to include a broader range of adaptive (e.g. autonomous motivation, self-efficacy, or deep-level learning) and maladaptive outcomes (e.g. frustration, disaffection, or disruptive behaviours) to achieve a fuller picture of the possible drawbacks of a supportive and thwarting approach in educational contexts. Thus, we could know in depth what kind of specific positive and negative consequences each of the interpersonal teaching profiles can have.

Practical implications

The results suggest that to promote students’ engagement, teachers should focus on supporting the three psychological needs. Specifically, to encourage autonomy support, it is important to generate a class atmosphere in which teachers consider the students’ opinions and initiatives. That is, teachers could allow students to express personal ideas and make their own decisions. To promote competence support, it is advisable to create clear expectations so that students know what is expected from them and to provide feedback reinforcing their work. Finally, to develop relatedness support, teachers can acknowledge students’ cooperative work through personal conversations with them. An inclusive class environment in which students can have conversations with classmates about how to resolve a task or promote an exchange of learnings will nurture students’ relatedness.

Conclusions

Four comparable motivating styles were identified in primary and secondary school PE students who rated their teachers’ support and thwarting in autonomy, competence and relatedness. Comparison of the motivating styles in terms of outcomes led to the conclusion that the presence of autonomy, competence and relatedness support, particularly in the absence of thwarting behaviours, relate to higher behavioural and emotional engagement. In contrast, motivating profiles characterized by high autonomy-, competence- and relatedness-thwarting seem to be associated with less engagement. Furthermore, the presence of autonomy-thwarting together with competence- and relatedness support does not seem to reduce students’ engagement in PE excessively.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the European Regional Development Fund (grant number IJC2019-040788-I, GR18102).

ORCID iDs

Francisco M. Leo

Juan J Pulido

Supplemental material

Supplemental material for this article is available online.

Notes

Author biographies

Francisco M. Leo is an Associate Professor at the Faculty of Teacher Training, University of Extremadura, Spain.

Juan J. Pulido is an Assistant Professor at the Faculty of Education, University of Extremadura, Spain.

David Sánchez-Oliva is a Postdoctoral Researcher at the Faculty of Sport Science, University of Extremadura, Spain.

Miguel A. López-Gajardo is a PhD student with Research Fellowship at the Faculty of Sport Science, University of Extremadura, Spain.

Athanasios Mouratidis is an Assistant Professor at the Faculty of Physical Education and Sport Science, National and Kapodistrian University of Athens, Greece.

References

Aelterman

Vansteenkiste

Haerens

, et al. (2019) Toward an integrative and fine-grained insight in motivating and demotivating teaching styles: The merits of a circumplex approach. Journal of Educational Psychology 111(3): 497–521.

Amoura

Berjot

Gillet

, et al. (2015) Autonomy-supportive and controlling styles of teaching: opposite or distinct teaching styles? Swiss Journal of Psychology 74(3): 141–158.

Assor

Kaplan

Roth

(2002) Choice is good, but relevance is excellent: autonomy-enhancing and suppressing teacher behaviours predicting students’ engagement in schoolwork. British Journal of Educational Psychology 72(2): 261–278.

Assor

Roth

Deci

(2004) The emotional costs of parents’ conditional regard: A self-determination theory analysis. Journal of Personality 72(1): 47–88.

Aunola

Nurmi

(2005) The role of parenting styles in children's problem behavior. Child Development 76: 1144–1159.

Belmont

Skinner

Wellborn

, et al. (1988). Teacher as Social Context: A Measure of Student Perceptions of Teacher Provision of Involvement, Structure, and Autonomy Support (Tech. Rep. No. 102). Rochester, NY: University of Rochester.

Christenson

Reschly

Wylie

. (2012). Handbook of Research on Student Engagement (Vol. 840). New York: Springer.

Cothran

Kulinna

(2020) Teachers’ and students’ experiences with and perceptions of the teaching styles. In: SueSee

Hewitt

Pill

(eds.) The Spectrum of Teaching Styles in Physical Education. London, UK: Routledge, pp. 128–138.

Cothran

Kulinna

Banville

, et al. (2005) A cross-cultural investigation of the use of teaching styles. Research Quarterly for Exercise and Sport 76(2): 193–201.

10.

Curran

Standage

(2017) Psychological needs and the quality of student engagement in physical education: teachers as key facilitators. Journal of Teaching in Physical Education 36(3): 262–276.

11.

De Meyer

Tallir

Soenens

, et al. (2014) Does observed controlling teaching behavior relate to students’ motivation in physical education? Journal of Educational Psychology 106(2): 541–554.

12.

De Muynck

Vansteenkiste

Delrue

, et al. (2017). The effects of feedback valence and style on need satisfaction, self-talk, and perseverance among tennis players: An experimental study. Journal of Sport and Exercise Psychology 39: 67–80.

13.

De Muynck

GJ.

Morbée

Soenen

, et al. (2020) Do both coaches and parents contribute to youth soccer players’ motivation and engagement? An examination of their unique (de) motivating roles. International Journal of Sport and Exercise Psychology 19(5): 761–779. DOI: 10.1080/1612197X.2020.1739111.

14.

Filak

Sheldon

(2008) Teacher support, student motivation, student need satisfaction, and college teacher course evaluations: testing a sequential path model. Educational Psychology 28(6): 711–724.

15.

Furrer

Skinner

(2003) Sense of relatedness as a factor in children’s academic engagement and performance. Journal of Educational Psychology 95(1): 148–162.

16.

Haerens

Aelterman

Vansteenkiste

, et al. (2015) Do perceived autonomy-supportive and controlling teaching relate to physical education students’ motivational experiences through unique pathways? Distinguishing between the bright and dark side of motivation. Psychology of Sport and Exercise 16(3): 26–36.

17.

Haerens

Vansteenkiste

De Meester

, et al. (2018) Different combinations of perceived autonomy support and control: identifying the most optimal motivating style. Physical Education and Sport Pedagogy 23(1): 16–36.

18.

Bentler

(1999) Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 6(1): 1–55.

19.

Jang

Reeve

Deci

(2010) Engaging students in learning activities: It is not autonomy support or structure but autonomy support and structure. Journal of Educational Psychology 102(3): 588–600.

20.

JASP Team (2018) JASP (Version 0.9.0.1). Computer software. https://jasp-stats.org/download/.

21.

Leo

Mouratidis

Pulido

, et al. (2020) Perceived teachers’ behavior and students’ engagement in physical education: The mediating role of basic psychological needs and self-determined motivation. Physical Education and Sport Pedagogy: 27(1): 59–76. DOI: 10.1080/17408989.2020.1850667.

22.

Leo

Sánchez-Oliva

Fernández-Rio

, et al. (2021). Validation of the teaching interpersonal style questionnaire in physical education. Journal of Sport Psychology: 30.

23.

Magnusson

(1988) Individual Development from an Interactional Perspective: A Longitudinal Study. Hillsdale: Erlbaum.

24.

Matosic

Cox

(2014) Athletes’ motivation regulations and need satisfaction across combinations of perceived coaching behaviors. Journal of Applied Sport Psychology 26(3): 302–317.

25.

Mitchell

Gray

Inchley

(2015) ‘This choice thing really works … ’ changes in experiences and engagement of adolescent girls in physical education classes, during a school-based physical activity programme. Physical Education and Sport Pedagogy 20(6): 593–611.

26.

Mouratidis

Vansteenkiste

Lens

, et al. (2008) The motivating role of positive feedback in sport and physical education: evidence for a motivational model. Journal of Sport and Exercise Psychology 30(2): 240–258.

27.

Nunnally

Bernstein

(1994) The assessment of reliability. In: Nunnally

Bernstein

(eds.) Psychometric Theory. New York: McGraw-Hall, pp. 248–292.

28.

Reeve

(2009) Why teachers adopt a controlling motivating style toward students and how they can become more autonomy supportive. Educational Psychologist 44(3): 159–175.

29.

Reeve

(2015) Giving and summoning autonomy support in hierarchical relationships. Social and Personality Psychology Compass 9(8): 406–418.

30.

Reeve

Jang

Carrell

, et al. (2004) Enhancing students’ engagement by increasing teachers’ autonomy support. Motivation and Emotion 28(2): 147–169.

31.

Reynders

van Puyenbroeck

Ceulemans

, et al. (2020) How do profiles of need-supportive and controlling coaching relate to team athletes’ motivational outcomes? A person-centered approach. Journal of Sport and Exercise Psychology 42(6): 452–462.

32.

Ryan

Deci

(2000) Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist 55(1): 68–78.

33.

Ryan

Deci

(2017) Self-Determination Theory: Basic Psychological Needs in Motivation, Development, and Wellness. New York: Guilford.

34.

Ryan

Deci

(2020) Intrinsic and extrinsic motivation from a self-determination theory perspective: definitions, theory, practices, and future directions. Contemporary Educational Psychology 61: 101860.

35.

Sánchez-Oliva

Mouratidis

Leo

, et al. (2020). Understanding physical activity intentions in physical education context: A multi-level analysis from the self-determination theory. International Journal of Environmental Research and Public Health 17(3): 99.

36.

Skinner

Belmont

(1993) Motivation in the classroom: reciprocal effects of teacher behavior and student engagement across the school year. Journal of Educational Psychology 85(4): 571–581.

37.

Skinner

Kindermann

Furrer

(2009) A motivational perspective on engagement and disaffection: conceptualization and assessment of children’s behavioral and emotional participation in academic activities in the classroom. Educational and Psychological Measurement 69(3): 493–525.

38.

Steinley

Brusco

(2011) Evaluating mixture modeling for clustering: recommendations and cautions. Psychological Methods 16(1): 63–79.

39.

Reeve

(2011) A meta-analysis of the effectiveness of intervention programs designed to support autonomy. Educational Psychology Review 23(1): 159–188.

40.

Syrmpas

Digelidis

Watt

, et al. (2017) Physical education teachers’ experiences and beliefs of production and reproduction teaching approaches. Teaching and Teacher Education 66: 184–194.

41.

Van den Berghe

Cardon

Tallir

, et al. (2016) Dynamics of need-supportive and need-thwarting teaching behavior: The bidirectional relationship with student engagement and disengagement in the beginning of a lesson. Physical Education and Sport Pedagogy 21(6): 653–670.

42.

Van den Berghe

Soenens

Vansteenkiste

, et al. (2013) Observed need-supportive and need-thwarting teaching behavior in physical education: Do teachers’ motivational orientations matter? Psychology of Sport and Exercise 14(5): 650–661.

43.

Van den Berghe

Tallir

Cardon

, et al. (2015) Student (dis)engagement and need-supportive teaching behavior: A multi-informant and multilevel approach. Journal of Sport and Exercise Psychology 37(4): 353–366.

44.

Vasconcellos

Parker

Hilland

, et al. (2019) Self-Determination theory applied to physical education: A systematic review and meta-analysis. Journal of Educational Psychology 112(7): 1444–1469.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.55 MB