Conducting the Pilot Study: A Neglected Part of the Research Process? Methodological Findings Supporting the Importance of Piloting in Qualitative Research Studies

Abstract

During the development of research to compare the processes and impact of inclusive education in Sweden with results obtained from a study undertaken in Ireland, a pilot study was conducted and documented. The pilot study had three aims: (1) to gather data to provide guidance for a substantive study adapted to Swedish conditions through modification of Irish research procedures and instruments, (2) to critically interrogate how we as researchers could most effectively conduct a pilot study utilizing observational and video-recorded data, and (3) to use the Irish theoretical model as a tool of analysis for studying inclusion in two Swedish schools. Although pilot studies are frequently conducted to assess the efficacy of research instruments for use in qualitative research projects, few publications have drawn upon empirical findings related to such studies. Additionally, while methodological texts recommend the use of pilot studies in qualitative research, there is a lack of reported research focusing on how to conduct such pilot studies. We argue that our methodological findings may contribute to greater awareness of the important role that a pilot study may have for full-scale qualitative research projects, for example, in case study research where semi-structured qualitative interviews are used. This argument is based on the assumption that researchers, and especially novice researchers, having conducted a pilot study will be better informed and prepared to face the challenges that are likely to arise in the substantive study and more confident in the instruments to be used for data collection. A proper analysis of the procedures and results from the pilot study facilitates the identification of weaknesses that may be addressed. A carefully organized and managed pilot study has the potential to increase the quality of the research as results from such studies can inform subsequent parts of the research process.

Keywords

case study methods in qualitative inquiry qualitative evaluation interpretive description mixed methods

Introduction

The proposal to investigate quality issues related to pilot studies in qualitative research was initiated by researchers from two Swedish universities, while working in partnership with academic colleagues from United Kingdom and Ireland who had come together to design a comparative study of inclusive education in Sweden and Ireland. It was proposed that a Swedish study would utilize data-collecting instruments previously used in the large-scale Irish project (IRIS; Rose, Shevlin, Winter, & O’Raw, 2015) with adaptation for the Swedish context. The proposed study, Inclusive Research in Swedish Schools (IRISS), would replicate the Irish approach of “interview teams” (several interviewers conducting interviews) in a number of case study schools. In order to compare the findings from two countries, it was deemed necessary to use interview guidelines and interview framework, which resembled those used in the Irish research. These could not, however, be identical as the school systems function in different contextual and cultural conditions (cf. Kim, 2010). In order to explore these differences, a substantial pilot study was regarded as an essential requirement in advance of the main study as a means of ensuring trustworthiness and utility. The interview instruments developed had to be flexible enough to make it possible to investigate conditions that are unique to Swedish schools.

The principal investigators from project IRIS played an important role in providing the Swedish researchers with guidance throughout the pilot study as procedures were adapted to Swedish conditions. This involved several steps including:

constructing interview guides that were similar to those used in Project IRIS,

translating research questions aligned to the format used in IRIS,

developing a way of posing interview questions similar to the IRIS study,

constructing training materials for researchers that would be a part of the IRISS main study, and

preparing for the pilot study work by reading and discussing the IRIS project and associated literature through research seminars focused on methodological and theoretical issues.

These preparatory steps were taken as part of a pilot study with two schools to ensure the quality of the main IRISS study, which has not been conducted yet. The Swedish researchers need to be prepared from the first day of fieldwork when they approach schools for collecting data. All procedures including interview guides need to be in place and the quality of the data to be collected require a process of evaluation. This necessitates that all procedures for conducting the research have to be adapted for Swedish conditions. While such preparations should be the norm in planning research studies, they are not always made explicit. Additionally, we found it necessary to design the pilot study in order to obtain methodological findings in a systematic way due to the poor availability of research publications about conducting effective pilot studies. Three aims were established for the pilot study:

to gather data to provide guidance for a substantive study adapted to Swedish conditions through modification of Irish research procedures and instruments;

to critically interrogate how we as researchers could most effectively conduct a pilot study utilizing observational and video-recorded data, which is a procedure coined by us as a “collaborative self-study approach”; and

to use the Irish theoretical model as a tool of analysis for studying inclusion in two Swedish schools, with a special focus on identifying “Commonalities” or “Exceptionalities,” related to inclusion (Rose & Shevlin, 2016).

Accordingly, three different types of findings were expected:

findings indicating how well the case study approach worked regarding the interview format, with the adaptations that were made (related to Aim 1);

findings from the investigation of the researchers’ handling of the pilot study within a “collaborative self-study approach,” which consisted of observational findings from the observers at the interview sessions and video-recorded documentation, from interviewees’ feedback, and from meta-analytical discussions. The “meta-analytical” discussions consisted of critical discussions about the data collection, held between the researchers during breaks between interviews and the day after the school visits (related to Aim 2); and

empirical findings about how the two pilot schools (C and F) provide special needs education, for instance, their work with special needs provision, and whether they have policy documents demonstrating a commitment to work toward inclusion (related to Aim 3).

Earlier Research

In this section of the article, we focus on methodological aspects of conducting pilot studies, in relation to the second and the third aims established above.

Different Types of Pilot Studies and Their Purposes

Pilot studies are commonly used within quantitative health-related inquiries in disciplines such as nursing and medicine (van Teijlingen & Hundley, 2001). These authors argue that the two main types of pilot studies used in social science are for the most part (1) smaller versions of studies, called feasibility studies, and (2) “the pre-testing or ‘trying out’ of a particular research instrument” (with reference to Baker, 1994). The feasibility study is used to assess the practicalities of the main study in respect of its implementation and utility and often includes an assessment of resources, such as time and costs, for the main study (Gudmundsdottir & Brock-Utne, 2010). In medicine, several types of feasibility studies exist according to Vogel and Draper-Rodi (2017), and they are often connected to the use of randomized control trials as main studies. The pilot study reported in this article may be viewed as a combination of the two types that van Teijlingen and Hundley (2001) propose but within a research project based on case study research where the collection of qualitative data is the main objective. Our pilot study also has another distinguishing feature as having an ambition to contribute to increased methodological knowledge as well as awareness about conducting pilot studies. It should be noted that the kind of pilot study reported in this article also has other unique features as being part of a comparative study of research that has been conducted before. The IRIS research team conducted pilot studies in advance of their main study, and we were thus able to avail of the research experiences gathered from the largest “inclusion” study conducted in Europe to date. In some ways, this may be regarded as a “pilot study,” or a “feasibility study,” in advance of the IRISS study. But, as the school systems and the culture in Ireland differ from Swedish conditions, we identified the need to conduct a pilot study in Sweden. We have restricted the planning of our research design to only include the case study approach used in IRIS together with a review of literature concerning inclusion in Sweden, which is similar to the review carried out in project IRIS. The findings from the literature review are not being reported in this article in which we focus only on aspects of the piloting process related to research quality.

One of the aims with conducting a pilot study is to increase research quality, and this may potentially be achieved in most aspects of a research process (Gudmundsdottir & Brock-Utne, 2010; van Teijlingen & Hundley, 2001). Gudmundsdottir and Brock-Utne (2010) especially emphasize its importance in enhancing reliability and validity in research. Therefore, a pilot study should be viewed as a crucial part of a research design (Kim, 2010; van Teijlingen & Hundley, 2001). In contrast to these judgments, “pilot studies have attracted limited attention in research literature” (Kim, 2010, p. 191). When pilot studies are found in research publications, they are seldom discussed in-depth, with few detailed descriptions of how they were conducted and how the main study was adapted in terms of changes of procedures, instruments, and other management issues (van Teijlingen & Hundley, 2001). The purpose of a pilot study is not simply to declare that this has been conducted or to justify the methods deployed without making any details explicit; rather, the focus should be to identify the necessity to modify questions or other procedures that do not elicit appropriate responses or enable the researchers to obtain rich data (cf. Gudmundsdottir & Brock-Utne, 2010; Kim, 2010). Chenail (2011) identifies other important issues which may arise during the piloting of qualitative interviewing approaches by emphasizing several challenges for researchers in terms of “instrumentation rigor” and management of bias. Poggenpoel and Myburgh (2003) share this concern and views the interviewer as an “instrument” when collecting interview data. In order to counteract threats to trustworthiness, they emphasize the importance for researchers using interviewing to spend time on the fieldwork processes to be used in order to be well-prepared. They also advise researchers to be humble, reflexive, and work in teams and thereby use peer evaluation.

An effective implementation is an important part of a pilot study, irrespective of the type of pilot study or if it is within qualitative or quantitative research. There are several critical aspects related to the implementation such as the pilot study size, the methods, and the content of the pilot study. Sometimes, especially in large research projects, a number of pilot studies may be needed and qualitative as well as quantitative methods may be used (van Teijlingen & Hundley, 2001). van Teijligen and Hundley (2001, p. 1) give an example of this:

The first phase of a pilot might involve using in-depth interviews of focus groups to establish the issues to be addressed in a large-scale questionnaire survey. Next the questionnaire, e. g. the wording and the order of the question, or the range of answers on multiple-choice questions, might be piloted. A final pilot could be conducted to test the research process, e. g. the different ways of distributing and collecting questionnaires.

It is clear from this example that pilot studies can play an important part in designing a research study and that they need to be adapted for the main study. This is, however, not sufficient according to van Teijlingen and Hundley (2001) who state that the use of pilot studies needs to be more widely discussed and experiences from pilot studies disseminated as these issues are related to research quality.

As previously stated, pilot studies are seldom published (Gudmundsdottir & Brock-Utne, 2010; Kim, 2010; Sampson, 2004; van Teijlingen & Hundley, 2001). van Teijlingen and Hundley outline a number of reasons for the limited publication which usually concern quantitative research studies. One of these is a tendency for journals to only accept papers that have statistically significant results. They propose few reasons for the scarcity of published pilot studies where qualitative methods have been used other than suggesting that the process may be of less importance to many qualitative researchers. van Teijlingen and Hundley (2001) refer to arguments from “qualitative” researchers that separate pilot studies are not necessary in interpretative inquiry. As authors, we acknowledge that in some iterative research, where small-scale exploratory investigations are undertaken with the sole purpose of determining whether an issue is suitable for a substantial research, a piloting process may be of lesser importance.

The Limited Discussion of Methodological Issues in Research Publications

As stated, several researchers report a paucity of research publications about pilot studies in qualitative research, particularly concerning methodological findings. Consequently, there are few guidelines in the research literature about how to conduct a pilot study of this nature. We investigated this by searching for methodological issues about conducting pilot studies in standard qualitative research methodological texts. These edited volumes (Bryman & Burgess, 1999; Denzin & Lincoln, 2005, 2011; Flick, 2018; Seale, Gobo, Gubrium, & Silverman, 2004) contain almost 5,000 pages and provide limited guidance about conducting pilot studies. In another internationally widely distributed methods book (Bryman, 2016), content about pilot studies was only to be found in the section focusing on quantitative methods. Apart from less systematic searches in Google Scholar, which we combined with Snowballing search techniques (Heyvaert, Hannes, & Onghena, 2017), we also conducted a systematic search for empirically based research publications in the Education Resources Information Center (ERIC) database. The two broad thesaurus terms “Pilot Projects” and “Research Methodology” were used together with the Boolean operator AND. The settings were as follows: subject: “Qualitative research,” limiters: “Peer Reviewed,” and publication date: 1992–2018. Only 16 publications were found. Most (13) contained empirical results of, for example, students’ achievements in Spanish or how the use of an alternate grading tool in higher education had worked in a pilot study. Only four articles contained methodological findings: one reported aspects of data collection with the use of multimedia interviews (Pratt & Yezierski, 2018), another focused on the use of diaries in library research (Pellegrino, 2014), a third contained an “interviewing the investigator approach” that can be used when other piloting is not possible or practical (Chenail, 2011), and the fourth reported the use of pilot studies in action research (Gudmundsdottir & Brock-Utne, 2010). Pratt and Yezierski (2018) and Pellegrino (2014) reported on the design of feasibility studies in advance of a larger main study, whereas Chenail (2011) suggested an alternative way to test interview questions instead of using a regular pilot study. While the pilot studies conducted by Pratt and Yezierski (2018) and Pellegrino (2014) were part of their research designs, Gudmundsdottir and Brock-Utne’s (2010) pilot was originally the first phase of an action research study:

The important phase in the beginning of the project was not planned as a pilot project either. The further we got into the action research study, the more we drew on information gathered in the first very tentative phase of the project. In a way, most of the problems we ran into could be foreseen from data gathered in the first phase, had we only taken the time then to analyse them and looked at them as piloting the action research project that later took place. (p. 366)

Published methodological results from a pilot study may be discovered “by coincidence,” as the research design was not intended to investigate methodological issues. This was reported in Pratt and Yezierski’s (2018) research approach. They stated that “the use of a pilot study to test the method and interview guide further adds credibility and dependability to the study” (p. 417). They also indicated that “pilot study interviews were used primarily to build researcher expertise in using/troubleshooting the interview platform” (p. 15), indicating that their main focus was not to contribute to methodological knowledge. Our research design, on the other hand, had triple purposes from the beginning: (1) to contribute to design and research method findings, (2) with findings about how we as researchers managed to conduct the pilot study, and (3) empirical findings ahead of the main study.

Our search was limited to the ERIC database, due to limitations of resources, with its main focus on research literature within education. It would have been preferable to conduct a full-scale scoping review (Arksey & O’Malley, 2005; Paré, Trudel, Jaana, & Kitsiou, 2015) in several research databases and across a range of disciplines. Such a large-scale search would have given more references but not necessarily within educational qualitative research. Our combined search strategy, which was based on a systematic search in ERIC, less structured searches by using Google Scholar, Snowballing search techniques in found research publications, and investigations in standard qualitative research methodological texts, all indicated that there is limited discussion of pilot studies in qualitative educational research. This is important as the way in which pilot studies are conducted, if they are used at all, may be critical for the quality of subsequent main studies (Gudmundsdottir & Brock-Utne, 2010; Kim, 2010; van Teijlingen & Hundley, 2001). The pilot study reported here is an example of an approach which, to our knowledge, has not been previously reported in the literature.

Method

There are several issues involved in conducting both a pilot study and an “introspective” study’ focusing on the management of the pilot study. The latter is an approach in which we are investigating how we are conducting our own study. We will approach these issues chronologically in four sections: (1) Designing the pilot study, (2) Planning and preparation of data collection, (3) Data collection, and (4) Data analysis and findings.

Designing the Pilot Study

The initial idea of conducting a study in Sweden, comparing educational inclusion in Ireland (IRISs) and Sweden (IRISS), started in 2015 when one of the Swedish authors visited the UK partner university. A team comprising researchers from two Swedish universities was established to partner the UK and Irish universities that managed the original Irish-based study. The team contained experienced researchers within the field of inclusive education, and there has been a series of meetings where the IRIS study was discussed. Due to the comparative research approach, the publications from the IRIS project have guided the design of the Swedish study with a strong emphasis on the pilot study which was central to that originally conducted in Ireland. The research team has disseminated the original study methodology in Sweden through a series of research seminars. Funding was sought in order to progress this initiative, and resources were secured for the pilot study which was conducted in two schools.

Planning and Preparation of Data Collection

Two schools were selected to represent different types of school settings in two municipalities in the south of Sweden. The two researchers, who visited the schools, had no previous contacts with these schools. When the principals were contacted, they agreed to take part in the pilot study and believed that it would be beneficial for their schools to take part in research studies as such experiences stimulate school development.

Both school settings had preschool classes to Grade 6, with ages ranging from 6 to 12 years old. One of the schools, school SED (a school in a Socio-Economically Disadvantaged catchment area), was a mainstream school with more than 300 pupils and was situated in a municipality with about 100,000 inhabitants. Ninety percent of the pupils had an ethnic background other than Swedish, and many were the children of newly arrived immigrants. The second school visited by the researchers, School LDU (a school with a Learning Disability Unit), had about 200 pupils on roll and was situated in a small municipality with about 10,000 inhabitants. The school included a unit for pupils with intellectual impairments. Potential interviewees were given written information about the research study, which contained details about the aims of the pilot study and its relationship to the main study, and notified that they would be asked to provide the researchers with critical feedback on the pilot instruments and the interview techniques that were used. It was made clear to them that this information would be used to assist us in making improvements and clarification to the research instruments. They were also informed that video recording would take place and that the video camera would focus only on the interviewer.

The original interview instruments used in Ireland were translated by the Swedish researchers. There was one interview guide for each category of interviewee, with many interview questions common to all categories. These instruments, or interview schedules, were examined by a third Swedish researcher who gave suggestions for changes. The interview instruments were discussed in two meetings, one in Sweden and one in Dublin. After the first data collection phase, experiences of using the instruments were discussed in the meeting in Dublin.

Participants

Based on the research team discussions, we concluded that we should interview people representing the same professions as in IRIS. At each school, interviews were conducted with the principal, one teacher, one special educator (the equivalent to support staff in IRIS), and one student assistant (special needs assistant in IRIS). Based on the important role student health teams have in Sweden, an additional interview was planned at each school with a member of this team. At both schools, this was a social worker.

Data Collection

During 1-day visits, five interviews were conducted in each school and school documentation was requested. In accordance with the introspective approach, two researchers participated in each interview, one as interviewer and one as observer of the interview process. All interviews were video-recorded, with the camera only directed toward the interviewer. The researcher acting as observer focused on the interviewer’s technique but remained silent throughout. After the interview, the interviewee was given the interview guide to assist with recall of the questions asked. The interviewee was asked to give feedback concerning the quality of the questions, the interviewer’s conduct, and whether there were further questions which might have revealed important issues related to the research topics. This part of the interview we termed “the feedback response.”

Video documentation of the interview procedures was important for three main reasons: (1) in order to study our own effectiveness in obtaining useful responses, (2) to produce instructional material for interviewees participating in the main study, and (3) to identify ethical issues, which were important for guiding the work and obtaining approval to proceed.

The interviews were scheduled by the principals, and each was planned to last approximately 45 min with an additional 15 min for feedback. In the analysis of the data, it was clear that these informal feedback sessions added important information that may have been a result of the “open” dialogue style adopted during these sessions. Among other advantages, they could discuss issues that they viewed as important and that were not part of the interview schedules.

Interviews with parents were also undertaken. These were conducted in a different way from those conducted by the IRIS researchers. Instead of face-to-face interviews, we used mobile phone interviews and agreed with parents not to report their experiences to the schools. This decision was made for ethical reasons, as no ethical approval had been applied for, and to avoid the possibility that the parents and consequently their children could have been identified by school staff through the experiences revealed in written reports. The parents were asked to use a phone setting which would make the phone number impossible to identify for the interviewer. They were given the same information and ethical guidelines as all other interviewees. In contrast to IRIS, no interviews with pupils were conducted and no observations were used as we had not applied for ethical approval.

Further data were obtained from documented meta-analytical discussions between the two interviewers at breaks between interviews and after the school visits. These discussions focused on experiences from the visits and are a part of the collaborative self-study approach.

Data Analysis and Findings

The analysis of data collated within the three areas, which previously have been described, was focused on:

the usefulness of the data collection procedures and instruments, which relates to the first aim;

the interviewers’ way of conducting the study, with a substantial focus on the management of the interviews, which is related to the second aim; and

the content of the interviewees’ answers relating to our study of inclusion in two Swedish schools, thereby addressing the third aim.

Analysis of the interview format

For the analysis of the interview format, data were used that were collected from the interviews, from the feedback response session, and from meta-discussions between the interviewers the day after the school visits. All interviews and feedback responses were transcribed verbatim. A matrix was constructed for the analysis, which consisted of 4 columns and 16 rows (Figure 1). The first column focused on the usefulness of the interview guides. The same three headings were used irrespective of interview guide: “Questions that needed to be modified,” “Questions that needed to be added,” and “Questions that needed to be withdrawn.”

Figure 1.

The analysis matrix for the interview format and the interviewers’ management of the interviews.

The second column contained notes about questions used in school SED, the third column had questions from school LDU. In the column, there were spaces for comments such as those obtained from one of the principals (School F) given as an example here. The principal commented in the feedback response part that she liked one question, which asked, “how do you assess the impact of your work upon the progress and attainment of pupils?” The principal said she had to think for a while because she was not used to considering issues in this way and that she said indicated that this was a very good question. The analysis also clarified the need for adding interview questions such as those about the student health units, their composition, objectives, and work within schools.

Analysis of the interviewers’ way of conducting the interviews

The performance and conduct of the two interviewers were examined through the transcribed material from the interviews and the feedback response sections and also from notes from two meta-discussions. This was systematic and structured and was based to a large extent on the matrix used in the analysis of the interview format (Figure 1). This material guided the researchers to the parts of the interviews where mistakes could have been made but also to areas where good practice was noted. Audio and, in some instances, video recordings were used to analyze these situations thoroughly. Using Flick’s (2014) terminology, this part of the analysis focused on how the interviewers handled the interviews through a combination of a detailed analysis and a “rough” analysis to get an overview and a summary (p. 5). In this analysis, documentation from the meta-analytical discussions was also used.

Analysis of the content in the interviewees’ answers

The interviewees’ answers and the two schools’ documentation were analyzed by using the theoretical model used in project IRIS (Rose et al., 2015, p. 46) which can be seen in Figure 2.

Figure 2.

The theoretical model used for the analysis of the content in the interviews.

In order to analyze to what extent the data collection was successful in collecting data about inclusion, two different analytical methods were used. The first was a mapping of the content into the four areas: policy, provision, experience, and outcomes. These were adopted from the approach used in the Irish research study in which the researchers (Rose et al., 2015) had identified four key areas of concern for both policy makers and professionals working in the area of inclusive education. The four areas of policy, provision, experience, and outcomes allowed the research team to organize emerging issues under these four themes and enabled the sorting and management of data to be conducted in a systematic manner. In order to make this explicit, this analysis was undertaken by coloring the interview texts with four colors, one for each area. The Project IRIS team in Ireland had further strengthened this model by identifying those sources such as teachers or parents from whom data had been obtained. This assisted greatly in the management of data at the interpretation and analysis stage of the project.

This part of the analysis provided a picture of each of the areas discussed during the interviews. In a subsequent step of this analysis, the content within each area was broadly measured. In the second analysis, the procedure used in IRIS was applied and the interview text was thematically coded (Rose et al., 2015). The interviews were analyzed by all three Swedish researchers in order to investigate the congruence of the codes used in project IRIS with Swedish conditions. In the analysis of the content, comparisons were also made between the two schools in order to map commonalities (findings common to both schools) and exceptionalities (findings unique for each school).

Results

1. Findings about how well the case study approach and the interview format worked, with the adaptations that were made (e.g., if the questions obtained the data required). These findings address the first aim.

Two questions that were included in all interview guides did not work as expected in school SED. These were modified in advance of the visit to school LDU and worked well there. In school LDU, the principal proposed modifications for two other questions including the question “What impact has the Disability Act had on your work?” which needed clarifications from the interviewer. The principal arrived eventually at an answer she was satisfied with but claimed that another way of formulating this question would have made it easier to answer.

From the responses obtained from five interviewees in school SED, two additional questions were suggested. It was decided that the specific kind of information we were advised to obtain through these questions, such as the proportion of immigrant pupils at the case study schools, will be collected through other means during the case study visits. The two interviewers arrived at the conclusion in the meta-discussion after the school visit to school SED that two additional questions were needed among the common questions that were used in all interviews. One of these focused on obtaining information about the function the special educator in providing special support. This question had been used in school SED and had provided valuable information. The other question interrogated understanding of the term inclusion and became the final question in all interview schedules. These two questions were added to the interview schedules and eventually used in school LDU. During the visit in school LDU, only one of the interviewees wanted to add a question. This is to consider how teachers work with pupil assistants, as this was perceived to be an important, complex, and sensitive issue.

None of the respondents in the two schools reported finding any question to be irrelevant or of less value but indicated that they were interesting and important.

We found that the order of interviews also was important. Both interviews with the principals were completed after others have been conducted. This was planned by the principals, not by us, as they were requested to schedule all interviews. During the “other” interviews, we sometimes received answers, which required additional information or clarification to be fully answered. In these cases, the interviewees referred to the principals as they could not fully answer some of our follow-up questions. We arrived at the conclusion that we would not have obtained the necessary information without first having different issues raised by other interviewees and second to have these issues followed-up and elaborated on by the principals. There was an agreement among researchers, staff, and principals that the interviews with the principals should therefore always be conducted last at each school. Another related finding was that the interviews with the principals required twice as much time as others.

Our results revealed the importance of having interviewees who only worked partly in the case study school and who had contemporary experiences from other schools. In our study, there were three interviewees, who also worked in other schools: the special educator/SENCo (Special Educational Needs Coordinator) in school SED and the two social workers, one in each school. Our analysis of data showed that they provided important information about the two case study schools in relation to their experiences from the other schools they worked in. Therefore, we will ensure that these kinds of interviewees with employment in other schools are included in the main study.

2. Findings from the investigation of the pilot study (such as video-recorded documentation, and the way of handling ethical issues): taking a critical stance toward conducting a pilot study, which address the second aim.

It was found that the two interviewers had followed the procedures that had been rehearsed in advance of the school visits. Both appeared to create a calm and friendly atmosphere while interviewing. Their body language, according to the video recordings, did not involve any exaggerated gestures. After each main question was posed, there was a natural conversation where the interviewees talked most of the time. The interviewers were silent but attentive most of the time, showing their interest in the interviewees’ narratives mainly by using facial expressions indicating “I’m listening.” The main questions in the interview schedules were posed without being viewed as “leading” questions, and follow-up questions were asked in a neutral way. Some of the participants reported that they had had positive experiences related to the interviewers’ way of posing questions. For example, the teacher at school SED said:

it was quite nice when you said that you had already asked me about this but you wanted me to elaborate it more, rather than just following the protocol…as if you had kept repeating the same question over and over again even though I had already answered the question, instead you behaved in a positive way towards me.

Our analysis revealed mistakes that we as researchers made. Some were minor, such as wordings that we will correct in advance of the main study. Another type of mistake occurred during the interview with the pupil assistant in school SED. The interviewer did not fully comprehend the assistants’ description of his work with an “anonymous” pupil, which caused confusion for a part of the interview. This was, however, corrected during the interview and further explained by the pupil assistant during the feedback response session. Without doubt, part of this interview was of lesser quality than the other interviews. It took longer to get all questions answered. Our conclusion was that this mistake may be viewed as a natural consequence of what easily happens in social interaction, especially when the persons involved have different mother tongues as was the case in this instance. In the main study, we will try to shape the planning so that we have time in reserve after each interview.

3. Empirical findings about how the two schools work with special needs education, for instance, their work with special needs provision, and whether they have policy documents showing a commitment to work toward inclusion or not, which address the third aim.

The initial analysis focusing on the content of the interviews showed that the dominant part, approximately 80% of the text, focused on “Provision.” The other three areas were each of similar size in respect of the data obtained and accounted for 20% of the text.

The analysis, which extracted data through a process of thematic coding, showed that the three Swedish researchers’ coding were in full agreement in 48% of the cases. In 42% of the cases, the same code was used by two of the three researchers. In 10% of the cases, there was a significant disagreement as all three researchers chose different codes.

The gathered data were considered to be “rich” as the interviewees provided elaborated answers to interview questions. These data provide clear guidance for issues that will be investigated more thoroughly in the main study. When the same kind of issue is described in two different school settings, this may be viewed as a “commonality” (Rose & Shevlin, 2016), and such commonalities across schools will be investigated in more detail in the main study. One example from the analysis of the pilot study data will be used to exemplify one commonality among the two different types of schools, which is within the special needs education area. Despite our finding that the two schools had catchment areas that were very different from each other, both schools seemed to share the same kind of educational challenges. Their greatest concerns were not issues related to immigrant pupils or pupils with neuropsychiatric disorders but a combination of conditions such as autism and intellectual disability or an immigrant background in combination with an intellectual disability. Both schools described having problems with providing a good school situation for students described as intellectually in the “gray zone” and therefore not being accepted for a placement in “schools for children with learning disabilities.” This was an important commonality found in our pilot study, and it will be elaborated here to show the possibility of identifying commonalities among schools and to convey their different solutions.

The school SED principal said:

Something that is tricky is these pupils with a low intellectual ability and where their parents say no to a school placement in a special school for pupils with intellectual impairments, that’s the way that school legislation works, but they don’t get the right support (in a regular school) and I know that a special school placement would be good for them. Of course, we are trying other kinds of support for them, maybe get a pupil integration or something like that.

The staff in school LDU stated that they had difficulties in dealing with pupils with a low intellectual ability and that the school used “reverse integration” as a solution. This, for example, was practiced for a pupil with low intellectual functioning but being on a too high level to be enrolled in the special school for pupils with intellectual disabilities. The pupil was described as having difficulties with the lively environment in a classroom with many students. The school practiced a teaching strategy where two teachers were “co-teaching” (cf. Scruggs, Mastropieri, & McDuffie, 2007) two classes at the same time, which was the opposite of what the pupil needed.

Discussion

The pilot study we have undertaken and presented in this article comprises several approaches which will influence the main study, to pretest the interview format, and to interrogate the way to conduct the pilot study. When we started planning the pilot study, we suspected early on that we would not obtain much guidance from research publications and methodological literature. We decided to systematically document our research procedures, partly as our main study design requires a large team of researchers. Another motive was that the video recordings, the matrices, and the analysis procedures would be parts of the training material that would be used in the main study. This article also has served the role of being useful discussion material used between researchers positioned at universities in different countries. As such, it has been beneficial for our collaboration and has served the purpose of enhancing the research quality.

Apart from being of importance to our own research, we argue that our findings indicate the value of elaborated pilot studies where the intention is to conduct research with a wider population. Our first two aims were to investigate in a systematic way the design, preparation, and accomplishment of our own pilot study. Even as we had made efforts to prepare ourselves, we found during the interviews, during feedback sessions, in meta-discussions, and during the analysis of the interviews that many things could be improved with modest effort. Some improvements were made between the visit to school SED and the visit to school LDU. During the 5 weeks between visiting these schools, an initial analysis of data was made, and the research team had a meeting in Dublin where all researchers could contribute their experiences and competencies in preparation for the second school visit. Without doubt, the data collection was better in the latter visit. The interview schedules as a result were better adapted to Swedish schools. Even so, we found more aspects to improve in the interview schedules. These were to some extent related to the type of school we had visited and had been difficult to foresee. It was also expected that we would find areas for improvement as we had chosen largely different types of schools for our study.

Our main source of data was from interviews, and the quality of these data was dependent on the interviewers’ competence and the interview techniques that had been used. The two researchers followed the planned procedures such as working with an understanding of the interview format, informing the interviewees about the study, and securing informed consent at the interviews. Neither the researchers nor the interviewees seemed to be affected by the presence of the video camera directed toward the interviewers. It should be noted that both researchers have conducted many research interviews before this study, and this could partly explain why few mistakes were made. Another explanation was that the two researchers were well-prepared and had received help from the researchers who had conducted the IRIS study. It cannot be ruled out that the awareness of the video recordings also made the interviewers more alert, hence contributing to fewer mistakes.

When considering the overall situation, from the preplanning phase in advance of the school visits, the evaluation and planning period between the school visits, to the assessment and analysis period after the school visits, several amendments have been made that will improve the quality of the main study. The school visits served the purpose of updating us about the Swedish context and making us more aware of educational issues that will be important to investigate in relation to inclusive education. As we are using the idea of “commonality” (Rose & Shevlin, 2016), it was interesting to note the common challenges the two largely different schools faced in our study. These findings were unexpected as much of the present debate in Sweden focuses on pupils’ disruptive behavior and the prevalence of neuropsychiatric diagnoses (cf. Malmqvist, 2018). This kind of behavior, however, was not the most challenging issue in the schools that we visited. Behavioral challenges were, for example, managed by teachers’ “low affective behavior response” in accordance to Ross Greene’s theory (Greene & Ablon, 2012) or by referring pupils to PRUs. This means that inclusive practices and exclusive practices were available and could address this kind of problem. Instead, it was the issue of low intellectual ability among some pupils that posed most problems in the two schools or, more specifically, a low intellectual ability in combination with, for example, a neuropsychiatric disorder, social issues, or being “newly arrived” in Sweden. Some of the students could not be admitted to special schools as they had too high an intellectual capacity. Other students had lower intellectual capacity, but their parents did not want special schools for their children, which is in line with school legislation. School SED seemed to struggle with finding a solution, and the interviewees declared that there were few options and they did not succeed with these pupils. School LDU, on the other hand, had started to use “reverse integration.” One example presented earlier also showed that the school had put co-teaching with very large classes into practice, a consequence of which was that the classroom environment addressed the pupil’s needs even less and would probably make it impossible for the “reversely integrated” pupil to reenter the regular classroom.

In the main study, the pupil in the example above, from school LDU, would have been in focus in all interviews. Due to the restrictions already mentioned, it was not possible to use the research design in this regard. Neither was it possible to present interview data from parents in this article due to the risk of pupils being identified. Preparations will, however, be made in order to ensure high-quality observations and interviews with pupils and parents in the main study.

Conclusions

As a result of the elaborated pilot study, we have been able to both modify our instruments and improve the research design which will inform the substantive study. This was a valuable part of the piloting process whereby issues were identified related to the effectiveness of the instruments and the transferability of the model adopted by researchers working in Ireland. As a consequence of what was learned through piloting, we were enabled to gain greater understanding of the complexities of working within a previously designed model and the ways in which instruments could be modified to be appropriate for a specific research environment. Such modifications and understanding may well not have been achieved without giving such detailed attention to the pilot project stage, and the planning of the main research project may therefore have been less effective.

van Teijlingen and Hundley (2001) refer to researchers conducting qualitative research who at times suggest that separate pilot studies are not necessary. They also refer to research with qualitative interviews, where there are gradual improvements of interview schedules and specific questions which emerge during the process of data collection in main studies. We would agree that some research studies, due to philosophical underpinnings or explorative ambitions, may require such an approach. Exploratory research, whereby the investigators are attempting to gain a broad understanding of a phenomenon, may well benefit from a more iterative process, which does not require the elaboration of instruments. However, in studies which are seeking a depth of understanding rather than a broad perspective, the necessity to use instruments that are consistent and well tested increases confidence in the trustworthiness of the data that may be obtained (Bassey, 1999). In the research that we are conducting, and based on our findings and experiences from this study and other studies considered through our review of the literature, we believe that a well-planned and thoroughly conducted pilot study is not only important but necessary to ensure high research quality when a depth of understanding is sought. It is therefore surprising considering the vast number of qualitative research studies within education that it is so difficult to find research publications which provide discussions about conducting pilot studies. This absence of detailed discussion of pilot studies is not confined to educational research. Researchers from other disciplines (Kim, 2010; van Teijlingen & Hundley, 2001) report that a similar situation pertains across the humanities and social sciences.

A lack of appreciation about how to conduct pilot studies may be particularly problematic for PhD students—especially if they are not part of a research team or have a supervisor who is not fully engaged in the PhD student’s study. The ability to conduct an effective piloting of instruments is often a factor considered essential by those who examine doctoral theses (Holbrook, Bourke, Lovat, & Dally, 2004). The methodological literature currently provides little guidance in respect of how pilot studies have influenced reported investigations. We contend that a raising of awareness of the ways in which pilot studies can influence understanding and assist in shaping quality research is an issue which justifies greater debate within the published literature.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Johan Malmqvist

References

Arksey

O’Malley

(2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8, 19–32. doi:10.1080/1364557032000119616

Baker

T. L.

(1994). Doing social research (2nd ed.). New York, NY: McGraw-Hill.

Bassey

(1999). Case study research in educational settings. Buckingham, England: Open University Press.

Bryman

(2016). Social research methods (5th ed.). Oxford, England: Oxford University Press.

Bryman

Burgess

R. G.

(Eds). (1999). Qualitative research (Vol. 1–4). London, England: Sage.

Chenail

R. J.

(2011). Interviewing the investigator: Strategies for addressing instrumentation and researcher bias concerns in qualitative research. The Qualitative Report, 16, 255–262.

Denzin

Lincoln

Y. S.

(2005). The SAGE handbook of qualitative research (3rd ed.). Thousand Oaks, CA: Sage.

Denzin

Lincoln

Y. S.

(2011). The SAGE handbook of qualitative research (4th ed.). Thousand Oaks, CA: Sage.

Flick

(2014). Mapping the field. In Flick

(Ed.), The SAGE handbook of qualitative data analysis (pp. 3–18). Thousand Oaks, CA: Sage.

10.

Flick

(Ed.). (2018). The SAGE handbook of qualitative data collection. London, England: Sage.

11.

Greene

R. W.

Ablon

J. S.

(2012). Att bemöta explosiva barn [treating explosive kids]. Lund, Sweden: Studentlitteratur.

12.

Gudmundsdottir

G. B.

Brock-Utne

(2010). An exploration of the importance of piloting and access as action research. Educational Action Research, 18, 359–372. doi:10.1080/09650792.2010.499815

13.

Heyvaert

Hannes

Onghena

(2017). Using mixed methods research synthesis for literature reviews. Thousand Oaks, CA: Sage.

14.

Holbrook

Bourke

Lovat

Dally

(2004). Investigating PhD thesis examination reports. International Journal of Educational Research, 14, 98–120. doi:10.1016/j.ijer.2005.04.008

15.

Kim

(2010). The pilot study in qualitative inquiry: Identifying issues and learning lessons for culturally competent research. Qualitative Social Work, 10, 190–206. doi:10.1177/1473325010362001

16.

Malmqvist

(2018). Has schooling of ADHD students reached a crossroads? Emotional and Behavioural Difficulties, 23, 389–409. doi:10-1080/13632752.2018.1462974

17.

Paré

Trudel

M. C.

Jaana

Kitsiou

(2015). Synthesizing information systems knowledge: A typology of literature reviews. Information & Management, 52, 183–199. doi:10.1016/j.im.2014.08.008

18.

Pellegrino

(2014). A preliminary methodology and a cautionary tale, for determining how students seek research help online. Libraries and the Academy, 14, 187–196. doi:10.1353/pla.2014.0000

19.

Poggenpoel

Myburgh

(2003). The researcher as research instrument in educational research: A possible threat to trustworthiness? Education, 124, 418–421, 320.

20.

Pratt

J. M.

Yezierski

E. J.

(2018). A novel qualitative method to improve access, elicitation, and sample diversification for enhanced transferability applied to studying chemistry outreach. Chemistry Education Research and Practice, 19, 410–430. doi:10.1039/c7rp00200a

21.

Rose

Shevlin

(2016). The development of case studies as a method within a longitudinal study of special educational needs provision in the Republic of Ireland. Journal of Research in Special Educational Needs, 16, 113–121. doi:10.1111/1471-3802.12066

22.

Rose

Shevlin

Winter

O’Raw

(2015). Project IRIS—Inclusive research in Irish Schools. A longitudinal study of the experiences of and outcomes for pupils with special educational needs (SEN) in Irish Schools. NCSE Research Reports No. 20. Trim: National Council for Special Education.

23.

Sampson

(2004). Navigating the waves: The usefulness of a pilot in qualitative research. Qualitative Research, 4, 383–402. doi:10.177/1468794104047236

24.

Scruggs

T. E.

Mastropieri

M. A.

McDuffie

K. A.

(2007). Co-teaching in inclusive classrooms: A metasynthesis of qualitative research. Exceptional Children, 73, 392–416.

25.

Seale

Gobo

Gubrium

J. F.

Silverman

(Eds.). (2004). Qualitative research practice. London, England: Sage.

26.

van Teijlingen

E. R.

Hundley

(2001). The importance of pilot studies. Social Research Update, Winter, ISSN: 1360–7898.

27.

Vogel

Draper-Rodi

(2017). The importance of pilot studies, how to write them and what they mean (Editorial). International Journal of Osteopathic Medicine, 23, 2–3. doi:10.1016/j.ijosm.2017.02.001