An Exploratory Investigation of Levels of Learning and Learning Efficiency Between Online and Face-to-face Instruction

Abstract

This study examined differences in levels of learning and perceived efficiency between online (n = 22) and face-to-face (n = 32) graduate level subjects. Participants were master’s students enrolled in an accredited counselor preparation program. Findings indicated that students in both groups equally exhibited gains in levels of learning; however, there was a significant difference between groups in perceived learning efficiency, favoring the online instructional modality. Limitations regarding this study are noted, particularly related to the research design. Additional studies examining levels of learning and perceived efficiency of students enrolled in online courses are recommended.

Keywords

online learning counselor education learning efficiency

Learning efficiency and levels of learning are critical variables when addressing distance education. Yet, surprisingly there is a paucity of well-grounded research that focused on these concepts. Distance education has been supported due to its ease of access and convenience, yet few studies in graduate education examined learning efficiency of online courses when compared to face-to-face instruction (Means, Toyama, Murphy, Bakia, & Jones, 2010). In addition, mixed results surfaced when examining differences in levels of learning between these two instructional methods (Bacow, Bowen, Guthrie, Lack, & Long, 2012; Means et al., 2010). Research examining online and face-to-face instruction at the postsecondary level has also been criticized for its lack of rigor (Bowen, Chingos, Lack, & Nygren, 2012). These research findings are confounded by a lack of (a) adequate control groups, (b) random selection procedures, (c) consistent measures of dependent variables, and (d) clear descriptors of online instructional content.

There are also mixed results regarding differences in perceptions of online and face-to-face instruction. In a meta-analysis examining distance learning at the K-12 levels, Cavanaugh (2001) found similar levels of achievement between distance education programs and classroom instruction. Shachar and Neumann (2003) found that students taking distance education courses outperformed those enrolled in traditional courses and Bernard and colleagues (2004), in a meta-analysis, reported a positive effect size favoring achievement levels of online learners. However, Parsons-Pollard, Diehl Lacks, and Hylton-Grant (2008) found that face-to-face students out performed online students in overall performance including final grades, while online learners performed significantly better on a knowledge-base examination.

Enriquez (2010) found that students receiving online instruction had higher grades and higher scores on homework assignments than students taking face-to-face classes. Yet, Dell, Low, and Wilker (2010) found mixed results with online undergraduate educational psychology students performing significantly higher on a graded report assignment, but not on an analysis of learning project. Plumb and LaMere (2011) compared online instruction to face-to-face learning in undergraduate classes and found no significant differences between online and face-to-face students with respect to their performance on final exams or final grades in the course. Rich and Dereshiwsky (2011) researched face-to-face and online instruction using 100 students in four sections of an undergraduate intermediate accounting course. A number of homework assignments, serving as dependent variables, were used to measure student performance. Researchers found no significant differences in performance between students enrolled online when compared to face-to-face instruction. Also, Wagner, Garippo, and Lovaas (2011) studied a moderately sized sample of undergraduates enrolled in beginning level business courses. Face-to-face learning was compared to online using final grade percentages as the dependent variable. Researchers reported no significant differences in performance levels between students completing courses online when compared to face-to-face instruction.

The aforementioned researchers focused on differences in levels of learning between online and face-to-face instruction. Researchers produced mixed results with some favoring online instruction and others supporting face-to-face coursework. Research investigating efficiency of learning between online and face-to-face instruction was unavailable.

A limited number of experimental studies have been published that examine difference in online and face-to-face instruction with graduate students enrolled in counselor preparation programs. One factor influencing the paucity of research is the concern of whether online instruction is the most appropriate teaching modality for preparing counselors, particularly in the development of clinical skills (Murphy, MacFadden, & Mitchell, 2008). Further concerns (Moorhead, Colburn, Edwards, & Erwin, 2013) point to the lack of faculty prepared to develop online courses, as well as the limited number of counselor education programs that include coursework for training students to develop, teach, and research online courses. Despite these concerns, online distance education courses in counselor preparation programs have increased exponentially (Benshoff & Gibbons, 2011; Granello &Wheaton, 2004; Layne & Hohenshil, 2005; Quinn, Hohenshil, & Fortune, 2002; Reicherzer, Dixon-Saxon, & Trippany, 2009). Research focusing on online versus face-to-face instruction in counselor preparation graduate programs has fallen behind the growth of online programs in this academic area. The few experimental studies that have been published examined learning style differences, self-efficacy beliefs, and instructional preferences of counselors in training (Berry, Srebalus, Cromer, & Takacs, 2003; Flamez, B. N., 2010; Flamez, B., Smith, Devlin, Ricard, & Luther, 2009; Watson, 2012).

Several researchers have examined the process and content of online instruction in counselor training programs. For example, Palmer and McBride (2012) examined student and faculty interactions and general satisfaction levels with distance education in a graduate counseling program. The authors point to the 81 positive interactions by students enrolled in online courses as an indication of student satisfaction. In another study, Perry (2012) investigated online clinical supervision experiences of both students and site supervisors. By combining semi-structured interviews with a phenomenological analysis, the researcher concluded that graduate counseling students developed professional identities through online group interactions. Perry concluded that online interactions are different but in “no sense inferior to in-person relationships” (p. 65). Researchers have also surmised that online education can provide students with a protected, convenient, immediate, and a cost-saving platform that fosters the development of a global counselor identity (Sells, Tan, Brogan, Dahlen, & Stupart, 2012). However, there has been a lack of investigations that involve students assessing their efficiency of learning comparing online to face-to-face instruction.

In summary, because of the dearth of empirical research on the use of technology in preparing counselors (Nelson, Nichter, & Henriksen, 2010), there is a need for studies that investigate online instruction used in counselor preparation programs. More specifically, there is a need to investigate students’ levels of learning and learning efficiency. For the purposes of this study, level of learning was operationally measured by perceptions of students on a researcher-designed measure. Level of learning was, therefore, defined by the participants’ own perceptions of learning in online and face-to-face courses. In this study, efficiency of learning was operationally measured by perceptions of students on two researcher-designed instruments since standardized measures were unavailable. Efficiency of learning was defined by the participants’ own perceptions that involved a general assessment of their efficiency of learning in the course that could include an estimated amount of time it took to learn the course content and address course assignments. The following research questions directed this study:

Are there significant differences in learning efficiency?

Are there significant differences in levels of learning?

Are there significant differences in attained course objectives?

Are there significant differences in course ratings?

Method

Participants were enrolled in a counseling program accredited by the council for accreditation of counseling and related educational programs located at a midsize, regional university in the southwestern United States. Data were gathered from 54 master’s level graduate students registered in one of the two courses: (a) Introduction to Marriage and Family Counseling or (b) Developmental Issues in Human Personality and Behavior. Two groups of participants were analyzed to determine whether there were significant differences between students enrolled in a traditional face-to-face course (n = 32; 60%) and students enrolled in an equivalent online course (n = 22; 40%). All student data resulting from research measures were complete and included for analysis.

The entire sample consisted of 54 graduate students. Demographic characteristics are summarized in Table 1. The online class consisted of 22 participants (14 females and 8 males) with an average age of 33.27 years (SD = 8.08). Within this group, the number of White alone, Black/African American alone, Asian/White, and other was 13, 2, 1, and 6, respectively. Ethnic background was collapsed into two categories: (a) non-Hispanic/non-Latino/a and (b) Hispanic or Latino/a, since there were few ethnic groups represented beyond Hispanic, considering the sample was drawn from a Hispanic serving institution. Ten students identified as non-Hispanic/Non-Latino and 12 identified as Hispanic or Latino/a.

Table 1.

Participant Demographic Characteristics by Group.

	Online (n = 22)		Face-to-Face (n = 32)
Characteristics	n	%	n	%
Gender
Male	8	36.4	9	28.1
Female	14	63.6	23	71.9
Race
White alone	13	59.1	22	68.8
Black/African American alone	2	9.1	2	6.3
Asian alone	0	0.0	1	3.1
American Indian/Alaskan Native/White	0	0.0	1	3.1
Asian/White	1	4.5	0	0.0
Other	6	27.3	6	18.8
Ethnicity
Non-Hispanic or non-Latino/a	10	45.5	19	59.4
Hispanic or Latino/a	12	54.5	13	46.6

	M	SD	M	SD
Age	33.27	8.08	33.09	10.20

The traditional, face-to-face classes consisted of 32 participants (23 females and 9 males) with an average age of 33.09 (SD = 10.20). Within this group, the number of White alone, Asian alone, Black/African American alone, American Indian/Alaskan Native/White, and other was 22, 1, 1, 1, and 6, respectively. Eighteen students identified themselves as non-Hispanic/non-Latino/a and 15 students identified themselves as Hispanic or Latino/a.

Group differences on the basis of gender, χ² (1, N = 54) = .12, p = .73, and ethnicity (recorded as non-Hispanic/non-Latino/a or Hispanic/Latino/a), χ² (1, N = 54) = .53, p = .53, were not statistically significant. Group differences on the basis of age, face-to-face group (M = 33.09, SD = 10.12) and online group (M = 33.27, SD = 8.08), t (52) = .07, p = .94, were not statistically significant. It was concluded that the two groups were equivalent with respect to gender, ethnicity, and age.

Students were asked to participate in the study and complete consent forms. After giving consent, participants completed a demographic questionnaire. After the 15-week course, both sections (online and face-to-face) were administered three, brief assessments in a posttest-only format. In addition to assessments, students rated the degree to which course objectives were met and the overall course.

Instrumentation

Instruments utilized in this study were developed by the primary researcher due to the lack of measures assessing constructs under investigation. Therefore, a limitation to the study includes the lack of psychometric properties. The assessments measured counseling students’ perceptions of (a) efficiency of learning, (b) level of learning, (c) course objectives attained, and (d) general course ratings.

Student perceptions’ of efficiency of learning emphasized their overall assessment of efficiency of their own learning that could include the estimated amount of time it took to learn the course content and address all course assignments. The students’ perceptions’ of efficiency of learning were assessed and operationally defined by researcher-created measures. Participants were introduced to the instruments as a general assessment of their efficiency of learning in the course that could include an estimated amount of time it took to learn the course content and address course assignments. The rationale for creating the scales was two-fold: (a) the absence of measures to assess this construct and (b) the utilization of two scales allowed for a test of concurrent validity. Both surveys focused on efficiency of learning as related to the amount of time students devoted to class. The efficiency of learning survey included statements in a Likert-type format ranging from 1 (low) to 7 (high). The items included:

The amount of time devoted to this class;

The effort expected by me to understand material in this class;

My time was used efficiently in completing this class; and

The efficiency of learning in this class.

The sum of the items was utilized as a global score for efficiency of learning.

The second measure to assess efficiency of learning employed a semantic differential (Osgood, Suci, & Tannenbaum, 1957) model. The semantic differential measures the connotative meaning of words that represent objects, events, or concepts (Osgood et al., 1957). “Words are represented as points in space and the direction and magnitude of a word’s coordinate on each dimension corresponds to the nature (positive or negative on that dimension) and intensity of the reaction elicited by the word” (Xiong, Gordon, & Jeffrey, 2006, p. 1452). Osgood, Suci, and Tannenbaum (1957) explained that respondents rate variables or concepts on a series of 7-point, bipolar scales of meaning such as good/bad, large/small, or strong/weak. The connotations measure an individual’s attitude or meaning toward the topic in question. Test–retest reliability of the semantic differential is robust (r = .85; Osgood et al., 1957). Empirical reliability, validity, and utility of the semantic differential paradigm is supported (Snider & Osgood, 1969). The semantic differential is used widely to measure attitude or meaning, due in part to the adaptability of the various adjective items (Himmelfarb, 1993; Xiong et al., 2006).

The semantic differential is often used to measure concepts for which current instruments do not exist. Participants were introduced to this instrument as a measure of general assessment of course efficiency that could include an estimated amount of time it took to learn the course content and address course assignments. The semantic differential measure for this study included 4 items that assessed participant responses to the concept, course learning efficiency. Ratings were recorded on a 7-point scale. Each scale was anchored with two opposing adjectives: good/bad, weak/strong, positive/negative, and poor/great. Respondents circled a number in response to the four adjective pairs. Two items were reverse scored. The sum of all four items was utilized as a global score of course learning efficiency.

Level of learning was assessed by perceptions of students. The researchers’ intentionally assessed students’ perceived level of learning, using student perception as an outcome variable since previous investigations revealed minimal differences between online and face-to-face courses’ level of learning when employing grade point averages and test scores as dependent variables. Perceived level of learning was measured with a researcher-created instrument. The survey of learning included 4 items; responses were recorded on a 7-point Likert-type scale. Ratings ranged from 1 (low) to 7 (high). Items included:

The level of learning in this class;

My level of understanding of class content;

I know the material in this class; and

My knowledge of course material.

The sum of all items was utilized as a global score for level of learning.

Attainment of course objectives was measured at the completion of the courses with the use of surveys. Surveys were created and included a comparable 4-item, 7-point Likert-type format. Each scale ranged from 1 (low) to 7 (high). Respondents were instructed to “rate the level at which the course learning objectives were met.” Two items utilized for the Developmental Issues in Human Personality and Behavior course were (1) understanding theories of learning and personality development and (2) applying strategies for facilitating optimum development over the lifespan. Two items from the scale administered for the introduction to marriage and family course were (1) understanding family system theories and (2) understanding family life cycles, healthy family functioning, and family structures. The sum of each of the independent scales was utilized as a global score for course objectives attained.

The final survey developed by the primary researcher assessed a general course rating. This instrument was informed by the semantic differential format (Osgood et al., 1957) and included 5 items displayed on a 7-point Likert-type scale with opposing adjectives placed at each end. Participants were instructed to circle a number related to their perception of the concept entitled The Course.

Adjective pairs included fun/boring, dull/exiting, energizing/energy draining, joyless/enjoyable, and humorous/humorless. Three items were reversed scored. The sum of all 5 items was utilized as a global score for general course ratings.

Procedures

Following Institutional Review Board approval, participants were recruited from student groups enrolled in one of the two courses: (a) Introduction to Marriage and Family Counseling or (b) Developmental Issues in Human Personality and Behavior. Students were informed that participation was voluntary and 64 students chose to participate. Due to student attrition and class absences, 56 students remained in the study and ultimately 54 (96%) of the 56 students completed the posttests. Prior to the first class session, students were given the option to choose a face-to-face or online course format. On the final day of class, posttests related to student perceptions of learning and efficiency of learning were distributed to the face-to-face and online classes. Students were permitted as much time as required to complete the surveys. Data were collected and global scores for each of the posttests were calculated.

The face-to-face courses (Introduction to Marriage and Family Counseling and Developmental Issues in Human Personality and Behavior) were cotaught by the primary and secondary authors with the secondary author also teaching the online equivalent courses. Online and face-to-face course formats and content were identical. The online and face-to-face courses included the same lectures, video clips, discussion papers, assignments, exams, and grading systems. In addition, both modalities included the same syllabi, textbook, Power Point presentations, group activities, and student presentation rubrics.

Results

Student perceptions of online and face-to-face learning were explored utilizing SPSS software. Descriptive statistics for each groups’ posttest mean scores were calculated (see Table 2). In four of the five measures, the posttest mean scores exhibited minimal disparity between groups, ranging from .38–1.01, indicating that student perceptions of online versus face-to-face learning were generally similar. However, the mean score difference for the efficiency of learning subscale was recorded at 3.39, supporting online instruction.

Table 2.

Mean Scores for Face-to-Face and Online Posttests.

	Online Group (n = 22)		Face-to-Face Group (n = 32)
Post test variables	M	SD	M	SD
Efficiency of learning: Likert scale	23.95	2.80	20.56	3.93
Course learning efficiency: semantic differential scale	23.82	3.29	23.22	4.46
Level of learning	23.45	3.46	22.44	2.85
Course objectives met	24.14	3.09	23.22	2.17
General course rating	26.09	5.42	26.47	5.21

A multiple analysis of variance (MANOVA) was utilized to compare online group scores with face-to-face group scores. A 2 (Instruction Method) × 5 (Perceptions of Learning) model was employed. A significant difference between groups was found, Wilk’s λ = .75, F(5, 48) = 3.14, p = .02. Effect size was large, η² = .25. According to Cohen (1988), η² effect size of .01 is interpreted as small, .06 (medium), and .17 (large). This result suggested a significant difference in student perceptions between online and face-to-face groups. Analyses of variances (ANOVAs) on the dependent variables were conducted as follow-up tests to the MANOVA. An effect size was calculated to determine practical significance (Field, 2005). Using the Bonferroni method, each ANOVA was tested at the .01 level. The ANOVA on efficiency of learning scores was significant, F(1, 54) = 12.01, p = .001, η²= .19. The ANOVAs on other posttest measures were not significant: course learning efficiency, F(1, 54) = .29, p = .59, η²= .01; level of learning, F(1, 54) = 1.39, p = .24, η²= .03; course objectives met, F(1, 54) = 1.65, p = .21, η²= .03; and general course rating, F(1, 54) = .07, p = .79, η²= .001.

The significance and large effect related to efficiency of learning, as measured by the researcher-generated Likert-type scale supported the online course format. Due to the dearth of previous research on efficacy of online and face-to-face learning, it is difficult to compare these findings to earlier studies. Significant differences were not found, however, between groups in the second learning efficiency survey. This measure was formatted in accordance with the semantic differential (Osgood et al., 1957). In this assessment, the concept of course learning efficiency was placed above the bipolar adjective scales. No specific references were made to time and effort within the items as in the Likert-type survey. Finally, the remaining assessments did not demonstrate statistically significant differences between groups. However, in comparing mean scores of the level of learning survey, there was a slight trend favoring the online course. A similar trend was evident in the course objectives survey: Students rated the online course with slightly higher scores. General course ratings were congruent between the two groups, slightly favoring the face-to-face format.

Discussion

Although studies comparing online and face-to-face instruction in graduate counselor preparation programs have been limited, widespread interest accompanied by a phenomenal growth of online offerings in counselor training programs remains. This study focused on two significant variables pertaining to distance education: efficiency of learning and levels of learning. Students completing online instruction perceived efficiency of learning to be significantly better than that offered through face-to-face instruction. However, a number of factors need to be considered in the context of these findings. The first factor involves students’ perception of the method of instruction. General technology presents itself as a method to expedite communication and effective interaction. Perhaps, students were responding to this phenomenon rather than actual efficiency of learning. It is believed that further research can clarify this issue by using mixed methods paradigms that include a qualitative phase asking participants to respond to queries regarding efficiency of learning and what this concept meant to them. A second factor to consider when discussing significant findings as related to efficiency of learning involves teacher–student interactions. Although not quantified in this investigation, perhaps the online professor had more frequent interactions with students who took courses online when compared with face-to-face instruction. Perhaps, there was a greater level of consistency of interactions that took place during online instruction. In face-to-face teaching, the instructor met with students once per week with minimal interaction between classes, whereas online interactions were timely and ongoing throughout the week between instructor and students. This point again can be explored via qualitative interviews with students in online coursework.

Findings on levels of learning from this investigation add to the existent literature on studies comparing online and face-to-face instruction. This study, and those previously conducted, point to the need for a clearer definition of levels of learning, performance, and student outcome. Early research studies focused on levels of learning and performance between online and face-to-face instruction found both modalities to be effective. Perhaps, it is time that we consider disembarking from attempts to prove one modality as superior to another and simply agree that both methods of instruction have been effective thus allowing researchers to focus on other variables. Perhaps, future research can focus on best approaches when teaching from each modality by examining concepts such as time investment, confidentiality, student self-awareness, and technological expertise (Haberstroh, Duffey, Evans, Gee, & Trepal, 2007). Another area of study involves use of blended courses and how they can be effectively implemented and formatted (Conn, Roberts, & Powell, 2009; McMillen & Pehrsson, 2009; Rambo-Igney & Brinthaupt, 2008).

Implications for Practice

Based on this study’s findings, there are several implications for practice. First, counselor education programs can continue to incorporate online coursework or technology in face-to-face coursework. Varied teaching modalities and technology (Renfro-Michel, O’Halloran, & Delaney, 2010) might benefit some students who are (a) independent thinkers, (b) have different learning styles, and (c) self-motivated to work outside of class. Also, counselor educators can inform students about potential positive and negative consequences of online coursework. As evidenced in this study, one possible benefit might include increased learning efficiency. When students are aware of direct benefits, they will be more likely to make an informed decision based on research and not convenience. Finally, counselor educators should continue to identify specific types of learners and coursework that are conducive to an online instruction format. We contend that certain coursework might be inappropriate for online instruction (e.g., statistics) and certain students might benefit from face-to-face coursework.

Conclusions

This study examined differences in learning and perceived learning efficiency between students enrolled in online and face-to-face coursework. Findings indicated a lack of difference in level of learning and a significant difference in efficiency of learning that favored the online instructional method. Due to the limitations of this study, implications made from significant findings related to efficiency of learning should be treated with caution. The small sample size and lack of randomization, as well as missing comparative precourse measures, are recognized as important limitations. However, the investigation’s focus on efficiency of learning between online and face-to-face instruction can contribute to future research of these two instructional modalities. It is hoped that significant findings of this study will encourage future investigations using a variety of research designs to further examine efficiency of learning between online and face-to-face instruction.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Bacow

L. S.

Bowen

W. G.

Guthrie

K. M.

Lack

K. A.

Long

M. P.

(2012). Barriers to adoption of online learning systems in U.S. higher education. ITHAKA S+R. Retrieved from http://www.sr.ithaka.org/sites/default/files/reports/sr-ithaka-interactive-learning-online-at-public-universities.pdf

Benshoff

J. M.

Gibbons

M. M.

(2011). Bringing life to e-learning: Incorporating a synchronous approach to online teaching in counselor education. The Professional Counselor: Research and Practice, 1, 21–28. Retrieved from http://tpcjournal.nbcc.org/wp-content/uploads/2011/03/Benshoff.pdf

Bernard

R. M.

Abrami

P. C.

Lou

Borokhovski

Wade

Wozney

… Huang

(2004). How does distance education compare to classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74, 379–439. doi:10.3102/00346543074003379

Berry

Srebalus

D. J.

Cromer

P. W.

Takacs

(2003). Counselor trainee technology use skills, learning styles, and preferred modes of instruction. Journal of Technology in Counseling, 3. Retrieved from http://jtc.columbusstate.edu/vol3_1/

Bowen

W. G.

Chingos

M. M.

Lack

K. A.

Nygren

T. I.

(2012). Interactive learning online at public universities: Evidence from randomized trials. ITHAKA S+R. Retrieved from http://www.sr.ithaka.org/researchpublications/interactive-learning-online-public-universities-evidence-randomized-trials

Cavanaugh

C. S.

(2001). The effectiveness of interactive distance education technologies in K-12 learning: A meta-analysis. International Journal of Telecommunications, 7, 73–88. Retrieved from http://www.ijates.org/index.php/ijates

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd Ed.). Hillsdale, NJ: Erlbaum.

Conn

S. R.

Roberts

R. L.

Powell

B. M.

(2009). Attitudes and satisfaction with a hybrid model of counseling supervision. Educational Technology & Society, 12, 298–306. Retrieved from http://www.ifets.info/journals/12_2/21.pdf

Dell

Low

Wilker

(2010). Comparing student achievement in online and face-to-face class formats. Journal of Online Learning and Teaching, 6, 30–42. Retrieved from http://jolt.merlot.org/vol6no1/dell_0310.htm

10.

Enriquez

A. G.

(2010). Enhancing student performance using tablet computers. College Teaching, 58, 77–84. doi:10.1080/87567550903263859.

11.

Field

(2005). Discovering statistics using SPSS (2nd Ed). London, England: Sage.

12.

Flamez

B. N.

(2010). A comparison of online and traditional graduate counseling courses: Learning style, instructional preferences, and educational environment (Doctoral dissertation). Retrieved from ProQuest. (3424444)

13.

Flamez

Smith

R. L.

Devlin

J. M.

Ricard

Luther

M. S.

(2009). Learning styles and instructional preferences: A comparison of an online and traditional counseling course. Journal of Technology in Counseling, 5. Retrieved from http://jtc.columbusstate.edu/Vol5_1/Flamez.htm

14.

Granello

D. H.

Wheaton

J. E.

(2004). On-line data collection: Strategies for research. Journal of Counseling and Development, 82, 387–393. doi:10.1002/j.1556-6678.2004.tb00325.x

15.

Haberstroh

Duffey

T. D.

Evans

Gee

Trepal

(2007). The experience of online counseling. The Journal of Mental Health Counseling, 29, 269–282. Retrieved from http://essential.metapress.com/content/j344651261w357v2/

16.

Himmelfarb

(1993). The measurement of attitudes. In Eagly

A. H.

Chaiken

(Eds.), Psychology of attitudes, (pp. 23–88). Belmont, CA: Thomson/Wadsworth.

17.

Layne

C. M.

Hohenshil

T. H.

(2005). High tech counseling: Revisited. Journal of Counseling and Development, 83, 222–226. doi:10.1002/j.1556-6678.2005.tb00599.x

18.

McMillen

P. S.

Pehrsson

(2009). Improving a counselor education web site through usability testing: The bibliotherapy education project. Counselor Education and Supervision, 49, 122–136. doi:10.1002/j.1556-6978.2009.tb00092.x

19.

Means

Toyama

Murphy

Bakia

Jones

(2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, DC: U.S. Department of Education Office of Planning, Evaluation, and Policy Development.

20.

Moorhead

H. H.

Neuwer-Colburn

A. A.

Edwards

N. N.

Erwin

K. T.

(2013). Beyond the myth of the pajama party: Delivering quality online counselor education and supervision. VISTAS, 2. Retrieved from http://counselingoutfitters.com/vistas/vistas13/article_33.pdf

21.

Murphy

MacFadden

Mitchell

(2008). Cybercounseling online: The development of a university- based training program for e-mail counseling. Journal of Technology in Human Services, 26, 447–469. doi:10.1080/15228830802102081

22.

Nelson

J. A.

Nichter

Henriksen

(2010). On-line supervision and face-to-face supervision in the counseling internship: An exploratory study of similarities and differences. Retrieved from http://counselingoutfitters.com/vistas/vistas10/Article_46.pdf

23.

Osgood

C. E.

Suci

G. J.

Tannenbaum

P. H.

(1957). The measurement of meaning. Champaign, IL: The Board of Trustees of the University of Illinois.

24.

Palmer

McBride

(2012). Assessing student and faculty satisfaction in a master of counseling distance education paradigm. (Online Submission). Retrieved from ERIC. (ED533905).

25.

Parsons-Pollard

Diehl Lacks

Hylton Grant

(2008). A comparative assessment of student learning outcomes in large online and traditional campus-based introduction to criminal justice courses. Criminal Justice Studies, 21, 239–251. doi:10.1080/14786010802355826

26.

Perry

(2012). Constructing professional identity in an online graduate clinical training program: Possibilities for online supervision. Journal of Systemic Therapies, 31, 53–67. doi:10.1521/jsyt.2012.31.3.53

27.

Plumb

LaMere

(2011). Comparing student learning in a required electrical engineering undergraduate course: Traditional, face-to-face vs online. Paper presented at the Annual Conference of the International Network for Engineering Education and Research, Belfast, England.

28.

Quinn

A. C.

Hohenshil

T. H.

Fortune

(2002). Utilization of technology in CACREP approved counselor education programs. Journal of Technology in Counseling, 2, 2. Retrieved from http://jtc.columbusstate.edu

29.

Rambo-Igney

B. C.

Brinthaupt

T. M.

(2008). Best practices of college teaching connect with hybrid online instruction in counselor education. Journal of Technology in Counseling, 5, 1–15. Retrieved from http://jtc.columbusstate.edu/Vol5_1/RamboBrinthaupt.htm%20

30.

Reicherzer

Dixon-Saxon

Trippany

(2009). Quality counselor training in a distance environment. Counseling Today, 51, 46–47. Retrieved from http://ct.counseling.org/

31.

Renfro-Michel

E. L.

O’Halloran

K. C.

Delaney

M. E.

(2010). Using technology to enhance adult learning in the counselor education classroom. Adultspan Journal, 9, 15–24. doi:10.1002/j.2161-0029.2010.tb00068.x

32.

Rich

A. J.

Dereshiwsky

J. I.

(2011). Assessing the comparative effectiveness of teaching undergraduate intermediate accounting in the online classroom format. Journal of College Teaching and Learning, 8, 19–27. Retrieved from http://www.cluteinstitute.com/ojs/index.php/TLC/article/view/5641/5722

33.

Sells

Tan

Brogan

Dahlen

Stupart

(2012). Preparing international counselor educators through online distance learning. International Journal for the Advancement of Counseling, 34, 39–54. Retrieved from http://www.editlib.org/p/110612

34.

Shachar

Neumann

(2003). Differences between traditional and distance education academic performances: A meta-analytic approach. Journal of Online Learning and Technology, 6, 318–334. Retrieved from http://jolt.merlot.org/vol6no2/shachar_0610.pdf

35.

Snider

J. G.

Osgood

C. E.

(1969). Semantic differential technique: A sourcebook. Chicago, IL: Aldine.

36.

Wagner

S. C.

Garippo

S. J.

Lovaas

(2011). A longitudinal comparison of online versus traditional instruction. MERLOT Journal of Online Learning and Teaching, 7, 68–73. Retrieved from http://jolt.merlot.org/vol7no1/wagner_0311.pdf

37.

Watson

J. C.

(2012). Online learning and the development of counseling self-efficacy beliefs. The Professional Counselor: Research and Practice, 2, 143–151. Retrieved from http://tpcjournal.nbcc.org/wp-content/uploads/2012/09/online-learning_Watson-Manuscript-p143-151.pdf

38.

Xiong

M. J.

Gordon

G. D.

Jeffrey

J. F.

(2006). Testing the semantic differential as a model of task processes with implicit association test. Memory & Cognition, 34, 1452–1463. doi: 10.3758/BF03195910