Information Services and Use

A unique collaborative project to identify interactive enhancements to conventional-print journal articles, and to evaluate their contribution to readers’ learning and satisfaction.

It was hypothesized that (a) the enhanced article would yield more knowledge acquisition than the original article; (b) the interactivity aspects of the enhanced article would measurably contribute to the acquisition of knowledge; and (c) the enhancements to the original article would increase reader acceptance.

Fifteen SNMA medical students, assumed to have a greater generational familiarity and comfort level with interactive electronic media, reviewed 12 articles published in three Elsevier clinical and basic science journals. They used the Student National Medical Association’s asynchronous online discussion forum over a four month period to suggest desired enhancements to improve learning. “Prognostic Factors in Stage T1 Bladder Cancer”, published in the journal Urology was selected by the investigators as presenting the best opportunity to incorporate many of the students’ suggested interactive and presentational enhancements in the limited timeframe available prior to the established test date. Educational, statistical, and medical consultants assisted in designing a test protocol in which 51 second to fourth year medical students were randomly assigned to experimental and control conditions, and were administered either the original or enhanced interactive version of the article on individual computer workstations. Test subjects consisted of 23 participants in the control group (8 males, 15 females) and 28 participants in the experimental group (9 males, 19 females). All subjects completed pre- and post-test instruments which measured their knowledge gain on 30 true-false and multiple-choice questions, along with 7 Likert-type questions measuring acceptance of the articles’ format. Time to completion was recorded with the experimental group taking 22 min on average compared to 18 min for the controls; pre- and post-test times were 6 and 7 min, respectively. Statistical comparisons were based on change scores using either the Student t-test or the Two Way Analysis of Variance or Covariance. Significance was set at α = 0.05 or better.

Results on the dependent measure of knowledge acquisition showed no difference overall on the 30 questions, but learning gain was statistically significant for the subset of 10 questions that measured gain on content that was accessible by the user-invoked interactive features of the enhanced article. Further analyses revealed significant interactions by student year and gender. Second year students (11 in the control group, 8 in the experimental group) were the best performers in terms of knowledge acquisition from both articles. The female medical students received a larger learning gain from journal enhancements and interactivity components than their male counterparts. Acceptance overall was greater for the experimental group who rated the experience more favorably than the controls.

Failure to consider human factors such as gender and learning style may obscure underlying differences and their impact on the interactive aspects of scientific publications. Preliminary findings suggest the need for further study to include a heavier focus on interactivity apart from presentational enhancements; a more rigorous treatment of time as a specific variable; and an expanded experimental design that evaluates acquisition, understanding, integration and acceptance as dependent measures.

Implement and evaluate online user access to interactive datasets that complement online scientific papers.

Software development; website evaluation, including web log data monitoring, online user survey.

An expert group of respondents found the interactive data component to have positive impacts on understanding and discovery. However a significant minority reported that the software interface, instructions, and/or download presented a barrier to effective use of the interactive data functionality.

This exploratory study indicates that interactive datasets offer potential benefits as a complement to published papers, but that further software improvements and more in-depth evaluation are needed to best discern future directions.