What Is Found Positive in Healthcare Information and Communication Technology Implementation?—The Results of a Nationwide Survey in Finland

Abstract

Objective: Considerable expectations have been placed on information and communication technology (ICT) in improving the processes and quality of healthcare. Our purpose was to find out which element is found positive in healthcare ICT implementation. Materials and Methods: An online questionnaire on e-Health implementation submitted to all Finnish public health service providers and a sample from the private sector included an open question about which the electronic working methods, systems, or applications have most positively influenced the fluency or quality of service processes. Results: The electronic health record was mentioned as an item that has positive influence by 52% of the respondents from the hospital districts, 27% of those from the primary healthcare centers, and 38% of those from the private providers. Digital radiology systems (including teleradiology) were mentioned by 52% of the hospital districts and 27% of the primary healthcare centers. The figures for digital laboratory systems (including telelaboratory) were 5% and 11%, respectively. The figures for teleradiology itself were 5% for the hospital districts and 15% for the primary healthcare centers; the figures for telelaboratory systems were 5% and 9%, respectively. Conclusions: The specialized healthcare seem to experience intraorganizational electronic services integrated to the electronic health record, such as digital radiology and laboratory services as exerting a positive influence, whereas the primary healthcare find such influence from different functions such as interorganizational data exchange and telemedicine services. These might indicate where the efforts should be focused when implementing ICT in healthcare.

Introduction

At the strategic level, information and communication technology (ICT) has been seen as a key factor and positive influence regarding the improved accessibility, quality, and efficiency of healthcare service delivery.^1

–6 At the operational level, the key tool is the implementation of a multifunctional electronic health record (EHR) system.^7

–12

Finland is scarcely populated, the population density being on average 16 persons per square km. In the northern parts where about one-tenth of the population lives, the population density is only 1–10 per square km and inhabitants' distances to their specialty hospitals may be 80–500 km.¹³ Four hospital districts out of 21, which are responsible for public specialized healthcare, and 45 primary healthcare centers (among them 39 rural and countryside) out of 229 are situated in the northern part of the country.¹³ These facts have accelerated implementation of telemedicine and e-health services in Finland, where the Ministry of Social Affairs and Health has continuously regulated and monitored the implementation of ICT in healthcare. Currently, an EHR is in use comprehensively at all levels of the Finnish healthcare system.^14

–17

The EHR systems are multifunctional and include patient administration, continuous narrative documents of diagnostics, treatment and care, order entries for laboratory and radiological exams, tools for reviewing their results, medication lists for generating prescriptions, interorganizational data exchange including telemedicine services, and at least optionally, a decision support system. The narrative, image, and biometric data of these EHRs are highly integrated with a unique nationwide patient identification code.

Broadband networks cover in practice the whole country, and secured communications for patient privacy are available for healthcare everywhere. ICT literacy is high among healthcare professionals.¹⁵ A typical remote northern community is, for instance, the city of Pudasjärvi (9,000 inhabitants) located 90 km from a central hospital, exploiting ICT for versatile data transfer with the specialized care and offering telepsychiatry and teledermatology services to citizens.

The transition from paper-based records to EHRs took place in Finnish primary healthcare centers during the late 1990s, and the usage of EHR is currently 100%. For hospitals, where the current usage is also 100%, the transition took place after 2000. For the private physician service providers, the EHR coverage, based on the weighted sampling, was 89%.¹⁷ Table 1 shows the rapid implementation of ICT systems in the Finnish public healthcare sector, which provides 85% of healthcare services. The coverage of many of them has doubled in recent years.

Table 1.

Progress of Information and Communication Technology Systems and Applications in Finnish Public Healthcare

	Public Healthcare
	Primary Healthcare Centers % (N = 229)			Hospital Districts % (N = 21)
System or Application User Organizations	2003	2005	2007/2008	2003	2005	2007/2008
Comprehensive integrated EHR	94	96	100	48	95	100
PACS	17	No data	49	48	71	100
Regional data exchange^a	38	64	72	62	95	95
E-referral letters	24	45	77	48	76	90
Electronic consultations^b	No data	35	55	No data	48	52
Televideo conferencing
Clinical use	13	12	17	33	48	67
Educational use	13	25	32	76	76	95
Teleradiology	10	29	61	62	76	81
Telelaboratory	38	64	72	43	90	90

Concerns narrative text, laboratory results, and linkage to PACS when appropriate.

Mode of action when general practitioner requests specialist's opinion without transferring patient to hospital.

EHR, electronic health record; PACS, picture archiving and communication systems.

Electronic decision support systems and clinical guidelines, accessed either via computer desktop or by manual navigation from the EHR, are in comprehensive use. All specialized hospitals are filmless. Picture archiving and communication systems (PACS) have been integrated with radiology information systems (RIS) and hospital information systems.

The coverage of services targeted for information exchange between healthcare organizations and patients (e-health systems for citizens) is low compared with those meant for professionals only, but is constantly increasing. The direct electronic ordering of an appointment for certain services was utilized in 38% of the hospital districts, in 8% of the primary healthcare centers, and in a third of the sample of the private service providers.

Electronic communications between patients and organizations by short message service (SMS) or e-mail was uncommon in 2008. The usage level of different applications (appointments, browsing laboratory results, and similar functions) varied between 5% and 10%.¹⁷ In 2010, the Finnish government has begun to fund development programs designed to enhance citizen participation by launching a nationwide Electronic Democracy program (SADe).¹⁸ It is expected that the usage level of electronic tools in patient–healthcare system communications will increase.

The comprehensive implementation of ICT for widespread telemedicine and e-health applications has now taken place within Finnish healthcare. Consequently, experiences on the change from paper-based system to electronic tools have begun to accumulate, and it is interesting to know which of the tools were seen to have the most positive influences. This question was asked in connection with the latest data collection for the national e-health report.¹⁷

Materials and Methods

The data collection about the implementation of ICT systems took place in late 2007 and early 2008 with a structured online questionnaire distributed to all medical directors of public health service providers and a weighted sampling of private providers.¹⁷ Because it was an official survey authorized by the Ministry of Social Affairs and Health, the respondents were asked to answer the questions as representatives of their organizations. If necessary, a reminder was sent after a couple of weeks, and if needed, phone calls were also made.

The questionnaire included an open question: “Which electronic working methods, systems, or applications used in your organization have, in your opinion, most positively influenced the fluency or quality of service processes?” The responses were analyzed by a qualitative content analysis method.¹⁹ The authors first read the responses separately and then made proposals for a common classification. One of the authors found 11, the second 15, and the third 23 different categories. Among them were 9 common categories, 2 held in common by two of the authors, and 11 proposed by one. The authors' discussions resulted in the nine categories presented in Table 2, after which separate proposals for inserting the replies in the agreed categories were made. Of the 246 responses, all three authors classified 185 (75.2%) of the responses at the level of the categories uniformly, two of the authors agreed on 60 (24.4%), and all three disagreed on the classification of 1 (0.4%) response. Subsequently, the authors agreed on the categories' classifications.

Table 2.

List of Electronic Working Methods, Systems, or Applications with the Most Positive Influences Mentioned by Respondents from Public Healthcare Organizations, the Number of Mentions, Their Prevalence, and the Statistical Analysis of Differences Between the Hospital Districts and Primary Healthcare Centers

	Positive Mentions/Users (%) of Application		Statistical Analysis
Method, System, or Application	Hospital Districts (N = 21)	Primary Healthcare Centers (N = 198)	Fischer's Exact Test
EHR system	11/21 (52)	57/198 (29)	p = 0.04
Digital radiology and related systems (PACS)^a including teleradiology	11/21 (52)	31/198 (16)	p = 0.003
Digital laboratory-related systems including telelaboratory	4/19 (21)	22/198 (11)	p = N.S.
Electronic referral-feedback system	1/19 (5)	38/164 (23)	p = 0.082
Regional information system	2/17 (12)	14/127 (11)	p = N.S.
Electronic patient management system	1/21 (5)	17/198 (9)	p = N.S.
Decision support system	0/21 (0)	15/198 (8)	p = N.S.
Other issues mentioned (see text for details)	6/21 (29)	17/198 (9)	p = 0.013
Cannot say or no answer	5/21 (24)	91/198 (46)	p = 0.065

Includes any application of digital radiology from single units to hospital-wide systems.

N.S., not significant.

Fischer's two-sided exact test was used to test the differences between the responses of the healthcare centers and the hospital districts.

Results

Responses to the questionnaire were obtained from all 21 Finnish hospital districts or specialized public healthcare units, 198 (87%) of the 229 Finnish primary healthcare centers, and 28 (62%) from the sampling of 45 private service providers.

The responses from the public sector are summarized in Table 2. The respondents mentioned 0–6 different positive issues. At least one item exerting a positive influence was mentioned by 16 (76%) of the hospital districts' respondents, 107 (54%) of the primary healthcare centers' respondents (besides 2 respondents mentioning only negative items), and 11 (46%) in the sample of private care providers.

Half of the hospital districts mentioned EHR and digital radiology including PACS as systems that had been experienced as exerting a positive influence. Only one district from the southern part of the country mentioned teleradiology specifically. Digital laboratory systems were rated in third place, but with a prevalence of one fifth. One respondent mentioned telelaboratory services.

Certain systems or applications received fewer mentions. In the hospital districts, there were (expressed as a user/existing system) two for cellular phones (2/21) and one each for digital dictation (1/14 assessed from 2005 data), electronic consultations (1/11), SMS messaging with patients (1/3), SMS message calls to professionals in emergency cases (1/no data), and hospital infection information system (1/no data).

In the primary healthcare centers, EHR systems were mentioned most often, but only by about a quarter of the respondents. The electronic referral feedback system had almost the same prevalence. Teleradiological services were cited by 18 (15%) of the 124 respondents utilizing them. Among those 18, there were 7 rural primary healthcare centers from the northern part of the country. Several different methods or applications were mentioned by about a tenth of the users: regional information systems, decision support systems, and electronic patient management systems (Table 2). Telelaboratory services were mentioned by 14 (9%) of the 148 respondents who utilized them. Of those 14 primary healthcare centers, 6 were both rural and situated in the northern part of the country. Among the 15 mentions concerning decision support systems, 13 concerned the national stand-alone online database Terveysportti for medical information and guidelines.

The primary healthcare centers respondents also mentioned e-mailing with patients (4/30), electronic consultations (3/110), televideo consultations (3/33), televideo education (2/61), and digital dictation (2/81 assessed from 2005 data), and there were single mentions of SMS messaging with patients (1/33), speech recognition (1/14), cellular phones (1/no data), remote homecare systems (1/no data), mobile phones (1/no data), multiprofessional collaboration (1/no data), and even the HL7 data transfer standard (1/75). Despite the request for positive influences only, one respondent stated that ICT had a negative influence on his work, and another stated that ICT had only increased the physicians' workload.

Among the sampled private service providers, the most favored system was an EHR (9 mentions), whereas digital radiology, electronic patient management systems, and electronic appointments achieved 3 mentions and the rest fewer.

Discussion

According to recent reports, Finland appears to be the global leader in the implementation of multifunctional EHR in both primary and specialized healthcare.^20,21 This pervasive EHR usage was reflected in the responses. EHR systems were most often mentioned as having a positive influence by all provider groups: hospital districts, primary healthcare centers, and private providers. The execution of modern medical services requires systems capable of providing a degree of mastery of data and coordination, improving safety, and supporting the means to identify changes in clinical status.² To achieve this, a highly developed multifunctional EHR holds the key position. It can improve the performance of healthcare through data management, offering easy access to patient information depositories, tools to execute actions for diagnostics and treatment, and information sharing between professionals.^8,10
–12

The relative prevalence of the positive impact of EHR was, however, twice as high in specialized healthcare as in primary healthcare. The relative prevalence of positive experiences with digital radiology and PACS, as well as digital laboratories, was also much higher in hospitals than in primary care. One possible explanation may be that in specialized healthcare, several departments simultaneously create and utilize patient documents in different parts of a hospital building. This means that in the era of paper-based systems and X-ray films, accessing patient data (narrative texts, imaging data, and laboratory results) was time consuming. In terms of accessibility, the multifunctional, integrated EHR represents centralized storage combined with the convenient management of patient records. Various studies have shown that PACS and RIS decrease radiology report turnaround times, thus making the imaging results more readily available. The integration of RIS into hospital information systems and EHR also makes digital order entries possible and further improves the workflow.^22
–24 In the Finnish context, all imaging results are available through the integrated EHR, thus enabling the physician in charge of care to manage the patient's information.^25,26 Various articles in the current literature discuss the importance of this integrated approach when providing high-quality services.^27,28

Because primary healthcare centers are generally small units, the transfer from paper-based to electronic systems locally has not significantly affected any changes in workflow. Several primary healthcare centers have used EHR for so many years and thus the effects of the switchover from paper-based to digital processes are difficult to recall.

In the hospital districts' responses, the main emphasis seems to be on intraorganizational systems such as digital radiology and laboratory services. Telemedicine services such as teleradiology and telelaboratory services, regional information systems, electronic referral feedback, and electronic consultation systems were mentioned only sporadically.

In the primary healthcare centers, telemedicine services or interorganizational data exchange appear to have evoked positive experiences. Primary healthcare centers alone cannot provide demanding specialist or examination services; they must be purchased from specialized healthcare. To accomplish this, telemedicine offers a time-saving, better coordinated, and more fluent way of working.²⁹ Electronic referral and discharge letters transmitted directly via the EHR facilitates fluent communications with the specialized care.³⁰ Teleradiology and telelaboratory services seemed to be mentioned as positive, especially at remote countryside healthcare centers.

Decision support systems have the potential to improve quality and performance in healthcare.^2,10,12 These were mentioned to a certain extent by the primary healthcare respondents, but not at all by those from specialized healthcare. This can be explained by the fact that the respondents in the hospital districts were administrative medical directors or information managers, but in the primary healthcare centers the medical directors were often also clinicians who require decision support systems in their practices.

Because electronic communication between patient and care provider was still in its formative stages at the time the survey was conducted, only a few organizations were able to provide any feedback on this item. Even so, some positive mentions were obtained. New tools may change the patient's role in healthcare services. This is an important developmental area requiring focused research.

Responding to an open question in a 2003 study conducted by the Finnish Medical Association among a random sampling of its members, a total of 2,060 respondents expressed their opinions concerning the positive and negative changes in their work. Among the 480 physicians (23%) who mentioned changes resulting from the use of information technology, 327 (68%) mentioned only positive changes such as improved efficiency and faster information retrieval. Positive and negative changes were mentioned by 44 (9%) respondents, and 109 (23%) mentioned only negative aspects. The latter group mentioned issues such as an excessive proportion of consulting hours taken up with electronic data management, the disturbance of contact with patients, and information overload.³¹ Owing to the pace of technological development, however, the experiences reported in 2003 and 2007/2008 are not fully comparable.

Healthcare provider organizations also cited the negative or challenging influences of ICT in healthcare through their answers to another question in the Finnish ICT implementation study.¹⁷ In Finland, the construction of a national ICT architecture in healthcare is currently ongoing. As planned, the new system will be in use (at least partially) in 2011.^32,33 Because many of the problems and negative effects cited related to the framework of the new architecture's implementation process, the results cannot be generalized as negative experiences pertaining to the present ICT systems or applications. The negative experiences published in the Finnish Medical Journal can be briefly summarized as follows.³⁴ Hospital districts most frequently reported issues related to information system compatibilities and tight timetables; primary healthcare centers reported insufficient resources, inadequate finances, and a shortage of the expertise required for implementation processes. The challenges for the private providers most often concerned what they considered inadequate directions from public authorities and insufficient financial resources.

The number of healthcare providers who mentioned the positive influences of electronic working methods, systems, or applications in healthcare was <50% of all respondents for all applications other than the EHR systems and digital radiology in specialized care. Other responses were scattered and a wide range of telemedicine and e-health tools were found to have a positive effect by certain respondents. It is worth noting that respondents were asked to answer a single open question concerning the issues that they considered to have the most positive effects on the fluency or quality of service processes. This can be regarded as a limitation of the study, because inquiring the positive effects by a structured list of ICT applications could have yielded more widespread information. This question tended to favor mechanisms that are in widespread use and was less favorable toward those with low user volumes because their influence remains in any case minimal from the standpoint of an entire organization or the overall healthcare system.

It is possible that many of the respondents did not mention any positive influence because they consider them nonexistent. In certain working processes, no improvements in ICT systems^35
–37or physician satisfaction³⁸ were reported, and ICT was even cited as a negative influence on doctor–patient communications and certain document management processes.^39,40 The clinical and technical infrastructure may have an effect on experiencing e-health. General ICT literacy and availability of computers and internet connections are good in Finland.¹⁵ The availability of training was not studied in the national survey. As training may have an influence on the experiences, more research on this is needed.

The responses indicated that complex organizations such as hospital districts usually found benefits in solutions that facilitate an improved integration of medical record systems' components, thus improving organizational, and possibly interorganizational, workflows. This might suggest where the most effort should be focused when reorganizing and implementing health ICT reforms in large organizations. On the other hand, primary healthcare units seemed to place a greater emphasis on support functions, interorganizational information flow, and the delivery of advice directly to a physician's desk. This might indicate that ICT developments should be targeted similarly.

The main strength of the survey was its extensive scope; the results can be considered as representing the entire country's healthcare situation. Subsequent surveys focusing on the use of electronic working methods, systems, or applications in Finnish healthcare offer valuable background data for assessing the results. This survey was, however, a qualitative survey in which classifications based on an open question were formulated based on interpretations of actual answers; this imposes limitations on the generalization of the results and possible future research.

In Finland, there are no direct business or financial incentives for providers to deliver ICT services to remote sites. The local administrators (municipalities) have the responsibility to organize healthcare service. The administrators in regions that are scarcely populated make decisions to use e-health services because it is expensive to provide traditional special services to remote villages.

Although the digitalization of healthcare in Finland began in late 1980s, it is possible that fundamental changes in healthcare processes have not yet occurred. The next step in the digitalization process will be the formulation of tightly integrated systems between organizations as well as between organizations and citizens. This may be the time when new effective means for providing services emerge. It is also possible that digitalization alone will not be the primary driver for change. The administrative structure of the Finnish healthcare system has been extremely stable for the past 20 years, and an administrative reorganization of the healthcare system that achieves greater flexibility is perhaps the top priority.

Footnotes

Disclosure Statement

No competing financial interests exist

References

Hersh

. Medical informatics—improving healthcare through information. JAMA, 2002; 288:1955–1958.

Bates

, Gawande

. Improving safety with information technology. NEJM, 2003; 348:2526–2534.

Hersh

. Healthcare Information Technology: Progress and Barriers. JAMA, 2004; 292:2273–2274.

Chaudhry

, Wang

, Wu

et al. Systematic review: Impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med, 2006; 144:742–752.

Hämäläinen

, Doupi

, Hyppönen

. eHealth Policy and Deployment in the European Union. Review and Analysis of Progress. Reports 26/2008. Helsinki: Stakes, 2008.

Liebovitz

. Cloud around the silver lining? Arch Intern Med, 2009; 169:924–926.

Walsh

. The clinician's perspective on electronic health records and how they can affect patient care. BMJ, 2004; 328:1184–1187.

Zandieh

, Kahyun

Y-F

, Kuperman

et al. Challenges to EHR implementation in electronic- versus paper-based office practices. J Gen Intern Med, 2008; 23:755–761.

Stroetmann

, Jones

, Dobrev

, Stroetmann

. eHealth is worth it: The economic benefits of implemented eHealth solutions at ten European sites. http://ec.europa.eu/information_society/activities/health/docs/publications/ehealthimpactsept2006.pdf. 2009 November 20.

10.

Shekelle

, Morton

, Keeler

. Costs and benefits of health information technology. Evid Rep Technol Assess, 2006; 132:1–71.

11.

Häyrinen

, Saranto

, Nykänen

. Definition, structure, content, use and impacts of electronic health records: A review of the research literature. Int J Med Inf, 2008; 77:291–304.

12.

Gartner on behalf of The Ministry of Health and Social Affairs in Sweden. eHealth for a Healthier Europe!–opportunities for a better use of healthcare resources. 2009. www.regeringen.se/sb/d/574/a/129815. 2009 November 18.

13.

Suomen Kuntaliitto (The Association of Finnish Local and Regional Authorities). Kunnat ja kuntayhtymät. www.kunnat.net/k_peruslistasivu.asp?path=1;29;374;36984;150755. 2010 August 17.

14.

Kiviaho

, Winblad

, Reponen

. Terveydenhuollon toimintaprosesseja ja asiointia tukevat atk-sovellukset Suomessa–kartoitus ja käyttöanalyysi [Use on information technology in the healthcare of Finland- mapping and analysis of use] Helsinki: National Network of Excellence Centers, 2004.

15.

Hämäläinen

, Reponen

, Winblad

. eHealth of Finland. Check point 2006. Report 1/2007. Helsinki: National Research and Development Centre for Welfare and Health, 2007.

16.

Reponen

, Winblad

, Hämäläinen

. Current status of national eHealth and telemedicine development in Finland. Stud Health Technol Inform, 2008; 134:199–208.

17.

Hämäläinen

, Reponen

, Winblad

. eHealth of Finland. Check point 2008. National Institute for Health and Welfare Report 1/2009. www.thl.fi/thl-client/pdfs/f5ca5a36-f2c6-4e94-ae95-a7b439b1169b. 2010 May 19.

18.

Ministry of Finance, SADe-ohjelma. www.vm.fi/vm/fi/05_hankkeet/023_sade/index.jsp. 2010 August 17(in Finnish).

19.

Krippendorff

. Content analysis: an introduction to its methodology, 2nd. Thousand Oaks, CA: Sage, cop, 2004.

20.

Dobrev

, Hoesner

, Husing

, Korte

, Meyer

. (Empirica). Benchmarking ICT use among General Practitioners in Europe. Final Report. Empirica on assignment of European Commission Information Society and Media Directorate General. Bonn April 2008. http://ec.europa.eu/information_society/eeurope/i2010/docs/benchmarking/gp_survey_final_report.pdf. 2009 November 20.

21.

Castro

. Explaining international IT leadership: Health IT. The Information Technology & Innovation. Foundation September 2009. www.itif.org/files/2009-leadership-healthit.pdf. 2009 January 5.

22.

Inamura

, Umeda

, Harauchi

et al. Time and flow study results before and after installation of a hospital information system and radiology information system and before clinical use of a picture archiving and communication system. J Digit Imaging, 1997; 10:1–9.

23.

Lepanto

, Paré

, Aubry

, Robillard

, Lesage

. Impact of PACS on dictation turnaround time and productivity. J Digit Imaging, 2006; 19:92–97.

24.

Mackinnon

, Billington

, Adam

, Dundas

, Patel

. Picture archiving and communication systems lead to sustained improvements in reporting times and productivity: results of a 5-year audit. Clin Radiol, 2008; 63:796–804.

25.

Reponen

. Radiology as a part of a comprehensive telemedicine and eHealth network in Northern Finland. Int J Circumpolar Health, 2004; 63:429–435.

26.

Reponen

. Teleradiology: A Northern Finland perspective. Kumar

, Krupinski

. Teleradiology. Berlin—Heidelberg, Germany: Springer-Verlag, 2008; 217–225.

27.

Costa

, Oliveira

, Silva

, Gama

. CARS 2003. An integrated access interface to multimedia EPR. Computer Assisted Radiology and Surgery: Proceedings of the 17th International Congress and Exhibition, London, June 25–28, 2003. Int Congress Series, 2003; 1256:880–886.

28.

Münch

, Engelmann

, Schröter

, Meinzer

. The integration of medical images with the electronic patient record and their web-based distribution. Acad Radiol, 2004; 11:661–668.

29.

Aas

. A qualitative study of the organizational consequences of telemedicine. J Telemed Telecare, 2001; 7:18–26.

30.

Reponen

, Marttila

, Paajanen

, Turula

. Extending a multimedia medical record to a regional service with electronic referral and discharge letters. J Telemed Telecare, 2004; 10,Suppl 1:81–83.

31.

Haukilahti

R-L

, Virjo

, Halila

. Enemmistö lääkäreistä pitää tietoteknologian muutosta myönteisenä [Information technology in healthcare evaluated by experienced physicians] Suomen Lääkärilehti (Finnish Medical Journal), 2008; 63:4223–4229(in Finnish).

32.

Act 159/2007. Laki sosiaali- ja terveydenhuollon asiakirjojen sähköisestä käsittelystä [Legislation on electronic archiving of documents in social and healthcare]

http://www.finlex.fi/fi/laki/ajantasa/2007/. 2010 August 17(in Finnish).

33.

Act 61/2007. Laki sähköisistä lääkemääräyksistä [legislation on electronic prescribing]

http://www.finlex.fi/fi/laki/ajantasa/2007/. 2010 August 17(in Finnish).

34.

Winblad

, Reponen

, Hämäläinen

. Kansallinen terveysarkisto - puolitoista vuotta liittymiseen Osa II: Mitä pelätään? [The forthcoming national archive for healthcare–challenges seen in healthcare] Suomen Lääkärilehti (Finnish Medical Journal), 2010; 65:1223–1229(in Finnish).

35.

Jiwa

, Skinner

, Coker

, Campbell

, Thompson

. Implementing referral guidelines: Lessons from a negative outcome cluster randomised factorial trial in general practice. BMC Fam Pract, 2006; 7:65.

36.

Ventres

, Kooeinga

, Vuckovic

, Marlin

, Nygren

, Stewart

. Physicians, patients, and the electronic health record: An ethnographic analysis. Ann Family Med, 2006; 4:124–131.

37.

Barber

, Cornford

, Klecun

. Qualitative evaluation of an electronic prescribing and administration system. Qual Saf Healthcare, 2007; 16:271–278.

38.

Johnson

, Serwint

, Fagan

et al. Computer-based documentation: Effect on parent and physician satisfaction during a pediatric health maintenance encounter. Arch Pediatr Adolesc Med, 2005; 159:250–254.

39.

Shachak

, Michal Hadas-Dayagi

, Amitai Ziv

, Reis

. Primary care physicians' use of an electronic medical record system: A cognitive task analysis. J Gen Int Med, 2009; 24:341–348.

40.

Winblad

, Hyppönen

, Salo

, Reinikainen

, Reponen

. Onko tietokone vastaanoton aikavaras? (Do computers steal too much attention during physician consultations?) Suomen Lääkärilehti (Finnish Medical Journal), 2009; 64:3956–3959(in Finnish).