Evaluation systems in local eldercare governance

Abstract

Summary

This article explores how three evaluation systems in eldercare governance, two national and one local, operate and interact at the municipal, administrative, and service levels in a Swedish municipality. The case study focuses on the three systems’ contributions to accountability and to improving eldercare quality. It is based on multiple sources, including 28 interviews with local key actors involved in local eldercare governance, and the results derive from a directed content analysis guided by four research questions.

Findings

The study demonstrates that the three evaluation systems support accountability and quality improvement in different ways and have different consequences for local actors. The systems create multiple accountability problems and have multiple constitutive effects, for example, creating different notions of what quality in eldercare means. The systems’ contributions to improving eldercare quality differed: the net effect of the two national systems was negative, whereas the local system has helped improve eldercare quality without any identified negative effects so far.

Applications

The article broadens our theoretical understanding and knowledge of regulatory mechanisms in eldercare governance. It has significance for eldercare policy by finding that policymakers and service providers must be aware of and manage multiple evaluation systems and accountability problems. Its implication for eldercare practice is that local actors must build evaluation capacity to manage existing evaluation systems in order to improve their own practices.

Keywords

Social work case study evaluation long-term care older people decision making

Introduction

The policy community has high hopes that recurrent evaluations can help governance improve quality in social services. Research into how evaluation works in practice is scarce, however, and existing research presents mixed findings regarding the quality improvement function of evaluation. This article closely examines three evaluation systems in Swedish eldercare governance, exploring how they are enacted in a municipality and how their accountability and quality improvement functions operate, with a special focus on the latter. The article provides insight into an issue that is both overlooked and contested, insight that can help local actors critically use evaluative data to inform their own practice.

Numerous evaluation systems are institutionalized in many countries to improve eldercare quality. For example, the Care Quality Commission regulates care homes in England (www.cqc.org.uk), and the Dutch Inspectorate for Health Care (http://www.igz.nl) does this in the Netherlands. These two countries have also developed quality assurance systems. In England, the Care Quality Commission uses the National Minimum Standards to assess eldercare quality. In the Netherlands, the Dutch Inspectorate uses the Quality Framework for Responsible Care to monitor quality in eldercare, while the Dutch Centre for Consumer Experience in Health Care (www.centrumklantervaringzorg.nl) is responsible for monitoring quality in eldercare from a consumer perspective according to a Consumer Quality Index. In Sweden, which is the focus of this article, the Health and Social Care Inspectorate¹ is responsible for undertaking state supervision (SSV) to monitor compliance with the statutes and ensure good quality in eldercare, while the National Board of Health and Welfare (NBHW) and the Swedish Association of Local Authorities and Regions (SALAR) run Open Comparison (OC), a national indicator-based evaluation system for benchmarking eldercare. SSV and OC are the two major national eldercare evaluation systems in Sweden. SSV can use legal sanctions whereas OC relies on voluntary action. In addition, local evaluation systems are in place in Sweden, as they are in other countries.

Although evaluation systems are key components of eldercare governance (Clarkson, 2010; Meagher & Szebehely, 2013; Ranci & Pavolini, 2013) and new public management (NPM) regimes (Van Dooren, Bouckaert, & Halligan, 2010), how they work in practice in local settings has scarcely been researched. This article explores how three evaluation systems (i.e. SSV, OC, and a local system) operate and interact in practice in a Swedish municipality. It seeks answers to four research questions:

RQ1: How are evaluation systems enacted at the political, administrative, and service levels?

RQ2: Do the systems support accountability and, if so, how?

RQ3: Do the systems support quality improvement in eldercare and, if so, how?

RQ4: What are the systems’ consequences for key actors’ eldercare work?

Sweden’s two national evaluation systems with radically different logics, SSV and OC, are the same for all municipalities, whereas local evaluation systems vary between municipalities (Lindgren, 2016). For the purpose of this article, it was important to choose a local evaluation system that is elaborated on and institutionalized in local eldercare governance, hence the choice of the Vanguard model and of East Point municipality. The Vanguard model was first introduced in one home-help services unit in East Point and is now used in all units. Sweden is a suitable illustrative case because it devotes considerable resources to eldercare (Huber, Rodrigues, Hoffman, Gasior, & Marin, 2009), has a decentralized political system with local discretion for eldercare governance, and employs public and private service providers. The privatization of eldercare (Meagher & Szebehely, 2013) and the development of evaluation systems reflect international trends, and exploring how they operate at the local level is relevant to actors operating in decentralized political systems.

Swedish eldercare governance

The Health and Medical Services Act (SFS, 1982:763) and the Social Services Act (SFS, 2001:453) frame and steer eldercare in Sweden, in which good care quality, individual rights, and the rule of law are fundamental values. Eldercare governance is divided between the state, county councils (including a few regions), and municipalities. The state governs eldercare through legislation, various policies, and supervision. Twenty-one county councils are responsible for supplying their citizens with health care services, including hospital care, primary care, psychiatric care, and dental care. At the local level, 290 municipalities are responsible for providing and running health care services to achieve national eldercare objectives.

Quality control of eldercare in Sweden

Eldercare services consist of home-help services in the elderly person’s private home and long-term residential care. Public and private providers, both publicly funded, provide these services, reflecting the fact that the universal model of eldercare services in Sweden has been replaced with a quasi-market model with customer choice (Meagher & Szebehely, 2013; Szebehely & Trydegård, 2012). There is a growing need for monitoring to ensure that public and private eldercare services meet national quality standards. Over the last two decades, many “quality-related” paragraphs have been added to the Swedish Social Services Act, and SSV has been strengthened to reinforce compliance with laws and regulations to ensure eldercare quality. Both public and private residential care homes are subjected to quality control by SSV. However, Sweden is a country where local governments also wield considerable freedom of choice. The strengthening and institutionalization of national evaluation systems have challenged local autonomy, and reinforced municipalities’ role as agents of the national government, although they are accountable to local citizens (Montin, 2015). Municipalities assess applications to operate as private care providers in home-help services, and continuously monitor if those admitted provide quality according to contract.

Brief research overview

Studies of inspection find that it can improve the quality of the services inspected, but that important aspects, such as mutual relationship building between caregivers and care-receivers, are somewhat overlooked (Braithwaite, Makkai, & Braithwaite, 2007; Downe & Martin, 2007; Furness, 2009). Moreover, studies find that while inspections and indicator-based systems measuring eldercare quality can drive quality improvement in some contexts, they also tend to have a number of negative effects, for example, overemphasizing targeted values at the expense of other important values (Braithwaite et al., 2007; Hood, 2012; Lindgren, 2016).

Research into evaluation systems demonstrates that these systems support different functions in public policy and governance (Hanberger, 2011; Rist & Stame, 2005). Ultimately, most systems are intended to enhance quality, although the ways they do this differ. Researchers have identified negative consequences of the external evaluation systems to which professionals are subjected, including hard (e.g. SSV) and soft (e.g. benchmarking systems) forms of evaluation for improving eldercare quality (Beddoe, 2012; Ubels, 2015). Braithwaite, Makkai, and Braithwaite (2007) demonstrate that the external regulation of nursing homes has the negative consequence of making them prioritize documentation over solving important problems. Other problems concern the low construct validity of the performance indicators used to measure eldercare and the generally assumed attribution between indicators (Lindgren, 2016). Moreover, the performance measures are detached from the context in which care is provided and do not reflect good and accountable care for actual persons (Jerak-Zuiderent, 2015).

Researchers studying the consequences of external control systems recognize a conflict between professionalism and the demands of NPM (Bradley & Höjer, 2009). Professional accountability is based on trust in the professional agent being qualified to make situated judgment and improve practice whereas external control is based on administratively predefined standards. Others note that professionals also have self-interest and that professional self-control is insufficient (Braithwaite et al., 2007; Brennan & Berwick, 1996).

Braithwaite et al. (2007), borrowing from Ermann (1976), note that key regulatory mechanisms have shifted in American health governance from era to era: professional self-regulation dominating during the Liberal “night-watchman state” (1800–1935), comprehensive planning and plans enforced by state command and control dominating during the welfare state (1935–1970), and local and informal social control together with markets and price mechanisms dominating during what they call “regulatory capitalism” (1970–present). The authors recognize that all three regulatory mechanisms have led to problems in eldercare, but argue that “we need now to tie these reeds together in new ways to weave a more robust and redundant policy fabric” (Braithwaite et al., 2007, p. 20).

Framework

In addition to the above-cited research, we briefly outline how the concepts evaluation systems, accountability and care quality are used. The research and concepts support our analysis.

Evaluation systems

Different types of evaluation systems exist serving different evaluative purposes (Leeuw & Furubo, 2008; Lindgren, Hanberger, & Lundström, 2016). The present article discusses three types of evaluation systems in eldercare governance in Sweden: inspection or SSV systems provide various kinds of data about compliance with laws and regulations; performance measurement systems provide quantitative data for predetermined indicators that gauge good service quality or performance; while local evaluation systems use various types of data to meet local information needs.

Accountability

The meaning of accountability extends in several directions (Mulgan, 2000). The present framework is based on Behn’s (2001) notion of democratic accountability, namely, that governments, their agencies, and service providers can be held to account for finances, fairness, and performance. Accountability for finances implies that those in power should explain what they have done with taxpayer money, not exceed budgets, and be held accountable for following the rules and spending the money as planned. Accountability for fairness concerns whether those in power or the responsible organizations have paid due attention to established rules and procedures and upheld ethical standards such as fairness and equity. Rules and procedural standards are set to support the values an organization is intended to uphold. There is also a need to hold governments and public organizations accountable for the implementation, outcomes, and consequences of public policy, that is, accountability for performance. There must be expectations as to what performance outcomes can be assessed, and defining such expectations is critical in a democracy. There is no consensus about what performance criteria are to be used in assessing public policies.

Quality in eldercare

Governments, stakeholders (care providers, professionals and older people/relatives), and researchers conceive of quality in eldercare in different ways, and there is no common understanding of what criteria should be used to capture such quality.

Care quality has been defined in relation to the structural attributes of eldercare settings (e.g. staff competence), care processes, how care is performed (e.g. continuity), and the outcomes of care (e.g. achievement of national eldercare objectives; Donabedian, 1988). Eldercare research emphasizes that care quality concerns the relationship between caregiver and care receiver, and that this relationship has several important qualities that make it personal and genuine (Szebehely & Trydegård, 2012; Waerness, 1984). Good quality in the meeting between caregiver and care receiver is based on mutuality and trust.

The notion of the quality of situated, concrete, and accountable care (Jerak-Zuiderent, 2015) questions the assumption that care and accountability can be separated. Instead, good and accountable care is situated in the concrete situations of actual persons, situations in which care and accountability are intertwined.

Methods

The case

East Point is a medium-sized Swedish municipality with a population of around 100,000 inhabitants, 5% of whom are over 80 years old—the average proportion for the country. The implementation of three radically different evaluation systems was a criterion for selecting this municipality. East Point was the first Swedish municipality to introduce Vanguard, and was also subjected to SSV and participated in OC during the studied period. We selected a municipality allowing private service providers, as evaluations are supposed to play an important function in today’s market-oriented eldercare governance. East Point’s political majority has shifted from right wing to red-green over the last two elections. The right-wing Social Welfare Committee decided to open home-help services to private providers in 2012.

Methods and material

The research questions guided the data collection. The material includes descriptions and reports of the three evaluation systems, interview transcripts, Social Welfare Committee minutes, 35 inspectorate decisions, OC reports, as well as websites and supplementary material covering the 2011–2015 period. Our interviews in East Point comprise 28 semi-structured interviews with the Chair of the Social Welfare Committee, the Head of the Social Welfare Department, two politicians, six administrators, two care managers, five care unit managers, and nine staff of residential homes and home-help services. Care receivers were not included in this study, but two representatives of elder advocacy organizations were also interviewed. The interviewees were selected to reflect different roles and power positions in local eldercare governance. The semi-structured interview questions focused on the research questions and covered the interviewees’ experiences and perceptions of the three evaluation systems and their effects and consequences. The interviews lasted between one and two hours were tape-recorded and transcribed.

We conducted a directed content analysis (Hsieh & Shannon, 2005), based on the research questions, of the operation of the evaluation systems at the municipal, administrative, and service levels. The first-level interpretations (made by eldercare actors) are reported descriptively and sometimes cited. Our own second-level interpretations (based on the framework and further knowledge of how evaluation systems operate) elaborate on these (Yanow, 2000). All material was treated as texts and triangulated to seek answers to the research questions. We summarize the results for each question together with citations and present our own interpretations successively and more elaborated in the discussion.

Three evaluation systems in Swedish eldercare

State supervision

SSV can be described as a hard evaluation system because it can use legal sanctions. The inspectorate, responsible for SSV, addresses complaints from individual citizens regarding health and social care services (including eldercare), handles mandatory and voluntary reporting of irregularities by staff. The handling of these complaints and reports can lead to further supervision. In addition, the inspectorate initiates its own supervisions and implements special government assignments for thematic supervision. The inspectorate also examines applications for permits for the private service licensing of privately owned residential homes.

Swedish SSV consists of desktop supervision as well as preannounced and unannounced inspections to determine whether eldercare services are complying with laws and regulations and other national eldercare standards. The inspectorate is guided by a supervision policy and national criteria for risk-based supervisions. For each supervision, a decision is issued stating insufficiencies and any needed improvement actions. A yearly summary report is compiled for all supervisions, containing conclusions regarding the state of eldercare and need for action. The inspectorate uses various methods, such as good examples and sanctions, in its supervisions, the latter making it a hard evaluation system. SSV is assumed to ensure a minimum level of eldercare service and that the elderly are provided safe high-quality eldercare.

Open comparisons

OC can be described as a soft evaluation system because it is voluntary and lacks formal sanctions. Its overall objective is to improve the performance of social services (e.g. eldercare) through open access to comparable information on quality, results, and costs, which presupposes local capacity to analyze OC data. OC has been developed and administered by NBHW, a central government agency, and by SALAR, a nongovernmental organization that represents and advocates the interests of local governments.

At present, around 40 indicators are used to capture the performance of eldercare services at the municipal level, and for some indicators at the service provider level as well. In annual reports, municipalities are compared and ranked using a relative green, yellow, and red scale. The indicators are based on data from official statistics, recurrent municipal surveys, user satisfaction surveys, and patient and pharmacy registers. The indicators capture measureable signs of eldercare service quality or performance, based on NBHW’s and SALAR’s interpretations and operationalizations of the eldercare requirements and objectives set forth in laws and other provisions, and of resources assumed to be needed to realize them. A third of the indicators measure structural aspects of services, another third care processes, and the final third outcomes (mainly user perceptions of aspects of services) (NBHW & SALAR, 2015).

Local evaluation system: Vanguard

Unlike OC, Vanguard is a bottom-up derived model or methods in which evaluation is integrated with organizational improvement. Vanguard was inspired by ideas from systems thinking and lean manufacturing, adapted for public-sector services by Seddon (2008). The aim was to establish a new model of work that could better match the actual needs of individual service receivers (in East Point for home-help services). The Vanguard methods’ usage by public sector organizations has been documented elsewhere (Gibson & O’Donovan, 2014). Following Seddon’s (2008) check-plan-do cycle, a group of staff (i.e. nurses, care workers, and administrators) first checked and analyzed the purpose and current way of working, including understanding customer demands, work flow, and obstacles to effective work flow. After they agreed about the purpose of home-help services from the customers’ perspective, they identified all unnecessary work in the current organization. The second stage was to plan how to “clean stream” the work flow to remove activities that did not help meet customer demands, and to develop performance measures that truly capture customer demands. In the “do-stage”, these solutions were implemented and care activities incrementally and continuously adjusted to customer needs (Swan & Blusi, 2012). The focus in the present article is on the service receivers’ own recurrent evaluation of care and services.

Enactment of evaluation systems in East Point

State supervision

Between 2012 and 2016, SSV was implemented in East Point through 35 supervision cases. The inspectorate handled 20 complaints initiated by individual patients and relatives concerning, for example, deficient care, treatment, and medical routines. Ten supervision cases were initiated in response to care staff reports concerning inadequacies and adverse events such as the exchange of medications. In addition, the inspectorate conducted five supervisions on their own initiative, four concerning reported shortcomings in individual residential care units and one concerning the quality of communication between all eldercare institutions.

At the municipal level, politicians in power and administrators were the main actors involved in enacting SSV, together with the service providers subjected to inspection. All these parties spent time reviewing the inspections to understand their meaning, and to respond to any criticisms. The inspections were administered through formal channels following the local government model of governance:

There have been a few inspections, and what we do is tell the department to go through the case, look at the problems, and suggest action to take. We are laypeople and cannot say what they should do in an individual case, except that we should follow the head nurse’s and the inspectorate’s recommendations. (Chair of Social Welfare Committee)

SSV is a recurring issue treated in Social Welfare Committee minutes. On several occasions, the Committee has discussed and decided on action to alleviate deficiencies noted in inspections, in the way described above.

The decision makers on the Social Welfare Committee regard legislation and national guidelines as governing eldercare, and believe that SSV adds little in terms of reinforcing compliance with the statutes.

At the administrative level, SSV is used to guide the administration’s quality control:

Yes, I use the supervision criteria in planning and steering, to see what they [i.e. the inspectorate] mean by quality control systems and that kind of thing. What are they focusing on—the central levels or the lower levels? (Senior administrator)

At the service level, a residential home was subjected to an unannounced inspection. This home, caring for elderly living with dementia, was found not to have enough night-time staff. One staff member was moved to increase the night-time staffing, so the home enacted SSV recommendation in one unit. The experience of care unit managers, however, is that SSV is not important for eldercare practice except in the few cases in which inspections are conducted and insufficiencies detected. The statutes and national guidelines are considered more important:

I try to prevent [deficiencies] and follow the guidelines, so it does not matter if an unannounced inspection happens . . . I try to comply with the prevailing norms. (Care unit manager)

However, it could be that the manager overlooks the impact of SSV and that it actually has greater influence than perceived, but none of the interviewees shared with us that they took action to avoid being criticized in an announced inspection.

Open comparison

The enactment of OC in East Point was voluntary, and was manifested by using OC indicators and reports at all three levels. Annual OC reports (2012–2015) reveal that in 2012, East Point scored above average (green) on six indicators, average (yellow) on 15, and below average (red) on four. Between 2012 and 2015, three indicators had improved from yellow to green, six stayed green, and five stayed yellow. In 2014, nine indicators scored red, but five of these returned to yellow in 2015. East Point’s overall ranking fluctuated, but was higher at the end of the studied period.

At the municipal level, OC was enacted as an eldercare performance objective in East Point’s eldercare policy. The municipality targeted scoring in the upper quartile (i.e. green, ranking among the first 81 municipalities) for all indicators measuring the quality of home-help services and residential care. In 2014, a report from East Point’s municipal audit office revealed that eldercare costs were alarmingly high, but OC indicators suggested that costs were average. The Chair of the Social Welfare Committee explained how they interpreted these contradictory results:

In OC we see that we are not as bad as we thought we were. This is, of course, very good and interesting too. It feels good . . . to know that we are not among the worst municipalities in the country – and of course not among the most excellent ones either. (Chair of Social Welfare Committee)

The Committee’s 2012–2015 minutes often discuss East Point performance on the OC indicators, and sometimes action has been taken with reference to these scores. The cost per customer is an indicator that attracts the most political interest, while indicators such as waiting times for residential care, customer satisfaction, and rehabilitation measures are less frequently discussed.

Implementation of OC at the administrative level is guided by East Point’s eldercare policy. Administrators spend time collecting relevant data and reporting on the indicators to the national, municipal, and service levels.

According to our interviews, politicians and administrators are the ones mostly engaged in OC, whereas care managers and staff at the service level are less engaged but still governed by the OC results:

We who work in eldercare operations receive instructions from the central administration or political level based on OC. We have, for example, reduced the number of staff in all residential care units. (Care unit manager)

East Point’s eldercare policy aims to score among the 25% best performing municipalities on all OC eldercare indicators, apparently cost indicators being an exception.

Vanguard

Vanguard was implemented as an experiment in 2012 in one home-help service unit in East Point. The experiment was successful, so Vanguard was implemented in all the municipality’s home-help services starting in 2016. A new way of monitoring eldercare has resulted from implementing Vanguard at the municipal and administrative levels. Politicians on the Social Welfare Committee and administrators use Vanguard reports and results as well as other follow-ups to inform themselves about the state of eldercare.

More flexible use of staff time and adaptation of care to customer needs characterize the implementation of Vanguard at the service level. A 2015 comparison of home help eldercare units that had employed Vanguard for over one year with comparable units employing the model for under one year or not at all found that the performance of the former units had improved in many ways. Staff continuity had improved, as had staff knowledge and observation of individual customer needs and health conditions. Staff sick leave had declined, and work satisfaction and customer satisfaction with services had improved (reflected in our interviews; Daneryd, 2015):

Vanguard works really well—customers are satisfied, staff are satisfied, not least because they have more influence over the organization of their work. But the biggest difference is the continuity. Now, they go to the same customers most of the time, while before they might meet the same customer only once in three weeks. That was very difficult. Before you never knew what to do or whom to visit the following day, whether to walk, cycle, or drive the car. (Care unit manager)

The positive response could also be related to perceived problems in eldercare governance, and to the time-consuming documentation and monitoring system that poorly reflected provided care at the service level.

The systems’ support of accountability and eldercare improvement

Support of accountability

All citizens are to be ensured good-quality eldercare on equal terms, irrespective of where they live. To uphold this right, local governments and service providers are held accountable by the inspectorate’s SSV. Administrators in East Point understood that the inspectorate held them accountable for the fair provision of eldercare:

We adapt to the statutes, and then it is up to them [i.e. the inspectorate] to follow up whether we have. But we are not designing our services just to make them look good in their eyes, but rather to adhere to the statutes. We are well aware of the laws and rules and of what is expected, but there are varying conditions and opportunities to do this. (Senior administrator)

As mentioned, the inspectorate held a residential home accountable for providing insufficient night-time staffing. The staffing was increased in line with the supervision, but this action also had a crowding-out effect, in that staff were moved from another unit reducing the care of other care receivers.

The main accountability function of OC in East Point was to support accountability for performance. By following up the OC scoring, care providers were held accountable for how East Point in general and their own units in particular scored on OC:

I have briefed the staff about what it was like and how we performed [in OC], and we have reached the level for many indicators. (Care unit manager)

Vanguard supported accountability for eldercare performance too, but in a different way. All care receivers were asked about what could be improved according to their needs and desires. They repeatedly held care providers and professionals accountable for the provided care.

Support to improve eldercare quality

Planning and steering, a structural quality considered important by SSV, was not monitored in East Point during the studied period. Interviewees at all levels experienced turbulence in the municipal eldercare organization and a period of insufficient planning and steering. Vanguard was introduced as a complementary planning and follow-up model and has improved bottom-up steering. But, none of the evaluation systems recognized any weakness in planning and steering in East Point.

As shown, the unannounced inspection contributed to quality improvement in one location, though staffing was reduced for other care recipients at the same residential home. East Point’s eldercare performed well according to OC, but this was used to justify a cut in eldercare staff in the organization at large: “We scored relatively high in OC before it was decided that East Point can cut down staffing” (Care unit manager).

Taken together, the SSV and OC support intended to improve eldercare quality, measured in terms of increased staffing, has had a negative effect.

OC monitored processual quality in terms of continuity and how care receivers perceive the relationship between caregiver and care receiver. A few care unit managers scoring low on these OC indicators said that they took action to improve these aspects by asking staff to consider how they could improve their work. In the Vanguard model, the individual care receiver’s needs and continuity were very important, and the monitoring of continuity was perceived to contribute significantly to improved eldercare quality:

We have worked on continuity [in Vanguard], and because we have good continuity, we do not fail in this as we used to when many staff members were involved. Now [name of caregiver] visits the same person, sometimes for a whole week, and then we do not miss things. (Care unit manager)

Vanguard’s monitoring of continuity also improved East Point’s OC score according to the Chair of Social Welfare Committee.Documentation, another indicator used to measure processual quality, was perceived to compete with care provision in East Point:

We end up writing instead of caring . . . you want to be together with the care receiver instead of sitting somewhere else taking notes. Some consider it as taking time from the elderly, and as a bit negative. (Care unit manager)

SSV monitored whether East Point had implemented care decisions in a way that ensures quality in eldercare and ensures that the elderly get the care to which they have a right, as captured by the indicator “follow up eldercare”:

A problem in East Point is the lack of residential homes, and this in turn has often prompted SSV to demand a statement of opinion from us. People may have been declared eligible for a place in a residential home but have not been offered one for a long time, and then they [i.e. the inspectorate] ask for a status report to check that these people have decent conditions while waiting. Sometimes the delay leads to the reconsideration of the first decision, because these people have become much worse and need more care and another type of home. (Care manager)

Care receiver satisfaction with the provided care, i.e. quality of results, is accounted for by all three systems. SSV and OC use the same national questionnaire to assess the elderly care receivers’ general satisfaction with the provided care. The interviewees have mixed opinions regarding this questionnaire, most being aware of its limitations (e.g. composition of respondents and low response rate), though they still use its results. The care receiver satisfaction responses collected in the Vanguard model are considered more valid and feasible, as they clearly capture individual care receiver assessments of the provided care.

Only Vanguard helps improve the quality of situated care, because this model is the only evaluation system designed to respond to individuals’ specific needs and desires.

A customer said that she did not like it when you could smell smoke on a staff member, and she wrote this down so that other staff members would know this. The next time there was a “ten” on this [on a 10-point scale] so I think that nobody will go there smelling of smoking . . . And that our ladies should be careful about cleaning—not missing the corners and those kinds of things, which disturb them—that is important to them. (Staff member at home-help service)

Taken together, the results indicate that all three systems support some aspects of structural quality, processual quality, and quality of results, but only Vanguard supports the improvement of situated care.

Consequences for key actors

Decision makers

While SSV holds eldercare committees to account for compliance with national standards and statutes, those in authority must take action in response to SSV criticism. One consequence is that SSV affects local priorities and tends to increase costs.

SSV governs the Social Welfare Committee, the Chair in particular, which acts as the long arm of the state to reallocate resources and ensure that subordinates meet national standards. This limits local autonomy by forcing local actors to prioritize the achievement of national objectives.

Contrary to the intentions guiding the Vanguard model and East Point’s own eldercare policy, stating that East Point should be among the top 25% of municipalities in all OC indicators for eldercare, OC has made it possible for those in power to justify cuts in eldercare service. Keeping to the budget is considered more important than scoring high on OC. In this way, OC increased local autonomy, which was then used to make cuts in eldercare.

The consequence of the Vanguard model is that decisions makers have to respond to and take into account bottom-up data on eldercare performance together with other statistics and national performance data, making eldercare policy, and management more challenging.

Care providers

The consequences of SSV have been rather limited for most service providers, as SSV has simply governed and constrained their priorities and resource allocation. OC was used by East Point to hold service providers accountable for their performance and to exert pressure to maintain green and improve red and yellow performance on OC indicators. OC also defined what eldercare qualities should be prioritized. Vanguard changed service providers’ management role from steering staff and helping them implement national and municipal standards and policy objectives, to coaching staff to improve care according to the care receivers’ individual needs.

Professionals

SSV did not directly steer professionals’ work, but indirectly affected their professional discretion by reinforcing their role as implementers of national eldercare policy in compliance with the statutes. Furthermore, to avoid criticism, professionals have had to improve the specific care attributes being inspected, which has meant working less on the soft unmeasured aspects of care. OC has induced professionals to pay attention to OC indicators, particularly if they score low on a specific indicator. Professionals working in a unit that scored high were acknowledged for this. Vanguard empowered professionals and changed their role from one of implementing directives from above to working with a customer orientation, which increased their work satisfaction, reduced sick-leave, and empowered them to govern themselves.

Care receivers

Surprisingly, neither of the interviewed representatives of elder advocacy organizations had experience of the three evaluation systems. Below we interpret the consequences of these systems for care receivers based on interviews with other actors.

SSV has helped ensure that care receivers receive the care to which they have a legal right. Whether or not the 20 supervisions initiated in response to complaints from care receivers and relatives had a positive effect on the individuals or in general has not been studied. In theory, OC scores could help customers make an informed choice of care provider, but none of the interviewees indicated any such use of OC by the elderly or their relatives. Care managers instead provided such decision-support information orally and in writing when an eldercare decision was being made. Because a lack of residential homes, however, customers could only exercise choice between different home-help services. Vanguard had the most obvious positive consequences for care receivers, as their needs, wishes, and demands were captured by the evaluation system, helping them get the care they valued, including increased staff continuity.

Limitations of the study

Service receivers’ experiences of the three evaluation systems were not investigated. Another limitation is that only one case has been explored. Further research is needed of how national and local evaluation systems operate in different local contexts and of how care-receivers and their relatives experience various systems.

Conclusions and discussion

Four overall conclusions can be drawn from this study. First, the three evaluation systems supported accountability and quality improvement in different ways with different consequences for local actors. Second, the systems created multiple accountability problems, that is, vagueness about to whom and for what service providers and professionals are accountable. Third, the net effects of SSV and OC have been negative as they have had a crowding-out effect and cutback effect, respectively, whereas Vanguard has helped improve eldercare without having any identified negative effects so far.

Fourth, the systems have a number of constitutive effects. All three create different notions of what quality in eldercare entails: SSV reinforces actors’ prescribed roles and responsibilities in national eldercare governance; OC helps make benchmarking a norm, something done recurrently to improve quality in eldercare; while Vanguard helps legitimatize bottom-up governance.

Returning to the research questions, the three evaluation systems have all been implemented differently (RQ1). The implementation of SSV was manifested in several supervisions initiated by care receivers, staff, and the inspectorate when insufficiencies were identified and violation of regulations suspected, whereas OC was voluntarily institutionalized in East Point’s eldercare policy and implemented top-down. In contrast, Vanguard started as an experiment at one home-help service unit and was implemented in all home-help services starting in 2016.

The operation of SSV reflects the division of power between the state and local government as well as key actors’ prescribed roles in eldercare governance. Although SSV is a hard evaluation system, it was not perceived as such in East Point. In contrast, OC, a seemingly softer evaluation system, has had a stronger effect on eldercare policy and practice. As indicated, important care qualities have been overlooked by SSV and OC. Vanguard seems to provide a solution to this problem by supporting the improvement of situated care (Jerak-Zuiderent, 2015) in line with care receivers’ needs and demands.

The three systems presented evaluation results at different times and regularity which could explain why they were not systematically integrated in eldercare governance, accountability, and quality work. This might also have been because of lack of time, or lack in analytical capacity. SSV demands local actors to comply with regulations any time insufficiencies are revealed, whereas OC promotes performance-oriented action targeting prioritized indicators once a year when a new OC report is presented. Vanguard provides recurrent feedback from individual care receivers on provided care.

The three systems support different kinds of accountability (RQ2). SSV helped hold decision makers accountable for fairness (i.e. compliance with national standards), while OC helped hold them and service providers accountable for performance as captured by OC indicators (Behn, 2001). Vanguard mainly helped hold service providers and staff accountable for situated care (Jerak-Zuiderent, 2015). These divergent focuses imply multiple accountability problems (Schillemans & Bovens, 2011). The different systems created uncertainty about to whom service providers and professionals were accountable and for what. Various actors could interpret the features and functions of an evaluation system, and come up with competing arguments as to what the data mean and how they should be acted upon. The multilevel Swedish eldercare governance system adds to the multiple accountability problems (Schillemans & Bovens, 2011), as does the involvement of semi-governmental and private evaluation actors (i.e. OC and Vanguard). This also reflects a tension between hierarchical accountability, characterized by lack of trust in professionals, and professional accountability, characterized by trust in professionals’ competences and ethics.

The systems’ contributions to improving eldercare quality also differed significantly from each other (RQ3). SSV and OC made mixed contributions to the quality improvement function. SSV helped improve aspects of care being inspected, but also had a crowding-out effect. OC was used to justify general cutbacks, but also helped improve the caregiver–care receiver relationship at some units. However, the net effect of SSV and OC was negative, whereas Vanguard helped improve eldercare quality without any identified negative effects so far. Vanguard was the only system that provided performance data in relation to care receivers’ own expressed objectives.

The results indicate that OC and Vanguard have had major consequences for the key actors, whereas SSV has mainly temporarily affected those inspected (RQ4). Vanguard has had positive consequences for care providers and professionals, mainly in that Vanguard made the staff focus on the right things in their own eyes and in the eyes of care receivers, reducing sick leave, increasing continuity, and increasing work satisfaction. The implications of this for care receivers are extensive. Asking care receivers about their personal needs and wants and adapting care in response, according to our interviews with staff, has been a major change brought about by Vanguard, a change highly valued by care receivers.

Whether Vanguard is a sustainable model for helping care providers and professionals do the right things for care receivers as their needs are changing, helping professionals build trust and provide good and accountable care within the budget, and whether this model will clash with other values, such as the rule of law, national policy goals, are questions needing further investigation. How this and other bottom-up evaluation systems are adapted to local conditions can vary significantly, which means that an evaluation system, such as Vanguard, can appear in many guises. More research is needed into various local evaluation systems, including how they change and interact with other local and national evaluation systems.

Braithwaite et al. (2007) advocate combining different regulatory mechanisms in a new way, uniting (1) professional self-regulation, (2) comprehensive planning enforced by state command and control, (3) control via local and informal social means as well as via markets and price mechanisms. In East Point these mechanisms are not well combined, the three systems do not quite reflect these three regulatory mechanisms. SSV is not just a top-down control system, as it permits care receivers to make complaints as one way to initiate SSV. The benchmarking system (OC) is based on “naming and blaming”, a mechanism not addressed by Braithwaite et al. (2007), and Vanguard is based on a combination of professional self-control, local and informal social control, and, most importantly, on control exerted by care receivers. The regulatory mechanisms discussed by Braithwaite et al. (2007) do not quite reflect the mechanisms involved in this case, so there is a need to reconsider what the new regulatory mechanisms are in eldercare governance.

Implications for policy and practice

The implication of this study for local actors is that they can continue to follow statutes and national guidelines and keep informed about SSV’s supervision policy and plans. This will help them avoid being inspected and prepare them for unannounced inspections. However, using knowledge from the inspectorate to enhance eldercare quality is problematic because SSV focuses on specific deficiencies and not on what is conceived as the most important in quality in eldercare, and not on what functions well. Using data from a national benchmarking system is also problematic because the data quality varies, and governing by numbers that are not trustworthy is risky. The development of eldercare policy on OC indicators is uncertain too because of the above and because the indicators have changed radically over time, and more fundamentally because the municipality’s ranking depends on others’ scoring. Benchmarking is not a solid mechanism for ensuring quality in eldercare. This is not to say that national evaluation data have no value for quality work, it has, but should be used critically and reflectively. The results can justify developing tailor-made evaluation systems to meet local actors’ knowledge needs and to build evaluation capacity. They need capacity to analyze data from different systems and to respond to national evaluations while deliberating and developing their own practice.

Footnotes

Ethics

Ethically approval is not of relevance in this article as it does not include service receivers.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: the Swedish research council FORTE (2012-0379).

Note

References

Beddoe

(2012). External supervision in social work: Power, space, risk, and the search for safety. Australian Social Work, 65(2), 197–213.

Behn

R. D.

(2001). Rethinking democratic accountability. Washington DC, CO: Brookings Institution Press.

Bradley

Höjer

(2009). Supervision reviewed: Reflections on two different social work models in England and Sweden. European Journal of Social Work, 12(1), 71–85.

Braithwaite

Makkai

Braithwaite

(2007). Regulating aged care: Ritualism and the new pyramid. Cheltenham/Northampton, MA: Edward Elgar.

Brennan

T. A.

Berwick

D.M.

(1996). New rules: Regulation, markets and the quality of American health care. San Francisco, CA: Jossey Bass.

Clarkson

(2010). Performance measurement in adult social care: Looking backwards and forwards. British Journal of Social Work, 40(1), 170–187.

Daneryd

(2015). Home help services in East Point – A demonstration of progress. Stockholm: Forum för Health Policy.

Donabedian

(1988). The quality of care: How can it be assessed? Archives of Pathology & Laboratory Medicine, 121(11), 1743–1748.

Downe

Martin

(2007). Regulation inside government: Processes and impacts of inspection of local services. Policy and Politics, 35(2), 215–232.

10.

Ermann

M. D.

(1976). The social control of organizations in the health care area. Milbank Memorial Fund Quarterly/Health and Society, 53, 167–183. Retrieved from https://www.milbank.org/quarterly/articles/the-social-control-of-organizations-in-the-health-care-area/.

11.

Furness

(2009). A hindrance or a help? The contribution of inspection to the quality of care in homes for older people. British Journal of Social Work, 39(3), 488–505.

12.

Gibson

O’Donovan

(2014). The vanguard method as applied to the design and management of English and Welsh Children’s Service Departments. Systemic Practice and Action Research, 27(1), 39–55.

13.

Hanberger

(2011). The real functions of evaluation and response systems. Evaluation, 17(4), 327–349.

14.

Hood

(2012). Public management by numbers as a performance-enhancing drug: Two hypotheses. Public Administration Review, 72(1), 85–S92.

15.

Hsieh

Shannon

(2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288.

16.

Huber

Rodrigues

Hoffman

Gasior

Marin

(2009). Facts and figures on long-term care. Europe and North America. Vienna: European Center.

17.

Jerak-Zuiderent

(2015). Accountability from somewhere and for someone: Relating with care. Science as Culture, 24(4), 412–435.

18.

Leeuw

F. L.

Furubo

J.-E.

(2008). Evaluation systems: What are they and why study them? Evaluation, 14(2), 157–169.

19.

Lindgren

(2016). Kvalitetsmätningars användning och effekter inom svensk äldreomsorg [Use and effects of quality measurements in Swedish eldercare]. Nordiske Organisajonsstudier, 18(1), 31–55.

20.

Lindgren

Hanberger

Lundström

(2016). Evaluation systems in a crowded policy space: Implications for local school governance. Education Inquiry, 7(3), 237–258.

21.

Meagher

Szebehely

(Eds) (2013). Marketisation in Nordic eldercare: A research report on legislation, oversight, extent and consequences. Stockholm University, Sweden: Normacare.

22.

Montin

(2015). Från tilltrobaserad till misstroendebaserad styrning. Relationen mellan stat och kommun och styrning av äldreomsorg [From trust-based to distrust-based steering. The relation between state and municipality in governing eldercare]. Nordisk Administrativ Tidskrift, 92(1), 58–75. Retrieved from https://www.djoef-forlag.dk/sites/nat/

23.

Mulgan

(2000). “Accountability”: An ever-expanding concept? Public Administration, 78(3), 555–573.

24.

NBHW & SALAR. (2015). Öppna jämförelser 2014. Vård och omsorg om äldre. [Open comparisons 2014. Eldercare]. Stockholm: The National Board of Health and Welfare.

25.

Ranci

Pavolini

(Eds) (2013). Reforms in long-term care policies in Europe. Investigating institutional change and social impacts. New York, NY: Springer.

26.

Rist

R. C.

Stame

(Eds) (2005). From studies to streams. Managing evaluative systems. New Brunswick, NJ: Transaction Publishers.

27.

Schillemans

Bovens

(2011). The challenge of multiple accountability: Does redundancy lead to overload? In Dubnick

M. J.

Frederickson

H. G.

(Eds) Accountable governance. Problems and promises (pp. 3–21). Armonk, NY: ME Sharpe.

28.

Seddon

(2008). Systems thinking in the public sector. The failure of the reform regime and a manifesto for a better way. New Brunswick, NJ: Triarchy Press.

29.

SFS. (1982:763). Hälso-och sjukvårdslagen [The Health and Medical Services Act].

30.

SFS. (2001:453). Socialtjänstlagen [The Social Services Act].

31.

Swan

Å.

Blusi

(2012). Hemtjänst – införandet av ett nytt arbetssätt med kunden i centrum. Härnösand: Kommunförbundet FoU Västernorrland.

32.

Szebehely

Trydegård

G.-B.

(2012). Home care for older people in Sweden: A universal model in transition. Health and Social Care in the Community, 20(3), 300–309.

33.

Ubels

G. M.

(2015). Narrative accountability and quality awareness: Learning about (re)presenting narrative care. Journal of Aging Studies, 34, 190–198.

34.

Van Dooren

Bouckaert

Halligan . (2010). Performance management in the in the public sector. Aningdon: Routledge.

35.

Waerness

(1984) The rationality of caring. Economic and Industrial Democracy, 5(2), 185–211. Retrieved from http://journals.sagepub.com.ezproxy.ub.gu.se/doi/pdf/10.1177/0143831X8452003

36.

Yanow

(2000). Conducting interpretive policy analysis. Newbury Park, CA. Sage.