The British Civil Service: Perspectives on ‘Decline’ and ‘Modernisation’

‘Decline of the Whitehall Model’

The Whitehall model is the most well-known analytical tool used to understand the functioning of the British central bureaucracy and its relationship to ministers and parliament.

Its key elements are: a career civil service governed by an ethos of impartiality; competitive recruitment systems geared toward producing generalist administrators; promotion on merit, mainly from within the organisation itself; accountability governed by the doctrine of ministerial responsibility for the work of departments; expert policy advice to ministers designed to enable them to function effectively in the parliamentary arena, from officials who are effectively the monopoly providers of advice; a civil service charged with implementing policies collectively agreed by ministers; and hierarchical bureaucratic structures, organised on a functional basis, as the mechanism through which policy is implemented (for discussions, see, for example, Drewry and Butcher 1991, 23–26, 150–153; Pyper 1995, 6–15; Burnham and Pyper 2008, 32, 37–41).

Many observers subscribed to the Whitehall model as an accurate general description of the civil service, and adhered to its implications in a positive spirit, seeing it as a desirable role model. However, as a guide to understanding the British civil service, this model is analytically impoverished, empirically ill-founded and historically weak in its reading of evidence. This can be illustrated with reference to four key points.

First, having identified the key characteristics of the traditional civil service as being based around bureaucratic hierarchies, the research focus of this model is then located on a very selective group of senior officials, and it analyses a limited range of their roles and functions, concentrating on relationships with ministers, high policy-making, accountability and issues surrounding impartiality and neutrality. This narrow focus on an elite aligns with the ‘club government’ analogy developed from Marquand's original concept by Michael Moran (2003). Evaluating changes to the civil service in relation to these particular features, adherents to the Whitehall model conclude that the organisation is ‘in decline’. These observers became increasingly concerned about the impact of change (particularly, but not solely, the structural and managerial reforms implemented by the Thatcher and Major governments and their successors) upon the key components of this model. Richard Chapman's work epitomises this outlook. A series of articles (Chapman 1992 and 1997; Chapman and O'Toole 1995 and 2010) charted what was perceived as the decline and fall of the traditional civil service by those who believed it had been encapsulated by the Whitehall model. Reviewing the effect of converting departments’ executive divisions into semiautonomous agencies (including the structural changes, managerial innovations and changes to accountability regimes) and taking into account other departures from the model, such as reforms enabling people to be recruited at senior level from the private sector and other public bodies, Chapman (1992) speculated as to whether these managerial projects might mark ‘the end of the civil service’, as traditionally understood. He later cast aside any doubt (Chapman 1997). He believed that there had been a fundamental erosion of the core values and functions of the civil service. He now saw the civil service as a weakened, fragmented organisation (or series of organisations), its functions reduced by privatisation and its role in policy-making marginalised by creeping politicisation. A similar conclusion was reached by Colin Campbell and Graham Wilson, who analysed ‘the end of Whitehall’, describing what they saw as the ‘death of a paradigm’ from a variety of causes including the reforms so heavily criticised by Chapman, the characteristics and conduct of the Thatcher and Major governments and more deeply rooted failings and weaknesses in the British constitution. ‘By the 1990s … the Whitehall model had come under such stress that its survival seemed questionable’ (Campbell and Wilson 1995, 20).

Second, this ‘decline and fall’ narrative exhibits a tendency to put past generations of senior civil servants on an artificially high pedestal, from which more recent senior officials appear to have fallen. There is an associated tendency to over-praise past performance in certain spheres without giving adequate recognition to problems and deficiencies. Some examples include:

Third, while overplaying and exaggerating the positive features of the ‘traditional’ civil service, the ‘Whitehall decline’ thesis underestimates past organisational and managerial failings. For example:

Fourth, as a corollary of its failure to take full account of these issues, the ‘Whitehall decline’ thesis underestimates the positive features of the changes brought about by the modernisation agendas of recent governments, including the new focus on service delivery (beyond and within the civil service) and improvements in the calibre of management at all levels. The new emphasis on ‘managing downwards’ (rather than the traditional focus on policy advice upwards) also had a positive impact on the old ‘primary’ role of policy development, because it served to remind the senior civil service of the need to understand implementation challenges and the reasons for delivery failures.

The ‘Governance/Fragmentation/Hollow State’ Critique

At the core of this critique lies an assumption that the British civil service is in decline because it has been engulfed by waves of changes which have eroded its status and previous hegemony in particular respects.

‘Governance’ is a term and an approach that enlarges the concept of ‘government’ to encompass the functioning networks of governmental and non-governmental bodies in a more inclusive notion, and at the same time asserts that the role of government is changing towards one of ‘steering’ policy delivery rather than itself doing the ‘rowing’. Drawing on research from the ‘ESRC Whitehall Programme’, Rod Rhodes (2000) synthesised the implications of this change from government to governance in terms of: the heightened significance of policy networks; a reduced role for government where the private sector is used for activities formerly provided directly by the state; increased attention to moral and ethical issues and standards of conduct, as the ‘in-house’, hierarchical systems of oversight are replaced by looser arrangements with outside providers that might not work to the same principles; the introduction of NPM initiatives such as the transfer of private sector management systems and techniques to the public sector, new structures for the delivery of public services and the introduction of quasi- (or pseudo-?) markets, contractorisation and consumerism; and ongoing programmes of institutional and constitutional reform which would have implications for policies, the political culture and the governing framework itself.

The ‘hollowing out of the state’ is thought by some authors to be the inevitable consequence of the move (or return) to governance. Simply stated, the hollowing-out thesis (Rhodes 1994 and 1997, 17–19, 87–111; see also Foster and Plowden 1996) argues that the British government has steadily ‘lost’ policy functions ‘upwards to the European Union, downwards to special-purpose bodies and outwards to agencies’. In fact, the drift ‘upwards’ involves not only the gradual ceding of power to the EU, but also the continuous process of engagement in the complex interdependencies of international politics and government, via negotiations and compromises with a wide range of state and transnational organisations, including foreign powers, international bodies like the UN and the World Bank, and multinational corporations. The drift ‘downwards’ includes dealing with the consequences of privatisation and contracting out, including a constant reconfiguring of the roles and responsibilities of departments as large areas of public service are delivered through market mechanisms by a wide array of public and private-sector organisations. The drift ‘outwards’ or sideways refers to the trend towards the deployment of arm's-length governmental bodies (such as executive agencies) as mechanisms through which policy can be implemented.

Those supporting the ‘hollowing-out’ thesis argue that it produced significant problems and challenges for Whitehall departments in relation to the creation and co-ordination of policy, and to the application of rules governing the accountability of ministers and civil servants. Some traditional Whitehall policy spheres (most obviously those within the scope of the EU treaties) became at the very least shared territories. In the domestic context, marketisation and contractualisation meant that, for example, the civil service (largely Treasury officialdom) was required to manage the central elements of the public–private partnerships scheme (PPP). However, as this last example demonstrates, ‘delegation’ of delivery responsibilities has been accompanied by such a strengthening of controls on budgetary and performance objectives at all governmental levels that the diagnosis of ‘hollowing out’ is by no means universally shared.

The negative effects of fragmentation on the British civil service have been given further emphasis within analyses based upon network theory. Again, the work of Rhodes lay at the heart of this conceptual approach, at least in its UK context, although others made significant contributions to the key ideas (see, for example, Richardson and Jordan 1979; Jordan and Richardson 1987; Jordan 1990; Marsh and Rhodes 1992). ‘Policy networks are a long-standing feature of British government … what is new is the multiplication of networks’ (Rhodes et al. 2003, 26–27). Where the civil service is concerned, the implication of the network theory approach is that the formal Whitehall structures and processes must be viewed as only one element of the policy-making and implementation process. As a result of these network approaches to the understanding of public policy-making, civil servants as individuals and as groups, and the departments they inhabit, are placed within a much broader context in which non-governmental bodies, groups and individuals assume important roles, and officials adjudicate between competing interests, negotiate and compromise to try to secure the outcomes desired by ministers.

Associated with these analyses, the ‘differentiated polity’ thesis synthesises elements of network theory, governance, the hollowing out of the state—and extends the analysis to give particular attention to the pre-existing and potential impact of devolution and decentralisation (see Rhodes et al. 2003). It notes that all three approaches emphasise specialisation and fragmentation, in contrast to the ‘collective’ and ‘unifying’ assumptions and principles of the Whitehall model. Policy networks divide the policy-making arena ‘vertically’ into policy segments, each focused around a particular department, or even a division of a department. The ‘governance’ theme draws attention to the fragmentation of policy programmes between numerous service providers, some in executive agencies still nominally part of the department but split off ‘horizontally’ from senior officials, and others connected only by contracts, funding agreements or looser arrangements, but each specialising in its own particular sector. The ‘hollowing-out thesis’ asserts that, unlike the Westminster-Whitehall portrayal of a well co-ordinated governmental system, the picture is of a very weakened state that struggles to exert a ‘joined-up’ control over functions that have been dispersed in all directions. To these aspects of the way in which the British state is divided into different sections, each with its own characteristics (the ‘differentiated polity’ as outlined by Rhodes 1997, 7), were added after 1999 the distinctive arrangements of the three nations/regions/provinces (Scotland, Wales, Northern Ireland), to which new powers were devolved (or restored). As the specialists of those areas knew all along, there was already a significant differentiation between the component parts of the UK, including notably in the way that the civil service operated in these areas.

While these perspectives in many respects provide us with a richer understanding of the place of the civil service in the polity, there are three main problems attached to the governance/fragmentation/hollow state critique when it is deployed in order to show that the civil service is ‘in decline’:

International Perspectives on Britain's New Public Management

First, let us locate more precisely the British civil service in relation to its counterparts abroad. The OECD has charted the adoption of managerial reforms under the banner of ‘modernisation’ (OECD 2005), a term adopted widely by OECD members though not by Thatcher's government. These reforms are equated with and synthesised as ‘new public management’, defined by Hood (1991, 4–5) as: disaggregation into units headed by a manager delegated substantial autonomy; emphasis on output performance rather than due process; stress on private-sector methods, such as flexible recruitment and pay systems; and ‘parsimony’ in resource use.

Britain is one of three countries that from the 1980s disaggregated large parts of departments into partly autonomous executive agencies—but ‘agencification … has a much longer history in Scandinavia’ (OECD 2004, 7). The OECD (2009) put the British civil service among the leaders on open recruitment, but well behind Australia, Sweden and New Zealand on delegation of personnel management. Britain is second in its use of performance-related pay (behind Australia), but the UK Treasury restricts managers’ autonomy by monitoring bargaining parameters for budgetary compliance, unlike its counterparts in Australia and New Zealand (Rexed et al. 2007, 5–6). Britain was later than Belgium, Canada and the USA in implementing programmes for human resource management, leadership and change management (OECD 2003, 7). The Hutton Report (2004) on Dr David Kelly's death confirmed the amateur nature of Whitehall's personnel management: only in 2006 did Whitehall recognise the ‘human resources profession’. Britain was only average in its use of performance assessments for making personnel decisions—leaving ample room for the Conservative–Liberal Democrat government to ‘make it easier to reward the best civil servants and remove the least effective’ (UK Government 2010, 27).

Britain came first in 2007 for short-term budgetary flexibility but ninth on performance budgeting and 15th on long-term budgeting (OECD 2009). Budgetary flexibility is a pragmatic alternative to planning, which the Thatcher government was notoriously against, yet the long-standing criticism of Whitehall's financial management (see below, and Public Accounts Committee 2008) makes it difficult to blame just the politicians. On ‘parsimony’, Britain came second to Canada in 2006 and first in 2008 for its use of regulatory impact assessments; and sixth in 1998 but equal first in 2005 on other reductions in ‘paperwork’. Finally, a European assessment of online public services put Britain second, behind the Netherlands, for its national e-portal (OECD 2009). The British civil service is therefore clearly among the leading current adopters of modern managerial techniques—though no more so than the Nordic countries and rather less than ‘Westminster system’ colleagues.

Furthermore, Britain's performance is enhanced by the weighting given to ‘programmes’ (OECD 1995 and 2009). Germany, Sweden and Switzerland, with already dispersed public administrations, had less need of disaggregation programmes than Britain with its centralised government and large departments. From a Norwegian perspective, ‘the focus on the UK example carries with it the danger of exaggerating the changes to public service provision’ [since] ‘some of the measures associated with NPM … are ‘long-standing features of many European states’ (Eliassen and Sitter 2008, 103). Another continental critic (Kickert 1997) sees NPM as an ‘Anglo-American’ private-sector approach, inadequate for the public sector. In anglophone literature continental states tend to be ‘portrayed simply as rather backward administrations’ (Pollitt 2007, 20). Hood et al. (2004) confirmed Whitehall's insularity when they discovered that the British language of public-sector regulation did not travel very effectively. Greater awareness in Whitehall of how service providers in the Netherlands were ‘steered’ (Kickert 1997) might well have reduced the numbers of British ‘waste watchers, quality police, and sleaze-busters’ (Hood et al. 1999). Deeper understanding of Germany's arrangements would have made redundant a Scottish devolution policy paper's request that the ‘Civil Service Code should be amended to recognise the importance of co-operation and mutual respect’ (Calman Commission 2009, para. 4.23).

European comparisons suggest too that civil service reform is facilitated in Britain by its constitutional and political system. Creating a hundred executive agencies without new legislation would be unimaginable in Germany, Sweden or France (Pollitt 2007, 17). British governments share with other core NPM states some advantageous levers over the civil service: a majoritarian party system; centralised authority; few statutory obstacles; and an individualist not collectivist culture (Pollitt and Bouckaert 2004, 39–64). With the prime minister able to use the Crown's residual powers to manage the civil service through Orders in Council, it was relatively easy to adapt the civil service towards new needs. Even so, announcements of reform were not always followed by action.

Historical Perspectives on British Civil Service Reform

Using the recommendations of government inquiries and parliament as a proxy for ‘present needs and modern views’ we can compare them with their outcomes to show the hesitant way in which the ‘modern pattern emerges’; the civil service has never been very willing to modernise itself (Gladden 1967, 18–40; the analysis below draws too on Hennessy 1990; Drewry and Butcher 1991; Pyper 1995; Theakston 1995; Burnham and Pyper 2008).

The Northcote–Trevelyan Report (Northcote and Trevelyan 1854) was the modernising project of the 19th century; and its recommendations remain the foundation of the ‘Whitehall model’. It proposed replacing corrupt patronage with the recruitment of young people to a unified service of two classes (intellectual, mechanical), through an open ‘literary examination’ run by an independent authority. The Civil Service Commission was set up in 1855 to examine and certify recruits as fit for service—yet many departments still appointed favoured candidates.

The Commons in 1860 reaffirmed its support for open competition; it was introduced 10 years later, though with temporary exemption for the Home Office and Foreign Office. The same Order in Council empowered the Treasury to introduce department-wide grades, but by 1874 neither they nor the two-class structure had materialised. The proposal of the Playfair Committee (1874–75) for lower-division officials to be recruited for service in any department was implemented immediately: the proposals for senior officials, including ‘outsider’ appointments where internal expertise was lacking, were not. The Ridley Commission (1886–90) recommended a single grade structure for senior administrators, but the higher pay at the Home and Foreign Offices held back implementation until 1919. At lower levels, unified recruitment and grading were made difficult by the growth of new functions. The labour exchanges (‘job centres’) and national insurance commissions (welfare provisions, delivered by the voluntary sector) required mature staff, who were recruited by competitive interview (Gladden 1967, 23). The MacDonnell Commission (1912–15) on recruitment did not respond to the new conditions (Hennessy 1990, 56). It criticised the use of outsiders on the insurance commissions, and advised that ‘professionals’ be recruited in separate open competitions. However, its deliberations were overtaken by the 1914–18 war, during which the civil service relied on temporary recruits of all ages from all sectors.

After the war, the Haldane Committee on the Machinery of Government (1917–18), Bradbury Committee on the civil service (1918–19) and parliament concurred on the need for Treasury-led unification. Orders in Council and Treasury circulars 1919–20 created general ‘clerical’, ‘executive’ and ‘administrative’ classes; gave the Treasury responsibility for civil service management; and transferred responsibility for senior appointments to the prime minister, advised by the Treasury Permanent Secretary, who was made Head of the Civil Service. Under Warren Fisher, its head 1919–39, the civil service became a less politicised, virtually self-managing service. Fisher said he used interdepartmental transfers to produce ‘general managers’, not ‘experts’ (Hennessy 1990, 75); he thereby created a cohesive but inbred elite. Although the 1939–45 war brought in outsiders, it was generally back to ‘business as usual’ in 1945 (Theakston 1995, 56–82) and a ‘missed opportunity’ (Hennessy 1990, 120–168). Yet there were reforms: the Robertson inquiry of 1941 led to the merging of the different corps linked to the Foreign Office—though also a separation from the Home Civil Service; and the Barlow report of 1943 enabled the scientific specialisms to unite, though not with the ‘general’ classes. Personnel policy became more progressive: the ban on married women was lifted in 1946; equal pay was conceded in 1952 and implemented from 1955; ‘adults’ were admitted to clerical grades from 1955, and the examinations for clerical and executive grades were replaced with educational qualifications followed by interview.

In 1946 top officials rejected the idea of an inquiry into their capacity to handle the new welfare and industrial functions (Theakston 1995, 65–67). Yet highly placed wartime ‘temporaries’ were already warning of the ‘considerable need for change’, since the functions were no longer regulatory but ‘more those of management’ (Lord Franks, in Hennessy 1990, 124). Parliamentary criticism of Treasury financial management led to the Plowden Report of 1961, which advised that administrative grades receive training in economics and management. Following defence procurement scandals, and with the Labour opposition (led by a wartime government economist, Harold Wilson) proposing radical reform, the Conservative government responded with a training scheme for administrative class recruits, and a few posts for ‘late entrants’ with managerial experience. Wilson's first government created career structures for economists (lacking in many departments) and information officers. His second, urged on by parliament, set up in 1966 under Lord Fulton, another wartime temporary, remains the last major inquiry on the civil service.

The Fulton Committee (1968) favoured a service that was better trained and managed, integrated generalists and specialists and was connected to the community. Its recommendation that administrative class officials specialise in economic and financial administration, or in social administration, and that specialists be trained in management, was addressed with a new Civil Service College. However administrative civil servants did not value the college, whose status grew only in the 1990s, when the Conservatives emphasised management and performance, and again from 2004, when Labour pressed officials to acquire ‘Professional Skills for Government’. In 1972 the top three administrative grades (only) were opened to specialists: further extension took place under the Thatcher government in the mid-1980s. Fulton wanted the senior service to be in touch with other sectors through late entry, short-term contracts, secondments and outward movements. There were few secondments to business until the 1990s, when the Major government set targets, and opened up competition for top posts, forcing civil servants to take their own experience record more seriously.

Fulton considered that efficiency would be increased by dividing departments into units accounting for their spending against defined objectives. Little happened until Thatcher's Financial Management Initiative of 1982, strongly supported by parliament, required departments to adopt such systems (but see Public Accounts Committee 2008 for elements outstanding). The Fulton committee hoped the government would ‘hive off’ executive activities to non-departmental agencies: some were set up by the management-minded Heath government of the early 1970s, but the idea was implemented more extensively under Thatcher as executive agencies. Opposition to agency officials accounting directly to parliament (part of the case for ‘decline’) was reduced by a procedure which recognised that officials are often best placed to provide information, while ministers retained political responsibility. Fulton also suggested that ministers should be able to draw on advice from a departmental policy planning unit, and from temporary specialist advisers. Forward planning remains of limited interest in Whitehall (the Transport Plan 2000–10 lasted two years); however Wilson and most subsequent prime ministers established small policy units in Number 10 (Heath created instead the Cabinet's Central Policy Review Staff) and the phenomenon of the ‘temporary civil servant’ making a party-political input into policy-making became generally accepted by Whitehall, if not by some traditionalists, and then by the Constitutional Reform Act in 2010.

The Fulton Report spawned supplementary inquiries into secrecy in policy-making and on administrative class (‘fast stream’) recruitment. The Franks Committee (1971–72) advised limiting the scope of the Official Secrets Act 1911; however the law was not reformed until 1989, by which time reformers’ demands had broadened to ‘open government’ and ‘freedom of information’—in part to improve the quality of policy-making. The Major government's 1994 Code of Practice automatically put more information into the public domain, but the 1997 Labour government had to overcome considerable departmental opposition before its Freedom of Information Act of 2000 was implemented in 2005 (Burnham and Pyper 2008, 187). Successive inquiries on ‘fast stream’ recruitment failed to satisfy parliament, and the critique expanded from possible Oxbridge bias to the independence of the Civil Service Commission and the barriers facing women and other social groups. Civil Service Commissioners were replaced partly and then wholly in 1991 by outside human resources experts (more evidence of decline for some analysts). Under pressure from the Commission for Racial Equality and Modernising Government (Cabinet Office 1999a, 59–60), procedures were changed to reduce cultural discrimination and increase diversity (Cabinet Office 1999b). Equality programmes introduced into the civil service from 1984 for women, and from the early 1990s for people from ethnic minorities or with disabilities, helped answer the Fulton Committee's demand for more ‘contact between the Service and the community it is there to serve’. Civil service leaders eventually asserted that the civil service needed ‘to reflect society’ in order ‘to be effective in delivering services in an increasingly personalised way’ (Cabinet Office 2008, 24).

The changes inaugurated by Margaret Thatcher seemed more intense because they included Fulton reforms, long delayed by top officials and trade unions and/or not actively pursued by ministers overwhelmed by economic problems (Theakston 1995, 106–114). The Thatcher reforms were underpinned by New Right theories on the bureaucracy, not by the Fabian assumptions of Fulton, and they were not proclaimed as ‘modernisation’, even if the strategies resembled other governments’ modernisation programmes. Christopher Pollitt and Geert Bouckaert (2004, 99–100) prefer the term ‘marketiser’ for Britain, reserving the term ‘modernisers’ for countries in which Weberian bureaucracies were being updated, not rejected. Nevertheless, ‘market’ aspects of the Thatcher reforms (contracting out, competition with outsiders for posts, quantified performance outcomes) were important stimuli to the self-modernisation of the civil service.

Furthermore, there was a modernising character to the market mechanisms added by John Major's Citizen's Charter of 1991 (setting service quality standards of direct concern to users), and stimulated by Major's dislike of the way ‘anonymous officialdom’ treated ordinary people (Seldon and Baston 1997, 188). User-oriented reforms responded to societal changes, such as the decline in deference (Pollitt and Bouckaert 2004, 293), which ministers recognised in putting more pressure on the Benefits Agency over quick turnaround time for clients than over accuracy of assessment (James 2003, 102–103). The Charter was implemented across ministries, although ‘possibly not since the Wilson government … had there been so much resistance … to a Prime Minister's wishes’ (Seldon and Baston 1997, 193). Citizens’ Charter-style performance standards were continued by the New Labour government, adding to the ‘quality police’ enumerated by Hood et al. (1999). Modernising Government and Modern Public Services challenged departments further by promising users ‘joined-up government’ between service deliverers of all types, and by introducing Public Service Agreements—performance standards for departments with the performance in question likely to involve steering their ‘partners’ in the public or private sector (that is, ‘governance’). For Moran (2003, 62–64, 125–129) the Thatcher-Major-Blair project was modernism, even hyper-modernism, replacing the ‘club government’ in Whitehall with formal regulatory powers: the tacit knowledge of the policy-making elite official based on experience of the Whitehall world has been downgraded in favour of explicit knowledge. Today's senior civil servant is expected to use managerial skills to produce measured, published outputs from a finely regulated and controlled workforce.

From the historical perspective the British civil service is no longer corrupt and politicised but more directly accountable, more open and interconnected with society and its demands. The record shows a progressive but often delayed adoption of reforms that help the service match up to ‘modern needs and views’ as defined by government and parliament. International comparisons show it to be among those civil services that have modernised most in the past two decades, but not all civil services have needed or been required to modernise in this particular way.