The Oral History of Evaluation

George, how did you first come to work as an operations research analyst in the Defense Department following your MA?

I needed a summer job when I was studying at Georgetown University. In those days the federal recruitment process was they had a big central file and you just sent your name in with many thousands of people, and somehow they picked mine out for a summer job. I started working for the Assistant Secretary of Defense Comptroller for the summertime. Then they asked me to come back for the next summer, and then they asked me in my last year of graduate school if I could work a few hours during the week, and then when that was over they asked me if I would be willing to work full time for the Assistant Secretary of the Navy for Financial Management.

I actually started working for Defense—1969 is the year that’s on my official record, but that’s an average of those summers—it was probably ’68 and then later on moved to Health and Human Services (HHS; then called Health Education and Welfare [HEW]).

I’m curious about the cultures of the two different places.

Very different but both extremely beneficial. I would say that for a rapid learning of how to work in the government, you should go to Defense, because they teach their young employees how to become civil servants the way they train high school students to take care of jet airplanes and everything else like that. They’re just very, very good at training. It’s formalized. They show you, “This is how you write a memo. These are the parts of it. This is the way the organization runs. This is who you have to talk to.” They really would take you and work you through it in a very gentle way. They were military officers of course, but they were still very humane. Where I worked they were also very attuned to the ideas of saving money. We were working for the Comptroller’s Office, so it was not a warlike atmosphere, but I was working for many military people who of course had served and who were very proud of their professions, and I picked that up and I gained a lot of respect for military people because of their dedication not only to the warlike aspects of their career but to the professional as well, and I had a lot to learn from them.

When I went over to HEW, the world was totally different. It was much less formal. If anything, to me—not just given that I started in Defense, but because of who I was, how I was raised, it was almost confusingly informal. I then had to learn another whole way of interacting and dealing with people. And they didn’t teach it to you. To give you an example, in Defense, let’s say that we had a simple thing like a report that had to be reviewed by the office. And let’s say I was the staff officer for it—basically it was my job to give it to everyone in the office, gather their comments up, write the memo with the comments, and send it around. Well the way the answer would go it would be something like “From: The Assistant Secretary of Defense Comptroller; To: The Chief of Naval Operations; Subject: Report number such-and-such; Reference A: That report again; Enclosure one: Comments under report.” Then the memo would be “One: Your study was reviewed as requested.” There might have been a “Reference B” which was a request for comments. “Your study was reviewed in accordance with reference B. The results of our review are attached as Enclosure One.” That’s it! It’s one, two, three. A, B, C. It was always with the greatest courtesy of rank and following carefully who the copy to-ers were. Everything was prescribed. It was very orderly. Now I get over to HEW and first of all, no one is wearing suits. Many of them have beards and corduroy coats, and many of them are wearing loafers, very casual. Their idea of a comment was, let’s say that you received something from CDC (Centers for Disease Control) and you were Assistant Secretary of Climate and evaluation was to look at it. The memo there was “Bill, Got your study. Great job! Thought you might be interested in the attached comments. You got any questions, call George Grob.”

It was a totally different world! Much of what I was taught seemed disrespectful to me. That’s how they operated. And they would be totally making fun of you if you said “Dear Mr. So-and-so.” Later on we did learn when you were supposed to behave, but there was a lot more of this just back and forth first-name basis, just simple things. Lower-level staff people being given these positions of coordinating things. Now you still have to get your clearances and all of that, but it was just very different:

“Hey come on in. I thought we’d talk about this,” whereas in the military you’d make an appointment with the Captain. Totally different world! I will tell you because of my upbringing it was a lot easier for me to absorb the military than it was this more casual thing, but that is the world I lived in ever since. I was always very much aware of the fact that the people you’re dealing with are people you can learn a lot from. So I never lost a chance to learn as much as I could.

Now the move came about—you left when the war was over?

I was an operations research analyst because I had this degree in mathematics. We were analyzing weapon systems acquisitions and cost overruns and whether they were being procured in an orderly way, whether the cost was under control, whether they were being done on schedule, and whether they were meeting specifications. We were doing independent estimates of the cost to compare to the estimates of the advocates of the programs, things like that. That was in the days when operations research analysis and that general kind of model building and analysis was in a boom era in the military world. They had started adopting it during World War II to do all the logistics for the huge enterprise, to keep track of everything, to find the fastest way to get stuff wherever it had to go. A lot of it was taken from the auto industry which was also being very much modernized by those people, and some of those people were there at Defense helping do that. They had a good reputation for being able to do this kind of analysis. And there were people in the domestic agencies that began to think “Well, maybe the same kind of analysis could be done for the domestic agencies.” So they put out feelers and I was brought in that way. My boss came to me and said “George, there’s quite a bit of interest over there in HEW and other domestic agencies to see if some of the things we’ve been doing over here could help with the domestic programs. Your name came to mind. Would you be interested in doing that?” That’s how I got switched over. It wasn’t the war dividend; it was more the fact that the field was catching on.

Another thing that was happening at the time was President Nixon wanted to shut down many of the Great Society programs including the War on Poverty and the Office of Economic Opportunity, OEO. He could not get authority to do so from the Congress, but he did so administratively. He moved the analytic staff from OEO into HEW right about the time I went over there. There used to be a tiny group of people who were the Assistant Secretary for Planning and Evaluation, and suddenly they had this influx of people who were doing quite a bit of good analytic work for the OEO. They couldn’t shut that office down, but they were able to move that analytic staff out which took a lot of power away from the policy development effort of the day. There were many people who really were very sad and very upset that he was trying to shut this down. However, he probably didn’t realize it, but what he had done was seed the largest domestic agency with this analytic power. And the people were really very good at this, people like John Palmer, Larry Orr, Michael Barth, and Tom Ault come to mind—I can’t remember all their names. For many years they were doing impact analysis using randomized experiments. They had, for example 5 million dollars a year for health insurance experiments where they enrolled people into maybe five different health insurance programs to see what the utilization of medical care would be like under different cost-sharing arrangements; same thing with welfare. They had theories about what combinations of work and welfare would best incentivize reductions in welfare enrollment. What’s the likelihood that people will leave welfare and get a job as opposed to saying “No, that cut is too steep; I’d rather just get welfare?” They actually ran these experiments for many years. They also started the Michigan Longitudinal Study. They set up the Poverty Institute in Wisconsin. And these today remain very vibrant things. That Michigan Longitudinal started with 5,000 low-income families and they have been keeping track of those families and their many generations ever since, so you could see over the long run what happens in a culture of economic need. Did welfare stay in the family? How did people break away from it? They were very systematic about it. They were doing very revolutionary things way back then, and that’s what defined the work.

Having not done that kind of work and being brand new, they put me in charge of the budget and the grant program. I thought that was wonderful because I could put my nose into anything I wanted—the one thing they were not good at was managing. They were all researchers and when they came over, they didn’t even know what grants they had! When the next budget was about to be renewed, they honestly did not know which grants were ending. I had to get all the grants, make tables, including these very large programs. Then when there were budget shortfalls, I had to come up with proposals as to how to keep these things going and which ones you could drop. It just thrust me into the middle of all that kind of stuff. These people were really outstanding at what they did and it was a pleasure to get to know them.

You were Director of Planning and Policy Coordination at ASPE (Assistant Secretary for Planning and Evaluation) from 1976 to 1988, preceding and then following the time that Joe Wholey was Deputy Assistant Secretary.

When I went to work for HEW they had to stick me some place, so they stuck me in an office that I would call the process office. For policy work, there were things you had to do. For example, for every year you also had to send in the legislative program of the President or the Secretary. For the budget you say “We only have so much money and we need more.” But simultaneously there is another appendix which is about all of the legislative proposals that you want. And that would mean that for every program that needed to be authorized, you’d have to have a specific proposal to reauthorize it. Unlike entitlement programs, discretionary programs were on a 3-year cycle, 4-year cycle, or 5 years. Of course, you would never just propose to say “Yeah, give me another 5 years of this.” They would want to improve it. I had the job of coordinating that effort. I had to be the producer of these proposals, which means I had to get everyone to submit their stuff, look it over, and make it pretty. And my jobs ranged everywhere from getting the typing done all the way up to suggesting that Medicare people might want to consider what the Social Security people did with a similar problem 5 years ago.

Another thing was every time a new Secretary came in they would want to put their mark on it. They would usually commission a series of studies or policies that would need work. And they would usually convene all the Assistant Secretaries and say “Okay, let’s have a day’s retreat here and figure out what we are going to focus on.” I had to arrange for those. Again, my job was everything from making sure I got the coffee order straight—who got cream and sugar—all the way up to preparing the papers with the agendas, and then just making sure that everything went right.

Then there were other things, planning things, and of course there was the evaluation function which was centralized in the sense that there was this one percent evaluation set aside. They actually tried to run it from the Assistant Secretary of Planning and Evaluation’s office. I had to help out with that. All the proposals throughout the department had to come and they would be reviewed and then have to be approved by the Assistant Secretary before they were funded. It was a job of running what I would call the policy making machinery. The offices were divided up as you would think they would be. There was an Assistant Secretary for Planning and Evaluation, and then there would be Deputy Assistant Secretaries, usually political appointees, responsible for health policy, responsible for Medicare and Medicaid policy, responsible for human development policy, Head Start, the aged. We also had the Education Department and the Social Security Administration. So there would be one in charge of income security, another one in charge of education which included elementary education, elementary and high school, post-secondary, graduate education. Plus there was a lot of vocational education. Then there were cross cutters like rehabilitation which was a combination of a human development program and an education and training program. And these giant programs like Head Start and the programs for the aging. We in the Secretary’s office had to be able to move in on any one of them. One time the guy who was in charge of disability policy got a job in the private sector so they needed someone to be the desk officer for retirement and disability programs. They said “Hey, George. Can you do that for the next 6 months or a year or so?” For a year I suddenly had to become the expert in disability.

But this planning and policy office where I worked was theoretically a process office, not a substantive one—except that it was, and here’s why: what would end up there would be things that either were of a management nature or were not of interest to the high-flying policy analysts. Examples were Indian sanitation facilities on Indian reservations; how much money should the northern Marianas get in social services next year? The analysts in the substantive offices had bigger fish to fry. They weren’t interested in that. We took care of that stuff. The other thing we took care of was the stuff that didn’t belong in any one of them. I’ll give you one really good example. There was a proposal that all child support enforcement agreements would include a requirement that if you were employed and you had the option to gain employer health insurance coverage for your family, then you would be required to take it if you were the absent parent of a child on child support. Now what kind of policy is it? Is that a health policy? A Medicaid policy? It wasn’t Medicaid because it was employer insurance. Was that a child support policy? So which office should analyze this proposal—the health policy office? The income security office? The human development office? The truth of the matter is that at that time the people who were working child support enforcement, welfare, and health care programs were not interested in what they thought was a relatively small issue. And each one was saying “It’s yours. It’s yours. It’s yours. It’s yours.” Our office would get stuff like that. So I ended up preparing the cost analysis for that one. And by the way, that one took off and turned out to be quite important after all. But it could be also bigger things, like maybe the Secretary would come in and want a major initiative for helping people with developmental disabilities. Well, what is that? Is that a social security policy? Is it a human development policy? What kind of policy is it? Some were “None of the above.” Maybe like how to help avoid children getting alcohol, that kind of thing that just didn’t fit neatly. But unlike things that weren’t wanted, there were also huge policies. Somebody had to administer those, so our office would get called in to help. That was a load of fun. We got involved in the really big things. And the attitude that I took on it was that a lot of the organizing of these big projects was administrative. “Well we need to have a meeting. Who should come to the meeting?” Then you sit down and you have “Well what are the issues? Can we come to an agreement? Will we agree to this? Okay, can you type it up and get it back to me tomorrow?”

A lot of times, this had two advantages for me. One, I was in on everything. I was in on all of these exciting policies, so I learned a lot. That turned out to be handy because I could do just what I said a moment ago. If they were struggling with a policy in one area, I could say “Well, you know, Social Security had a similar problem 5 years ago. Here’s what they did. Obviously you don’t have to do this, but I just thought you ought to know. Here are the names of five people you can call. Here’s a copy of the documents.” I became a cross-fertilizing resource for these people, and they were glad to have that. They never resented my doing it because I wasn’t stepping into their world and taking it from them. I was always offering to help them. But where I did start getting more substantively involved was when no one wanted to do what I called the nuisance work. So you’re at a meeting and its 5:00 o’clock at night and you’ve had this really hard meeting and they’ve all decided “Okay, it comes down to this. We finally got it. We’ve got three options and we know what they are A, B, C. We’ve got to have a memo like that tomorrow at noon. We have a rough draft of it, but my kid has a soccer game right now and I can’t do that. George, would you do it?”

What I decided to do was basically say: “I’ll never say no.” When I had openings, I filled them with people who could do that. I had the key for every Xerox machine on any floor of the department. I would be going around handing things out on whatever floor I was on. I talked to somebody and they’d say “George, that’s great. By the way, you have such and such?” I’d say “I do. I’ll make that copy for you right now.” Then I’d go down the hall, make a copy and I’d bring it back to them. I told myself “I’m never going to say we can’t get it because I can’t get it typed or because I can’t get it copied or because I can’t distribute it.” I said “Okay, look, I’ll tell you what I’ll do. We’ll get it copied. I’ll get it to everybody tonight and we’ll give everyone until 10:00 o’clock tomorrow morning to get their comments in. I’ll do that, and if they can’t reach you, they can call me. I’ll get you their comments.” Then someone had to write up the summary of that. And no one was interested in that stuff. That was peon’s work. I said “I’ll do it for you.” Now it is 2:00 o’clock the next day and you’ve got to run this thing around to everybody to get the clearances for the Secretary. You’ve got to get four or five Assistant Secretaries who were at the meeting to sign off on it. How are we going to do it? Somebody has got to walk it around, and I’d say “I’ll walk it around.” I got to the point where “George. Oh hey. How you doing? Could you explain this to me?” You know how it goes. By the time a few years were up, I knew every Assistant Secretary by name, every Deputy Assistant Secretary by name, they knew me and if they wanted something, they knew I could get it to them. And suddenly, gradually, I was involved in questions of substance, but never taking away from the expertise that people with prime responsibility had. I was raised with a sense of duty. I didn’t care who got credit for it. I just wanted to make sure it got done.

I learned how to help without being a nuisance or detracting from the lead analysts. For example, I’d go up to people and say “I don’t know what we’re going to do. These people in the front office, they want this thing by 10:00 am tomorrow morning. What do you want me to do? Do you want to draft it and let me get it typed and correct the little stuff, or would you rather have me take a whack at it and then you can make sure all my things are good. Which would you rather do?”

I learned how to be of service to these people, preserve not just their sense of ownership, but to actually exploit what they were good at. To say “We’ve got really great thinkers around here. We can really do a lot of stuff, but someone needs to help them out without imposing on them.” I don’t care whether they thought that I was the guy to get the coffee or whether I was the guy who hit all the cross-cutting ideas. I didn’t care. The trick was “Can we make it happen? Can we get this done? Can we get options? Are they credible? Is it professional? Will it be accepted?” That was the attitude I had. It was very empowering. I met all these people. I learned a lot about everything.

This is really the beginning of evaluation!

Oh absolutely! It really was. It’s really difficult to distinguish the boundaries between policy work and evaluation. If you’re evaluating—if you’re looking at developing policies, you’re evaluating current policies on the fly. They were consuming studies. One of my first jobs was to set up a little library. This was long before electronic things, so that any time someone published a study we would get a copy of it, and we would have it in our library. Again, the idea was give people the resources they need to get their jobs done. I was glad to be the librarian if that’s what it took. Whatever it would take, just get it done.

Did you begin to think of yourself as an evaluator at that point?

No. In fact, I didn’t like evaluation. I thought that I was a program analyst, and then eventually a policy analyst. Now I recognize that a lot of what I did was evaluation work, but I didn’t think of it that way because no one was looking for evaluative work in those days. I would divide the world up into three groups. There was the kind of thing that was research in the sense that you’d have to carefully do something, and then it would take you 2, 3, 4, 5, 10 years to get the answer; the long-term, carefully orchestrated work. Then there was the kind of thing we needed to get all of this down and write a decision memo for the Secretary in 3 days. Again, you were using the evaluation studies and you were really evaluating things, but it was on the fly and doing it quickly. Then there was the other stuff that was in the middle, which would be the evaluation study that’s going to take you 6 months or a year to do. So those were the three worlds of evaluation. Most people thought of that middle world as being the evaluation world. They thought the other two were policy research. And they thought that what my office did was not quite policy analysis but policy coordination.

I never thought of myself as an evaluator. In my mind the evaluators seemed very arrogant. In those days the question was “What do we really mean about evaluation?” You would get into these discussions with people, and everyone would have their own opinion “Oh, it really means this. It really means that.” I knew what evaluation means. It means to evaluate something. I don’t need to get bogged down on “What do we really mean by evaluation?” But people had some strong opinions on that and seemed to believe that only they and no one else really understood.

Then there came a turn, and I was interested in whether or not you could actually change things. I was in a world now with people that did that for a living. Secretaries would come in and say “The world’s going to be different after I leave.” And they meant it. And for many of them it was true. I picked up a lot on the legislative side I mentioned before, the reauthorizations. We had to prepare all that paper and then we had to work with the General Counsel’s office to prepare draft bills, then they had to track it and make sure where it was—in which congressional committee, “Did it get through two committees yet? What vehicle is it in?” What I started doing was every 2 years, when that Congress was over, I would pretend I was busy doing what they wanted me to, but instead would close the door or go out some place, I would look at what happened during the Congress. I’d say “What made it? What didn’t?” We had two or three-hundred proposals we sent over there, and only a handful would ever make it through that Congress. A lot of them after several congresses, they’d make it. Remember, “making it” was actually passed by the Congress and signed by the President. I would also look at “What got through the Congress but didn’t make it past the President?” I would go down the list and I would compare it to what we had proposed. It was a minority of things that would get through. So I asked myself the question: which ones made it and why? You can see some obvious answers. One was support of the President, support of the Secretary, that kind of thing. But at a more mechanical level, what made it? What I discovered over the years was there was an amazing correspondence between the proposals that made it and those that had what I call a “nifty” evaluation report. A nifty evaluation report would be 20 pages long, a two-page executive summary, an attractive sounding title, and a very compelling argument that we had done some work and this seemed to prove that this would be an effective approach. Remember, long before e-mail, long before anything like that, I found that the existence of a nifty evaluation report corresponded highly to ones that made it. I began to see if I could find any evaluation reports and I’d make sure that the people developing the reauthorization of this program had at their disposal any reports or other studies there were germane. I would try to help them or I would get people to help them and say “Okay, did you read this? Did you read that? You might be interested in knowing that.” I thought that would improve the odds that you’d be successful.

Then I guess that takes us up to the point where I got this invitation to become the Director of Evaluation and Inspections in the Inspector General’s office. This was 15 years in. Now I knew a lot about that function because it was one of the policies that I worked on under Secretary Califano. He was frustrated with his inability to get relevant policy information that was professionally done quick enough for him to use and that was not tainted by the opinions of the program advocates who were sending it to him. For example, if he really had some misgivings about how Head Start was working in a particular area, he could ask the head of the Head Start program, the Assistant Secretary of Human Development Services, for a quick response, but he’d never get it quick and when he got it, he didn’t trust it. He wanted to set up some kind of an operation that would enable him to get studies very, very fast that were professionally done but that were totally independent of the program proponents. My boss at that time was given the job of figuring out how to do this and I had to help him out. What they came up with was the idea that they could assign this function to the Inspector General’s Office because they were independent and they were located all over the country. They commissioned that. And then after a few years the people who came over to see me had no idea that I had played a minor role in setting that up, they asked me if I wanted to be the field director for the evaluations. They didn’t call it that. It was then called the “Office of Analysis and Inspections.” That fit me well because I said “I’m a policy analyst and a program analyst.” Somehow they knew that I could organize all the rest and their primary job now was they had maybe 100 people, and they had to do all these studies every year, and they just needed someone who could organize it.

So I ended up with a staff of about 100 people who were doing maybe 20 or 30 studies a year. And they were saying “That’s just not enough for that kind of a resource.” They said “George, we need you to really make this office productive.” In about 3 or 4 years we got up to 60 studies a year. The other thing was I knew where all the policy machinery was, and I knew all the ropes and bells. There has got to be a lot of interaction with the policy people from the time you start the study to the time you end it because if you really want them to act on what you’ve done, then you’ve got to be dealing with them. You still need to be independent, but you didn’t want to work on things that were totally irrelevant to them, and you didn’t want to show your ignorance of policy by going over ground they’ve been over for the last 10 or 15 years. We had to learn it from scratch. They actually had brought me over there because they wanted that connection. They were one of the first IG offices to ever deliberately set out to do that, to connect their independent evaluations with the policy-making system. That’s why they brought me over there, because they knew I was running this other operation and had been there 15 years so I really knew the ropes. Then I looked at some of the stuff they were doing which was really quite good, and I was a little fearful. I saw their studies and said to myself “They’d bring me over to do this?! I don’t know that I could do better than what they are already doing.” But I went.

Fortunately the reason I could do better than that was because Mike Mangano was the head and he had this wonderful sense of management. He developed this staff over there so that it was a powerhouse. They had these GS 15’s who I had never heard of before, but boy they were really good. And they cultivated their staff from the beginning. They would only hire college graduates who were in the top 10 percent of their class AND who got along with others. They didn’t care what their field was; that’s who they hired. Then they would bring them up. They would require annual training. They would put them on teams right away. They were just very, very solicitous about developing their staff. Now later on we changed that policy to look for a mix of people, as well as people with a master’s degree because they turned out to be more mature. More mature analytically, but also they knew what it was like to worry about earning a living and just what it took. You had to know how to deal with people or you wouldn’t have food tomorrow night. We strengthened the office considerably. We continued to adopt policies to really run this first class office. So when we were doing 60 studies instead of 20 or 30, they were REALLY good studies. The staff that Mike Mangano had put together was really what today we would call a learning environment. It was important that everybody get good at what they do, and that was part of the culture.

Once we started delivering products, well there was a market for it. Members of Congress, the Secretary wanted this stuff. There was a part of them that was of conservative bent, you know- save money. The attitude was, “We love the poor, but dog-gonnit don’t blow this money. You could help a lot more people.” But they really did want to help people. So we issued a whole series of studies to help people in nursing homes where there were increases in bed sores, increases in accidents, increases in malnutrition; trends over the years that were running contrary to modern society. Why were these conditions deteriorating? We did studies of wasted money for medical supplies in Medicare, sending the money to supposed suppliers that didn’t really exist—where the medical supply store was the trunk of a car, or four mailboxes in a row, or a store in which there was a receptionist and nothing in there. We started sending people out to do inspections and to analyze all this kind of stuff. And suddenly we were saving billions of dollars a year.

Then there are some really hard problems. One of them was Medicare home health. In the early 90's it was a 3-billion dollar a year program. Then it took off by 1996 or 1997, it was up to 18 billion dollars, and the next year it was going to be 20 billion. Now it might have been a boon that suddenly people who didn’t have a home health benefit, finally realized that there was one. Advocates said “People should get what they deserve.” On the other hand, maybe it was fraud. Maybe it was mismanagement. I remember when we discovered there was a lot of fraud and there was a lot of mismanagement. It wasn’t all evil. It was like somebody would have their wife in terrible shape and they needed home care, but the rules for home care and insurance are very stringent. For example, even in our insurance you had to have a prior hospitalization to be eligible for home health care. The fact that you need home care doesn’t mean your insurance company will cover it. So the rule then was that you had need for skilled care in the home, and if you had that, then Medicare would pay for everything. If somebody would go up to the doctor and say “I don’t know what to do. I’ve got to go to work. We can’t live without my income. I can’t afford a nursing home, and my wife is not sick enough to go to a nursing home. I don’t know what to do.” Then the doctor will say “Listen. I’ll tell you what. Bring her in. I’ll diagnose her and I’ll make sure the diagnosis comes out that she needs skilled care at home then Medicare will pay for it.” So then you say “He’s a good doctor, right?”

A lot of it was that. But a lot of new companies were coming in and just exploiting this. The average number of visits went from 23 to like 100 because Medicare paid by the visit. I remember taking a plane ride to one of my regional offices, and I was looking for something to read in the front pocket of my seat. In one of the magazines I read an article about “Ten ways to run a business from your home” One of them was home health. All you basically needed was a recipe box, recipe cards, and the names of a few nurses, and you could own a home health agency. We started studying home health like crazy. We ended up doing maybe 10 or 15 studies. I remember my staff and I looking at this and saying “I can’t stand this! People need home health by all means, but this isn’t right.” And I recall thinking “I want to see that thing come down.” I had a lot of staff say “George, you know you’re not going to succeed. Maybe we can stop the rate of growth, but it’s still going to go up.” And then they added “Well, George, you're up against a large industry. We’re out-lawyered every place. You’ve got to get anything you do through the Department, through OMB, the Secretary has got to agree. You’ve got two or three committees in each Congressional chamber with subcommittees and committees. They’ve all got to approve it. And then you’re going to have the industry come down on you like crazy. You’re naive.” I remember saying to them “You know, those lawyers, when they came out of the womb, they didn’t know any more than you knew when you came out of the womb. Everything they know they learned. If they can learn it, we can learn it. You guys are smart. There’s no reason why they’re so much smarter than we are.” And they said “Well George it’s going to take 5, 6, 7, 8 years, and you’ve got to convince all these people.” And I remember saying “Well, you know, you haven’t told us we can’t do it, you just told us how to do it. Give yourself 5 to 7 years and here’s a list of the people you have to persuade.” And so we’ll persuade every one of them and we’ll get the press behind us and we’ll do it. “Anyone in? I’m not going to make anybody do this with me, but are you in?” And so, some said, “Yes, I’m in.” And, 7 years later, they passed a law, and it changed the way that Medicare pays for home health. Instead of paying by the visit, you pay like hospitals. Patients get a fixed period of care for a fixed payment. The doctors and home health agencies then decide how many visits they get. The next year the home health dropped by 40 percent. It dropped down from 18 billion to 8.5 billion. The auditors had done studies showing that they thought there was a 40 percent error rate. They were right. It was exactly 40 percent. We got the Congress to do this. It took long but we did it. From that point on, some of us had the attitude “Don’t tell me we can’t do this. We can do this.”

And so we took on big things. We took on the prescription drug community, and we got huge savings there because they were charging exorbitant amounts of money for Medicare drugs. We took on child support enforcement. The numbers were like 25 percent of the kids actually were getting any money at all from the absent parent. The agency said “Let’s make a goal to get it to at least 50 percent.” We did one study after another to determine what was in the way. Why couldn’t you identify who the parent was? What would it take to establish paternity? What would it take to establish an order? We discovered new ways of doing things and promoted them. One change was to require cheek swabs for identifying paternity. This was a great improvement over drawing blood. Some nurses discovered that if the father was in the hospital on the date of birth he would say “That’s my boy!” But the next day he would say, “No sir, it’s not my child.” So the nurses started getting paternity agreements the day their child was born. We found out about that so we wrote that up and passed it around. We found all kinds of things that you could really do. For example, for child support there’s a rule that if the absent parent wanted to have the child support payment withdrawn from his paycheck for convenience’s sake, they could request it, and then the employer would have to do it. That got changed to “You don’t get a choice. The default is it’s going to get it withdrawn from your paycheck unless you get an exception, and you have to have a darn good reason; you’re going to have to apply for it.” Now, it is routine: if you’re the absent parent, when you get any job right now you have to acknowledge that and they will immediately begin withholding the money from your paycheck.

We did it. We helped get those numbers up. Of course, not alone, but we contributed to the enterprise of policy change. Those are examples of things that we did during that period, all because there was this band of people in the office who were really interested in making something happen. And they all wanted to be people who could say “Geez, this is what we did.” We used to celebrate these successes. It was really a life-giving activity.

We had a rule. Every report had to be no more than 20 pages. It had to have an executive summary, no more than 2 pages. In those days all reports were physical. By the time I left the office, they didn’t print reports any more. You sent it to everyone on the Internet. None of our readers wanted a printed copy. But in those days you had to have printed copies. Another rule was that the cover was a simple blue cover, but inside the first page was the same cover again in black and white. We allowed printing on only one side of the paper, never two. The only binding we wanted was a staple in the upper left-hand corner. Why? Because we knew then that no one was going to act on our recommendations unless they read the report. We wanted the report to be short enough that they would read it. If they wanted more copies fast they could easily make them by sticking it in the Xerox machine and make 10 copies for everybody, right? You simply remove the staple. You throw the cover away because if you Xerox that it’s going to look like mud. You want them to be able to read the words. And when you stick the thing in the Xerox machine, you’re not going to be missing every other page because no one knows how to copy double-sided pages if they have to take it out and turn the document around. They always screw it up. My motto was that your report has to be like the rock band’s garage-produced tape. All they’re ever going to know about you and your music is playing that cassette tape. That’s who you are. Same with our reports. You’ve done all this work. What you get is a 20-page report printed on one side in black and white—no color please because it looks like mud and all the graphs don’t make any sense—with a simple staple they can remove and stick it in a Xerox machine and make it real quick. That’s how we get across to people.

We paid attention to stuff like that. What does it take to get it through? Who should you talk to? Who’s on the distribution list? How do you get your public affairs staff to get it to the Congress, to get it to the press? We got plenty of that. They were interested. They need stories. But, also the professional staffer, the people representing all the professional rags. Make sure that the health people know about this. They’re going to hate it, but they’re still going to publish it and then make fun of you. So get it out there. Make sure everybody knows. So how do you make that happen? We just took this process apart, and we said “Okay, we’re going to change the world out there.” We did that.

I’m still struck that it still comes up, this distinction between evaluation and auditing. I wonder what you think about the two communities, how they are different or similar and what they can learn from one another.

I can tell you the view at the time. I think it was the auditors who cared more about the distinction than the evaluators did. They have their professional standards (including the so called “Yellow Book,” which is the government’s audit standards) and they are very proud of those standards. They feel you have to follow them exactly. Auditors have several products. One is a financial audit. Evaluators would never do that. The second one was an attestation study where the auditors go in and say “We’ve checked out your systems and they’re okay or they’re not okay.” We don’t do that. But they invented two other things.

The first is what they call “non-audit services.” These products are usually the result of requests from agency officials to gather and analyze facts or to provide advice on a management or technical issue. Such requests may be made, for example, because of the technical expertise of the auditors stemming from their body of work on a particular topic or because the agency wants truly independent advice and are confident that they can get that from auditors. Auditors are sometimes willing to provide such services, but when they do they include a statement that the report does not comply with audit standards. I say “sometimes” because if auditors provide advice on how to set up a management system then they will not be allowed to audit that system in the future. Evaluators are a lot more comfortable doing studies like that. However, in an OIG setting they are also restricted from advising on management systems that their auditor colleagues might someday have to audit.

Of far more importance was their second innovation—the performance audit, which really was a true evaluation, and a very practical one for government work. That was a good model for evaluations for government work. What standards are you going to use? I think for a lot of government work that’s very applicable. There are things where you would check performance as opposed to spending against standards to see if they’re either performing or they’re not performing according to whatever standard you’re using. Although if there was no government standard you could say “Well, what do they use in the private sector?” Or you can say “Well, what has it been like over the last 5 years? What’s the going rate? What are the top 80 percent like? What do they expect?” “Yellow Book” describes the principles that auditors follow in conducting their studies. It includes principles and guidelines for performance audits that auditors have now widely adopted. In the Inspectors General community, we evaluators have developed our own standards, which we call the Blue Book, and which was basically written around the performance audit.

But the evaluations we did were much more open than that. To give you an example, we did the very first survey of Medicare beneficiaries in terms of how they were being treated by their Medicare processors, the carriers and contractors who pay their bills. We would take random samples of maybe 500 Medicare beneficiaries and talk about that. “The last time you had any interaction with Medicare, how were you treated? Were you treated respectfully? Did they answer your questions?” And then we would ask them questions about “When you needed to be referred to a specialist, how many times did you have to call them before you could get in the door? How many days did you have to wait?” We’d ask them about “Overall, when you were done dealing with the Medicare carrier or contractor did you feel that your questions were answered?” I don’t know what the standard for that is. We found an 80 percent satisfaction rate, which was very, VERY high. GAO had done the first such beneficiary satisfaction survey for social security, and then after that they turned it over to us. So every year we did a survey of social security beneficiaries. Same idea. There wasn’t a standard there, but once we started doing the studies a standard emerged. Did beneficiaries' satisfaction it go up or did it go down? We had the sense for large businesses that few of them had gotten an 80 percent satisfaction rate. Medicare and social security both did.

Eventually the Social Security Administration started doing its own survey, but to a much larger extent, a much larger sample. Our people had helped them initially. So it’s now 20 years later and I recently met the person who is in charge of that survey at Social Security. Their satisfaction rate is now 85 percent and they were upset when it went down to 83 percent one year. Clearly this is an evaluation that actually is part of the way they live and breathe over there to run their program. But there was no standard before that.

I remember one our office did early on. There were these runaway and homeless youth hotlines. If you were a kid you could call an 800 number and say “Yeah, I’m lost. Send a message to my mother”, or “I don’t know what I’m going to do. Help me out.” We had all of our young people pretend to be homeless youths and call the lines. They tallied up how many times was the line busy; how many times they called before they spoke to somebody. I called it the “Try it out” evaluation. Again, there’s really no standard. There may be now, but in those days there wasn’t a standard.

I’d like to address something the chronology doesn’t get at in terms of what really matters. What shapes your career? First of all, I’ve had these experiences that run several years at a whack, you know, Defense - 4 years maybe, HHS - 15 which gradually changed from being kind of an administrative person to being more heavy-duty into the cross-cutting analysis and policy world, and then the IG world - 16, and then the consultant world, and now I’m back in another area entirely. Every one of those changed what I did. In every one of those I picked up ideas and met people who were very dedicated, doing very good work. I learned from every one of them that there’s all these very different but very legitimate ways to think about what we do. And rather than try to conclude which one is right, I said “hey, they all have a lot of merit to them.” Every one gave me something about how this profession is and what you can do with it.

The other thing is I always worked for giants. My first boss in the Navy, after I did the summer work for DOD, was the Assistant Secretary of the Navy Comptroller that hired me for my permanent job, Chuck Bowsher. My first guy eventually became the Comptroller General of the United States. I knew every secretary in the Department of Health and Human Services from ’73 to 2006. In doing the evaluation work in ASPE as the Director of Planning and Policy Coordination and in the IG’s office, I knew them all. Plus all the Deputy Secretaries. Plus all the Assistant Secretaries, and all the Deputy Assistant Secretaries, and all the really good civil servants. It was the world I inhabited. And it wasn’t because I was like one of them. I just brought the tea and cookies and helped them out in different ways. But I could see what they were doing. And so I made up my mind early that I would watch them and try to figure out what the good ones were like. And I’d take informal notes and began to formulate what a really successful program analyst or policy analyst or evaluator was like. What were their qualities? I figured—may be some are inherent, but if I could imitate them I would. I really did this very systematically. Every one of them who came to Washington, Democrats because it’s in their soul, Republicans because it’s an act of patriotism, they could make more money someplace else. But they are young 35 or 40 year-old Deputy Assistant Secretaries who came because they want to be there. They knew they had just a few years. They said “When I leave, I want it to be better.”

You have to be non-cynical. You have to be open to what people do. And you have to forgive a lot. And people have to forgive a lot of me. That’s true for everybody. To really be successful and to really make a difference, you have to learn your trade. And it’s not just how we evaluate things; it’s how the policy process works. How you deal with it is just as important as how you do it. I learned after a while that you have to know all these things. If you really wanted it to matter you have to learn all these worlds and you basically had to recognize this is not going to happen in a year. This is going to take time. I used to take a subject a year. I said “This year I’m going to become an expert in how Medicare pays physicians.” A year later, I knew all about it. Next year I took another topic. I did 30 years’ of that I said “I’m just going to learn all there is, and I’ll know as much as anybody does”.

I have one other thing too that I thought was important. I studied philosophy in college. My minor was English and then later on I got a degree in mathematics. And I think that those three things were the perfect combination for being an evaluator. You know I was beaten down by my teachers to write well. And the philosophy got me a lot in terms of the logic of the human mind and how you prove things and how you think about things. And the mathematics, math is like an alphabet for me. I do it a lot but not much above really good high school math or first year of college; it never holds me back. No one can show me a mathematical formula and say that I’m not competent to understand it. I can read it like I can read English. With that kind of background and also with the seminary training, to try to be humble, try to be of service, try to be decent, try to treat people right, try to care about everything, maybe all of that added up that I was more successful in this world of policy and evaluation than I might have been as a baseball player or an engineer or something else. Maybe I just lucked into something that was compatible with my upbringing, my early training and each experience up to that. So I ended up as a very lucky person and I remain that way today. It’s like getting paid to play baseball. It’s just really fun.

One of the things that struck me when you were talking about your early experiences is that you seemed to think very carefully about how to build the demand to create use. How has your thinking about use changed over time?

It has been reinforced. I also don’t think there’s many people at all who had the view that I had and some of my colleagues that you really could change the world. You don't do that with just one study. You need a family of studies. My goals were things like wanting to see the percentage of children who are entitled to support payments and getting them go up, even though I’m not in charge of it. But I don’t find very many evaluators who want to think that way. I think they think it’s naïve of me to think that you could really change the world. And they still think too much in my opinion in terms of an evaluation as opposed to a body of work. I think it’s the body of work that gets you in there. What is it going to take? You’ve got to meet people. You’ve got to cooperate with a bunch of other people. You can’t do just one study. You’ve got to stick at it. You’ve got to bring new information to the table.

What do you look forward to in evaluation’s future?

What I look forward to is working with the younger evaluators and trying to help them understand what they can do and how to do it. I’m always worried about the future because I’ve got employees and I want them to do good work and I want them to thrive in their profession and I’ll do anything I can to help them do that, so I’m almost constantly trying to work with the newcomers or older ones who are really interested in becoming good at what they do. I’ll try to find a way to talk to them and to work with them, trying to get people to understand that there really is a connection here between what you do and decisions that are made. And don’t be so cynical. There is a lot of cynicism.

I think for the future that I really want to see the young ones plus the more experienced people who are now the doers of evaluation, the people who actually do it for a living, to really understand how they can make a difference. It was like that office I moved into, that IG office. They systematically did this. The reason their staff was so good was because they decided that their staff was going to be really good. So they set up the training programs. They set up personal performance assessments evaluations which were very strong on “What did you accomplish last year?” rather than simply “What did you do last year?” Remember the days when it was always about “Mission, vision, and value” and all that kind of stuff? During the Clinton years with Vice President Gore, you were required to go these meeting that would wear you down discussing, “What’s our mission? What’s our vision?” We had bi-annual conferences where 120 people would come to a week’s worth of training.

At one of these training sessions, our agency at the time decided we had to carry out this drill. So we had one of those exhausting days where people worked in small groups and put down lists and then someone reported out, and they voted with stickies and all that kind of stuff. And after a whole day the question was “What’s most important to you?” Number one, they wanted to make a difference. Number one! Below that was a lot of white space. At the next level you picked up that people wanted two things at work: they wanted to be supported with what they needed to do their jobs and they wanted an opportunity to develop themselves. They wanted to grow in their profession. Now you drop down one level below that and they wanted decent working conditions. They did not expect to be millionaires, just a decent standard of living. Then below that there was a bunch of other stuff. But those are the big ones. “I want to make a difference,” then “I want to develop if I can, and I want to learn more” and then “I want to have a decent place to work.”

So I remember we said to them after that, “Okay. You got it. That’s what you want, that’s what we’re going to do. Your performance ratings from now on will be what you accomplished. It won’t be how many reports you did, although it will be very hard to accomplish something if you don’t do any reports, right? So from now on when we evaluate you, it’s going to be ‘What did you accomplish this year?’ We want to know that there was a law passed that you had an input to, or regulation was enacted that you had something to do with. The budget was changed. Money was saved and the people who saved it said that one of the things that contributed to it was a report that you did. You come to us with that, you’re going to get a bonus. You come to us and say you’ve issued five reports? No bonus for that. This is what you said you wanted. So we’re going to give it to you.” That lit the place up. And I don’t mean on that day. I can tell you that for the next 4 or 5 years it was like magic.

I remember one year when the Congress passed one of those mammoth spending bills and reconciliation bills, and they needed to save slews of money. We had a book that we put out with the auditors. We call it the red book. Every recommendation that either audit or we made to save money had a one-pager in it in that book. Some of the stuff we had worked on for years. Nursing home reform, I mentioned to you, improved quality of care in a nursing home plus saving money. Prescription drugs, medical equipment, just all kinds of stuff. The Congress took it all. They took almost all of it and they passed this humongous bill and there were these sections in there on stuff we had worked on. Of course everyone was working on these issues, but in some cases we knew we were the first. We were raising nursing home issues when no one was raising nursing home issues. So that was a big year for us. And it was inspired by this idea of we’re taking it seriously. Show us the law. Show us the savings. Show us where they give you credit for contributing to it.

We took this gigantic bill, and we identified the sections that were things our staff had worked. For each one we took the cover page of the law and the cover page of that section of the law, like nursing home reform, that was part of the reconciliation act. There were about 20 such sections in all. And we put them out on tables, and I found lost in the cabinets those old government pens, those little Skilcraft ball-point pens—they were black and they were leaky. We put the boxes out on the tables. And we said “Okay, now any of you who worked on any of the bills”—we had the name on each table, nursing home, medical equipment, child support enforcement, whatever it was—“You come up there and you sign that.” I said “Look, we’ve been working these fields for some time, so we’re having our own signing ceremony. You’re going to sign the cover of that bill. We’re going to take your picture and you get to keep the pen.” Then when I got that done, I went up to Inspector General June Gibbs Brown and I brought them all into her. She signed the cover, and I had my picture taken with her. And then we gave everybody a picture of me with June plus their picture, plus a copy of the bill. And they got to keep the pen. The President has a signing ceremony, why can’t we? It’s the bottom line here and that’s what we’re working for.