A Conversation With David Roberts,AES Fellow,2025

David – its really terrific to talk to you. Although I have known you for many years can I get you to introduce yourself in terms of what you are doing now and how long you have been in evaluation?

I guess in terms of how long I’ve been in evaluation it’s a bit complicated as I (like many others) have moved in and out of the field over the years. I started back in about 1978 when I was working in Social Security. They had a couple of little projects they wanted doing and I volunteered. So that is where I started in evaluation.

One was an organisation and methods study, but the other was a process evaluation. And I didn’t think about that as evaluation at the time. Shortly afterwards, I wrote an essay for my anthropology degree describing and analysing the organisation and operations of the Department of Social Security (DSS). After that I did some community development work as a volunteer. I built up some skills; and then got a job doing community development work in the Riverland region in South Australia. That was fairly intense. After 2 years, I hadn’t had a holiday. I took 6 weeks off but unfortunately got sick the second day of my break and didn’t get better until the last week of leave. So, I decided to change jobs. In 1992 I got offered a job in Canberra doing evaluation work for what was then the Department of Employment, Education and Training. I stayed there for about 3 years before I changed departments. I moved around for a few years in policy and program management roles, but I kept on gravitating back into evaluation areas. For example, I did a couple of major policy reviews, worked on counter-terrorism evaluation (interesting!) and then to Centrelink in an evaluation role.

It sounds like you have been in and out of evaluation – like a lot of us!

Yes, but I think I’ve been full time in evaluation since about 2002.

I believe you then started your own company?

It was in 2010 and it is still going. I am not really looking for work at the moment, but people contact me and ask me to do things. I’ve been getting really interesting work and in particular the work I enjoy is peer reviews. I have always enjoyed it and feel that not enough is done in that area.

Well – tell me what got you into evaluation, was it a choice or something that you fell into?!

Well, I’ve always liked research, especially people and systems and structures. My first degree was in anthropology. In fact, I started a master’s in anthropology, but that was when my first daughter was born which really put a hole in my efforts. I did a first draft of a thesis, but that needs a lot more work. Those questions about how things work have always been interesting to me.

Yes, it’s that curiosity thing, isn’t it? Why? Why is that so? What does that mean?

Yes and the other part of it has always been about making things better.

The second was just about what brought you into the field of evaluation

That is fairly basic – I got offered a job. But I’d been sort of doing projects for ages. And it aligned pretty much with my interests in qualitative work, which aligned with the community development work. I’ve also got basic qualifications in maths and statistics.

When you look back in your career, what do you think have been the major challenges for you?

As an evaluator, I encourage people to go beyond the conventional ideas about evaluation, and challenge their ideas about how best to do an evaluation and explore methods that might tell commissioners what they want to know. What I try and do is start the process by asking, “What do you need to know? And what are you going to do with it?” Then I explore the different ways we can go about getting the information they need and talk about some of the questions that they might ask. Of course, not everybody’s interested. In fact, some evaluations, especially internal evaluations are not done because they want answers. They’re doing it because they have to do an evaluation. So that was always a challenge.

Although it’s the challenges that are interesting

The other part, which our Indigenous brothers and sisters keep coming back to, is about who owns the data. And for me, it’s always been the case that I’m sharing stories or information that come from the people we interview, and they need to know what’s happened to it.

Of course, it is their data, isn’t it?

Yes. It’s their data. As far as I’m concerned, it doesn’t belong to the Department of Education or whatever. It belongs to them. And so there needs to be a process of showing them what’s happened in the program, as well as what they did and being faithful to them and demonstrating to them that you’ve been faithful to the data received.

So it’s a matter of trust then as well, isn’t it?

Yes it is. I often frame it for commissioners by saying that we need to do data checking. We need to validate the data with the people to make sure we’ve got it right. I always try and do that. But quite often governments are not interested.

So are you generally now working for government?

Mostly. I’ve done some work with not-for-profits, as well. I did some work for ACOSS, a number of not-for-profits and for the Ethnic Community Council in Victoria, people like that. They usually don’t have as much money. And they often don’t understand how much work’s involved in getting the answers they want. You have to tell them in a respectful way what they can get for the money they have to spend.

Well, that’s often the way with evaluation, trying to price something they do not fully understand

Yes we’ve all gone through that. That is very frustrating

Not really knowing what they’ve done, or what they think about it. This is one of the reasons I like doing peer review. I'm trying to encourage peer reviews so that at least evaluators get feedback about the quality of their work.

So can you comment on how you think the field of evaluation changed over the years that you’ve been involved?

I wasn’t linked into evaluation as a profession until 1992. But when I first started every agency in the Commonwealth Government, as well as most States, were requiring programs to be evaluated on a regular basis and publishing the evaluation reports. But then it went through a hiatus where basically people weren’t doing evaluations or only doing minimal evaluations. Now we are seeing an upswing of sorts. But we’re still not getting many evaluations published. So they’re not in the public domain. And to be frank, some part of that is about us as evaluators not writing well.

Well, you are right – not publishing, and that is very true

Most of the work I have done is not published because it’s all confidential and not in the public domain. Clearly, it’s up to the Commissioner to agree to publication. But that said, I feel that evaluations with people should, by and large, be in the public domain so people can see what’s been said about programs. That’s a battle that we still have to win. It’s easier with the not-for-profit sector many of whom want the results in public domain so they can get funding for their projects.

So if you look at evaluation as it is now, what do you consider would be, say, three or four breakthrough ideas over the last 10 or 15 years that have really helped to define and influence modern evaluation? So really major, major things that have happened

During the recent Canberra conference lots of people were talking about how you engage community in the process of evaluation, how you help community – not just with First Nations peoples, but also in other contexts. How do you engage people and help and encourage them to do evaluative thinking and support change and acting as a force for good.

Another issue is that we still tend to portray a simplistic picture of a program. “The program runs like this, has these effects on 90% of people and 60% of people think this or that.”

I think there’s a fundamental problem with programs. Programs are developed, by and large, because somebody sees there’s a problem, and they feel a need to fix it. Take for example a problem like homeless kids. They develop a program to fix the problem of “homelessness.”

However, a program is inherently limited. First the problem is defined “children are living without homes, they need shelter and supervision.” Usually such definitions assume that the individuals can be characterised fairly simply. It is rare for such definitions to recognise there are a wide range of complicated and diverse individuals who share the “problem.” In one of my earliest evaluations I talked to some homeless Indigenous people who were either too young for Job Seeker/Youth Allowance, or couldn’t meet the requirements. Some of those I talked to refused to use the shelters. They survived by crime.

So the design of programs tends to treat individuals as elements to be acted on rather than as agents.

A second limitation is that there is usually only a limited amount of money so the program has to be focused. And criteria are developed about what can be funded and who can receive the funds. Some people in need are almost always excluded.

Some programs are much more flexible, but the basic process is about fixing a problem, not about helping individuals. We need to encourage funders to think beyond typical programs to individualised services that respond to a person’s lived reality.

It may also be working for some and not working for others

And maybe as evaluators we can say this program does work for 90% of people. But what about the other 10%? What’s happening with them? And how well does it work for all of that 90%?

Another thing is that I’m not a fan of the assumptions underlying randomised controlled trials. This applies to surveys as well. If you ask why an evaluator is using a random sample, they’ll tell you that it's about controlling for variables.

However, then they go and differentiate between men and women, differentiate between income groups and education standards. So, okay, you’ve identified these variables that you know affect the result and can’t be fixed by randomisation. So why do you think randomisation is going to work for other variables? Look, in most cases the uncontrolled variables aren’t important, and I think randomised control trials are useful in many ways: but I think people overstate how useful they are.

So what do you think are the main skills or competencies that evaluators need to have or develop to keep pace with emerging trends in evaluation as well as the emerging trends in the world?

What’s probably most important is that evaluators recognise what skills they have; and what skills they don’t have; and look to work with people, and projects that utilise their skills; and work with people who’ve got complementary skills. You’ve got to have an inquiring mind. And I think you’ve got to be able to listen and ask questions that help people. I’m talking now about questions that help clients clarify what they want.

We as evaluators also need to go beyond the superficial. At the moment we have people saying, “We’re going to design a survey to get as close as possible to the true value.” And I’m saying, what’s this true value you’re talking about? We need to recognise that people are complicated individuals: the same person can say or do one thing in one situation and then do or say something completely different in another situation. At the very least, we should be reporting the range of an individual’s response.

There is an even greater distinction between what people “say” and what they “do.” A very crude example is when a person hobbles into an interview room and when asked reports, “I’m fine.” We need to go beyond the presentation and read the situation.

People either don’t want to know, or it's not appropriate, or you don’t want the answer

Unfortunately, many interviews and survey designs have a list of questions and they just go through them mechanically. They think that gives them good data. It doesn’t necessarily. One of the things I learned, from social workers mainly, but also later on in cognitive psychology, was that when people generalise or use a metaphor you need to ask them more, to be more specific.

Also, when they’re talking about other people, you can ask, “Can you give me an example of that?”

You actually need to be able to probe. And so you need that flexibility in the way you do things. The same thing is true for surveys, except you can’t do it the same way.

No, you can’t. That's a good comment

Because they might say, no, I’m sorry, that's really important. So it is being very upfront and saying, what is it you're looking at? What are your objectives? What do you want to get out of this? And where do you want them?

Sometimes it’s about incremental change, sometimes it’s not. When I was in Centrelink, I had two instances in which the program area was saying, aren’t we wonderful? We’ve done all this wonderful stuff. And I was saying, “‘Well, no!’ The experiences I’m hearing about don’t reflect what you’re talking about: and my own background working in Social Security tells me that what you’re doing is problematic.”

For one of the evaluations, I ran a big workshop with the program area and I told them, “This is what the data is saying. Let’s validate that data and what it means. And then work out where we go from there? What sorts of recommendations are useful to you to make that happen?”

Anyway, the deputy CEO was there, and she listened to what we were saying. Everybody else was saying, “No, you’re wrong.” At the end of the workshop, the Deputy CEO said, “All right, as of Monday, 2 weeks from now, the following things are going to happen.” And so that changed.

And the other evaluation was of a big new IT system that they spent millions of dollars developing. They wanted us to evaluate it. We’d only just started and alarm bells were ringing for me already. So I talked to my boss about it, said, “Look, this doesn’t seem to be working and they are looking for us to give them a rubber stamp. Maybe we should talk to them a bit more about it.” He said, let me get back to you. Anyway, we put the evaluation on hold for a week, and then there was an announcement that the Deputy CEO had canned the IT program. I don’t know. I suspect she was a very canny woman.

I suspect the other things were going on

Yeah, I suspect she knew that there were issues. Maybe what we did was enough to give her the support to go ahead and do what she was going to do anyway.

Also I am not precious about the recommendation I make. Essentially, my view is that if you want to bring about change, you can only change people as far as they’re able to go, and you don’t necessarily know what the constraints are? It’s important. That’s why I like to develop the recommendations with the decision-makers.

And it’s the only way to do it. Otherwise, they’ll say, oh, no, that that wasn’t what we want. So, you know, you really need to go through a process

A lot of that came from Patton!

If you consider where evaluation was when you started your career, where it is now, what do you see as the key issues that evaluators ought to be thinking about as well as seeking to resolve in the next decade? And I guess we have to also consider whether we should be seeking to resolve things?

I think we sometimes give ourselves too much credit. I think we think too much of ourselves sometimes.

Oh, that’s a good one. Yeah, that’s a good point

It comes down to how do you define evaluation? And if you use the definition of determining the merit, worth, and value, you are making a judgement. The wording of that definition places the evaluator in a position of privilege as somebody who’s got more understanding than other people. And I think that if we’re going to make change, we need to change that mode of thinking. We need to see ourselves as change agents, rather than as adjudicators. …. And that’s what I want.

No, that’s a good one. So what, going back to career, what for you have been the highlights of your career, do you reckon?

I really enjoy doing the work I’ve been doing with National Emergency Management, and during bushfire situations. Because it brings together some thoughts of mine. And because in disaster the focus is on community and local empowerment and those sorts of things, so it resonates with me. But there’s also a disconnect between that and the way it is funded. They have funding programs, but many of the funding programs are not community based. They don’t say, “Community X needs $Y and the community should work out how to meet the needs.” Instead most of them focus on specific “problems” in the abstract, much as I was talking about earlier, rather than the needs as experienced at the community level. Going back to the Glasser review I talked about earlier on, it doesn’t talk about how you build community.

Are there other highlights that you want to mention?

I really enjoyed working on my Master’s thesis, explaining the ways people develop answers to questions in surveys and interviews and how that is often different from what people do in real life. We know that most respondents “satisfice,” go with an answer that seems to fit. My research showed that most people, though by no means everybody, respond very quickly to a question. Answers are rarely considered, they are most often almost automatic responses to the presenting situation.

We don’t know enough about the underlying bases of behaviour. Most researchers and evaluators have implicit models for how behaviours and words are generated and we use those models in questions and interviews. However, the cognitive sciences shows that it is much more complicated than many of the models. One thing the science tells us that each individual’s own behaviour is quite different at different times in what appear to be the same circumstances. Sure individuals have “preferred behaviours” but they often go outside the pattern and do something else. Asking a question in a research interview or survey, creates a context and that context generates an automatic response, an answer specific to that question in that context. Schema theory helps to explain why and how those responses are generated and how close such responses might be to the individual’s behaviour.

I hope that my work will challenge people to think differently about their own methods and practice and maybe do further work to find out more about the factors that produce the responses we get in interviews and survey questions.

Going back to evaluation practice, I suspect that the biggest impacts of my work were at a small community level, but I’m probably not as aware of them as I’d like to be.

One of the things I did was a review of small business access to the legal system. And that was complicated because it was a Commonwealth review. But most small business disputes are dealt with at state level. One of the things we found very clearly was that most small businesses didn’t want to go to mediation. In contrast the legal people said, “We need to mediate these disputes.”

Very few of the small businesses, something like 10%, are interested in relationships and relationship building. They’re the ones that grow. The rest aren’t interested in that or in mediation. They just want to get on with their thing and do it quickly. If they have a fight with somebody, they want somebody to say, “This is what is going to happen.” So they actually want somebody to make a judgement for them. Our recommendations around that contributed to the formation of the Federal Magistrates Court. That was a fairly big systemic change.

The food regulation review was another interesting project. We did a review of bilateral relationships with New Zealand on food and drugs.

One other evaluation that I thought was useful, was an evaluation of Centrelink’s policies on homelessness. Originally, I was really struggling to get any clear questions from the people who were managing it. They were just ticking the box that said evaluation. But the management of the program got changed over to social work team. So I had another meeting to talk about what we were doing, and all of a sudden they were clear about what they wanted to know. We made 12 recommendations all of which were adopted and, from what I’ve heard, it went well.

That’s incredibly positive. Let’s move on to the AES. You were president… Oh, first of all, how long have you been a member of the AES? Can you remember?

Well, I first joined in 92. But there was a period of 5 years where I was doing other things and work wasn’t going to pay the fees. I started getting involved again in about 98 in the local Canberra committee. I was on the Board then and I went to my first AES Conference in Darwin as the ACT rep.

I pulled back a bit after that though I did work on things like the Awards. Then in 2012 Anne Markiewicz asked me if I would nominate as President? So when Alan finished up, I nominated. Then we found out there were all sorts of issues about our legal status because we were incorporated in the ACT but we had just moved into the University of Melbourne. We were also having issues about the bank account in New Zealand. So that’s when we decided we would change to a company limited by guaranteed structure. I lead quite a lot of that. But we also had issues with CEOs. I think we had three CEOs before I persuaded Bill to take on the role.

Happily the changes we made gave the AES a sustainable base for the future with a new Constitution and a Preamble about inclusiveness and the importance of having First Nations people on the Board.

How relevant has the Society been for you?

Central to me, working on your own its great to be talking to others. As a person in my own business, it is great and really important to be exposed to new ideas. To have people you can just talk to about stuff outside of your meeting workplace. And it also helped me running workshops, which gave me an income stream, as well.

So thinking now of the society, how can it best position itself to be still relevant in the future. What are your thoughts on that? I mean, you might say it’s fine as it is going the right direction

I know someone who was a member of the Research Society but he decided to join the AES as well. He is critical of us for a number of reasons. He feels that we are a bit “stick in the mud” in terms of approaches to evaluation. I don’t know why, but if you talk to market researchers about what they do, they’re much more innovative than we are. I feel that they are good at selling what they do because they’re doing different and new things. Companies buy, and so do government agencies.

I think the first thing we need to do is find out why we are so conservative in our approach. And then I think we need to be having conversations with significant people about what evaluation can and can’t do. So, for example, I’ve managed to talk to a couple of Secretaries when I was president, but not to the board of secretaries, which in Canberra are the secretaries of departments. I don’t think anybody’s ever talked to them, although we have talked quite a lot to the current Minister for Finance, which I think is fantastic, but we’re not getting much traction.

We do have a good relationship with the Centre for Evaluation?

Yes, that’s important. But the AES needs to be talking beyond those people to ministers. And I think what we need to do is find issues where people are interested in finding information. You know, politicians talk a lot about bang for the buck. And I think if we were saying, okay, let’s talk about what you mean, can we have a conversation? Maybe we start with the back benches or people like David Pocock.

That sounds good. Those are all of my formal questions, actually. Is there anything that I have missed out that you would like to contribute?

I need to know more about what the fellows are doing. I’m happy to contribute in whichever way I can.