Knowledge and Innovation is a social research institute, whose mission is to promote, plan and carry out applied research projects in different areas connected with the processes of change that are affecting contemporary societies. K&I is the leading partner responsible for the evaluation component of the project. The evaluation exercise adopts a participatory method aiming to involve CSI teams at every stage of the evaluation process. Luciano d’Andrea, Team Leader at K&I, has been engaged for over forty years in the field of sociological research. Over the last decade, his research interests have mainly focused on science-society relationships, understood as a privileged observation field to analyse the pervasive and profound transformation processes affecting contemporary societies as a whole.
In Step Change, the evaluation exercise adopts a Developmental Evaluation perspective, fostering equal relationships among all those involved. Could you tell us more about the developmental approach that you are using?
Traditionally, evaluation is intended as an assessment aimed at verifying whether and how a given project is implemented based on a given plan. In this view, implementing a project is understood as a linear process and evaluation simply as a judgmental exercise made by external evaluators. Developmental Evaluation, like other similar approaches defined in the last decades, is based on a more realistic assumption: implementing a project does not mean executing a plan, since there is always a gap, often remarkable, between what was planned and what occurs. Consequently, the success of a project cannot be measured with the level of stickiness of the implementation process to the original plan. The role of evaluation changes accordingly. Especially when complex social projects are concerned (and Citizen Science projects are undoubtedly complex), evaluation should be a service provided to the project team to help it timely identify, manage, and, as far as possible, anticipate risks, opportunities, and unexpected events, so as to find effective solutions, and learn from experience. This implies regular and close contacts between the project team and the evaluators throughout the project, the adoption of mechanisms and tools allowing the participation of stakeholders, and a specific focus on the processes activated by the project rather than only on its outputs. All in all, Developmental Evaluation helps evaluators to adopt a proactive and engaged point of view, not limiting themselves to simply recording the gap between a set of established objectives and the actual project outcomes.
You have led the setting up of local evaluation units composed of both researchers and citizen scientists. Could you tell us more about challenges and opportunities related to this?
Putting together researchers, citizen scientists, and stakeholders in the local evaluation units and making them work together with the professional evaluators in charge of the entire evaluation process, the so-called “Central Evaluation Unit”, is itself a challenge, under different angles. It is necessary to find common terminology and language, coordinate the work effectively, and develop a shared understanding of problems and solutions. However, the establishment of local evaluation units offers the opportunity to ground evaluation on first-hand information produced by those who are engaged in implementing the Citizen Science Initiative (CSI), albeit with different roles. This allows to build a comprehensive and reliable picture of the CSIs, in which also CSIs’ internal dynamics and intangible aspects like, for example, the participants’ sense of ownership, motivations, and satisfaction, become analysable.
Could you explain how participatory evaluation is increasing and will increase CSIs’ self-reflexivity?
When it is based on participatory mechanisms, evaluation becomes an organisational function, with its methods and timeline, explicitly devoted to favour self-reflection. It urges the CSI teams to stop for a moment and reflect on their projects from a more detached and objective observation point. The participation of evaluators not directly involved with the project as well as of stakeholders or beneficiaries, helps teams to break away from immediate implementation problems and recognise other perspectives from which their project can be observed. For these reasons, if organising a conventional evaluation exercise is indeed more difficult, participatory evaluation is more productive since it offers the possibility to go in depth into critical points and opportunities, favouring rapid personal and institutional learning processes, otherwise impossible to reach.
The evaluation of the CSIs has been designed and is being implemented as a Citizen Social Science Initiative, which can be considered as an additional and cross cutting experiment within the project. Could you explain?
Evaluation, beyond being a service, is also a powerful research method to collect fresh data, not only on the CSIs but, more broadly, on how Citizen Science itself works in general. Step Change offers the incredible opportunity to observe, through evaluation, five different CSIs at the same time, and this opportunity could not be missed. This is the reason why we decided to consider evaluation as another CSI, involving both professional experts and citizens and based on specific theoretical assumptions, research questions, and objectives. Approaching evaluation as a citizen science research project is also important to provide the knowledge basis to develop, in the last phases of Step Change, a model and a hands-on tool (the Citizen Science Navigator) to use citizen science as a means to help research organisations and scientists to better interact with society while doing research.
