Evaluation can be creative – and more effective

By Alice Wilkin, Research Officer, Program Development and Evaluation, Alcohol and Drug Foundation

The best people to evaluate community-based social projects are the community organisers running it, together with the ‘target population group’ they’ve been working with.

So how can these project organisers best engage the community in meaningful evaluations?

Any community-based program evaluation (including the ADF’s Drug Action Team programs), shouldn’t simply be a dry number-crunching exercise – this risks community disengagement from the process and the loss of valuable insights.

Instead, in many cases, creative evaluation can be applied and adapted to suit the community’s capabilities and needs.

Effective community evaluations can use a variety of tailored mediums. Art, photos, sculpture, photographs, storytelling, music and video can all be ways to engage the wider community in sharing their experiences.

PhotoVoice

For example, many community programs have undertaken PhotoVoice and similar art projects to express and evaluate an element of their lived experience within the community. This artwork can then be analysed to produce reports, as well as being exhibited with a wider audience.

This approach can really enhance the community engagement experiences of all stakeholders involved in a project; it can be particularly effective in cross-cultural interventions.

‘Creative evaluation’ generates data that is different from standard evaluations as:

• not everyone can express themselves in a standard evaluation survey format;
• it can reveal perspectives and provide data which is co-written/produced;
• it’s not just about what’s happening, it’s about how and why; and
• the use of images can place control of the interview process with the participant, bringing out issues that are meaningful to them¹

Aboriginal Voice and Vision Photovoice

The following photos and quotes are examples of the PhotoVoice project ‘Aboriginal Voice and Vision: Aboriginal Women’s experience of working in the Victorian health system’² .

The two photos and the related interviews gave an insight into health workers’ coping mechanisms.

Through this creative evaluation, participants openly discussed the barriers and enablers in their work locations and structures. The results enabled an understanding of:

• unsafe working areas;
• why they were unable to conduct preventative health measures in their workplace;
• that the main enabler was their desire to change the health outcomes of Indigenous Australians; and
• all the participants had coping mechanisms to deal with stress caused by their work.

Additionally, the feedback from the participants was that the method of talking about their own photographs was a culturally appropriate and a cathartic experience³.

Other methods can include:

• maps, diagrams and matrices⁴
• photographs and video footage^{5 6}and;

collage and drawings ^{7 8}

Already seeing great examples

While still in the early days of the LDAT program, we’re already seeing great examples of creative evaluation:

• We are making moving photograph videos with audio of a basketball match to promote a Sydney LDAT titled ‘Peer education with at-risk CALD communities’ events – the raw data is also going to their progress reports.
• The ADF is also using narrative storytelling to capture participants’ experiences of attending an AOD presentation by Indigenous former NRL player and boxer, Joe Williams. (This event was hosted by the Gundagai LDAT: Gundagai Community Back On Track.)
• Development of craft activities to evaluate cultural activities.

More creative evaluation by LDATs

One of the Queensland LDATs has a vision of creating positive community connections to Indigenous culture as a protective factor against the risk of AOD misuse in the local Aboriginal community.

LDAT grant funding has allowed this team to commence programs that engage community members in AOD protective factors. The activities — which include cultural camps, historic site visits, scrapbooking, photography lessons and craft sessions — are allowing participants to express themselves creatively, contributing to healthy culture and spirit, which in turn is building healthier communities.

Working with LDATs to harness what works

With help from the ADF, one Queensland LDAT is using creative evaluation and harnessing what already works. The LDAT project participants enjoy engaging in creative activities. And they are interested in their own (and neighbouring communities) cultural heritage. Using these interests as its starting point, the LDAT has harnessed existing strengths to minimise AOD harms.

Working with this LDAT, I put together a simple schedule, a list of broad and adaptable questions, and suggested a number of new techniques to help them evaluate the elements of their project.

This included using the craft sessions as a means to reflect and evaluate the visits to culturally important sites.

For example, after the cultural camp, the group will have a reflective art and craft session to express what they learned from the visit. There will be no tick-box evaluation; instead, the evaluation is itself the art work that the group is already doing!

This important and engaging work then analysed with help from the ADF who are upskilling the LDAT on how to conduct analysis and generate the data that will subsequently inform their six and 12-month progress reports.

By using a thematic analysis of artwork and interviews you can gain greater insight into the lived experiences of your participants while identifying issues in a systematic and reportable manner. This analysis will also provide evidence of program successes and allow the LDAT to build a case for future Drug Action Team program or other funding.

As the ADF’s LDAT Monitoring and Evaluation Officer, I have used visual evaluation in a range of communities and I’m available to support the development of monitoring and evaluation plan to suit every LDAT.

If your team is looking for support with evaluation, please contact me via your respective ADF state or territory contact.