September 13, 2017
By Alice Wilkin, Research Officer, Program Development and Evaluation, Alcohol and Drug Foundation
The best people to evaluate community-based social projects are the community organisers running it, together with the ‘target population group’ they’ve been working with.
So how can these project organisers best engage the community in meaningful evaluations?
Any community-based program evaluation (including the ADF’s Drug Action Team programs), shouldn’t simply be a dry number-crunching exercise – this risks community disengagement from the process and the loss of valuable insights.
Instead, in many cases, creative evaluation can be applied and adapted to suit the community’s capabilities and needs.
Effective community evaluations can use a variety of tailored mediums. Art, photos, sculpture, photographs, storytelling, music and video can all be ways to engage the wider community in sharing their experiences.
For example, many community programs have undertaken PhotoVoice and similar art projects to express and evaluate an element of their lived experience within the community. This artwork can then be analysed to produce reports, as well as being exhibited with a wider audience.
PhotoVoice is a way to evaluate ‘lived experience’ – it is a way to get to know your community, but also to break down barriers.
Broadly, the aims of these techniques are to engage participants and communities with an evaluation that is respectful to their needs, competencies and skill-set. It’s about creating a process in which they feel comfortable to open-up and really enable them to contribute.
This approach can really enhance the community engagement experiences of all stakeholders involved in a project; it can be particularly effective in cross-cultural interventions.
‘Creative evaluation’ generates data that is different from standard evaluations as:
The following photos and quotes are examples of the PhotoVoice project ‘Aboriginal Voice and Vision: Aboriginal Women’s experience of working in the Victorian health system’2 .
The two photos and the related interviews gave an insight into health workers’ coping mechanisms.
“I took [this] photo because water is very healing to me and it shows it’s calmness; also the colour. The colour of the water and the smell of the water the ocean. The magnificent blue it was just crystal clear – it’s just like a real cleansing. I loved that it was a beautiful, beautiful healing for me, water and the colour and the smell.”
Aboriginal health worker
“I go here for a smoke. People go there to talk one-on-one; it’s a nice quiet little place for people. Because it’s such a friendly place if one or two people are talking everyone is welcome to join in. But sometimes it’s not a public issue that you want to talk about, or you need a bit of a break. Sometimes I get a bit peopled out.”
Aboriginal health worker
Through this creative evaluation, participants openly discussed the barriers and enablers in their work locations and structures. The results enabled an understanding of:
Additionally, the feedback from the participants was that the method of talking about their own photographs was a culturally appropriate and a cathartic experience3.
Other methods can include:
collage and drawings 7 8
While still in the early days of the LDAT program, we’re already seeing great examples of creative evaluation:
One of the Queensland LDATs has a vision of creating positive community connections to Indigenous culture as a protective factor against the risk of AOD misuse in the local Aboriginal community.
LDAT grant funding has allowed this team to commence programs that engage community members in AOD protective factors. The activities — which include cultural camps, historic site visits, scrapbooking, photography lessons and craft sessions — are allowing participants to express themselves creatively, contributing to healthy culture and spirit, which in turn is building healthier communities.
With help from the ADF, one Queensland LDAT is using creative evaluation and harnessing what already works. The LDAT project participants enjoy engaging in creative activities. And they are interested in their own (and neighbouring communities) cultural heritage. Using these interests as its starting point, the LDAT has harnessed existing strengths to minimise AOD harms.
Working with this LDAT, I put together a simple schedule, a list of broad and adaptable questions, and suggested a number of new techniques to help them evaluate the elements of their project.
This included using the craft sessions as a means to reflect and evaluate the visits to culturally important sites.
For example, after the cultural camp, the group will have a reflective art and craft session to express what they learned from the visit. There will be no tick-box evaluation; instead, the evaluation is itself the art work that the group is already doing!
This important and engaging work then analysed with help from the ADF who are upskilling the LDAT on how to conduct analysis and generate the data that will subsequently inform their six and 12-month progress reports.
By using a thematic analysis of artwork and interviews you can gain greater insight into the lived experiences of your participants while identifying issues in a systematic and reportable manner. This analysis will also provide evidence of program successes and allow the LDAT to build a case for future Drug Action Team program or other funding.
As the ADF’s LDAT Monitoring and Evaluation Officer, I have used visual evaluation in a range of communities and I’m available to support the development of monitoring and evaluation plan to suit every LDAT.
If your team is looking for support with evaluation, please contact me via your respective ADF state or territory contact.