PRINT

September 13, 2017

Evaluation can be creative – and more effective

By Alice Wilkin, Research Officer, Program Development and Evaluation, Alcohol and Drug Foundation

The best people to evaluate community-based social projects are the community organisers running it, together with the ‘target population group’ they’ve been working with.

So how can these project organisers best engage the community in meaningful evaluations?

Any community-based program evaluation (including the ADF’s Drug Action Team programs), shouldn’t simply be a dry number-crunching exercise – this risks community disengagement from the process and the loss of valuable insights.

Instead, in many cases, creative evaluation can be applied and adapted to suit the community’s capabilities and needs.

Effective community evaluations can use a variety of tailored mediums. Art, photos, sculpture, photographs, storytelling, music and video can all be ways to engage the wider community in sharing their experiences.

PhotoVoice

For example, many community programs have undertaken PhotoVoice and similar art projects to express and evaluate an element of their lived experience within the community. This artwork can then be analysed to produce reports, as well as being exhibited with a wider audience.

PhotoVoice example

PhotoVoice is a way to evaluate ‘lived experience’ – it is a way to get to know your community, but also to break down barriers.

Source

Broadly, the aims of these techniques are to engage participants and communities with an evaluation that is respectful to their needs, competencies and skill-set. It’s about creating a process in which they feel comfortable to open-up and really enable them to contribute.

 

This approach can really enhance the community engagement experiences of all stakeholders involved in a project; it can be particularly effective in cross-cultural interventions.

‘Creative evaluation’ generates data that is different from standard evaluations as:

  • not everyone can express themselves in a standard evaluation survey format;
  • it can reveal perspectives and provide data which is co-written/produced;
  • it’s not just about what’s happening, it’s about how and why; and
  • the use of images can place control of the interview process with the participant, bringing out issues that are meaningful to them1

Aboriginal Voice and Vision Photovoice

The following photos and quotes are examples of the PhotoVoice project ‘Aboriginal Voice and Vision: Aboriginal Women’s experience of working in the Victorian health system’2 .

The two photos and the related interviews gave an insight into health workers’ coping mechanisms.

PhotoVoice water

 

“I took [this] photo because water is very healing to me and it shows it’s calmness; also the colour. The colour of the water and the smell of the water the ocean. The magnificent blue it was just crystal clear – it’s just like a real cleansing. I loved that it was a beautiful, beautiful healing for me, water and the colour and the smell.”

Aboriginal health worker

 

“I go here for a smoke.  People go there to talk one-on-one; it’s a nice quiet little place for people.  Because it’s such a friendly place if one or two people are talking everyone is welcome to join in. But sometimes it’s not a public issue that you want to talk about, or you need a bit of a break.  Sometimes I get a bit peopled out.”

Aboriginal health worker

PhotoVoice Safe place

Through this creative evaluation, participants openly discussed the barriers and enablers in their work locations and structures. The results enabled an understanding of:

  • unsafe working areas;
  • why they were unable to conduct preventative health measures in their workplace;
  • that the main enabler was their desire to change the health outcomes of Indigenous Australians; and
  • all the participants had coping mechanisms to deal with stress caused by their work.

Additionally, the feedback from the participants was that the method of talking about their own photographs was a culturally appropriate and a cathartic experience3.

Other methods can include:

  • maps, diagrams and matrices4
  • photographs and video footage5 6and;

collage and drawings 7 8

Already seeing great examples

While still in the early days of the LDAT program, we’re already seeing great examples of creative evaluation:

  • We are making moving photograph videos with audio of a basketball match to promote a Sydney LDAT titled ‘Peer education with at-risk CALD communities’ events – the raw data is also going to their progress reports.
  • The ADF is also using narrative storytelling to capture participants’ experiences of attending an AOD presentation by Indigenous former NRL player and boxer, Joe Williams. (This event was hosted by the Gundagai LDAT: Gundagai Community Back On Track.)
  • Development of craft activities to evaluate cultural activities.

More creative evaluation by LDATs

One of the Queensland LDATs has a vision of creating positive community connections to Indigenous culture as a protective factor against the risk of AOD misuse in the local Aboriginal community.

LDAT grant funding has allowed this team to commence programs that engage community members in AOD protective factors. The activities — which include cultural camps, historic site visits, scrapbooking, photography lessons and craft sessions — are allowing participants to express themselves creatively, contributing to healthy culture and spirit, which in turn is building healthier communities.

Working with LDATs to harness what works

With help from the ADF, one Queensland LDAT is using creative evaluation and harnessing what already works. The LDAT project participants enjoy engaging in creative activities. And they are interested in their own (and neighbouring communities) cultural heritage. Using these interests as its starting point, the LDAT has harnessed existing strengths to minimise AOD harms.

Working with this LDAT, I put together a simple schedule, a list of broad and adaptable questions, and suggested a number of new techniques to help them evaluate the elements of their project.

This included using the craft sessions as a means to reflect and evaluate the visits to culturally important sites.

For example, after the cultural camp, the group will have a reflective art and craft session to express what they learned from the visit. There will be no tick-box evaluation; instead, the evaluation is itself the art work that the group is already doing!

This important and engaging work then analysed with help from the ADF who are upskilling the LDAT on how to conduct analysis and generate the data that will subsequently inform their six and 12-month progress reports.

By using a thematic analysis of artwork and interviews you can gain greater insight into the lived experiences of your participants while identifying issues in a systematic and reportable manner. This analysis will also provide evidence of program successes and allow the LDAT to build a case for future Drug Action Team program or other funding.

As the ADF’s LDAT Monitoring and Evaluation Officer, I have used visual evaluation in a range of communities and I’m available to support the development of monitoring and evaluation plan to suit every LDAT.

If your team is looking for support with evaluation, please contact me via your respective ADF state or territory contact.

References
  1. Frith, H., Riley, S., Archer, L. & Gleeson, K. (2005). Imag(in)ing visual methodologies [Editorial], Qualitative Research in Psychology2(3), 187-198. Retrieved from: http://dx.doi.org/10.1191/1478088705qp037ed
  2. Wilkin A (2008) Aboriginal Voice and Vision: Aboriginal women’s experience of working in the Victorian health system. Unpublished honours thesis, School of Public Health, La Trobe University, Melbourne, Australia.
  3. Wilkin A, Liamputtong P (2011) Indigenous Voice and Vision: The Experiences of Indigenous women working in the Victorian health system through photovoice. Journal of Indigenous Issues, 13(4), 45-68
  4. Copeland, A.J & Agosto, D.E. (2012) Diagrams and Relational Maps. International Journal of Qualitative Methods
  5. Radley, A., Hodgetts, D. & Cullen, A. (2005). Visualizing homelessness: A study in photography and estrangement. Journal of Community & Applied Social Psychology, 15, 273–295. doi: 10.1002/casp.825
  6. Ross, N.J., Renold, E., Holland, S. & Hillman, A. (2009). Moving stories: using mobile methods to explore the everyday lives of young people in public care. Qualitative Research, 9(5) 605-623. doi: 10.1177/1468794109343629
  7. Bagnoli, A. (2004). Researching identities with multi-method autobiographies. Sociological Research Online, 9(2). Retrieved from: http://www.socresonline.org.uk/9/2/bagnoli.html
  8. Guillemin, M. (2004). Understanding Illness: Using Drawings as a Research Method. Qual Health Res, 14(2), 272-289. doi: 10.1177/1049732303260445

Further reading