September 13, 2017
Evaluation can be creative – and more effective
The best people to evaluate community-based social projects are the community organisers running it, together with the ‘target population group’ they’ve been working with.
So how can these project organisers best engage the community in meaningful evaluations?
Any community-based program evaluation including the ADF’s Local Drug Action Team program, shouldn’t simply be a dry number-crunching exercise - this risks community disengagement from the process and the loss of valuable insights.
Instead, in many cases, creative evaluation can be applied and adapted to suit the community’s capabilities and needs.
Effective community evaluations can use a variety of tailored mediums. Art, photos, sculpture, photographs, storytelling, music and video can all be ways to engage the wider community in sharing their experiences.
For example, many community programs have undertaken PhotoVoice and similar art projects to express and evaluate an element of their lived experience within the community. This artwork can then be analysed to produce reports, as well as being exhibited with a wider audience.
PhotoVoice is a way to evaluate ‘lived experience’ – it is a way to get to know your community, but also to break down barriers
Broadly, the aims of these techniques are to engage participants and communities with an evaluation that is respectful to their needs, competencies and skill-set. It’s about creating a process in which they feel comfortable to open-up and really enable them to contribute.
This approach can really enhance the community engagement experiences of all stakeholders involved in a project; it can be particularly effective in cross-cultural interventions.
‘Creative evaluation’ generates data that is different from standard evaluations as:
- not everyone can express themselves in a standard evaluation survey format
- it can reveal perspectives and provide data which is co-written/produced
- it’s not just about what’s happening, it’s about how and why
- the use of images can place control of the interview process with the participant, bringing out issues that are meaningful to them.1
Aboriginal Voice and Vision Photovoice
The following photos and quotes are examples of the PhotoVoice project Aboriginal Voice and Vision: Aboriginal Women’s experience of working in the Victorian health system.2
The two photos and the related interviews gave an insight into health workers’ coping mechanisms.
“I took [this] photo because water is very healing to me and it shows it’s calmness; also the colour. The colour of the water and the smell of the water the ocean. The magnificent blue it was just crystal clear – it’s just like a real cleansing. I loved that it was a beautiful, beautiful healing for me, water and the colour and the smell.”
Aboriginal health worker
“I go here for a smoke. People go there to talk one-on-one; it's a nice quiet little place for people. Because it's such a friendly place if one or two people are talking everyone is welcome to join in. But sometimes it's not a public issue that you want to talk about, or you need a bit of a break. Sometimes I get a bit peopled out.”
Aboriginal health worker
Through this creative evaluation, participants openly discussed the barriers and enablers in their work locations and structures. The results enabled an understanding of:
- unsafe working areas;
- why they were unable to conduct preventative health measures in their workplace;
- that the main enabler was their desire to change the health outcomes of Indigenous Australians; and
- all the participants had coping mechanisms to deal with stress caused by their work.
Additionally, the feedback from the participants was that the method of talking about their own photographs was a culturally appropriate and a cathartic experience3.
Other methods can include:
- maps, diagrams and matrices4
- photographs and video footage5,6
- collage and drawings.7,8
While still in the early days of the LDAT program, we’re already seeing great examples of creative evaluation:
- photograph videos with audio of a basketball match to promote a Sydney LDAT titled 'Peer education with at-risk CALD communities' events - the raw data is also going to their progress reports.
- narrative storytelling to capture participants’ experiences of attending an AOD presentation by Indigenous former NRL player and boxer, Joe Williams. (This event was hosted by the Gundagai LDAT: Gundagai Community Back On Track.)
- development of craft activities to evaluate cultural activities.
More creative evaluation by LDATs
One of the Queensland LDATs has a vision of creating positive community connections to Indigenous culture as a protective factor against the risk of AOD misuse in the local Aboriginal community.
LDAT grant funding has allowed this team to commence programs that engage community members in AOD protective factors. The activities — which include cultural camps, historic site visits, scrapbooking, photography lessons and craft sessions — are allowing participants to express themselves creatively, contributing to healthy culture and spirit, which in turn is building healthier communities.
Working with LDATs to harness what works
With help from the ADF, one Queensland LDAT is using creative evaluation and harnessing what already works. The LDAT project participants enjoy engaging in creative activities. And they are interested in their own (and neighbouring communities) cultural heritage. Using these interests as its starting point, the LDAT has harnessed existing strengths to minimise AOD harms.
Working with this LDAT, I put together a simple schedule, a list of broad and adaptable questions, and suggested a number of new techniques to help them evaluate the elements of their project.
This included using the craft sessions as a means to reflect and evaluate the visits to culturally important sites.
For example, after the cultural camp, the group will have a reflective art and craft session to express what they learned from the visit. There will be no tick-box evaluation; instead, the evaluation is itself the art work that the group is already doing!
This important and engaging work then analysed with help from the ADF who are upskilling the LDAT on how to conduct analysis and generate the data that will subsequently inform their six and 12-month progress reports.
By using a thematic analysis of artwork and interviews you can gain greater insight into the lived experiences of your participants while identifying issues in a systematic and reportable manner. This analysis will also provide evidence of program successes and allow the LDAT to build a case for future Drug Action Team program or other funding.
- Frith, H., Riley, S., Archer, L. & Gleeson, K. (2005). Imag(in)ing visual methodologies [Editorial], Qualitative Research in Psychology, 2(3), 187-198. Retrieved from: http://dx.doi.org/10.1191/1478088705qp037ed
- Wilkin A (2008) Aboriginal Voice and Vision: Aboriginal women’s experience of working in the Victorian health system. Unpublished honours thesis, School of Public Health, La Trobe University, Melbourne, Australia.
- Wilkin A, Liamputtong P (2011) Indigenous Voice and Vision: The Experiences of Indigenous women working in the Victorian health system through photovoice. Journal of Indigenous Issues, 13(4), 45-68
- Copeland, A.J & Agosto, D.E. (2012) Diagrams and Relational Maps. International Journal of Qualitative Methods
- Radley, A., Hodgetts, D. & Cullen, A. (2005). Visualizing homelessness: A study in photography and estrangement. Journal of Community & Applied Social Psychology, 15, 273–295. doi: 10.1002/casp.825
- Ross, N.J., Renold, E., Holland, S. & Hillman, A. (2009). Moving stories: using mobile methods to explore the everyday lives of young people in public care. Qualitative Research, 9(5) 605-623. doi: 10.1177/1468794109343629
- Bagnoli, A. (2004). Researching identities with multi-method autobiographies. Sociological Research Online, 9(2). Retrieved from: http://www.socresonline.org.uk/9/2/bagnoli.html
- Guillemin, M. (2004). Understanding Illness: Using Drawings as a Research Method. Qual Health Res, 14(2), 272-289. doi: 10.1177/1049732303260445
- Wilkin A, Liamputtong P (2010) The Photovoice Method: Researching the experiences of Aboriginal health workers through photographs. The Australian Journal of Primary Health, 16, 231-239.
- Rouse, L. (2013). A change of view: Using visual methods to explore experience in qualitative research. JEPS Bulletin. Available from http://blog.efpsa.org/2013/05/15/a-change-of-view-using-visual-methods-to-explore-experience-in-qualitative-research/