Surveys may be one of the first things we consider when we think about evaluation, but they aren’t always as easy to design as they might seem. Below are some examples of the things that you need to consider when compiling a survey.
A well-written survey carefully considers:
There are a number of important considerations when writing survey questions, as well as some common traps when it comes to wording your questions clearly. We go through these below, and include a relevant example question where appropriate.
Often, we use pre-and-post survey to get feedback on a program from our target audience. It’s easy to confuse a participant’s enjoyment of a program or activity with the program or activity meeting its objectives.
For example, the Victorian Police, as part of the evaluation of their SSMART ASSK program with young people, stopped using beer goggles in part because student feedback indicated many found them to be “one of the best things about the program” with an overwhelming emphasis on how “fun” they were. This supported previous concerns that the “fun” nature of the activity was undermining the knowledge it was intended to provide, and that it risked glamorising the effects of alcohol instead. After conducting a review of the evidence and finding it was not best practice, the use of beer goggles was discontinued.
It is important to be mindful of ethical considerations is a critical part of engaging with any target audience, and there are some specific issues to keep in mind.
Why do we need to consider ethics? It is essential that the way we collect our information does not harm to those we are collecting it from, as this will undermine our overall aims.
What do we mean by harm? Harm can be experienced in many different ways by people, in relation to AOD is it important to make sure that we don’t make people feel isolated, stigmatised or judged by the questions that we are asking. When working with vulnerable groups this is even more important. When working with young people we need to make sure that our surveys are age appropriate. Some ways to ensure our work is ethical is to be mindful of the following:
It can be tempting to ask our respondents everything we’ve ever wanted to know about their demographic. However, care must be taken to restrict our data collection to that which will be directly useful to our program in respect of their privacy and time. Additionally, people are more likely to complete shorter surveys.
Working with some populations, like under 18s, require both ethics committee clearance and the consent of their parent/guardian. Working with other ‘vulnerable populations’ may require ethics clearance.
Particularly when it comes to alcohol and other drugs, care needs to be taken not to reinforce exaggerated perceptions about how many people use or are dependent on drugs, the messaging that all drug use is dependent use, the idea that illicit drugs cause more harm than alcohol, etc.
This requires careful wording, and age-appropriate, context-sensitive questions.
These are unintended consequences of certain types of questions or question wordings.
These are common pitfalls when writing questions that can make your survey hard to understand, take longer to complete, and may even compromise your data.
Stigma is a major barrier to people seeking treatment. Eliminating stigmatising language of all kinds should be a key aim within your CDAT. For example:
Do you consider drug addiction an issue in our community? If yes, why?
Consider the implication of using stigmatising language in this question, as well as the value of the information you will receive.
When you know what you’re trying to ask, it can be tricky to see how it might be confusing for a respondent. For example:
What is your experience with drug use (including weed, alcohol, meth, inhalants, prescription pills and other drugs):
This question is unclear; was use once or multiples times?
One question that is really two separate questions. For example:
How satisfied are you with the health services and drug treatment services available in your community?
Ask either about health services OR social services – they might be satisfied with one but not the other.
These unintentionally sway a respondent towards a particular response, instead of being neutral.
If you have had some experience with drug use (including weed, alcohol, meth, inhalants, prescription pills and other drugs) how much did peer pressure influence you to try drugs?
You’re automatically swaying the respondent towards stating it had influence.