Writing surveys

Surveys may be one of the first things we consider when we think about evaluation, but they aren’t always as easy to design as they might seem. Below are some examples of the things that you need to consider when compiling a survey.

ADF graphic, aerial park

A well-written survey carefully considers:

  • Do I really need to be asking this, or is this information already available (e.g. within an existing body of literature, has the data been collected by another organisation)?
  • How am I going to directly use the data I gather? (can this information be used to inform or enhance my program?
  • Are there ethical considerations for my target audience (e.g. young people, vulnerable populations)?
  • How am I going to phrase my questions so they’re:
    • Clear and easy to understand
    • Get me the data I need
    • Minimise the time spent

Critical considerations

There are a number of important considerations when writing survey questions, as well as some common traps when it comes to wording your questions clearly. We go through these below, and include a relevant example question where appropriate.

Be wary of confusing fun with effective!

Often, we use pre-and-post survey to get feedback on a program from our target audience. It’s easy to confuse a participant’s enjoyment of a program or activity with the program or activity meeting its objectives.

For example, the Victorian Police, as part of the evaluation of their SSMART ASSK program with young people, stopped using beer goggles in part because student feedback indicated many found them to be “one of the best things about the program” with an overwhelming emphasis on how “fun” they were. This supported previous concerns that the “fun” nature of the activity was undermining the knowledge it was intended to provide, and that it risked glamorising the effects of alcohol instead. After conducting a review of the evidence and finding it was not best practice, the use of beer goggles was discontinued.

Ethics issues

It is important to be mindful of ethical considerations is a critical part of engaging with any target audience, and there are some specific issues to keep in mind.

Why do we need to consider ethics? It is essential that the way we collect our information does not harm to those we are collecting it from, as this will undermine our overall aims.

What do we mean by harm? Harm can be experienced in many different ways by people, in relation to AOD is it important to make sure that we don’t make people feel isolated, stigmatised or judged by the questions that we are asking. When working with vulnerable groups this is even more important. When working with young people we need to make sure that our surveys are age appropriate. Some ways to ensure our work is ethical is to be mindful of the following:

Collecting unnecessary data

It can be tempting to ask our respondents everything we’ve ever wanted to know about their demographic. However, care must be taken to restrict our data collection to that which will be directly useful to our program in respect of their privacy and time. Additionally, people are more likely to complete shorter surveys.

Under 18s and ‘vulnerable populations’

Working with some populations, like under 18s, require both ethics committee clearance and the consent of their parent/guardian. Working with other ‘vulnerable populations’ may require ethics clearance.

Reinforcing misperceptions

Particularly when it comes to alcohol and other drugs, care needs to be taken not to reinforce exaggerated perceptions about how many people use or are dependent on drugs, the messaging that all drug use is dependent use, the idea that illicit drugs cause more harm than alcohol, etc.

This requires careful wording, and age-appropriate, context-sensitive questions.

Participation barriers

These are unintended consequences of certain types of questions or question wordings.

Wording problems

These are common pitfalls when writing questions that can make your survey hard to understand, take longer to complete, and may even compromise your data.

Stigma is a major barrier to people seeking treatment. Eliminating stigmatising language of all kinds should be a key aim within your CDAT. For example:

Do you consider drug addiction an issue in our community? If yes, why?

Consider the implication of using stigmatising language in this question, as well as the value of the information you will receive.

When you know what you’re trying to ask, it can be tricky to see how it might be confusing for a respondent. For example:

What is your experience with drug use (including weed, alcohol, meth, inhalants, prescription pills and other drugs):

  • Currently in-recovery
  • Previously used
  • Currently using
  • Never used

This question is unclear; was use once or multiples times?

One question that is really two separate questions. For example:

How satisfied are you with the health services and drug treatment services available in your community?

Ask either about health services OR social services – they might be satisfied with one but not the other.

These unintentionally sway a respondent towards a particular response, instead of being neutral.

If you have had some experience with drug use (including weed, alcohol, meth, inhalants, prescription pills and other drugs) how much did peer pressure influence you to try drugs?

You’re automatically swaying the respondent towards stating it had influence.