PRINT

Writing surveys

Surveys may be one of the first things we consider when we think about evaluation, but they aren’t always as easy to design as they might seem. Below are some examples of the things that you need to consider when compiling a survey.

rounded-slice15

A well-written survey carefully considers:

  • Do I really need to be asking this, or is this information already available (e.g. within an existing body of literature, has the data been collected by another organisation)?
  • How am I going to directly use the data I gather? (can this information be used to inform or enhance my program?
  • Are there ethical considerations for my target audience (e.g. young people, vulnerable populations)?
  • How am I going to phrase my questions so they’re:
    • Clear and easy to understand
    • Get me the data I need
    • Minimise the time spent

Critical considerations

There are a number of important considerations when writing survey questions, as well as some common traps when it comes to wording your questions clearly. We go through these below, and include a relevant example question where appropriate.

Be wary of confusing fun with effective!

Often, we use pre-and-post survey to get feedback on a program from our target audience. It’s easy to confuse a participant’s enjoyment of a program or activity with the program or activity meeting its objectives.

For example, the Victorian Police, as part of the evaluation of their SSMART ASSK program with young people, stopped using beer goggles in part because student feedback indicated many found them to be “one of the best things about the program” with an overwhelming emphasis on how “fun” they were. This supported previous concerns that the “fun” nature of the activity was undermining the knowledge it was intended to provide, and that it risked glamorising the effects of alcohol instead.  After conducting a review of the evidence and finding it was not best practice, the use of beer goggles was discontinued.

Ethics issues

It is important to be mindful of ethical considerations is a critical part of engaging with any target audience, and there are some specific issues to keep in mind.

Why do we need to consider ethics? It is essential that the way we collect our information does not harm to those we are collecting it from, as this will undermine our overall aims.

What do we mean by harm? Harm can be experienced in many different ways by people, in relation to AOD is it important to make sure that we don’t make people feel isolated, stigmatised or judged by the questions that we are asking. When working with vulnerable groups this is even more important. When working with young people we need to make sure that our surveys are age appropriate. Some ways to ensure our work is ethical is to be mindful of the following:

Collecting unnecessary data

It can be tempting to ask our respondents everything we’ve ever wanted to know about their demographic. However, care must be taken to restrict our data collection to that which will be directly useful to our program in respect of their privacy and time. Additionally, people are more likely to complete shorter surveys.

Under 18s and ‘vulnerable populations’

Working with some populations, like under 18s, require both ethics committee clearance and the consent of their parent/guardian. Working with other ‘vulnerable populations’ may require ethics clearance.

For example:

Thinking about you, or someone you know, what do you think are some of the things in life that can sometimes influence someone to start using drugs? (including alcohol, weed, meth, inhalants, prescription pills and other drugs).

Please tick all those that apply:

  • Feeling like I don’t belong, don’t want to be at home. Disengagement from family, or community
  • Boredom and/or a lack of things to do in the community
  • Not going to school or work or unable to find work
  • Relationships that are violent or controlling
  • Easy access to drugs
  • No one to provide positive, healthy support or supervision
  • Pain management
  • Peer pressure and social pressure from friends, family, community or media
  • Experimentation
  • Lack of awareness and knowledge of the risks of drug use
  • Depression and or anxiety
  • Lack of community understanding to mental health issues, feeling judged/isolated
  • Other:

Should include a “recreational use” option. Also, if intended for a younger audience, this question may be inappropriate because introducing the idea of drugs to a young person without context and evidence-based education can risk giving them the impression that:

  • There is a drug problem in their school
  • Their peers are using drugs
  • Drug use is common/ normal

Asking this question may be unnecessary if we can source the data from existing research. For example, there is already a body of literature asking why young people start to use drugs that we can draw upon.

 

 Reinforcing misperceptions

Particularly when it comes to alcohol and other drugs, care needs to be taken not to reinforce exaggerated perceptions about how many people use or are dependent on drugs, the messaging that all drug use is dependent use, the idea that illicit drugs cause more harm than alcohol, etc.

This requires careful wording, and age-appropriate, context-sensitive questions.

Participation barriers

These are unintended consequences of certain types of questions or question wordings.

    • Potential for identification
    • Many open-ended questions – how will you collate data collected from open ended questions
    • Long surveys
    • Inaccessible language
For example:

I live:

  • In the Greater Faketown area (between suburb 1 and suburb 2)
  • In the Central Faketown area (between suburb 2 and suburb 3)
  • Outside these regions

This question may be a participation barrier because people may feel that they can be identified by their answer. You can still ask it, but consider whether the value of the results outweigh the possible barrier created.

 

Wording problems

These are common pitfalls when writing questions that can make your survey hard to understand, take longer to complete, and may even compromise your data.

Stigmatising language

Stigma is a major barrier to people seeking treatment. Eliminating stigmatising language of all kinds should be a key aim within your CDAT. For example:

Do you consider drug addiction an issue in our community? If yes, why?

Consider the implication of using stigmatising language in this question, as well as the value of the information you will receive.

Clarity

When you know what you’re trying to ask, it can be tricky to see how it might be confusing for a respondent. For example:

What is your experience with drug use (including weed, alcohol, meth, inhalants, prescription pills and other drugs):

  • Currently in-recovery
  • Previously used
  • Currently using
  • Never used

This question is unclear; was use once or multiples times?

Double-barrelled questions

One question that is really two separate questions. For example:

How satisfied are you with the health services and drug treatment services available in your community?

Ask either about health services OR social services – they might be satisfied with one but not the other.

Leading/ loaded questions

These unintentionally sway a respondent towards a particular response, instead of being neutral.

If you have had some experience with drug use (including weed, alcohol, meth, inhalants, prescription pills and other drugs) how much did peer pressure influence you to try drugs?

You’re automatically swaying the respondent towards stating it had influence.