How to make a GOD (Goal-Oriented Design) Survey

 

This blog was originally written on embeemobile.com on March 26, 2019. It is copied here for posterity. Pictures have had to be removed.

March 26, 2019

ResearchDesk™ allows easy and cost-effective access to a high-quality mobile panel in the US, but the value in the results is as much about the quality of the survey as the quality of the panel.

When creating a survey, the hardest part is working out what to ask. There are times when you already have a previous survey, and times when you just brainstorm questions, but the best approach to a pithy survey that achieves your goals is to start with the goals.

This is GOD: “Goal-Orientated Design”.

What is Goal-Orientated Design (GOD)?

Surveys created using GOD start with the question: “What are the killer charts that will absolutely tell my story?”. You then design your survey specifically to populate the data needed to make those charts.

GOD front-loads the work of the final deliverable. In fact, writing any questions at all is one of the last things you do.
The process has the following steps:

1.      Decide what you want to know

2.      Create detailed mock-ups of the charts that will tell you what you want to know

3.      If needed, get consensus from all stakeholders of the study

4.      Create a list of every piece of information that you need to know

5.      Write the questions that provide all the information that you need to know

6.      Code the questions into a survey platform

7.      Estimate roughly how the responses will fall (assuming you have some idea of this), and determine how many completes you’ll need

8.      Set up a ResearchDesk™ study to get the responses

9.      Create the report for stakeholders

Why write surveys with GOD?

Surveys written with GOD are pithier and deliver more compelling results which serve business objectives. Because they tend to be shorter, and nicer to take, drop out rates are lower and respondents concentrate more, giving higher quality answers.

Sadly, most surveys are not Goal-Orientated. It is more usual to start with the questions, run a survey, and then spend weeks trying to make sense of it. These sorts of surveys come when the questions are thought of as “What sort of things might be interesting?”, and then everything that might be interesting is thrown together into a survey.

These question-orientated surveys tend to be longer (often significantly), and don’t hang together very well. Drop out rates are high, and respondent concentration is diminished causing lower quality answers. There is always a temptation to use questions you’ve used before, which might be inconsistent, out-of-date, or simply not appropriate for a mobile platform.

Embee Mobile does not recommend re-running a survey that has previously been fielded through another method to continue a longitudinal study (e.g. a tracker), because results cannot be meaningfully compared between two surveys where the methodology was different.

When migrating a long-running survey to ResearchDesk™, you should take the opportunity to refresh the questionnaire using the GOD approach, and then run a few waves using both methodologies. It’s likely there will be differences in the results, and it’s not a foregone conclusion that the old methodology is more accurate than the ResearchDesk™ methodology, just because it’s older.

GOD: Step-by-Step

Step 1: Decide what you want to know

Although this sounds trivial, it is often the hardest step. This is especially true when you started with the idea of running a survey, before the problem you are trying to solve has been defined.

The emphasis here is to get your stakeholders to think about the following questions, in this order:

“What decisions do I need to make?”

“What information do I need to make that decision?”.

Step 2: Create detailed mock-ups of the charts that will tell you what you want to know

For each bit of information you need to know, create using data that looks like what you’d expect, a picture of the killer chart that would absolutely answer the question.

For example, you might have in your mind a killer chart to show where people spend their money by retail outlet, distributed by the outlet they consider their primary retail outlet:

You might want to actually do all or most of the formatting of the graphic already, assuming that the graphic is being created dynamically from data (see step 9).

In the above example case, the chart was created in Excel, and the data required to make it was:

Step 3: If needed, get consensus from all stakeholders of the study

Once you have created all the charts that will answer all the business objectives, you have a mock-up of the results you plan to deliver.

(Yes – you have actually already written the report! All you’ll need to do in Step 9 is change the data!)

Why not send it round to stakeholders… they’ll be able to approve or suggest edits so much faster if they can see it.

Step 4: Create a list of every piece of information that you need to know

So, you now have all the charts you need to create, and an idea of what data is going to be needed to make each chart.

We’re about to start to think about the survey.

For each chart, you can now list all the information that is needed. There will likely be some overlap between charts, but that’s OK.

In the above chart, we need the following information:

· Age and Gender (because we’re creating this chart multiple times for each age/gender combination, as you can see in the title)

· Which is their preferred bricks and mortar retail location

· How much they spend in each of the stores

Don’t forgot that if a chart is only for a subset of respondents, then the data to define that subset also needs to be determined.

Step 5: Write the questions that provide all the information that you need to know

Now that you have a list of the information that you need, you can write the questions. Age and Gender are already provided by ResearchDesk™, so we don’t need to ask that, but the following questions would get the preferred bricks and mortar location, and their spend.

1.                   Which of these shops do you shop in? [list, incl. none of the above]

2.                   if not checked none of the above, or only one outlet: Which of these shops do you consider your primary retail outlet? [list of those they checked]

3.                   if not checked none of the above: How much do you spend per month in [list of those they shop in, as a rotational group]? [Answer options could be ranges, slider or text box].

You may note that Q1 is not actually providing any information that we need. When writing a survey for a mobile, it is always better to have multiple short questions rather than fewer longer questions. Q1 is designed to make the survey nicer to take, by limiting unnecessary options made available in Q2 and Q3.

Step 6: Code the questions into a survey platform

At this stage, you have all your questions, together with some of the logic that you’ll do.

You don’t need the questions to be in the same order as you thought of them. The only requirements for question ordering is to ensure that any logic required for a question is based on questions that come before it. In general, you should start with screener questions, then a few easy “warm-up” questions, and finish with the harder questions, but the prevailing rule is that the entire survey feels like a journey and not a collection of pieces.

The next step is to open up your survey platform, and code the questions, and the logic in. Most survey platforms will allow you to copy-paste from other applications.

Be sure to check every question satisfies all the Golden Rules.

As always, you need to test your survey thoroughly at this stage, using a mobile device. (Note – preview for mobile surveys on some platforms isn’t always accurate, so as a final check, always check using an real small Android phone).

Step 7: Estimate roughly how the responses will fall (assuming you have some idea of this), and determine how many completes you’ll need

For this, you’ll need to decide on a confidence interval and an acceptable margin of error.

Calculating the number of completes is explained in a previous blog. This provides an explanation and a calculator to help you.

Step 8: Set up a ResearchDesk™ study to get the responses

The last step is simply to set this up in ResearchDesk™, click Launch and wait for the results to come in.

Step 9: Create the report for stakeholders

Step 9 has almost already been done in Step 2. Extract the data from the survey, and plug it into the data that creates your charts, assuming that the charts are dynamically linked to the data.

Add conclusions and titles – something you couldn’t do before you had data, and if you are adding in a statistical comment, then add that in too. (Statistical comments are often written as a small footnote on a slide, which states the total number of panelists asked, sometimes the text of the question, sometimes the margin of error, and if not all panelists, a description of which panelists have been asked).

Now you can deliver the report, maybe on the same day that the survey fielding closed!

 

The Golden Rules of Survey Question Design for Mobile

 

This blog was originally written on embeemobile.com on March 26, 2019 and is copied here for posterity. Pictures have had to be removed.

March 26, 2019

Writing successful and unambiguous surveys depends on having clear and unambiguous questions. Furthermore, mobile surveys are constrained by space on the screen, and it’s essential to ensure that questions all appear above the fold on all screens.

Combining general rules and specific rules for mobiles provides a checklist of Golden Rules that should be checked before any survey is fielded. When you submit a survey in ResearchDesk™, adherence to these Golden Rules is checked.

The purpose of these rules is two-fold: quality and quality. You want respondents to absolutely answer the question you posed (not what they thought you asked), and you want respondents to remain engaged throughout the survey giving the most thoughtful answers they can, without dropping out.

This blog details the following Golden rules:

General Survey Structure

·                  Don’t waste respondents time

·                  Clear language

·                  Phrase questions as ACTION – QUESTION – CONTEXT

·                  Highlight differences between similar sequential questions

·                  One question per page

·                  Eliminate jargon (or thoroughly explain it)

·                  Short sentences

·                  Eliminate cultural references

·                  Never go below the fold

·                  Require every question

·                  Remind the user what they said

·                  Allow going back to change answers

For specific question types

·                  Radio Button and Checkbox questions

·             Inclusivity

·             Uniqueness

·             Monoconcept

·             Exclusive answers

·             Number of answer options

·             Flipping and randomizing answer options

·                  Matrix questions

·                  Ranking questions

·                  Text questions

 

General Survey Structure

Don’t waste respondents’ time

If there is a set of questions that some respondents might not be interested in, then add a question before the group, and allow the respondent to skip all the questions.

For example: Don’t ask a respondent to rate a set of films that have seen at the cinema in a series of 20 questions that ask them to rate each film. Instead, ask them a question “Have you been to see a film at the cinema in the last 6 months?” (Yes, no), then “Which of the following films did you see?”, and give them the list of 20, and then only ask them about the films they’ve seen.

Online (desktop/PC) respondents can easily quite easily scroll through long lists, especially when they have a wheel on their mouse. However, because the equivalent is more cumbersome on a small touchscreen (without missing bits), it’s really important to minimize the amount of unnecessary words that a mobile respondent needs to absorb.

Clear language

Every question and answer choice should be in the clearest possible language. Many professional survey designers are at their peak when they have an eight-year old child. Eight-year olds make the best proof readers of survey questions. Design your survey for an eight year old.

The reason for this is quality. It’s bad when a respondent does not understand the question and gives you an ill-considered answer. It’s much worse when you don’t know if the respondent has given you an ill-considered answer.

If your text is ambiguous, then you can’t know what the respondent understood by the question. The results are worse than useless: they could be completely misleading.

Phrase questions as ACTION – QUESTION – CONTEXT

When wording a question, try to get the actual action as close to the beginning of the sentence. This helps the respondent to be reading the actual question in light of what you want them to do about it.

For example: (bad example) “Considering all the times that you have been to see a movie in a movie theater in the last six months, select the three most important reasons why you buy popcorn.”.

There are usually three things in a question:

Action: (what you want the respondent to do) — “select the three most important reasons”

Question: (what you want to know) — “reasons people buy popcorn”

Context: (when/where etc) — “at movie theaters in the last six months”.

The structure of a question should typically follow the pattern:

ACTION — QUESTION — CONTEXT

Putting the action first helps the respondent to be prepared to know what they need to do and can be thinking about it while the read the rest of the question.

Having the question next is important because we want to absolutely have the respondent thinking.

Putting the context last is usually appropriate, because most often this is going to be the same context as the previous questions. We need to have the minimum amount of the respondents attention at this point, because they are probably already aware of the context.

For example: (rewritten) “Select the three main reasons you bought popcorn when you went to a movie theater in the last six months.”

When writing for a desktop, you have more space on the screen. It’s very common to drop the action to a new line after the question, in a different font. This is fine on a desktop, but is too greedy on real-estate to be appropriate for a mobile survey.

There are times when you are fundamentally changing the context. When this happens, you sometimes see a desktop-focus question that starts “Considering all the times that you’ve been to the cinema in the last six months…”. This is ok on a PC, but the question and action get lost in a plethora of text when using a mobile. You sometimes see questions written like this for every question, and in this writers opinion, that is not even ok on a PC, and absolutely not OK on a mobile.

So, what should you do when you are transitioning the context, and the survey is a mobile survey?

Answer: you should recognise that changing the context mid-survey is like a section break in a book. You should create a page specifically to change the context, then carry on with the ACTION-QUESTION-CONTEXT approach, starting on a new page.

For example: you have been talking about buying popcorn in grocery stores, and now you have a bunch of question about buying popcorn in movie theaters.

Create a page that says something like “Thank you. Now we’re going to ask about popcorn in movie theaters”, on its own, with nothing but a Next/OK button to go to the first of the second set of questions.

Highlight differences between similar sequential questions

When you need to ask the same question about multiple things (which in olden days was a matrix question), the respondents’ experience is to physically see one question per page, and each page looks almost identical.

Therefore, you should draw the respondents’ attention to what has changed. Bolding the change is usually perfect for the job.

This is not just to be helpful. If the difference from the previous question is buried deep in the sentence, then a respondent might end up accidentally checking a response to a different question. This happens often when the original question was a matrix question. (See Matrix questions below).

This is particularly important with mobile surveys, because there’s often less white space, and the other questions are not likely to be on the same page.

You’re trying to help the respondent in the scenario where they have read the question, but been interrupted and have forgotten precisely what you were asking about. Help them to visually find it without having to read the whole question again.

Alchemer Tip: you can change the formatting of the question part of a matrix question centrally in the Style section of Alchemer. There’s a section called “Custom HTML” (or “Custom CSS”), and you can insert the following to change the colour for the question of all matrix questions:

/* Write your custom CSS here */

th.sg-first-cell{color:#ff6600; font-weight: 600}

label{border:0 !important;}

In this case, we’re changing the color to orange, and making it bold. font-weight is normally 600, but is a value between 0 and 1000. 600 works well for this.
We’re also changing the label to have no border. This makes Alchemer look less busy.

One question per page

Ensure that there is one question per page. Every question should fit on the page, and you the respondent should never need to scroll.

It is always better to have lots of pages rather than crowded pages.

In addition, try to find whatever option works to minimize the number of times that the respondent has to find and press something. For example, in SurveyMonkey, they have an option called “Question at a time”, which puts one question on each page. However, this actually presents an “OK” button, followed by a “Next” button. By switching this option off, and then adding a page break between each question means that the respondent only has an “OK” button, which turns the page.

Eliminate jargon (or thoroughly explain it)

You’d have thought this would be obvious, but industry jargon is the biggest reason why consumer surveys fail. You should use language that all respondents will understand unambiguously. Remember the eight-year-old rule.

However, it’s not always easy to avoid industry jargon when your survey is about something that only the industry knows about. When this happens, you need to explain what you mean.

You can even test the respondent. Why not, it’s fun! Respondents will just as merrily spend 20 minutes doing something fun, rather than 3 minutes doing something mind-numbing.

For example (made-up):

Q1: “This survey is about Sprankots. A Sprankot is a photo-torpedic device embedded in your brain, which sends images from one location to another, which could be used to telepathically send images of what you are seeing to a friend who also has a sprankot implant”.

—(page break)—

Q2: “Where would you find a sprankot?” [“on the Play store”, “in my head”, “on my television”, “in a laser guided weapon”, “none of these”].

Q2 would have validation so that if they get the answer wrong, they would be instructed to go back and re-read.

Short sentences

The purpose of a survey question is to accurately extract information/opinion from the respondent into your dataset. Keep it as simple as possible.

Eliminate cultural references

Unless you explicitly know that the respondent will understand a colloquialism or a cultural reference, don’t use one. You don’t know the respondent, so you can say with certainty that you don’t know for sure they will understand.

For example, don’t refer to “scoring an own goal”, or “shopping on Broadway”.

Sports analogies very often end up in surveys which end up going international… these don’t work, especially since few people around the world play or understand the sports played in the USA, and vice versa.

Never go below the fold

When planning for a mobile, long questions are the start of failure for a survey. Respondents often can’t read all the question, and if they have to scroll up and down in order to understand what you’re asking, they will lose patience and give up, or they will get confused and be more likely to give an incorrect answer.

It is ALWAYS better on a mobile to break a long question into smaller questions whenever possible.

Three short questions takes less time than one long question.

Some experienced survey designers tend to link long questions, because their experience comes from the online survey design world, where real estate was less of a problem, and because 10 years ago, sample houses (firms who provided access to respondents) charged by the question. Therefore, they have been brought up believing that one long question is necessarily better than 3 short questions, because it’s cheaper.

This is NOT the case on mobile.

Require every question

It’s much easier to ‘fat finger’ (accidentally press the wrong thing) on a mobile than it is on a PC. It’s good practice to require that every question has an answer, in case they click the “OK” button twice by accident. If a question is required, the second time they press the “OK” button, they will not be allowed to continue.

Remind the user what they said

It’s often a good idea to start a question with “You said you shopped at Kmart”…

It not only provides context to the next question, but allows the respondent to spot if they have mis-clicked previously. This is especially good if you have a question that was not worded in the same way (because you were trying not to lead the question before). Here, you’re describing your interpretation of the previous answer, and giving them a chance to ensure they have not mislead you.

It also gives the impression that you are listening and care about what they are saying.

This is especially useful in the first question after a branch, where the consequences of branching based on an incorrect answer are more serious.

(See also the ACTION-QUESTION-CONTEXT Golden Rule above)

Allow going back to change answers

Respondents can fat-finger, or simply realize that they’d misread a previous question. There’s (almost) never a good reason not to allow them to go back and give you a better answer.

The only exception to this rule is if you have complex screener requirements, and there’s some structure in your questions that would make a respondent know that they were about to get screened out, but they haven’t actually been screened out at that point.

For specific question types:

Radio Button and Checkbox questions

 Inclusivity

Every imaginable scenario that the respondent could have must be included. If this is not possible, then an “other” or “none of the above” answer should be allowed.

 Uniqueness

For radio button questions, in every imaginable scenario, there should not be two possible answer choices. If the question requires that multiple answers are possible, the question must ask the user to choose the one that most applies.

Monoconcept

No survey question should have more than one concept. It is always better to break a question mixing multiple concepts into two (or more) questions. This applies equally to answer choices.

For example: “Why do you like Acme Inc?”. Answer choices: “Because they are competitively priced and I can save money”. This answer option has two concepts: that about them being competitively priced, and that being the respondent thinks he can save money.

When concepts are mixed in the question, the respondent might be answering about one concept, the other concept, either concept or only if both concepts apply. As a survey designer, you will have no idea on what question the respondent answered, and the question has no value and should be purged.

 Exclusive answers

For checkbox questions: These are “none of the above” answer choices. When a respondent clicks on one of these, they should not be allowed to answer any other option. Most survey platforms allow an answer to be ‘exclusive’ or to write question/page validation to ensure this.

If you allow the possibility for a respondent to check “None of the Above” as well as one of the answer choices, you will need to make business rules to decide whether to keep that response, or to purge it. This is unnecessary. It is always better to make it so that it’s not possible to have to make that business rule.

 Number of answer options

The number of answer options should not be more than about 5. This is because, on a mobile, some of the options will fall below the fold, and these are less likely to be clicked.

If you have more than five answer options, try to break it into more than one question, or pipe in a reduced number from a previous question.

If you still have more than 5 answer options, you should consider using a dropdown instead. In general, survey platforms do a good job of rendering dropdown questions, though you should check what it actually looks like on an Android phone. However, nothing beats restricting the answer options.

The exception to this rule is where the respondent will know the answer without reading the answer choices (e.g. “In what state do you currently reside?”). In these cases, dropdowns with many options are perfectly acceptable (see Flipping and randomizing answer options).

Flipping and randomizing answer options

Flipping or randomizing answer choices is often a good idea, because some respondents are more likely to check answers that are higher up the list than lower. ResearchDesk panelists are better than many, but there will always be respondents who don’t read an entire question before selecting answer choices.

Randomizing puts the answer choices in a random order. Your survey platform should allow you to ensure that any exclusive answer choices are not randomized and remain at the bottom of the list.

Flipping randomly turns the answer options up-side down for half of the respondents.

The first thing you need to decide is the way that a respondent will think about answering your question. You need to distinguish between questions where the respondent knows the answer before reading the answer choices, versus one where the respondent is choosing from the answer choices.

If the respondent knows the answer in advance (e.g. “What state do you live in?”), then your objective is to help the respondent find the answer as easily as possible. In this case, many answer options are OK (though should be in a dropdown), but they should be alphabetical.

Most questions, however, the user is selecting from the answer choices provided. The question to ask yourself is “are the answer choices cardinals or ordinals?”.

A set of cardinal answer choices are where there is no natural order (e.g. apples, bananas and couscous).

A set of ordinal answer choices are one where there is a natural order (e.g. “love”, “like”, “indifferent”, “dislike”, “hate”).

Ordinal answer options should never be randomized.

Ordinal answer choices can be flipped, but it’s generally better to ensure that the flip is the same for each respondent. So, if a respondent has been randomly allocated to be a “flip” respondent, then they should see all ordinal questions flipped, or none flipped, but not some mix of the two.

Cardinal questions should be randomized, but if a series of answer options is going to be repeated regularly, then the randomization should be the same if possible, especially if the list of options is long.

 Matrix questions

A matrix question used in an desktop survey is where you see a grid of answers. It’s very common to see questions written for mobiles which were originally designed for desktop, but which are being rendered as a mobile survey.

Matrix questions rarely work well on mobiles, though some survey platforms will do the best they can. Many survey tools will convert a matrix question into a series of questions that are exactly the same as each row.

These are really unpleasant for the user. The respondent sees a sequence of very similar questions. In these cases, it is essential that the difference between the questions is highlighted to the respondent.

None-the-less, repeating the same question over and over is bad practice on a mobile. The respondent does not get the benefit of being able to see a consistent structure as they would on a desktop, and it feels repetitive. Expect high drop-outs and low quality answers.

If you absolutely must use a matrix question, make sure that you have detailed branching to ensure that only highly relevant questions will appear for the user. If necessary, ask a question before the matrix question to find out what is relevant. Also, make sure that the bit that changes in each question if visibly different. (See CSS modification above). 

Ranking questions

Ranking questions are often great questions. These require the user to select some or all of the answer choices and put them into an order and is sometimes called “Forced Ranking”. Ranking questions are only appropriate for cardinal answer choices.

Most survey platforms do a good job of rendering ranking questions on mobiles, but you should test this (on a mobile) to be sure.

If you have lots of answer options, then you should break the question into two questions. The first question asks them which are relevant, and the next question (which is branched assuming there’s more than one that is relevant) asks them to rank those that are relevant.

Ranking questions should be randomized, but when splitting a question with lots of answer choices into two, you should ensure that the answer options are in the same sequence for the two questions. Most survey platforms will naturally pipe answer options from a previous question in the order they were presented, but you should check this is the case with your survey platform.

 Text questions (Open Ends)

Mobile phones are not as easy to type into as online surveys. You should avoid text questions. Where you do need a text question, structure the question so that a short answer is adequate, and if you have question validation based on the length of the question, then this should not be too onerous.

Questions sometimes ask for brand names to be entered unprompted. (e.g. “List any movie theater chains you can think of”). These questions are very hard to handle on a mobile, because many respondents will answer, but the results you get will, in many cases, be the spellcheck version of the brand. Because of this, you should consider any other ways of getting unprompted information from the respondent without depending on them typing the names precisely.

Alchemer tip: within input boxes within Alchemer, there is a validation option where you can write a very long list (thousands if need be) of possible answer choices. You paste in the list of possible choices into the answer validation, and make a setting to auto complete when the respondent has typed enough letters. This is a great way to get a validation response when there are thousands to choose from (e.g. movie chains), and you don’t actually want to list them out. When using this, there should be an option for panelists to write in a movie chain without passing the validation (see inclusivity rule above).