Scripting a counterbalanced survey on Alchemer (formerly SurveyGizmo)
Let’s say we have a survey that asks questions about coffee and tea. How should we order the question sets? Although the answer is likely it depends, there are practices around ordering that can help reduce confounding variables. Counterbalancing is one [1]. In this example, it means:
- 50% respondents answer coffee questions first, tea questions second
- The other 50% answer tea questions first, coffee questions second
Why should we care about splitting participants equally into different ordered groups? It’s to reduce order effects [2]. Let’s say we always show the coffee questions first. Potential consequences could include:
- Carryover effect. Maybe exposure to coffee questions affected the participants; this could then affect the way they answer our tea questions.
- Fatigue effect. Maybe after answering the coffee questions, participants become bored or tired. The fatigue then could affect the way they answer our tea questions: not reading the questions as carefully, rushing through to finish it, etc.
Now, how do we implement counterbalancing in Alchemer? Beyond using embedded logic features, we can use scripts. Scripting is a flexible and robust way to manipulate the survey flow. This article outlines how to implement a counterbalanced survey. I’d like for this to be a 101 intro; please feel free to try it out even if scripting is new to you! Before we dive in, three things —
- The article is fairly long; you can cmd+F “STEP” to jump between steps. I’ll avoid using that word often so as to not saturate the search results.
- Throughout the article, I added pushpin emojis 📌 next to some tips. For ease of access, I’ve included the list of tips below, so you can view them without going through the whole article.
- Disclaimer: you might be able to find ways to achieve the same result without any scripting. I’ve found that scripting has worked better for our past project needs. Personal preferences are also a factor here; do what works for you.
📌 Tips
- Do print statements to make sure the action script is registering and firing correctly. Example:
print("Hello World"). - In preview mode, action scripts are turned off by default. To see the results of your labor, you have to turn it ON by toggling “Fire Actions.”
- Comment your script/code. It helps provide context for future references.
- Progressively add complexity to your script. Whenever you add something new, check if the logic is correct and if the printed statements are what you expect. Don’t go all out at once. Incremental additions makes it easier to isolate errors and debug.
STEP 1. Set up your survey structure.
Here’s the survey structure we’re following: name, coffee, tea, raffle entry, and the final thank-you confirmation. Our goal is to make it so that there are two flow options:
- Name → coffee → tea → raffle entry → confirmation
- Name → tea → coffee → raffle entry → confirmation
This mirrors a recent survey we launched, except I’ve grossly simplified it and swapped out the questions from cloud technology to beverage. In Alchemer, set up your survey as follows:
Page: Name-----------------------------------Page: Coffee section
Coffee question 1
Coffee question 2-----------------------------------Page: Tea section
Tea question 1
Tea question 2-----------------------------------Page: Raffle entry
Email for optional entry-----------------------------------Page: Confirmation
STEP 2. Add a new page wherever the script actions happen.
Think about it in terms of the overall flow. Where should the participant be directed or redirected? Whenever we might evaluate an entered response or jump to a specific page, that’s where we add a new page so that we can add our action scripts there. Continuing with our example, let’s see where we would add a new page and why.
Page: Name-----------------------------------Page: ADD ACTION HERE 🔸
We add it here so that when a participant clicks "Next" after filling out their name, some script action fires to randomly direct the participant to either the coffee page OR the tea page.
See STEP 3 on how to do this.-----------------------------------Page: Coffee section
Coffee question 1
Coffee question 2-----------------------------------Page: ADD ACTION HERE 🔸
We add it here so that when a participant completes the coffee section, we decide where to send them. Remember, 50% participants see the coffee section first. If the tea section has not been completed, we take them there. If the tea section has already been completed, we take them to the raffle entry. See STEP 4.-----------------------------------Page: Tea section
Tea question 1
Tea question 2-----------------------------------Page: ADD ACTION HERE 🔸
We add it here so that when a participant completes the tea section, we decide where to send them. Remember, 50% participants see the tea section first. If the coffee section has not been completed, we take them there. If the coffee section has already been completed, we take them to the raffle entry. See STEP 5 on how to do this.-----------------------------------Page: Raffle entry
Email for optional entry-----------------------------------Page: Thank-you confirmation
STEP 3. Action script no. 1 of 3: direct participants randomly to either the coffee section OR the tea section.
Select ADD NEW ACTION on the page, then scroll to the bottom where it says “Custom Script.” Alchemer uses the programming language, Lua [3]. We’ll use Lua here (goes without saying — you can use JavaScript if you’re already familiar with the language, too).
To start, I like to do a print statement to make sure that the action script is registering and firing correctly 📌. A print statement prints whatever you feed it to onto the console or the webpage. Enter the following into the script box:
Save the script and click on the preview button. One reminder here: in preview mode, action scripts are turned off by default. So to see the results of your labor, you have to turn it ON. 📌 There are so many times when I’m debugging and I’d think there’s something wrong with my script — but no, I just forgot to turn it on.
Once we make sure that the action is firing properly, we can dive into the scripting logic itself. Revisiting our intention here: the expected flow is that the participant will enter their name → land on this page → get directed randomly. This means the following needs to take place:
- Generate a random number (either 1 or 2)
- Direct the person to coffee page if the number is 1
- Direct the person to tea page if the number is 2
Note that -- means comment. If you look at line 4 and line 8, those lines are meant to be human-readable. The system will ignore these lines that start with the double dash. The reason I add these is to document what the code is doing. Commenting helps provide more context for future references. 📌
Line 5 says we’re generating a random number that’s either 1 or 2. On line 6, we print out the number that’s generated — again, just a nice way for us to check what’s actually going on behind the curtain.
Line 9 — 13 is your basic if-else logic. If the pageOrder is 1, then we take that person to the coffee page (id = 3). If the pageOrder is 2, we jump to the tea page (id = 4). Note that jumptopage(id) takes the page ID, NOT the page number [4]. There are multiple ways to check a page ID:
- Go to the question page and view the little ID tag next to the title
- In the action script panel, find the “Question & Page ID Lookup” dropdown
You might be thinking, what’s different between a page ID and a page number? Each page and question in Alchemer is assigned a unique, persistent ID. The page number depends on the ordering while the ID does not. In other words, changing the ordering of your pages and questions does NOT change this ID.
You might also notice that the IDs in my example are not identical to yours. Make sure to use what’s shown in YOUR survey builder — that’s how your script can find the right pages within the context of your survey.
Once you’re done, save the script, and do a quick preview. If it functions correctly, right after entering your name, you should sometimes see the coffee page first and sometimes see the tea page first. Don’t worry about hiding this action page from participants — the action fires instantly, so the system is likely to finish executing the jumptopage() command and directing the participant BEFORE the participant has a chance to see this action page.
STEP 4. Action script no. 2 of 3: when a participant finishes the coffee section, take them to the tea section OR the email raffle page.
When a participant finishes a coffee page, there are two possibilities:
- They saw the coffee page first. Now they need to proceed to tea.
- They saw the tea page first already. Now they are done with both coffee and tea sections; it’s time to take them to the email raffle page.
We can know this by checking whether the tea questions are answered. There is a getvalue(questionID) method that grabs a question by its unique ID and returns its value (aka the response) [5]. One of the tea questions I have is:
You answered "I drank tea this morning." Which of the following tea(s) did you drink? Select all that apply.[ ] Green tea
[ ] Black tea
[ ] Earl grey
[ ] Other - write-in
12 is the question ID, sogetvalue(12) effectively gets me its response. Alchemer stores the returned value in an array format [6]. If the array is empty, that means there are no responses yet. Conversely, if the array is not empty, that means there are responses recorded.
The diagram below depicts getvalue(12). On the left is what we would get if the participant has not answered the question. On the right is what we would get if the participant answered green tea and black tea.
Ready to move on? This snippet below shows how we can calculate whether the array is empty:
On line 5, we declare a variable called count and assign it the value of 0. On line 6, we do a for-loop to traverse through the data: as long as there is something valid left, we run line 7, which is to increase count by 1. The following illustrations demonstrate the traversal:
Again, if count ends up being 0, the tea question response array is empty AND we should direct the participant to the tea page. ELSE, the participant has completed both the coffee and tea sections, therefore is now ready to go to the email raffle page. This is the full script:
Now, when I’m still writing the script, I usually leave out the jumptopage() statements — so, line 25 and line 29 above would be commented out. Instead, I print out what’s supposed to happen (lines 23–24 and lines 27–28). This is because I prefer to examine the survey in preview mode first and look at the printed statements. If the logic is correct and the printed statements are what I expect, then I proceed with adding spicier actions. 📌 Otherwise, if the logic is incorrect yet the spicier actions are there, all that hopping between pages would make it trickier to debug/troubleshoot.
STEP 5. Action script no. 3 out of 3: when a participant finishes the tea section, take them to the coffee section OR the email raffle page.
This is very similar to how we did it previously, so you’re welcome to try it out before reading this section! If you prefer to follow along, read on. When a participant finishes the tea page, there are two possibilities:
- They saw the coffee page first. Now they are ready for the raffle page.
- They saw the tea page first and have not completed the coffee questions. They need to go to the coffee section.
We can know this by checking whether the coffee questions are completed. Use getvalue(questionID) to grab a question by its unique ID and return its response value. One of my coffee question response ID is 7, sogetvalue(7) effectively gets me the coffee question response. If the getvalue(7) is empty, that means there are no responses yet. Conversely, if getvalue(7) is not empty, that means there are recorded responses.
This snippet below shows how we can calculate whether the array is empty:
On line 5, we declare a variable called count and assign it the value of 0. On line 6, we do a for-loop to traverse through the data: as long as there is something valid left, we run line 7, which increases count by 1.
This means if count ends up being 0, the coffee question response array is empty, and we should direct the participant to the coffee page. ELSE, the participant has completed both the coffee and tea sections, therefore is now ready to go to the email raffle page. This is the full script:
Save the script, test the survey, voila! This is how the survey flow should behave now:
Thanks so much for your time. Hopefully this is of value to you. If you have other tips and tricks you’d like to share about scripting in Alchemer, I’m all ears.
