Cameron Brick

The Qualtrics help pages are actually quite good. Learn about basic topics like survey flow, randomization, branching, etc.

Recruitment

Free

1. If at the UvA: our participant pool of psychology undergrads can provide hundreds per term. Get ethics permission through rms.uva.nl, and then when approved go to lab.uva.nl, log in as a researcher, and go to the "recruitment" tab (upper left). Then search for your project under "search projects" and typing FMG-2090 into "ethic title". You’ll need to provide more information about the recruitment (see bsl.socsci.uva.nl), then you can request publication.

2. StudySwap

   • http://daniellakens.blogspot.com/2017/08/towards-more-collaborative-science-with.html

3. /r/SampleSize

4. SurveyCircle

Paid

1. Qualtrics, Kantar, Nielsen, YouGov, Gallup, Harris Polls, Respondi, Dyanata, MSI, Prolific, MTurk (these last three are perhaps most affordable)

Qualtrics

1. If using MTurk or Prolific, use embedded data to automatically capture information like their Participant ID. Embedded data can also capture mobile vs. desktop, where they clicked on the survey link from, etc. See image above.

2. Options: Security: Anonymize responses: “Don’t record respondents’ IP Address, location data, and contact info”

3. Use separate blocks for sections. It’s often easier to use separate blocks than to use page breaks, depending on the situation.

4. Learn about survey flow, randomization, and branching.

5. Use request response rather than force response: it’s more ethical not to force participants.

6. Relabel all variables from V1, V2, etc. to informative labels: age, female, etc. Notice that female is preferable over gender because the latter is ambiguous as to high vs. low. Keep the labels short and don’t use spaces or special characters (numbers are ok).

7. There are three levels of randomization: of response options, of items within a scale, and of questionnaire stages/sections. I recommend randomization within variables (e.g., the order of the four identity questions); for the rest, please think through a plan that makes sense to you to minimize order effects, etc.

8. Manipulation checks: in experiments, it’s good to add a question or two that require having paid attention to the manipulated content, whether or not they were convinced or changed their beliefs or preferences in any way. These manipulation checks measure exposure and basic comprehension of the treatment.

9. Add timing questions to some critical pages, e.g., the manipulation.

10. To control participant behavior, for example to maximize treatment exposure, consider using code to show/hide the next button for a specified period, allow users to start a video but not pause it, etc.

11. To record clicking: [2/22 some are reporting issues with this code: be sure to pilot test]

    A) Add the embedded variable to survey flow before the question block:

    clicked = 0

    B) Add the link to the question text (in HTML mode):

    This is a link: <a href="http://www.google.com" target="_blank" id="extLink">Click here</a>

    C) Add the following JavaScript to the question where "extLink" matches the id of the link in (2) and "clicked" matches the embedded variable in (A).

    Qualtrics.SurveyEngine.addOnload(function() {     $('extLink').on('click', function(name, event) {         Qualtrics.SurveyEngine.setEmbeddedData('clicked', '1');     }); });

12. Pilot test your study before launching it! A lot! See https://twitter.com/CameronBrick/status/1468880976092741635