What If Anyone in the World Could be a Participant in Your User Study?

From Anita Borg Institute Wiki

Please see below notes by Nare Hayrapetyan, Uma Murthy and Keita Del Valle. More information about the panel is available here. This page has resources to conduct remote user studies, including links to tutorials and tools.

Presentation Slides‎

Panel Notes

Introductions

• Uma Murthy - parts of documents, subimage fish identification study overview
  • Study was done over skype
  • What worked - having a partner and detailed planning of all activities
• Tejinder Judge
  • PhD project - connecting families
  • Challenges -
    • getting people to participate
    • everything that could go wrong, went wrong (systems were down) - people were annoyed and hostile
    • people weren’t using the system so getting data was a challenge
    • no one used the diaries they were given
    • she was on call 24/7 for tech support
    • one thing that really helped was being flexible with her schedule
• Erika Poole - project share files with friends etc
  • Challenges -
    • software broke, didn’t work on other people’s computers
    • people do get mad, people would threaten to quit the study because they'd get so mad
    • advice - if you don't have to put software on participant's system, it's best - they will blame you for anything that goes wrong with their system. If you must put software on their system, choose a language that will let you push updates, be resilient.
• Sharoda - analyzing data from social networks
  • question - what type of questions people ask their friends on social networking websites
  • Challenges:
    • didn’t know who the people were, had to make certain assumptions about them
    • wanted to differentiate between questions and statements
    • for help analyzing the large amount of info she used Mechanical Turk, a marketplace for human tasks where Turkers (workers) get paid a few cents to do your task. But she had to make sure data coming from Turkers was valid. You don't know their motivations and you don't know if they understood task.
    • make sure your data is valid. Her advice - you need to think really carefully how to ensure validity of data, what kind of controls can be put in place.
    • she had to figure out how to deal with people who were unhappy with payment. She designed something to weed out people that she think just clicked through task without really reading it and wouldn't pay them. But people would get upset and say they really did the work - she didn't have a way to know for sure so she paid them anyway

Tips/questions/issues

• Establish good relationship with the people you work with (users).
• Use a market research firm to do the recruiting for you.
• Be prepared to deal with awkward moments - e.g. you find out that the daughter is looking at porn on the laptop.
• Connecting with participants
  • Find motivation for users to participate in your study.
  • Try to find something to bond over, ask people about them, establish a connection.
• Bring second person with you to have another representation for the study researcher.
• Get people from diverse backgrounds to participate and also within your team.
• Children can be hard to interview - some are more talkative than others.
  • Try to interview them beforehand (phone or mock interview) to find out if they’re talkative and articulate.
• Incentives for participants - should you pay participants?
  • Question - was there difference between the data you got from people who wanted to get paid and those who didn’t care about monetary reward.
  • Bring the families dinner, you’ll be more equal and not so formal.
  • Keep in mind with monetary incentives you are changing the dynamic - you become in a position of power which could change data
  • Sometimes incentives work great and sometimes not. Money will get you participants fast but expectations can change.
• Before you start your research, think about what your data is going to look like, then it’s more focused.
• Don't make assumptions - E.g. don't design online games for kids in middle school, schools might not even have computers.
• Collect as much data as you can.
• Be respectful and nice to your users.
• Remember people have many other commitments, don't burden them with too much, be respectful of their time.
• You have to make a commitment to your study - if you just throw it out there, it will be more qualitative in nature and will be more subject to interpretation.
• Diaries are not dead yet, if they're tied to specific tasks