Popular in Professional

Culture and Society

The City as a Growth Machine: Toward a Political Economy of Place

New York University Harvey Moltoch

Education

The Role of Education in Reducing Racial Inequality

University of Southern California Pedro A. Noguera

Law

Diversity and Social Mobility in the UK legal profession

Law Society I. Stephanie Boyce

Internet Law

The Law of AI for Good

University of San Diego Orly Lobel

Pharmacy

Cytotoxic Activity and Anti-Migratory Effect of Berberine-Phytantriol Liquid Crystalline on Non-Small-Cell Lung Cancer

University of Technology Sydney Kamal Dua

Human Rights Law

Transforming Restorative Justice for Transitional Settings

University of Nottingham Kerry Clamp

Education

Improving the Statistical Power and Reliability of Research

View more insights in Professional - Accounting

View more figures from University of Massachusetts at Amherst

View the page of Jeremy Bentley

Read the Study

Jeremy Bentley describes various reasons why MTurk research may have power and reliability issues in this study. These issues can be addressed by leveraging MTurk’s strengths.

Image courtesy of interviewee. January 26, 2024

Generating Workspace

Export as PDF

This video video is a detailed discussion about the challenges and considerations of conducting behavioral research using Amazon Mechanical Turk (MTurk) as a participant source. It outlines the difficulties researchers face in recruiting participants traditionally, particularly from limited pools such as students or alumni, and how MTurk can provide a larger and more diverse participant base.

However, the speaker emphasizes that MTurk is not a perfect solution and introduces various sources of noise that can affect the reliability and validity of research findings conducted through this platform. Key points include:

1. **Unqualified Participants**: There is a risk of low-quality responses from participants who may not meet the necessary criteria for the study, including potential bots. The speaker suggests pre-screening participants to ensure they are qualified.

2. **Lack of Attention**: Participants may be distracted or rushed, which can lead to unreliable data. The speaker recommends using attention checks during experiments to ensure participants are engaged and understand their tasks.

3. **Lack of Experimental Control**: MTurk participants are more heterogeneous than traditional student samples, which can add variability to the data. Researchers must decide whether to embrace this diversity or restrict participant demographics through prescreening.

4. **Nonna Participants**: Frequent MTurk users might have preconceived notions about study manipulations due to their experience with similar tasks. The speaker advises using novel experimental paradigms to engage such participants meaningfully and maintain data integrity.

5. **Addressing Effect Sizes**: The speaker discusses the importance of sample size in detecting effects, particularly when working with small effect sizes typically observed in behavioral research. Suggestions for calculating appropriate sample sizes based on the nature of the effects are provided.

In summary, while MTurk can be a valuable tool for researchers, careful consideration must be given to participant selection, attention, and potential biases. Strategies for mitigating these issues are discussed to improve the reliability and validity of research findings obtained from MTurk.

Language:

Export as PDF

Export as PDF
Here are some article suggestions that discuss similar themes regarding the use of Amazon Mechanical Turk (MTurk) for conducting behavioral research, addressing challenges faced by researchers, and strategies for improving data quality.

1. **"Using Amazon Mechanical Turk to Collect Data for Your Research: Advantages and Challenges"** - This article explores the practical aspects of using MTurk, highlighting both its benefits for reaching diverse participant pools and the potential pitfalls that researchers should be aware of.

2. **"The Role of Attention Checks in Online Research: Ensuring Participant Engagement on MTurk"** - A focused discussion on the importance of implementing attention checks in online research, emphasizing how these can help mitigate issues of participant distraction and maintain data integrity.

3. **"Best Practices for Recruiting Participants on Amazon Mechanical Turk: A Guide for Researchers"** - This guide outlines effective strategies for recruiting high-quality participants on MTurk, including pre-screening techniques and considerations for demographic diversity.

4. **"The Heterogeneity of MTurk Participants: Implications for Behavioral Research"** - An exploration of the diversity in participant backgrounds on MTurk and how this variability can impact research findings, along with suggestions for managing this diversity in study design.

5. **"Addressing Bias in Online Samples: Strategies for Improved Validity in Behavioral Research"** - This article discusses biases that can arise in online participant samples, including issues specific to MTurk, and offers methods for researchers to enhance the validity of their studies.

6. **"Understanding Effect Sizes in Behavioral Research: Sample Size Considerations for MTurk Studies"** - A discussion on the importance of calculating appropriate sample sizes in behavioral research, particularly when using MTurk, with a focus on detecting small effect sizes.

7. **"The Ethical Considerations of Using MTurk for Research: Participant Welfare and Data Integrity"** - This article delves into the ethical implications of using MTurk as a participant source, discussing how researchers can ensure the welfare of participants while maintaining data quality.

These articles will provide a deeper understanding of the opportunities and challenges associated with using MTurk for behavioral research, as well as practical strategies to enhance research outcomes.

X-TIMESTAMP-MAP:LOCAL=00:00:00.000,MPEGTS=900000

00:00.185 --> 00:03.525
Anyone who has done behavioral research knows

00:03.825 --> 00:06.725
how difficult it is to get participants, right?

00:06.815 --> 00:11.165
We're recruiting from our own students or from alumni

00:11.625 --> 00:14.365
or people in the the neighborhoods, something like that.

00:14.385 --> 00:17.165
And it's just a hassle, especially if you're trying

00:17.165 --> 00:20.405
to run multiple studies within a given semester, right?

00:20.785 --> 00:21.965
If I'm using my own students,

00:22.315 --> 00:24.885
I've got maybe a hundred students a semester that I can use,

00:25.345 --> 00:27.045
and I can't do any follow-up experiments

00:27.045 --> 00:29.005
because they already did the first experiment.

00:30.065 --> 00:31.485
So it's a challenge.

00:32.185 --> 00:33.845
And income's, mechanical Turk,

00:34.455 --> 00:37.085
mechanical turk isn't really meant for research.

00:37.635 --> 00:41.085
It's meant for any type of, task

00:41.195 --> 00:44.045
that can't really be automated well by it.

00:45.385 --> 00:48.285
And it's meant to outsource that as kind

00:48.285 --> 00:49.525
of little micro jobs.

00:49.785 --> 00:53.565
So you could think of it kind of like the, the Uber of

00:54.545 --> 00:56.885
any type of computer task, right?

00:56.965 --> 00:59.565
Where we get someone to transcribe something

00:59.585 --> 01:02.485
or you get someone to go look up some information on the

01:02.645 --> 01:04.965
internet or, something like that.

01:06.155 --> 01:08.295
And it's great for research in

01:08.295 --> 01:10.895
that it provides this huge sample of participants

01:11.285 --> 01:13.975
that you can use to do your studies.

01:15.595 --> 01:19.735
Um, so that's kinda the background on Amazon Mechanical Turk

01:19.955 --> 01:22.455
and why a lot of people find it really useful.

01:22.745 --> 01:25.975
There had been a lot of research studies that I had seen,

01:25.975 --> 01:28.375
especially when I was a PhD student, kind

01:28.375 --> 01:30.935
of touting the glory of Mechanical Turk

01:30.955 --> 01:33.495
and saying, rah, ru haha, it's wonderful.

01:34.295 --> 01:36.655
Everyone should use it because it's better than students.

01:36.765 --> 01:39.015
It's better than this. It's, it works in every way.

01:39.855 --> 01:41.995
And when I started doing some research, I went,

01:42.575 --> 01:44.115
you know, it, it's great.

01:44.255 --> 01:47.555
It is very helpful, but it has some drawbacks.

01:47.945 --> 01:49.435
It's not the magic bullet.

01:50.875 --> 01:55.655
And so the, the idea behind this paper was, you know,

01:55.815 --> 01:57.735
I ran into a lot of challenges when I

01:57.735 --> 01:58.815
started doing this research.

01:59.545 --> 02:03.175
Maybe I could walk some people through, some of those,

02:03.475 --> 02:06.015
road bumps I ran into

02:06.515 --> 02:08.135
and see if I can help them avoid them.

02:08.595 --> 02:12.615
So statistical power refers to how likely you are

02:12.915 --> 02:16.185
to find a result that is true, right?

02:16.325 --> 02:18.145
So if you know, say you're trying to find,

02:18.485 --> 02:19.865
are men taller than women?

02:20.255 --> 02:22.185
Well, we know that on average that's true.

02:23.245 --> 02:24.705
How likely are you to find that

02:24.705 --> 02:26.345
with a different sample size, right?

02:26.565 --> 02:29.305
If you just take two random people, one man, one woman,

02:29.845 --> 02:31.865
it may be true, it may not be true,

02:31.925 --> 02:33.345
but if you take a hundred

02:33.445 --> 02:36.225
or 200 of each, then you're gonna find that statistically.

02:36.905 --> 02:40.365
So that's referring as kind of type one error, the chance

02:40.435 --> 02:43.565
that something is true, but you fail to find it.

02:43.565 --> 02:46.755
Statistically, reliability is kind

02:46.755 --> 02:47.795
of the opposite side of that.

02:48.525 --> 02:51.175
It's the chance that you will find a statistically

02:51.175 --> 02:53.975
significant result totally by chance.

02:54.555 --> 02:58.535
So it's an error. Your significance isn't reflective

02:58.535 --> 02:59.845
of the real world, right?

03:00.025 --> 03:02.485
That's the chance that you pair up two people

03:02.705 --> 03:06.605
and you find happen to get a woman that's taller than a man.

03:06.915 --> 03:08.245
Sure, that happens all the time.

03:08.635 --> 03:10.245
That doesn't mean it's generalizable.

03:11.565 --> 03:15.505
So research has typically focused on the type two error.

03:15.845 --> 03:18.145
That's the one that reviewers

03:18.365 --> 03:21.065
and editors are generally more concerned about.

03:21.775 --> 03:25.265
Type two error is the stuff that then can border on fraud

03:25.365 --> 03:28.785
or misconduct or just wrong inferences

03:28.965 --> 03:31.745
and people making real world decisions

03:32.275 --> 03:34.025
based on something that's not true.

03:34.845 --> 03:37.345
So it's important to reduce type two error,

03:38.535 --> 03:40.945
but you know, as researchers,

03:41.645 --> 03:43.665
we also should care about the type one error,

03:44.135 --> 03:47.025
because I don't want to run a study that

03:47.535 --> 03:50.145
doesn't have any chance of finding results.

03:50.825 --> 03:51.905
I don't want to, if I were doing

03:51.905 --> 03:55.345
that man versus woman study, I wouldn't want to

03:56.785 --> 03:58.485
run it with too small of a sample size

03:58.485 --> 04:02.445
and go, I guess men aren't taller than women, simply

04:02.445 --> 04:04.605
because I designed my study wrong,

04:04.785 --> 04:07.565
or I included the wrong control variables,

04:07.565 --> 04:09.445
or didn't measure certain things in the right way.

04:10.355 --> 04:13.615
So while my paper touches a little bit on the type two

04:13.705 --> 04:18.655
error, that reliability, it's a lot more focused on power,

04:19.415 --> 04:20.975
reducing that type one error

04:21.595 --> 04:24.775
and making it so researchers can be more successful

04:24.795 --> 04:27.255
and have a better chance of finding results

04:27.285 --> 04:28.655
that are really there.

04:29.195 --> 04:31.975
And they're important to find. The biggest reason why

04:32.795 --> 04:36.895
you might not find an effect is that your

04:37.975 --> 04:40.975
intended effect isn't the only thing mattering in the real

04:40.975 --> 04:42.465
world, right?

04:42.765 --> 04:45.705
So gender is not the only thing

04:45.705 --> 04:46.905
that determines someone's height.

04:47.405 --> 04:49.585
Yes, it matters, but there are a whole lot

04:49.585 --> 04:53.745
of other things out there, and noise

04:54.285 --> 04:57.665
is kind of our way of defining, broadly speaking, all

04:57.665 --> 04:58.905
of those other factors.

05:00.985 --> 05:04.085
Now, experimentalists are used to dealing with noise,

05:04.665 --> 05:07.525
but in my paper, I outline a couple of sources of noise

05:07.525 --> 05:08.885
that you face with MTurk

05:09.275 --> 05:13.085
that you wouldn't face in a more traditional lab environment

05:13.595 --> 05:17.085
like using, you know, student population.

05:18.025 --> 05:20.885
And I try and discuss those five sources of noise

05:20.945 --> 05:24.405
and present ideas of how to deal with them

05:24.505 --> 05:27.205
or reduce that influence of that noise in your research.

05:28.495 --> 05:32.755
Uh, the first source of noise is unqualified participants.

05:34.315 --> 05:36.775
And the basic idea here is you might not have,

05:36.875 --> 05:38.775
you might have people who are participating

05:38.915 --> 05:41.575
who aren't really the people you want.

05:42.115 --> 05:44.455
In fact, they might not even be people at all, right?

05:44.455 --> 05:47.215
They might be bots, and you don't want

05:47.215 --> 05:49.815
to be doing some behavioral study thinking.

05:50.015 --> 05:51.655
You've got humans responding

05:51.675 --> 05:54.935
and later finding out that you were actually interfacing

05:54.935 --> 05:56.535
with chat GPT, right?

05:56.735 --> 05:57.965
That that would be a bad thing.

05:59.085 --> 06:00.785
Or even if they are real people,

06:01.245 --> 06:02.385
are they the people you want?

06:03.005 --> 06:04.705
Are they even fluent in the language you're

06:04.705 --> 06:05.945
administering the experiments on?

06:06.405 --> 06:08.305
Or are they kind of randomly guessing?

06:08.885 --> 06:10.585
Do they have the required backgrounds?

06:11.245 --> 06:14.145
In my research, I often want people who are managers,

06:14.645 --> 06:17.545
or at least who've been in a full-time job

06:17.715 --> 06:20.065
where they've got someone looking over their shoulder

06:20.285 --> 06:22.825
and kind of giving them a bonus of some kind.

06:23.565 --> 06:27.545
If I had a 16-year-old doing the experiment, it just,

06:28.145 --> 06:30.305
I don't think they would understand enough about the real

06:30.305 --> 06:31.785
world to really react.

06:33.395 --> 06:36.535
And then not just do they have the background?

06:36.535 --> 06:37.935
Do they have the knowledge they need?

06:38.635 --> 06:41.535
Um, so for example, in my dissertation, I needed people

06:41.535 --> 06:42.535
who knew how to play chess.

06:43.855 --> 06:46.905
And if I had someone who didn't really know

06:47.055 --> 06:49.265
that wouldn't work, I've got some other papers

06:49.265 --> 06:51.665
where I need people who are really avid basketball fans,

06:52.745 --> 06:55.285
and I needed to make sure they had that just

06:55.285 --> 06:56.965
so my experiment would have a chance

06:56.965 --> 06:58.005
of working the way I wanted.

06:59.305 --> 07:00.725
So how do you deal with this?

07:00.785 --> 07:03.445
How do you make sure you get the right people in your study?

07:04.645 --> 07:08.175
Well, the solution I suggest in the paper is prescreening.

07:08.965 --> 07:11.065
That's where you do some types of checks

07:11.765 --> 07:14.425
before you actually run your experiment

07:14.965 --> 07:16.705
to make sure you're getting the people you want.

07:18.315 --> 07:20.015
Now, it's important when you do these checks

07:20.015 --> 07:21.935
that you're not just asking them things like,

07:22.275 --> 07:23.495
Hey, are you good at chess?

07:23.795 --> 07:26.335
Or do you like basketball? Right?

07:26.395 --> 07:31.375
That's, that's not gonna work because people lie, right?

07:32.155 --> 07:34.655
If you tell me, Hey, I need an expert in chess,

07:34.725 --> 07:36.655
I'll give you a million dollars if you're an expert in

07:36.655 --> 07:38.775
chess, I'll say, yeah, I'm an expert.

07:39.605 --> 07:42.055
It's just, it's kind of natural.

07:42.195 --> 07:44.215
And so if you just ask that self-report,

07:44.835 --> 07:48.175
you can end up screening in liars instead

07:48.175 --> 07:50.415
of screening in the people you actually want.

07:51.725 --> 07:54.305
So what I suggest is that you use knowledge

07:54.305 --> 07:57.305
or ability checks instead, right?

07:58.005 --> 08:01.545
You may be able to claim that you are

08:02.545 --> 08:06.655
a regular stock market trader and you're a financial expert,

08:07.155 --> 08:09.335
but it's a completely different thing to be able

08:09.335 --> 08:11.695
to compute financial ratios on the fly

08:12.515 --> 08:15.535
or to be able to analyze an in-progress chess game.

08:16.565 --> 08:19.345
Now, it really helps that these kinda screening questions

08:19.345 --> 08:21.225
you're asking are something hard,

08:21.535 --> 08:23.665
that you can't really Google the answer to

08:25.185 --> 08:26.405
the things that people need to know.

08:26.425 --> 08:29.045
And Intuit, that's the way you can tell if they're

08:29.045 --> 08:30.685
really qualified.

08:31.885 --> 08:34.905
Now, you do need to watch out for lucky guesses, right?

08:34.965 --> 08:36.505
You wanna think, well, what percent

08:36.505 --> 08:39.465
of people could pass this question just by guessing?

08:40.335 --> 08:43.875
Um, but those are the types of questions you want.

08:45.455 --> 08:47.475
Uh, the second, source

08:47.475 --> 08:50.195
of noise I discuss is a lack of attention.

08:51.485 --> 08:54.055
This is when you have people who are maybe just distracted

08:54.195 --> 08:55.965
or in a rush, right?

08:56.425 --> 08:59.125
So maybe, you know, they're, they're working from home

08:59.305 --> 09:00.445
and they're watching TV

09:01.265 --> 09:04.445
or they're, you know, kind of doing something else.

09:04.555 --> 09:07.445
They're, they're sitting on a computer doing another

09:07.445 --> 09:11.085
full-time job and kind of on the side doing this MTurk task.

09:11.785 --> 09:13.805
We don't know who these people are and

09:13.905 --> 09:15.365
and how much attention they're paying.

09:16.635 --> 09:19.535
So the solution I propose here is you

09:19.595 --> 09:20.735
try and measure their attention.

09:21.555 --> 09:24.775
You used some, check questions as you're going along,

09:25.435 --> 09:28.215
making sure they understood the important material you're

09:28.215 --> 09:29.865
giving them, right?

09:29.885 --> 09:33.545
So if I'm setting up some incentive in a certain way,

09:33.725 --> 09:36.705
or I'm making them be the employee versus the supervisor,

09:37.505 --> 09:40.545
I ask them, Hey, you know, what's your role in this?

09:40.605 --> 09:41.865
Or How are you going to behave?

09:41.885 --> 09:43.745
What's your, your job in this task?

09:44.525 --> 09:46.105
And make sure they're understanding the

09:46.105 --> 09:47.185
experiment as they go along.

09:47.925 --> 09:50.785
And if they fail, you kick 'em out.

09:50.965 --> 09:53.595
You don't have 'em participate so

09:53.595 --> 09:55.595
that you make sure you're getting the good people in.

09:57.365 --> 10:00.585
Now, even that situation, you may end up with some people

10:00.605 --> 10:03.505
who aren't that great, who kind of guessed

10:03.505 --> 10:04.585
or who have paid attention.

10:05.445 --> 10:07.105
You may have to drop them after the fact.

10:08.125 --> 10:11.375
Um, but at least you're trying

10:12.115 --> 10:13.415
and you can deal with that.

10:13.635 --> 10:17.255
Um, I also suggest using some, different ways

10:17.275 --> 10:19.895
and being very transparent in how you choose

10:19.955 --> 10:21.375
to exclude participants.

10:22.045 --> 10:25.015
Make sure that everyone knows what your design choices were

10:25.015 --> 10:26.735
so you're not being opportunistic in this.

10:29.245 --> 10:34.225
Um, the third source of noise I talk about is a lack

10:34.225 --> 10:37.865
of experimental control, which I'll include in

10:37.865 --> 10:40.065
that you have a, a heterogeneous population.

10:41.905 --> 10:44.955
When we did research, maybe 10

10:44.955 --> 10:47.155
or 20 years ago, we were using students,

10:48.255 --> 10:50.995
and students are sometimes,

10:51.275 --> 10:54.195
colloquially called the weird population.

10:54.705 --> 10:58.875
They're western, educated, industrialized rich,

10:59.175 --> 11:02.335
and demographic, sorry, democratic.

11:02.635 --> 11:06.815
So they are taking this very narrow subset

11:06.995 --> 11:08.175
of the population.

11:09.355 --> 11:11.295
Now, that can be good and bad.

11:12.325 --> 11:14.215
It's good because it means

11:14.755 --> 11:17.215
you don't have all this extra source of noise,

11:17.395 --> 11:20.145
you don't have for purposes

11:20.145 --> 11:22.105
of your experiment baggage, right?

11:22.105 --> 11:24.825
You don't have people having substantially different

11:25.775 --> 11:30.055
upbringings and backgrounds, that that means

11:30.055 --> 11:31.455
that you have greater power.

11:31.965 --> 11:33.335
It's bad because maybe you can't

11:33.335 --> 11:34.655
generalize your results as well.

11:35.915 --> 11:37.735
But then when you move to the MTurk world,

11:38.035 --> 11:39.255
you've completely flipped that.

11:40.055 --> 11:41.855
Suddenly you have people from all over the world

11:42.245 --> 11:44.335
with all different upbringings

11:44.355 --> 11:45.695
and all different backgrounds,

11:46.195 --> 11:48.535
and that's really good for generalizing research,

11:49.475 --> 11:52.895
but it's bad in that it means

11:53.035 --> 11:55.365
They have their own ideas that they're bringing in.

11:55.875 --> 11:58.405
They're not all rich and educated.

11:58.755 --> 12:02.565
They're not all 20 to 23-year-old students.

12:03.735 --> 12:07.155
And that difference of backgrounds means that they're going

12:07.155 --> 12:09.235
to respond a little bit differently

12:09.235 --> 12:12.555
and add some more variation to your analysis,

12:12.565 --> 12:14.235
which just reduces your power.

12:15.445 --> 12:18.755
They may also be using different technology, right?

12:18.755 --> 12:20.155
They could be Googling answers,

12:20.155 --> 12:22.275
they could be using auto clicking software,

12:23.215 --> 12:25.995
and you can prevent them from doing that in a lab.

12:26.015 --> 12:29.375
You can't on MTurk. So what do you do?

12:30.245 --> 12:32.215
Well, you kinda have two options.

12:32.395 --> 12:34.455
Option one is you embrace the differences.

12:35.355 --> 12:38.695
You try and measure all of the different aspects, right?

12:39.075 --> 12:41.215
Ask 'em their age, ask 'em their education,

12:41.475 --> 12:43.295
ask 'em their their income.

12:43.515 --> 12:45.655
Ask 'em where they live, all of these things

12:45.655 --> 12:47.055
that might matter in the real world.

12:47.805 --> 12:50.255
Measure them, make them covariates

12:50.355 --> 12:52.735
and include them in part of your analysis.

12:53.265 --> 12:54.935
Learn from all of that information.

12:56.485 --> 12:58.865
Option two is say, I don't want that.

12:59.245 --> 13:00.705
Uh, that's not the point of my paper.

13:01.205 --> 13:03.945
And you try and restrict people, right?

13:03.995 --> 13:06.105
Maybe you say, no, I I only want

13:06.105 --> 13:10.265
to do research on people located in this geographical area

13:10.765 --> 13:13.825
who speak this language, who are in this age group,

13:13.965 --> 13:17.145
or who are, you know, this, this, this, this, tighten

13:17.295 --> 13:20.265
that group through the use of prescreening.

13:21.045 --> 13:23.665
You can also do some restrictions on technology, right?

13:23.735 --> 13:25.105
With, you know, things you can't,

13:25.245 --> 13:27.185
you stop them from opening a different browser

13:27.565 --> 13:29.785
or you tell them they have to take it on a computer,

13:29.965 --> 13:32.185
not on a phone, something like that.

13:33.485 --> 13:36.545
Um, but my recommendation is be very deliberate.

13:36.805 --> 13:38.665
Either embrace it, measure it,

13:39.285 --> 13:41.865
or intentionally restrict things.

13:43.385 --> 13:45.805
Now the final, source

13:45.805 --> 13:50.495
of noise I talk about in the paper is that you might have

13:51.385 --> 13:52.855
nonna participants.

13:55.065 --> 13:57.795
What I mean here is with a typical student,

13:58.785 --> 14:01.035
they may do a couple of studies a semester,

14:01.935 --> 14:03.915
and every study's kind of unique to them.

14:05.235 --> 14:08.605
With an m eker, they may do a couple of studies an hour.

14:09.715 --> 14:12.125
That means they've seen the prisoner's dilemma,

14:12.265 --> 14:14.845
the ultimatum game, the this, the, that.

14:15.155 --> 14:17.645
They know that researchers are trying to find racism.

14:18.155 --> 14:19.805
They know that researchers are trying

14:19.805 --> 14:21.285
to show this or that effect.

14:21.865 --> 14:24.205
And in fact, they may have participated in one

14:24.205 --> 14:26.525
of your own prior studies or papers.

14:28.105 --> 14:30.645
And that becomes a problem if they start

14:30.825 --> 14:33.325
to either have default responses

14:33.585 --> 14:35.165
and say, Nope, I always do this.

14:35.905 --> 14:38.125
Or if they start to guess your manipulations.

14:39.375 --> 14:42.075
Uh, just as an example, when I was an undergrad,

14:42.435 --> 14:46.075
I participated in a study where we saw a video of someone

14:46.175 --> 14:48.675
who was, supposed to be the CEO of a company,

14:49.495 --> 14:51.405
and we were supposed to see if we believed

14:51.405 --> 14:52.805
their statement or not.

14:54.115 --> 14:56.375
And the experimenter manipulated

14:56.645 --> 14:59.535
what university paraphernalia was in the background.

15:00.985 --> 15:02.725
And as soon as I saw the study

15:02.725 --> 15:03.765
and start participating,

15:04.525 --> 15:07.565
I saw this paraphernalia from our rival university,

15:08.025 --> 15:10.005
and I knew that was the manipulation.

15:11.145 --> 15:14.005
And so suddenly I wasn't really reacting.

15:14.325 --> 15:17.005
Normally, I was in the head

15:17.005 --> 15:18.605
of the experiment, or at least a little bit.

15:19.655 --> 15:22.915
Now, multiply that 10,000 fold for MTurk

15:23.865 --> 15:25.375
where they've seen these experiments

15:25.475 --> 15:26.935
and they know how you're working

15:26.935 --> 15:28.655
because they've done it over and over

15:28.975 --> 15:31.875
and over, it just makes it harder to

15:32.495 --> 15:34.355
really see what's going on in their heads

15:34.905 --> 15:37.235
because they're not acting naturally anymore.

15:38.965 --> 15:40.425
So what do I suggest?

15:40.905 --> 15:43.585
I suggest, first you try to use a novel paradigm.

15:44.345 --> 15:47.295
Don't just give them that same old economic game

15:47.325 --> 15:48.695
that everyone else uses.

15:49.695 --> 15:51.715
Do something unique, right?

15:51.775 --> 15:53.195
That's why in my experiments,

15:53.275 --> 15:56.115
I have them make betts on in-progress chess games,

15:56.335 --> 15:59.915
or try to develop a new NBA basketball team

16:00.575 --> 16:02.475
or something like that where

16:03.145 --> 16:04.595
they don't know what's going on.

16:04.595 --> 16:07.675
They're, this is a task they've never experienced before.

16:07.705 --> 16:09.635
They, it suddenly breaks 'em out of their

16:10.285 --> 16:11.715
MTurk shell a little bit,

16:12.135 --> 16:16.435
and they become kind of real people again, instead

16:16.435 --> 16:17.835
of get this done as fast

16:17.835 --> 16:19.435
as I can do the same thing I always do.

16:20.465 --> 16:23.605
Um, another thing you can do is, is keep track

16:23.605 --> 16:26.885
of who's participated in your own experiments in the past

16:27.785 --> 16:29.645
and try to screen them out.

16:29.735 --> 16:31.005
Don't let them participate.

16:31.005 --> 16:34.625
Again, that's pretty easy to do with MTurk qualifiers.

16:35.375 --> 16:37.385
Just, you know, make sure they're

16:37.385 --> 16:38.785
not repeating your studies.

16:40.675 --> 16:42.735
Uh, a final piece of advice on this one is

16:43.555 --> 16:46.095
be careful about your debriefing.

16:47.085 --> 16:50.665
So a debrief is when the experimenter goes in and

16:50.875 --> 16:55.345
after the experiment's over tells participants, Hey, just

16:55.345 --> 16:56.745
so you know, here's what we did

16:56.765 --> 16:58.985
and here's, you know, people saw different things.

17:00.705 --> 17:03.525
Now, sometimes that kind of debriefing is helpful

17:04.355 --> 17:06.325
because it can help you exclude people

17:06.325 --> 17:07.485
who maybe knew too much,

17:08.495 --> 17:11.275
but sometimes it can poison the pool for future researchers.

17:12.515 --> 17:14.255
Um, let me give an example here.

17:15.265 --> 17:17.265
I once heard a research presentation

17:17.915 --> 17:19.345
where they talked about their study,

17:19.365 --> 17:22.665
and then they said, well, at the end, early on,

17:22.665 --> 17:24.865
we told everyone we'd pay them based on their actual

17:24.865 --> 17:26.625
performance, but in the end,

17:26.765 --> 17:28.345
we paid everyone the exact same amount.

17:28.805 --> 17:30.185
And we told them, yeah,

17:30.315 --> 17:31.865
we're paying you all the maximum amount

17:31.865 --> 17:34.865
because that's what our institutional review

17:34.865 --> 17:35.985
board required us to do.

17:36.765 --> 17:39.185
So yeah, everyone's getting paid the same amount.

17:40.855 --> 17:43.115
No, suddenly those participants are going

17:43.355 --> 17:45.635
to distrust all future researchers.

17:46.605 --> 17:49.325
They think, this isn't, this doesn't matter.

17:49.425 --> 17:50.445
My decisions don't matter

17:50.445 --> 17:52.805
because you know you're gonna pay me

17:52.805 --> 17:53.925
the same amount no matter what.

17:55.065 --> 17:56.605
Be careful about doing that.

17:57.035 --> 17:58.805
Well, hopefully your IRB doesn't

17:58.805 --> 17:59.885
make you do that kind of thing.

18:00.505 --> 18:03.875
But if they do, just be a little bit considerate

18:03.875 --> 18:07.755
for the rest of us that at least in my experiments,

18:08.475 --> 18:10.835
I want them to go in thinking that I'm going

18:10.835 --> 18:13.675
to pay them based on how they perform because I do.

18:14.695 --> 18:16.875
And if they don't believe that statement,

18:17.895 --> 18:19.785
then the whole experiment's wasted

18:20.405 --> 18:23.425
effect sizes in behavioral research are often quite

18:23.515 --> 18:24.985
small, right?

18:25.035 --> 18:26.985
Where manipulating a couple of things,

18:27.495 --> 18:28.585
they're important things,

18:28.965 --> 18:32.585
but there's a lot of other things going on in people's head

18:34.545 --> 18:38.195
that becomes especially true in MTurk, where we have all

18:38.195 --> 18:40.515
of those extra uncontrolled things we just talked

18:40.515 --> 18:41.555
about in attention.

18:42.425 --> 18:45.275
Unqualified guessers ahead of genius population,

18:46.385 --> 18:48.075
that reduces your effect size

18:48.075 --> 18:49.515
because it's adding extra noise.

18:51.065 --> 18:54.075
So our effect sizes are often small.

18:54.375 --> 18:57.315
How do we deal with that? Well, usually

18:57.855 --> 19:00.555
in research we've been doing too small of sample sizes.

19:01.255 --> 19:04.915
The best way to detect something with a small effect size is

19:05.115 --> 19:06.835
to have a larger sample size.

19:07.485 --> 19:09.935
There's this old rule of thumb in research,

19:10.035 --> 19:13.535
you need 20 observations per experiment, manipulation

19:14.155 --> 19:18.615
or per cell that that's great for reducing type two error,

19:18.835 --> 19:21.535
but it is not enough to solve type one errors.

19:22.975 --> 19:26.635
So what I did in my meta-analysis, I just looked back at

19:27.275 --> 19:31.995
a whole bunch of recent, research in accounting,

19:32.055 --> 19:34.155
but I think it would generalize to other types

19:34.255 --> 19:35.515
of behavioral research

19:36.415 --> 19:39.275
and these studies that used Amazon Mechanical Turk.

19:39.275 --> 19:41.835
And I said, well, given what they found,

19:42.345 --> 19:44.275
what was the actual effect size?

19:45.175 --> 19:49.835
And if you had to design a new study kind of related,

19:50.815 --> 19:52.235
how many participants would you need

19:52.335 --> 19:53.875
to have a good chance of detecting this?

19:55.945 --> 19:58.245
The whole idea of this expected effect size

19:58.265 --> 20:02.205
and sample size is that you can calculate the number

20:02.205 --> 20:04.245
of participants you need to have such

20:04.245 --> 20:06.005
and such chance of finding an effect.

20:07.615 --> 20:11.115
But that's hard if we don't know our own effect sizes.

20:12.165 --> 20:15.225
So I, I broke this down in, in my meta-analysis

20:16.095 --> 20:19.105
into effect sizes based on different types of predictions,

20:19.825 --> 20:22.865
a main effect, a dis discoordinate interaction

20:23.005 --> 20:24.425
and an ordinal interaction.

20:25.755 --> 20:28.255
Uh, let me explain those three types of effects

20:28.395 --> 20:31.455
and then talk about my recommendations for sample sizes.

20:32.775 --> 20:36.215
A main effect is just a prediction that X causes y

20:36.235 --> 20:38.935
to increase, or X causes y to decrease.

20:39.515 --> 20:42.055
But there's no, nothing really complicated about it.

20:42.055 --> 20:44.135
It's just that one step.

20:45.285 --> 20:48.285
And my analysis says you probably need around 150,

20:48.285 --> 20:52.885
maybe 200 total MTurk participants to find that effect.

20:53.105 --> 20:56.245
If it's really there. That would tend to find most

20:56.305 --> 21:00.525
of the effects that, I found in my meta-analysis.

21:02.395 --> 21:05.555
A disorder interaction is when lines are expected

21:05.555 --> 21:07.155
to go in opposite directions.

21:07.935 --> 21:11.275
So X causes a positive effect in one situation

21:11.415 --> 21:13.555
and a negative effect in another situation.

21:14.655 --> 21:18.325
These interactions are really quite strong most of the time,

21:18.865 --> 21:20.645
so fewer participants are needed.

21:21.435 --> 21:23.335
Um, in my own research, I would aim for

21:23.335 --> 21:27.135
around a hundred total participants, if that's the kind

21:27.135 --> 21:29.055
of effect that I'm predicting to find.

21:30.795 --> 21:34.815
The final effect is called an in an ordinal interaction,

21:35.395 --> 21:38.055
and this is the least powerful type of interaction.

21:39.195 --> 21:42.765
This occurs when x causes a positive effect in both

21:42.815 --> 21:46.205
conditions, but it's more positive in one

21:46.205 --> 21:47.445
condition than another.

21:48.705 --> 21:51.525
In this situation, researchers typically need north

21:51.525 --> 21:55.005
of like 400 participants, maybe even 500

21:55.025 --> 21:58.325
or a thousand participants to have enough power to detect

21:58.325 --> 22:02.045
that effect, even if the effect is real and truly exists.

22:02.955 --> 22:04.965
It's just, it's really hard to find it.

22:05.105 --> 22:06.925
So you need to think kind of order

22:06.925 --> 22:10.365
of magnitude larger than maybe you were thinking with a,

22:10.925 --> 22:15.525
a lab experiment if you want to find that effect.

22:16.385 --> 22:21.205
My big conclusion here is there's a lot of value

22:21.385 --> 22:24.405
to Amazon Mechanical Turk when you're conducting

22:24.405 --> 22:28.405
experimental research, but it's not a magic bullet.

22:28.935 --> 22:30.525
There are a lot of sources of noise,

22:31.185 --> 22:33.845
and we can deal with those sources of noise

22:34.625 --> 22:36.765
by kinda using some of these different tools,

22:37.015 --> 22:41.045
leveraging pre-screening and attention check questions,

22:41.305 --> 22:44.445
and controlling for variables in a way that's going

22:44.445 --> 22:48.165
to make it, more effective, increase your chances

22:48.165 --> 22:49.485
of finding real work.

22:50.755 --> 22:53.775
Um, something about kinda future research.

22:54.115 --> 22:57.295
Uh, I've got some research going on right now that

22:57.995 --> 23:00.855
is looking at how Amazon Mechanical Turk has changed,

23:02.175 --> 23:05.195
and in particular, the participants that have changed.

23:06.135 --> 23:09.235
One thing I've noticed is that, we're getting more

23:09.375 --> 23:11.755
and more, both bots

23:12.695 --> 23:14.155
and people who I,

23:14.995 --> 23:16.755
I can't quite figure out what's going on with them.

23:17.375 --> 23:19.595
Uh, the best way I can describe them is kind

23:19.595 --> 23:21.035
of copy and pasting.

23:21.775 --> 23:24.915
Uh, it's where they're passing attention check questions,

23:25.845 --> 23:28.465
but then they're giving exact same answers

23:28.605 --> 23:31.785
to free text responses as some other participant.

23:32.675 --> 23:36.455
And so I had a study we ran last year where this happened,

23:36.835 --> 23:40.765
and we saw a large number of participants.

23:40.765 --> 23:44.765
We ran, over a thousand people and we saw

23:44.805 --> 23:47.205
A large number of participants giving the exact same

23:47.485 --> 23:51.005
verbatim word for word response as another participant.

23:52.105 --> 23:54.725
And they were in theory, in different parts of the world,

23:54.725 --> 23:56.045
different IP addresses,

23:56.785 --> 23:59.525
but these answers were totally, were definitely not random.

24:00.875 --> 24:04.055
Um, we still haven't figured out exactly why it's going on,

24:04.835 --> 24:06.335
but you should watch out for that.

24:06.335 --> 24:09.135
Then realize that even with all of these prescreening things

24:09.135 --> 24:11.695
that we've talked about, people are still going

24:11.695 --> 24:12.935
to try and beat the system.

24:13.395 --> 24:17.375
If you're paying $10 for an MTurk study, well that's a lot

24:17.375 --> 24:19.535
of money for five minutes of work or 10 minutes of work,

24:20.195 --> 24:21.335
and people are going to try

24:21.355 --> 24:23.495
and get around the controls putting into place.

24:24.235 --> 24:27.415
So watch out, treat everything a little bit with a grain

24:27.415 --> 24:31.895
of salt, and be careful to use MTurk,

24:31.895 --> 24:32.975
but use it appropriately

24:33.915 --> 24:36.655
and don't just assume everything's going to be easy from it.

00:00 - Anyone who has done behavioral research knows how difficult it is to get participants, right? We're recruiting from our own students or from alumni or people in the the neighborhoods, something like that. And it's just a hassle, especially if you're trying to run multiple studies within a given semester, right?

00:20 - If I'm using my own students, I've got maybe a hundred students a semester that I can use, and I can't do any follow-up experiments because they already did the first experiment. So it's a challenge. And income's, mechanical Turk, mechanical turk isn't really meant for research. It's meant for any type of, task that can't really be automated well by it. And it's meant to outsource that as kind

00:48 - of little micro jobs. So you could think of it kind of like the, the Uber of any type of computer task, right? Where we get someone to transcribe something or you get someone to go look up some information on the internet or, something like that. And it's great for research in that it provides this huge sample of participants that you can use to do your studies. so that's kinda the background on Amazon Mechanical Turk

01:19 - and why a lot of people find it really useful. There had been a lot of research studies that I had seen, especially when I was a PhD student, kind of touting the glory of Mechanical Turk and saying, rah, ru haha, it's wonderful. Everyone should use it because it's better than students. It's better than this. It's, it works in every way. And when I started doing some research, I went, you know, it, it's great. It is very helpful, but it has some drawbacks.

01:47 - It's not the magic bullet. And so the, the idea behind this paper was, you know, I ran into a lot of challenges when I started doing this research. Maybe I could walk some people through, some of those, road bumps I ran into and see if I can help them avoid them. So statistical power refers to how likely you are to find a result that is true, right? So if you know, say you're trying to find,

02:18 - are men taller than women? Well, we know that on average that's true. How likely are you to find that with a different sample size, right? If you just take two random people, one man, one woman, it may be true, it may not be true, but if you take a hundred or 200 of each, then you're gonna find that statistically. So that's referring as kind of type one error, the chance that something is true, but you fail to find it.

02:43 - Statistically, reliability is kind of the opposite side of that. It's the chance that you will find a statistically significant result totally by chance. So it's an error. Your significance isn't reflective of the real world, right? That's the chance that you pair up two people and you find happen to get a woman that's taller than a man. Sure, that happens all the time. That doesn't mean it's generalizable.

03:11 - So research has typically focused on the type two error. That's the one that reviewers and editors are generally more concerned about. Type two error is the stuff that then can border on fraud or misconduct or just wrong inferences and people making real world decisions based on something that's not true. So it's important to reduce type two error, but you know, as researchers, we also should care about the type one error,

03:44 - because I don't want to run a study that doesn't have any chance of finding results. I don't want to, if I were doing that man versus woman study, I wouldn't want to run it with too small of a sample size and go, I guess men aren't taller than women, simply because I designed my study wrong, or I included the wrong control variables, or didn't measure certain things in the right way. So while my paper touches a little bit on the type two

04:13 - error, that reliability, it's a lot more focused on power, reducing that type one error and making it so researchers can be more successful and have a better chance of finding results that are really there. And they're important to find. The biggest reason why you might not find an effect is that your intended effect isn't the only thing mattering in the real world, right? So gender is not the only thing

04:45 - that determines someone's height. Yes, it matters, but there are a whole lot of other things out there, and noise is kind of our way of defining, broadly speaking, all of those other factors. Now, experimentalists are used to dealing with noise, but in my paper, I outline a couple of sources of noise that you face with MTurk that you wouldn't face in a more traditional lab environment like using, you know, student population.

05:18 - And I try and discuss those five sources of noise and present ideas of how to deal with them or reduce that influence of that noise in your research. the first source of noise is unqualified participants. And the basic idea here is you might not have, you might have people who are participating who aren't really the people you want. In fact, they might not even be people at all, right? They might be bots, and you don't want to be doing some behavioral study thinking.

05:50 - You've got humans responding and later finding out that you were actually interfacing with chat GPT, right? That that would be a bad thing. Or even if they are real people, are they the people you want? Are they even fluent in the language you're administering the experiments on? Or are they kind of randomly guessing? Do they have the required backgrounds?

06:11 - In my research, I often want people who are managers, or at least who've been in a full-time job where they've got someone looking over their shoulder and kind of giving them a bonus of some kind. If I had a 16-year-old doing the experiment, it just, I don't think they would understand enough about the real world to really react. And then not just do they have the background? Do they have the knowledge they need? so for example, in my dissertation, I needed people

06:41 - who knew how to play chess. And if I had someone who didn't really know that wouldn't work, I've got some other papers where I need people who are really avid basketball fans, and I needed to make sure they had that just so my experiment would have a chance of working the way I wanted. So how do you deal with this? How do you make sure you get the right people in your study? Well, the solution I suggest in the paper is prescreening.

07:08 - That's where you do some types of checks before you actually run your experiment to make sure you're getting the people you want. Now, it's important when you do these checks that you're not just asking them things like, Hey, are you good at chess? Or do you like basketball? Right? That's, that's not gonna work because people lie, right? If you tell me, Hey, I need an expert in chess, I'll give you a million dollars if you're an expert in

07:36 - chess, I'll say, yeah, I'm an expert. It's just, it's kind of natural. And so if you just ask that self-report, you can end up screening in liars instead of screening in the people you actually want. So what I suggest is that you use knowledge or ability checks instead, right? You may be able to claim that you are a regular stock market trader and you're a financial expert, but it's a completely different thing to be able

08:09 - to compute financial ratios on the fly or to be able to analyze an in-progress chess game. Now, it really helps that these kinda screening questions you're asking are something hard, that you can't really Google the answer to the things that people need to know. And Intuit, that's the way you can tell if they're really qualified. Now, you do need to watch out for lucky guesses, right? You wanna think, well, what percent

08:36 - of people could pass this question just by guessing? but those are the types of questions you want. the second, source of noise I discuss is a lack of attention. This is when you have people who are maybe just distracted or in a rush, right? So maybe, you know, they're, they're working from home and they're watching TV or they're, you know, kind of doing something else. They're, they're sitting on a computer doing another

09:07 - full-time job and kind of on the side doing this MTurk task. We don't know who these people are and and how much attention they're paying. So the solution I propose here is you try and measure their attention. You used some, check questions as you're going along, making sure they understood the important material you're giving them, right? So if I'm setting up some incentive in a certain way, or I'm making them be the employee versus the supervisor,

09:37 - I ask them, Hey, you know, what's your role in this? Or How are you going to behave? What's your, your job in this task? And make sure they're understanding the experiment as they go along. And if they fail, you kick 'em out. You don't have 'em participate so that you make sure you're getting the good people in. Now, even that situation, you may end up with some people who aren't that great, who kind of guessed

10:03 - or who have paid attention. You may have to drop them after the fact. but at least you're trying and you can deal with that. I also suggest using some, different ways and being very transparent in how you choose to exclude participants. Make sure that everyone knows what your design choices were so you're not being opportunistic in this. the third source of noise I talk about is a lack

10:34 - of experimental control, which I'll include in that you have a, a heterogeneous population. When we did research, maybe 10 or 20 years ago, we were using students, and students are sometimes, colloquially called the weird population. They're western, educated, industrialized rich, and demographic, sorry, democratic. So they are taking this very narrow subset of the population.

11:09 - Now, that can be good and bad. It's good because it means you don't have all this extra source of noise, you don't have for purposes of your experiment baggage, right? You don't have people having substantially different upbringings and backgrounds, that that means that you have greater power. It's bad because maybe you can't generalize your results as well.

11:35 - But then when you move to the MTurk world, you've completely flipped that. Suddenly you have people from all over the world with all different upbringings and all different backgrounds, and that's really good for generalizing research, but it's bad in that it means They have their own ideas that they're bringing in. They're not all rich and educated. They're not all 20 to 23-year-old students.

12:03 - And that difference of backgrounds means that they're going to respond a little bit differently and add some more variation to your analysis, which just reduces your power. They may also be using different technology, right? They could be Googling answers, they could be using auto clicking software, and you can prevent them from doing that in a lab. You can't on MTurk. So what do you do? Well, you kinda have two options.

12:32 - Option one is you embrace the differences. You try and measure all of the different aspects, right? Ask 'em their age, ask 'em their education, ask 'em their their income. Ask 'em where they live, all of these things that might matter in the real world. Measure them, make them covariates and include them in part of your analysis. Learn from all of that information. Option two is say, I don't want that.

12:59 - that's not the point of my paper. And you try and restrict people, right? Maybe you say, no, I I only want to do research on people located in this geographical area who speak this language, who are in this age group, or who are, you know, this, this, this, this, tighten that group through the use of prescreening. You can also do some restrictions on technology, right? With, you know, things you can't, you stop them from opening a different browser

13:27 - or you tell them they have to take it on a computer, not on a phone, something like that. but my recommendation is be very deliberate. Either embrace it, measure it, or intentionally restrict things. Now the final, source of noise I talk about in the paper is that you might have nonna participants. What I mean here is with a typical student, they may do a couple of studies a semester,

14:01 - and every study's kind of unique to them. With an m eker, they may do a couple of studies an hour. That means they've seen the prisoner's dilemma, the ultimatum game, the this, the, that. They know that researchers are trying to find racism. They know that researchers are trying to show this or that effect. And in fact, they may have participated in one of your own prior studies or papers. And that becomes a problem if they start

14:30 - to either have default responses and say, Nope, I always do this. Or if they start to guess your manipulations. just as an example, when I was an undergrad, I participated in a study where we saw a video of someone who was, supposed to be the CEO of a company, and we were supposed to see if we believed their statement or not. And the experimenter manipulated what university paraphernalia was in the background.

15:00 - And as soon as I saw the study and start participating, I saw this paraphernalia from our rival university, and I knew that was the manipulation. And so suddenly I wasn't really reacting. Normally, I was in the head of the experiment, or at least a little bit. Now, multiply that 10,000 fold for MTurk where they've seen these experiments and they know how you're working

15:26 - because they've done it over and over and over, it just makes it harder to really see what's going on in their heads because they're not acting naturally anymore. So what do I suggest? I suggest, first you try to use a novel paradigm. Don't just give them that same old economic game that everyone else uses. Do something unique, right? That's why in my experiments,

15:53 - I have them make betts on in-progress chess games, or try to develop a new NBA basketball team or something like that where they don't know what's going on. They're, this is a task they've never experienced before. They, it suddenly breaks 'em out of their MTurk shell a little bit, and they become kind of real people again, instead of get this done as fast as I can do the same thing I always do.

16:20 - another thing you can do is, is keep track of who's participated in your own experiments in the past and try to screen them out. Don't let them participate. Again, that's pretty easy to do with MTurk qualifiers. Just, you know, make sure they're not repeating your studies. a final piece of advice on this one is be careful about your debriefing. So a debrief is when the experimenter goes in and

16:50 - after the experiment's over tells participants, Hey, just so you know, here's what we did and here's, you know, people saw different things. Now, sometimes that kind of debriefing is helpful because it can help you exclude people who maybe knew too much, but sometimes it can poison the pool for future researchers. let me give an example here. I once heard a research presentation where they talked about their study,

17:19 - and then they said, well, at the end, early on, we told everyone we'd pay them based on their actual performance, but in the end, we paid everyone the exact same amount. And we told them, yeah, we're paying you all the maximum amount because that's what our institutional review board required us to do. So yeah, everyone's getting paid the same amount. No, suddenly those participants are going

17:43 - to distrust all future researchers. They think, this isn't, this doesn't matter. My decisions don't matter because you know you're gonna pay me the same amount no matter what. Be careful about doing that. Well, hopefully your IRB doesn't make you do that kind of thing. But if they do, just be a little bit considerate for the rest of us that at least in my experiments,

18:08 - I want them to go in thinking that I'm going to pay them based on how they perform because I do. And if they don't believe that statement, then the whole experiment's wasted effect sizes in behavioral research are often quite small, right? Where manipulating a couple of things, they're important things, but there's a lot of other things going on in people's head that becomes especially true in MTurk, where we have all

18:38 - of those extra uncontrolled things we just talked about in attention. Unqualified guessers ahead of genius population, that reduces your effect size because it's adding extra noise. So our effect sizes are often small. How do we deal with that? Well, usually in research we've been doing too small of sample sizes. The best way to detect something with a small effect size is to have a larger sample size.

19:07 - There's this old rule of thumb in research, you need 20 observations per experiment, manipulation or per cell that that's great for reducing type two error, but it is not enough to solve type one errors. So what I did in my meta-analysis, I just looked back at a whole bunch of recent, research in accounting, but I think it would generalize to other types of behavioral research and these studies that used Amazon Mechanical Turk. And I said, well, given what they found,

19:42 - what was the actual effect size? And if you had to design a new study kind of related, how many participants would you need to have a good chance of detecting this? The whole idea of this expected effect size and sample size is that you can calculate the number of participants you need to have such and such chance of finding an effect. But that's hard if we don't know our own effect sizes. So I, I broke this down in, in my meta-analysis

20:16 - into effect sizes based on different types of predictions, a main effect, a dis discoordinate interaction and an ordinal interaction. let me explain those three types of effects and then talk about my recommendations for sample sizes. A main effect is just a prediction that X causes y to increase, or X causes y to decrease. But there's no, nothing really complicated about it. It's just that one step. And my analysis says you probably need around 150,

20:48 - maybe 200 total MTurk participants to find that effect. If it's really there. That would tend to find most of the effects that, I found in my meta-analysis. A disorder interaction is when lines are expected to go in opposite directions. So X causes a positive effect in one situation and a negative effect in another situation. These interactions are really quite strong most of the time, so fewer participants are needed. in my own research, I would aim for

21:23 - around a hundred total participants, if that's the kind of effect that I'm predicting to find. The final effect is called an in an ordinal interaction, and this is the least powerful type of interaction. This occurs when x causes a positive effect in both conditions, but it's more positive in one condition than another. In this situation, researchers typically need north of like 400 participants, maybe even 500 or a thousand participants to have enough power to detect

21:58 - that effect, even if the effect is real and truly exists. It's just, it's really hard to find it. So you need to think kind of order of magnitude larger than maybe you were thinking with a, a lab experiment if you want to find that effect. My big conclusion here is there's a lot of value to Amazon Mechanical Turk when you're conducting experimental research, but it's not a magic bullet. There are a lot of sources of noise, and we can deal with those sources of noise

22:34 - by kinda using some of these different tools, leveraging pre-screening and attention check questions, and controlling for variables in a way that's going to make it, more effective, increase your chances of finding real work. something about kinda future research. I've got some research going on right now that is looking at how Amazon Mechanical Turk has changed, and in particular, the participants that have changed. One thing I've noticed is that, we're getting more

23:09 - and more, both bots and people who I, I can't quite figure out what's going on with them. the best way I can describe them is kind of copy and pasting. it's where they're passing attention check questions, but then they're giving exact same answers to free text responses as some other participant. And so I had a study we ran last year where this happened, and we saw a large number of participants.

23:40 - We ran, over a thousand people and we saw A large number of participants giving the exact same verbatim word for word response as another participant. And they were in theory, in different parts of the world, different IP addresses, but these answers were totally, were definitely not random. we still haven't figured out exactly why it's going on, but you should watch out for that. Then realize that even with all of these prescreening things that we've talked about, people are still going

24:11 - to try and beat the system. If you're paying $10 for an MTurk study, well that's a lot of money for five minutes of work or 10 minutes of work, and people are going to try and get around the controls putting into place. So watch out, treat everything a little bit with a grain of salt, and be careful to use MTurk, but use it appropriately and don't just assume everything's going to be easy from it.

Faculti AI can make mistakes