Popular in Social Sciences

Books

City Time: On Being Sentenced to Rikers Island

Independent Scholar David A. Campbell

Culture and Society

The City as a Growth Machine: Toward a Political Economy of Place

New York University Harvey Moltoch

Economics

Gender and Economics

University of California Santa Barbara Shelly Lundberg

American Politics

Keeping the Republic

Boston College Marc Landy

Banks and Policy Organisations

Greening the financial sector

European Investment Bank Matteo Ferrazzi

British Politics

Entrepreneurial thinking: the politics and practice of policy impact

University of Sheffield Matthew Flinders

Culture and Society

Carceral Entrapments: Views from the Prison/Street Interface in India

Indian Institute of Technology Delhi Mahuya Bandyopadhya

Enumerator Experiences in Violent Research Environments

View more insights in Social Sciences - Political Science

View more figures from UC Berkeley

View the page of Martha Wilfahrt

Read the Study

Martha Wilfahrt and Justine Davis discuss how survey enumerators in violent research settings face unique security concerns that affect data collection. Personal insecurity and exposure to violence influence their challenges and compromises.

Image courtesy of interviewee. December 14, 2023

Generating Workspace

Export as PDF

This video video discusses the rise in academic surveys and field experiments conducted in the Global South, emphasizing the important role of survey enumerators in the process. These enumerators, who often have a university education, typically work as gig workers, contributing their skills to collect data for various research projects. They take pride in their work, valuing the professional development and opportunities for learning about their countries.

The authors highlight a lack of understanding regarding the experiences and working conditions of enumerators, especially in insecure contexts marked by violence or conflict. Risks faced by enumerators can impact data quality due to reported instances of violence, threats, and specific challenges in engaging with respondents. This video video study aims to shed light on this issue and emphasize the ethical considerations of placing enumerators in potentially dangerous situations.

Significant findings include the high percentage of enumerators feeling unsafe or fearing for their physical safety while conducting interviews. The study reveals challenges encountered, including violent encounters and difficulties ensuring respondents' cooperation. The authors advocate for better engagement with enumerators in the research design process, including considerations for their safety and mental health, as well as adapting survey questions to local contexts.

Lastly, the paper offers several recommendations for researchers to improve enumerator welfare and enhance data validity, such as establishing disciplinary norms for research in insecure areas, adapting research designs to minimize enumerator risk, and collecting feedback on enumerator experiences before and after data collection. By addressing these issues, the authors hope to foster a more ethical and effective research environment.

Language:

Export as PDF

Export as PDF
Here are some key quotes drawn from the video you :

1. **Growing Research in the Global South**:
- "The past decade or two has seen a sharp uptick in the number of academic surveys and field experiments being conducted in the global south."

2. **The Role of Survey Enumerators**:
- "These survey enumerators... are very proud of the work that they do."
- "They report really valuing the professional skills that they can develop in the course of doing academic research."

3. **Navigating Insecure Convideos**:
- "We don’t actually know a lot about the research experience... of these people who are doing a large number of surveys every day."
- "We think that these concerns would be particularly acute because we're just further from the data collection process."

4. **Survey Enumerator Challenges**:
- "Almost 60% of the enumerators in our study felt unsafe, at least sometimes while conducting academic surveys."
- "Over 75% reported fearing for their physical safety."

5. **Experience of Violence**:
- "About a third had been followed," and "26% had been threatened with physical violence."
- "Enumerators also witnessed violence while on the job, including violent protests, threats, harassment and physical violence."

6. **Impact on Data Quality**:
- "When enumerators face insecurity, they're more likely to overcome challenges by adapting or breaking survey protocols."
- "These breaches hold potentially serious consequences for the quality of the data we collect."

7. **Recommendations for Improvement**:
- "We urge researchers to continue efforts towards establishing disciplinary norms around research in insecure settings."
- "Understanding the working conditions can help alleviate some of these risks."

8. **Emphasis on Collaboration**:
- "Including enumerators in all aspects of the research design to make sure that we're asking culturally sensitive questions that are adapted to local realities."

These quotes encapsulate the primary themes of the video, highlighting the experiences of survey enumerators in insecure convideos and the implications for both ethical considerations and data quality in research.

Export as PDF

X-TIMESTAMP-MAP:LOCAL=00:00:00.000,MPEGTS=900000

00:00.025 --> 00:02.685
So the past decade or two has seen a sharp uptick in the

00:02.685 --> 00:04.405
number of academic surveys

00:04.405 --> 00:06.725
and field experiments being conducted in the global south.

00:07.145 --> 00:08.405
And one consequence of this,

00:08.405 --> 00:10.565
or perhaps a driver of it, is that there's a growing number

00:10.625 --> 00:12.765
of firms, many of which are very high quality,

00:12.835 --> 00:14.525
that are available to do this kind

00:14.525 --> 00:15.645
of research for researchers.

00:16.185 --> 00:18.445
So based on our data, or sorry,

00:18.445 --> 00:20.565
and I should say, so those firms then need to hire people

00:20.625 --> 00:22.965
to go out in the field and, you know,

00:22.995 --> 00:24.285
collect our surveys.

00:24.285 --> 00:25.725
And so those are survey enumerators.

00:26.185 --> 00:28.925
So based on our data, and anecdotally from other research

00:28.925 --> 00:30.845
that Jesse and I have both done in West Africa,

00:31.055 --> 00:32.845
these survey animators are generally,

00:32.865 --> 00:34.005
you know, pretty well educated.

00:34.005 --> 00:36.565
They're often completing university or finished university,

00:36.865 --> 00:38.365
and they're essentially doing gig work,

00:38.365 --> 00:39.805
which means they might take a contract

00:39.805 --> 00:41.405
to run an academic survey one week,

00:41.625 --> 00:43.885
and then a few weeks later they work on the national

00:43.885 --> 00:45.085
census or something like this.

00:45.505 --> 00:47.365
Um, and so we wanna stress at the outset

00:47.635 --> 00:49.565
that these survey enumerators,

00:49.745 --> 00:51.365
are very proud of the work that they do.

00:51.425 --> 00:53.685
And so in both close and open-ended questions,

00:54.035 --> 00:56.445
they report really valuing the professional skills

00:56.645 --> 00:57.845
that they can develop in the course

00:57.865 --> 00:59.365
of doing academic research.

00:59.755 --> 01:01.085
They like to be able to travel

01:01.145 --> 01:02.405
and learn about their countries, which

01:02.405 --> 01:04.325
otherwise might not always be options for a lot

01:04.325 --> 01:06.765
of these people that like participate in the research

01:06.915 --> 01:09.405
process and they hope that they can help in some way.

01:09.405 --> 01:10.565
They're co citizens in the process

01:10.585 --> 01:11.605
of doing academic research.

01:11.985 --> 01:13.685
So these are people who are hired to do their job

01:14.105 --> 01:16.365
and are trying to do the best of their, ability.

01:17.065 --> 01:18.725
But what we've realized in sort

01:18.725 --> 01:20.925
of thinking about our own research experiences is

01:20.925 --> 01:23.285
that we don't actually know a lot about the research,

01:23.745 --> 01:25.605
you know, experience or sort of their actual sort

01:25.605 --> 01:27.005
of work experience of these people

01:27.065 --> 01:29.805
who are doing a large number of surveys every day

01:29.865 --> 01:31.965
for weeks on end to sort of, you know, produce data

01:31.965 --> 01:33.685
that arrives to us in our inboxes.

01:34.105 --> 01:36.245
And so we really wanted to think about this in particular,

01:36.245 --> 01:38.885
in the case of insecure context, by which we mean countries

01:38.885 --> 01:40.365
that might have an ongoing conflict.

01:40.515 --> 01:43.405
There's a growing discussion about research ethics in

01:43.405 --> 01:45.045
violent context, in political science

01:45.065 --> 01:47.285
and adjacent disciplines, but also

01:47.285 --> 01:48.965
because we think that this might impact the data

01:48.965 --> 01:51.165
that we receive in ways that we would wanna understand

01:51.185 --> 01:53.285
as researchers if there's systematic sources

01:53.305 --> 01:54.765
of biases in one way or the other.

01:56.185 --> 01:58.445
So to the extent that research enumerators

01:58.465 --> 02:00.565
or survey enumerators go out into communities, they have

02:00.565 --> 02:02.645
to knock on doors, they have to attempt to sort

02:02.645 --> 02:04.485
of get respondents to agree to participate.

02:04.715 --> 02:06.885
They have to walk them through very lengthy

02:06.885 --> 02:09.965
and very legalistic consent forms that might be very foreign

02:10.265 --> 02:12.845
to a lot of, you know, perhaps not particular, you know,

02:12.845 --> 02:14.485
people without high literacy skills.

02:14.795 --> 02:16.405
They have to sit with those people for some length

02:16.405 --> 02:17.605
of time asking them

02:17.635 --> 02:19.685
what sometimes are pretty sensitive questions.

02:20.185 --> 02:21.565
Um, to that extent, we think

02:21.565 --> 02:23.765
that these numerator are engaged in really important acts

02:23.785 --> 02:26.045
of brokerage on our behalf as researchers.

02:26.585 --> 02:28.325
So there's been some sustained attention

02:28.425 --> 02:31.045
to thinking about research assistance as brokers.

02:31.115 --> 02:32.805
It's largely been done by anthropologists

02:32.805 --> 02:35.325
or demographers people who tend to spend a fair amount

02:35.325 --> 02:37.605
of time in the field and tend to work with one

02:37.625 --> 02:39.005
or a small number of brokers.

02:39.785 --> 02:41.445
And this work has tended to emphasize sort

02:41.445 --> 02:43.645
of the unequal relationship between researchers

02:43.825 --> 02:44.885
and research brokers.

02:45.665 --> 02:47.085
Um, and that these are, you know,

02:47.085 --> 02:49.165
this work is also really thought about the ways in which,

02:49.345 --> 02:52.165
how that in inequity is acute in conflict settings,

02:52.165 --> 02:54.125
where brokers not only are providing us with data,

02:54.425 --> 02:55.685
but often are helping sort

02:55.685 --> 02:57.365
of navigate security concerns like

02:57.365 --> 02:59.005
roadblocks for researchers.

02:59.625 --> 03:01.565
So given this uptick in remote research

03:01.585 --> 03:03.885
and the ease with which we can now do, you know,

03:04.025 --> 03:06.565
big surveys in the field, well, you know, you know,

03:06.765 --> 03:08.805
researchers often based in the global north sit at

03:08.805 --> 03:10.285
home, you know, in our offices.

03:11.105 --> 03:13.845
Um, we thought sort of, we wanna sort of suggest that sort

03:13.845 --> 03:16.125
of, we bring some of these insights from this work on

03:16.405 --> 03:19.525
research brokerage from adjacent disciplines to think about

03:19.525 --> 03:21.525
what the implications are for survey research.

03:21.785 --> 03:24.205
And we think that these concerns would be particularly acute

03:24.395 --> 03:27.085
because we're just further from the data collection process

03:27.355 --> 03:28.405
when we work with a firm

03:28.465 --> 03:30.765
or sort of employ a large number of survey enumerators,

03:30.765 --> 03:32.565
even if we hire those enumerators ourselves.

03:33.425 --> 03:35.725
And so when we have a survey, there's just more people,

03:35.865 --> 03:37.365
it tends to be more decentralized.

03:37.545 --> 03:39.925
And so you have a bunch of enumerators working under a team

03:39.925 --> 03:41.805
leader working, you know, hired by a firm.

03:41.865 --> 03:43.965
And we might be able to communicate with the firm, maybe

03:43.965 --> 03:46.165
with the team leader, but we don't necessarily know

03:46.165 --> 03:47.885
what information is being filtered up to us.

03:47.905 --> 03:50.165
And so we wanted to understand sort of, if we talk

03:50.165 --> 03:51.365
to enumerators themselves, right,

03:51.365 --> 03:52.845
what are their working conditions?

03:53.355 --> 03:56.005
They are really important to the research effort.

03:56.545 --> 03:59.205
But often in things like IRB,

03:59.505 --> 04:01.405
we aren't asked questions about how we're going

04:01.405 --> 04:03.605
to protect their security,

04:03.705 --> 04:06.645
or ensure that they themselves don't face, you know,

04:06.885 --> 04:08.845
dangerous context, despite the fact

04:08.845 --> 04:10.085
that they are the ones on the ground

04:10.085 --> 04:11.445
collecting the data for us.

04:11.985 --> 04:14.125
Um, and so what that was another motivation

04:14.225 --> 04:17.085
for pursuing this is that we often just kind of think

04:17.085 --> 04:18.965
of them as neutral actors

04:18.995 --> 04:20.885
that are just out there doing this work for us.

04:21.225 --> 04:22.765
Um, but in reality they have

04:22.765 --> 04:25.405
to navigate these contexts just like the respondents

04:25.405 --> 04:27.085
that are answering our surveys,

04:27.345 --> 04:28.965
do in in insecure settings

04:28.965 --> 04:31.525
that can include violence witnessing violent violence,

04:31.525 --> 04:33.045
experiencing violence themselves.

04:33.825 --> 04:36.165
We certainly have paid attention to survey enumerators

04:36.165 --> 04:37.605
before, but the vast majority

04:37.605 --> 04:39.325
of work is focused on sort of two things.

04:39.745 --> 04:42.685
So first we know a growing amount about the enumerator

04:42.685 --> 04:43.765
respondent dyad.

04:44.025 --> 04:46.845
So most work here has focused on how characteristics

04:47.025 --> 04:49.325
of the enumerators might elicit certain types

04:49.325 --> 04:51.405
of responses by respondents.

04:51.665 --> 04:53.325
So there's been work by CLA data

04:53.545 --> 04:56.525
and a number of co-authors on how ethnicity a respondent

04:56.525 --> 04:58.885
or co enumerators ethnicity might cue certain types

04:58.885 --> 05:00.765
of respondents from either co-ethnic

05:00.765 --> 05:03.005
or non co-ethnic survey respondents.

05:03.195 --> 05:05.405
There's been work on gender, there's been work on sort

05:05.405 --> 05:07.965
of religious dress by Lindsay Bedstead and others.

05:08.265 --> 05:11.045
Um, and it's all been shown to influence sort of

05:11.105 --> 05:12.125
how an enumerator,

05:12.125 --> 05:14.765
something about the enumerators characteristics are shaping

05:14.825 --> 05:16.605
the responses they get from respondents.

05:17.205 --> 05:18.405
Secondly, there's been some work

05:18.405 --> 05:20.685
that I think is encapsulated well in a,

05:20.685 --> 05:23.085
piece in the annual review of political science by no Lupa

05:23.305 --> 05:25.845
and Kristen Michel that sort of has thought about

05:25.845 --> 05:28.085
how enumerators own behavior might

05:28.085 --> 05:29.245
shape the data generating process.

05:29.345 --> 05:33.965
And in particular about concerns over, fabricated data.

05:34.265 --> 05:36.165
But there's also been work that's just thought about

05:36.185 --> 05:38.445
how enumerators might accelerate surveys.

05:38.445 --> 05:40.765
They might develop priors that are gonna influence

05:40.825 --> 05:42.685
how they approach certain questions, et cetera.

05:42.985 --> 05:45.245
And so the biggest concern here, of course, is really about

05:45.245 --> 05:47.285
how enumerators themselves are doing something

05:47.305 --> 05:48.765
to impact data quality.

05:49.465 --> 05:52.005
Um, so both of these literatures have distinct implications.

05:52.105 --> 05:54.245
So if it's the former, if we think it's something about

05:54.245 --> 05:56.925
enumerator characteristics that are influencing responses,

05:57.235 --> 05:59.525
then something like including enumerator fixed effects

05:59.625 --> 06:02.365
or a dummy variable for an enumerator agenda in our,

06:02.365 --> 06:05.045
you know, models that we might run could maybe address that

06:05.745 --> 06:07.805
if we think it's something about that enumerators are doing,

06:07.835 --> 06:09.565
like, perhaps more related

06:09.565 --> 06:11.205
to like their professionalism, let's say.

06:11.205 --> 06:12.925
Right? Then, you know, there's also techniques.

06:12.925 --> 06:16.285
So increasingly surveys are run on tablets,

06:16.505 --> 06:17.965
and so we might be able to sort of see

06:17.985 --> 06:19.325
how long a survey took, right?

06:19.325 --> 06:20.605
And a survey didn't take long enough.

06:20.625 --> 06:23.005
We might have questions about the quality of the survey,

06:23.465 --> 06:25.725
or sometimes we can geolocate that so we know whether

06:25.725 --> 06:27.565
or not a survey enumerator is actually going

06:27.565 --> 06:29.605
to different households or just sitting in one spot filling

06:29.605 --> 06:30.605
out surveys at the end of the day.

06:31.385 --> 06:34.365
Um, and of course we can also think about other techniques

06:34.425 --> 06:36.165
to sort of, you know, design questions

06:36.165 --> 06:38.045
that are particularly clever to maybe address some

06:38.045 --> 06:39.045
of these concerns, et cetera.

06:39.465 --> 06:42.605
So that literature all exists and we think it's important,

06:42.665 --> 06:44.805
but we don't think it's actually capturing sort

06:44.805 --> 06:47.245
of the political context that sort

06:47.245 --> 06:48.765
of enumerators are working in, which we think is sort

06:48.765 --> 06:50.965
of a distinct driver of,

06:51.115 --> 06:53.645
both enumerator work experiences that we care about

06:53.645 --> 06:55.965
for normative reasons, but also data quality reasons

06:56.035 --> 06:57.645
that we'll talk about more in a few minutes.

06:58.145 --> 07:01.485
But we thought about, our own experiences

07:01.545 --> 07:04.845
and I've, I've had a lot of experience working in TiVo,

07:04.915 --> 07:06.165
with Enumerators there.

07:06.465 --> 07:09.565
And so what we did, is we recruited a diverse set

07:09.585 --> 07:12.085
of roughly 300,

07:12.085 --> 07:15.365
enumerators from the top three research firms in co dvo.

07:16.025 --> 07:19.365
Um, the reason that co dvo is actually really interesting,

07:19.745 --> 07:21.165
is several fold.

07:21.225 --> 07:24.405
So first, it is a site of,

07:24.555 --> 07:26.005
surveys and academic research.

07:26.265 --> 07:30.245
Um, there's a number of publications, that have,

07:30.605 --> 07:31.805
included data from TiVo.

07:32.265 --> 07:35.245
Um, the Afro Barometer has been conducted there several

07:35.305 --> 07:39.165
rounds, but crucially, the political context is

07:39.165 --> 07:40.885
what we are particularly interested in.

07:40.945 --> 07:43.285
Um, and TiVo unfortunately,

07:43.285 --> 07:46.845
experienced the civil war from 2002 to 2011,

07:47.105 --> 07:50.365
and has experienced a, a high level of,

07:50.645 --> 07:53.165
election related violence, ethnic violence.

07:53.665 --> 07:56.445
Um, and so we see a high level of insecurity,

07:56.545 --> 07:59.245
for people living in the country as well as enumerators

07:59.305 --> 08:00.645
who are conducting this research.

08:01.225 --> 08:05.045
Um, and so the goal that we had then was to get,

08:05.385 --> 08:07.645
an understanding of these enumerator experiences.

08:07.785 --> 08:09.805
So we recruited them from these top three firms.

08:10.305 --> 08:12.485
Um, they ultimately represented all

08:12.485 --> 08:14.445
of the ethnic groups in co TiVo,

08:15.225 --> 08:17.285
and had substantial experience in the sector.

08:17.825 --> 08:20.525
Um, as Martha said, these folks tend

08:20.525 --> 08:23.605
to be more educated than the general population,

08:23.825 --> 08:26.645
and to have like master's degrees and that sort,

08:26.785 --> 08:29.005
and they tend to come from more urban settings,

08:29.285 --> 08:30.285
although, one

08:30.285 --> 08:32.765
of the nice things about our recruitment strategy is

08:32.765 --> 08:35.165
that some of these survey firms like to recruit locally,

08:35.305 --> 08:37.725
and so many of the enumerators,

08:37.875 --> 08:39.765
come from sometimes even rural villages.

08:40.185 --> 08:41.885
Um, and so what we asked them to do was

08:41.885 --> 08:44.925
to complete an online survey about the sum

08:44.985 --> 08:46.725
of their enumerator experiences.

08:47.225 --> 08:50.685
Um, we asked them about, just general, you know,

08:50.685 --> 08:53.005
where have you worked, what did you do,

08:53.145 --> 08:55.550
the challenges they faced while implementing projects,

08:55.550 --> 08:57.445
the challenges they encountered as a function

08:57.445 --> 08:59.245
of the survey content,

08:59.265 --> 09:01.445
and then solutions they may have employed when they

09:01.445 --> 09:02.485
ran into these challenges.

09:02.985 --> 09:05.085
We also included several open-ended questions

09:05.085 --> 09:08.885
because we really wanted to gather, stories of how,

09:09.075 --> 09:10.365
they experienced challenges,

09:10.365 --> 09:11.765
but also how they navigated them.

09:12.425 --> 09:15.245
So one of the limitations of this data, just to kind

09:15.245 --> 09:16.605
of start there, is that it's collected

09:16.605 --> 09:17.685
at the enumerator level.

09:18.185 --> 09:20.565
We were particularly interested in the sum

09:20.585 --> 09:23.925
of their experiences because first, descriptively,

09:23.925 --> 09:25.685
this hasn't really been investigated.

09:25.785 --> 09:27.725
We don't know how often,

09:27.725 --> 09:29.885
enumerators encounter challenges in these settings.

09:29.885 --> 09:32.205
We wanted to actually get kind of a baseline for that.

09:32.945 --> 09:36.205
Um, but the limitation about that is

09:36.205 --> 09:38.805
that we don't know if a specific project

09:38.985 --> 09:41.685
or specific survey led to more challenges

09:41.705 --> 09:42.845
or fewer challenges.

09:43.305 --> 09:45.805
Um, so that's one particular limitation.

09:45.805 --> 09:48.725
And we think that future research certainly,

09:48.785 --> 09:51.645
and we, we have a lengthy implication section in the paper,

09:52.065 --> 09:55.285
but future research researchers should consider asking

09:55.285 --> 09:56.925
these questions at the end of every project

09:57.305 --> 09:59.405
and potentially even at the beginning to find out

09:59.675 --> 10:01.885
what has been the experiences of the enumerators

10:01.885 --> 10:03.325
that you're employing on your project.

10:04.025 --> 10:06.285
Um, so nonetheless, we wanted to know the sum

10:06.285 --> 10:08.205
of their experiences to understand the scope

10:08.545 --> 10:09.645
and the magnitude

10:09.645 --> 10:11.005
of the challenges they face when

10:11.005 --> 10:12.285
conducting this type of research.

10:13.145 --> 10:15.205
Um, and again, I also wanna reiterate

10:15.205 --> 10:17.445
that this project is primarily descriptive.

10:17.825 --> 10:19.605
Um, we wanted to detail the challenges

10:19.605 --> 10:21.445
that enumerators faced,

10:21.515 --> 10:23.045
because they're often overlooked,

10:23.045 --> 10:25.805
but extremely important actor in our research endeavor.

10:26.665 --> 10:29.645
Um, so that's kind of covering the, the data

10:29.985 --> 10:32.045
and, why we selected co vo.

10:32.345 --> 10:34.485
One other note that I'll say about co vo,

10:34.485 --> 10:36.165
people often are like, it's a small country.

10:36.745 --> 10:38.925
Um, you know, what can that tell us about Africa?

10:39.385 --> 10:43.645
Um, but in reality, co vo can represent a case of,

10:43.795 --> 10:47.005
high levels of insecurity, a lot

10:47.005 --> 10:49.165
of polarization on both ethnic

10:49.185 --> 10:53.925
and political lines, which could definitely generalize

10:53.925 --> 10:55.805
to places like Molly Burkina Faso,

10:55.855 --> 10:57.605
Madagascar, South Africa.

10:57.825 --> 10:59.725
And in the paper we also include,

10:59.725 --> 11:01.925
some data from the ARO Barometer to show

11:01.925 --> 11:04.005
that people across the continent,

11:04.005 --> 11:05.725
enumerators across the continent,

11:06.025 --> 11:08.605
are experiencing insecurity when they conduct this research.

11:08.825 --> 11:10.205
And this is something that we often

11:10.215 --> 11:11.525
don't think about or consider.

11:11.985 --> 11:13.765
Um, and this is happening everywhere.

11:13.865 --> 11:17.165
So though Covar is a small case, we think it kind

11:17.165 --> 11:19.125
of represents and is a good launching

11:19.125 --> 11:20.245
pad for this research agenda.

11:20.865 --> 11:24.005
But we do, in the paper though, is show relationships

11:24.035 --> 11:25.125
between,

11:25.125 --> 11:27.285
and I'm, I can start talking about, the,

11:27.345 --> 11:28.525
the findings in particular.

11:28.945 --> 11:31.965
Um, but we show relationships between experiences

11:32.065 --> 11:34.565
of violence or witnessing violence,

11:34.705 --> 11:37.205
and higher likelihood of facing these challenges.

11:37.585 --> 11:40.885
Um, and while we can't say, you know, enumerator x

11:41.835 --> 11:45.245
experienced violence and then also had this challenge in the

11:45.245 --> 11:46.965
same survey, and that's one of the limitations

11:46.965 --> 11:50.165
with the data, what we can say is that,

11:50.795 --> 11:52.805
unfortunately we document

11:52.835 --> 11:55.645
that these enumerators are experiencing high levels

11:55.665 --> 11:57.245
of violence, witnessing violence,

11:57.825 --> 11:59.685
and are facing a number

11:59.685 --> 12:02.285
of challenges when they're conducting our research in ways

12:02.285 --> 12:04.845
that often as researchers we don't consider

12:04.865 --> 12:06.045
or hadn't even thought about.

12:06.385 --> 12:08.085
Um, I was kind of queued to this,

12:08.105 --> 12:09.205
and it kind of was the reason

12:09.205 --> 12:13.005
that we started this project when I did a project in Cote Vo

12:13.005 --> 12:14.285
about, election violence.

12:14.705 --> 12:17.405
And the enumerators were sending me WhatsApp messages like,

12:17.745 --> 12:19.325
we're being followed when we

12:19.325 --> 12:20.365
were trying to conduct this survey.

12:20.505 --> 12:22.685
And so that kind of made me think,

12:22.945 --> 12:24.285
how often does this actually happen?

12:24.345 --> 12:26.085
And so that's the, the point of the paper.

12:26.545 --> 12:29.045
Um, we don't see it, we see it kind of

12:29.045 --> 12:31.485
as opening a research agenda as opposed to kind

12:31.485 --> 12:33.045
of having a final word on this topic.

12:33.465 --> 12:35.805
One of the most concerning result that we had was

12:35.805 --> 12:40.005
that almost 60% of the enumerators in our study felt unsafe,

12:40.225 --> 12:42.605
at least sometimes while conducting academic surveys,

12:42.775 --> 12:46.285
while over 75% reported fearing for their physical safety.

12:46.865 --> 12:48.165
And this is enormous.

12:48.745 --> 12:50.805
Um, this means that these are folks

12:50.905 --> 12:54.325
who every day in their work potentially are feeling this

12:54.325 --> 12:56.765
way, feeling unsafe, feeling, threatened.

12:57.265 --> 13:00.605
Um, three quarters of enumerated surveyed reported fearing

13:00.605 --> 13:02.005
for their personal physical safety.

13:02.625 --> 13:05.285
Um, 50% felt unsafe.

13:06.185 --> 13:08.685
Um, and what does this kind of, these threats

13:08.745 --> 13:09.765
or violence look like?

13:10.345 --> 13:12.805
Um, so about a third had been followed.

13:13.585 --> 13:16.925
Um, 26% had been threatened with physical violence.

13:17.105 --> 13:19.045
And to just give you an example,

13:19.045 --> 13:21.765
an enumerator told us in the open-ended question,

13:22.025 --> 13:24.925
during the war when, when they were conducting a survey

13:24.955 --> 13:28.085
that rebels were everywhere, they were following us,

13:28.205 --> 13:30.445
a respondent told us that since we had been

13:31.005 --> 13:32.005
identified we were being followed.

13:32.385 --> 13:33.965
And so this is something that came up

13:34.215 --> 13:35.605
often in, in the survey.

13:36.505 --> 13:38.365
Um, in addition to experiencing violence,

13:38.435 --> 13:40.685
enumerators also witnessed violence while on the job,

13:40.875 --> 13:43.725
including violent protests, threats, harassment

13:43.905 --> 13:44.925
and physical violence.

13:45.825 --> 13:48.205
So in addition to experiencing this violence

13:48.345 --> 13:50.365
and witnessing it, but they face a number of,

13:50.465 --> 13:51.885
challenges in the field.

13:52.195 --> 13:54.165
They feel unsafe due the con due

13:54.165 --> 13:55.485
to the conditions in which they work,

13:55.485 --> 13:57.605
including the polarized political environment,

13:57.675 --> 14:00.285
high crime rates and isolation of the field sites, feeling

14:00.285 --> 14:02.485
that they wouldn't be able to escape if they

14:02.725 --> 14:03.805
were encountered with violence.

14:04.665 --> 14:07.205
Um, they reported that the respondents

14:07.265 --> 14:10.485
and their families were the most difficult for them,

14:10.485 --> 14:13.605
sometimes even an extreme that families would threaten them

14:13.605 --> 14:14.645
with physical aggression

14:14.705 --> 14:17.365
or even keep them, hostage in their homes.

14:17.945 --> 14:19.285
Um, verbal threats

14:19.305 --> 14:22.005
and harassment increase the sense of discomfort

14:22.025 --> 14:23.565
and fear that these eners have.

14:24.505 --> 14:26.965
So what we show is that experiencing violence

14:27.185 --> 14:31.685
and working in difficult in a context, are more likely

14:31.865 --> 14:34.965
to talk about how, they have this,

14:34.965 --> 14:36.645
that it's psychologically challenging for them

14:36.645 --> 14:38.325
to do this work,

14:38.745 --> 14:41.325
and that they faced more challenges from,

14:41.475 --> 14:42.965
from the local population.

14:43.705 --> 14:45.045
So that's first the ethical

14:45.185 --> 14:47.085
and normative concern that we have, right?

14:47.585 --> 14:51.405
Um, that perhaps when we ask them to do, do this work

14:51.405 --> 14:53.285
for us, we're putting them both in physical

14:53.345 --> 14:54.485
and psychological danger,

14:55.305 --> 14:57.645
but we also show that there's serious ramifications

14:57.645 --> 14:58.725
for data validity.

14:58.955 --> 15:01.445
When enumerators face insecurity, they're more likely

15:01.465 --> 15:03.365
to overcome challenges by adapting

15:03.505 --> 15:05.045
or breaking survey protocols.

15:05.305 --> 15:08.005
And these breaches hold potentially serious consequences

15:08.105 --> 15:10.045
for the quality of the data we collect.

15:10.545 --> 15:13.405
And even if it seems benign, like skipping a household,

15:13.545 --> 15:16.285
for example, or deviating from a random walk protocol

15:16.305 --> 15:17.685
to avoid dangerous neighborhoods,

15:18.035 --> 15:19.805
this can bias the data that we collect.

15:20.315 --> 15:23.645
Another common, solution

15:23.645 --> 15:26.565
that the enumerators used was emphasizing their political

15:26.565 --> 15:29.005
or ethnic identity to be able to kind

15:29.005 --> 15:31.125
of make the respondent comfortable.

15:31.505 --> 15:34.125
But what that can do is influence the respondent's answers,

15:34.555 --> 15:37.365
even if it does protect the enumerator themselves.

15:37.585 --> 15:40.605
And so in general, we show that greater exposure to violence

15:41.065 --> 15:43.765
or working in violence affected communities correlates

15:43.765 --> 15:44.965
with an increased likelihood

15:44.965 --> 15:47.005
of deviating from survey protocols

15:47.585 --> 15:50.525
and across a range of, of various variables,

15:50.525 --> 15:52.885
that we show in the paper in numerous face.

15:53.125 --> 15:55.525
Personal insecurity during data collection report,

15:55.525 --> 15:58.525
more challenges from local populations, express difficulties

15:58.555 --> 16:00.205
with some types of survey questions

16:00.545 --> 16:02.525
and deviate from the research protocol.

16:02.865 --> 16:04.845
Oh, I just wanna add one thing about implications,

16:04.845 --> 16:06.045
which I think is that sort of,

16:06.045 --> 16:07.925
and this is gonna sort of build on something Justine just

16:07.925 --> 16:10.245
said, which is I think one key finding for us

16:10.635 --> 16:13.325
that is takes a sort of step away from the enumerator is

16:13.325 --> 16:15.165
that it's often survey respondents

16:15.165 --> 16:16.365
who are posing the biggest hurdles

16:16.365 --> 16:18.205
and obstacles to survey enumerators.

16:18.505 --> 16:21.405
So as Jesse mentioned in open-end questions, they,

16:21.495 --> 16:24.085
about 50% of our enumerators said that, you know,

16:24.085 --> 16:26.245
mentioned some problems that they'd had with respondents

16:26.245 --> 16:28.005
and how hard it's to get, you know, respondents

16:28.005 --> 16:29.885
to complete interviews, to, to consent to interviews,

16:29.905 --> 16:31.885
to understand what they're there for, et cetera.

16:32.345 --> 16:33.565
And so, you know, as much

16:33.565 --> 16:35.485
as this paper is about enumerators,

16:35.545 --> 16:38.045
we think there's this sort of secondary implication about

16:38.255 --> 16:39.485
respondent hesitancy,

16:39.485 --> 16:40.565
which is something researchers

16:40.585 --> 16:41.925
should all be concerned about.

16:42.265 --> 16:43.965
So of course, some of this might be

16:43.965 --> 16:45.845
because of research fatigue.

16:45.905 --> 16:48.405
We know that in some communities, researchers, you know,

16:48.405 --> 16:50.245
there's certain work in like refugee camps that, you know,

16:50.365 --> 16:51.685
researchers are constantly running, you know,

16:51.825 --> 16:53.805
and not just researchers, but also like the un like

16:53.965 --> 16:55.085
everyone's running a million surveys

16:55.085 --> 16:56.565
and people just get tired of participating.

16:56.945 --> 16:59.085
That's one problem. There's also, of course, an issue

16:59.085 --> 17:01.965
of institutional trust that, you know, is, is, you know,

17:02.015 --> 17:03.925
often low in many of these settings,

17:04.225 --> 17:06.525
but we also think that there's a misunderstanding about the

17:06.725 --> 17:07.725
research process in general

17:07.835 --> 17:11.405
that I think says something about our failure as researchers

17:11.405 --> 17:14.445
to sort of translate why we're, you know, our results back

17:14.445 --> 17:16.405
to communities and to help them understand why there might

17:16.405 --> 17:18.325
be some value to what we're doing.

17:18.985 --> 17:21.605
Um, and so in this sense, we, I just wanna reiterate

17:21.605 --> 17:23.725
that I think that, you know, we think there's implications

17:23.725 --> 17:26.405
here for how we work with, you know, the people we hired

17:26.545 --> 17:28.485
to collect the data that we're interested in.

17:28.705 --> 17:30.965
But we also think that there's implications here for sort

17:30.965 --> 17:33.685
of, you know, why respondents don't really wanna partake in

17:33.805 --> 17:35.165
research that they don't see any benefit from,

17:35.165 --> 17:36.885
or they don't necessarily fully understand.

17:37.025 --> 17:40.165
And, and that that's also something that we, you know,

17:40.315 --> 17:42.525
hope people will sort of start taking seriously.

17:42.745 --> 17:44.005
But also think about sort of

17:44.105 --> 17:45.805
how respondents do understand

17:45.805 --> 17:47.005
the research that they're taking part in.

17:47.465 --> 17:49.885
One of the things we really wanted to do at this paper is

17:49.885 --> 17:53.725
to like, not just kind of detail all of the challenges

17:53.725 --> 17:54.805
that these enumerators face,

17:54.805 --> 17:56.205
but to actually offer solutions.

17:56.625 --> 17:58.965
Um, and so we have kind of a lengthy,

17:59.395 --> 18:01.845
implication section with a broad range of solutions.

18:02.225 --> 18:03.725
So first we urge researchers

18:03.725 --> 18:06.805
to continue efforts towards establishing disc disciplinary

18:06.935 --> 18:09.325
norms around research and insecure settings.

18:09.515 --> 18:10.685
This is already happening.

18:10.905 --> 18:14.325
Um, there's ad hoc committees on ethics across,

18:14.435 --> 18:17.405
political science, but we wanna keep pushing that

18:17.405 --> 18:20.845
and especially think thinking about our, enumerators.

18:21.685 --> 18:25.725
Secondly, we hope that our findings offer a call

18:25.785 --> 18:27.525
to alter our research designs

18:27.745 --> 18:29.605
to minimize risk to field staffs.

18:29.825 --> 18:31.445
And there's a lot of ways that we can do this.

18:31.545 --> 18:34.645
So, it can include being attentive to the timing

18:34.745 --> 18:35.965
and content of surveys.

18:36.075 --> 18:38.685
Enumerators said that it's kind of the worst timing

18:38.865 --> 18:40.885
to do a survey in a polarized

18:41.485 --> 18:43.285
insecure setting right before an election.

18:43.625 --> 18:46.405
And often in political science, we don't actually need

18:46.405 --> 18:47.645
to do it right before an election.

18:47.645 --> 18:49.165
Sometimes we're testing a mechanism.

18:49.705 --> 18:52.765
Um, we're just collecting data on political opinions outside

18:52.765 --> 18:54.325
of the electoral, setting.

18:54.665 --> 18:57.845
And so perhaps we can move our surveys up by a few months

18:58.265 --> 19:00.805
and reduce the risk that enumerators are seen

19:00.805 --> 19:01.925
as political actors.

19:02.865 --> 19:05.765
Um, we also encourage researchers

19:05.825 --> 19:08.325
to stay informed about enumerators working conditions.

19:08.585 --> 19:11.845
Um, as Martha mentioned, there is a distance often

19:11.895 --> 19:14.605
where we only interact with the survey firms,

19:14.665 --> 19:16.925
so we don't actually know how much the enumerators are being

19:16.925 --> 19:20.045
paid, whether they can get health insurance,

19:20.045 --> 19:22.285
whether they, are able to leave

19:22.385 --> 19:24.045
and not face any consequences.

19:24.505 --> 19:27.205
Um, and so understanding the working conditions,

19:27.425 --> 19:29.405
can help a alleviate some of these risks.

19:30.225 --> 19:33.365
Um, another thing we encourage is including enumerators in

19:33.365 --> 19:35.845
all aspects of the research design to make sure

19:35.845 --> 19:37.965
that we're asking culturally sensitive questions

19:37.965 --> 19:39.765
that are adapted to local realities.

19:40.105 --> 19:42.685
Um, so piloting the questions with the enumerators,

19:42.905 --> 19:45.245
asking them, you know, should I ask the question this way

19:45.265 --> 19:47.965
or that way, can help improve,

19:48.105 --> 19:50.285
not just their experience, but also the data quality.

19:50.345 --> 19:52.605
So we don't have enumerators changing the wording

19:52.605 --> 19:53.685
of questions on the fly.

19:53.875 --> 19:55.725
Instead everyone's asking the same question

19:55.725 --> 19:57.725
that's been adapted to the local context.

19:58.945 --> 20:02.885
Uh, one thought, that can improve, especially the fact

20:02.885 --> 20:03.885
that we, we see

20:03.885 --> 20:06.765
that enumerators felt particularly psychologically taxed

20:06.765 --> 20:09.685
by this research is considering psychosocial support

20:09.705 --> 20:12.445
for field staff, in country if possible.

20:13.065 --> 20:16.005
And then finally collecting data on enumerator experiences

20:16.005 --> 20:17.765
before and after the data collection.

20:18.145 --> 20:19.605
Um, we're the first to do this.

20:19.865 --> 20:22.005
Um, but there's ways that,

20:22.605 --> 20:24.605
researchers can improve if they collect data

20:24.625 --> 20:27.245
before, if they say, have you ever experienced violence?

20:27.305 --> 20:28.325
If the enumerators have,

20:28.325 --> 20:31.285
they can ask them more questions about how can we help you

20:31.345 --> 20:32.645
to avoid these contexts?

20:32.905 --> 20:35.645
Um, and then finally, after the research, we wanna make sure

20:35.765 --> 20:38.365
that, the enumerators are working in a safe environment.

20:39.025 --> 20:43.005
And so we hope that our paper contributes to,

20:43.035 --> 20:44.125
both our understanding

20:44.345 --> 20:46.965
and renders these enumerators experiences visible.

20:47.665 --> 20:51.125
Um, but that also we can improve both the way

20:51.125 --> 20:54.045
that data's collected and protect these enumerators

20:54.625 --> 20:58.165
and also, improve the data validity of our projects.

20:58.785 --> 21:00.565
You know, often as researchers, we spend a lot

21:00.565 --> 21:01.685
of time working on our surveys

21:01.685 --> 21:03.405
and especially if there's an experimental treatment

21:03.405 --> 21:05.205
of any kind, you know, really workshopping

21:05.385 --> 21:07.085
to make sure everything is, is tight and,

21:07.105 --> 21:09.125
and sort of doing what we hope it's doing.

21:09.665 --> 21:12.205
And so I think it can be hard if you're training enumerators

21:12.235 --> 21:13.965
that suddenly people are sort of saying, well this is,

21:13.965 --> 21:15.485
makes this, you know, to sort of be like, you know,

21:15.755 --> 21:17.645
it's last minute, you already have IRB approval, right?

21:17.645 --> 21:19.645
That this can pose challenges on our end.

21:19.645 --> 21:21.845
And I think sometimes it can lead to a reticence to sort of

21:22.455 --> 21:24.885
adapt things too much when we've spent, you know, months

21:24.885 --> 21:26.205
and months working on surveys.

21:26.505 --> 21:28.365
But on the other hand, I think, you know, if

21:28.365 --> 21:30.925
what we care about is capturing the reality,

21:30.925 --> 21:32.765
then I think we should be really open to these things.

21:32.905 --> 21:35.085
So, I mean, it's not a insecure context,

21:35.145 --> 21:37.085
but I was writing a survey in Senegal this past summer

21:37.185 --> 21:39.685
and the survey new had amazing suggestions for sort of,

21:39.685 --> 21:43.205
how do you know, come up with more, locally

21:44.485 --> 21:47.125
accurate sort of examples of practices

21:47.125 --> 21:48.125
that I was interested in asking

21:48.125 --> 21:49.685
with my co-authors asking questions about.

21:49.705 --> 21:51.125
And some of 'em said, you know, we know this happens

21:51.125 --> 21:52.285
elsewhere in Senegal,

21:52.285 --> 21:54.365
but in the area that we're working, here's a better idea.

21:54.785 --> 21:56.445
And, you know, that made the survey a lot better

21:56.465 --> 21:58.645
and it, I think it helps us get better data, right?

21:58.705 --> 22:01.285
And so I think it can, I,

22:01.445 --> 22:03.485
I understand why sometimes there's a bit of a hesitancy

22:03.825 --> 22:06.445
to sort of involve enumerators too much in the feedback,

22:06.445 --> 22:07.965
but I think we should also lean into the fact

22:07.965 --> 22:10.165
that these are experienced individuals who know the context,

22:10.545 --> 22:11.925
who, you know, I think are committed

22:11.925 --> 22:14.405
to having a good survey experience for everyone, including

22:15.085 --> 22:16.285
researchers getting the data that they want.

00:00 - So the past decade or two has seen a sharp uptick in the number of academic surveys and field experiments being conducted in the global south. And one consequence of this, or perhaps a driver of it, is that there's a growing number of firms, many of which are very high quality,

00:12 - that are available to do this kind of research for researchers. So based on our data, or sorry, and I should say, so those firms then need to hire people to go out in the field and, you know, collect our surveys. And so those are survey enumerators. So based on our data, and anecdotally from other research that Jesse and I have both done in West Africa, these survey animators are generally,

00:32 - you know, pretty well educated. They're often completing university or finished university, and they're essentially doing gig work, which means they might take a contract to run an academic survey one week, and then a few weeks later they work on the national census or something like this. and so we wanna stress at the outset that these survey enumerators, are very proud of the work that they do.

00:51 - And so in both close and open-ended questions, they report really valuing the professional skills that they can develop in the course of doing academic research. They like to be able to travel and learn about their countries, which otherwise might not always be options for a lot of these people that like participate in the research process and they hope that they can help in some way. They're co citizens in the process

01:10 - of doing academic research. So these are people who are hired to do their job and are trying to do the best of their, ability. But what we've realized in sort of thinking about our own research experiences is that we don't actually know a lot about the research, you know, experience or sort of their actual sort of work experience of these people who are doing a large number of surveys every day for weeks on end to sort of, you know, produce data

01:31 - that arrives to us in our inboxes. And so we really wanted to think about this in particular, in the case of insecure context, by which we mean countries that might have an ongoing conflict. There's a growing discussion about research ethics in violent context, in political science and adjacent disciplines, but also because we think that this might impact the data that we receive in ways that we would wanna understand as researchers if there's systematic sources

01:53 - of biases in one way or the other. So to the extent that research enumerators or survey enumerators go out into communities, they have to knock on doors, they have to attempt to sort of get respondents to agree to participate. They have to walk them through very lengthy and very legalistic consent forms that might be very foreign to a lot of, you know, perhaps not particular, you know, people without high literacy skills. They have to sit with those people for some length

02:16 - of time asking them what sometimes are pretty sensitive questions. to that extent, we think that these numerator are engaged in really important acts of brokerage on our behalf as researchers. So there's been some sustained attention to thinking about research assistance as brokers. It's largely been done by anthropologists or demographers people who tend to spend a fair amount of time in the field and tend to work with one

02:37 - or a small number of brokers. And this work has tended to emphasize sort of the unequal relationship between researchers and research brokers. and that these are, you know, this work is also really thought about the ways in which, how that in inequity is acute in conflict settings, where brokers not only are providing us with data, but often are helping sort of navigate security concerns like

02:57 - roadblocks for researchers. So given this uptick in remote research and the ease with which we can now do, you know, big surveys in the field, well, you know, you know, researchers often based in the global north sit at home, you know, in our offices. we thought sort of, we wanna sort of suggest that sort of, we bring some of these insights from this work on research brokerage from adjacent disciplines to think about what the implications are for survey research.

03:21 - And we think that these concerns would be particularly acute because we're just further from the data collection process when we work with a firm or sort of employ a large number of survey enumerators, even if we hire those enumerators ourselves. And so when we have a survey, there's just more people, it tends to be more decentralized. And so you have a bunch of enumerators working under a team leader working, you know, hired by a firm. And we might be able to communicate with the firm, maybe

03:43 - with the team leader, but we don't necessarily know what information is being filtered up to us. And so we wanted to understand sort of, if we talk to enumerators themselves, right, what are their working conditions? They are really important to the research effort. But often in things like IRB, we aren't asked questions about how we're going to protect their security, or ensure that they themselves don't face, you know,

04:06 - dangerous context, despite the fact that they are the ones on the ground collecting the data for us. and so what that was another motivation for pursuing this is that we often just kind of think of them as neutral actors that are just out there doing this work for us. but in reality they have to navigate these contexts just like the respondents that are answering our surveys,

04:27 - do in in insecure settings that can include violence witnessing violent violence, experiencing violence themselves. We certainly have paid attention to survey enumerators before, but the vast majority of work is focused on sort of two things. So first we know a growing amount about the enumerator respondent dyad. So most work here has focused on how characteristics of the enumerators might elicit certain types

04:49 - of responses by respondents. So there's been work by CLA data and a number of co-authors on how ethnicity a respondent or co enumerators ethnicity might cue certain types of respondents from either co-ethnic or non co-ethnic survey respondents. There's been work on gender, there's been work on sort of religious dress by Lindsay Bedstead and others. and it's all been shown to influence sort of how an enumerator,

05:12 - something about the enumerators characteristics are shaping the responses they get from respondents. Secondly, there's been some work that I think is encapsulated well in a, piece in the annual review of political science by no Lupa and Kristen Michel that sort of has thought about how enumerators own behavior might shape the data generating process. And in particular about concerns over, fabricated data. But there's also been work that's just thought about

05:36 - how enumerators might accelerate surveys. They might develop priors that are gonna influence how they approach certain questions, et cetera. And so the biggest concern here, of course, is really about how enumerators themselves are doing something to impact data quality. so both of these literatures have distinct implications. So if it's the former, if we think it's something about enumerator characteristics that are influencing responses, then something like including enumerator fixed effects

05:59 - or a dummy variable for an enumerator agenda in our, you know, models that we might run could maybe address that if we think it's something about that enumerators are doing, like, perhaps more related to like their professionalism, let's say. Right? Then, you know, there's also techniques. So increasingly surveys are run on tablets, and so we might be able to sort of see how long a survey took, right? And a survey didn't take long enough.

06:20 - We might have questions about the quality of the survey, or sometimes we can geolocate that so we know whether or not a survey enumerator is actually going to different households or just sitting in one spot filling out surveys at the end of the day. and of course we can also think about other techniques to sort of, you know, design questions that are particularly clever to maybe address some of these concerns, et cetera. So that literature all exists and we think it's important,

06:42 - but we don't think it's actually capturing sort of the political context that sort of enumerators are working in, which we think is sort of a distinct driver of, both enumerator work experiences that we care about for normative reasons, but also data quality reasons that we'll talk about more in a few minutes. But we thought about, our own experiences and I've, I've had a lot of experience working in TiVo, with Enumerators there.

07:06 - And so what we did, is we recruited a diverse set of roughly 300, enumerators from the top three research firms in co dvo. the reason that co dvo is actually really interesting, is several fold. So first, it is a site of, surveys and academic research. there's a number of publications, that have, included data from TiVo. the Afro Barometer has been conducted there several

07:35 - rounds, but crucially, the political context is what we are particularly interested in. and TiVo unfortunately, experienced the civil war from 2002 to 2011, and has experienced a, a high level of, election related violence, ethnic violence. and so we see a high level of insecurity, for people living in the country as well as enumerators who are conducting this research. and so the goal that we had then was to get,

08:05 - an understanding of these enumerator experiences. So we recruited them from these top three firms. they ultimately represented all of the ethnic groups in co TiVo, and had substantial experience in the sector. as Martha said, these folks tend to be more educated than the general population, and to have like master's degrees and that sort, and they tend to come from more urban settings, although, one

08:30 - of the nice things about our recruitment strategy is that some of these survey firms like to recruit locally, and so many of the enumerators, come from sometimes even rural villages. and so what we asked them to do was to complete an online survey about the sum of their enumerator experiences. we asked them about, just general, you know, where have you worked, what did you do, the challenges they faced while implementing projects,

08:55 - the challenges they encountered as a function of the survey content, and then solutions they may have employed when they ran into these challenges. We also included several open-ended questions because we really wanted to gather, stories of how, they experienced challenges, but also how they navigated them. So one of the limitations of this data, just to kind of start there, is that it's collected

09:16 - at the enumerator level. We were particularly interested in the sum of their experiences because first, descriptively, this hasn't really been investigated. We don't know how often, enumerators encounter challenges in these settings. We wanted to actually get kind of a baseline for that. but the limitation about that is that we don't know if a specific project or specific survey led to more challenges

09:41 - or fewer challenges. so that's one particular limitation. And we think that future research certainly, and we, we have a lengthy implication section in the paper, but future research researchers should consider asking these questions at the end of every project and potentially even at the beginning to find out what has been the experiences of the enumerators that you're employing on your project. so nonetheless, we wanted to know the sum

10:06 - of their experiences to understand the scope and the magnitude of the challenges they face when conducting this type of research. and again, I also wanna reiterate that this project is primarily descriptive. we wanted to detail the challenges that enumerators faced, because they're often overlooked, but extremely important actor in our research endeavor.

10:26 - so that's kind of covering the, the data and, why we selected co vo. One other note that I'll say about co vo, people often are like, it's a small country. you know, what can that tell us about Africa? but in reality, co vo can represent a case of, high levels of insecurity, a lot of polarization on both ethnic and political lines, which could definitely generalize to places like Molly Burkina Faso,

10:55 - Madagascar, South Africa. And in the paper we also include, some data from the ARO Barometer to show that people across the continent, enumerators across the continent, are experiencing insecurity when they conduct this research. And this is something that we often don't think about or consider. and this is happening everywhere. So though Covar is a small case, we think it kind

11:17 - of represents and is a good launching pad for this research agenda. But we do, in the paper though, is show relationships between, and I'm, I can start talking about, the, the findings in particular. but we show relationships between experiences of violence or witnessing violence, and higher likelihood of facing these challenges. and while we can't say, you know, enumerator x

11:41 - experienced violence and then also had this challenge in the same survey, and that's one of the limitations with the data, what we can say is that, unfortunately we document that these enumerators are experiencing high levels of violence, witnessing violence, and are facing a number of challenges when they're conducting our research in ways that often as researchers we don't consider or hadn't even thought about.

12:06 - I was kind of queued to this, and it kind of was the reason that we started this project when I did a project in Cote Vo about, election violence. And the enumerators were sending me WhatsApp messages like, we're being followed when we were trying to conduct this survey. And so that kind of made me think, how often does this actually happen? And so that's the, the point of the paper.

12:26 - we don't see it, we see it kind of as opening a research agenda as opposed to kind of having a final word on this topic. One of the most concerning result that we had was that almost 60% of the enumerators in our study felt unsafe, at least sometimes while conducting academic surveys, while over 75% reported fearing for their physical safety. And this is enormous. this means that these are folks who every day in their work potentially are feeling this

12:54 - way, feeling unsafe, feeling, threatened. three quarters of enumerated surveyed reported fearing for their personal physical safety. 50% felt unsafe. and what does this kind of, these threats or violence look like? so about a third had been followed. 26% had been threatened with physical violence. And to just give you an example, an enumerator told us in the open-ended question,

13:22 - during the war when, when they were conducting a survey that rebels were everywhere, they were following us, a respondent told us that since we had been identified we were being followed. And so this is something that came up often in, in the survey. in addition to experiencing violence, enumerators also witnessed violence while on the job, including violent protests, threats, harassment and physical violence.

13:45 - So in addition to experiencing this violence and witnessing it, but they face a number of, challenges in the field. They feel unsafe due the con due to the conditions in which they work, including the polarized political environment, high crime rates and isolation of the field sites, feeling that they wouldn't be able to escape if they were encountered with violence. they reported that the respondents

14:07 - and their families were the most difficult for them, sometimes even an extreme that families would threaten them with physical aggression or even keep them, hostage in their homes. verbal threats and harassment increase the sense of discomfort and fear that these eners have. So what we show is that experiencing violence and working in difficult in a context, are more likely to talk about how, they have this,

14:34 - that it's psychologically challenging for them to do this work, and that they faced more challenges from, from the local population. So that's first the ethical and normative concern that we have, right? that perhaps when we ask them to do, do this work for us, we're putting them both in physical and psychological danger, but we also show that there's serious ramifications

14:57 - for data validity. When enumerators face insecurity, they're more likely to overcome challenges by adapting or breaking survey protocols. And these breaches hold potentially serious consequences for the quality of the data we collect. And even if it seems benign, like skipping a household, for example, or deviating from a random walk protocol to avoid dangerous neighborhoods, this can bias the data that we collect.

15:20 - Another common, solution that the enumerators used was emphasizing their political or ethnic identity to be able to kind of make the respondent comfortable. But what that can do is influence the respondent's answers, even if it does protect the enumerator themselves. And so in general, we show that greater exposure to violence or working in violence affected communities correlates with an increased likelihood of deviating from survey protocols

15:47 - and across a range of, of various variables, that we show in the paper in numerous face. Personal insecurity during data collection report, more challenges from local populations, express difficulties with some types of survey questions and deviate from the research protocol. I just wanna add one thing about implications, which I think is that sort of, and this is gonna sort of build on something Justine just said, which is I think one key finding for us

16:10 - that is takes a sort of step away from the enumerator is that it's often survey respondents who are posing the biggest hurdles and obstacles to survey enumerators. So as Jesse mentioned in open-end questions, they, about 50% of our enumerators said that, you know, mentioned some problems that they'd had with respondents and how hard it's to get, you know, respondents to complete interviews, to, to consent to interviews, to understand what they're there for, et cetera.

16:32 - And so, you know, as much as this paper is about enumerators, we think there's this sort of secondary implication about respondent hesitancy, which is something researchers should all be concerned about. So of course, some of this might be because of research fatigue. We know that in some communities, researchers, you know, there's certain work in like refugee camps that, you know,

16:50 - researchers are constantly running, you know, and not just researchers, but also like the un like everyone's running a million surveys and people just get tired of participating. That's one problem. There's also, of course, an issue of institutional trust that, you know, is, is, you know, often low in many of these settings, but we also think that there's a misunderstanding about the research process in general that I think says something about our failure as researchers

17:11 - to sort of translate why we're, you know, our results back to communities and to help them understand why there might be some value to what we're doing. and so in this sense, we, I just wanna reiterate that I think that, you know, we think there's implications here for how we work with, you know, the people we hired to collect the data that we're interested in. But we also think that there's implications here for sort of, you know, why respondents don't really wanna partake in research that they don't see any benefit from,

17:35 - or they don't necessarily fully understand. And, and that that's also something that we, you know, hope people will sort of start taking seriously. But also think about sort of how respondents do understand the research that they're taking part in. One of the things we really wanted to do at this paper is to like, not just kind of detail all of the challenges that these enumerators face, but to actually offer solutions.

17:56 - and so we have kind of a lengthy, implication section with a broad range of solutions. So first we urge researchers to continue efforts towards establishing disc disciplinary norms around research and insecure settings. This is already happening. there's ad hoc committees on ethics across, political science, but we wanna keep pushing that and especially think thinking about our, enumerators. Secondly, we hope that our findings offer a call

18:25 - to alter our research designs to minimize risk to field staffs. And there's a lot of ways that we can do this. So, it can include being attentive to the timing and content of surveys. Enumerators said that it's kind of the worst timing to do a survey in a polarized insecure setting right before an election. And often in political science, we don't actually need to do it right before an election.

18:47 - Sometimes we're testing a mechanism. we're just collecting data on political opinions outside of the electoral, setting. And so perhaps we can move our surveys up by a few months and reduce the risk that enumerators are seen as political actors. we also encourage researchers to stay informed about enumerators working conditions. as Martha mentioned, there is a distance often where we only interact with the survey firms,

19:14 - so we don't actually know how much the enumerators are being paid, whether they can get health insurance, whether they, are able to leave and not face any consequences. and so understanding the working conditions, can help a alleviate some of these risks. another thing we encourage is including enumerators in all aspects of the research design to make sure that we're asking culturally sensitive questions that are adapted to local realities.

19:40 - so piloting the questions with the enumerators, asking them, you know, should I ask the question this way or that way, can help improve, not just their experience, but also the data quality. So we don't have enumerators changing the wording of questions on the fly. Instead everyone's asking the same question that's been adapted to the local context. one thought, that can improve, especially the fact that we, we see

20:03 - that enumerators felt particularly psychologically taxed by this research is considering psychosocial support for field staff, in country if possible. And then finally collecting data on enumerator experiences before and after the data collection. we're the first to do this. but there's ways that, researchers can improve if they collect data before, if they say, have you ever experienced violence? If the enumerators have,

20:28 - they can ask them more questions about how can we help you to avoid these contexts? and then finally, after the research, we wanna make sure that, the enumerators are working in a safe environment. And so we hope that our paper contributes to, both our understanding and renders these enumerators experiences visible. but that also we can improve both the way that data's collected and protect these enumerators and also, improve the data validity of our projects.

20:58 - You know, often as researchers, we spend a lot of time working on our surveys and especially if there's an experimental treatment of any kind, you know, really workshopping to make sure everything is, is tight and, and sort of doing what we hope it's doing. And so I think it can be hard if you're training enumerators that suddenly people are sort of saying, well this is, makes this, you know, to sort of be like, you know, it's last minute, you already have IRB approval, right?

21:17 - That this can pose challenges on our end. And I think sometimes it can lead to a reticence to sort of adapt things too much when we've spent, you know, months and months working on surveys. But on the other hand, I think, you know, if what we care about is capturing the reality, then I think we should be really open to these things. So, I mean, it's not a insecure context, but I was writing a survey in Senegal this past summer and the survey new had amazing suggestions for sort of,

21:39 - how do you know, come up with more, locally accurate sort of examples of practices that I was interested in asking with my co-authors asking questions about. And some of 'em said, you know, we know this happens elsewhere in Senegal, but in the area that we're working, here's a better idea. And, you know, that made the survey a lot better and it, I think it helps us get better data, right? And so I think it can, I,

22:01 - I understand why sometimes there's a bit of a hesitancy to sort of involve enumerators too much in the feedback, but I think we should also lean into the fact that these are experienced individuals who know the context, who, you know, I think are committed to having a good survey experience for everyone, including researchers getting the data that they want.

Faculti AI can make mistakes