Popular in Social Sciences

Books

City Time: On Being Sentenced to Rikers Island

Independent Scholar David A. Campbell

Culture and Society

The City as a Growth Machine: Toward a Political Economy of Place

New York University Harvey Moltoch

Economics

Gender and Economics

University of California Santa Barbara Shelly Lundberg

American Politics

Keeping the Republic

Boston College Marc Landy

Banks and Policy Organisations

Greening the financial sector

European Investment Bank Matteo Ferrazzi

British Politics

Entrepreneurial thinking: the politics and practice of policy impact

University of Sheffield Matthew Flinders

Culture and Society

Carceral Entrapments: Views from the Prison/Street Interface in India

Indian Institute of Technology Delhi Mahuya Bandyopadhya

How Credible is the Credibility Revolution?

View more insights in Social Sciences - Economics

View more figures from Boston University

View the page of Kevin Lang

Read the Study

When economists analyze a well-conducted RCT or natural experiment and find a statistically significant effect, they conclude the null of no effect is unlikely to be true. But how frequently is this conclusion warranted? Kevin Lang discusses.

Image courtesy of interviewee. August 1, 2024

Generating Workspace

Export as PDF

This video is a transcription of an interview discussing the "credibility revolution" in economics and how it relates to evaluating the effectiveness of policies. The key points covered include:

1. **Credibility Revolution**: This video movement aims to improve the methods used in econometric studies to better compare different groups and isolate the effects of specific variables, such as military service on earnings. Techniques like utilizing random assignment and draft lotteries enhance the reliability of causal inferences.

2. **Null Hypothesis Testing**: Economists often start with a null hypothesis, which states that there is no effect from a policy or intervention. They analyze data to determine if they can reject this hypothesis. The confidence in rejecting the null hypothesis hinges on the statistical significance of the findings, typically assessed through p-values.

3. **Findings on False Discoveries**: The research suggests that a significant proportion of studies published in leading economics journals may contain false discoveries. Specifically, 41% of rejected null hypotheses might be erroneous, and for studies with marginal p-values around 0.05, the false discovery rate could soar to 65%.

4. **Implications for Policy**: The findings prompt a cautious approach to policy implementation based on economic studies. Economists must be more discerning about the robustness of evidence before endorsing policies, especially when the consequences of incorrect conclusions could be substantial.

5. **Future Research Directions**: The interviewee expresses a commitment to focus on development economics and plans to explore further the accuracy of findings in that area, emphasizing the need for more rigorous replication studies in the field.

Overall, the discussion highlights the importance of scrutinizing research methods in economics and the need for a cautious interpretation of results, especially when formulating public policy.

Language:

Export as PDF

Export as PDF
Here are some key quotes extracted from the video you've :

1. **On Traditional Economic Analysis:**
"Historically, when economists tried to figure out whether something had an effect... they would compare people who went to Vietnam with people who did not go to Vietnam."

2. **On the Credibility Revolution:**
"The credibility revolution tried to find ways to make that comparison better."

3. **Using the Draft Lottery Example:**
"One of the ways that we did that was to take advantage of... the draft lottery."

4. **On the Null Hypothesis:**
"The way economists normally think about this is to begin with what we call the null hypothesis."

5. **On Statistical Significance:**
"We generally say that the result is significant if the probability of the event happening by chance is less than 5%."

6. **Questioning the Null Hypothesis:**
"The real question is how likely is the null hypothesis to be true?"

7. **On False Discoveries:**
"41% of all rejected null hypotheses in these leading economics journals are false discoveries."

8. **On Narrow Rejections:**
"If I look at very narrow rejections, ones where the probability is right around 0.05... then we're talking about 65% of those being false discoveries."

9. **Reflection on Empirical Research:**
"I think we have to be cautious. We have to have a lot less confidence in calling so-called significant results as clear policy implications."

10. **Concern Over Evidence:**
"I got worried that we fund a lot of really good studies... but that means we're sometimes gonna find that policies work when they really don't work."

These quotes encapsulate the core ideas presented in the speech about economic analysis, the relevance of statistical testing, the nuances of the credibility revolution, and the concerns regarding the reliability of findings in empirical economics.

Export as PDF
Here are some articles that share similar themes with the video you , focusing on the "credibility revolution" in economics, the challenges of empirical research in the field, and implications for policy evaluation:

1. **"The Power of Zero: The Credibility Revolution in Economics"** - This article discusses the credibility revolution and contrasts traditional methods in economics with new approaches that prioritize experimental designs and causal inference techniques. It emphasizes ongoing changes in economic research practices and their implications for policymakers.

2. **"Econometric Methodology: Understanding Null Hypothesis Testing"** - A thorough examination of null hypothesis significance testing in economics. This piece critiques common practices, such as reliance on p-values, and discusses their contributions to false discoveries in published research.

3. **"Revisiting the False Discovery Rate in Economics: A Meta-Analysis"** - This research article analyzes the prevalence of false discoveries in economics literature, similar to the findings mentioned in your transcription. It discusses implications for future research and how economists can mitigate the risks of erroneous conclusions.

4. **"Policy Implications and the Role of Evidence in Economic Research"** - This paper delves into how the credibility revolution impacts policy formulation. It discusses the necessity for robust evidence before adopting policies based on econometric studies, reflecting the cautious approach highlighted in your video.

5. **"Challenges in Replication: The Case of Development Economics"** - An exploration of the importance of replication in economics, especially in the area of development economics. The article addresses the insights gained from unsuccessful replication attempts and stresses the need for rigorous methodologies.

6. **"The Evolution of Causal Inference in Economics: From Correlation to Causation"** - A comprehensive review of how economic research has evolved, from reliance on correlational methods to adopting more rigorous causal inference techniques, highlighting the impact of the credibility revolution on the discipline.

These articles collectively illuminate the challenges of empirical economics, the significance of rigorous methodologies, and the vital role that smart policy-making must play in the face of potential pitfalls in research outcomes.

X-TIMESTAMP-MAP:LOCAL=00:00:00.000,MPEGTS=900000

00:00.445 --> 00:03.925
Historically, when economists tried to figure out whether

00:04.865 --> 00:08.985
something had an effect, say whether, going

00:08.985 --> 00:13.105
to Vietnam had an impact on your eventual earnings,

00:13.765 --> 00:17.225
what they would do is they would compare people who went

00:17.225 --> 00:20.705
to Vietnam with people who did not go to Vietnam,

00:21.365 --> 00:24.665
and they would try to look at people who looked similar.

00:25.405 --> 00:28.345
So maybe what they would do is to,

00:28.575 --> 00:32.345
look only at white people, who went to Vietnam

00:32.405 --> 00:33.545
and who didn't go to Vietnam,

00:33.545 --> 00:37.625
or maybe even white high school dropouts who went

00:37.625 --> 00:40.465
to Vietnam versus white high school dropouts who didn't.

00:41.125 --> 00:44.585
But we were never very good at being convincing

00:44.585 --> 00:47.265
that we had found all of the things that were different

00:47.265 --> 00:50.465
between the people who went to Vietnam and those who didn't.

00:51.985 --> 00:56.895
And so what the credibility revolution tried

00:56.895 --> 01:01.655
to do is to find ways to make that comparison better.

01:02.565 --> 01:06.925
So they tried to find somebody who looked even more similar

01:07.265 --> 01:08.605
to somebody who went to Vietnam.

01:09.625 --> 01:13.685
And one of the ways that we did that was to take advantage

01:14.145 --> 01:17.125
of something like in the case of Vietnam,

01:18.105 --> 01:19.205
the draft lottery.

01:20.035 --> 01:21.575
So the draft lottery was based

01:21.575 --> 01:22.735
on the day that you were born.

01:23.745 --> 01:26.165
So it might be that if you were born on January 5th,

01:26.265 --> 01:28.245
you had a low lottery number, which meant

01:28.245 --> 01:29.965
that you were very likely to be drafted.

01:30.305 --> 01:32.885
But if you were born on January 6th, you happened

01:32.885 --> 01:34.765
to have a very high lottery number

01:34.905 --> 01:37.045
and were unlikely to be drafted.

01:37.465 --> 01:39.685
And there's really almost no difference between the people

01:39.685 --> 01:42.445
who were born on January 5th and people born on January 6th.

01:42.825 --> 01:45.725
So that seems like a pretty good, pretty good comparison.

01:45.865 --> 01:48.405
And we could look at what was the difference in the

01:48.565 --> 01:50.045
proportion of people who went

01:50.065 --> 01:52.125
to Vietnam based on their birthdate

01:52.705 --> 01:54.925
and what were the differences in their outcome,

01:55.265 --> 01:58.685
and figure out the impact of going to Vietnam.

01:58.825 --> 02:01.445
Now, that example is a little bit more precise than we've

02:01.445 --> 02:04.125
ever been able to do, but it's the basic idea,

02:05.225 --> 02:09.525
the credibility revolution basically popularized a set

02:09.525 --> 02:12.605
of techniques that compare people in situations

02:13.175 --> 02:14.925
where they seemed like they were pretty

02:14.925 --> 02:16.405
nearly randomly assigned.

02:17.485 --> 02:19.465
In the best cases, they actually were,

02:19.615 --> 02:22.345
they actually ran experiments just the way physicians do.

02:23.105 --> 02:25.645
Um, so we might in a situation where

02:26.185 --> 02:30.555
the training program was oversubscribed, randomly assigned,

02:30.735 --> 02:33.035
who of the people wanted to go into the training program,

02:33.895 --> 02:36.115
got into the training program, compare outcomes.

02:36.755 --> 02:37.945
There, there are a bunch of other

02:37.945 --> 02:39.265
techniques in the interest of time.

02:39.345 --> 02:40.665
I don't think I should go into them,

02:40.965 --> 02:44.105
but basically it's about it finding ways to do a better job

02:44.105 --> 02:46.585
of comparing apples to apples into apples oranges.

02:46.685 --> 02:50.345
The critical issue that I wanted to address is,

02:50.875 --> 02:52.505
let's take one of these very well

02:53.145 --> 02:55.185
executed credibility revolution studies,

02:56.065 --> 02:59.325
and it says that, you know, some policy is effective, say,

02:59.445 --> 03:01.405
I don't know, for example,

03:01.415 --> 03:04.685
maybe giving students more tutoring has a significant effect

03:04.685 --> 03:07.205
on their earnings after they leave high school

03:07.345 --> 03:09.565
or secondary school to be more international.

03:10.145 --> 03:12.205
Um, how confident should we be

03:12.205 --> 03:13.405
that the policy actually works?

03:13.405 --> 03:14.765
That's the question I want to answer.

03:15.945 --> 03:19.325
Now, the way economists normally think about this is

03:19.325 --> 03:21.885
to begin with what we call the null hypothesis, right?

03:21.945 --> 03:25.125
In this case, the null hypothesis is at the effect

03:25.125 --> 03:28.245
of giving the extra tutoring on later earnings this year,

03:28.665 --> 03:31.645
and it has no effect on outcomes, right?

03:32.345 --> 03:36.085
And normally what the economist is going to say is,

03:36.555 --> 03:38.485
suppose I do find an effect.

03:39.695 --> 03:42.355
How likely is it that if there were no effect,

03:43.295 --> 03:46.255
I would actually find an effect that the, that's this large.

03:47.195 --> 03:50.815
So to give a a simple example, you know,

03:50.815 --> 03:52.535
think about tossing a coin a hundred times.

03:53.255 --> 03:54.615
I won't always get 50 heads.

03:54.725 --> 03:59.215
Sometimes I got 48 heads, sometimes I get 52 heads, right?

03:59.235 --> 04:01.695
The same thing can happen if you're tutoring people.

04:02.005 --> 04:05.855
Sometimes kids you happen to randomly sign to get tutored

04:06.745 --> 04:09.405
better than the ones that you happen to randomly sign not

04:09.405 --> 04:13.045
to get tutoring, or they just happen to get lucky on the,

04:13.345 --> 04:15.205
on the test, they guess some multiple

04:15.205 --> 04:16.685
choice questions, right?

04:17.425 --> 04:20.285
Um, and so what we wanna ask is how likely,

04:20.665 --> 04:24.325
if there really was no difference, would I find an effect

04:24.325 --> 04:26.525
that's this large, that thing's called the P value?

04:27.345 --> 04:29.605
And we generally say that the result is significant

04:30.265 --> 04:33.485
if the probability of the event happening by chance

04:34.105 --> 04:36.085
is less than 5%, right?

04:37.385 --> 04:40.725
So in that case, we would say that the effect that we found

04:40.745 --> 04:45.005
of tutoring, that we found a probability

04:45.035 --> 04:48.885
that it has no effect is sufficiently unlikely

04:49.475 --> 04:53.205
that we would reject the hypothesis that it has no effect.

04:54.255 --> 04:57.915
And as a result, since the alternative to having no effect

04:58.535 --> 05:01.355
is that it has an effect, we implicitly,

05:01.355 --> 05:04.275
or I would say often explicitly conclude

05:04.275 --> 05:07.225
that the tutoring did have an effect, right?

05:07.485 --> 05:11.745
And that in turn would point to supporting a policy

05:11.925 --> 05:12.945
of implementing tutoring.

05:13.585 --> 05:15.185
I wanna be careful here, right?

05:15.185 --> 05:17.665
There are other considerations like cost, right?

05:18.085 --> 05:20.465
Uh, but now at least we know that tutoring is effective.

05:21.235 --> 05:25.475
So, so the question that I ask is how likely is it

05:25.475 --> 05:26.675
that tutoring is effective?

05:26.935 --> 05:29.315
And notice that's a different question, right?

05:30.355 --> 05:32.355
Normally what we ask is,

05:32.585 --> 05:34.715
suppose tutoring doesn't have an effect.

05:35.175 --> 05:37.845
How likely is my result? Right?

05:38.185 --> 05:42.405
We never ask, assume that the tutoring does have an effect.

05:43.465 --> 05:45.045
How unlikely is my result?

05:45.995 --> 05:47.815
And we need to know both of those, right?

05:47.835 --> 05:50.095
So the real question is

05:50.315 --> 05:52.895
how likely is the null hypothesis to be true?

05:54.175 --> 05:56.595
And let me see if I can give you an extreme example.

05:57.265 --> 05:59.965
Uh, but I think a helpful one in terms of the intuition.

06:01.475 --> 06:03.485
Suppose I have a coin, right?

06:03.825 --> 06:05.525
And I know, or at least I believe

06:05.875 --> 06:09.605
that it's either a fair coin produced by the US mint,

06:10.185 --> 06:11.725
or it's a fake coin with two heads.

06:12.255 --> 06:15.765
Right? Now, if it's produced by the US mint,

06:15.785 --> 06:19.285
and if I flip it 50% of the times, it'll give heads 50%

06:19.285 --> 06:20.645
of the time it'll give me tails.

06:22.205 --> 06:24.825
Of course, if it's a fake coin, I always get two heads.

06:24.885 --> 06:26.385
But let's not worry about that yet.

06:27.205 --> 06:29.705
Now, as an economist, I start with a null hypothesis

06:29.705 --> 06:31.465
that it's a two headed coin, right?

06:31.875 --> 06:35.495
So I flip it some number 20 times, let's say,

06:35.795 --> 06:37.695
and I get, let's say 17 hits.

06:38.465 --> 06:40.205
Um, if the coin's truly fair,

06:41.065 --> 06:43.565
that's a really unlikely outcome, right?

06:43.625 --> 06:46.285
And since my null hypothesis is that the coin is fair,

06:47.045 --> 06:48.125
I have to reject it, right?

06:48.125 --> 06:50.325
Because the outcome that I've observed is unlikely under

06:50.325 --> 06:51.405
the null hypothesis.

06:52.545 --> 06:55.725
And since my only choices are either that it's a fair coin

06:56.025 --> 06:59.395
or it's a two-headed coin under the traditional way

06:59.395 --> 07:01.835
of thinking about it, I reject the null

07:02.095 --> 07:04.515
and accept the alternative, which is

07:04.515 --> 07:05.595
that it's a two-headed coin.

07:06.215 --> 07:07.635
And that, of course is ridiculous

07:07.635 --> 07:10.355
because I couldn't possibly get even one tail,

07:10.615 --> 07:14.195
let alone three, if it were a two headed point.

07:15.645 --> 07:18.425
So the question I wanna ask is how likely

07:18.965 --> 07:21.545
is the outcome under the alternative versus

07:21.685 --> 07:23.385
how likely it is under the null?

07:24.425 --> 07:26.325
And we also have to ask, you know,

07:26.325 --> 07:27.685
originally when I started out,

07:27.745 --> 07:29.925
how likely did I think the alternative and the,

07:30.305 --> 07:32.285
and the null were right?

07:32.475 --> 07:35.605
Because if I were totally convinced that the coin

07:36.675 --> 07:37.975
was fair, right?

07:38.035 --> 07:41.495
It would take a lot of evidence to to change my mind, right?

07:41.715 --> 07:43.775
So if I thought it was pretty likely that it was unfair,

07:43.775 --> 07:45.215
then it wouldn't take so much evidence.

07:45.595 --> 07:47.295
So I also have to take that into account

07:47.835 --> 07:50.455
and putting those two things together, I can calculate

07:51.195 --> 07:55.375
the probability that the null hypothesis is true.

07:55.905 --> 07:57.815
There are two things that I need to know in order

07:57.835 --> 08:00.535
to decide whether null hypothesis is true or not.

08:00.675 --> 08:03.055
That's what I was just talking about, right?

08:03.395 --> 08:04.575
How likely is the outcome

08:04.575 --> 08:06.655
that I observed not only under the null,

08:07.195 --> 08:08.855
but also under the alternative,

08:10.005 --> 08:13.785
and also how likely was the null a priority?

08:13.785 --> 08:15.505
What did I think it was, right?

08:15.695 --> 08:17.705
There's a formula for putting those things together,

08:17.765 --> 08:18.945
and it's called the base there.

08:20.045 --> 08:23.265
So the goal of my research is to take the body

08:23.265 --> 08:24.345
of empirical research

08:24.375 --> 08:27.065
that broadly fits into the credibility revelation

08:27.905 --> 08:31.005
and ask how likely is it that a rejected null

08:31.345 --> 08:33.965
that's been published in a good journal

08:34.755 --> 08:36.055
should not have been rejected?

08:36.445 --> 08:39.695
Another way of putting it is what's the false discovery

08:39.845 --> 08:41.245
rate, right?

08:41.665 --> 08:44.045
And what particularly what is the false discovery rate

08:44.505 --> 08:45.845
and empirical economics

08:45.845 --> 08:48.005
that draws on the credibility revolution?

08:50.185 --> 08:53.685
So the way I proceed is I think of economists

08:53.745 --> 08:56.480
as producing results from two different situations.

08:57.785 --> 09:01.085
In the first situation, they're actually testing

09:01.845 --> 09:02.925
hypotheses that are true.

09:04.225 --> 09:08.015
And in that case, I'm gonna use our most common measure of

09:08.015 --> 09:10.775
how likely the finding is under the null, which is

09:10.775 --> 09:12.215
what we call the T statistic.

09:12.995 --> 09:15.295
And in the situation where the null is true,

09:15.955 --> 09:18.175
the t statistic has the T distribution.

09:18.395 --> 09:22.265
So it's very easy. And for those people who are not familiar

09:22.265 --> 09:24.545
with the T distribution, you can think of it

09:24.545 --> 09:27.465
for most purposes as having a normal distribution

09:28.185 --> 09:30.825
centered on zero and a standard deviation of one.

09:31.445 --> 09:33.315
So that's, that's some fraction

09:33.475 --> 09:35.315
of the null hypotheses that are being tested.

09:36.775 --> 09:39.395
And then there are the null hypotheses that are false.

09:40.915 --> 09:42.655
And the average t statistic we're going

09:42.655 --> 09:44.775
to find in those cases is gonna depend on

09:44.775 --> 09:46.095
how large the true effect is

09:46.755 --> 09:48.455
and how precise our experiment is.

09:48.915 --> 09:51.615
So the more data we have, everything else equal,

09:51.675 --> 09:53.775
the higher the T statistic will tend to be,

09:54.855 --> 09:57.195
and that t statistic turns out we'll have something

09:57.195 --> 09:59.315
that's called the non-central T distribution.

10:01.155 --> 10:04.175
So what I then do is I assume a fairly general

10:05.505 --> 10:09.045
raw distribution from which those expected Ts

10:09.395 --> 10:10.645
that are the central point

10:10.645 --> 10:14.525
of the non-central T distribution, like the, the

10:15.085 --> 10:16.485
distribution from which they are drawn.

10:17.595 --> 10:20.015
And then my objective is to measure the proportion

10:20.015 --> 10:23.175
of hypotheses that economists study that are true.

10:24.825 --> 10:27.485
In other words, the ones that should not be rejected,

10:28.265 --> 10:31.655
and the distribution of TT statistics, the false nulls

10:31.725 --> 10:32.935
that should be rejected.

10:34.165 --> 10:35.295
Okay? So to do this,

10:36.235 --> 10:38.615
had my research assistant collect data from a set

10:38.615 --> 10:42.285
of leading journals, I have to give her credit,

10:42.755 --> 10:44.765
they read the paper sufficiently carefully

10:44.785 --> 10:47.445
to identify the core null hypothesis.

10:48.035 --> 10:49.535
And, and that's important, right?

10:49.815 --> 10:53.055
'cause we didn't include robustness, checks,

10:54.145 --> 10:55.695
heterogene analysis

10:55.695 --> 10:58.215
unless they were core to the paper, right?

10:58.515 --> 11:00.735
And we then took those data

11:01.515 --> 11:03.375
and found the parameters of the model

11:03.885 --> 11:05.895
that best fit the empirical distribution

11:05.895 --> 11:06.975
of the two statistics.

11:07.675 --> 11:09.895
Now, I've kind of simplified things.

11:09.955 --> 11:12.775
The final model is a little more complicated than

11:12.775 --> 11:13.895
what I just described.

11:14.755 --> 11:18.135
For example, I allow the null hypothesis

11:18.135 --> 11:19.615
to be only approximately true.

11:20.355 --> 11:24.745
So that if you know, the null is zero is true, even if the,

11:25.535 --> 11:27.385
it's just a little bit different from zero.

11:28.165 --> 11:29.785
Uh, but I think I've described

11:30.635 --> 11:32.405
what capture the spirit of what I do.

11:32.555 --> 11:34.445
This work has been done,

11:34.725 --> 11:36.485
I think in a less fully convincing way,

11:36.585 --> 11:38.645
but in a similar way in the medical literature.

11:39.345 --> 11:42.725
Um, and there they find a pretty high false discovery rate,

11:43.455 --> 11:44.875
but they're not finding kind of numbers

11:44.875 --> 11:46.915
that I I'm about to tell you about.

11:47.975 --> 11:49.755
Um, so the best estimate

11:49.765 --> 11:54.565
after I've kind of generalized the model is that 41%

11:55.475 --> 11:57.655
of all rejected null hypotheses

11:58.235 --> 12:01.495
in these leading economics journals are false

12:01.855 --> 12:03.155
discoveries, right?

12:03.775 --> 12:05.075
I'm talking about studies that use

12:05.075 --> 12:06.475
credibility revolution methods.

12:06.975 --> 12:08.915
I'm talking about leading economics journals.

12:10.465 --> 12:13.385
I should not, I should add, okay, it's important,

12:14.015 --> 12:17.905
that I'm excluding some extreme T statistics.

12:17.905 --> 12:20.185
So ones with the T statistics that go for 10,

12:20.525 --> 12:23.505
and I exclude them because in most cases,

12:23.595 --> 12:26.065
those really aren't trying to test the all hypothesis.

12:26.065 --> 12:27.825
They're looking at trying

12:27.825 --> 12:29.425
to get a more precise parameter

12:29.665 --> 12:30.785
estimate or something like that.

12:31.745 --> 12:35.245
Now, even worse is, if I look at very narrow rejections,

12:35.715 --> 12:39.705
ones where the probability is right around 0.05 i

12:39.715 --> 12:42.865
where there's about a 5% chance of getting something

12:42.865 --> 12:46.935
that's extreme or larger, then we're talking about 65%

12:47.555 --> 12:49.175
of those being false discoveries.

12:50.675 --> 12:53.895
And first, to re really only have a 5% chance

12:53.895 --> 12:54.975
of a false discovery,

12:55.355 --> 12:58.135
we need a much larger T than we've been using.

12:58.675 --> 13:01.175
We would need a tee of about 5.48,

13:01.785 --> 13:04.005
and that would eliminate over 80%

13:04.105 --> 13:05.605
of the published rejected nulls.

13:06.265 --> 13:08.885
So I wanna be careful here, right?

13:09.765 --> 13:13.685
I I'm not suggesting there's any malpractice on the part

13:13.685 --> 13:15.525
of any economist or economist, right?

13:16.335 --> 13:19.625
This happens because a lot of economists think of a lot

13:19.625 --> 13:22.825
of interesting male hypotheses which they hope to reject,

13:23.245 --> 13:26.675
but most of which are true or approximately true, right?

13:27.215 --> 13:29.555
And when we reject, that's often the

13:29.555 --> 13:30.635
most interesting finding.

13:31.135 --> 13:34.295
So those are the ones that get published in the journals

13:34.295 --> 13:37.255
and we publish a lot of false discoveries, right?

13:37.555 --> 13:40.595
So one thing I don't find is that a lot

13:40.595 --> 13:42.955
of false discoveries are due to what's called P hacking.

13:44.005 --> 13:46.445
P hacking refers to the case where, well,

13:46.445 --> 13:48.245
you try your estimates a certain way

13:48.865 --> 13:51.685
and you don't get quite a, the T statistic

13:51.755 --> 13:54.285
with an absent value of above 1.96.

13:55.735 --> 13:58.235
So you don't quite make the 0.05 cutoff.

13:58.695 --> 14:00.515
So, which we like to use, you say, well,

14:00.515 --> 14:02.155
maybe I should limit the sample in this way.

14:02.865 --> 14:04.355
That doesn't quite get you there either.

14:04.815 --> 14:06.795
And you say, well, maybe instead

14:06.795 --> 14:08.995
of looking at the difference in years of education,

14:09.715 --> 14:11.355
I should look at the difference in

14:11.355 --> 14:12.595
high school graduation rates.

14:13.095 --> 14:14.635
And eventually you find something

14:14.635 --> 14:16.035
that's statistically significant.

14:16.615 --> 14:18.755
And if that happens, you would get a lot

14:18.755 --> 14:20.395
of false discoveries, right?

14:20.655 --> 14:23.315
But there are actually a bunch of tests out there,

14:23.315 --> 14:26.075
in the literature for looking at p hacking and I,

14:26.075 --> 14:28.875
and I find them, and I don't find any evidence of p hacking.

14:29.335 --> 14:31.675
Uh, in fact, I find some evidence that the shortage

14:31.695 --> 14:34.395
of T statistics just above 1.96.

14:35.195 --> 14:37.015
And frankly, I think that makes sense.

14:37.675 --> 14:39.775
Uh, a lot of the times we're going to be skeptical

14:39.795 --> 14:43.095
of our ability to reject an all hypothesis and formally

14:43.235 --> 14:45.935
or informally we're skeptical of findings

14:45.935 --> 14:47.775
that are barely statistically significant.

14:48.625 --> 14:51.245
And so those papers are less likely to get published.

14:51.545 --> 14:54.685
And so we actually find a shortage of two statistics just

14:54.685 --> 14:55.845
above 1.96.

14:55.945 --> 14:57.125
Let me start by explaining

14:57.125 --> 14:58.525
how I got interested in the question.

14:58.925 --> 15:01.985
I think that'll help. So somewhat surprisingly,

15:01.985 --> 15:03.665
because I'm not a development communist,

15:03.815 --> 15:06.745
I've served on a review panel for grants in policy

15:07.265 --> 15:08.705
oriented development economics

15:08.765 --> 15:10.585
for quite a number of years now.

15:11.945 --> 15:15.765
And the funder is very interested when we find something

15:15.765 --> 15:18.125
that seems to work in figuring out how

15:18.125 --> 15:19.245
to get people to use it.

15:20.505 --> 15:25.085
And so what happened is I got worried that we fund a lot

15:25.085 --> 15:26.525
of really good studies,

15:27.185 --> 15:30.235
but that means we're sometimes gonna find

15:30.235 --> 15:32.435
that policies work when they really don't work.

15:33.315 --> 15:35.335
And I want it to be able to tell the funder

15:35.435 --> 15:39.415
how confident should we be about promoting policies based on

15:39.415 --> 15:40.575
the studies that we funded.

15:41.965 --> 15:44.625
And so I think the implication is that we have to be

15:45.825 --> 15:49.795
much more careful about how strong the evidence has to be

15:50.765 --> 15:52.225
before we encourage people

15:52.445 --> 15:54.585
to adopt some policy or intervention.

15:55.215 --> 15:58.945
Okay? Now again, that's not only going to depend on

15:58.945 --> 16:01.865
how likely the result is to be a false discovery, right?

16:02.285 --> 16:04.585
It also depends on the cost of implementing the policy,

16:05.755 --> 16:08.245
both when it works and when it doesn't work,

16:08.465 --> 16:09.605
and how well it works.

16:10.415 --> 16:12.475
So, you know, if something is virtually costless

16:13.055 --> 16:15.035
and we have very weak evidence that it works,

16:15.735 --> 16:17.155
why not go ahead and do it?

16:18.265 --> 16:22.365
Um, but it would be great if we could figure out how to fund

16:22.585 --> 16:24.165
and value more replication.

16:24.165 --> 16:25.245
Studies and economics

16:25.825 --> 16:27.325
and economics to rewards

16:27.325 --> 16:29.245
for replications are usually pretty low.

16:30.695 --> 16:35.475
And it's also a case, excuse me, that by its very nature,

16:36.255 --> 16:39.075
the credibility revolution often favors studies

16:39.665 --> 16:42.315
that cannot be replicated in any fundamental sense.

16:43.335 --> 16:46.845
We can look at the effect of military service in other wars,

16:47.225 --> 16:48.645
but there's only one Vietnam.

16:50.005 --> 16:54.005
And even when we use randomized controlled trials, many

16:54.005 --> 16:57.725
of them are sufficiently expensive that we're not going

16:57.725 --> 16:59.365
to do many of them right?

16:59.535 --> 17:01.605
After a lot of digging, we found that moving

17:01.605 --> 17:04.725
to opportunity had positive effects on young children.

17:05.585 --> 17:07.685
Now, I, I will say, my gut tells me that

17:07.685 --> 17:08.965
that result is probably right,

17:09.955 --> 17:12.935
but the important point is we're not likely to get a bunch

17:12.935 --> 17:15.375
of other moving opportunity experiments,

17:15.715 --> 17:16.815
in the near future.

17:18.115 --> 17:20.375
So, so mostly I would say the implication

17:20.375 --> 17:21.495
is that we have to be cautious.

17:22.075 --> 17:25.095
We have to have a lot less confidence in calling.

17:25.565 --> 17:27.855
So-called significant results

17:28.755 --> 17:31.255
as clear policy implications

17:31.955 --> 17:34.855
and not being fooled by our own rhetoric, including

17:34.855 --> 17:36.535
by the term credibility revolution.

17:37.255 --> 17:39.635
Uh, I wanna be clear, I'm a big fan

17:39.635 --> 17:40.795
of the credibility revolution,

17:41.495 --> 17:44.115
but I think it's led to an ex excessive degree

17:44.115 --> 17:45.395
of confidence in our results

17:46.455 --> 17:48.835
and moving to the future, I'm working

17:48.895 --> 17:52.045
Right now hoping to get funding for it,

17:52.525 --> 17:54.165
focusing on development economics

17:54.365 --> 17:56.485
'cause that's what got me most interested.

17:56.905 --> 17:58.565
And it's an area where we have a lot

17:58.785 --> 18:00.925
of randomized controlled trials.

18:01.625 --> 18:04.405
So it will be interesting to see if these issues

18:04.405 --> 18:07.165
that I'm raising are more important

18:07.625 --> 18:09.965
or less important, in that area.

18:10.265 --> 18:12.125
But I don't have any answers for that yet.

00:00 - Historically, when economists tried to figure out whether something had an effect, say whether, going to Vietnam had an impact on your eventual earnings, what they would do is they would compare people who went to Vietnam with people who did not go to Vietnam, and they would try to look at people who looked similar.

00:25 - So maybe what they would do is to, look only at white people, who went to Vietnam and who didn't go to Vietnam, or maybe even white high school dropouts who went to Vietnam versus white high school dropouts who didn't. But we were never very good at being convincing that we had found all of the things that were different between the people who went to Vietnam and those who didn't. And so what the credibility revolution tried to do is to find ways to make that comparison better.

01:02 - So they tried to find somebody who looked even more similar to somebody who went to Vietnam. And one of the ways that we did that was to take advantage of something like in the case of Vietnam, the draft lottery. So the draft lottery was based on the day that you were born. So it might be that if you were born on January 5th, you had a low lottery number, which meant that you were very likely to be drafted.

01:30 - But if you were born on January 6th, you happened to have a very high lottery number and were unlikely to be drafted. And there's really almost no difference between the people who were born on January 5th and people born on January 6th. So that seems like a pretty good, pretty good comparison. And we could look at what was the difference in the proportion of people who went to Vietnam based on their birthdate and what were the differences in their outcome,

01:55 - and figure out the impact of going to Vietnam. Now, that example is a little bit more precise than we've ever been able to do, but it's the basic idea, the credibility revolution basically popularized a set of techniques that compare people in situations where they seemed like they were pretty nearly randomly assigned. In the best cases, they actually were, they actually ran experiments just the way physicians do. so we might in a situation where

02:26 - the training program was oversubscribed, randomly assigned, who of the people wanted to go into the training program, got into the training program, compare outcomes. There, there are a bunch of other techniques in the interest of time. I don't think I should go into them, but basically it's about it finding ways to do a better job of comparing apples to apples into apples oranges. The critical issue that I wanted to address is, let's take one of these very well

02:53 - executed credibility revolution studies, and it says that, you know, some policy is effective, say, I don't know, for example, maybe giving students more tutoring has a significant effect on their earnings after they leave high school or secondary school to be more international. how confident should we be that the policy actually works? That's the question I want to answer. Now, the way economists normally think about this is

03:19 - to begin with what we call the null hypothesis, right? In this case, the null hypothesis is at the effect of giving the extra tutoring on later earnings this year, and it has no effect on outcomes, right? And normally what the economist is going to say is, suppose I do find an effect. How likely is it that if there were no effect, I would actually find an effect that the, that's this large. So to give a a simple example, you know, think about tossing a coin a hundred times.

03:53 - I won't always get 50 heads. Sometimes I got 48 heads, sometimes I get 52 heads, right? The same thing can happen if you're tutoring people. Sometimes kids you happen to randomly sign to get tutored better than the ones that you happen to randomly sign not to get tutoring, or they just happen to get lucky on the, on the test, they guess some multiple choice questions, right? and so what we wanna ask is how likely, if there really was no difference, would I find an effect

04:24 - that's this large, that thing's called the P value? And we generally say that the result is significant if the probability of the event happening by chance is less than 5%, right? So in that case, we would say that the effect that we found of tutoring, that we found a probability that it has no effect is sufficiently unlikely that we would reject the hypothesis that it has no effect. And as a result, since the alternative to having no effect is that it has an effect, we implicitly,

05:01 - or I would say often explicitly conclude that the tutoring did have an effect, right? And that in turn would point to supporting a policy of implementing tutoring. I wanna be careful here, right? There are other considerations like cost, right? but now at least we know that tutoring is effective. So, so the question that I ask is how likely is it that tutoring is effective? And notice that's a different question, right?

05:30 - Normally what we ask is, suppose tutoring doesn't have an effect. How likely is my result? Right? We never ask, assume that the tutoring does have an effect. How unlikely is my result? And we need to know both of those, right? So the real question is how likely is the null hypothesis to be true? And let me see if I can give you an extreme example. but I think a helpful one in terms of the intuition.

06:01 - Suppose I have a coin, right? And I know, or at least I believe that it's either a fair coin produced by the US mint, or it's a fake coin with two heads. Right? Now, if it's produced by the US mint, and if I flip it 50% of the times, it'll give heads 50% of the time it'll give me tails. Of course, if it's a fake coin, I always get two heads. But let's not worry about that yet. Now, as an economist, I start with a null hypothesis

06:29 - that it's a two headed coin, right? So I flip it some number 20 times, let's say, and I get, let's say 17 hits. if the coin's truly fair, that's a really unlikely outcome, right? And since my null hypothesis is that the coin is fair, I have to reject it, right? Because the outcome that I've observed is unlikely under the null hypothesis. And since my only choices are either that it's a fair coin

06:56 - or it's a two-headed coin under the traditional way of thinking about it, I reject the null and accept the alternative, which is that it's a two-headed coin. And that, of course is ridiculous because I couldn't possibly get even one tail, let alone three, if it were a two headed point. So the question I wanna ask is how likely is the outcome under the alternative versus how likely it is under the null?

07:24 - And we also have to ask, you know, originally when I started out, how likely did I think the alternative and the, and the null were right? Because if I were totally convinced that the coin was fair, right? It would take a lot of evidence to to change my mind, right? So if I thought it was pretty likely that it was unfair, then it wouldn't take so much evidence. So I also have to take that into account

07:47 - and putting those two things together, I can calculate the probability that the null hypothesis is true. There are two things that I need to know in order to decide whether null hypothesis is true or not. That's what I was just talking about, right? How likely is the outcome that I observed not only under the null, but also under the alternative, and also how likely was the null a priority? What did I think it was, right?

08:15 - There's a formula for putting those things together, and it's called the base there. So the goal of my research is to take the body of empirical research that broadly fits into the credibility revelation and ask how likely is it that a rejected null that's been published in a good journal should not have been rejected? Another way of putting it is what's the false discovery rate, right?

08:41 - And what particularly what is the false discovery rate and empirical economics that draws on the credibility revolution? So the way I proceed is I think of economists as producing results from two different situations. In the first situation, they're actually testing hypotheses that are true. And in that case, I'm gonna use our most common measure of how likely the finding is under the null, which is what we call the T statistic.

09:12 - And in the situation where the null is true, the t statistic has the T distribution. So it's very easy. And for those people who are not familiar with the T distribution, you can think of it for most purposes as having a normal distribution centered on zero and a standard deviation of one. So that's, that's some fraction of the null hypotheses that are being tested. And then there are the null hypotheses that are false. And the average t statistic we're going

09:42 - to find in those cases is gonna depend on how large the true effect is and how precise our experiment is. So the more data we have, everything else equal, the higher the T statistic will tend to be, and that t statistic turns out we'll have something that's called the non-central T distribution. So what I then do is I assume a fairly general raw distribution from which those expected Ts that are the central point

10:10 - of the non-central T distribution, like the, the distribution from which they are drawn. And then my objective is to measure the proportion of hypotheses that economists study that are true. In other words, the ones that should not be rejected, and the distribution of TT statistics, the false nulls that should be rejected. Okay? So to do this, had my research assistant collect data from a set of leading journals, I have to give her credit,

10:42 - they read the paper sufficiently carefully to identify the core null hypothesis. And, and that's important, right? 'cause we didn't include robustness, checks, heterogene analysis unless they were core to the paper, right? And we then took those data and found the parameters of the model that best fit the empirical distribution of the two statistics.

11:07 - Now, I've kind of simplified things. The final model is a little more complicated than what I just described. For example, I allow the null hypothesis to be only approximately true. So that if you know, the null is zero is true, even if the, it's just a little bit different from zero. but I think I've described what capture the spirit of what I do. This work has been done,

11:34 - I think in a less fully convincing way, but in a similar way in the medical literature. and there they find a pretty high false discovery rate, but they're not finding kind of numbers that I I'm about to tell you about. so the best estimate after I've kind of generalized the model is that 41% of all rejected null hypotheses in these leading economics journals are false discoveries, right?

12:03 - I'm talking about studies that use credibility revolution methods. I'm talking about leading economics journals. I should not, I should add, okay, it's important, that I'm excluding some extreme T statistics. So ones with the T statistics that go for 10, and I exclude them because in most cases, those really aren't trying to test the all hypothesis. They're looking at trying to get a more precise parameter

12:29 - estimate or something like that. Now, even worse is, if I look at very narrow rejections, ones where the probability is right around 0.05 i where there's about a 5% chance of getting something that's extreme or larger, then we're talking about 65% of those being false discoveries. And first, to re really only have a 5% chance of a false discovery, we need a much larger T than we've been using. We would need a tee of about 5.48,

13:01 - and that would eliminate over 80% of the published rejected nulls. So I wanna be careful here, right? I I'm not suggesting there's any malpractice on the part of any economist or economist, right? This happens because a lot of economists think of a lot of interesting male hypotheses which they hope to reject, but most of which are true or approximately true, right? And when we reject, that's often the most interesting finding.

13:31 - So those are the ones that get published in the journals and we publish a lot of false discoveries, right? So one thing I don't find is that a lot of false discoveries are due to what's called P hacking. P hacking refers to the case where, well, you try your estimates a certain way and you don't get quite a, the T statistic with an absent value of above 1.96. So you don't quite make the 0.05 cutoff. So, which we like to use, you say, well,

14:00 - maybe I should limit the sample in this way. That doesn't quite get you there either. And you say, well, maybe instead of looking at the difference in years of education, I should look at the difference in high school graduation rates. And eventually you find something that's statistically significant. And if that happens, you would get a lot of false discoveries, right?

14:20 - But there are actually a bunch of tests out there, in the literature for looking at p hacking and I, and I find them, and I don't find any evidence of p hacking. in fact, I find some evidence that the shortage of T statistics just above 1.96. And frankly, I think that makes sense. a lot of the times we're going to be skeptical of our ability to reject an all hypothesis and formally or informally we're skeptical of findings that are barely statistically significant.

14:48 - And so those papers are less likely to get published. And so we actually find a shortage of two statistics just above 1.96. Let me start by explaining how I got interested in the question. I think that'll help. So somewhat surprisingly, because I'm not a development communist, I've served on a review panel for grants in policy oriented development economics for quite a number of years now.

15:11 - And the funder is very interested when we find something that seems to work in figuring out how to get people to use it. And so what happened is I got worried that we fund a lot of really good studies, but that means we're sometimes gonna find that policies work when they really don't work. And I want it to be able to tell the funder how confident should we be about promoting policies based on the studies that we funded.

15:41 - And so I think the implication is that we have to be much more careful about how strong the evidence has to be before we encourage people to adopt some policy or intervention. Okay? Now again, that's not only going to depend on how likely the result is to be a false discovery, right? It also depends on the cost of implementing the policy, both when it works and when it doesn't work, and how well it works. So, you know, if something is virtually costless

16:13 - and we have very weak evidence that it works, why not go ahead and do it? but it would be great if we could figure out how to fund and value more replication. Studies and economics and economics to rewards for replications are usually pretty low. And it's also a case, excuse me, that by its very nature, the credibility revolution often favors studies that cannot be replicated in any fundamental sense.

16:43 - We can look at the effect of military service in other wars, but there's only one Vietnam. And even when we use randomized controlled trials, many of them are sufficiently expensive that we're not going to do many of them right? After a lot of digging, we found that moving to opportunity had positive effects on young children. Now, I, I will say, my gut tells me that that result is probably right, but the important point is we're not likely to get a bunch

17:12 - of other moving opportunity experiments, in the near future. So, so mostly I would say the implication is that we have to be cautious. We have to have a lot less confidence in calling. So-called significant results as clear policy implications and not being fooled by our own rhetoric, including by the term credibility revolution. I wanna be clear, I'm a big fan

17:39 - of the credibility revolution, but I think it's led to an ex excessive degree of confidence in our results and moving to the future, I'm working Right now hoping to get funding for it, focusing on development economics 'cause that's what got me most interested. And it's an area where we have a lot of randomized controlled trials. So it will be interesting to see if these issues

18:04 - that I'm raising are more important or less important, in that area. But I don't have any answers for that yet.

Faculti AI can make mistakes