In this way, replicability is at the heart of the grand project to give everyone a shot at a decent life.

But in economics, and the social sciences generally, replication doesn’t happen much. Why? No one ever won a Nobel Prize by copying others. And as I think John Kenneth Galbraith once observed about competition, people like replication until it happens to them.

But times are changing. The World Wide Web was invented in 1990 to help researchers share data, and it is slowly shifting norms about openness and collaboration. Communities converge faster to agreement, dare I say the “truth,” when analysis is public, collective, and iterative. Back when data were stored in punchcards and scientific discourse took place via ungainly bound volumes, statisticians could easily bar outside observers from their private digital laboratories. Now expectations of transparency are much higher.

So here’s a cool thing. The International Initiative for Impact Evaluation (3ie) is seeking nominations by May 31 for impact studies that it should pay to have replicated. 3ie was born out of a recommendation in a CGD report led by Ruth Levine, Nancy Birdsall, and Bill Savedoff; its mission is to support new, high-quality research on what works in development, and at what cost. The just-launched replication program, on whose advisory board I serve, embodies an astute recognition that 3ie can also contribute by rigorously reexamining research done by others.

Are there studies in your field that you think are influential enough to deserve reexamination—and confirmation, contradiction, or something in between? How strong is the evidence base for the idea that girl education is an investment with particularly high returns? How sure should we be that, roughly speaking, men spend all their money on beer while women invest in the children? Or that financial sector expansion increases economic growth?

“Replication” has a range of meanings. I could replicate the Hyderabad microcredit impact study by performing a similar experiment in Mumbai, what 3ie calls “external replication.” That is important, but not what 3ie has in mind here. Or I could field my own survey team in Hyderabad to collect and analyze new data on the impacts of the microcredit experiment there, what psychologist John Hunter called statistical replication. Or, since I am lazy and lack managerial prowess, I could download the original study’s data and reanalyze it, for a “pure replication.” 3ie looks to support work of these latter kinds.

Actually, I’d draw a further distinction within the “pure replication” category. There is what I consider true, pure replication, which strives to exactly reproduce a published analysis in order to scrutinize it. Partly for lack of imagination, that’s what I usually do, as in my work with Jonathan Morduch on the Pitt & Khandker microcredit impact study. And there is what I could call reanalysis, which applies different methods to same data, as Jonathan did for Pitt & Khandker in 1998 and as Maren Duvendack and Richard Palmer-Jones (who also advises the 3ie program) did in 2011.

Perhaps the biggest challenge will be persuading the authors of the chosen studies to share their data. Should that problem arise, I hope it will be solved by the public exposure generated by this exercise.

Importantly, the expectation is that results will be posted regardless of outcome. Everyone loves a conflict, but confirmations matter as much as contradictions.

Send your nominations to replication@3ieimpact.org. But surely the list should be crowdsourced, so I encourage you to post your ideas below as well. And if you want to achieve riches by replicating (any starving grad students out there?), stay tuned.

One of the useful things the federal government does for the economy is produce information as a public good. And the American Community Survey is chock full of information that’s useful to researchers, companies, curious individuals, policymakers at different levels of government, etc. But House Republicans have decided that they want to kill it, and it seems clear that some of them have a passion for the cause that completely exceeds their understanding of the issue. Representative Daniel Webster, for example, is a sponsor of the anti-ACS survey in part because he thinks $70 per survey respondent is “not cost effective … especially since in the end this is not a scientific survey. It’s a random survey.” [Emphasis added.]

The other is on the site of the Whole Planet Foundation (you have to dig a bit on this page or go here). The Whole Planet Foundation’s mission is “poverty alleviation through microcredit in communities worldwide that supply Whole Foods Market stores with products.” I gather that when you check out at Whole Foods you can donate your change and then some to the foundation.

Reading the latter one, by Executive Program Director Steve Wanta, led me to reflect on what can make a review persuasive. I thought of two things. One is eminence. The reviewer’s achievements and experience and position—who he is—can make him credible when he describes a book as insipid or brilliant. I think of Amartya Sen’s review of Bill Easterly’s second book and Bill Clinton’s take on Robert Caro’s latest opus on LBJ. But that avenue is not open to Wanta, not so much because he lacks the stature of Sen or Clinton, but because he has a professional, vested interest in what I critique. Who he is should make him compelling only to the converted.

The other avenue is evidence. I.e., backing up statements like “Roodman does not demonstrate the same statistical rigor for the potential adverse effects as he does for the positive effects of community building reported by the industry” with specifics. I tried to do something like what I have in mind in my biting review of Dambisa Moyo’s Dead Aid, which the Whole Planet Foundation deems “Compelling reading and a must read for anyone interested in African development.” Or see Alex Counts’s thorough examination of my book, which I still think is the best so far. At any rate, I think the Whole Planet review of my book does not persuade through either eminence or evidence.

From the outside, KGFS seems like alchemy. No longer is a single, low-quality financial service like group microcredit mass-produced for the poor. Instead, poor people get customized financial services the middle class might envy. A client walks into a branch and meets with a KGFS employee who inventories her income and spending, assets and liability, worries and goals, punches them into a computer, and prints out a recommended portfolio of services that may include loans, insurance, savings, and even retirement savings. KGFS then provides all the recommended services, either on its own account, as with loans, or as an agent, as with insurance. The financial assessment, and presumably the service portfolio preferred, is updated every six months. Somehow, despite all this custom service and the forays into services most others have lost money on, KGFS turns a profit. When Nachiket Mor, who I think is the driving force behind IFMR, described it at a CGAP meeting last December, someone in the audience aptly commented that she would like access to something that good.

There is much that is refreshing about KGFS: taking client needs as the starting point; using of high tech to cut the cost of assessing and meeting those needs; partnering with insurance companies and other financial titans, with KGFS the retail agent; and delivering a set of services. The big question in my mind has been whether it can pay for itself and grow.

So far it seems to have. CGAP has just put out a report and a video interview with Bindu.

From the report, I learned that in the last few years the IFMR Trust has licensed the KGFS system to five independent for-profit entities, which have raised all their capital from private investors. Together they have reached 200,000 people. Even more impressive is that mature branches, ones approaching three years in age, reach nearly 70% of people in the vicinity, or “catchment area.”

I commend both the report and the video to you as introductions to the KGFS model. They are clear, accessible, not too long. Having met and talked shop with Bindu, as well as her CGAP coauthors on the report, Greg Chen and Stephen Rasmussen, I have great respect for judgment and intelligence of all those involved.

That said, you should recognize that these materials do not represent an independent inquiry into KGFS, which is what I, as someone already familiar with the model, was most hoping to see. Both are clean presentations of the KGFS story as the KGFS people want to tell it. That’s fine as long as it’s seen for what it is.

At this point, I have a couple of critical questions not confronted by the new CGAP report. Writing for CGAP, Rich Rosenberg has warned that demand for microcredit may be lower than is often assumed when people talk about the billions who still lack access to it. If forced to pick a number, I bet he’d estimate the typical demand at closer to 10% of poor households than 70%. Yet according to the new report, 87% of the 70% of people reached by a mature KGFS branch within its catchment area take loans. That works out to 61% of the population, way higher than the statistics we’ve seen for national microcredit penetration anywhere else. Should we worry about this spread of indebtedness? A lot of the loans are group loans, presumably not unlike those in traditional microfinance.

In the same vein, which products generate the profits? I think it’s fantastic that customizing multiple services to meet actual needs is hard-wired into KGFS. Yet if some products make money and others lose it, that generates incentives that are hard to ignore, and could easily push the practice, whatever the theory, toward credit. The “stylized branch income and expense statement” on page 14 of the report oddly displays net income from lending (so we see it’s profitable) and gross income from everything else (so we can’t tell what in the array of other services is profitable and what not).

I am also curious about who is investing in KGFS, and how many would consider themselves “social investors.” Beth Rhyne has argued that a core cause of the Andhra Pradesh debacle is India’s prohibition on non-profits owning parts of for-profits. So when Indian microcreditors went for-profit in order to raise capital and grow, mission-driven actors were structural excluded, unable to counterbalance the drive for profits. What does that imply for KGFS?

Not that I have reason to be deeply suspicious of KGFS. Indeed, I assume Greg and Stephen only lent their names to this report after serious, behind-the-scenes due diligence. But KGFS is so striking a departure from the past, so potentially important, that it deserves close, independent scrutiny, which would not be coauthored by one of the promoters. The model I have in mind is MicroSave’s tracking (with DFID funding?) of the roll-out of Grameen II a decade ago. They hired Stuart Rutherford to study it from the client point of view, and I assume gathered other kinds of evidence through examination of financial records and interviews with staff from Muhummad Yunus on down. The result was a credible, penetrating, balanced analysis with important lessons for the rest of the microfinance world.

Update: I forgot to mention how great it is that the IFMR Trust has commissioned a randomized evaluation of KGFS, to be led by Rohini Pande and Erica Field. Especially exciting is that the experiment will run for three years instead of the usual 12–18 months, giving us a glimpse of longer-term impacts.

The powers in Bangladesh did not appreciate Hillary’s contribution. Two top officials has gone after Clinton and Yunus, equally publicly. The rather less-famous finance minister who helped Yunus found the Grameen Bank, opined:

“I think this comment is very unwarranted,” finance minister AMA Muhith told reporters.

“Grameen Bank is a government institution. The government has created it and brought it thus far. It is because of the government that Mr. Yunus could come this far.”

Syed Ashraful Islam, general secretary of the ruling political party and Minister for Local Government and Rural Development & Co-operatives, attacked Yunus in the presence of his Prime Minister:

The country cannot be developed through multilevel marketing companies and NGOs based on illegal money from abroad. Microcredit won’t change our fates and it won’t be able to change Bangladesh in the next 100 years.
…
His basic subject was Economics. But he did not get the prize for Economics. He got the Nobel prize in Peace though he did not play any role in stopping a war.

Now, many people in our country know how to ‘get a Nobel prize’.

If you think about it, this is a pretty stark spectacle. It is hard for me to imagine two members of Obama’s cabinet attacking a single citizen in public, even ridiculing him in front of the President—much less one so highly celebrated. It’s clear that governance in Bangladesh, not just politics, is war by other means. I hope that as regards the safety of Yunus and the Grameen Bank staff, and the integrity of the institution, the threat from the government is offset by the equally unusual endorsement from a powerful foreign official.

He has conceded to me the argument over what the economic studies say. So his main thesis now is:

I think David, in his faith in the power of numbers, has strayed too far from the path of Common Sense. Example: If, as David concludes, there is value to be found in the creation of an industry that has corrected—for many millions of people—the “market failure” of financial systems in Developing Countries, is it logical to then conclude that the impact on global poverty of this gargantuan investment has been zero?

Are we to believe that joining millions of people who earn their living each day in the informal sector, and who previously had no access to credit or savings services, and now do, are not materially better off as a result of it? I think these two conclusions marry up like drunkards in a Las Vegas wedding. Let’s get them across the border to Mexico for an annulment.

He then tells a couple of good stories of encounters he has had with borrowers, the first being of a borrower who says she climbed from poverty thanks to microcredit. He concludes:

David would say “No causality!” Rupert would say “Hey, listen to the woman! Maybe she’s not making it up!”

I’d respectfully correct a few things Rupert says about me:

• The thrust of my analysis of the econometric impact literature is actually about my lack of faith in the power of numbers—about why I doubt most microfinance impact studies. That said, Rupert’s description is fair to the extent that I do think the results of the new, experimental studies need to be taken seriously, and that in my judgment they trump small, ad hoc assemblages of stories.
• I don’t hate stories. Chapter 3, 4, and 7 include lots of them, and not for the purpose of disparagement. I know that people understand and explain the world through stories. I do think though that in talking of microfinance, in particular of whether it should receive charity, we should tell stories that are representative. And we should judge representativeness through rigorous analysis of evidence of both the narrative and numerical kinds. I don’t see Rupert doing that. I would say “Listen to that woman, and to a bunch of other women who are selected in some transparent and representative way.”
• I do not conclude that the impact of Rupert’s career on poverty has been zero. I conclude, rather, that zero is the best estimate given limited rigorous evidence of the impact of microcredit on the income and spending levels of clients. For more see Why Celebrate Institutions that Deliver a Service that Doesn’t Reduce Poverty. Yes, that is a pretty serious challenge to Rupert’s work, especially since I think it should affect funding for microfinance. But I’d like to avoid exaggerations of my certainty that imply sloppy reasoning.
• Just for the record, I don’t describe microfinance as reducing “market failure.” The term is jargon and it elevates an unrealistic economic abstraction to the level of an ideal.

A last point: as lovely as Rupert’s metaphor is, the one about drunkards in matrimony, I don’t think it applies. It doesn’t violate common sense to suggest that a massive ramp-up in a lending industry could hurt people.

It’s clear from the work done so far (led, I believe, by Asli Demirguc-Kunt, and co-authored by Leora Klapper), as well as from the outreach effort, that the World Bank was the right institution for the job. The data set appears to be of high quality. Ditto for the “metadata,” which tell number-crunchers like me exactly how variables are defined and stored. The project’s home page offers an almost bewildering slate of options for exploring the data. The technical paper is quite readable and contains lots of good figures and tables. CGAP’s main blog is running a series on the implications of the new data. (Don’t miss Jonathan Morduch’s piece reconciling the macroview from this data set with the micro view from Portfolios of the Poor.)

I have to admit I’m not very excited about this database. Probably that’s a reflection on my lack of imagination as a researcher. While it’s true that I didn’t know before that 52.9853% of adults in Afghanistan who have an account at a formal institution make 1–2 withdrawals per month, and that 67.14356% make 1–2 deposits, I’m likely to forget those facts as fast as I type them. Meanwhile, the big picture is unsurprising. Half the world is still unbanked. And, what do you know, more people have bank accounts in countries where more people have money.

But even if I don’t feel the excitement in my gut, my head tells me this is important. Here’s why. This graph shows the relationship between one of the dataset’s key variables, % of adults with an account at a formal financial institution, and a standard measure of national wealth, gross domestic product per person. The relationship is strong. And it is S-shaped, which makes sense since no matter how poor a country, its financial inclusion rate can’t fall below 0, and no matter how rich, it can’t rise about 100%. Each dot in the graph represents a country, and I’ve labelled the dots with 3-letter country codes (list here), which don’t collide with each other on the screen as much as full country names would. So that’s Burundi, Liberia, and the former Zaire in the impoverished bottom left and Singapore and Luxembourg in the affluent upper right:

I ran a few regressions, looking at how much of the cross-country differences in this measure of financial inclusion can be explained by such factors as GDP/capita, % of the population that lives in cities, and what continent a country is on. The answer: about two-thirds. To this extent, financial inclusion appears to be driven by factors beyond the influence of well-meaning outsiders such as the Gates Foundation and David Roodman’s Microfinance Open Book Blog. As I wrote in my book without much originality, financial inclusion appears to be more a consequence than a cause of rising wealth.

But two-thirds is not all. Some countries, such as Bangladesh and Kenya, score much higher on financial inclusion than their GDP/capita peers. Their dots are well above the smooth red line in the graph above (look for BGD and KEN). Others, necessarily, are below average. This suggests that a country’s level and quality of financial inclusion are not beyond intentional influence. There may be substantial room for domestic policymakers and foreign actors to boost financial inclusion in many countries. In fact, data like this may provoke such change, especially when it comes in the form of comparisons between neighbors. (India (IND), why despite the largest financial inclusion drive in the history of developing countries are you so far behind Sri Lanka (LKA)?) Apparently that’s part of what happened in Kenya: financial inclusion data from FSD Kenya reached the eyes of the central bank governor and made him more open to giving the nascent M-PESA some regulatory leeway. Think how many insouciant comparisons can be made now.

I should add that we don’t have to just guess as to why people in some countries are less apt to open formal financial accounts. The pollsters asked them. From page 19 of the technical paper:

There is much more to the data set. Pollsters collected demographic information—income, sex, education level, etc.—so many of the results are broken out along these dimensions. Informal services, such as moneylending and savings clubs, also get attention. All in all, it is an impressive piece of work. OK, I admit it, I’m starting to get excited about it.