Endings and Beginnings

So after almost a year, this will be my last post on this blog.  Now, don’t worry, I’m starting a new blog (here), so you can continue to hear about my musings on various topics.

So why end one blog just to start another?  Well, I’m currently in a moment of transition, of endings and beginnings, and felt that now is the right time to shake up my blog as well.  After almost 2 and a half years, I’ve left Guatemala, and will be undertaking a new adventure in London as Program Officer for Impact and Learning with the Transparency and Accountability Initiative.  In addition, I just defended my Ph.D. dissertation, so now I’m Dr. Brendan Halloran.  Both of these transitions entail going from a more specific set of issues and questions that I’ve been grappling with in this blog to a broader array of matters.  My new blog will reflect these wider interests.

So please follow the next iteration of my blog and see what new ideas I’ll be thinking about moving forward.

Throwing a rock at Evidence?

Originally posted at www.politicsandideas.org by Nilakshi De Silva on the challenges of data, evidence and development policy & practice:

I always feel a little uncomfortable when we refer to research findings as ‘evidence’. The term is very close to the idea of ‘proof’, and conjures up images of court rooms and lawyers presenting Exhibit A with a lot of fanfare to prove the case beyond any shadow of a doubt. This image is a far cry from development research and how it actually happens. We work in the imperfect real world, which is changing even as we study it. Researchers are also hampered by funding and other constraints, and research designs are often the best we can manage under these multiple constraints. Yet, even if we do manage to design the best possible research design, how much of what we do is really, beyond any doubt, ‘the truth’ – indisputable and constant?

One of the criteria by which we judge the quality of research is generalisability or the extent to which the findings may be applied to places and cases which have not been studied.  For example, can we use the findings from an impact evaluation of an employment generation scheme in Colombo and apply it to other cities and towns in Sri Lanka? In theory, of course we can – especially if the study has provided for impact heterogeneity and can show effects on different sub groups and types. Yet, there is a need to be cautious. The evaluation is limited to a specific time and place, and the world moves even as we speak; what worked in this place may not work even in this same place in a few years´ time, let alone in other non-studied locations. In Sri Lanka for example, there is no other place that is exactly like Colombo, though some of its characteristics can be seen elsewhere. The city is also undergoing palpable change and many who return to it after a few years find it has transformed, sometimes beyond recognition. Would a program that performed successfully here 10 years ago, still work? It is hard to say, which is why a good evaluation needs to tell us, not just about what worked here, but also identify under what circumstances.

Evidence of what worked in a particular place and time doesn’t travel well for another reason. Pilot interventions on which the research findings are based may be implemented in a certain way, which is often not replicated (and sometimes not replicable) in a scaled up version of the same intervention. Important checks and balances may be missing in the larger version, and we find that the impact that was seen in the studied intervention is not there under the scaled up policy that is implemented. This does not mean that the idea on which the intervention was based is a bad one, but too often this kind of outcome ends in throwing out the idea as bad policy based on bad evidence.

In our conversations about evidence – based policy making, we often ignore the contested concept of ‘knowledge’.  I was at a discussion forum on post structuralism recently where the question of whether there is anything that can be termed ‘objective’ knowledge came up. Derrida for example suggests that all knowledge is subjective, in that there is someone who goes out, interacts with the objects of the study, gathers information and subjects it to some interpretation. Knowledge is not, and he suggests it never can be, directly that of the object that is studied, but is always shaped by the object coming into contact with the person who studies the object and which is interpreted by the researcher. Many of us, who are long past our university days, may no longer engage with the philosophy of science but these ideas are critical to what we do every day.  Knowledge is our business, but do we really understand enough about how knowledge (a.k.a evidence) is produced? In other words, as development researchers, it is not enough that we are development specialists, we also need to understand what words like ‘knowledge’ and ‘evidence’ mean.

From this perspective, a healthy skepticism is a good way to approach evidence to inform policy and practice. Our knowledge is always time bound and context specific. Rather than jumping through increasingly more complex hoops in an effort to make the evidence water tight, a more useful way forward may be to focus on how we recommend that this knowledge be used.

What does all this mean, if anything, for research influencing policy? I think it means that we need to be very careful about the policy recommendations we make and how the replication and scaling up are rolled out. Whatever the evidence on which the policy is based, the implementation of the policy still needs to be carefully crafted to monitor if it is working, what is not working and adjust and adapt as you go. If we accept that there are limitations and uncertainties in the knowledge that research produces, this becomes even more important. There are no guarantees, and this needs to be borne in mind and incorporated into the recommendations as well as scale up plans. In the end, this may very well mean that the seed of the idea may remain the same as that from which the evidence was drawn, but the intervention / policy that is implemented is more grounded in the context to which the policy applies. A good example are conditional cash transfer programmes, which are implemented in many parts of the world, but which often have different designs which take into account local realities.

As development researchers, our work is driven by the implicit goal of changing the world for the better. The underlying premise is that we can ‘know’ what works and what doesn´t, by carefully designing and carrying out research studies, which can then provide ‘evidence’ for better policies and practice. While this version of the world is naturally attractive and helps to inspire and validate our work, there is also a need to be real. We study a complex world, in which intricate relationships are continuously interacting in multiple, complex ways.  This understanding should not cripple us or alternatively, push us towards over simplifying the world in order to be able to study it. Rather it should enrich our work and help development research to be more truly a useful service.

Some thoughts on tackling corruption

I’ve been seeing a few things coming up on corruption lately, so I thought I would try to pull them together with some thoughts of my own.  A recent post on the World Bank’s “People, Spaces, Deliberation” blog discusses the state of global corruption.  The author notes that there are some indications that global corruption has increased over the past several years.  At the same time, however, there are a number of more promising trends in the struggle against corruption, including:

  • Moves by rich countries to target tax havens
  • A multitude of novel approaches to addressing corruption around the globe
  • Perhaps most importantly, a new generation of social movements, largely middle class, throwing off the resignation of corruption as ‘business as usual’ and taking to the streets in places like India, Brazil, Turkey and elsewhere

The article concludes by noting that international organizations seeking to address corruption are more effective when the support these existing efforts, than when they try to create such initiatives from nothing.

However, in order to address corruption, we must know must know where it exists and how it impacts citizens.  Alex Cobham of the Center for Global Development argues that the most widely-cited corruption indicator (which is also utilized by the World Bank blog’s author), Transparency International’s Corruption Perceptions Index (CPI), is deeply flawed.  He observes that the CPI draws on too few ‘expert’ opinions and has too much room for bias, and instead calls for ‘barometers’ that draw on a diverse population and utilize a number of questions that reveal citizens’ experiences of corruption in each country.  While citizens certainly experience demands for bribes and other direct expressions of corruption, they may not have a full understanding of the depth of corrupt practices that occur behind closed doors.  Corruption is often a hidden practice, after all.  Thus, a broad survey of citizen perceptions could be buttressed by a more robust expert analysis, perhaps along the lines of the Global Integrity Report, in order to guide policy makers towards the most prevalent, and pernicious, forms of corruption in each country.

Finally, once the dynamics and sources of corruption have been established, international organizations need effective tools to combat it.  As mentioned, building off existing efforts and movements is key.  Too often, international actors have focused exclusively on helping countries strengthen formal laws and institutions to address corruption.  Yet corruption persists.  According to the Global Integrity Report, Guatemala’s legal framework for combating corruption scores 90 of 100, while the implementation of those laws rates at 43, a massive gap between form and function.

Over the past several years, however, research and analysis has revealed improved principles for catalyzing meaningful institutional change through foreign aid.  The Fragile States Research Center highlights several recent books, articles and reports that argue for a new thinking about how to support institutional reform.  The authors of these pieces agree that promoting real institutional change involves focusing on the informal aspects of institutional functioning, such as organizational culture, power relationships, incentive structures, etc.  The authors note that international aid agencies have struggled to put these new insights into practice, despite new analytical tools to examine political economy factors, for example.  Thus, the challenge remains for development decision-makers and practitioners to incorporate the latest learning in order to help support what may be a rising tide against global corruption.

Macro Context and Micro Dynamics: Research with Policy Implications

I recently reviewed a research proposal for an ethnographic study of the dynamics of violence and the responses by local actors in a low-income barrio in Guatemala City.  The methodology was focused on seeing violence through the eyes of neighborhood residents by means of observation and interviews.  The proposal made some mention of examining the macro-social contextual factors that affected the area as well, such as economic and political marginalization, which are generally understood to be drivers of violence in such communities.

In my comments, I suggested that the researchers focus much more on this last element.  I mentioned that there existed a number of studies such as the one they were proposing, oriented towards understanding the character of violence in vulnerable neighborhoods in Latin America.  Researchers have also studied the types of responses local actors have carried out in these situations.  However, the linkage between underlying structures of economic, political and social exclusion, on the one hand, and local dynamics of violence, on the other, have not been as fully explored, at least not empirically.  This, I argued, would be the most policy-relevant area of potential investigation: understanding the relationship between macro-structures and micro-dynamics.

As long as the linkages between structural exclusion and neighborhood violence remain murky, efforts to promote citizen security will continue to focus on the violence itself as the cause of security problems, rather than a symptom of an underlying cause.  Efforts to address violence often treat violence as a characteristic of an individual, or possibly group, rather than an element of the structures in which those actors are imbedded.  If we could better understand the relationship between structural exclusion and violence, we could better address the underlying issues that were making individuals and communities more prone to violence.  That would be a step forward in the journey towards addressing spiraling levels of violence in many cities throughout Latin America.

In my own research on vulnerable urban communities in Guatemala, I have attempted to delve into this connection between macro and micro, particularly concerning political dynamics.  In Guatemala, the nature of political clientelism, a much-studied phenomenon in Latin America, is more short-term and transactional, better described as vote buying.  Political parties are too weak and fragmented to build and sustain the durable forms of clientelism associated with parties like Mexico’s PRI and Argentina’s Peronist Party, which have endured for generations.  Rather, parties focus their resources on more simple and direct exchanges of material goods for votes.

On the ground, I found that parties interfaced with community leaders in an attempt to secure votes.  They deployed promises, both of future benefits for the neighborhood and for special consideration for local leaders (often cash or a commitment to provide a government job), building materials such as bags of cement or corrugated metal sheets, or a tangible physical project in the community.  This last option was actually quite rare, as the cost to the party often outweighed the benefit.  Voting is secret and there were over a dozen parties to choose from, so political operatives were hesitant to commit significant resources in an uncertain exchange.

The impacts of the macro-political context were several.  First and most importantly, the interchange between community leaders and political parties, given the total lack of credibility of the later, weakened the position of those leaders in the community.  Association with a party opened local leaders up to charges of corruption and manipulation.  Second, political parties did not build any lasting machinery or affiliates on the ground; they simply disappeared once the elections were finished.  Thus, political parties remained weak and were not a mechanism to channel the interests of citizens to the government.  Vote buying is a substitute for a more programmatic basis for party identity, and a means of marginalizing citizens, by exchanging political rights (a citizen’s vote) for a material good at the time of election, rather than responsive service throughout the elected official’s term in office.  The absence of credible political parties and the presence of vote buying practices sustained the political marginalization of poor urban communities, leaving them systematically ignored by elected politicians.

Studying the link between broader structural factors and local dynamics can be challenging, as the relationships are often complex and indirect.  Isolating relevant contextual elements can be difficult, as can the search for causal linkages.  Yet such research can yield promising policy recommendations to long-standing and challenging problems.  Specifically, it can allow decision-makers to differentiate symptoms and causes, and thus channel resources to the most effective leverage points to promote positive change.

10 Things To Know About Rule of Law Reform

Brendan Halloran:

These are important insights about promoting institutional change in developing countries. The emphasis on informal dynamics (power, incentives, culture, politics) rather than just formal rules and functions is applicable to public sector institutions more broadly. Donors need to support broad change coalitions over the long term and take advantage of windows of opportunity. Finally, reform and change needs to be, above all else, relevant to the local population.

Originally posted on Reinventing the Rules:

Dr. Rachel Kleinfeld, Founder of the Truman National Security Project, discusses techniques to improve the rule of law sector.

[Posted from INPROL’s blog]

Credit: Supriya Biswas

Credit: Supriya Biswas

1)     The most important elements to change are a country’s power structure and culture –its popular and professional norms.  Most first-generation rule of law programs focused on improving laws and institutions.  Reformers would build new court buildings, providing equipment to police, rewrite commercial laws, and provide computers to improve case management.  But no matter how badly it appears that these material goods are needed, second generation reformers have discovered that they are secondary to real reform.  Until power structures and professional and popular culture support the rule of law, politically powerful individuals can ignore it, and laws and institutions will continue to malfunction.  Therefore, the crucial realization of second generation reform is that countries lack the rule of law not because…

View original 1,134 more words

Review of “Development Aid Confronts Politics: The Almost Revolution” by Thomas Carothers and Diane De Gramont

I recently finished reading “Development Aid Confronts Politics: The Almost Revolution”, by Carothers and De Gramont.  I found it to be a comprehensive and insightful look at the interaction between international development practice and politics.  By politics, the authors refer to political analysis and politically-informed interventions, as well as explicitly political goals, such as promoting democratic governance.  This book is extremely relevant and timely, coming as moment when the international development industry is in a state of flux driven by budget austerity in Western nations, loud criticism and skepticism about aid from numerous sectors, and competition from investment from countries such as China and massive flows of remittances to poor countries.

The authors demonstrate that the history of incorporating political goals and methods into international development practice dates back at least five decades.  Criticism of the technocratic and apolitical nature of development has existed since the inception of the foreign aid enterprise, yet beginning in the 1990’s movement to more meaningfully address politics began to gain traction, driven by voices calling for reform both from within and outside development institutions.  The authors trace the rise of policy statements on the importance of politics and governance throughout international development work, but the very uneven progress of integrating more political analysis, methods and goals into development practice.  Indeed, although discussion of “governance” issues has spread throughout most aid institutions and across development sectors, much “governance work” is undertaken through profoundly apolitical methods, such as “technical capacity building” of state institutions and personnel while ignoring incentives and power structures that affect the way such organizations and their staff operate on the ground.

Many development organizations, including USAID, have begun to pay more attention to political economy analysis to inform goals, strategies and interventions.  Yet the authors argue that political analyses, particularly at the country level, often raise more questions than they answer and seldom have clear programmatic implications.  These analyses can often be considered “interesting, but impractical” by development professionals, and quickly find their way to the archives.  Nonetheless, the authors see the continued spread of political analysis as a potentially positive step, but note that the incorporation of politics into development practice involves sailing against strong headwinds:

“Politically informed methods thus imply a whole set of operational characteristics— such as flexibility, open-endedness, toleration of uncertainty, labor intensiveness, significant investments in preparatory analysis, and devolution of control— that cut directly against many of the central imperatives and habits of mainstream aid organizations….taken together they represent a fundamental challenge to long-established ways of operating.” p. 263

The authors believe that the conversation about politics within the international development community is a step forward, but warn that the road towards truly politically-informed development practice will be both long and challenging.

“The movement over the last two decades to incorporate political thinking and action into development aid is not a fad. It is not one more item in the seemingly ever-changing series of development fashions that capture the attention of aid providers before quickly fading away. It is, potentially at least, a fundamental advance in the aid community’s conception of the substance and process of development itself and how outside actors can play a useful role. But as we have attempted to show in this book, it remains a partially realized revolution, at best.” p. 275

Hopefully, however, the progress thus far will mean that there is no turning back now.  For as the authors state in their conclusion:

“Political methods and goals do not provide any magic bullets— developmental change is always complex and usually difficult. Although political approaches are not sufficient for sustainable development progress, there is good evidence that taking politics more fully into account is necessary.” p. 284