Throwing a rock at Evidence?

Originally posted at www.politicsandideas.org by Nilakshi De Silva on the challenges of data, evidence and development policy & practice:

I always feel a little uncomfortable when we refer to research findings as ‘evidence’. The term is very close to the idea of ‘proof’, and conjures up images of court rooms and lawyers presenting Exhibit A with a lot of fanfare to prove the case beyond any shadow of a doubt. This image is a far cry from development research and how it actually happens. We work in the imperfect real world, which is changing even as we study it. Researchers are also hampered by funding and other constraints, and research designs are often the best we can manage under these multiple constraints. Yet, even if we do manage to design the best possible research design, how much of what we do is really, beyond any doubt, ‘the truth’ – indisputable and constant?

One of the criteria by which we judge the quality of research is generalisability or the extent to which the findings may be applied to places and cases which have not been studied.  For example, can we use the findings from an impact evaluation of an employment generation scheme in Colombo and apply it to other cities and towns in Sri Lanka? In theory, of course we can – especially if the study has provided for impact heterogeneity and can show effects on different sub groups and types. Yet, there is a need to be cautious. The evaluation is limited to a specific time and place, and the world moves even as we speak; what worked in this place may not work even in this same place in a few years´ time, let alone in other non-studied locations. In Sri Lanka for example, there is no other place that is exactly like Colombo, though some of its characteristics can be seen elsewhere. The city is also undergoing palpable change and many who return to it after a few years find it has transformed, sometimes beyond recognition. Would a program that performed successfully here 10 years ago, still work? It is hard to say, which is why a good evaluation needs to tell us, not just about what worked here, but also identify under what circumstances.

Evidence of what worked in a particular place and time doesn’t travel well for another reason. Pilot interventions on which the research findings are based may be implemented in a certain way, which is often not replicated (and sometimes not replicable) in a scaled up version of the same intervention. Important checks and balances may be missing in the larger version, and we find that the impact that was seen in the studied intervention is not there under the scaled up policy that is implemented. This does not mean that the idea on which the intervention was based is a bad one, but too often this kind of outcome ends in throwing out the idea as bad policy based on bad evidence.

In our conversations about evidence – based policy making, we often ignore the contested concept of ‘knowledge’.  I was at a discussion forum on post structuralism recently where the question of whether there is anything that can be termed ‘objective’ knowledge came up. Derrida for example suggests that all knowledge is subjective, in that there is someone who goes out, interacts with the objects of the study, gathers information and subjects it to some interpretation. Knowledge is not, and he suggests it never can be, directly that of the object that is studied, but is always shaped by the object coming into contact with the person who studies the object and which is interpreted by the researcher. Many of us, who are long past our university days, may no longer engage with the philosophy of science but these ideas are critical to what we do every day.  Knowledge is our business, but do we really understand enough about how knowledge (a.k.a evidence) is produced? In other words, as development researchers, it is not enough that we are development specialists, we also need to understand what words like ‘knowledge’ and ‘evidence’ mean.

From this perspective, a healthy skepticism is a good way to approach evidence to inform policy and practice. Our knowledge is always time bound and context specific. Rather than jumping through increasingly more complex hoops in an effort to make the evidence water tight, a more useful way forward may be to focus on how we recommend that this knowledge be used.

What does all this mean, if anything, for research influencing policy? I think it means that we need to be very careful about the policy recommendations we make and how the replication and scaling up are rolled out. Whatever the evidence on which the policy is based, the implementation of the policy still needs to be carefully crafted to monitor if it is working, what is not working and adjust and adapt as you go. If we accept that there are limitations and uncertainties in the knowledge that research produces, this becomes even more important. There are no guarantees, and this needs to be borne in mind and incorporated into the recommendations as well as scale up plans. In the end, this may very well mean that the seed of the idea may remain the same as that from which the evidence was drawn, but the intervention / policy that is implemented is more grounded in the context to which the policy applies. A good example are conditional cash transfer programmes, which are implemented in many parts of the world, but which often have different designs which take into account local realities.

As development researchers, our work is driven by the implicit goal of changing the world for the better. The underlying premise is that we can ‘know’ what works and what doesn´t, by carefully designing and carrying out research studies, which can then provide ‘evidence’ for better policies and practice. While this version of the world is naturally attractive and helps to inspire and validate our work, there is also a need to be real. We study a complex world, in which intricate relationships are continuously interacting in multiple, complex ways.  This understanding should not cripple us or alternatively, push us towards over simplifying the world in order to be able to study it. Rather it should enrich our work and help development research to be more truly a useful service.

Macro Context and Micro Dynamics: Research with Policy Implications

I recently reviewed a research proposal for an ethnographic study of the dynamics of violence and the responses by local actors in a low-income barrio in Guatemala City.  The methodology was focused on seeing violence through the eyes of neighborhood residents by means of observation and interviews.  The proposal made some mention of examining the macro-social contextual factors that affected the area as well, such as economic and political marginalization, which are generally understood to be drivers of violence in such communities.

In my comments, I suggested that the researchers focus much more on this last element.  I mentioned that there existed a number of studies such as the one they were proposing, oriented towards understanding the character of violence in vulnerable neighborhoods in Latin America.  Researchers have also studied the types of responses local actors have carried out in these situations.  However, the linkage between underlying structures of economic, political and social exclusion, on the one hand, and local dynamics of violence, on the other, have not been as fully explored, at least not empirically.  This, I argued, would be the most policy-relevant area of potential investigation: understanding the relationship between macro-structures and micro-dynamics.

As long as the linkages between structural exclusion and neighborhood violence remain murky, efforts to promote citizen security will continue to focus on the violence itself as the cause of security problems, rather than a symptom of an underlying cause.  Efforts to address violence often treat violence as a characteristic of an individual, or possibly group, rather than an element of the structures in which those actors are imbedded.  If we could better understand the relationship between structural exclusion and violence, we could better address the underlying issues that were making individuals and communities more prone to violence.  That would be a step forward in the journey towards addressing spiraling levels of violence in many cities throughout Latin America.

In my own research on vulnerable urban communities in Guatemala, I have attempted to delve into this connection between macro and micro, particularly concerning political dynamics.  In Guatemala, the nature of political clientelism, a much-studied phenomenon in Latin America, is more short-term and transactional, better described as vote buying.  Political parties are too weak and fragmented to build and sustain the durable forms of clientelism associated with parties like Mexico’s PRI and Argentina’s Peronist Party, which have endured for generations.  Rather, parties focus their resources on more simple and direct exchanges of material goods for votes.

On the ground, I found that parties interfaced with community leaders in an attempt to secure votes.  They deployed promises, both of future benefits for the neighborhood and for special consideration for local leaders (often cash or a commitment to provide a government job), building materials such as bags of cement or corrugated metal sheets, or a tangible physical project in the community.  This last option was actually quite rare, as the cost to the party often outweighed the benefit.  Voting is secret and there were over a dozen parties to choose from, so political operatives were hesitant to commit significant resources in an uncertain exchange.

The impacts of the macro-political context were several.  First and most importantly, the interchange between community leaders and political parties, given the total lack of credibility of the later, weakened the position of those leaders in the community.  Association with a party opened local leaders up to charges of corruption and manipulation.  Second, political parties did not build any lasting machinery or affiliates on the ground; they simply disappeared once the elections were finished.  Thus, political parties remained weak and were not a mechanism to channel the interests of citizens to the government.  Vote buying is a substitute for a more programmatic basis for party identity, and a means of marginalizing citizens, by exchanging political rights (a citizen’s vote) for a material good at the time of election, rather than responsive service throughout the elected official’s term in office.  The absence of credible political parties and the presence of vote buying practices sustained the political marginalization of poor urban communities, leaving them systematically ignored by elected politicians.

Studying the link between broader structural factors and local dynamics can be challenging, as the relationships are often complex and indirect.  Isolating relevant contextual elements can be difficult, as can the search for causal linkages.  Yet such research can yield promising policy recommendations to long-standing and challenging problems.  Specifically, it can allow decision-makers to differentiate symptoms and causes, and thus channel resources to the most effective leverage points to promote positive change.

My new article on USAID and democratic local governance

Have a look at my new article published in Public Knowledge Journal entitled “AIDing Democratic Governance? Examining USAID Efforts to Strengthen Participatory Governance in Guatemala and Bolivia”.

You can find it here.

Check out the other articles in the current issue on Interventions here.

Feel free to comment on the issues raised.

And, as always, the views expressed in the article are my own and not those of USAID.

Vote Buying in Urban Guatemala: Some Evidence

Vote buying is a phenomenon found all over the world.  Vote buying differs slightly from clientelism in that the latter often entails more durable relationships based on repeated interactions, relationships and symbolism.  The buying of votes, on the other hand, can be viewed as a more episodic and transactional event.  As a political strategy, clientelism is infinitely preferable, as it is associated with much high predictability of outcomes.  A Mexican PRIista or Argentinian Peronista has likely been supporting the same party for years and is well enmeshed in a system of favors, relationships, symbolic exchanges and functions to demonstrate expressions of party loyalty (see studies by Javier Auyero for the case of Peronist clientelism).

In Guatemala, however, political parties do not have nearly the resources, organization, or ideology to establish such structures and relationships.  Guatemalan political parties are incredibly ephemeral, often functioning for only 2 or 3 elections before fading into irrelevance and being disbanded.  Thus, parties often rely on more abstract promises and proposals to influence voter preference.  But parties have also had a long history of offering more tangible inducements, such as food, fertilizer, building materials, etc.  A recent (2012) study the National Democratic Institute (NDI), Accion Ciudadana and the University of Notre Dame examined the phenomenon of vote buying in Guatemala and provides some concrete evidence on the prevalence of this activity.

The investigation found that 37% of respondents reported being aware of vote buying in their locality.  Only 4.4% of respondents reported that their vote was influenced by vote buying activities however.  Due to the likelihood of under reporting  the study utilized a methodology to establish a more accurate estimate of actual participation in vote buying, arriving at a figure of 17%.  Given Guatemala’s poverty and lack of established political parties, one might view this figure as rather low.  It would seem logical that the cost of a vote for Guatemala’s poor majority would be rather economical for political parties versus the rather high cost of advertising, especially buying television air time.  Yet vote buying is unpredictable.  Individuals may accept material inducements from many parties and vote for none of them on election day.  Improvements in electoral practices, particularly disallowing individuals to enter with cell phones, and thus photograph their ballot to confirm their loyalty, have reduced the efficiency of one-off vote buying transactions.

Yet in certain localities, vote buying is prevalent.  The following table presents the results of a survey in 7 neighborhoods of a low income urban area on the periphery of Guatemala City.  Survey respondents were asked to identify any of the following 4 activities undertaken by political candidates and parties in their neighborhood during the most recent elections in 2011: General promises and commitments; Promised personal benefits; Campaign paraphernalia, food and trinkets; and materials or a community project.

Neighborhood General
promises/   Commitments
Promised
personal
benefits
Shirts/Food/
Trinkets
Materials/ Project
A 49% 82% 49% 55%
B 43% 52% 29% 10%
C 95% 75% 55% 80%
D 95% 66% 66% 14%
E 73% 41% 45% 23%
F 83% 83% 83% 17%
G 10% 90% 25% 10%
AVERAGE 64% 70% 50% 30%

The evidence from the survey shows that political candidates and party operators are much more prone to promises, of a general or personal nature, than to opening the coffers to provide tangible benefits.  And when they do opt to make use of campaign funds for material inducements, they are more likely to be of an insignificant nature (party shirts and hats, food, etc) than a contribution of materials to a community project, or carrying out a project on their own.

Is this good news for political consolidation in Guatemala?  Its hard to say.  If it is a demonstration that political parties are shifting away from vote buying to more ideological and policy based platforms, this would be a positive step.  However, if it is simply a political calculation that investing in material vote buying is less cost effective than broader advertising campaigns, then it may not signify any improvement in Guatemala’s weak political party system.  In either case, in the absence of more stable political parties that represent the interests of the majority of Guatemalans, the country’s democracy will remain fragile and even superficial.

polis: Governance is Key in Addressing Urban Poverty

polis: Governance is Key in Addressing Urban Poverty.

Check out my post on the blog Polis related to urban governance.  It draws a few quick and basic lessons from my dissertation research on low-income urban areas in Guatemala.  The basic idea is that a lack of transparency and accountability mechanisms at the municipal level, combined with little organization and political agency by the urban poor, leads to a situation where such communities are not well served by local infrastructure and services and subject to political manipulation by local political actors.

polis: Can ‘Co-Production’ Bring Infrastructure to Informal Settlements?

polis: Can ‘Co-Production’ Bring Infrastructure to Informal Settlements?.

I came across this blog post while (re)rewriting one of the essays for my dissertation.  I was intrigued because it was written by a master’s student working for SPARC, an organization in Mumbai that I have been a fan of for a while now.  The topic of the blog is co-production in slum upgrading, focusing on evidence from Nairobi, Kenya (I’ll be examining co-production-ish activities in slums in Guatemala City).  The author highlights the real possibility of co-production schemes solidifying informal (and second class) service-provision arrangements, but notes that “But when done well, [co-production] can address gaps in services and policy that government or private firms alone can’t solve.”  However, if one posits that governments (or private firms) “can’t solve” a challenge like urban service provision to slum communities, the caveat must be added “under the dominant paradigm of neo-liberal urban governance”.  I’ll again refer back (this is becoming a habit) to Raj Desai’s piece I reviewed a few weeks ago.  The issue for governments isn’t so much “can’t” as “isn’t induced to”, because the urban poor do not (generally) constitute a cohesive political block that can effectively wield political agency.  City budgets respond, to a greater or lesser extent, to those with political clout (which can also take the form of financial clout).  Case in point: on the outskirts of Guatemala City the government constructed a new $3 million overpass to facilitate the entrance and exit to a gated community for wealthy families.  Three million dollars were invested to shave a few minutes off the daily commute of (at the most) a couple hundred people.  How many families living in Guatemala City’s many slums could have been provided with basic water and sewer service for $3 million?

Dissertation Progress: A light at the end of the tunnel

“So how’s that dissertation coming along?”  The question most dreaded by those making their way through the long and tortuous PhD process.  Seeing as how I am nearing (fingers crossed) the end of my own journey in this regard, I thought it would be and appropriate time for a short status update and reflection.

First, the background.  I finished all my PhD requirements except the dissertation in May of 2011 and started my research in June.  I have been studying the socio-political dynamics of community development in a diverse, but predominantly low-income, urban area on the outskirts of Guatemala City.  From July through December I did a lot of interviews with key actors, and some with randomly selected residents, surveys of those who participate in community organizations and those who do not, and participant observation of community group meetings.  My research design evolved considerably during the first couple of months as I was learning more about the community and what would be the most productive avenues of investigation.  Oh, if I had only known what I know now when I began my research!  But I adapted as new opportunities and challenges presented themselves.

I analyzed my data as I went (using Nvivo software to organize my qualitative data) and starting with the new year in 2012, I began to put pen to paper.  So I have been in the writing phase for 10 months (and counting).  Of course this did not mean that I was done analyzing my data, or even collecting data for that matter.  I continued as a participant observer and also carrying out interviews when I came across a new actor who might be able to provide some additional insights.  Thus, these past months have been a back-and-forth process of writing some, looking back at the data to see if they support my hunches, writing some more, rewriting, looking at the data one more time, and rewriting again.  I have framed my writing in 4 essays (plus an essay bringing together conclusions, policy recommendations and suggestions for further research).  Each essay stands on its own, but together they present a more integrated analysis of the community dynamics that I have been studying.  The essay themes are the following:

  1. An ethnographic exploration of spaces for citizen participation
  2. An analysis of community development, focusing on actors and approaches
  3. An investigation of citizen participation and empowerment
  4. A study of politics at the community level and its interaction with community organization and community development

Some clear lessons learned have come out of this research, which have direct policy implications.  I’ll discuss these in a future post, because they directly pertain to a new NGO I’m helping to found that will hopefully be able to take some action on the key issues I have identified using the strategies that have emerged as most successful in this context.  I am also integrating the ideas and lessons from my research in my work at USAID, bringing a critical perspective on community dynamics to our projects here in Guatemala (well, at least those that I have a chance to work on).

So what is the plan?  Well, I expect to have a completed first draft of my dissertation by the end of this year.  Actually, as noted above, it will already have gone through a couple rounds of revisions.  My advisor has provided helpful feedback on the first 2 essays and has the 3rd on his desk for his review.  I’m putting the last touches on the final essay (which has been the most complicated and lead to the most head scratching on my part).  Once I have a working draft, I will circulate it to my entire committee and hope for feedback during the Christmas break.  Then I will go into another intensive round of revisions to come up with a final, defensible product.  Logistics will be the primary determinant of when I defend my dissertation, but ideally sometime in the late spring.

Finally, as with most freshly-minted PhDs, I will be seeking publication venues for my work.  That will likely be another long and tortuous process (one I’ve already been through a couple times, with mixed results), which I’m not particularly looking forward to.  But I do think that I have generated some valuable new(ish) knowledge and I’d like to get it “out there”.  Ideally I would publish in academic journals geared towards development practitioners (such as Development in Practice or Environment and Urbanization).  I’ll be on the lookout for other venues as well.  And no doubt some tidbits will make their way into this blog as well.  I’m sure you are all waiting with bated breath!

That that is my dissertation process in a nutshell.  Wish me luck as I continue in the final stages, believe me that my level of emotional (and probably intellectual) energy dedicated to my dissertation have been steadily eroding over time, but I think I have one last push left in me.  We shall see.