{"id":9730,"date":"2018-09-05T11:03:53","date_gmt":"2018-09-05T09:03:53","guid":{"rendered":"https:\/\/openasia.org\/en\/?p=9730"},"modified":"2018-09-05T11:03:53","modified_gmt":"2018-09-05T09:03:53","slug":"technical-learning-from-the-meta-analysis-of-womens-empowerment-projects","status":"publish","type":"post","link":"https:\/\/openasia.org\/en\/2018\/09\/technical-learning-from-the-meta-analysis-of-womens-empowerment-projects\/","title":{"rendered":"Technical learning from the meta-analysis of women\u2019s empowerment projects"},"content":{"rendered":"

OXFAM<\/span><\/a><\/p>\n

By Kristen McCollum<\/p>\n

\"\"<\/a>

PHOTO CREDIT: JENNIFER CIOCHON\/OXFAM<\/p><\/div>\n

Kristen McCollum, consultant at the International Fund for Agricultural Development (IFAD), shares with us the learning from Oxfam\u2019s meta-analyses on women\u2019s empowerment.<\/strong><\/p>\n

When we first decided to conduct a meta-analysis on women\u2019s empowerment Effectiveness Reviews (ERs), the idea was to go where no impact evaluation had gone before. While the\u00a0Effectiveness Reviews<\/a>\u00a0give us a rigorous measurement of the impact of specific projects in their contexts, we wanted to be able to draw conclusions across studies for an organisational perspective, as well as learning about measurement and impact from a birds-eye perspective.<\/p>\n

Results were enlightening in some unexpected ways, both of their implications for programme learning, as well as for their technical implications on measurement. We already presented the\u00a0main results and programme learning in a previous blog<\/a>. Here we outline four major technical learning we obtained from the meta-analysis.<\/p>\n

1) There\u2019s\u00a0power<\/em>\u00a0in numbers<\/strong><\/h2>\n

Sample sizes for impact evaluations are always calculated with their potential power in mind.\u00a0 That is, we make sure that the number of people interviewed is sufficient to be able to detect statistically significant impact at a level we think is substantial.\u00a0 Two caveats here: first, we all know that budget is often an equally powerful determinant of sample size. Second: as women\u2019s empowerment is notoriously difficult to measure, it is difficult to define when an impact is \u2018large enough\u2019 to be deemed important to detect.<\/p>\n

The meta-analysis gives us some effect sizes we can use for conducting power analysis and drawing conclusions on the required sample size for future studies. Moreover, the beauty of meta-analysis is that it acts as value-added research which takes advantage of the latent power hidden in a collection of impact evaluations.\u00a0 Though one study might not have the sufficient sample size to detect the impact, a combination of studies might. This is exactly what happened with our analysis.<\/p>\n

Of the nine ERs that examined the impact on violence against women, not a single one detected a significant change in women\u2019s experience of violence. However,\u00a0meta-analysing the studies allowed us to find an impact that was too small to be detected by individual studies.<\/strong>\u00a0Overall, women in the projects were\u00a0more<\/em>\u00a0likely to report having experienced violence than women in comparison groups. Exploring the data further, it seems likely this is at least in part a result of the way violence was measured \u2014 leading us to our next technical lesson:<\/p>\n

Meta-analysing the studies allowed us to find an impact that was too small to be detected by individual studies<\/p><\/blockquote>\n

2) Stronger measurement tools<\/strong><\/h2>\n

Seven of the ER\u2019s measuring experience of violence employed the\u00a0neighbourhood method<\/a>.\u00a0 With this method, women are asked if they\u00a0or anyone close to them<\/em>\u00a0had experienced physical, sexual, or emotional violence. While this method allows us to ask about violence in a more sensitive manner suited to particularly vulnerable contexts, it may give results that are difficult to interpret. Are women reporting more violence because it is happening more often, or are more empowered women more likely to identify violence (particularly emotional violence) happening to their peers?\u00a0 This question is difficult to answer now. What we learned is\u00a0the need for strengthening our measurement of violence<\/strong>\u00a0and for programmes to more accurately detect, prevent and manage any possible unintended negative impact on violence. As a direct response to these findings in the most recent Effectiveness Reviews of women\u2019s empowerment projects, we moved to investigate direct exposure to violence. We also need to pay more attention on how to capture this important information.<\/p>\n

3) Flexibility does not prejudice results<\/strong><\/h2>\n

In addition to the above, other measurement questions emerged while conducting the analysis.\u00a0 A meta-analysis can give us overall effect sizes, but also allow us to regress the effect sizes of our ERs on a variety of characteristics to see if the way we construct the Women\u2019s Empowerment Index drastically influences its output. Since the Index is meant to be a flexible tool that allows for a context-specific measurement of empowerment while still being reasonably comparable across projects, we wanted to verify that small changes to the Index wouldn\u2019t be a determinant of the impact it detected. For example: is the Index more likely to result in a positive measure of empowerment if it includes more indicators? Here, we don\u2019t find any statistically significant difference. So what does this mean? It means so far,\u00a0there\u2019s nothing to suggest that the differences in the construction of the Index between studies are influencing the results we get<\/strong>. This is good news \u2014 the flexibility of the Index doesn\u2019t appear to be compromising our results.<\/p>\n

4) Let\u2019s measure impacts not outputs<\/strong><\/h2>\n

Lastly, a final finding called our attention to the underlying assumptions of how we conceptualize empowerment. We realised that 13 of the ERs had used group participation measurement in their Index. This is pretty standard, and in line with\u00a0how OPHI and IFPRI measure empowerment<\/a>. However, this may not be as appropriate in the context of our Effectiveness Reviews, or impact evaluations more in general. This is because the vast majority of the evaluated projects have a group-forming component. So we would expect this indicator to always have a positive effect on the treatment group, as a de-facto condition of their being project participants (indeed, the overall effect size was large and significant). So do we leave it out? Well, it probably depends.\u00a0Participation in groups is a key facet of empowerment, but it is important to capture the\u00a0impact<\/em>\u00a0of that participation<\/strong>\u00a0without simply throwing in an\u00a0output<\/em>\u00a0measurement into an impact-focused Index. So in future Effectiveness Reviews, we will give priority to indicators such as \u2018influence within groups\u2019 rather than \u2018group participation\u2019, and we will decide on whether to include group participation if this not a pre-condition for project involvement.<\/p>\n

To conclude, the meta-analysis was not only useful for measuring organisational impact, but also to direct the future of how Oxfam measures women\u2019s empowerment and the assumptions we make about the underlying mechanisms of projects.<\/p>\n

FOR MORE INFORMATION, read the Impact evaluation of the project \u2018AMAL:\u00a0Supporting Women\u2019s Transformative\u00a0Leadership\u2019 in Tunisia<\/a>.<\/p>\n

 <\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

OXFAM By Kristen McCollum Kristen McCollum, consultant at the International Fund for Agricultural Development (IFAD), shares with us the learning from Oxfam\u2019s meta-analyses on women\u2019s empowerment. When we first decided to conduct a meta-analysis on […]<\/p>\n","protected":false},"author":15,"featured_media":9731,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[43,11,17,49,10],"tags":[957,638,958],"class_list":["post-9730","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-human-rights-online-library","category-issues","category-women","category-womens-rights","category-world","tag-research","tag-tunisia","tag-womens-empowerment","Documents-conventions"],"acf":[],"_links":{"self":[{"href":"https:\/\/openasia.org\/en\/wp-json\/wp\/v2\/posts\/9730","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/openasia.org\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/openasia.org\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/openasia.org\/en\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/openasia.org\/en\/wp-json\/wp\/v2\/comments?post=9730"}],"version-history":[{"count":3,"href":"https:\/\/openasia.org\/en\/wp-json\/wp\/v2\/posts\/9730\/revisions"}],"predecessor-version":[{"id":9768,"href":"https:\/\/openasia.org\/en\/wp-json\/wp\/v2\/posts\/9730\/revisions\/9768"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/openasia.org\/en\/wp-json\/wp\/v2\/media\/9731"}],"wp:attachment":[{"href":"https:\/\/openasia.org\/en\/wp-json\/wp\/v2\/media?parent=9730"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/openasia.org\/en\/wp-json\/wp\/v2\/categories?post=9730"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/openasia.org\/en\/wp-json\/wp\/v2\/tags?post=9730"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}