METHODOLOGY—Basic general questions answered
Background: the post-2015 global development agenda
- The Millennium Development Goals (MDGs) have made a huge impact on the lives of billions, but there is still much more to be done to address extreme poverty, diseases and environmental crises.
- The global development agenda for the period after 2015 will build on the progress made on the MDGs while confronting persistent inequalities and new challenges facing people and planet.
- The new development agenda must result from a truly open and inclusive process that engages people from all parts of the world and all sections of society.
- The UN and the Secretary General’s High Level Panel on post-2015 want to hear directly from people across the world about what is important to them, so that their views can be taken into account in this global conversation. MY World is conceived as a tool to deliver this.
What is MY World?
- The United Nations and partners have teamed up to put in place an innovative global survey, known as MY World, so people can participate in the post-2015 process telling us the changes that would make the most difference to their lives.
- It is an options survey which, through creative online and offline methods, allows people across the world to tell the United Nations, global leaders – and in particular the Secretary General’s High Level Panel, the most important issues they would like the post-2015 agenda to address.
- MY World asks individuals which six of sixteen possible issues they think would make the most difference to their lives. The sixteen issues have been built up from the priorities expressed by poor people in existing research and polling exercises, and from the ongoing technical and political discussions about possible future goals. They cover the existing MDGs, plus issues of sustainability, security, governance and transparency.
How will it work?
- The survey is available online at http://www.myworld2015.org in the six UN official languages. Citizens will also be able to participate in this survey through mobile technologies such as SMS and IVR (toll-free phone numbers) from January 2013.
- The survey will also be available offline in paper form – distributed through a network of grass roots organizations, faith based communities, youth groups, private sector bodies and NGO partners around the world. The support of these organizations is vital in reaching out directly into communities and drawing the digitally disconnected, illiterate and poorest communities into the global debate.
- Participants will be asked their gender, age and country, to allow for disaggregation of data and to present decision makers with an accurate global picture of what citizens think.
When and where will this all happen?
- From now until 2015, we want as many people in as many countries as possible to be involved: citizens of all ages, genders and backgrounds, particularly the world’s poor and marginalized communities.
- MY World will be available from early December 2012, so countries hosting national consultations can take it into account. However, the global launch of MY World is scheduled for late January 2013 (at that time a comprehensive communication toolkit will be available for all).
What are the sixteen options?
- Better job opportunities
- Support for people who can´t work
- A good education
- Better healthcare
- Affordable and nutritious food
- Phone and internet access
- Better transport and roads
- Access to clean water and sanitation
- Reliable energy at home
- Action taken on climate change
- Protecting forests, rivers and oceans
- Equality between men and women
- Protection against crime and violence
- Political freedom
- An honest and responsive government
- Freedom from discrimination and persecution
What will happen to the survey results?
- Data from mobile phones, website and offline surveys will be continuously consolidated and available on the MY World website.
- Results will be submitted to the Secretary General’s High Level Panel on the post-2015 development agenda before they present their final report to the Secretary General of the United Nations in June 2013. MY World will continue gathering people´s voices and results will be shared with the Secretary General and global leaders as they prepare the next development agenda in the run up to 2015.
Basic methodology questions answered
What is this survey for?
MY World is a way of listening to what people think a new global agreement on development should look like. It will complement other initiatives bringing citizens voices into the post-2015 debate, such as the ‘Participate’ project, run by the Institute for Development Studies and the Beyond 2015 coalition of civil society organisations, which is doing in-depth qualitative research into people’s priorities, and the large number of national and thematic consultations organised by the United Nations across the world.
MY World complements these by providing quantitative data on priorities. The options in the survey are deliberately framed in a broad way: ‘better healthcare’ for example, or ‘equality between men and women’ to make these issues understandable to ordinary citizens. The other consultations will be complementary and provide additional information to allow policy makers to understand people’s views and priorities within each of these broad headings, and to consider if and how these could be translated into goals, targets and indicators.
MY World provides a bridge between the other consultation processes, and the political process focusing on how to prioritise to create a manageable global agenda. MY World will allow the views of citizens to be included into that.
The intention is not to create a definite ‘answer’ as to what the world’s priorities should be , but to contribute information to global decision makers to ensure that what emerges at the end of the post 2015 process is informed by the views of as many people as possible.
Why these 16 options?
Choosing a list of options for a proposal like this is a very difficult task. This is the process we went through to choose the 16 options:
- An initial list of 24 top-line options and explanatory paragraphs was drawn up based on the current Millennium Development Goals issues, a review of civil society and other proposals for post-2015, various Sustainable Development Goals proposals, and existing data from participatory research and polling. The options chosen were those which emerged as clear themes from these sources, and which were likely issues for global.
- This list then went out for consultation, among a group of individuals from NGOs, policy makers, academics and UN staff in a large number of countries and regions. Following the feedback, it was decided to reduce the list to 15, in order to make the survey simpler and more accessible. This list then went out for a second round of consultations among a smaller group, including an Africa-based opinion polling company, for a second round.
- Following this consultation, it was decided to increase the number to 16 to accommodate the new round of comments.
- A final test was conducted through an SMS survey in Uganda, where an open question was asked and the responses coded back to the 16 options. When asked about issues of most importance to individuals and their families, less than four per cent of responses did not fall under the 16 options categories.
We firmly believe that we have a balanced set of options, which reflect the range of issues which people around the world are concerned about and issues around which new goals might be developed. It doesn’t include everything that matters to people, but it does include the range of issues that goals could address and which we know are important to people, to organisations, and to governments. The introduction to the survey, both online and offline, will stress that all of these options are important but that we are trying to identify those that matter most, to inform the eventual selection of a few global goals.
But what if I want to choose a different option?
The survey provides an opportunity for participants to enter their own priority, through a 17th ‘your own option’ choice. Answer provided here will analysed very carefully to see if there are missing themes or new ideas that are generating support. The analysis of these free-form options will be made available.
Why are the options worded like this?
The options have to be understandable to everyone everywhere, not just development experts. We worked with an Africa-based polling company and with communications experts in three organizations to revise the language to ensure that it is accessible and as free as possible of jargon or culturally specific references.
Why have you asked ‘what matters most to you and your family’?
While a post-2015 agreement will be a global one, ultimately its purpose is to make life better for people. So asking what is important to people seems a better and clearer starting point than asking people to imagine a remote global agreement and what should be in it. The survey is framed around individual priorities, but we recognise that for many people their personal interests and choices are inextricably linked to those of their families (parents, for example, might answer the question with their aspirations for their children in mind). We have therefore tried to get across the idea that while we are asking people what matters to them as individuals, we recognise that their happiness is bound up with that of others.
In addition, wording the question like this means that there is less room for respondents to interpret the question differently, which is a risk if we asked a broader question on the priorities for global governance, or one about country or community priorities.
Why limit people to 16 options …couldn’t you just ask an open question?
MY World is just one part of a global consultation effort on post-2015, and all the others involve open questions. There are lots of ways for people to put forward their points of view, and these different methods can complement each other. The role of MY World is to offer insights about people’s priorities, and for that it was necessary to predetermine the list of options. As described above, great efforts have been made to ensure that the priorities are representative of the things people have said are important .
In addition, there are two technical and methodological reasons for asking closed questions:
- Given that the survey is going to be done in multiple languages, and through different mediums (web, SMS and offline), current technology is not good enough for the responses to an open question to be grouped consistently and credibly into clusters for analysis using available software. Manual coding of the several million responses we are hoping for in this project would make the project prohibitively expensive and time-consuming.
- Major polling partners have shared their experience that participation rates are very significantly lower in surveys with open questions. This is not surprising as they require more thought and effort from the participants. Therefore, in order to maximise participation and make the survey as accessible as possible, offering closed options is a better solution.
But you can’t really tackle poverty through a list of single issues – everything is connected.
We agree. That’s why it was important to give people a large number of options and to allow them to choose more than one. Choosing 6 out of the 16 will allow people to think about what combinations of actions might produce the best effect. It will also allow people to select options which tackle the same problem in different ways: a parent concerned that their daughter should get a good education, for example, might want to choose both ‘equality between men and women’ and ‘a good education’ to represent the different dimensions of the changes they want to see.
We tested the poll with three pilot groups to see whether it made a difference if people were asked to choose 3, 5, or 7 options. Respondents were asked whether the survey was harder/easier and more or less enjoyable. No noticeable differences were found.
How will you know if people answering the survey are representative of the population as a whole or of different groups within it?
We are going to find out as much as we can about the people answering the survey, so we can say as much as possible about whom they are. But we will be extremely careful not to over-claim, by being open and transparent about the content and limitations of the data. In addition, we are taking advice from a global expert opinion polling company about how to analyse and present the data. The levels of information are as follows:
- Gender – all respondents will be asked their gender so this applies to all data
- Age – again, all respondents
- Country – all respondents
- Method of response – we will be able to analyse the data according to how people take the survey, which will establish any broad differences between the three groups.
- Date and time of response – to evaluate the impact of any marketing campaigns and to pick up any automated programmes that could rig the results.
- Educational level – all off-line and SMS participants will be asked this question. In addition, all online participants will be asked this question as an optional extra (this is to minimise the dropout rate)
- Sub-national region – we will have this information about SMS participants from the mobile operators, and about offline participants from the surveyors. Online participants will again be asked this question as an optional extra, to minimise dropout rates.
The information we will collect will be of a similar level of detail to that collected by the Gallup World Survey and other globally representative samples. While we will not be randomly sampling in the way that Gallup does, we will use the social and demographic data we collect to establish how well our sample in each country reflects its whole population, and make that clear when we publish the data.
How will you present the data?
It will be possible to view the data from the web survey in real time, and data from the SMS and offline surveys will also be added as rapidly as possible to ensure the results are up to date.
In addition, the extra information collected from people taking the survey will be used to disaggregate the data to provide a picture of how priorities differ between countries, demographic and socio-economic groups. We will report on the priorities of different groups and countries separately.
We will also aggregate the results by country, region and globally. In doing so, we will use the information collected from participants to weight the responses so that when data is aggregated, it is done so in a way that provides as representative a picture as possible of the whole population. Options for weighting include:
- Demographic factors: weighting the responses from each country in line with population size, from men and women and different age groups to reflect the balance in the population.
- Socio-economic factors and sub-national regional demographics. We expect to have information about level of education and sub-national region of residence for a sizeable sub-group of the total respondents. Once the numbers are large enough it will therefore be possible to weight responses in a way that reflects the socioeconomic as well as demographic composition of countries.
- Proxies for income: to some extent the three response methods are proxies for different income groups, with the wealthiest being more likely to have access to the web and the poorest most likely to be among those participating offline. This is particularly the case given that the focus for the offline survey will be the poorest regions in the different countries in which it is administered. We will experiment with different ways of weighting the responses according to how the survey was carried out, to reflect regional population patterns and the distribution of income groups within countries.
If the post-2015 agenda is about eradicating poverty…why should we care about the priorities of people who aren’t poor?
We have taken great care in developing the methodology to ensure that the priorities of the poorest people are captured in this survey, and that it will be possible to analyse their responses separately from the rest. We have used household survey data to identify the regions within countries where the poorest people live, and will be focusing the off-line efforts in those regions to ensure coverage of the poorest communities. In addition, we are planning to work with a group of NGOs who can give us advice on how to do the survey with specific hard-to-reach groups such as children, older people and disabled people, and many of those same NGOs will do the survey with the communities they work with. So it will be possible to use this data to focus only on the extremely poor, if that is the focus. When the data is aggregated at the global level, we can use weighting techniques to ensure that the responses of poor people count in proportion to their numbers in the whole population, and are not drowned out by a large web response in high income countries.
But the survey is designed to be global because everyone has aspirations and faces problems in their lives. And very often the priorities of different groups are more similar than we might expect. Understanding how people in every country view their lives and the things they would like to change will make it easier tobuild the political constituencies to push for a new agreement. The more a framework reflects a global consensus about priorities, the easier it will be to build the politics that can make it happen.
Isn’t there a bit more to a new framework than just short-listing the most important issues to people round the world?
The MY World survey is just one of a whole range of consultations and political processes that will contribute to a new agreement. And knowing about priorities for different groups will be important in shaping those politics.
In framing the 16 options, we were very aware that this process is about designing global goals, so the options all represent issues that could usefully and feasibly be tackled through international action. We excluded some things, like, for example, religion, which we know are very important to people but which would be less appropriate as subjects for new global goals.