Benchmarking of Mobile Data Collection Solutions - 2021

This resource was originally published on the CartoBlog in 2021 by CartONG. It was transferred to the IM Resource Portal in May 2021. The CartoBlog will be decommissioned in early 2022.

This resource is available as a ZIP file containing the benchmarking in pdf format and one Excel file.

CartONG released a first substantial benchmarking of MDC solutions in 2017, focusing on applications and services running on low-cost Android devices, and for contexts where cellular networks can be unreliable. Since then, technologies have evolved quickly and functionalities that were standing out of the crowd are now widespread – hence it seemed interesting to come out with a new version of the benchmarking focusing more particularly on features that are differentiating factors nowadays.

It was therefore perfect timing when Welthungerhilfe solicited a collaboration with CartONG in order to conduct a benchmarking of solutions for the internal use of the organisation. This led to testing and comparing several Mobile Data Collection tools to see if any of them could be compatible with Welthungerhilfe’s strategy to deploy a well-conceived, robust and integrated solution for data collection, processing and analysis for the whole organisation.

This benchmark, composed of a PDF and an Excel, aimed to explore more particularly the tools’ user friendliness (of both web platform and app), simplicity of processes and features enhancing data protection and quality, as these were key aspects for Welthungerhilfe.

To this end, the focus was put on the following aspects:

• Organisational management – Organisation-wide control available, including centralised management of users and surveys, and the possibility to create a dedicated workspace for country teams, including the promotion of standardisation and organisation learning (use of global template and standards methodologies).
• User experience – Intuitive tool that is easy to adopt, and also both as interoperable as possible with partner organisations’ data (therefore using humanitarian widely used formats, such as XLS form), as well as with visualisation tools (through an API for instance).
• Data quality – Availability of dedicated features such as advanced constraints and skip logics as well as extensive possibilities to test and review the forms and the data, including validation processes and the possibility to monitor any changes.
• Data protection – Ensuring data protection and GDPR compliance through dedicated features to flag and limit access to PII2 and sensitive data, high granularity of roles, and adequate storage location.
• Case Management – Availability of features and workflow facilitating case management and/or longitudinal data collection.

This benchmark covers a reduced number of solutions as it focuses on the solutions that were most likely to meet the needs of Welthungerhilfe. To that end, 18 solutions were pre-assessed and 8 solutions were compared in depth.

The initial list was composed of: ActivityInfo, AkvoFlow, CommCare, DeviceMagic, Dharma, Fulcrum, iFormbuilder, KoBoToolbox, Magpi, Mobenzi Researcher, ODK, ONA, Poimapper, Survey 123, Survey CTO, SDK, SMAP, and Taroworks.

A pre-assessment focusing on 20 specific requirements based on Welthungerhilfe’s needs allowed us to reduce the list to the 8 solutions presented in this benchmark. The benchmark thus only focuses on the results of the in-depth assessment conducted for the said 8 solutions: AkvoFlow, CommCare, KoBoToolbox, Magpi, Mobenzi Researcher, ODK, ONA, and Survey CTO.

Please note this report is based on tests conducted in December 2020 and January 2021. We therefore advise readers to verify features and functions of newer releases when interested in a particular tool/platform.

-----

This resource was produced by CartONG and co-funded by the French Development Agency (AFD) and Welthungerhilfe. Nevertheless, the ideas and opinions presented in this document do not necessarily represent those of Welthungerhilfe and AFD.