Integrity Analytics: How AI can help the development sector to streamline learning and knowledge management

Organizational learning and knowledge management is a continuous challenge for large organizations where institutional knowledge often relies on narrative documents and reports. While written information can provide rich detail on past lessons learned, it is time-consuming to analyze. Recent advances in artificial intelligence (AI) and large-language models (LLMs) can help solve some of these challenges, writes Naji Harfouche, Integrity’s Principal Manager, Data Analytics.

Integrity recently completed a project with USAID’s Bureau for Conflict Prevention and Stabilization (CPS) that demonstrates the real-world potential of AI and LLMs to understand and analyze decades of USAID-funded peace and security programming. Conducted through the Integrity-led Peacebuilding Evaluation, Analysis, Research, and Learning (PEARL) Activity, this project focused on extracting insights from long-form PDF reports stored on USAID’s Development Experience Clearinghouse (DEC). These documents often lack abstracts and are difficult for users to locate. CPS asked Integrity to help identify documents in the DEC relevant to peace and security, develop abstract-style summaries for each document, and build a dashboard for USAID users to access evidence and learning more readily.

Integrity’s approach to using data science methods and AI tools draws on a commitment to transparency and sustainability. We aim to use open-source tools so our work is reproducible, scalable, and can be updated at minimal cost. We give clients full visibility into our work, allowing them to scrutinize our methods and confidently use our findings. Our approach also offers value for money by empowering clients to re-use these tools independently.

To meet CPS objectives, we used innovative approaches at every step of the process. This involved a combination of data science, machine learning methods, and AI tools including LLMs. While human language is nuanced, AI-based approaches inherently tend toward generalization because they use patterns identified in training data to produce results. Additionally, generative AI tools are trained to produce outputs that are written convincingly, sometimes oversimplifying concepts or concealing inaccuracies. To address these shortcomings, we designed a robust prompting and quality assurance process informed by human expert judgment to ensure the requested summaries were not only well written, but also an accurate reflection of document content.

Working closely with CPS counterparts, we captured the results of our work in a user-friendly Peacebuilding Research, Evidence, and Programming Portal. This dashboard provides USAID users with an accessible interface to explore the evidence on peace and security stored within the DEC and more easily find relevant documents based on geographic and thematic focus. USAID is already using the dashboard to access evidence and inform research, program design, and adaptation based on 20 years of analyzing what works in different contexts.

The performance of machine learning tools for text analysis and generation (natural language processing) has improved dramatically over the last few years. Our work demonstrates that careful design and collaboration with human experts can overcome limitations and effectively respond to organizational learning challenges. These approaches present major opportunities for the development sector to better use existing evidence to improve the delivery of aid.

Integrity’s data analytics team delivers data science, digital development, and knowledge management services. We blend cutting-edge technologies and methods to improve the value, efficiency, and effectiveness of international development projects. We combine data analytics with Integrity’s other services to provide integrated products that equip decision makers with the evidence they need to tell the story of their work. Go here for more details of Integrity’s recent projects.