James Gathogo
← Back to Portfolio

MEL System Design & Evaluation Methodology

Harmonised monitoring architecture for a global alliance of 1,100+ member organisations across 70+ countries, paired with rigorous evaluation frameworks for measuring programme impact.

System Architecture Indicator Harmonisation OECD DAC Contribution Analysis Stepped-Wedge Design

A. MEL System Architecture

A centralised data architecture designed to harmonise reporting from diverse member organisations using different tools, languages, and formats.

System Architecture: Data Flow

The architecture below illustrates how data flows from 1,100+ member organisations through validation, harmonisation, and governance layers into a unified indicator database powering multilingual dashboards and automated reporting.

KoBoToolbox
Mobile data collection
30+ countries
SurveyCTO
Regional partner surveys
8 regional offices
Excel / CSV
Manual uploads
Smaller orgs
DHIS2 Feeds
Health information
Routine HIS
API Connectors
REST APIs & Power Query ETL pipelines
Validation Rules
Range checks, skip logic, deduplication
Indicator Mapping
Maps local indicators to global framework
Indicator Database
Harmonised indicator values across members
Master Dictionary
Standardised definitions & metadata
Governance Layer
PII rules, access control, audit trail
Power BI
Dashboards
EN/FR/PT/ES
Donor Reports
Automated narrative + data exports
Scorecards
Partner performance benchmarking
Global Results
Aggregated reach & outcome figures

Key Design Considerations

Language Diversity

Member orgs operate in 40+ languages. The integration layer normalises indicator labels and definitions into four official languages (EN, FR, PT, ES) with fallback mappings for local variants.

Tool Heterogeneity

Members use KoBoToolbox, SurveyCTO, ODK, CommCare, and bespoke Excel templates. API connectors handle structured feeds; Power Query transformations standardise flat-file uploads.

Progressive Onboarding

Not all members can report on all indicators at once. The system supports tiered reporting: a core set of 8 mandatory indicators and 24 optional indicators phased in over three years.

Indicator Dictionary: Sample Extract

The master indicator dictionary standardises definitions, disaggregation requirements, and reporting frequencies across all member organisations. Below is an extract of the core mandatory indicators.

CodeIndicatorDefinitionDisaggregationSourceFrequencyTargetBaseline
IND-001 Organisations reporting against harmonised indicators Count of member orgs that submitted at least one complete dataset in the reporting period By region, org size KoBoToolbox Quarterly 850 312
IND-002 Members meeting governance threshold % of member orgs scoring ≥70% on the governance self-assessment tool By region Self-assessment Annual 65% 41%
IND-003 Children reached through member programmes Cumulative unique children directly benefiting from member-delivered services Sex, age (0–5, 6–11, 12–17), disability status Partner reports Semi-annual 14.2M 9.8M
IND-004 Members with functional M&E systems % of members with dedicated M&E staff, documented plan, and evidence of data use By region, org size Capacity assessment Annual 55% 28%
IND-005 Gender-transformative practice score Average score on the 5-point Gender Integration Continuum Scale By programme type Self-assessment Annual 3.5 2.1
IND-006 Advocacy actions taken Number of documented advocacy events, statements, or policy submissions by members By type (event, statement, submission) Event tracking Quarterly 240 87
IND-007 Community feedback response rate % of community feedback cases acknowledged and responded to within 30 days By feedback channel (hotline, suggestion box, digital) CRM system Quarterly 85% 52%
IND-008 Data quality assessment score Average DQA score across all reporting members (composite of completeness, timeliness, accuracy, consistency) By region, org size DQA tool Annual 80% 54%

Data Governance Framework

Data Classification

Public - aggregated results, published reports
Internal - operational data, partner scorecards
Confidential - individual programme data, staff records
Restricted - PII, child protection data, GPS coordinates

Access Control Tiers

Tier 1: Global administrator - full read/write
Tier 2: Regional coordinator - regional data read/write
Tier 3: Country focal point - country-level data
Tier 4: Member organisation - own data only
Tier 5: External / donor - aggregated dashboards only

PII Handling

Anonymisation: k-anonymity (k≥5) on all individual records
Consent: digital consent tracked per data subject
Retention: 5 years post-programme, then secure deletion
Transfer: encrypted at rest (AES-256) and in transit (TLS 1.3)

Quality Assurance: DQA Checklist

Completeness - all required fields populated (≥95%)
Timeliness - data submitted within 15 days of period close
Accuracy - cross-checked against source documents (5% sample)
Consistency - no contradictions between related indicators

B. Evaluation Methodology Showcase

Rigorous evaluation designs applied to education and development programmes, drawing on OECD DAC criteria and mixed-methods approaches.

OECD DAC Evaluation Criteria

R Relevance

Was the intervention aligned with the needs and priorities of beneficiaries?

Key question: To what extent did the education programme address girls' barriers to school attendance?
Source: Baseline survey, community FGDs
Method: Needs-alignment matrix, stakeholder interviews

C Coherence

Does the intervention fit with other policies and programmes in the context?

Key question: How well did the programme complement the national education sector plan?
Source: Policy documents, coordination minutes
Method: Policy mapping, key informant interviews

E Effectiveness

Did the intervention achieve its intended objectives and results?

Key question: Did girls' learning outcomes improve by the target 15% in literacy and numeracy?
Source: EGRA/EGMA assessments
Method: Pre-post comparison, difference-in-difference

E Efficiency

Were resources used optimally to deliver results?

Key question: What was the cost per learning-adjusted year of schooling gained?
Source: Financial records, output data
Method: Cost-effectiveness analysis, unit cost benchmarking

I Impact

What broader changes - positive and negative - did the intervention produce?

Key question: Did the programme contribute to reduced child marriage rates in target communities?
Source: DHS data, longitudinal cohort
Method: Contribution analysis, quasi-experimental comparison

S Sustainability

Will the benefits and systems continue after funding ends?

Key question: Have county education offices institutionalised teacher mentoring practices?
Source: Government budgets, institutional assessments
Method: Sustainability scorecard, exit interviews

Sampling Strategy: Multi-stage Cluster Sampling

GEC endline evaluation: estimating learning outcomes for marginalised girls

Stage 1: Purposive selection
4 counties selected (from 12) based on programme footprint & ecological diversity
Criteria: ≥500 beneficiaries, urban/rural mix, conflict sensitivity
Stage 2: Random selection
21 schools per county (84 total from a frame of 332 schools)
Simple random sample using randomised school list
Stage 3: Systematic random sampling
25 students per school - every k-th student from attendance register
k = (enrolled ÷ 25), random start point
Total Sample: 2,100 students
95%
Confidence Level
±5%
Margin of Error
1.5
Design Effect (DEFF)
+10%
Non-response Adjustment

Contribution Analysis: Logic Model

Education & bicycle mobility programme: mapping causal pathways from inputs to impact

Inputs
Funding ($2.4M), bicycles (3,200), trained mentors (84), school materials
Activities
Bicycle distribution, teacher training, community dialogues, bursary payments
Outputs
3,200 girls with bicycles, 84 mentors active, 42 community groups formed
Outcomes
↑ Attendance (+22%), ↑ Literacy scores (+15%), ↓ Dropout (−8pp)
Impact
Reduced child marriage, improved livelihoods, community norm change
A1: Bicycles reduce travel time sufficiently to enable daily attendance
A2: Trained mentors remain in post and apply new pedagogies
A3: Improved attendance translates to improved learning
A4: Education gains delay marriage and improve economic outcomes

Evidence Mapping by Results Level

Outputs → Quantitative: Distribution records, training attendance registers, monitoring visit checklists
Outcomes → Mixed: EGRA/EGMA scores (quant), most significant change stories (qual), attendance registers (admin)
Impact → Quasi-experimental: DHS secondary data analysis, longitudinal cohort tracking, community FGDs
Assumptions → Participatory: Community perception surveys, teacher interviews, programme implementation fidelity audits

Stepped-Wedge Cluster Randomised Design

Youth economic empowerment programme - ethical rollout with built-in counterfactual

T1
T2
T3
T4
T5
T6
Cluster A
Control
Intervention
Intervention
Intervention
Intervention
Intervention
Cluster B
Control
Control
Intervention
Intervention
Intervention
Intervention
Cluster C
Control
Control
Control
Intervention
Intervention
Intervention
Cluster D
Control
Control
Control
Control
Intervention
Intervention
Why stepped-wedge? All clusters eventually receive the intervention (ethical requirement), while the staggered rollout creates a within-cluster counterfactual. Each cluster serves as its own control before crossover, strengthening causal inference. Practical for programmes where simultaneous rollout is logistically infeasible and withholding treatment indefinitely is unacceptable.

Design Parameters

Clusters: 4 districts (8–12 communities each)
Time periods: 6 quarters (18 months total)
Primary outcome: youth employment rate at 6 months post-training
Analysis: mixed-effects regression with cluster random effects

Ethical Advantages

No community permanently denied the intervention
Phased rollout matches operational capacity constraints
Learning from early clusters can improve later rollout
Within-cluster comparison reduces confounding bias

Technical Methods Summary

Core methodological competencies applied across programme evaluations and MEL system design engagements.

Mixed methods (OECD DAC)
Contribution analysis
Difference-in-Difference
Stepped-wedge designs
Pre-post analysis
Qualitative synthesis (thematic coding)
Sampling frameworks
Data quality assurance
Theory of change development
Cost-effectiveness analysis
Indicator harmonisation
Ethical review & safeguarding
12+
Evaluation designs led
70+
Countries covered by MEL systems
1,100+
Member organisations supported