NPIPIntelligence Platform
header_live
header_india_year
🕸️

Data Sources & Scraper Intelligence

Government data sources powering nPIP · Active scrapers · Upcoming integrations · Scraping methodology

11
Active Sources
701
Districts Scored
14
Upcoming Sources
6
Scraper Methods
2,483
Village Records
0
Officials Indexed

Active Data Sources (11)

NFHS-5 (2019–21)
National Family Health Survey Round 5
Health📍 701 districts🔄 5-yearly
ACTIVE
Ministry: Ministry of Health & Family Welfare
Indicators
Stunting, wasting, underweight (<5 yrs)Anaemia in children & womenInstitutional delivery %Children fully immunized %Infant/neonatal/U5 mortalityANC 4-visit coverage
🔧 REST API available via DHS Program (api.dhsprogram.com)
🔗 rchiips.org/nfhs/nfhs-5_FCTS/home.html
HMIS (2023–24)
Health Management Information System
Health📍 697 districts🔄 Monthly
ACTIVE
Ministry: NHM / MoHFW
Indicators
ANC registrationsInstitutional deliveriesImmunisation countsSub-Centre / PHC / CHC functional
🔧 Monthly Excel/CSV reports downloadable. Scraper: HTTP download + pandas parse
🔗 hmis.nhp.gov.in/downloadReport
UDISE+ (2022–23)
Unified District Information System for Education Plus
Education📍 697 districts🔄 Annual
ACTIVE
Ministry: DSEL, Ministry of Education
Indicators
Enrollment (boys/girls/SC/ST)Dropout & pass ratesPupil-teacher ratioSchool infrastructure (electricity, toilets, water, library, computer lab)
🔧 District/state-level reports downloadable as Excel. State-wise URLs follow predictable pattern.
🔗 udiseplus.gov.in/#/en/data/dataAnaReport
JJM Dashboard
Jal Jeevan Mission – Har Ghar Jal
Water📍 8 districts🔄 Daily
ACTIVE
Ministry: Ministry of Jal Shakti
Indicators
FHTC household connectionsVillage tap coverage %State-wise progressPipeline length
🔧 Live API: https://ejalshakti.gov.in/jjmreport/api/... (JSON). Can scrape daily.
🔗 ejalshakti.gov.in/jjmreport/JJMIndia.aspx
SBM-G Dashboard
Swachh Bharat Mission Gramin
Sanitation📍 698 districts🔄 Daily
ACTIVE
Ministry: Ministry of Jal Shakti (MoJS)
Indicators
IHHL construction progressODF village/district countToilet coverage %
🔧 Download reports from SBM portal as CSV/Excel. API partially available.
🔗 sbm.gov.in/sbmReport/home.aspx
Saubhagya Portal
Pradhan Mantri Sahaj Bijli Har Ghar Yojana
Energy📍 698 districts🔄 Daily
ACTIVE
Ministry: Ministry of Power
Indicators
Household electrification %Village electrificationRemaining households
🔧 JSON API at saubhagya.gov.in/api/. State/district breakdown available.
🔗 saubhagya.gov.in
MoAFW Crop Statistics
Ministry of Agriculture & Farmers Welfare
Agriculture📍 697 districts🔄 Annual
ACTIVE
Ministry: Ministry of Agriculture
Indicators
Area, production, yield per cropIrrigated vs rainfed areaCrop diversity
🔧 Download Excel from DACNET APY portal. State + district disaggregation available.
🔗 aps.dac.gov.in/APY/Index.htm
PMGSY-MIS
Pradhan Mantri Gram Sadak Yojana Management Info System
Infrastructure📍 698 districts🔄 Quarterly
ACTIVE
Ministry: Ministry of Rural Development
Indicators
Road length sanctioned/completedHabitations connectedAll-weather road coverage
🔧 State-wise PDF/Excel from OMMS portal. Habitation data in CSV format.
🔗 omms.nic.in
Census 2011 Village Abstract
Primary Census Abstract for Villages
Demographics📍 698 districts · 2,483 villages🔄 Decennial
ACTIVE
Ministry: Office of the Registrar General of India
Indicators
Population (total/SC/ST)Literacy rateHousehold countUrban/rural split
🔧 Excel downloads from Census India NADA portal. Census 2024 expected soon.
🔗 censusindia.gov.in/nada/index.php/catalog
MGNREGS MIS
Mahatma Gandhi NREGS Management Information System
Employment📍 697 districts🔄 Daily
ACTIVE
Ministry: Ministry of Rural Development
Indicators
Person-days generatedWage disbursementActive job cardsWork demand vs supply
🔧 Live API and Excel downloads available. nrega.nic.in/netnrega/home.aspx#
🔗 nregsportal.nic.in
Representatives (Synthetic)
Elected & Administrative Officials Directory
Governance📍 0 officials🔄 Per election
ACTIVE
Ministry: Election Commission of India / Lok Sabha / MoHA
Indicators
District Collector/DMSuperintendent of PoliceCEO Zila ParishadMLAs, MPs per constituency
🔧 ECI portal for electoral data; LS/RS websites for MP data; individual state portals for IAS/IPS postings
🔗 eci.gov.in, loksabhaph.nic.in

Upcoming Integrations (14)— Sources to add from forthepeople.in & government portals

PM-KISAN Beneficiary Database
High
Agriculture/FinanceMinistry of Agriculture
State/district-wise beneficiary counts, amount disbursed per installment, excluded categories
🔗 https://pmkisan.gov.in
PMAY-G Dashboard
High
HousingMinistry of Rural Development
Houses sanctioned, completed, geo-tagged, category-wise (SC/ST/minorities)
🔗 https://rhreporting.nic.in
Aadhaar Enrollment Data
Medium
IdentityUIDAI
District-wise Aadhaar saturation, seeding rates for DBT schemes
🔗 https://uidai.gov.in
VHSND / Village Health Data
High
HealthNHM State Societies
Village Health Nutrition Days data, AWC-level nutrition tracking
🔗 https://nhm.gov.in
ECI Electoral Rolls & Results
High
GovernanceElection Commission of India
Constituency boundaries, voter registration, election results 2014-2024, candidate profiles
🔗 https://eci.gov.in, https://results.eci.gov.in
Lok Sabha / Rajya Sabha Members
High
GovernanceParliament of India
MP profiles, attendance, questions raised, funds (MPLAD) utilised
🔗 http://loksabhaph.nic.in, https://rajyasabha.nic.i
State MLA Data
Medium
GovernanceVarious State Governments
MLA profiles, constituency maps, MLALAD funds, attendance records
🔗 State Legislative Assembly portals
Block-level Health Facilities
High
HealthNHM / MoHFW
Health Facility Registry – Sub-Centres, PHCs, CHCs, District Hospitals with GPS coordinates
🔗 https://hfr.nhp.gov.in
e-Shram Workers Database
Medium
LabourMinistry of Labour
Unorganised sector worker registration, district-wise count, occupation categories
🔗 https://eshram.gov.in
DBT Mission Dashboard
High
FinanceCabinet Secretariat / DBT Mission
All DBT scheme transfers, beneficiaries, districts covered, amount transferred
🔗 https://dbtbharat.gov.in
Open Government Data (OGD)
High
AllMeitY / NIC
6000+ datasets from ministries. Agriculture, education, health, infrastructure, justice
🔗 https://data.gov.in
SECC 2011 (Socio-Economic Survey)
Medium
SocialMinistry of Rural Development
Deprivation indices, auto-included/excluded households, caste/occupation profiles
🔗 https://secc.gov.in
MoPR / e-Panchayat Data
Medium
GovernanceMinistry of Panchayati Raj
Gram Panchayat development plans, SFC reports, elected members
🔗 https://egramswaraj.gov.in
IndiaWRIS (Water Resources)
Medium
WaterMinistry of Jal Shakti
River basin data, reservoir levels, groundwater levels, flood-prone areas
🔗 https://indiawris.gov.in

⚙️Scraper Architecture & Methods

🔌
REST API Scraping
Many NIC portals expose undocumented JSON APIs. Identify via browser DevTools → Network tab.
Tools
axios / node-fetchrate limiting with p-limitRedis cache for responses
Used For
• JJM ejalshakti.gov.in/api• Saubhagya API• MGNREGS nregsportal.nic.in• DBT dbtbharat.gov.in
📊
Scheduled Excel/CSV Download
Most government portals provide downloadable reports. Automate via cron + wget/curl.
Tools
node-cron for schedulingxlsx / csv-parse for parsingPostgreSQL COPY for bulk load
Used For
• HMIS monthly reports• UDISE+ annual district data• APY/MoAFW crop stats• PMGSY district reports
🤖
Headless Browser (Puppeteer)
For JS-rendered portals that require login or complex navigation.
Tools
Puppeteer / PlaywrightStealth plugin to avoid bot detectionScreenshot for verification
Used For
• State assembly portals• Some state revenue portals• ECI results portal
📄
PDF Parsing
State government reports are often PDFs. Extract tabular data programmatically.
Tools
pdf-parse / pdfplumber (Python)Tabula-py for table extractionInfraBoard AI for unstructured PDF parsing
Used For
• Census primary abstracts• State budget documents• Annual reports from ministries
🌐
Open Data Portals
data.gov.in and state OGD portals have structured datasets with API access.
Tools
CKAN API (data.gov.in uses CKAN)OAuth tokens for authenticated accessDataset version tracking
Used For
• data.gov.in (6000+ datasets)• State data portals (Maharashtra, Karnataka, etc.)• Wris.gov.in
🧠
AI-Powered Extraction
Use InfraBoard AI to parse unstructured government text, PDFs, and press releases.
Tools
InfraBoard AI gatewayStructured output schemasConfidence scoring for extracted data
Used For
• State cabinet decisions• Ministry press releases• Planning Commission reports

🚀 Recommended Scraper Stack for nPIP

Layer 1: Orchestration
✓ Node-cron for scheduling
✓ Bull/Redis for job queues
✓ Retry logic with exponential backoff
✓ Dead letter queue for failures
Layer 2: Data Fetch
✓ axios-retry for HTTP scraping
✓ Puppeteer for JS-rendered portals
✓ pdf-parse + tabula for PDFs
✓ SFTP for ministry data drops
Layer 3: Parsing & Validation
✓ Zod schemas for type safety
✓ InfraBoard AI for unstructured data
✓ Data quality checks (null %, range validation)
✓ Deduplication before insert
Layer 4: Storage & Freshness
✓ PostgreSQL COPY for bulk load
✓ data_refresh_log for tracking
✓ UPSERT with conflict resolution
✓ Version history tables
📊 Data Coverage Report🗂️ Dataset Registry📖 Data Dictionary📋 Cron Logs