Download Script Fix + PVmap Changes in Mongolia Imports (Mongolia_Education, Mongolia_Health, Mongolia_Employment, Mongolia_Demographics

[kartik-s21]

This PR addresses data validation failures, point deletions, and execution timeouts across all four Mongolia imports (mongolia_education, mongolia_demographics, mongolia_health, and mongolia_employment).

With these fixes, all four datasets generate cleanly with 0.00% point deletions and execute without API network timeouts or connection dropouts.

Key Changes:

1. Common Infrastructure (common_download_script.py)
   Updated table extraction, parsing logic, and retry formatting for Education, Demographics, and Health datasets to prevent data truncation and column shifts during downloading.

2. Mongolia Education (mongolia_education)
   Added explicit date mappings for survey years 2024 and 2025 across all 6 education PVMAP tables.
   Updated run.sh and place mappings to cleanly handle regional groupings, resolving all historical point deletions (+350 newly recovered observation points).
   Document for deletions - link
   CL for missing references - link
   Validation Output - input0input1input2input3input4input5

3. Mongolia Demographics (mongolia_demographics)
   Fixed 19.35% record deletion in marital status statistics (resident_population_by_agegroup_15_and_over_and_maritalstatus_pvmap.csv) by re-enabling the missing Age Group:Total mapping.
   Added explicit column key prefixes (Marital Status:, Age Group:, Gender:) to prevent ambiguous "Total" matches from creating duplicate observation conflicts.
   Validation Output - input0input1input2input3input4input5input6

4. Mongolia Health (mongolia_health)
   Corrected column extraction and monthly time-series parsing logic for births, deaths, infant mortality, and child delivery tables, eliminating all dropped historical observation dates through 2025.
   Document for deletions - link
   Validation Output - input0input1input2input3input4input5

5. Mongolia Employment (mongolia_employment)
   Fixed Cloud Batch container timeout (InvalidDCInstanceError) caused by over 100,000 fallback API calls during place resolution.
   Regenerated places_resolved.csv with 134 exhaustive mappings that account for leading whitespace hierarchy indentation.
   Validation Output - link

PR Checklist - link

[gemini-code-assist]

Code Review

This pull request updates the Mongolian demographics and education import scripts. It refactors common_download_script.py to dynamically resolve table paths and fetch metadata (English and Mongolian) from the API, mapping the POST response data to the expected format. It also extends observation dates up to 2030 in several mapping files and updates place resolution configurations. Feedback focuses on optimizing performance by caching API catalog requests and pre-building lookup dictionaries to avoid inefficient .index() lookups in loops. Additionally, it is recommended to increase the retry count for robust downloads and strip whitespaces from keys to normalize place names in places_resolved.csv.

[saanikaaa]

Pls add differ results with the latest changes

[saanikaaa]

pls add output_counter

[kartik-s21]

pls add output_counter

done

[kartik-s21]

Pls add differ results with the latest changes

done

[saanikaaa]

the key in source only says Total, and we have mapped it to age 15 and above. Is it correct? Pls attach source reference to validated

[saanikaaa]

we should avoid adding this file, as missing reference wont show up in cloud lint and we will fail to add a new schema if any introduced.
Also, if we are adding this now because we were getting any missing reference pls create schema CL with new schema.

[saanikaaa]

Same comment across entire file