BranchesTags

Repository files navigation

SANDAG PopulationSim Repository

Introduction

This repository is dedicated to running PopulationSim, a powerful demographic simulation tool used by SANDAG to translate marginal control totals produced by the Estimates & Forecast team to micro-simulated households and persons for use by SANDAG's Activity-Based Model team.

PopulationSim is well-suited for generating detailed household and person-level synthetic populations based on sample data and control totals. It is a important component in urban planning and transportation modeling. Learn more about PopulationSim in its official documentation.

Getting Started

Running PopulationSim

  1. Clone the Repository and ensure an installation of uv exists. Use the pyproject.toml file in the root directory of the project to create the Python virtual environment needed to run the project with uv sync.

  2. Configuration of Private Data in secrets.yml In order to avoid exposing certain data to the public this repository uses a secrets file to store sensitive configurations in addition to a standard configuration file. This file is stored in the root directory of the repository as secrets.yml and is included in the .gitignore intentionally to avoid it ever being committed to the repository.

The secrets.yml should mirror the following structure.

sql:
server: "<SQLInstanceName>"# SQL instance containing seed and control dataschema: "<[SQLSchemaName]>"# E&F team Series 15 UDM schema to use for control dataoutput_database: "<SQLoutputDatabaseName>"# Optional PopulationSim output SQL database
  1. Update the config.yml configuration file in the project root directory
sql:
seed_households: "sql/seed_households.sql"# household seed data queryseed_persons: "sql/seed_persons.sql"# person seed data querymgra_controls: "sql/mgra_controls.sql"# mgra controls data queryregion_controls: "sql/region_controls.sql"# region controls data querymgrabase: "sql/mgrabase.sql"# mgrabase file generation data queryload_to_database: False # Set to True to Load results to databaseeconomic_controls: "data/Economic Team Region Controls.csv"# region economic controls provided by SANDAG's Economics Teamyears: # years for which to generate controls and run populationsim
- 2022
- 2026
- 2029
- 2032
- 2035
- 2040
- 2050

  1. Update PopulationSim configuration files (if necessary)

    • SANDAG commonly sets the populationsim/conigs_mp/settings.yaml file such that multiprocess: True, num_processes: 22, multiprocess_steps: num_processes: 22 to enable the maximum level of multiprocessing using the 22 San Diego PUMAS as the slice_geography: PUMA. If at least 22 logical processors are not available (not advised due to long run times), it is suggested to set both num_processes: configurations to the number of logical processors.
    • See the PopulationSim official documentation

  2. Run the main.py entry point file from the project root directory: On Windows:

uv run main.py

Outputs of PopulationSim

Once completed, the output folder will contain subfolders for each year specified in the config.yml file. Each subfolder will contain the following files.

FileDescription
synthetic_persons_gq.csvPopulationSim output synthetic group quarters persons
synthetic_persons.csvPopulationSim output synthetic persons (non-group quarters)
syntheticpersonsyear.csvCombined synthetic persons file for use by the Activity-Based Model team
synthetic_households_gq.csvPopulationSim output synthetic group quarters households
synthetic_households.csvPopulationSim output synthetic households (non-group quarters)
synthetichouseholdsyear.csvCombined synthetic households file for use by the Activity-Based Model team
mgra15based_inputyear.csvThe mgrabase file for use by the Activity-Based Model team
timing_log.csvPopulationSim log of process runtimes

If running PopulationSim as an official run for use by SANDAG's QA and/or Activity-Based Model teams, update the version tracker at: \\sandag.org\transdata\socioec\Current_Projects\SR15\version_history.xlsx

Note: This is temporary until ABM team feels comfortable with use of production SQL database

Production Database

This repository contains the option in the config.yml to load PopulationSim outputs into a production database. The schema for the database is shown below. input

Streamlit Report App

This repository contains a Streamlit app that generates validation reports for PopulationSim outputs stored in SANDAG's production database. You can use it to visualize the results of the run interactively using Streamlit's easy-to-use interface. The documentation can be found here https://docs.streamlit.io/.

Prerequisites

Before generating the report, ensure that you have the following:

  • Set the proper SQL instance and database containing PopulationSim outputs in the secrets.yml.
  • Are running in a Python virtual environment with all required dependencies listed in the environment.yml.

Generate validation reports

Run the Streamlit app in the base project directory with the following command.

uv run -- streamlit run report/report.py

About

SANDAG's Process to run the ActivitySim PopulationSim

github.com/ActivitySim/populationsim

Topics

transportation-modeling

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

2 watching

Forks

0 forks
Report repository

Releases

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
BranchesTags
Repository files navigation
SANDAG PopulationSim Repository
Introduction
This repository is dedicated to running PopulationSim, a powerful demographic simulation tool used by SANDAG to translate marginal control totals produced by the Estimates & Forecast team to micro-simulated households and persons for use by SANDAG's Activity-Based Model team.
PopulationSim is well-suited for generating detailed household and person-level synthetic populations based on sample data and control totals. It is a important component in urban planning and transportation modeling. Learn more about PopulationSim in its official documentation.
Getting Started
Running PopulationSim
  1. Clone the Repository and ensure an installation of uv exists. Use the pyproject.toml file in the root directory of the project to create the Python virtual environment needed to run the project with uv sync.
  2. Configuration of Private Data in secrets.yml
In order to avoid exposing certain data to the public this repository uses a secrets file to store sensitive configurations in addition to a standard configuration file. This file is stored in the root directory of the repository as secrets.yml and is included in the .gitignore intentionally to avoid it ever being committed to the repository.
The secrets.yml should mirror the following structure.
sql:
server: "<SQLInstanceName>"# SQL instance containing seed and control dataschema: "<[SQLSchemaName]>"# E&F team Series 15 UDM schema to use for control dataoutput_database: "<SQLoutputDatabaseName>"# Optional PopulationSim output SQL database
  1. Update the config.yml configuration file in the project root directory
sql:
seed_households: "sql/seed_households.sql"# household seed data queryseed_persons: "sql/seed_persons.sql"# person seed data querymgra_controls: "sql/mgra_controls.sql"# mgra controls data queryregion_controls: "sql/region_controls.sql"# region controls data querymgrabase: "sql/mgrabase.sql"# mgrabase file generation data queryload_to_database: False # Set to True to Load results to databaseeconomic_controls: "data/Economic Team Region Controls.csv"# region economic controls provided by SANDAG's Economics Teamyears: # years for which to generate controls and run populationsim
- 2022
- 2026
- 2029
- 2032
- 2035
- 2040
- 2050
  1. Update PopulationSim configuration files (if necessary)
    • SANDAG commonly sets the populationsim/conigs_mp/settings.yaml file such that multiprocess: True, num_processes: 22, multiprocess_steps: num_processes: 22 to enable the maximum level of multiprocessing using the 22 San Diego PUMAS as the slice_geography: PUMA. If at least 22 logical processors are not available (not advised due to long run times), it is suggested to set both num_processes: configurations to the number of logical processors.
    • See the PopulationSim official documentation
  2. Run the main.py entry point file from the project root directory:
On Windows:
uv run main.py
Outputs of PopulationSim
Once completed, the output folder will contain subfolders for each year specified in the config.yml file. Each subfolder will contain the following files.
FileDescription
synthetic_persons_gq.csvPopulationSim output synthetic group quarters persons
synthetic_persons.csvPopulationSim output synthetic persons (non-group quarters)
syntheticpersonsyear.csvCombined synthetic persons file for use by the Activity-Based Model team
synthetic_households_gq.csvPopulationSim output synthetic group quarters households
synthetic_households.csvPopulationSim output synthetic households (non-group quarters)
synthetichouseholdsyear.csvCombined synthetic households file for use by the Activity-Based Model team
mgra15based_inputyear.csvThe mgrabase file for use by the Activity-Based Model team
timing_log.csvPopulationSim log of process runtimes
If running PopulationSim as an official run for use by SANDAG's QA and/or Activity-Based Model teams, update the version tracker at: \\sandag.org\transdata\socioec\Current_Projects\SR15\version_history.xlsx
Note: This is temporary until ABM team feels comfortable with use of production SQL database
Production Database
This repository contains the option in the config.yml to load PopulationSim outputs into a production database. The schema for the database is shown below.
input
Streamlit Report App
This repository contains a Streamlit app that generates validation reports for PopulationSim outputs stored in SANDAG's production database. You can use it to visualize the results of the run interactively using Streamlit's easy-to-use interface. The documentation can be found here https://docs.streamlit.io/.
Prerequisites
Before generating the report, ensure that you have the following:
  • Set the proper SQL instance and database containing PopulationSim outputs in the secrets.yml.
  • Are running in a Python virtual environment with all required dependencies listed in the environment.yml.
Generate validation reports
Run the Streamlit app in the base project directory with the following command.
uv run -- streamlit run report/report.py
About
SANDAG's Process to run the ActivitySim PopulationSim
github.com/ActivitySim/populationsim
Topics
transportation-modeling
Resources
Readme
Activity
Custom properties
Stars
0 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
SANDAG PopulationSim Repository
Introduction
This repository is dedicated to running PopulationSim, a powerful demographic simulation tool used by SANDAG to translate marginal control totals produced by the Estimates & Forecast team to micro-simulated households and persons for use by SANDAG's Activity-Based Model team.
PopulationSim is well-suited for generating detailed household and person-level synthetic populations based on sample data and control totals. It is a important component in urban planning and transportation modeling. Learn more about PopulationSim in its official documentation.
Getting Started
Running PopulationSim
  1. Clone the Repository and ensure an installation of uv exists. Use the pyproject.toml file in the root directory of the project to create the Python virtual environment needed to run the project with uv sync.
  2. Configuration of Private Data in secrets.yml
In order to avoid exposing certain data to the public this repository uses a secrets file to store sensitive configurations in addition to a standard configuration file. This file is stored in the root directory of the repository as secrets.yml and is included in the .gitignore intentionally to avoid it ever being committed to the repository.
The secrets.yml should mirror the following structure.
sql:
server: "<SQLInstanceName>"# SQL instance containing seed and control dataschema: "<[SQLSchemaName]>"# E&F team Series 15 UDM schema to use for control dataoutput_database: "<SQLoutputDatabaseName>"# Optional PopulationSim output SQL database
  1. Update the config.yml configuration file in the project root directory
sql:
seed_households: "sql/seed_households.sql"# household seed data queryseed_persons: "sql/seed_persons.sql"# person seed data querymgra_controls: "sql/mgra_controls.sql"# mgra controls data queryregion_controls: "sql/region_controls.sql"# region controls data querymgrabase: "sql/mgrabase.sql"# mgrabase file generation data queryload_to_database: False # Set to True to Load results to databaseeconomic_controls: "data/Economic Team Region Controls.csv"# region economic controls provided by SANDAG's Economics Teamyears: # years for which to generate controls and run populationsim
- 2022
- 2026
- 2029
- 2032
- 2035
- 2040
- 2050
  1. Update PopulationSim configuration files (if necessary)
    • SANDAG commonly sets the populationsim/conigs_mp/settings.yaml file such that multiprocess: True, num_processes: 22, multiprocess_steps: num_processes: 22 to enable the maximum level of multiprocessing using the 22 San Diego PUMAS as the slice_geography: PUMA. If at least 22 logical processors are not available (not advised due to long run times), it is suggested to set both num_processes: configurations to the number of logical processors.
    • See the PopulationSim official documentation
  2. Run the main.py entry point file from the project root directory:
On Windows:
uv run main.py
Outputs of PopulationSim
Once completed, the output folder will contain subfolders for each year specified in the config.yml file. Each subfolder will contain the following files.
FileDescription
synthetic_persons_gq.csvPopulationSim output synthetic group quarters persons
synthetic_persons.csvPopulationSim output synthetic persons (non-group quarters)
syntheticpersonsyear.csvCombined synthetic persons file for use by the Activity-Based Model team
synthetic_households_gq.csvPopulationSim output synthetic group quarters households
synthetic_households.csvPopulationSim output synthetic households (non-group quarters)
synthetichouseholdsyear.csvCombined synthetic households file for use by the Activity-Based Model team
mgra15based_inputyear.csvThe mgrabase file for use by the Activity-Based Model team
timing_log.csvPopulationSim log of process runtimes
If running PopulationSim as an official run for use by SANDAG's QA and/or Activity-Based Model teams, update the version tracker at: \\sandag.org\transdata\socioec\Current_Projects\SR15\version_history.xlsx
Note: This is temporary until ABM team feels comfortable with use of production SQL database
Production Database
This repository contains the option in the config.yml to load PopulationSim outputs into a production database. The schema for the database is shown below.
input
Streamlit Report App
This repository contains a Streamlit app that generates validation reports for PopulationSim outputs stored in SANDAG's production database. You can use it to visualize the results of the run interactively using Streamlit's easy-to-use interface. The documentation can be found here https://docs.streamlit.io/.
Prerequisites
Before generating the report, ensure that you have the following:
  • Set the proper SQL instance and database containing PopulationSim outputs in the secrets.yml.
  • Are running in a Python virtual environment with all required dependencies listed in the environment.yml.
Generate validation reports
Run the Streamlit app in the base project directory with the following command.
uv run -- streamlit run report/report.py
About
SANDAG's Process to run the ActivitySim PopulationSim
github.com/ActivitySim/populationsim
Topics
transportation-modeling
Resources
Readme
Activity
Custom properties
Stars
0 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
SANDAG PopulationSim Repository
Introduction
This repository is dedicated to running PopulationSim, a powerful demographic simulation tool used by SANDAG to translate marginal control totals produced by the Estimates & Forecast team to micro-simulated households and persons for use by SANDAG's Activity-Based Model team.
PopulationSim is well-suited for generating detailed household and person-level synthetic populations based on sample data and control totals. It is a important component in urban planning and transportation modeling. Learn more about PopulationSim in its official documentation.
Getting Started
Running PopulationSim
  1. Clone the Repository and ensure an installation of uv exists. Use the pyproject.toml file in the root directory of the project to create the Python virtual environment needed to run the project with uv sync.
  2. Configuration of Private Data in secrets.yml
In order to avoid exposing certain data to the public this repository uses a secrets file to store sensitive configurations in addition to a standard configuration file. This file is stored in the root directory of the repository as secrets.yml and is included in the .gitignore intentionally to avoid it ever being committed to the repository.
The secrets.yml should mirror the following structure.
sql:
server: "<SQLInstanceName>"# SQL instance containing seed and control dataschema: "<[SQLSchemaName]>"# E&F team Series 15 UDM schema to use for control dataoutput_database: "<SQLoutputDatabaseName>"# Optional PopulationSim output SQL database
  1. Update the config.yml configuration file in the project root directory
sql:
seed_households: "sql/seed_households.sql"# household seed data queryseed_persons: "sql/seed_persons.sql"# person seed data querymgra_controls: "sql/mgra_controls.sql"# mgra controls data queryregion_controls: "sql/region_controls.sql"# region controls data querymgrabase: "sql/mgrabase.sql"# mgrabase file generation data queryload_to_database: False # Set to True to Load results to databaseeconomic_controls: "data/Economic Team Region Controls.csv"# region economic controls provided by SANDAG's Economics Teamyears: # years for which to generate controls and run populationsim
- 2022
- 2026
- 2029
- 2032
- 2035
- 2040
- 2050
  1. Update PopulationSim configuration files (if necessary)
    • SANDAG commonly sets the populationsim/conigs_mp/settings.yaml file such that multiprocess: True, num_processes: 22, multiprocess_steps: num_processes: 22 to enable the maximum level of multiprocessing using the 22 San Diego PUMAS as the slice_geography: PUMA. If at least 22 logical processors are not available (not advised due to long run times), it is suggested to set both num_processes: configurations to the number of logical processors.
    • See the PopulationSim official documentation
  2. Run the main.py entry point file from the project root directory:
On Windows:
uv run main.py
Outputs of PopulationSim
Once completed, the output folder will contain subfolders for each year specified in the config.yml file. Each subfolder will contain the following files.
FileDescription
synthetic_persons_gq.csvPopulationSim output synthetic group quarters persons
synthetic_persons.csvPopulationSim output synthetic persons (non-group quarters)
syntheticpersonsyear.csvCombined synthetic persons file for use by the Activity-Based Model team
synthetic_households_gq.csvPopulationSim output synthetic group quarters households
synthetic_households.csvPopulationSim output synthetic households (non-group quarters)
synthetichouseholdsyear.csvCombined synthetic households file for use by the Activity-Based Model team
mgra15based_inputyear.csvThe mgrabase file for use by the Activity-Based Model team
timing_log.csvPopulationSim log of process runtimes
If running PopulationSim as an official run for use by SANDAG's QA and/or Activity-Based Model teams, update the version tracker at: \\sandag.org\transdata\socioec\Current_Projects\SR15\version_history.xlsx
Note: This is temporary until ABM team feels comfortable with use of production SQL database
Production Database
This repository contains the option in the config.yml to load PopulationSim outputs into a production database. The schema for the database is shown below.
input
Streamlit Report App
This repository contains a Streamlit app that generates validation reports for PopulationSim outputs stored in SANDAG's production database. You can use it to visualize the results of the run interactively using Streamlit's easy-to-use interface. The documentation can be found here https://docs.streamlit.io/.
Prerequisites
Before generating the report, ensure that you have the following:
  • Set the proper SQL instance and database containing PopulationSim outputs in the secrets.yml.
  • Are running in a Python virtual environment with all required dependencies listed in the environment.yml.
Generate validation reports
Run the Streamlit app in the base project directory with the following command.
uv run -- streamlit run report/report.py
About
SANDAG's Process to run the ActivitySim PopulationSim
github.com/ActivitySim/populationsim
Topics
transportation-modeling
Resources
Readme
Activity
Custom properties
Stars
0 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
BranchesTags
Repository files navigation
SANDAG PopulationSim Repository
Introduction
This repository is dedicated to running PopulationSim, a powerful demographic simulation tool used by SANDAG to translate marginal control totals produced by the Estimates & Forecast team to micro-simulated households and persons for use by SANDAG's Activity-Based Model team.
PopulationSim is well-suited for generating detailed household and person-level synthetic populations based on sample data and control totals. It is a important component in urban planning and transportation modeling. Learn more about PopulationSim in its official documentation.
Getting Started
Running PopulationSim
  1. Clone the Repository and ensure an installation of uv exists. Use the pyproject.toml file in the root directory of the project to create the Python virtual environment needed to run the project with uv sync.
  2. Configuration of Private Data in secrets.yml
In order to avoid exposing certain data to the public this repository uses a secrets file to store sensitive configurations in addition to a standard configuration file. This file is stored in the root directory of the repository as secrets.yml and is included in the .gitignore intentionally to avoid it ever being committed to the repository.
The secrets.yml should mirror the following structure.
sql:
server: "<SQLInstanceName>"# SQL instance containing seed and control dataschema: "<[SQLSchemaName]>"# E&F team Series 15 UDM schema to use for control dataoutput_database: "<SQLoutputDatabaseName>"# Optional PopulationSim output SQL database
  1. Update the config.yml configuration file in the project root directory
sql:
seed_households: "sql/seed_households.sql"# household seed data queryseed_persons: "sql/seed_persons.sql"# person seed data querymgra_controls: "sql/mgra_controls.sql"# mgra controls data queryregion_controls: "sql/region_controls.sql"# region controls data querymgrabase: "sql/mgrabase.sql"# mgrabase file generation data queryload_to_database: False # Set to True to Load results to databaseeconomic_controls: "data/Economic Team Region Controls.csv"# region economic controls provided by SANDAG's Economics Teamyears: # years for which to generate controls and run populationsim
- 2022
- 2026
- 2029
- 2032
- 2035
- 2040
- 2050
  1. Update PopulationSim configuration files (if necessary)
    • SANDAG commonly sets the populationsim/conigs_mp/settings.yaml file such that multiprocess: True, num_processes: 22, multiprocess_steps: num_processes: 22 to enable the maximum level of multiprocessing using the 22 San Diego PUMAS as the slice_geography: PUMA. If at least 22 logical processors are not available (not advised due to long run times), it is suggested to set both num_processes: configurations to the number of logical processors.
    • See the PopulationSim official documentation
  2. Run the main.py entry point file from the project root directory:
On Windows:
uv run main.py
Outputs of PopulationSim
Once completed, the output folder will contain subfolders for each year specified in the config.yml file. Each subfolder will contain the following files.
FileDescription
synthetic_persons_gq.csvPopulationSim output synthetic group quarters persons
synthetic_persons.csvPopulationSim output synthetic persons (non-group quarters)
syntheticpersonsyear.csvCombined synthetic persons file for use by the Activity-Based Model team
synthetic_households_gq.csvPopulationSim output synthetic group quarters households
synthetic_households.csvPopulationSim output synthetic households (non-group quarters)
synthetichouseholdsyear.csvCombined synthetic households file for use by the Activity-Based Model team
mgra15based_inputyear.csvThe mgrabase file for use by the Activity-Based Model team
timing_log.csvPopulationSim log of process runtimes
If running PopulationSim as an official run for use by SANDAG's QA and/or Activity-Based Model teams, update the version tracker at: \\sandag.org\transdata\socioec\Current_Projects\SR15\version_history.xlsx
Note: This is temporary until ABM team feels comfortable with use of production SQL database
Production Database
This repository contains the option in the config.yml to load PopulationSim outputs into a production database. The schema for the database is shown below.
input
Streamlit Report App
This repository contains a Streamlit app that generates validation reports for PopulationSim outputs stored in SANDAG's production database. You can use it to visualize the results of the run interactively using Streamlit's easy-to-use interface. The documentation can be found here https://docs.streamlit.io/.
Prerequisites
Before generating the report, ensure that you have the following:
  • Set the proper SQL instance and database containing PopulationSim outputs in the secrets.yml.
  • Are running in a Python virtual environment with all required dependencies listed in the environment.yml.
Generate validation reports
Run the Streamlit app in the base project directory with the following command.
uv run -- streamlit run report/report.py
About
SANDAG's Process to run the ActivitySim PopulationSim
github.com/ActivitySim/populationsim
Topics
transportation-modeling
Resources
Readme
Activity
Custom properties
Stars
0 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
SANDAG PopulationSim Repository
Introduction
This repository is dedicated to running PopulationSim, a powerful demographic simulation tool used by SANDAG to translate marginal control totals produced by the Estimates & Forecast team to micro-simulated households and persons for use by SANDAG's Activity-Based Model team.
PopulationSim is well-suited for generating detailed household and person-level synthetic populations based on sample data and control totals. It is a important component in urban planning and transportation modeling. Learn more about PopulationSim in its official documentation.
Getting Started
Running PopulationSim
  1. Clone the Repository and ensure an installation of uv exists. Use the pyproject.toml file in the root directory of the project to create the Python virtual environment needed to run the project with uv sync.
  2. Configuration of Private Data in secrets.yml
In order to avoid exposing certain data to the public this repository uses a secrets file to store sensitive configurations in addition to a standard configuration file. This file is stored in the root directory of the repository as secrets.yml and is included in the .gitignore intentionally to avoid it ever being committed to the repository.
The secrets.yml should mirror the following structure.
sql:
server: "<SQLInstanceName>"# SQL instance containing seed and control dataschema: "<[SQLSchemaName]>"# E&F team Series 15 UDM schema to use for control dataoutput_database: "<SQLoutputDatabaseName>"# Optional PopulationSim output SQL database
  1. Update the config.yml configuration file in the project root directory
sql:
seed_households: "sql/seed_households.sql"# household seed data queryseed_persons: "sql/seed_persons.sql"# person seed data querymgra_controls: "sql/mgra_controls.sql"# mgra controls data queryregion_controls: "sql/region_controls.sql"# region controls data querymgrabase: "sql/mgrabase.sql"# mgrabase file generation data queryload_to_database: False # Set to True to Load results to databaseeconomic_controls: "data/Economic Team Region Controls.csv"# region economic controls provided by SANDAG's Economics Teamyears: # years for which to generate controls and run populationsim
- 2022
- 2026
- 2029
- 2032
- 2035
- 2040
- 2050
  1. Update PopulationSim configuration files (if necessary)
    • SANDAG commonly sets the populationsim/conigs_mp/settings.yaml file such that multiprocess: True, num_processes: 22, multiprocess_steps: num_processes: 22 to enable the maximum level of multiprocessing using the 22 San Diego PUMAS as the slice_geography: PUMA. If at least 22 logical processors are not available (not advised due to long run times), it is suggested to set both num_processes: configurations to the number of logical processors.
    • See the PopulationSim official documentation
  2. Run the main.py entry point file from the project root directory:
On Windows:
uv run main.py
Outputs of PopulationSim
Once completed, the output folder will contain subfolders for each year specified in the config.yml file. Each subfolder will contain the following files.
FileDescription
synthetic_persons_gq.csvPopulationSim output synthetic group quarters persons
synthetic_persons.csvPopulationSim output synthetic persons (non-group quarters)
syntheticpersonsyear.csvCombined synthetic persons file for use by the Activity-Based Model team
synthetic_households_gq.csvPopulationSim output synthetic group quarters households
synthetic_households.csvPopulationSim output synthetic households (non-group quarters)
synthetichouseholdsyear.csvCombined synthetic households file for use by the Activity-Based Model team
mgra15based_inputyear.csvThe mgrabase file for use by the Activity-Based Model team
timing_log.csvPopulationSim log of process runtimes
If running PopulationSim as an official run for use by SANDAG's QA and/or Activity-Based Model teams, update the version tracker at: \\sandag.org\transdata\socioec\Current_Projects\SR15\version_history.xlsx
Note: This is temporary until ABM team feels comfortable with use of production SQL database
Production Database
This repository contains the option in the config.yml to load PopulationSim outputs into a production database. The schema for the database is shown below.
input
Streamlit Report App
This repository contains a Streamlit app that generates validation reports for PopulationSim outputs stored in SANDAG's production database. You can use it to visualize the results of the run interactively using Streamlit's easy-to-use interface. The documentation can be found here https://docs.streamlit.io/.
Prerequisites
Before generating the report, ensure that you have the following:
  • Set the proper SQL instance and database containing PopulationSim outputs in the secrets.yml.
  • Are running in a Python virtual environment with all required dependencies listed in the environment.yml.
Generate validation reports
Run the Streamlit app in the base project directory with the following command.
uv run -- streamlit run report/report.py
About
SANDAG's Process to run the ActivitySim PopulationSim
github.com/ActivitySim/populationsim
Topics
transportation-modeling
Resources
Readme
Activity
Custom properties
Stars
0 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
SANDAG PopulationSim Repository
Introduction
This repository is dedicated to running PopulationSim, a powerful demographic simulation tool used by SANDAG to translate marginal control totals produced by the Estimates & Forecast team to micro-simulated households and persons for use by SANDAG's Activity-Based Model team.
PopulationSim is well-suited for generating detailed household and person-level synthetic populations based on sample data and control totals. It is a important component in urban planning and transportation modeling. Learn more about PopulationSim in its official documentation.
Getting Started
Running PopulationSim
  1. Clone the Repository and ensure an installation of uv exists. Use the pyproject.toml file in the root directory of the project to create the Python virtual environment needed to run the project with uv sync.
  2. Configuration of Private Data in secrets.yml
In order to avoid exposing certain data to the public this repository uses a secrets file to store sensitive configurations in addition to a standard configuration file. This file is stored in the root directory of the repository as secrets.yml and is included in the .gitignore intentionally to avoid it ever being committed to the repository.
The secrets.yml should mirror the following structure.
sql:
server: "<SQLInstanceName>"# SQL instance containing seed and control dataschema: "<[SQLSchemaName]>"# E&F team Series 15 UDM schema to use for control dataoutput_database: "<SQLoutputDatabaseName>"# Optional PopulationSim output SQL database
  1. Update the config.yml configuration file in the project root directory
sql:
seed_households: "sql/seed_households.sql"# household seed data queryseed_persons: "sql/seed_persons.sql"# person seed data querymgra_controls: "sql/mgra_controls.sql"# mgra controls data queryregion_controls: "sql/region_controls.sql"# region controls data querymgrabase: "sql/mgrabase.sql"# mgrabase file generation data queryload_to_database: False # Set to True to Load results to databaseeconomic_controls: "data/Economic Team Region Controls.csv"# region economic controls provided by SANDAG's Economics Teamyears: # years for which to generate controls and run populationsim
- 2022
- 2026
- 2029
- 2032
- 2035
- 2040
- 2050
  1. Update PopulationSim configuration files (if necessary)
    • SANDAG commonly sets the populationsim/conigs_mp/settings.yaml file such that multiprocess: True, num_processes: 22, multiprocess_steps: num_processes: 22 to enable the maximum level of multiprocessing using the 22 San Diego PUMAS as the slice_geography: PUMA. If at least 22 logical processors are not available (not advised due to long run times), it is suggested to set both num_processes: configurations to the number of logical processors.
    • See the PopulationSim official documentation
  2. Run the main.py entry point file from the project root directory:
On Windows:
uv run main.py
Outputs of PopulationSim
Once completed, the output folder will contain subfolders for each year specified in the config.yml file. Each subfolder will contain the following files.
FileDescription
synthetic_persons_gq.csvPopulationSim output synthetic group quarters persons
synthetic_persons.csvPopulationSim output synthetic persons (non-group quarters)
syntheticpersonsyear.csvCombined synthetic persons file for use by the Activity-Based Model team
synthetic_households_gq.csvPopulationSim output synthetic group quarters households
synthetic_households.csvPopulationSim output synthetic households (non-group quarters)
synthetichouseholdsyear.csvCombined synthetic households file for use by the Activity-Based Model team
mgra15based_inputyear.csvThe mgrabase file for use by the Activity-Based Model team
timing_log.csvPopulationSim log of process runtimes
If running PopulationSim as an official run for use by SANDAG's QA and/or Activity-Based Model teams, update the version tracker at: \\sandag.org\transdata\socioec\Current_Projects\SR15\version_history.xlsx
Note: This is temporary until ABM team feels comfortable with use of production SQL database
Production Database
This repository contains the option in the config.yml to load PopulationSim outputs into a production database. The schema for the database is shown below.
input
Streamlit Report App
This repository contains a Streamlit app that generates validation reports for PopulationSim outputs stored in SANDAG's production database. You can use it to visualize the results of the run interactively using Streamlit's easy-to-use interface. The documentation can be found here https://docs.streamlit.io/.
Prerequisites
Before generating the report, ensure that you have the following:
  • Set the proper SQL instance and database containing PopulationSim outputs in the secrets.yml.
  • Are running in a Python virtual environment with all required dependencies listed in the environment.yml.
Generate validation reports
Run the Streamlit app in the base project directory with the following command.
uv run -- streamlit run report/report.py
About
SANDAG's Process to run the ActivitySim PopulationSim
github.com/ActivitySim/populationsim
Topics
transportation-modeling
Resources
Readme
Activity
Custom properties
Stars
0 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
BranchesTags
Repository files navigation
SANDAG PopulationSim Repository
Introduction
This repository is dedicated to running PopulationSim, a powerful demographic simulation tool used by SANDAG to translate marginal control totals produced by the Estimates & Forecast team to micro-simulated households and persons for use by SANDAG's Activity-Based Model team.
PopulationSim is well-suited for generating detailed household and person-level synthetic populations based on sample data and control totals. It is a important component in urban planning and transportation modeling. Learn more about PopulationSim in its official documentation.
Getting Started
Running PopulationSim
  1. Clone the Repository and ensure an installation of uv exists. Use the pyproject.toml file in the root directory of the project to create the Python virtual environment needed to run the project with uv sync.
  2. Configuration of Private Data in secrets.yml
In order to avoid exposing certain data to the public this repository uses a secrets file to store sensitive configurations in addition to a standard configuration file. This file is stored in the root directory of the repository as secrets.yml and is included in the .gitignore intentionally to avoid it ever being committed to the repository.
The secrets.yml should mirror the following structure.
sql:
server: "<SQLInstanceName>"# SQL instance containing seed and control dataschema: "<[SQLSchemaName]>"# E&F team Series 15 UDM schema to use for control dataoutput_database: "<SQLoutputDatabaseName>"# Optional PopulationSim output SQL database
  1. Update the config.yml configuration file in the project root directory
sql:
seed_households: "sql/seed_households.sql"# household seed data queryseed_persons: "sql/seed_persons.sql"# person seed data querymgra_controls: "sql/mgra_controls.sql"# mgra controls data queryregion_controls: "sql/region_controls.sql"# region controls data querymgrabase: "sql/mgrabase.sql"# mgrabase file generation data queryload_to_database: False # Set to True to Load results to databaseeconomic_controls: "data/Economic Team Region Controls.csv"# region economic controls provided by SANDAG's Economics Teamyears: # years for which to generate controls and run populationsim
- 2022
- 2026
- 2029
- 2032
- 2035
- 2040
- 2050
  1. Update PopulationSim configuration files (if necessary)
    • SANDAG commonly sets the populationsim/conigs_mp/settings.yaml file such that multiprocess: True, num_processes: 22, multiprocess_steps: num_processes: 22 to enable the maximum level of multiprocessing using the 22 San Diego PUMAS as the slice_geography: PUMA. If at least 22 logical processors are not available (not advised due to long run times), it is suggested to set both num_processes: configurations to the number of logical processors.
    • See the PopulationSim official documentation
  2. Run the main.py entry point file from the project root directory:
On Windows:
uv run main.py
Outputs of PopulationSim
Once completed, the output folder will contain subfolders for each year specified in the config.yml file. Each subfolder will contain the following files.
FileDescription
synthetic_persons_gq.csvPopulationSim output synthetic group quarters persons
synthetic_persons.csvPopulationSim output synthetic persons (non-group quarters)
syntheticpersonsyear.csvCombined synthetic persons file for use by the Activity-Based Model team
synthetic_households_gq.csvPopulationSim output synthetic group quarters households
synthetic_households.csvPopulationSim output synthetic households (non-group quarters)
synthetichouseholdsyear.csvCombined synthetic households file for use by the Activity-Based Model team
mgra15based_inputyear.csvThe mgrabase file for use by the Activity-Based Model team
timing_log.csvPopulationSim log of process runtimes
If running PopulationSim as an official run for use by SANDAG's QA and/or Activity-Based Model teams, update the version tracker at: \\sandag.org\transdata\socioec\Current_Projects\SR15\version_history.xlsx
Note: This is temporary until ABM team feels comfortable with use of production SQL database
Production Database
This repository contains the option in the config.yml to load PopulationSim outputs into a production database. The schema for the database is shown below.
input
Streamlit Report App
This repository contains a Streamlit app that generates validation reports for PopulationSim outputs stored in SANDAG's production database. You can use it to visualize the results of the run interactively using Streamlit's easy-to-use interface. The documentation can be found here https://docs.streamlit.io/.
Prerequisites
Before generating the report, ensure that you have the following:
  • Set the proper SQL instance and database containing PopulationSim outputs in the secrets.yml.
  • Are running in a Python virtual environment with all required dependencies listed in the environment.yml.
Generate validation reports
Run the Streamlit app in the base project directory with the following command.
uv run -- streamlit run report/report.py
About
SANDAG's Process to run the ActivitySim PopulationSim
github.com/ActivitySim/populationsim
Topics
transportation-modeling
Resources
Readme
Activity
Custom properties
Stars
0 stars
Watchers
2 watching
Forks
0 forks
Report repository
Releases
Contributors
Languages