Preprocessing & Annotation functionality - #957

Merged
jpn-- merged 9 commits into
ActivitySim:mainfrom
RSGInc:preprocessing
Jul 17, 2025
Merged

Preprocessing & Annotation functionality#957
jpn-- merged 9 commits into
ActivitySim:mainfrom
RSGInc:preprocessing

Conversation

@dhensle

@dhensledhensle commented Jun 17, 2025

Copy link
Copy Markdown
Contributor

This pull request standardizes preprocessing and annotation capabilities across all activitysim abm models.

Refactoring of Preprocessing and Table Annotation

  • Standardized Preprocessing:

    • Replaced custom preprocessing logic using expressions.assign_columns with expressions.annotate_preprocessors in models such as auto_ownership, free_parking, and joint_tour_composition. This change ensures consistency across models and reduces duplication. [1][2][3]

  • Unified Table Annotation:

    • Replaced individual table annotation logic with expressions.annotate_tables in models like atwork_subtour_destination, cdap, and joint_tour_frequency. This provides a consistent and reusable approach for post-model table updates. [1][2][3]

Simplification of Model Settings

  • Removal of Unused Fields:

    • Removed unnecessary fields like preprocessor and annotate_* from model settings classes such as AtworkSubtourFrequencySettings, AutoOwnershipSettings, and FreeParkingSettings. This cleanup eliminates unused or redundant configuration options. [1][2][3]

  • Reorganization in cdap:

    • Consolidated preprocessor and annotation settings into a single unified structure for better clarity and alignment with other models.

Summary table of model changes:

Model nameChoosers PreprocessorAlts PreprocessorAnnotatorNotes
accessibilityexistedN/AN/Amodel is essentially a preprocessor
atwork_subtour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
atwork_subtour_frequencyupdatedN/Aadded
atwork_subtour_mode_choiceupdatedN/Aadded
atwork_subtour_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
auto_ownershipupdatedN/Aadded
cdapaddedN/Aupdated
disaggregate_accessibilityexistedN/Aexistednow can use alts preprocessors in logsum calculations through tour and location choice models
free_parkingupdatedN/Aupdated
initializeN/AN/AupdatedModel is annotator
initialize_losN/AN/AN/A
initialize_toursN/AN/AupdatedModel is annotator
input_checkerN/AN/AN/A
joint_tour_compositionupdatedN/Aupdated
joint_tour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
joint_tour_frequencyupdatedN/Aadded
joint_tour_frequency_compositionupdatedupdatedupdated
joint_tour_participationupdatedN/Aadded
joint_tour_schedulingupdatedaddedupdatedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
location_choiceaddedaddedupdatedadded preprocessing in both sample and simulate step
mandatory_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
mandatory_tour_frequencyupdatedN/Aadded
non_mandatory_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
non_mandatory_schedulingaddedaddedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
non_mandatory_tour_frequencyupdatedaddedadded
parking_location_choiceupdatedaddedadded
school_escortingexistedaddedaddeduniquely has 3 preprocessors
stop_frequencyupdatedN/Aadded
summarizeN/AN/AexistedModel is annotator
telecommute_frequencyupdatedN/Aadded
tour_mode_choiceupdatedN/Aadded
tour_od_choiceupdatedaddedaddedadded preprocessing in both sample and simulate step
tour_scheduling_probabilisticN/AN/Aadded
transit_pass_ownershipupdatedN/Aadded
transit_pass_subsidyupdatedN/Aadded
trip_departure_choiceexistedaddedadded
trip_destinationupdatedupdatedaddedadded preprocessing in both sample and simulate step
trip_matricesexistedN/AN/A
trip_mode_choiceexistedN/Aupdated
trip_purposeupdatedN/Aadded
trip_purpose_and_destinationN/AN/Aadded
trip_schedulingupdatedN/Aaddedis probabilistic model
trip_scheduling_choiceupdatedaddedadded
vehicle_allocationupdatedN/Aadded
vehicle_type_choiceupdatedupdatedupdated
work_from_homeupdatedN/Aupdated

@jpn--
jpn-- requested a review from CopilotJune 17, 2025 18:45
CopilotAI reviewed Jun 17, 2025
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request standardizes the preprocessing and table annotation functionality across multiple ABM models by replacing custom assign‐columns logic with unified calls to expressions.annotate_preprocessors and expressions.annotate_tables. Key changes include:

  • Replacing custom preprocessing calls with standardized annotation functions.
  • Removing unused/deprecated settings and consolidating configuration fields.
  • Updating multiple model files (e.g., auto_ownership, free_parking, joint_tour_frequency, etc.) to support the new standardized approach.

Reviewed Changes

Copilot reviewed 47 out of 47 changed files in this pull request and generated 1 comment.

Show a summary per file
FileDescription
activitysim/abm/models/parking_location_choice.pyAdded deprecation warning handling and introduced an alternative preprocessor field.
activitysim/abm/models/non_mandatory_tour_frequency.pyReplaced assign_columns with annotate_preprocessors and updated locals dictionary usage.
activitysim/abm/models/non_mandatory_scheduling.pyUpdated preprocessing and table annotation calls with new expressions functions.
activitysim/abm/models/non_mandatory_destination.pyRemoved legacy annotation function calls and standardized table annotation.
activitysim/abm/models/mandatory_tour_frequency.pyConsolidated preprocessing logic into standardized annotate_preprocessors and annotate_tables calls.
activitysim/abm/models/location_choice.pyUpdated both chooser and alternative preprocessing with standardized annotation calls.
activitysim/abm/models/joint_*.pyUnified preprocessing and annotation functions across joint tour models.
activitysim/abm/models/free_parking.pyUpdated preprocessing calls to use expressions.annotate_preprocessors and annotate_tables.
activitysim/abm/models/cdap.pyAdded preprocessing and post-model annotation functions; restructured settings fields.
activitysim/abm/models/auto_ownership.pyRemoved unused preprocessor settings by moving to a pass-through structure.
activitysim/abm/models/atwork_* (scheduling, mode_choice, frequency, destination)Consolidated preprocessing and table annotation logic with the new standardized approach.
Comments suppressed due to low confidence (1)

activitysim/abm/models/location_choice.py:184

  • [nitpick] The alternative preprocessor setting name 'alts_preprocessor_sample' differs from similar settings in other models. Consider adopting a consistent naming convention (e.g., 'alts_preprocessor') across the codebase.
 preprocessor_setting_name="alts_preprocessor_sample",

Comment threadactivitysim/abm/models/non_mandatory_tour_frequency.py
@jpn--
jpn-- self-requested a review June 17, 2025 18:52
@jpn--

jpn-- commented Jun 17, 2025

Copy link
Copy Markdown
Member

A few thoughts before I dig into this more:

  1. Estimation mode tests are failing.
  2. Other than estimation mode, existing tests do show we have not broken anything that was tested. But this PR appears to add significant functionality, and lacks any testing of that new functionality.

@dhensle

dhensle commented Jun 17, 2025

Copy link
Copy Markdown
ContributorAuthor

Hey @jpn--, thanks for your first look. As mentioned during the call today, I have a few more things to clean up on the CI tests (which is why the PR is listed as a "draft"). I will ping you in the next few days when my commits are complete and this PR is ready for review.

@dhensle
dhensle marked this pull request as ready for review June 18, 2025 23:35
@dhensle

dhensle commented Jun 18, 2025

Copy link
Copy Markdown
ContributorAuthor

@jpn-- Ok, this is ready for your review. I fixed the estimation mode issue and added unit tests for both the preprocessor and annotation functions which include some common expressions including reindex, skims lookups (in multiple ways), and some groupby calculations across tables. If you have other CI test ideas, I am happy to hear them.

jpn--
jpn-- requested changes Jun 27, 2025
View reviewed changes

@jpn--jpn-- left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! I found very little substantive to address, just a few things to clean up. Thanks for all the hard work on this.

Comment threadactivitysim/abm/models/vehicle_allocation.py Outdated
"""Setting for the preprocessor."""

OCCUPANCY_LEVELS: list = [1] # TODO Check this
OCCUPANCY_LEVELS: list = [1, 2, 3.5]

@jpn--jpn--Jun 27, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearly, the previous default value here of [1] was not a great solution. But it seems the occupancy levels set here needs to jive with calls to the generated variable names set in spec files for other components. "df.vehicle_occup_1" is going to show up as a variable name elsewhere in other components. I think it would be better to make that clear in the docstring, and have no default at all (i.e. failure to define this in the config for this model is an error).

@jpn--jpn--Jun 27, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this appears to be basically unrelated to the fundamental nature of this PR, maybe just leave it out here, and we can address it in a separate specific PR for this.

@dhensledhensleJul 1, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, we can address this separately. Reverted.

Comment threadactivitysim/abm/models/mandatory_tour_frequency.py

@jpn--jpn--Jun 27, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MandatoryTourFrequencySettings still has a redundant annotate_persons defined, which it also inherits from LogitComponentSettings

@dhensledhensleJul 1, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, removed.

Comment threadactivitysim/abm/models/cdap.py

preprocessor: PreprocessorSettings | None = None
"""Preprocess choosers tables before running the model."""
annotate_persons: PreprocessorSettings | None = None

@jpn--jpn--Jun 27, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automatic documentation tools will work better if annotate_persons has its own docstring, even though it's clear to a human reader the one docstring below applies to both.

@dhensledhensleJul 1, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment threadactivitysim/core/configuration/logit.py
utility expressions.
"""

annotate_households: PreprocessorSettings | None = None

@jpn--jpn--Jun 27, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For autodoc, have each annotate_* field have its own docstring.

@dhensledhensleJul 1, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment threadactivitysim/core/util.py Outdated
unique_variables_in_spec.update(re.findall(pattern, custom_chooser_lines))

logger.info("Dropping unused variables in chooser table")
logger.debug("Dropping unused variables in chooser table")

@jpn--jpn--Jun 27, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is clearly unrelated to this PR.

I do think we should address our logging, by default we are logging far too much

@dhensledhensleJul 1, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I put this in my code cause it was annoying to look at while debugging. Happy to address this in a separate logging task. reverted.

@dhensle

dhensle commented Jul 1, 2025

Copy link
Copy Markdown
ContributorAuthor

@jpn-- thanks for your comments. I have addressed them, but please let me know if there's anything else you find!

@jpn--
jpn-- merged commit ba4a12e into ActivitySim:mainJul 17, 2025
@dhensledhensle mentioned this pull request Aug 1, 2025
Merged
@dhensle
dhensle deleted the preprocessing branch August 1, 2025 17:38
@dhensledhensle mentioned this pull request Aug 13, 2025
Closed
@dhensledhensle mentioned this pull request Oct 28, 2025
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@jpn--jpn--jpn-- approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dhensle@jpn--@aletzdy
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Preprocessing & Annotation functionality - #957
Merged
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Jul 17, 2025
Merged
Preprocessing & Annotation functionality#957
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Conversation
@dhensle
@dhensledhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

Contributor
This pull request standardizes preprocessing and annotation capabilities across all activitysim abm models.
Refactoring of Preprocessing and Table Annotation
  • Standardized Preprocessing:
    • Replaced custom preprocessing logic using expressions.assign_columns with expressions.annotate_preprocessors in models such as auto_ownership, free_parking, and joint_tour_composition. This change ensures consistency across models and reduces duplication. [1][2][3]
  • Unified Table Annotation:
    • Replaced individual table annotation logic with expressions.annotate_tables in models like atwork_subtour_destination, cdap, and joint_tour_frequency. This provides a consistent and reusable approach for post-model table updates. [1][2][3]
Simplification of Model Settings
  • Removal of Unused Fields:
    • Removed unnecessary fields like preprocessor and annotate_* from model settings classes such as AtworkSubtourFrequencySettings, AutoOwnershipSettings, and FreeParkingSettings. This cleanup eliminates unused or redundant configuration options. [1][2][3]
  • Reorganization in cdap:
    • Consolidated preprocessor and annotation settings into a single unified structure for better clarity and alignment with other models.
Summary table of model changes:
Model nameChoosers PreprocessorAlts PreprocessorAnnotatorNotes
accessibilityexistedN/AN/Amodel is essentially a preprocessor
atwork_subtour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
atwork_subtour_frequencyupdatedN/Aadded
atwork_subtour_mode_choiceupdatedN/Aadded
atwork_subtour_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
auto_ownershipupdatedN/Aadded
cdapaddedN/Aupdated
disaggregate_accessibilityexistedN/Aexistednow can use alts preprocessors in logsum calculations through tour and location choice models
free_parkingupdatedN/Aupdated
initializeN/AN/AupdatedModel is annotator
initialize_losN/AN/AN/A
initialize_toursN/AN/AupdatedModel is annotator
input_checkerN/AN/AN/A
joint_tour_compositionupdatedN/Aupdated
joint_tour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
joint_tour_frequencyupdatedN/Aadded
joint_tour_frequency_compositionupdatedupdatedupdated
joint_tour_participationupdatedN/Aadded
joint_tour_schedulingupdatedaddedupdatedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
location_choiceaddedaddedupdatedadded preprocessing in both sample and simulate step
mandatory_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
mandatory_tour_frequencyupdatedN/Aadded
non_mandatory_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
non_mandatory_schedulingaddedaddedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
non_mandatory_tour_frequencyupdatedaddedadded
parking_location_choiceupdatedaddedadded
school_escortingexistedaddedaddeduniquely has 3 preprocessors
stop_frequencyupdatedN/Aadded
summarizeN/AN/AexistedModel is annotator
telecommute_frequencyupdatedN/Aadded
tour_mode_choiceupdatedN/Aadded
tour_od_choiceupdatedaddedaddedadded preprocessing in both sample and simulate step
tour_scheduling_probabilisticN/AN/Aadded
transit_pass_ownershipupdatedN/Aadded
transit_pass_subsidyupdatedN/Aadded
trip_departure_choiceexistedaddedadded
trip_destinationupdatedupdatedaddedadded preprocessing in both sample and simulate step
trip_matricesexistedN/AN/A
trip_mode_choiceexistedN/Aupdated
trip_purposeupdatedN/Aadded
trip_purpose_and_destinationN/AN/Aadded
trip_schedulingupdatedN/Aaddedis probabilistic model
trip_scheduling_choiceupdatedaddedadded
vehicle_allocationupdatedN/Aadded
vehicle_type_choiceupdatedupdatedupdated
work_from_homeupdatedN/Aupdated
@jpn--
jpn--
requested a review
from CopilotJune 17, 2025 18:45
CopilotAI
reviewed
Jun 17, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This pull request standardizes the preprocessing and table annotation functionality across multiple ABM models by replacing custom assign‐columns logic with unified calls to expressions.annotate_preprocessors and expressions.annotate_tables. Key changes include:
  • Replacing custom preprocessing calls with standardized annotation functions.
  • Removing unused/deprecated settings and consolidating configuration fields.
  • Updating multiple model files (e.g., auto_ownership, free_parking, joint_tour_frequency, etc.) to support the new standardized approach.
Reviewed Changes
Copilot reviewed 47 out of 47 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
activitysim/abm/models/parking_location_choice.pyAdded deprecation warning handling and introduced an alternative preprocessor field.
activitysim/abm/models/non_mandatory_tour_frequency.pyReplaced assign_columns with annotate_preprocessors and updated locals dictionary usage.
activitysim/abm/models/non_mandatory_scheduling.pyUpdated preprocessing and table annotation calls with new expressions functions.
activitysim/abm/models/non_mandatory_destination.pyRemoved legacy annotation function calls and standardized table annotation.
activitysim/abm/models/mandatory_tour_frequency.pyConsolidated preprocessing logic into standardized annotate_preprocessors and annotate_tables calls.
activitysim/abm/models/location_choice.pyUpdated both chooser and alternative preprocessing with standardized annotation calls.
activitysim/abm/models/joint_*.pyUnified preprocessing and annotation functions across joint tour models.
activitysim/abm/models/free_parking.pyUpdated preprocessing calls to use expressions.annotate_preprocessors and annotate_tables.
activitysim/abm/models/cdap.pyAdded preprocessing and post-model annotation functions; restructured settings fields.
activitysim/abm/models/auto_ownership.pyRemoved unused preprocessor settings by moving to a pass-through structure.
activitysim/abm/models/atwork_* (scheduling, mode_choice, frequency, destination)Consolidated preprocessing and table annotation logic with the new standardized approach.
Comments suppressed due to low confidence (1)
activitysim/abm/models/location_choice.py:184
  • [nitpick] The alternative preprocessor setting name 'alts_preprocessor_sample' differs from similar settings in other models. Consider adopting a consistent naming convention (e.g., 'alts_preprocessor') across the codebase.
 preprocessor_setting_name="alts_preprocessor_sample",

Comment threadactivitysim/abm/models/non_mandatory_tour_frequency.py
@jpn--
jpn--
self-requested a review
June 17, 2025 18:52
@jpn--
jpn--
commented
Jun 17, 2025

Copy link


Copy Markdown

Member
A few thoughts before I dig into this more:
  1. Estimation mode tests are failing.
  2. Other than estimation mode, existing tests do show we have not broken anything that was tested. But this PR appears to add significant functionality, and lacks any testing of that new functionality.
@dhensle
dhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

ContributorAuthor
Hey @jpn--, thanks for your first look. As mentioned during the call today, I have a few more things to clean up on the CI tests (which is why the PR is listed as a "draft"). I will ping you in the next few days when my commits are complete and this PR is ready for review.
@dhensle
dhensle
marked this pull request as ready for review
June 18, 2025 23:35
@dhensle
dhensle
commented
Jun 18, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- Ok, this is ready for your review. I fixed the estimation mode issue and added unit tests for both the preprocessor and annotation functions which include some common expressions including reindex, skims lookups (in multiple ways), and some groupby calculations across tables. If you have other CI test ideas, I am happy to hear them.
jpn--
jpn--
requested changes
Jun 27, 2025
View reviewed changes
@jpn--jpn--
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! I found very little substantive to address, just a few things to clean up. Thanks for all the hard work on this.
Comment threadactivitysim/abm/models/vehicle_allocation.py
Outdated

 """Setting for the preprocessor."""

 OCCUPANCY_LEVELS: list = [1] # TODO Check this
 OCCUPANCY_LEVELS: list = [1, 2, 3.5]
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearly, the previous default value here of [1] was not a great solution. But it seems the occupancy levels set here needs to jive with calls to the generated variable names set in spec files for other components. "df.vehicle_occup_1" is going to show up as a variable name elsewhere in other components. I think it would be better to make that clear in the docstring, and have no default at all (i.e. failure to define this in the config for this model is an error).
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this appears to be basically unrelated to the fundamental nature of this PR, maybe just leave it out here, and we can address it in a separate specific PR for this.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, we can address this separately. Reverted.
Comment threadactivitysim/abm/models/mandatory_tour_frequency.py
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MandatoryTourFrequencySettings still has a redundant annotate_persons defined, which it also inherits from LogitComponentSettings
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, removed.
Comment threadactivitysim/abm/models/cdap.py

 preprocessor: PreprocessorSettings | None = None
 """Preprocess choosers tables before running the model."""
 annotate_persons: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automatic documentation tools will work better if annotate_persons has its own docstring, even though it's clear to a human reader the one docstring below applies to both.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/configuration/logit.py
 utility expressions.
 """

 annotate_households: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For autodoc, have each annotate_* field have its own docstring.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/util.py
Outdated

 unique_variables_in_spec.update(re.findall(pattern, custom_chooser_lines))

 logger.info("Dropping unused variables in chooser table")
 logger.debug("Dropping unused variables in chooser table")
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is clearly unrelated to this PR.
I do think we should address our logging, by default we are logging far too much
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I put this in my code cause it was annoying to look at while debugging. Happy to address this in a separate logging task. reverted.
@dhensle
dhensle
commented
Jul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- thanks for your comments. I have addressed them, but please let me know if there's anything else you find!
@jpn--
jpn--
merged commit ba4a12e
into
ActivitySim:mainJul 17, 2025
@dhensledhensle
mentioned this pull request
Aug 1, 2025
 Merged

@dhensle
dhensle
deleted the

preprocessing

branch
August 1, 2025 17:38
@dhensledhensle
mentioned this pull request
Aug 13, 2025

Closed

@dhensledhensle
mentioned this pull request
Oct 28, 2025

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@jpn--jpn--jpn-- approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@dhensle@jpn--@aletzdy

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Preprocessing & Annotation functionality - #957
Merged
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Jul 17, 2025
Merged
Preprocessing & Annotation functionality#957
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Conversation
@dhensle
@dhensledhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

Contributor
This pull request standardizes preprocessing and annotation capabilities across all activitysim abm models.
Refactoring of Preprocessing and Table Annotation
  • Standardized Preprocessing:
    • Replaced custom preprocessing logic using expressions.assign_columns with expressions.annotate_preprocessors in models such as auto_ownership, free_parking, and joint_tour_composition. This change ensures consistency across models and reduces duplication. [1][2][3]
  • Unified Table Annotation:
    • Replaced individual table annotation logic with expressions.annotate_tables in models like atwork_subtour_destination, cdap, and joint_tour_frequency. This provides a consistent and reusable approach for post-model table updates. [1][2][3]
Simplification of Model Settings
  • Removal of Unused Fields:
    • Removed unnecessary fields like preprocessor and annotate_* from model settings classes such as AtworkSubtourFrequencySettings, AutoOwnershipSettings, and FreeParkingSettings. This cleanup eliminates unused or redundant configuration options. [1][2][3]
  • Reorganization in cdap:
    • Consolidated preprocessor and annotation settings into a single unified structure for better clarity and alignment with other models.
Summary table of model changes:
Model nameChoosers PreprocessorAlts PreprocessorAnnotatorNotes
accessibilityexistedN/AN/Amodel is essentially a preprocessor
atwork_subtour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
atwork_subtour_frequencyupdatedN/Aadded
atwork_subtour_mode_choiceupdatedN/Aadded
atwork_subtour_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
auto_ownershipupdatedN/Aadded
cdapaddedN/Aupdated
disaggregate_accessibilityexistedN/Aexistednow can use alts preprocessors in logsum calculations through tour and location choice models
free_parkingupdatedN/Aupdated
initializeN/AN/AupdatedModel is annotator
initialize_losN/AN/AN/A
initialize_toursN/AN/AupdatedModel is annotator
input_checkerN/AN/AN/A
joint_tour_compositionupdatedN/Aupdated
joint_tour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
joint_tour_frequencyupdatedN/Aadded
joint_tour_frequency_compositionupdatedupdatedupdated
joint_tour_participationupdatedN/Aadded
joint_tour_schedulingupdatedaddedupdatedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
location_choiceaddedaddedupdatedadded preprocessing in both sample and simulate step
mandatory_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
mandatory_tour_frequencyupdatedN/Aadded
non_mandatory_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
non_mandatory_schedulingaddedaddedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
non_mandatory_tour_frequencyupdatedaddedadded
parking_location_choiceupdatedaddedadded
school_escortingexistedaddedaddeduniquely has 3 preprocessors
stop_frequencyupdatedN/Aadded
summarizeN/AN/AexistedModel is annotator
telecommute_frequencyupdatedN/Aadded
tour_mode_choiceupdatedN/Aadded
tour_od_choiceupdatedaddedaddedadded preprocessing in both sample and simulate step
tour_scheduling_probabilisticN/AN/Aadded
transit_pass_ownershipupdatedN/Aadded
transit_pass_subsidyupdatedN/Aadded
trip_departure_choiceexistedaddedadded
trip_destinationupdatedupdatedaddedadded preprocessing in both sample and simulate step
trip_matricesexistedN/AN/A
trip_mode_choiceexistedN/Aupdated
trip_purposeupdatedN/Aadded
trip_purpose_and_destinationN/AN/Aadded
trip_schedulingupdatedN/Aaddedis probabilistic model
trip_scheduling_choiceupdatedaddedadded
vehicle_allocationupdatedN/Aadded
vehicle_type_choiceupdatedupdatedupdated
work_from_homeupdatedN/Aupdated
@jpn--
jpn--
requested a review
from CopilotJune 17, 2025 18:45
CopilotAI
reviewed
Jun 17, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This pull request standardizes the preprocessing and table annotation functionality across multiple ABM models by replacing custom assign‐columns logic with unified calls to expressions.annotate_preprocessors and expressions.annotate_tables. Key changes include:
  • Replacing custom preprocessing calls with standardized annotation functions.
  • Removing unused/deprecated settings and consolidating configuration fields.
  • Updating multiple model files (e.g., auto_ownership, free_parking, joint_tour_frequency, etc.) to support the new standardized approach.
Reviewed Changes
Copilot reviewed 47 out of 47 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
activitysim/abm/models/parking_location_choice.pyAdded deprecation warning handling and introduced an alternative preprocessor field.
activitysim/abm/models/non_mandatory_tour_frequency.pyReplaced assign_columns with annotate_preprocessors and updated locals dictionary usage.
activitysim/abm/models/non_mandatory_scheduling.pyUpdated preprocessing and table annotation calls with new expressions functions.
activitysim/abm/models/non_mandatory_destination.pyRemoved legacy annotation function calls and standardized table annotation.
activitysim/abm/models/mandatory_tour_frequency.pyConsolidated preprocessing logic into standardized annotate_preprocessors and annotate_tables calls.
activitysim/abm/models/location_choice.pyUpdated both chooser and alternative preprocessing with standardized annotation calls.
activitysim/abm/models/joint_*.pyUnified preprocessing and annotation functions across joint tour models.
activitysim/abm/models/free_parking.pyUpdated preprocessing calls to use expressions.annotate_preprocessors and annotate_tables.
activitysim/abm/models/cdap.pyAdded preprocessing and post-model annotation functions; restructured settings fields.
activitysim/abm/models/auto_ownership.pyRemoved unused preprocessor settings by moving to a pass-through structure.
activitysim/abm/models/atwork_* (scheduling, mode_choice, frequency, destination)Consolidated preprocessing and table annotation logic with the new standardized approach.
Comments suppressed due to low confidence (1)
activitysim/abm/models/location_choice.py:184
  • [nitpick] The alternative preprocessor setting name 'alts_preprocessor_sample' differs from similar settings in other models. Consider adopting a consistent naming convention (e.g., 'alts_preprocessor') across the codebase.
 preprocessor_setting_name="alts_preprocessor_sample",

Comment threadactivitysim/abm/models/non_mandatory_tour_frequency.py
@jpn--
jpn--
self-requested a review
June 17, 2025 18:52
@jpn--
jpn--
commented
Jun 17, 2025

Copy link


Copy Markdown

Member
A few thoughts before I dig into this more:
  1. Estimation mode tests are failing.
  2. Other than estimation mode, existing tests do show we have not broken anything that was tested. But this PR appears to add significant functionality, and lacks any testing of that new functionality.
@dhensle
dhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

ContributorAuthor
Hey @jpn--, thanks for your first look. As mentioned during the call today, I have a few more things to clean up on the CI tests (which is why the PR is listed as a "draft"). I will ping you in the next few days when my commits are complete and this PR is ready for review.
@dhensle
dhensle
marked this pull request as ready for review
June 18, 2025 23:35
@dhensle
dhensle
commented
Jun 18, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- Ok, this is ready for your review. I fixed the estimation mode issue and added unit tests for both the preprocessor and annotation functions which include some common expressions including reindex, skims lookups (in multiple ways), and some groupby calculations across tables. If you have other CI test ideas, I am happy to hear them.
jpn--
jpn--
requested changes
Jun 27, 2025
View reviewed changes
@jpn--jpn--
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! I found very little substantive to address, just a few things to clean up. Thanks for all the hard work on this.
Comment threadactivitysim/abm/models/vehicle_allocation.py
Outdated

 """Setting for the preprocessor."""

 OCCUPANCY_LEVELS: list = [1] # TODO Check this
 OCCUPANCY_LEVELS: list = [1, 2, 3.5]
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearly, the previous default value here of [1] was not a great solution. But it seems the occupancy levels set here needs to jive with calls to the generated variable names set in spec files for other components. "df.vehicle_occup_1" is going to show up as a variable name elsewhere in other components. I think it would be better to make that clear in the docstring, and have no default at all (i.e. failure to define this in the config for this model is an error).
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this appears to be basically unrelated to the fundamental nature of this PR, maybe just leave it out here, and we can address it in a separate specific PR for this.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, we can address this separately. Reverted.
Comment threadactivitysim/abm/models/mandatory_tour_frequency.py
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MandatoryTourFrequencySettings still has a redundant annotate_persons defined, which it also inherits from LogitComponentSettings
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, removed.
Comment threadactivitysim/abm/models/cdap.py

 preprocessor: PreprocessorSettings | None = None
 """Preprocess choosers tables before running the model."""
 annotate_persons: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automatic documentation tools will work better if annotate_persons has its own docstring, even though it's clear to a human reader the one docstring below applies to both.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/configuration/logit.py
 utility expressions.
 """

 annotate_households: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For autodoc, have each annotate_* field have its own docstring.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/util.py
Outdated

 unique_variables_in_spec.update(re.findall(pattern, custom_chooser_lines))

 logger.info("Dropping unused variables in chooser table")
 logger.debug("Dropping unused variables in chooser table")
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is clearly unrelated to this PR.
I do think we should address our logging, by default we are logging far too much
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I put this in my code cause it was annoying to look at while debugging. Happy to address this in a separate logging task. reverted.
@dhensle
dhensle
commented
Jul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- thanks for your comments. I have addressed them, but please let me know if there's anything else you find!
@jpn--
jpn--
merged commit ba4a12e
into
ActivitySim:mainJul 17, 2025
@dhensledhensle
mentioned this pull request
Aug 1, 2025
 Merged

@dhensle
dhensle
deleted the

preprocessing

branch
August 1, 2025 17:38
@dhensledhensle
mentioned this pull request
Aug 13, 2025

Closed

@dhensledhensle
mentioned this pull request
Oct 28, 2025

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@jpn--jpn--jpn-- approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@dhensle@jpn--@aletzdy

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Preprocessing & Annotation functionality - #957
Merged
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Jul 17, 2025
Merged
Preprocessing & Annotation functionality#957
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Conversation
@dhensle
@dhensledhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

Contributor
This pull request standardizes preprocessing and annotation capabilities across all activitysim abm models.
Refactoring of Preprocessing and Table Annotation
  • Standardized Preprocessing:
    • Replaced custom preprocessing logic using expressions.assign_columns with expressions.annotate_preprocessors in models such as auto_ownership, free_parking, and joint_tour_composition. This change ensures consistency across models and reduces duplication. [1][2][3]
  • Unified Table Annotation:
    • Replaced individual table annotation logic with expressions.annotate_tables in models like atwork_subtour_destination, cdap, and joint_tour_frequency. This provides a consistent and reusable approach for post-model table updates. [1][2][3]
Simplification of Model Settings
  • Removal of Unused Fields:
    • Removed unnecessary fields like preprocessor and annotate_* from model settings classes such as AtworkSubtourFrequencySettings, AutoOwnershipSettings, and FreeParkingSettings. This cleanup eliminates unused or redundant configuration options. [1][2][3]
  • Reorganization in cdap:
    • Consolidated preprocessor and annotation settings into a single unified structure for better clarity and alignment with other models.
Summary table of model changes:
Model nameChoosers PreprocessorAlts PreprocessorAnnotatorNotes
accessibilityexistedN/AN/Amodel is essentially a preprocessor
atwork_subtour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
atwork_subtour_frequencyupdatedN/Aadded
atwork_subtour_mode_choiceupdatedN/Aadded
atwork_subtour_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
auto_ownershipupdatedN/Aadded
cdapaddedN/Aupdated
disaggregate_accessibilityexistedN/Aexistednow can use alts preprocessors in logsum calculations through tour and location choice models
free_parkingupdatedN/Aupdated
initializeN/AN/AupdatedModel is annotator
initialize_losN/AN/AN/A
initialize_toursN/AN/AupdatedModel is annotator
input_checkerN/AN/AN/A
joint_tour_compositionupdatedN/Aupdated
joint_tour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
joint_tour_frequencyupdatedN/Aadded
joint_tour_frequency_compositionupdatedupdatedupdated
joint_tour_participationupdatedN/Aadded
joint_tour_schedulingupdatedaddedupdatedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
location_choiceaddedaddedupdatedadded preprocessing in both sample and simulate step
mandatory_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
mandatory_tour_frequencyupdatedN/Aadded
non_mandatory_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
non_mandatory_schedulingaddedaddedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
non_mandatory_tour_frequencyupdatedaddedadded
parking_location_choiceupdatedaddedadded
school_escortingexistedaddedaddeduniquely has 3 preprocessors
stop_frequencyupdatedN/Aadded
summarizeN/AN/AexistedModel is annotator
telecommute_frequencyupdatedN/Aadded
tour_mode_choiceupdatedN/Aadded
tour_od_choiceupdatedaddedaddedadded preprocessing in both sample and simulate step
tour_scheduling_probabilisticN/AN/Aadded
transit_pass_ownershipupdatedN/Aadded
transit_pass_subsidyupdatedN/Aadded
trip_departure_choiceexistedaddedadded
trip_destinationupdatedupdatedaddedadded preprocessing in both sample and simulate step
trip_matricesexistedN/AN/A
trip_mode_choiceexistedN/Aupdated
trip_purposeupdatedN/Aadded
trip_purpose_and_destinationN/AN/Aadded
trip_schedulingupdatedN/Aaddedis probabilistic model
trip_scheduling_choiceupdatedaddedadded
vehicle_allocationupdatedN/Aadded
vehicle_type_choiceupdatedupdatedupdated
work_from_homeupdatedN/Aupdated
@jpn--
jpn--
requested a review
from CopilotJune 17, 2025 18:45
CopilotAI
reviewed
Jun 17, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This pull request standardizes the preprocessing and table annotation functionality across multiple ABM models by replacing custom assign‐columns logic with unified calls to expressions.annotate_preprocessors and expressions.annotate_tables. Key changes include:
  • Replacing custom preprocessing calls with standardized annotation functions.
  • Removing unused/deprecated settings and consolidating configuration fields.
  • Updating multiple model files (e.g., auto_ownership, free_parking, joint_tour_frequency, etc.) to support the new standardized approach.
Reviewed Changes
Copilot reviewed 47 out of 47 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
activitysim/abm/models/parking_location_choice.pyAdded deprecation warning handling and introduced an alternative preprocessor field.
activitysim/abm/models/non_mandatory_tour_frequency.pyReplaced assign_columns with annotate_preprocessors and updated locals dictionary usage.
activitysim/abm/models/non_mandatory_scheduling.pyUpdated preprocessing and table annotation calls with new expressions functions.
activitysim/abm/models/non_mandatory_destination.pyRemoved legacy annotation function calls and standardized table annotation.
activitysim/abm/models/mandatory_tour_frequency.pyConsolidated preprocessing logic into standardized annotate_preprocessors and annotate_tables calls.
activitysim/abm/models/location_choice.pyUpdated both chooser and alternative preprocessing with standardized annotation calls.
activitysim/abm/models/joint_*.pyUnified preprocessing and annotation functions across joint tour models.
activitysim/abm/models/free_parking.pyUpdated preprocessing calls to use expressions.annotate_preprocessors and annotate_tables.
activitysim/abm/models/cdap.pyAdded preprocessing and post-model annotation functions; restructured settings fields.
activitysim/abm/models/auto_ownership.pyRemoved unused preprocessor settings by moving to a pass-through structure.
activitysim/abm/models/atwork_* (scheduling, mode_choice, frequency, destination)Consolidated preprocessing and table annotation logic with the new standardized approach.
Comments suppressed due to low confidence (1)
activitysim/abm/models/location_choice.py:184
  • [nitpick] The alternative preprocessor setting name 'alts_preprocessor_sample' differs from similar settings in other models. Consider adopting a consistent naming convention (e.g., 'alts_preprocessor') across the codebase.
 preprocessor_setting_name="alts_preprocessor_sample",

Comment threadactivitysim/abm/models/non_mandatory_tour_frequency.py
@jpn--
jpn--
self-requested a review
June 17, 2025 18:52
@jpn--
jpn--
commented
Jun 17, 2025

Copy link


Copy Markdown

Member
A few thoughts before I dig into this more:
  1. Estimation mode tests are failing.
  2. Other than estimation mode, existing tests do show we have not broken anything that was tested. But this PR appears to add significant functionality, and lacks any testing of that new functionality.
@dhensle
dhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

ContributorAuthor
Hey @jpn--, thanks for your first look. As mentioned during the call today, I have a few more things to clean up on the CI tests (which is why the PR is listed as a "draft"). I will ping you in the next few days when my commits are complete and this PR is ready for review.
@dhensle
dhensle
marked this pull request as ready for review
June 18, 2025 23:35
@dhensle
dhensle
commented
Jun 18, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- Ok, this is ready for your review. I fixed the estimation mode issue and added unit tests for both the preprocessor and annotation functions which include some common expressions including reindex, skims lookups (in multiple ways), and some groupby calculations across tables. If you have other CI test ideas, I am happy to hear them.
jpn--
jpn--
requested changes
Jun 27, 2025
View reviewed changes
@jpn--jpn--
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! I found very little substantive to address, just a few things to clean up. Thanks for all the hard work on this.
Comment threadactivitysim/abm/models/vehicle_allocation.py
Outdated

 """Setting for the preprocessor."""

 OCCUPANCY_LEVELS: list = [1] # TODO Check this
 OCCUPANCY_LEVELS: list = [1, 2, 3.5]
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearly, the previous default value here of [1] was not a great solution. But it seems the occupancy levels set here needs to jive with calls to the generated variable names set in spec files for other components. "df.vehicle_occup_1" is going to show up as a variable name elsewhere in other components. I think it would be better to make that clear in the docstring, and have no default at all (i.e. failure to define this in the config for this model is an error).
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this appears to be basically unrelated to the fundamental nature of this PR, maybe just leave it out here, and we can address it in a separate specific PR for this.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, we can address this separately. Reverted.
Comment threadactivitysim/abm/models/mandatory_tour_frequency.py
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MandatoryTourFrequencySettings still has a redundant annotate_persons defined, which it also inherits from LogitComponentSettings
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, removed.
Comment threadactivitysim/abm/models/cdap.py

 preprocessor: PreprocessorSettings | None = None
 """Preprocess choosers tables before running the model."""
 annotate_persons: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automatic documentation tools will work better if annotate_persons has its own docstring, even though it's clear to a human reader the one docstring below applies to both.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/configuration/logit.py
 utility expressions.
 """

 annotate_households: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For autodoc, have each annotate_* field have its own docstring.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/util.py
Outdated

 unique_variables_in_spec.update(re.findall(pattern, custom_chooser_lines))

 logger.info("Dropping unused variables in chooser table")
 logger.debug("Dropping unused variables in chooser table")
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is clearly unrelated to this PR.
I do think we should address our logging, by default we are logging far too much
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I put this in my code cause it was annoying to look at while debugging. Happy to address this in a separate logging task. reverted.
@dhensle
dhensle
commented
Jul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- thanks for your comments. I have addressed them, but please let me know if there's anything else you find!
@jpn--
jpn--
merged commit ba4a12e
into
ActivitySim:mainJul 17, 2025
@dhensledhensle
mentioned this pull request
Aug 1, 2025
 Merged

@dhensle
dhensle
deleted the

preprocessing

branch
August 1, 2025 17:38
@dhensledhensle
mentioned this pull request
Aug 13, 2025

Closed

@dhensledhensle
mentioned this pull request
Oct 28, 2025

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@jpn--jpn--jpn-- approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@dhensle@jpn--@aletzdy

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Preprocessing & Annotation functionality - #957
Merged
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Jul 17, 2025
Merged
Preprocessing & Annotation functionality#957
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Conversation
@dhensle
@dhensledhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

Contributor
This pull request standardizes preprocessing and annotation capabilities across all activitysim abm models.
Refactoring of Preprocessing and Table Annotation
  • Standardized Preprocessing:
    • Replaced custom preprocessing logic using expressions.assign_columns with expressions.annotate_preprocessors in models such as auto_ownership, free_parking, and joint_tour_composition. This change ensures consistency across models and reduces duplication. [1][2][3]
  • Unified Table Annotation:
    • Replaced individual table annotation logic with expressions.annotate_tables in models like atwork_subtour_destination, cdap, and joint_tour_frequency. This provides a consistent and reusable approach for post-model table updates. [1][2][3]
Simplification of Model Settings
  • Removal of Unused Fields:
    • Removed unnecessary fields like preprocessor and annotate_* from model settings classes such as AtworkSubtourFrequencySettings, AutoOwnershipSettings, and FreeParkingSettings. This cleanup eliminates unused or redundant configuration options. [1][2][3]
  • Reorganization in cdap:
    • Consolidated preprocessor and annotation settings into a single unified structure for better clarity and alignment with other models.
Summary table of model changes:
Model nameChoosers PreprocessorAlts PreprocessorAnnotatorNotes
accessibilityexistedN/AN/Amodel is essentially a preprocessor
atwork_subtour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
atwork_subtour_frequencyupdatedN/Aadded
atwork_subtour_mode_choiceupdatedN/Aadded
atwork_subtour_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
auto_ownershipupdatedN/Aadded
cdapaddedN/Aupdated
disaggregate_accessibilityexistedN/Aexistednow can use alts preprocessors in logsum calculations through tour and location choice models
free_parkingupdatedN/Aupdated
initializeN/AN/AupdatedModel is annotator
initialize_losN/AN/AN/A
initialize_toursN/AN/AupdatedModel is annotator
input_checkerN/AN/AN/A
joint_tour_compositionupdatedN/Aupdated
joint_tour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
joint_tour_frequencyupdatedN/Aadded
joint_tour_frequency_compositionupdatedupdatedupdated
joint_tour_participationupdatedN/Aadded
joint_tour_schedulingupdatedaddedupdatedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
location_choiceaddedaddedupdatedadded preprocessing in both sample and simulate step
mandatory_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
mandatory_tour_frequencyupdatedN/Aadded
non_mandatory_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
non_mandatory_schedulingaddedaddedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
non_mandatory_tour_frequencyupdatedaddedadded
parking_location_choiceupdatedaddedadded
school_escortingexistedaddedaddeduniquely has 3 preprocessors
stop_frequencyupdatedN/Aadded
summarizeN/AN/AexistedModel is annotator
telecommute_frequencyupdatedN/Aadded
tour_mode_choiceupdatedN/Aadded
tour_od_choiceupdatedaddedaddedadded preprocessing in both sample and simulate step
tour_scheduling_probabilisticN/AN/Aadded
transit_pass_ownershipupdatedN/Aadded
transit_pass_subsidyupdatedN/Aadded
trip_departure_choiceexistedaddedadded
trip_destinationupdatedupdatedaddedadded preprocessing in both sample and simulate step
trip_matricesexistedN/AN/A
trip_mode_choiceexistedN/Aupdated
trip_purposeupdatedN/Aadded
trip_purpose_and_destinationN/AN/Aadded
trip_schedulingupdatedN/Aaddedis probabilistic model
trip_scheduling_choiceupdatedaddedadded
vehicle_allocationupdatedN/Aadded
vehicle_type_choiceupdatedupdatedupdated
work_from_homeupdatedN/Aupdated
@jpn--
jpn--
requested a review
from CopilotJune 17, 2025 18:45
CopilotAI
reviewed
Jun 17, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This pull request standardizes the preprocessing and table annotation functionality across multiple ABM models by replacing custom assign‐columns logic with unified calls to expressions.annotate_preprocessors and expressions.annotate_tables. Key changes include:
  • Replacing custom preprocessing calls with standardized annotation functions.
  • Removing unused/deprecated settings and consolidating configuration fields.
  • Updating multiple model files (e.g., auto_ownership, free_parking, joint_tour_frequency, etc.) to support the new standardized approach.
Reviewed Changes
Copilot reviewed 47 out of 47 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
activitysim/abm/models/parking_location_choice.pyAdded deprecation warning handling and introduced an alternative preprocessor field.
activitysim/abm/models/non_mandatory_tour_frequency.pyReplaced assign_columns with annotate_preprocessors and updated locals dictionary usage.
activitysim/abm/models/non_mandatory_scheduling.pyUpdated preprocessing and table annotation calls with new expressions functions.
activitysim/abm/models/non_mandatory_destination.pyRemoved legacy annotation function calls and standardized table annotation.
activitysim/abm/models/mandatory_tour_frequency.pyConsolidated preprocessing logic into standardized annotate_preprocessors and annotate_tables calls.
activitysim/abm/models/location_choice.pyUpdated both chooser and alternative preprocessing with standardized annotation calls.
activitysim/abm/models/joint_*.pyUnified preprocessing and annotation functions across joint tour models.
activitysim/abm/models/free_parking.pyUpdated preprocessing calls to use expressions.annotate_preprocessors and annotate_tables.
activitysim/abm/models/cdap.pyAdded preprocessing and post-model annotation functions; restructured settings fields.
activitysim/abm/models/auto_ownership.pyRemoved unused preprocessor settings by moving to a pass-through structure.
activitysim/abm/models/atwork_* (scheduling, mode_choice, frequency, destination)Consolidated preprocessing and table annotation logic with the new standardized approach.
Comments suppressed due to low confidence (1)
activitysim/abm/models/location_choice.py:184
  • [nitpick] The alternative preprocessor setting name 'alts_preprocessor_sample' differs from similar settings in other models. Consider adopting a consistent naming convention (e.g., 'alts_preprocessor') across the codebase.
 preprocessor_setting_name="alts_preprocessor_sample",

Comment threadactivitysim/abm/models/non_mandatory_tour_frequency.py
@jpn--
jpn--
self-requested a review
June 17, 2025 18:52
@jpn--
jpn--
commented
Jun 17, 2025

Copy link


Copy Markdown

Member
A few thoughts before I dig into this more:
  1. Estimation mode tests are failing.
  2. Other than estimation mode, existing tests do show we have not broken anything that was tested. But this PR appears to add significant functionality, and lacks any testing of that new functionality.
@dhensle
dhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

ContributorAuthor
Hey @jpn--, thanks for your first look. As mentioned during the call today, I have a few more things to clean up on the CI tests (which is why the PR is listed as a "draft"). I will ping you in the next few days when my commits are complete and this PR is ready for review.
@dhensle
dhensle
marked this pull request as ready for review
June 18, 2025 23:35
@dhensle
dhensle
commented
Jun 18, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- Ok, this is ready for your review. I fixed the estimation mode issue and added unit tests for both the preprocessor and annotation functions which include some common expressions including reindex, skims lookups (in multiple ways), and some groupby calculations across tables. If you have other CI test ideas, I am happy to hear them.
jpn--
jpn--
requested changes
Jun 27, 2025
View reviewed changes
@jpn--jpn--
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! I found very little substantive to address, just a few things to clean up. Thanks for all the hard work on this.
Comment threadactivitysim/abm/models/vehicle_allocation.py
Outdated

 """Setting for the preprocessor."""

 OCCUPANCY_LEVELS: list = [1] # TODO Check this
 OCCUPANCY_LEVELS: list = [1, 2, 3.5]
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearly, the previous default value here of [1] was not a great solution. But it seems the occupancy levels set here needs to jive with calls to the generated variable names set in spec files for other components. "df.vehicle_occup_1" is going to show up as a variable name elsewhere in other components. I think it would be better to make that clear in the docstring, and have no default at all (i.e. failure to define this in the config for this model is an error).
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this appears to be basically unrelated to the fundamental nature of this PR, maybe just leave it out here, and we can address it in a separate specific PR for this.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, we can address this separately. Reverted.
Comment threadactivitysim/abm/models/mandatory_tour_frequency.py
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MandatoryTourFrequencySettings still has a redundant annotate_persons defined, which it also inherits from LogitComponentSettings
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, removed.
Comment threadactivitysim/abm/models/cdap.py

 preprocessor: PreprocessorSettings | None = None
 """Preprocess choosers tables before running the model."""
 annotate_persons: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automatic documentation tools will work better if annotate_persons has its own docstring, even though it's clear to a human reader the one docstring below applies to both.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/configuration/logit.py
 utility expressions.
 """

 annotate_households: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For autodoc, have each annotate_* field have its own docstring.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/util.py
Outdated

 unique_variables_in_spec.update(re.findall(pattern, custom_chooser_lines))

 logger.info("Dropping unused variables in chooser table")
 logger.debug("Dropping unused variables in chooser table")
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is clearly unrelated to this PR.
I do think we should address our logging, by default we are logging far too much
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I put this in my code cause it was annoying to look at while debugging. Happy to address this in a separate logging task. reverted.
@dhensle
dhensle
commented
Jul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- thanks for your comments. I have addressed them, but please let me know if there's anything else you find!
@jpn--
jpn--
merged commit ba4a12e
into
ActivitySim:mainJul 17, 2025
@dhensledhensle
mentioned this pull request
Aug 1, 2025
 Merged

@dhensle
dhensle
deleted the

preprocessing

branch
August 1, 2025 17:38
@dhensledhensle
mentioned this pull request
Aug 13, 2025

Closed

@dhensledhensle
mentioned this pull request
Oct 28, 2025

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@jpn--jpn--jpn-- approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@dhensle@jpn--@aletzdy

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Preprocessing & Annotation functionality - #957
Merged
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Jul 17, 2025
Merged
Preprocessing & Annotation functionality#957
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Conversation
@dhensle
@dhensledhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

Contributor
This pull request standardizes preprocessing and annotation capabilities across all activitysim abm models.
Refactoring of Preprocessing and Table Annotation
  • Standardized Preprocessing:
    • Replaced custom preprocessing logic using expressions.assign_columns with expressions.annotate_preprocessors in models such as auto_ownership, free_parking, and joint_tour_composition. This change ensures consistency across models and reduces duplication. [1][2][3]
  • Unified Table Annotation:
    • Replaced individual table annotation logic with expressions.annotate_tables in models like atwork_subtour_destination, cdap, and joint_tour_frequency. This provides a consistent and reusable approach for post-model table updates. [1][2][3]
Simplification of Model Settings
  • Removal of Unused Fields:
    • Removed unnecessary fields like preprocessor and annotate_* from model settings classes such as AtworkSubtourFrequencySettings, AutoOwnershipSettings, and FreeParkingSettings. This cleanup eliminates unused or redundant configuration options. [1][2][3]
  • Reorganization in cdap:
    • Consolidated preprocessor and annotation settings into a single unified structure for better clarity and alignment with other models.
Summary table of model changes:
Model nameChoosers PreprocessorAlts PreprocessorAnnotatorNotes
accessibilityexistedN/AN/Amodel is essentially a preprocessor
atwork_subtour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
atwork_subtour_frequencyupdatedN/Aadded
atwork_subtour_mode_choiceupdatedN/Aadded
atwork_subtour_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
auto_ownershipupdatedN/Aadded
cdapaddedN/Aupdated
disaggregate_accessibilityexistedN/Aexistednow can use alts preprocessors in logsum calculations through tour and location choice models
free_parkingupdatedN/Aupdated
initializeN/AN/AupdatedModel is annotator
initialize_losN/AN/AN/A
initialize_toursN/AN/AupdatedModel is annotator
input_checkerN/AN/AN/A
joint_tour_compositionupdatedN/Aupdated
joint_tour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
joint_tour_frequencyupdatedN/Aadded
joint_tour_frequency_compositionupdatedupdatedupdated
joint_tour_participationupdatedN/Aadded
joint_tour_schedulingupdatedaddedupdatedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
location_choiceaddedaddedupdatedadded preprocessing in both sample and simulate step
mandatory_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
mandatory_tour_frequencyupdatedN/Aadded
non_mandatory_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
non_mandatory_schedulingaddedaddedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
non_mandatory_tour_frequencyupdatedaddedadded
parking_location_choiceupdatedaddedadded
school_escortingexistedaddedaddeduniquely has 3 preprocessors
stop_frequencyupdatedN/Aadded
summarizeN/AN/AexistedModel is annotator
telecommute_frequencyupdatedN/Aadded
tour_mode_choiceupdatedN/Aadded
tour_od_choiceupdatedaddedaddedadded preprocessing in both sample and simulate step
tour_scheduling_probabilisticN/AN/Aadded
transit_pass_ownershipupdatedN/Aadded
transit_pass_subsidyupdatedN/Aadded
trip_departure_choiceexistedaddedadded
trip_destinationupdatedupdatedaddedadded preprocessing in both sample and simulate step
trip_matricesexistedN/AN/A
trip_mode_choiceexistedN/Aupdated
trip_purposeupdatedN/Aadded
trip_purpose_and_destinationN/AN/Aadded
trip_schedulingupdatedN/Aaddedis probabilistic model
trip_scheduling_choiceupdatedaddedadded
vehicle_allocationupdatedN/Aadded
vehicle_type_choiceupdatedupdatedupdated
work_from_homeupdatedN/Aupdated
@jpn--
jpn--
requested a review
from CopilotJune 17, 2025 18:45
CopilotAI
reviewed
Jun 17, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This pull request standardizes the preprocessing and table annotation functionality across multiple ABM models by replacing custom assign‐columns logic with unified calls to expressions.annotate_preprocessors and expressions.annotate_tables. Key changes include:
  • Replacing custom preprocessing calls with standardized annotation functions.
  • Removing unused/deprecated settings and consolidating configuration fields.
  • Updating multiple model files (e.g., auto_ownership, free_parking, joint_tour_frequency, etc.) to support the new standardized approach.
Reviewed Changes
Copilot reviewed 47 out of 47 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
activitysim/abm/models/parking_location_choice.pyAdded deprecation warning handling and introduced an alternative preprocessor field.
activitysim/abm/models/non_mandatory_tour_frequency.pyReplaced assign_columns with annotate_preprocessors and updated locals dictionary usage.
activitysim/abm/models/non_mandatory_scheduling.pyUpdated preprocessing and table annotation calls with new expressions functions.
activitysim/abm/models/non_mandatory_destination.pyRemoved legacy annotation function calls and standardized table annotation.
activitysim/abm/models/mandatory_tour_frequency.pyConsolidated preprocessing logic into standardized annotate_preprocessors and annotate_tables calls.
activitysim/abm/models/location_choice.pyUpdated both chooser and alternative preprocessing with standardized annotation calls.
activitysim/abm/models/joint_*.pyUnified preprocessing and annotation functions across joint tour models.
activitysim/abm/models/free_parking.pyUpdated preprocessing calls to use expressions.annotate_preprocessors and annotate_tables.
activitysim/abm/models/cdap.pyAdded preprocessing and post-model annotation functions; restructured settings fields.
activitysim/abm/models/auto_ownership.pyRemoved unused preprocessor settings by moving to a pass-through structure.
activitysim/abm/models/atwork_* (scheduling, mode_choice, frequency, destination)Consolidated preprocessing and table annotation logic with the new standardized approach.
Comments suppressed due to low confidence (1)
activitysim/abm/models/location_choice.py:184
  • [nitpick] The alternative preprocessor setting name 'alts_preprocessor_sample' differs from similar settings in other models. Consider adopting a consistent naming convention (e.g., 'alts_preprocessor') across the codebase.
 preprocessor_setting_name="alts_preprocessor_sample",

Comment threadactivitysim/abm/models/non_mandatory_tour_frequency.py
@jpn--
jpn--
self-requested a review
June 17, 2025 18:52
@jpn--
jpn--
commented
Jun 17, 2025

Copy link


Copy Markdown

Member
A few thoughts before I dig into this more:
  1. Estimation mode tests are failing.
  2. Other than estimation mode, existing tests do show we have not broken anything that was tested. But this PR appears to add significant functionality, and lacks any testing of that new functionality.
@dhensle
dhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

ContributorAuthor
Hey @jpn--, thanks for your first look. As mentioned during the call today, I have a few more things to clean up on the CI tests (which is why the PR is listed as a "draft"). I will ping you in the next few days when my commits are complete and this PR is ready for review.
@dhensle
dhensle
marked this pull request as ready for review
June 18, 2025 23:35
@dhensle
dhensle
commented
Jun 18, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- Ok, this is ready for your review. I fixed the estimation mode issue and added unit tests for both the preprocessor and annotation functions which include some common expressions including reindex, skims lookups (in multiple ways), and some groupby calculations across tables. If you have other CI test ideas, I am happy to hear them.
jpn--
jpn--
requested changes
Jun 27, 2025
View reviewed changes
@jpn--jpn--
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! I found very little substantive to address, just a few things to clean up. Thanks for all the hard work on this.
Comment threadactivitysim/abm/models/vehicle_allocation.py
Outdated

 """Setting for the preprocessor."""

 OCCUPANCY_LEVELS: list = [1] # TODO Check this
 OCCUPANCY_LEVELS: list = [1, 2, 3.5]
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearly, the previous default value here of [1] was not a great solution. But it seems the occupancy levels set here needs to jive with calls to the generated variable names set in spec files for other components. "df.vehicle_occup_1" is going to show up as a variable name elsewhere in other components. I think it would be better to make that clear in the docstring, and have no default at all (i.e. failure to define this in the config for this model is an error).
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this appears to be basically unrelated to the fundamental nature of this PR, maybe just leave it out here, and we can address it in a separate specific PR for this.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, we can address this separately. Reverted.
Comment threadactivitysim/abm/models/mandatory_tour_frequency.py
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MandatoryTourFrequencySettings still has a redundant annotate_persons defined, which it also inherits from LogitComponentSettings
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, removed.
Comment threadactivitysim/abm/models/cdap.py

 preprocessor: PreprocessorSettings | None = None
 """Preprocess choosers tables before running the model."""
 annotate_persons: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automatic documentation tools will work better if annotate_persons has its own docstring, even though it's clear to a human reader the one docstring below applies to both.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/configuration/logit.py
 utility expressions.
 """

 annotate_households: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For autodoc, have each annotate_* field have its own docstring.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/util.py
Outdated

 unique_variables_in_spec.update(re.findall(pattern, custom_chooser_lines))

 logger.info("Dropping unused variables in chooser table")
 logger.debug("Dropping unused variables in chooser table")
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is clearly unrelated to this PR.
I do think we should address our logging, by default we are logging far too much
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I put this in my code cause it was annoying to look at while debugging. Happy to address this in a separate logging task. reverted.
@dhensle
dhensle
commented
Jul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- thanks for your comments. I have addressed them, but please let me know if there's anything else you find!
@jpn--
jpn--
merged commit ba4a12e
into
ActivitySim:mainJul 17, 2025
@dhensledhensle
mentioned this pull request
Aug 1, 2025
 Merged

@dhensle
dhensle
deleted the

preprocessing

branch
August 1, 2025 17:38
@dhensledhensle
mentioned this pull request
Aug 13, 2025

Closed

@dhensledhensle
mentioned this pull request
Oct 28, 2025

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@jpn--jpn--jpn-- approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@dhensle@jpn--@aletzdy

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Preprocessing & Annotation functionality - #957
Merged
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Jul 17, 2025
Merged
Preprocessing & Annotation functionality#957
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Conversation
@dhensle
@dhensledhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

Contributor
This pull request standardizes preprocessing and annotation capabilities across all activitysim abm models.
Refactoring of Preprocessing and Table Annotation
  • Standardized Preprocessing:
    • Replaced custom preprocessing logic using expressions.assign_columns with expressions.annotate_preprocessors in models such as auto_ownership, free_parking, and joint_tour_composition. This change ensures consistency across models and reduces duplication. [1][2][3]
  • Unified Table Annotation:
    • Replaced individual table annotation logic with expressions.annotate_tables in models like atwork_subtour_destination, cdap, and joint_tour_frequency. This provides a consistent and reusable approach for post-model table updates. [1][2][3]
Simplification of Model Settings
  • Removal of Unused Fields:
    • Removed unnecessary fields like preprocessor and annotate_* from model settings classes such as AtworkSubtourFrequencySettings, AutoOwnershipSettings, and FreeParkingSettings. This cleanup eliminates unused or redundant configuration options. [1][2][3]
  • Reorganization in cdap:
    • Consolidated preprocessor and annotation settings into a single unified structure for better clarity and alignment with other models.
Summary table of model changes:
Model nameChoosers PreprocessorAlts PreprocessorAnnotatorNotes
accessibilityexistedN/AN/Amodel is essentially a preprocessor
atwork_subtour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
atwork_subtour_frequencyupdatedN/Aadded
atwork_subtour_mode_choiceupdatedN/Aadded
atwork_subtour_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
auto_ownershipupdatedN/Aadded
cdapaddedN/Aupdated
disaggregate_accessibilityexistedN/Aexistednow can use alts preprocessors in logsum calculations through tour and location choice models
free_parkingupdatedN/Aupdated
initializeN/AN/AupdatedModel is annotator
initialize_losN/AN/AN/A
initialize_toursN/AN/AupdatedModel is annotator
input_checkerN/AN/AN/A
joint_tour_compositionupdatedN/Aupdated
joint_tour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
joint_tour_frequencyupdatedN/Aadded
joint_tour_frequency_compositionupdatedupdatedupdated
joint_tour_participationupdatedN/Aadded
joint_tour_schedulingupdatedaddedupdatedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
location_choiceaddedaddedupdatedadded preprocessing in both sample and simulate step
mandatory_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
mandatory_tour_frequencyupdatedN/Aadded
non_mandatory_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
non_mandatory_schedulingaddedaddedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
non_mandatory_tour_frequencyupdatedaddedadded
parking_location_choiceupdatedaddedadded
school_escortingexistedaddedaddeduniquely has 3 preprocessors
stop_frequencyupdatedN/Aadded
summarizeN/AN/AexistedModel is annotator
telecommute_frequencyupdatedN/Aadded
tour_mode_choiceupdatedN/Aadded
tour_od_choiceupdatedaddedaddedadded preprocessing in both sample and simulate step
tour_scheduling_probabilisticN/AN/Aadded
transit_pass_ownershipupdatedN/Aadded
transit_pass_subsidyupdatedN/Aadded
trip_departure_choiceexistedaddedadded
trip_destinationupdatedupdatedaddedadded preprocessing in both sample and simulate step
trip_matricesexistedN/AN/A
trip_mode_choiceexistedN/Aupdated
trip_purposeupdatedN/Aadded
trip_purpose_and_destinationN/AN/Aadded
trip_schedulingupdatedN/Aaddedis probabilistic model
trip_scheduling_choiceupdatedaddedadded
vehicle_allocationupdatedN/Aadded
vehicle_type_choiceupdatedupdatedupdated
work_from_homeupdatedN/Aupdated
@jpn--
jpn--
requested a review
from CopilotJune 17, 2025 18:45
CopilotAI
reviewed
Jun 17, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This pull request standardizes the preprocessing and table annotation functionality across multiple ABM models by replacing custom assign‐columns logic with unified calls to expressions.annotate_preprocessors and expressions.annotate_tables. Key changes include:
  • Replacing custom preprocessing calls with standardized annotation functions.
  • Removing unused/deprecated settings and consolidating configuration fields.
  • Updating multiple model files (e.g., auto_ownership, free_parking, joint_tour_frequency, etc.) to support the new standardized approach.
Reviewed Changes
Copilot reviewed 47 out of 47 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
activitysim/abm/models/parking_location_choice.pyAdded deprecation warning handling and introduced an alternative preprocessor field.
activitysim/abm/models/non_mandatory_tour_frequency.pyReplaced assign_columns with annotate_preprocessors and updated locals dictionary usage.
activitysim/abm/models/non_mandatory_scheduling.pyUpdated preprocessing and table annotation calls with new expressions functions.
activitysim/abm/models/non_mandatory_destination.pyRemoved legacy annotation function calls and standardized table annotation.
activitysim/abm/models/mandatory_tour_frequency.pyConsolidated preprocessing logic into standardized annotate_preprocessors and annotate_tables calls.
activitysim/abm/models/location_choice.pyUpdated both chooser and alternative preprocessing with standardized annotation calls.
activitysim/abm/models/joint_*.pyUnified preprocessing and annotation functions across joint tour models.
activitysim/abm/models/free_parking.pyUpdated preprocessing calls to use expressions.annotate_preprocessors and annotate_tables.
activitysim/abm/models/cdap.pyAdded preprocessing and post-model annotation functions; restructured settings fields.
activitysim/abm/models/auto_ownership.pyRemoved unused preprocessor settings by moving to a pass-through structure.
activitysim/abm/models/atwork_* (scheduling, mode_choice, frequency, destination)Consolidated preprocessing and table annotation logic with the new standardized approach.
Comments suppressed due to low confidence (1)
activitysim/abm/models/location_choice.py:184
  • [nitpick] The alternative preprocessor setting name 'alts_preprocessor_sample' differs from similar settings in other models. Consider adopting a consistent naming convention (e.g., 'alts_preprocessor') across the codebase.
 preprocessor_setting_name="alts_preprocessor_sample",

Comment threadactivitysim/abm/models/non_mandatory_tour_frequency.py
@jpn--
jpn--
self-requested a review
June 17, 2025 18:52
@jpn--
jpn--
commented
Jun 17, 2025

Copy link


Copy Markdown

Member
A few thoughts before I dig into this more:
  1. Estimation mode tests are failing.
  2. Other than estimation mode, existing tests do show we have not broken anything that was tested. But this PR appears to add significant functionality, and lacks any testing of that new functionality.
@dhensle
dhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

ContributorAuthor
Hey @jpn--, thanks for your first look. As mentioned during the call today, I have a few more things to clean up on the CI tests (which is why the PR is listed as a "draft"). I will ping you in the next few days when my commits are complete and this PR is ready for review.
@dhensle
dhensle
marked this pull request as ready for review
June 18, 2025 23:35
@dhensle
dhensle
commented
Jun 18, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- Ok, this is ready for your review. I fixed the estimation mode issue and added unit tests for both the preprocessor and annotation functions which include some common expressions including reindex, skims lookups (in multiple ways), and some groupby calculations across tables. If you have other CI test ideas, I am happy to hear them.
jpn--
jpn--
requested changes
Jun 27, 2025
View reviewed changes
@jpn--jpn--
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! I found very little substantive to address, just a few things to clean up. Thanks for all the hard work on this.
Comment threadactivitysim/abm/models/vehicle_allocation.py
Outdated

 """Setting for the preprocessor."""

 OCCUPANCY_LEVELS: list = [1] # TODO Check this
 OCCUPANCY_LEVELS: list = [1, 2, 3.5]
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearly, the previous default value here of [1] was not a great solution. But it seems the occupancy levels set here needs to jive with calls to the generated variable names set in spec files for other components. "df.vehicle_occup_1" is going to show up as a variable name elsewhere in other components. I think it would be better to make that clear in the docstring, and have no default at all (i.e. failure to define this in the config for this model is an error).
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this appears to be basically unrelated to the fundamental nature of this PR, maybe just leave it out here, and we can address it in a separate specific PR for this.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, we can address this separately. Reverted.
Comment threadactivitysim/abm/models/mandatory_tour_frequency.py
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MandatoryTourFrequencySettings still has a redundant annotate_persons defined, which it also inherits from LogitComponentSettings
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, removed.
Comment threadactivitysim/abm/models/cdap.py

 preprocessor: PreprocessorSettings | None = None
 """Preprocess choosers tables before running the model."""
 annotate_persons: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automatic documentation tools will work better if annotate_persons has its own docstring, even though it's clear to a human reader the one docstring below applies to both.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/configuration/logit.py
 utility expressions.
 """

 annotate_households: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For autodoc, have each annotate_* field have its own docstring.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/util.py
Outdated

 unique_variables_in_spec.update(re.findall(pattern, custom_chooser_lines))

 logger.info("Dropping unused variables in chooser table")
 logger.debug("Dropping unused variables in chooser table")
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is clearly unrelated to this PR.
I do think we should address our logging, by default we are logging far too much
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I put this in my code cause it was annoying to look at while debugging. Happy to address this in a separate logging task. reverted.
@dhensle
dhensle
commented
Jul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- thanks for your comments. I have addressed them, but please let me know if there's anything else you find!
@jpn--
jpn--
merged commit ba4a12e
into
ActivitySim:mainJul 17, 2025
@dhensledhensle
mentioned this pull request
Aug 1, 2025
 Merged

@dhensle
dhensle
deleted the

preprocessing

branch
August 1, 2025 17:38
@dhensledhensle
mentioned this pull request
Aug 13, 2025

Closed

@dhensledhensle
mentioned this pull request
Oct 28, 2025

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@jpn--jpn--jpn-- approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@dhensle@jpn--@aletzdy

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Preprocessing & Annotation functionality - #957
Merged
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Jul 17, 2025
Merged
Preprocessing & Annotation functionality#957
jpn-- merged 9 commits into
ActivitySim:mainfrom 
RSGInc:preprocessing
Conversation
@dhensle
@dhensledhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

Contributor
This pull request standardizes preprocessing and annotation capabilities across all activitysim abm models.
Refactoring of Preprocessing and Table Annotation
  • Standardized Preprocessing:
    • Replaced custom preprocessing logic using expressions.assign_columns with expressions.annotate_preprocessors in models such as auto_ownership, free_parking, and joint_tour_composition. This change ensures consistency across models and reduces duplication. [1][2][3]
  • Unified Table Annotation:
    • Replaced individual table annotation logic with expressions.annotate_tables in models like atwork_subtour_destination, cdap, and joint_tour_frequency. This provides a consistent and reusable approach for post-model table updates. [1][2][3]
Simplification of Model Settings
  • Removal of Unused Fields:
    • Removed unnecessary fields like preprocessor and annotate_* from model settings classes such as AtworkSubtourFrequencySettings, AutoOwnershipSettings, and FreeParkingSettings. This cleanup eliminates unused or redundant configuration options. [1][2][3]
  • Reorganization in cdap:
    • Consolidated preprocessor and annotation settings into a single unified structure for better clarity and alignment with other models.
Summary table of model changes:
Model nameChoosers PreprocessorAlts PreprocessorAnnotatorNotes
accessibilityexistedN/AN/Amodel is essentially a preprocessor
atwork_subtour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
atwork_subtour_frequencyupdatedN/Aadded
atwork_subtour_mode_choiceupdatedN/Aadded
atwork_subtour_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
auto_ownershipupdatedN/Aadded
cdapaddedN/Aupdated
disaggregate_accessibilityexistedN/Aexistednow can use alts preprocessors in logsum calculations through tour and location choice models
free_parkingupdatedN/Aupdated
initializeN/AN/AupdatedModel is annotator
initialize_losN/AN/AN/A
initialize_toursN/AN/AupdatedModel is annotator
input_checkerN/AN/AN/A
joint_tour_compositionupdatedN/Aupdated
joint_tour_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
joint_tour_frequencyupdatedN/Aadded
joint_tour_frequency_compositionupdatedupdatedupdated
joint_tour_participationupdatedN/Aadded
joint_tour_schedulingupdatedaddedupdatedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
location_choiceaddedaddedupdatedadded preprocessing in both sample and simulate step
mandatory_schedulingaddedexistedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
mandatory_tour_frequencyupdatedN/Aadded
non_mandatory_destinationaddedaddedaddedadded preprocessing in both sample and simulate step in util.tour_destination.py
non_mandatory_schedulingaddedaddedaddedALTS_PREPROCESSOR segmented by tour purpose, added in util.vectorize_tour_scheduling.py
non_mandatory_tour_frequencyupdatedaddedadded
parking_location_choiceupdatedaddedadded
school_escortingexistedaddedaddeduniquely has 3 preprocessors
stop_frequencyupdatedN/Aadded
summarizeN/AN/AexistedModel is annotator
telecommute_frequencyupdatedN/Aadded
tour_mode_choiceupdatedN/Aadded
tour_od_choiceupdatedaddedaddedadded preprocessing in both sample and simulate step
tour_scheduling_probabilisticN/AN/Aadded
transit_pass_ownershipupdatedN/Aadded
transit_pass_subsidyupdatedN/Aadded
trip_departure_choiceexistedaddedadded
trip_destinationupdatedupdatedaddedadded preprocessing in both sample and simulate step
trip_matricesexistedN/AN/A
trip_mode_choiceexistedN/Aupdated
trip_purposeupdatedN/Aadded
trip_purpose_and_destinationN/AN/Aadded
trip_schedulingupdatedN/Aaddedis probabilistic model
trip_scheduling_choiceupdatedaddedadded
vehicle_allocationupdatedN/Aadded
vehicle_type_choiceupdatedupdatedupdated
work_from_homeupdatedN/Aupdated
@jpn--
jpn--
requested a review
from CopilotJune 17, 2025 18:45
CopilotAI
reviewed
Jun 17, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This pull request standardizes the preprocessing and table annotation functionality across multiple ABM models by replacing custom assign‐columns logic with unified calls to expressions.annotate_preprocessors and expressions.annotate_tables. Key changes include:
  • Replacing custom preprocessing calls with standardized annotation functions.
  • Removing unused/deprecated settings and consolidating configuration fields.
  • Updating multiple model files (e.g., auto_ownership, free_parking, joint_tour_frequency, etc.) to support the new standardized approach.
Reviewed Changes
Copilot reviewed 47 out of 47 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
activitysim/abm/models/parking_location_choice.pyAdded deprecation warning handling and introduced an alternative preprocessor field.
activitysim/abm/models/non_mandatory_tour_frequency.pyReplaced assign_columns with annotate_preprocessors and updated locals dictionary usage.
activitysim/abm/models/non_mandatory_scheduling.pyUpdated preprocessing and table annotation calls with new expressions functions.
activitysim/abm/models/non_mandatory_destination.pyRemoved legacy annotation function calls and standardized table annotation.
activitysim/abm/models/mandatory_tour_frequency.pyConsolidated preprocessing logic into standardized annotate_preprocessors and annotate_tables calls.
activitysim/abm/models/location_choice.pyUpdated both chooser and alternative preprocessing with standardized annotation calls.
activitysim/abm/models/joint_*.pyUnified preprocessing and annotation functions across joint tour models.
activitysim/abm/models/free_parking.pyUpdated preprocessing calls to use expressions.annotate_preprocessors and annotate_tables.
activitysim/abm/models/cdap.pyAdded preprocessing and post-model annotation functions; restructured settings fields.
activitysim/abm/models/auto_ownership.pyRemoved unused preprocessor settings by moving to a pass-through structure.
activitysim/abm/models/atwork_* (scheduling, mode_choice, frequency, destination)Consolidated preprocessing and table annotation logic with the new standardized approach.
Comments suppressed due to low confidence (1)
activitysim/abm/models/location_choice.py:184
  • [nitpick] The alternative preprocessor setting name 'alts_preprocessor_sample' differs from similar settings in other models. Consider adopting a consistent naming convention (e.g., 'alts_preprocessor') across the codebase.
 preprocessor_setting_name="alts_preprocessor_sample",

Comment threadactivitysim/abm/models/non_mandatory_tour_frequency.py
@jpn--
jpn--
self-requested a review
June 17, 2025 18:52
@jpn--
jpn--
commented
Jun 17, 2025

Copy link


Copy Markdown

Member
A few thoughts before I dig into this more:
  1. Estimation mode tests are failing.
  2. Other than estimation mode, existing tests do show we have not broken anything that was tested. But this PR appears to add significant functionality, and lacks any testing of that new functionality.
@dhensle
dhensle
commented
Jun 17, 2025

Copy link


Copy Markdown

ContributorAuthor
Hey @jpn--, thanks for your first look. As mentioned during the call today, I have a few more things to clean up on the CI tests (which is why the PR is listed as a "draft"). I will ping you in the next few days when my commits are complete and this PR is ready for review.
@dhensle
dhensle
marked this pull request as ready for review
June 18, 2025 23:35
@dhensle
dhensle
commented
Jun 18, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- Ok, this is ready for your review. I fixed the estimation mode issue and added unit tests for both the preprocessor and annotation functions which include some common expressions including reindex, skims lookups (in multiple ways), and some groupby calculations across tables. If you have other CI test ideas, I am happy to hear them.
jpn--
jpn--
requested changes
Jun 27, 2025
View reviewed changes
@jpn--jpn--
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! I found very little substantive to address, just a few things to clean up. Thanks for all the hard work on this.
Comment threadactivitysim/abm/models/vehicle_allocation.py
Outdated

 """Setting for the preprocessor."""

 OCCUPANCY_LEVELS: list = [1] # TODO Check this
 OCCUPANCY_LEVELS: list = [1, 2, 3.5]
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearly, the previous default value here of [1] was not a great solution. But it seems the occupancy levels set here needs to jive with calls to the generated variable names set in spec files for other components. "df.vehicle_occup_1" is going to show up as a variable name elsewhere in other components. I think it would be better to make that clear in the docstring, and have no default at all (i.e. failure to define this in the config for this model is an error).
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this appears to be basically unrelated to the fundamental nature of this PR, maybe just leave it out here, and we can address it in a separate specific PR for this.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, we can address this separately. Reverted.
Comment threadactivitysim/abm/models/mandatory_tour_frequency.py
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MandatoryTourFrequencySettings still has a redundant annotate_persons defined, which it also inherits from LogitComponentSettings
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, removed.
Comment threadactivitysim/abm/models/cdap.py

 preprocessor: PreprocessorSettings | None = None
 """Preprocess choosers tables before running the model."""
 annotate_persons: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automatic documentation tools will work better if annotate_persons has its own docstring, even though it's clear to a human reader the one docstring below applies to both.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/configuration/logit.py
 utility expressions.
 """

 annotate_households: PreprocessorSettings | None = None
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For autodoc, have each annotate_* field have its own docstring.
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done
Comment threadactivitysim/core/util.py
Outdated

 unique_variables_in_spec.update(re.findall(pattern, custom_chooser_lines))

 logger.info("Dropping unused variables in chooser table")
 logger.debug("Dropping unused variables in chooser table")
@jpn--jpn--Jun 27, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is clearly unrelated to this PR.
I do think we should address our logging, by default we are logging far too much
@dhensledhensleJul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I put this in my code cause it was annoying to look at while debugging. Happy to address this in a separate logging task. reverted.
@dhensle
dhensle
commented
Jul 1, 2025

Copy link


Copy Markdown

ContributorAuthor
@jpn-- thanks for your comments. I have addressed them, but please let me know if there's anything else you find!
@jpn--
jpn--
merged commit ba4a12e
into
ActivitySim:mainJul 17, 2025
@dhensledhensle
mentioned this pull request
Aug 1, 2025
 Merged

@dhensle
dhensle
deleted the

preprocessing

branch
August 1, 2025 17:38
@dhensledhensle
mentioned this pull request
Aug 13, 2025

Closed

@dhensledhensle
mentioned this pull request
Oct 28, 2025

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@jpn--jpn--jpn-- approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@dhensle@jpn--@aletzdy