Automatically drop unneeded columns in choosers table - #833

Merged
jpn-- merged 19 commits into
ActivitySim:mainfrom
wsp-sag:auto-drop-vars
Apr 21, 2024
Merged

Automatically drop unneeded columns in choosers table#833
jpn-- merged 19 commits into
ActivitySim:mainfrom
wsp-sag:auto-drop-vars

Conversation

@i-am-sijia

@i-am-sijiai-am-sijia commented Mar 18, 2024•
edited
Loading

Copy link
Copy Markdown
Member

This PR addresses #792

  • Drop unused columns automatically based on UECs
  • Pick up variables used/defined in source code, e.g. custom chooser
  • Skip dropping when running household debug tracing
  • Skip dropping when running estimation mode
  • Pass all existing CI tests
  • Run benchmarking runs
  • Clean up codes, consolidate dup codes into util funtions

@i-am-sijia

i-am-sijia commented Mar 26, 2024

Copy link
Copy Markdown
MemberAuthor

Note that when dropping columns in pandas df, the memory actually goes up momentarily before it goes down. see discussion here pandas-dev/pandas#17092, this issue was "closed" but it did not get resolved.

Looking at my code changes last week, I realized I was also dropping the columns a bit too late for interaction simulate models. The columns should be dropped before interaction df is created. I then made those changes.

Below shows the 1-zone benchmarking runs with main branch (top) and with this PR (bottom). In the non-sharrow mode, the peak memory went down from 352 GB to 261 GB, a 25% reduction.

run time comp

Component-wise memory reduction, sorted by memory saving:

image

I'm still dealing with some crashes in the sharrow mode.

@jpn--

jpn-- commented Mar 26, 2024

Copy link
Copy Markdown
Member

this issue was "closed" but it did not get resolved

If you read the thread closely, you'll note that that "this is not going to be solved in pandas 1." The behind-the-curtain memory management of pandas 1.x doesn't offer any way to drop unused columns without consuming extra RAM. This is alleviated by moving to pandas 2... after the transition, there are now ways to get the memory benefit. This is (part of) the reason I've worked this past week to try to get us pandas-2 compatible.

@i-am-sijia

i-am-sijia commented Apr 1, 2024

Copy link
Copy Markdown
MemberAuthor

I have some updates for the Sharrow runs. Below are three 1-zone benchmarking runs with Sharrow. The one on the top (3/20) used the main branch at 6d817be, the middle (3/28) and bottom (3/29) ones both used this PR with the latest code changes I made last week. There is a lot to discuss here.

  1. The peak memory of the entire model did not reduce much after dropping unused variables, although there were reductions in most components. With the main branch, the memory peak was Mandatory Tour Scheduling (156 GB), and a close runner-up was Write Trip Matrices (155 GB). After dropping unused variables, Mandatory Tour Scheduling reduced to 101 GB, but Write Trip Matrices reduced to 147 GB. So the peak memory usage only reduced from 156 GB to 147 GB.

  2. The two runs after dropping unused variables showed different memory usage patterns, even though they used the same source code. The one on 3/28 showed accumulation of memory in Trip Destination, while the one on 3/29 showed accumulation of memory in Mandatory Tour Scheduling (that caused School Escorting having a high mark to start with). This looks relevant to issue Double check Sharrow memory usage fix (#751) is merged and implemented #816. We have "memory leaks" that seem to show up sometimes but not all times.

  3. The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours. This could mean that dropping unused variables might have caused longer run time, even though I did not observe longer run time in the Non Sharrow runs reported week. In the Non Sharrow runs, in addition to a 25% peak memory reduction, dropping unused variable also reduced run time by 10 mins. One possibility is that dropping variables in the Sharrow mode has implications on run time that I have not realized. Another possibility is that the machine needs a reboot as things piled up that slowed it down (although this is a virtual machine with nothing else running).

run time comp sharrow

<style> </style>
EventWSP Sharrow On 3-20 MainWSP Sharrow On 3-28 PR 833WSP Sharrow On 3-29 PR 833
Max Memory (GB)156.1146.8150.5
mandatory_tour_scheduling156.1101.8109.9
write_trip_matrices155.0146.7147.2
trip_destination132.8146.8130.4
school_escorting124.8114.3150.5
non_mandatory_tour_scheduling116.6106.788.6
atwork_subtour_scheduling89.343.142.0
workplace_location82.872.981.2
atwork_subtour_mode_choice79.640.337.9
trip_mode_choice66.660.261.0
stop_frequency64.262.265.1
non_mandatory_tour_destination61.767.964.8
vehicle_allocation60.199.188.2
write_data_dictionary59.748.348.5
trip_scheduling58.950.350.7
school_location58.957.963.6
trip_purpose_and_destination55.750.748.3
atwork_subtour_destination54.351.048.0
track_skim_usage53.241.541.8
write_tables52.343.644.0
finalizing48.641.441.7
tour_mode_choice_simulate47.952.945.4
joint_tour_destination47.248.049.8
non_mandatory_tour_frequency42.550.741.7
trip_purpose39.438.138.4
joint_tour_scheduling36.536.033.0
atwork_subtour_frequency30.830.229.3
vehicle_type_choice28.828.727.6
joint_tour_participation28.829.928.5
cdap_simulate24.824.025.5
joint_tour_frequency24.224.324.4
free_parking21.623.323.5
joint_tour_composition21.321.221.2
compute_disaggregate_accessibility20.723.121.2
initialize_households20.421.222.3
mandatory_tour_frequency18.523.023.7
compute_accessibility12.715.516.9
auto_ownership_simulate11.112.011.8
initialize_landuse7.114.011.4
initialize_proto_population6.46.46.4
input_checker1.91.61.6
preload_injectables0.30.30.3
na0.30.30.3

@i-am-sijia
i-am-sijia marked this pull request as ready for review April 2, 2024 19:05
@i-am-sijia
i-am-sijia requested a review from jpn--April 2, 2024 19:13
i-am-sijia
i-am-sijia commented Apr 3, 2024
View reviewed changes
Comment threadactivitysim/core/util.py
@i-am-sijia

i-am-sijia commented Apr 8, 2024•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

Update RE:

The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours.

This seems to be because flow.load time increased.

For example, in the 3/20 run:

18/03/202419:27:38.294-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF218/03/202419:27:38.294-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07700018/03/202419:27:38.294-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:02:27.026987school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedapply_flowin0:02:27.103988 ...
18/03/202420:09:15.654-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:09:15.995-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X18/03/202420:09:16.193-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.55200118/03/202420:09:16.193-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:03:48.747827vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedapply_flowin0:03:49.299828 ...

In the 3/29 run:

29/03/202417:41:59.160-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF229/03/202417:41:59.160-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07813929/03/202417:41:59.160-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:11:32.047284school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedapply_flowin0:11:32.125423 ...
29/03/202419:11:49.595-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:11:49.924-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X29/03/202419:11:49.955-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.39058029/03/202419:11:49.955-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:09:37.196634vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedapply_flowin0:09:37.587213 ...

Not sure if this is because of recent sharrow updates, or because of recent updates in the way unused columns are dropped.

@jpn--

jpn-- commented Apr 11, 2024

Copy link
Copy Markdown
Member

I am not sure what is happening that has cause the code to run slower for @i-am-sijia , I am unable to replicate the problem.

I have run the full scale model on my laptop through school location choice (the first problematic model shown above) and got these results:

sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow as of 3/28)
activitysim commit e5d9878 (this PR as of 3/28)

[08:20.92] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:03.380284 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed apply_flow in 0:01:03.416361

sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7, in use 3/20)
activitysim commit 94c4db8 (main branch as of 3/20)

[08:11.76] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:00.648809 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed apply_flow in 0:01:00.678720

@jpn--

jpn-- commented Apr 16, 2024

Copy link
Copy Markdown
Member

I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.

@i-am-sijia

i-am-sijia commented Apr 16, 2024•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

I reran the reported "3/20" and "3/29" runs, see Test 1 and Test 2 below. Last time they were run on different machines. This time they are run on the same machine. Good news is that this time I'm not seeing a huge runtime difference. Test 3 uses the same commits as one of Jeff's test, apparently the run time on Win is longer than Mac.

Test 1 (Rerunning the "3/20" run)

Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)
ActivitySim commit 6d817be (main, as of 3/18)
activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)

16/04/202413:41:49.585-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF216/04/202413:41:49.587-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.11417416/04/202413:41:49.587-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:07:21.590254school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedapply_flowin0:07:21.704428

Test 2 (Rerunning the "3/29" run)

Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)
ActivitySim commit e5d9878 (this PR as of 3/28)
activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)

12/04/202420:47:53.356-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF212/04/202420:47:53.356-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07812812/04/202420:47:53.356-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:08:50.832949school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedapply_flowin0:08:50.911078

Test 3 (use the latest sharrow)

Sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow v2.8.2 as of 3/28)
ActivitySim commit e5d9878 (this PR as of 3/28)
activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)

15/04/202413:55:03.917-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF215/04/202413:55:03.917-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.08288215/04/202413:55:03.917-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:06:49.217218school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedapply_flowin0:06:49.300100

@dhensledhensle mentioned this pull request Apr 16, 2024
Closed
@i-am-sijia

i-am-sijia commented Apr 17, 2024•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.

@jpn-- , I made the key code changes needed for turning this feature on and off for individual components, I haven't pushed them. It's essentially passing a Boolean drop_unused_columns from component model settings into simple_simulate, interaction_simulate, interaction_sample etc. I modified almost all component .py (like auto_ownership.py) to feed in this setting.

Then I paused and read your latest comment on #824. Since you are generalizing the sharrow_settings to compute_settings and I checked that they are consumed at the same place where drop_unused_columns are required, it may make sense to include drop_unused_columns in compute_settings, or I can create a sibling of compute_settings to host drop_unused_columns and any other settings that might be created in the future. Thoughts?

@jpn--

jpn-- commented Apr 18, 2024

Copy link
Copy Markdown
Member

it may make sense to include drop_unused_columns in compute_settings

I like this idea. Let's finish the review/merge of #824 and then it should be easy to update this PR and put this straight into the compute_settings structure.

@jpn--jpn-- mentioned this pull request Apr 18, 2024
Merged
@i-am-sijia

i-am-sijia commented Apr 21, 2024

Copy link
Copy Markdown
MemberAuthor

@jpn-- , I implemented drop_unused_columns in compute_settings, cherry-picked the parking location changes in #849 and the protect additional variables in https://github.com/camsys/activitysim/commit/574ee0eb444e0e97c5b2ac8c9215ed9df55bd53d, so that everything is included in this PR for efficient review and merge.

I tested setting compute_settings.drop_unused_columns to false in auto ownership, non-mandatory tour destination and scheduling, school escorting, and trip mode choice. It worked as expected.

@jpn--
jpn-- merged commit 239f415 into ActivitySim:mainApr 21, 2024
@dhensledhensle mentioned this pull request Jan 28, 2025
Merged
11 tasks
@i-am-sijia
i-am-sijia deleted the auto-drop-vars branch October 27, 2025 15:45
@jpn--jpn-- mentioned this pull request Aug 13, 2026
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jpn--jpn--jpn-- approved these changes

Assignees

@i-am-sijiai-am-sijia

Labels

None yet

Projects

Phase 9 Work
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@i-am-sijia@jpn--
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Automatically drop unneeded columns in choosers table - #833
Merged
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Apr 21, 2024
Merged
Automatically drop unneeded columns in choosers table#833
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Conversation
@i-am-sijia
@i-am-sijiai-am-sijia
commented
Mar 18, 2024•

edited

Loading

Copy link


Copy Markdown

Member
This PR addresses #792
  •  Drop unused columns automatically based on UECs
  •  Pick up variables used/defined in source code, e.g. custom chooser
  •  Skip dropping when running household debug tracing
  •  Skip dropping when running estimation mode
  •  Pass all existing CI tests
  •  Run benchmarking runs
  •  Clean up codes, consolidate dup codes into util funtions
@i-am-sijia
i-am-sijia
commented
Mar 26, 2024

Copy link


Copy Markdown

MemberAuthor
Note that when dropping columns in pandas df, the memory actually goes up momentarily before it goes down. see discussion here pandas-dev/pandas#17092, this issue was "closed" but it did not get resolved.
Looking at my code changes last week, I realized I was also dropping the columns a bit too late for interaction simulate models. The columns should be dropped before interaction df is created. I then made those changes.
Below shows the 1-zone benchmarking runs with main branch (top) and with this PR (bottom). In the non-sharrow mode, the peak memory went down from 352 GB to 261 GB, a 25% reduction.
run time comp
Component-wise memory reduction, sorted by memory saving:
image
I'm still dealing with some crashes in the sharrow mode.
@jpn--
jpn--
commented
Mar 26, 2024

Copy link


Copy Markdown

Member
this issue was "closed" but it did not get resolved
If you read the thread closely, you'll note that that "this is not going to be solved in pandas 1." The behind-the-curtain memory management of pandas 1.x doesn't offer any way to drop unused columns without consuming extra RAM. This is alleviated by moving to pandas 2... after the transition, there are now ways to get the memory benefit. This is (part of) the reason I've worked this past week to try to get us pandas-2 compatible.
@i-am-sijia
i-am-sijia
commented
Apr 1, 2024

Copy link


Copy Markdown

MemberAuthor
I have some updates for the Sharrow runs. Below are three 1-zone benchmarking runs with Sharrow. The one on the top (3/20) used the main branch at 6d817be, the middle (3/28) and bottom (3/29) ones both used this PR with the latest code changes I made last week. There is a lot to discuss here.
  1. The peak memory of the entire model did not reduce much after dropping unused variables, although there were reductions in most components. With the main branch, the memory peak was Mandatory Tour Scheduling (156 GB), and a close runner-up was Write Trip Matrices (155 GB). After dropping unused variables, Mandatory Tour Scheduling reduced to 101 GB, but Write Trip Matrices reduced to 147 GB. So the peak memory usage only reduced from 156 GB to 147 GB.
  2. The two runs after dropping unused variables showed different memory usage patterns, even though they used the same source code. The one on 3/28 showed accumulation of memory in Trip Destination, while the one on 3/29 showed accumulation of memory in Mandatory Tour Scheduling (that caused School Escorting having a high mark to start with). This looks relevant to issue Double check Sharrow memory usage fix (#751) is merged and implemented #816. We have "memory leaks" that seem to show up sometimes but not all times.
  3. The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours. This could mean that dropping unused variables might have caused longer run time, even though I did not observe longer run time in the Non Sharrow runs reported week. In the Non Sharrow runs, in addition to a 25% peak memory reduction, dropping unused variable also reduced run time by 10 mins. One possibility is that dropping variables in the Sharrow mode has implications on run time that I have not realized. Another possibility is that the machine needs a reboot as things piled up that slowed it down (although this is a virtual machine with nothing else running).
run time comp sharrow

<style>
</style>

EventWSP Sharrow On 3-20 MainWSP Sharrow On 3-28 PR 833WSP Sharrow On 3-29 PR 833
Max Memory (GB)156.1146.8150.5
mandatory_tour_scheduling156.1101.8109.9
write_trip_matrices155.0146.7147.2
trip_destination132.8146.8130.4
school_escorting124.8114.3150.5
non_mandatory_tour_scheduling116.6106.788.6
atwork_subtour_scheduling89.343.142.0
workplace_location82.872.981.2
atwork_subtour_mode_choice79.640.337.9
trip_mode_choice66.660.261.0
stop_frequency64.262.265.1
non_mandatory_tour_destination61.767.964.8
vehicle_allocation60.199.188.2
write_data_dictionary59.748.348.5
trip_scheduling58.950.350.7
school_location58.957.963.6
trip_purpose_and_destination55.750.748.3
atwork_subtour_destination54.351.048.0
track_skim_usage53.241.541.8
write_tables52.343.644.0
finalizing48.641.441.7
tour_mode_choice_simulate47.952.945.4
joint_tour_destination47.248.049.8
non_mandatory_tour_frequency42.550.741.7
trip_purpose39.438.138.4
joint_tour_scheduling36.536.033.0
atwork_subtour_frequency30.830.229.3
vehicle_type_choice28.828.727.6
joint_tour_participation28.829.928.5
cdap_simulate24.824.025.5
joint_tour_frequency24.224.324.4
free_parking21.623.323.5
joint_tour_composition21.321.221.2
compute_disaggregate_accessibility20.723.121.2
initialize_households20.421.222.3
mandatory_tour_frequency18.523.023.7
compute_accessibility12.715.516.9
auto_ownership_simulate11.112.011.8
initialize_landuse7.114.011.4
initialize_proto_population6.46.46.4
input_checker1.91.61.6
preload_injectables0.30.30.3
na0.30.30.3
@i-am-sijia
i-am-sijia
marked this pull request as ready for review
April 2, 2024 19:05
@i-am-sijia
i-am-sijia
requested a review
from jpn--April 2, 2024 19:13
i-am-sijia
i-am-sijia
commented
Apr 3, 2024
View reviewed changes
Comment threadactivitysim/core/util.py
@i-am-sijia
i-am-sijia
commented
Apr 8, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Update RE:
The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours.
This seems to be because flow.load time increased.
For example, in the 3/20 run:
18/03/202419:27:38.294-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF218/03/202419:27:38.294-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07700018/03/202419:27:38.294-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:02:27.026987school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedapply_flowin0:02:27.103988 ...
18/03/202420:09:15.654-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:09:15.995-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X18/03/202420:09:16.193-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.55200118/03/202420:09:16.193-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:03:48.747827vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedapply_flowin0:03:49.299828 ...
In the 3/29 run:
29/03/202417:41:59.160-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF229/03/202417:41:59.160-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07813929/03/202417:41:59.160-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:11:32.047284school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedapply_flowin0:11:32.125423 ...
29/03/202419:11:49.595-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:11:49.924-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X29/03/202419:11:49.955-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.39058029/03/202419:11:49.955-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:09:37.196634vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedapply_flowin0:09:37.587213 ...
Not sure if this is because of recent sharrow updates, or because of recent updates in the way unused columns are dropped.
@jpn--
jpn--
commented
Apr 11, 2024

Copy link


Copy Markdown

Member
I am not sure what is happening that has cause the code to run slower for @i-am-sijia , I am unable to replicate the problem.
I have run the full scale model on my laptop through school location choice (the first problematic model shown above) and got these results:
sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow as of 3/28)

activitysim commit e5d9878 (this PR as of 3/28)
[08:20.92] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:03.380284 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed apply_flow in 0:01:03.416361 
sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7, in use 3/20)

activitysim commit 94c4db8 (main branch as of 3/20)
[08:11.76] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:00.648809 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed apply_flow in 0:01:00.678720 
@jpn--
jpn--
commented
Apr 16, 2024

Copy link


Copy Markdown

Member
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@i-am-sijia
i-am-sijia
commented
Apr 16, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I reran the reported "3/20" and "3/29" runs, see Test 1 and Test 2 below. Last time they were run on different machines. This time they are run on the same machine. Good news is that this time I'm not seeing a huge runtime difference. Test 3 uses the same commits as one of Jeff's test, apparently the run time on Win is longer than Mac.
Test 1 (Rerunning the "3/20" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit 6d817be (main, as of 3/18)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
16/04/202413:41:49.585-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF216/04/202413:41:49.587-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.11417416/04/202413:41:49.587-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:07:21.590254school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedapply_flowin0:07:21.704428
Test 2 (Rerunning the "3/29" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
12/04/202420:47:53.356-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF212/04/202420:47:53.356-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07812812/04/202420:47:53.356-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:08:50.832949school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedapply_flowin0:08:50.911078
Test 3 (use the latest sharrow)
Sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow v2.8.2 as of 3/28)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
15/04/202413:55:03.917-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF215/04/202413:55:03.917-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.08288215/04/202413:55:03.917-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:06:49.217218school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedapply_flowin0:06:49.300100
@dhensledhensle
mentioned this pull request
Apr 16, 2024
 Closed

@i-am-sijia
i-am-sijia
commented
Apr 17, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@jpn-- , I made the key code changes needed for turning this feature on and off for individual components, I haven't pushed them. It's essentially passing a Boolean drop_unused_columns from component model settings into simple_simulate, interaction_simulate, interaction_sample etc. I modified almost all component .py (like auto_ownership.py) to feed in this setting.
Then I paused and read your latest comment on #824. Since you are generalizing the sharrow_settings to compute_settings and I checked that they are consumed at the same place where drop_unused_columns are required, it may make sense to include drop_unused_columns in compute_settings, or I can create a sibling of compute_settings to host drop_unused_columns and any other settings that might be created in the future. Thoughts?
@jpn--
jpn--
commented
Apr 18, 2024

Copy link


Copy Markdown

Member
it may make sense to include drop_unused_columns in compute_settings
I like this idea. Let's finish the review/merge of #824 and then it should be easy to update this PR and put this straight into the compute_settings structure.
@jpn--jpn--
mentioned this pull request
Apr 18, 2024
 Merged

@i-am-sijia
i-am-sijia
commented
Apr 21, 2024

Copy link


Copy Markdown

MemberAuthor
@jpn-- , I implemented drop_unused_columns in compute_settings, cherry-picked the parking location changes in #849 and the protect additional variables in https://github.com/camsys/activitysim/commit/574ee0eb444e0e97c5b2ac8c9215ed9df55bd53d, so that everything is included in this PR for efficient review and merge.
I tested setting compute_settings.drop_unused_columns to false in auto ownership, non-mandatory tour destination and scheduling, school escorting, and trip mode choice. It worked as expected.
@jpn--
jpn--
merged commit 239f415
into
ActivitySim:mainApr 21, 2024
@dhensledhensle
mentioned this pull request
Jan 28, 2025
 Merged

11 tasks
@i-am-sijia
i-am-sijia
deleted the

auto-drop-vars

branch
October 27, 2025 15:45
@jpn--jpn--
mentioned this pull request
Aug 13, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jpn--jpn--jpn-- approved these changes

Assignees

@i-am-sijiai-am-sijia

Labels


None yet


Projects

Phase 9 Work

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@i-am-sijia@jpn--

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Automatically drop unneeded columns in choosers table - #833
Merged
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Apr 21, 2024
Merged
Automatically drop unneeded columns in choosers table#833
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Conversation
@i-am-sijia
@i-am-sijiai-am-sijia
commented
Mar 18, 2024•

edited

Loading

Copy link


Copy Markdown

Member
This PR addresses #792
  •  Drop unused columns automatically based on UECs
  •  Pick up variables used/defined in source code, e.g. custom chooser
  •  Skip dropping when running household debug tracing
  •  Skip dropping when running estimation mode
  •  Pass all existing CI tests
  •  Run benchmarking runs
  •  Clean up codes, consolidate dup codes into util funtions
@i-am-sijia
i-am-sijia
commented
Mar 26, 2024

Copy link


Copy Markdown

MemberAuthor
Note that when dropping columns in pandas df, the memory actually goes up momentarily before it goes down. see discussion here pandas-dev/pandas#17092, this issue was "closed" but it did not get resolved.
Looking at my code changes last week, I realized I was also dropping the columns a bit too late for interaction simulate models. The columns should be dropped before interaction df is created. I then made those changes.
Below shows the 1-zone benchmarking runs with main branch (top) and with this PR (bottom). In the non-sharrow mode, the peak memory went down from 352 GB to 261 GB, a 25% reduction.
run time comp
Component-wise memory reduction, sorted by memory saving:
image
I'm still dealing with some crashes in the sharrow mode.
@jpn--
jpn--
commented
Mar 26, 2024

Copy link


Copy Markdown

Member
this issue was "closed" but it did not get resolved
If you read the thread closely, you'll note that that "this is not going to be solved in pandas 1." The behind-the-curtain memory management of pandas 1.x doesn't offer any way to drop unused columns without consuming extra RAM. This is alleviated by moving to pandas 2... after the transition, there are now ways to get the memory benefit. This is (part of) the reason I've worked this past week to try to get us pandas-2 compatible.
@i-am-sijia
i-am-sijia
commented
Apr 1, 2024

Copy link


Copy Markdown

MemberAuthor
I have some updates for the Sharrow runs. Below are three 1-zone benchmarking runs with Sharrow. The one on the top (3/20) used the main branch at 6d817be, the middle (3/28) and bottom (3/29) ones both used this PR with the latest code changes I made last week. There is a lot to discuss here.
  1. The peak memory of the entire model did not reduce much after dropping unused variables, although there were reductions in most components. With the main branch, the memory peak was Mandatory Tour Scheduling (156 GB), and a close runner-up was Write Trip Matrices (155 GB). After dropping unused variables, Mandatory Tour Scheduling reduced to 101 GB, but Write Trip Matrices reduced to 147 GB. So the peak memory usage only reduced from 156 GB to 147 GB.
  2. The two runs after dropping unused variables showed different memory usage patterns, even though they used the same source code. The one on 3/28 showed accumulation of memory in Trip Destination, while the one on 3/29 showed accumulation of memory in Mandatory Tour Scheduling (that caused School Escorting having a high mark to start with). This looks relevant to issue Double check Sharrow memory usage fix (#751) is merged and implemented #816. We have "memory leaks" that seem to show up sometimes but not all times.
  3. The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours. This could mean that dropping unused variables might have caused longer run time, even though I did not observe longer run time in the Non Sharrow runs reported week. In the Non Sharrow runs, in addition to a 25% peak memory reduction, dropping unused variable also reduced run time by 10 mins. One possibility is that dropping variables in the Sharrow mode has implications on run time that I have not realized. Another possibility is that the machine needs a reboot as things piled up that slowed it down (although this is a virtual machine with nothing else running).
run time comp sharrow

<style>
</style>

EventWSP Sharrow On 3-20 MainWSP Sharrow On 3-28 PR 833WSP Sharrow On 3-29 PR 833
Max Memory (GB)156.1146.8150.5
mandatory_tour_scheduling156.1101.8109.9
write_trip_matrices155.0146.7147.2
trip_destination132.8146.8130.4
school_escorting124.8114.3150.5
non_mandatory_tour_scheduling116.6106.788.6
atwork_subtour_scheduling89.343.142.0
workplace_location82.872.981.2
atwork_subtour_mode_choice79.640.337.9
trip_mode_choice66.660.261.0
stop_frequency64.262.265.1
non_mandatory_tour_destination61.767.964.8
vehicle_allocation60.199.188.2
write_data_dictionary59.748.348.5
trip_scheduling58.950.350.7
school_location58.957.963.6
trip_purpose_and_destination55.750.748.3
atwork_subtour_destination54.351.048.0
track_skim_usage53.241.541.8
write_tables52.343.644.0
finalizing48.641.441.7
tour_mode_choice_simulate47.952.945.4
joint_tour_destination47.248.049.8
non_mandatory_tour_frequency42.550.741.7
trip_purpose39.438.138.4
joint_tour_scheduling36.536.033.0
atwork_subtour_frequency30.830.229.3
vehicle_type_choice28.828.727.6
joint_tour_participation28.829.928.5
cdap_simulate24.824.025.5
joint_tour_frequency24.224.324.4
free_parking21.623.323.5
joint_tour_composition21.321.221.2
compute_disaggregate_accessibility20.723.121.2
initialize_households20.421.222.3
mandatory_tour_frequency18.523.023.7
compute_accessibility12.715.516.9
auto_ownership_simulate11.112.011.8
initialize_landuse7.114.011.4
initialize_proto_population6.46.46.4
input_checker1.91.61.6
preload_injectables0.30.30.3
na0.30.30.3
@i-am-sijia
i-am-sijia
marked this pull request as ready for review
April 2, 2024 19:05
@i-am-sijia
i-am-sijia
requested a review
from jpn--April 2, 2024 19:13
i-am-sijia
i-am-sijia
commented
Apr 3, 2024
View reviewed changes
Comment threadactivitysim/core/util.py
@i-am-sijia
i-am-sijia
commented
Apr 8, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Update RE:
The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours.
This seems to be because flow.load time increased.
For example, in the 3/20 run:
18/03/202419:27:38.294-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF218/03/202419:27:38.294-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07700018/03/202419:27:38.294-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:02:27.026987school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedapply_flowin0:02:27.103988 ...
18/03/202420:09:15.654-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:09:15.995-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X18/03/202420:09:16.193-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.55200118/03/202420:09:16.193-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:03:48.747827vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedapply_flowin0:03:49.299828 ...
In the 3/29 run:
29/03/202417:41:59.160-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF229/03/202417:41:59.160-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07813929/03/202417:41:59.160-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:11:32.047284school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedapply_flowin0:11:32.125423 ...
29/03/202419:11:49.595-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:11:49.924-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X29/03/202419:11:49.955-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.39058029/03/202419:11:49.955-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:09:37.196634vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedapply_flowin0:09:37.587213 ...
Not sure if this is because of recent sharrow updates, or because of recent updates in the way unused columns are dropped.
@jpn--
jpn--
commented
Apr 11, 2024

Copy link


Copy Markdown

Member
I am not sure what is happening that has cause the code to run slower for @i-am-sijia , I am unable to replicate the problem.
I have run the full scale model on my laptop through school location choice (the first problematic model shown above) and got these results:
sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow as of 3/28)

activitysim commit e5d9878 (this PR as of 3/28)
[08:20.92] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:03.380284 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed apply_flow in 0:01:03.416361 
sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7, in use 3/20)

activitysim commit 94c4db8 (main branch as of 3/20)
[08:11.76] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:00.648809 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed apply_flow in 0:01:00.678720 
@jpn--
jpn--
commented
Apr 16, 2024

Copy link


Copy Markdown

Member
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@i-am-sijia
i-am-sijia
commented
Apr 16, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I reran the reported "3/20" and "3/29" runs, see Test 1 and Test 2 below. Last time they were run on different machines. This time they are run on the same machine. Good news is that this time I'm not seeing a huge runtime difference. Test 3 uses the same commits as one of Jeff's test, apparently the run time on Win is longer than Mac.
Test 1 (Rerunning the "3/20" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit 6d817be (main, as of 3/18)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
16/04/202413:41:49.585-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF216/04/202413:41:49.587-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.11417416/04/202413:41:49.587-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:07:21.590254school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedapply_flowin0:07:21.704428
Test 2 (Rerunning the "3/29" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
12/04/202420:47:53.356-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF212/04/202420:47:53.356-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07812812/04/202420:47:53.356-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:08:50.832949school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedapply_flowin0:08:50.911078
Test 3 (use the latest sharrow)
Sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow v2.8.2 as of 3/28)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
15/04/202413:55:03.917-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF215/04/202413:55:03.917-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.08288215/04/202413:55:03.917-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:06:49.217218school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedapply_flowin0:06:49.300100
@dhensledhensle
mentioned this pull request
Apr 16, 2024
 Closed

@i-am-sijia
i-am-sijia
commented
Apr 17, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@jpn-- , I made the key code changes needed for turning this feature on and off for individual components, I haven't pushed them. It's essentially passing a Boolean drop_unused_columns from component model settings into simple_simulate, interaction_simulate, interaction_sample etc. I modified almost all component .py (like auto_ownership.py) to feed in this setting.
Then I paused and read your latest comment on #824. Since you are generalizing the sharrow_settings to compute_settings and I checked that they are consumed at the same place where drop_unused_columns are required, it may make sense to include drop_unused_columns in compute_settings, or I can create a sibling of compute_settings to host drop_unused_columns and any other settings that might be created in the future. Thoughts?
@jpn--
jpn--
commented
Apr 18, 2024

Copy link


Copy Markdown

Member
it may make sense to include drop_unused_columns in compute_settings
I like this idea. Let's finish the review/merge of #824 and then it should be easy to update this PR and put this straight into the compute_settings structure.
@jpn--jpn--
mentioned this pull request
Apr 18, 2024
 Merged

@i-am-sijia
i-am-sijia
commented
Apr 21, 2024

Copy link


Copy Markdown

MemberAuthor
@jpn-- , I implemented drop_unused_columns in compute_settings, cherry-picked the parking location changes in #849 and the protect additional variables in https://github.com/camsys/activitysim/commit/574ee0eb444e0e97c5b2ac8c9215ed9df55bd53d, so that everything is included in this PR for efficient review and merge.
I tested setting compute_settings.drop_unused_columns to false in auto ownership, non-mandatory tour destination and scheduling, school escorting, and trip mode choice. It worked as expected.
@jpn--
jpn--
merged commit 239f415
into
ActivitySim:mainApr 21, 2024
@dhensledhensle
mentioned this pull request
Jan 28, 2025
 Merged

11 tasks
@i-am-sijia
i-am-sijia
deleted the

auto-drop-vars

branch
October 27, 2025 15:45
@jpn--jpn--
mentioned this pull request
Aug 13, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jpn--jpn--jpn-- approved these changes

Assignees

@i-am-sijiai-am-sijia

Labels


None yet


Projects

Phase 9 Work

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@i-am-sijia@jpn--

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Automatically drop unneeded columns in choosers table - #833
Merged
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Apr 21, 2024
Merged
Automatically drop unneeded columns in choosers table#833
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Conversation
@i-am-sijia
@i-am-sijiai-am-sijia
commented
Mar 18, 2024•

edited

Loading

Copy link


Copy Markdown

Member
This PR addresses #792
  •  Drop unused columns automatically based on UECs
  •  Pick up variables used/defined in source code, e.g. custom chooser
  •  Skip dropping when running household debug tracing
  •  Skip dropping when running estimation mode
  •  Pass all existing CI tests
  •  Run benchmarking runs
  •  Clean up codes, consolidate dup codes into util funtions
@i-am-sijia
i-am-sijia
commented
Mar 26, 2024

Copy link


Copy Markdown

MemberAuthor
Note that when dropping columns in pandas df, the memory actually goes up momentarily before it goes down. see discussion here pandas-dev/pandas#17092, this issue was "closed" but it did not get resolved.
Looking at my code changes last week, I realized I was also dropping the columns a bit too late for interaction simulate models. The columns should be dropped before interaction df is created. I then made those changes.
Below shows the 1-zone benchmarking runs with main branch (top) and with this PR (bottom). In the non-sharrow mode, the peak memory went down from 352 GB to 261 GB, a 25% reduction.
run time comp
Component-wise memory reduction, sorted by memory saving:
image
I'm still dealing with some crashes in the sharrow mode.
@jpn--
jpn--
commented
Mar 26, 2024

Copy link


Copy Markdown

Member
this issue was "closed" but it did not get resolved
If you read the thread closely, you'll note that that "this is not going to be solved in pandas 1." The behind-the-curtain memory management of pandas 1.x doesn't offer any way to drop unused columns without consuming extra RAM. This is alleviated by moving to pandas 2... after the transition, there are now ways to get the memory benefit. This is (part of) the reason I've worked this past week to try to get us pandas-2 compatible.
@i-am-sijia
i-am-sijia
commented
Apr 1, 2024

Copy link


Copy Markdown

MemberAuthor
I have some updates for the Sharrow runs. Below are three 1-zone benchmarking runs with Sharrow. The one on the top (3/20) used the main branch at 6d817be, the middle (3/28) and bottom (3/29) ones both used this PR with the latest code changes I made last week. There is a lot to discuss here.
  1. The peak memory of the entire model did not reduce much after dropping unused variables, although there were reductions in most components. With the main branch, the memory peak was Mandatory Tour Scheduling (156 GB), and a close runner-up was Write Trip Matrices (155 GB). After dropping unused variables, Mandatory Tour Scheduling reduced to 101 GB, but Write Trip Matrices reduced to 147 GB. So the peak memory usage only reduced from 156 GB to 147 GB.
  2. The two runs after dropping unused variables showed different memory usage patterns, even though they used the same source code. The one on 3/28 showed accumulation of memory in Trip Destination, while the one on 3/29 showed accumulation of memory in Mandatory Tour Scheduling (that caused School Escorting having a high mark to start with). This looks relevant to issue Double check Sharrow memory usage fix (#751) is merged and implemented #816. We have "memory leaks" that seem to show up sometimes but not all times.
  3. The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours. This could mean that dropping unused variables might have caused longer run time, even though I did not observe longer run time in the Non Sharrow runs reported week. In the Non Sharrow runs, in addition to a 25% peak memory reduction, dropping unused variable also reduced run time by 10 mins. One possibility is that dropping variables in the Sharrow mode has implications on run time that I have not realized. Another possibility is that the machine needs a reboot as things piled up that slowed it down (although this is a virtual machine with nothing else running).
run time comp sharrow

<style>
</style>

EventWSP Sharrow On 3-20 MainWSP Sharrow On 3-28 PR 833WSP Sharrow On 3-29 PR 833
Max Memory (GB)156.1146.8150.5
mandatory_tour_scheduling156.1101.8109.9
write_trip_matrices155.0146.7147.2
trip_destination132.8146.8130.4
school_escorting124.8114.3150.5
non_mandatory_tour_scheduling116.6106.788.6
atwork_subtour_scheduling89.343.142.0
workplace_location82.872.981.2
atwork_subtour_mode_choice79.640.337.9
trip_mode_choice66.660.261.0
stop_frequency64.262.265.1
non_mandatory_tour_destination61.767.964.8
vehicle_allocation60.199.188.2
write_data_dictionary59.748.348.5
trip_scheduling58.950.350.7
school_location58.957.963.6
trip_purpose_and_destination55.750.748.3
atwork_subtour_destination54.351.048.0
track_skim_usage53.241.541.8
write_tables52.343.644.0
finalizing48.641.441.7
tour_mode_choice_simulate47.952.945.4
joint_tour_destination47.248.049.8
non_mandatory_tour_frequency42.550.741.7
trip_purpose39.438.138.4
joint_tour_scheduling36.536.033.0
atwork_subtour_frequency30.830.229.3
vehicle_type_choice28.828.727.6
joint_tour_participation28.829.928.5
cdap_simulate24.824.025.5
joint_tour_frequency24.224.324.4
free_parking21.623.323.5
joint_tour_composition21.321.221.2
compute_disaggregate_accessibility20.723.121.2
initialize_households20.421.222.3
mandatory_tour_frequency18.523.023.7
compute_accessibility12.715.516.9
auto_ownership_simulate11.112.011.8
initialize_landuse7.114.011.4
initialize_proto_population6.46.46.4
input_checker1.91.61.6
preload_injectables0.30.30.3
na0.30.30.3
@i-am-sijia
i-am-sijia
marked this pull request as ready for review
April 2, 2024 19:05
@i-am-sijia
i-am-sijia
requested a review
from jpn--April 2, 2024 19:13
i-am-sijia
i-am-sijia
commented
Apr 3, 2024
View reviewed changes
Comment threadactivitysim/core/util.py
@i-am-sijia
i-am-sijia
commented
Apr 8, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Update RE:
The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours.
This seems to be because flow.load time increased.
For example, in the 3/20 run:
18/03/202419:27:38.294-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF218/03/202419:27:38.294-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07700018/03/202419:27:38.294-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:02:27.026987school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedapply_flowin0:02:27.103988 ...
18/03/202420:09:15.654-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:09:15.995-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X18/03/202420:09:16.193-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.55200118/03/202420:09:16.193-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:03:48.747827vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedapply_flowin0:03:49.299828 ...
In the 3/29 run:
29/03/202417:41:59.160-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF229/03/202417:41:59.160-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07813929/03/202417:41:59.160-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:11:32.047284school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedapply_flowin0:11:32.125423 ...
29/03/202419:11:49.595-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:11:49.924-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X29/03/202419:11:49.955-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.39058029/03/202419:11:49.955-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:09:37.196634vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedapply_flowin0:09:37.587213 ...
Not sure if this is because of recent sharrow updates, or because of recent updates in the way unused columns are dropped.
@jpn--
jpn--
commented
Apr 11, 2024

Copy link


Copy Markdown

Member
I am not sure what is happening that has cause the code to run slower for @i-am-sijia , I am unable to replicate the problem.
I have run the full scale model on my laptop through school location choice (the first problematic model shown above) and got these results:
sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow as of 3/28)

activitysim commit e5d9878 (this PR as of 3/28)
[08:20.92] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:03.380284 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed apply_flow in 0:01:03.416361 
sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7, in use 3/20)

activitysim commit 94c4db8 (main branch as of 3/20)
[08:11.76] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:00.648809 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed apply_flow in 0:01:00.678720 
@jpn--
jpn--
commented
Apr 16, 2024

Copy link


Copy Markdown

Member
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@i-am-sijia
i-am-sijia
commented
Apr 16, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I reran the reported "3/20" and "3/29" runs, see Test 1 and Test 2 below. Last time they were run on different machines. This time they are run on the same machine. Good news is that this time I'm not seeing a huge runtime difference. Test 3 uses the same commits as one of Jeff's test, apparently the run time on Win is longer than Mac.
Test 1 (Rerunning the "3/20" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit 6d817be (main, as of 3/18)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
16/04/202413:41:49.585-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF216/04/202413:41:49.587-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.11417416/04/202413:41:49.587-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:07:21.590254school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedapply_flowin0:07:21.704428
Test 2 (Rerunning the "3/29" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
12/04/202420:47:53.356-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF212/04/202420:47:53.356-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07812812/04/202420:47:53.356-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:08:50.832949school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedapply_flowin0:08:50.911078
Test 3 (use the latest sharrow)
Sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow v2.8.2 as of 3/28)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
15/04/202413:55:03.917-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF215/04/202413:55:03.917-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.08288215/04/202413:55:03.917-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:06:49.217218school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedapply_flowin0:06:49.300100
@dhensledhensle
mentioned this pull request
Apr 16, 2024
 Closed

@i-am-sijia
i-am-sijia
commented
Apr 17, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@jpn-- , I made the key code changes needed for turning this feature on and off for individual components, I haven't pushed them. It's essentially passing a Boolean drop_unused_columns from component model settings into simple_simulate, interaction_simulate, interaction_sample etc. I modified almost all component .py (like auto_ownership.py) to feed in this setting.
Then I paused and read your latest comment on #824. Since you are generalizing the sharrow_settings to compute_settings and I checked that they are consumed at the same place where drop_unused_columns are required, it may make sense to include drop_unused_columns in compute_settings, or I can create a sibling of compute_settings to host drop_unused_columns and any other settings that might be created in the future. Thoughts?
@jpn--
jpn--
commented
Apr 18, 2024

Copy link


Copy Markdown

Member
it may make sense to include drop_unused_columns in compute_settings
I like this idea. Let's finish the review/merge of #824 and then it should be easy to update this PR and put this straight into the compute_settings structure.
@jpn--jpn--
mentioned this pull request
Apr 18, 2024
 Merged

@i-am-sijia
i-am-sijia
commented
Apr 21, 2024

Copy link


Copy Markdown

MemberAuthor
@jpn-- , I implemented drop_unused_columns in compute_settings, cherry-picked the parking location changes in #849 and the protect additional variables in https://github.com/camsys/activitysim/commit/574ee0eb444e0e97c5b2ac8c9215ed9df55bd53d, so that everything is included in this PR for efficient review and merge.
I tested setting compute_settings.drop_unused_columns to false in auto ownership, non-mandatory tour destination and scheduling, school escorting, and trip mode choice. It worked as expected.
@jpn--
jpn--
merged commit 239f415
into
ActivitySim:mainApr 21, 2024
@dhensledhensle
mentioned this pull request
Jan 28, 2025
 Merged

11 tasks
@i-am-sijia
i-am-sijia
deleted the

auto-drop-vars

branch
October 27, 2025 15:45
@jpn--jpn--
mentioned this pull request
Aug 13, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jpn--jpn--jpn-- approved these changes

Assignees

@i-am-sijiai-am-sijia

Labels


None yet


Projects

Phase 9 Work

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@i-am-sijia@jpn--

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Automatically drop unneeded columns in choosers table - #833
Merged
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Apr 21, 2024
Merged
Automatically drop unneeded columns in choosers table#833
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Conversation
@i-am-sijia
@i-am-sijiai-am-sijia
commented
Mar 18, 2024•

edited

Loading

Copy link


Copy Markdown

Member
This PR addresses #792
  •  Drop unused columns automatically based on UECs
  •  Pick up variables used/defined in source code, e.g. custom chooser
  •  Skip dropping when running household debug tracing
  •  Skip dropping when running estimation mode
  •  Pass all existing CI tests
  •  Run benchmarking runs
  •  Clean up codes, consolidate dup codes into util funtions
@i-am-sijia
i-am-sijia
commented
Mar 26, 2024

Copy link


Copy Markdown

MemberAuthor
Note that when dropping columns in pandas df, the memory actually goes up momentarily before it goes down. see discussion here pandas-dev/pandas#17092, this issue was "closed" but it did not get resolved.
Looking at my code changes last week, I realized I was also dropping the columns a bit too late for interaction simulate models. The columns should be dropped before interaction df is created. I then made those changes.
Below shows the 1-zone benchmarking runs with main branch (top) and with this PR (bottom). In the non-sharrow mode, the peak memory went down from 352 GB to 261 GB, a 25% reduction.
run time comp
Component-wise memory reduction, sorted by memory saving:
image
I'm still dealing with some crashes in the sharrow mode.
@jpn--
jpn--
commented
Mar 26, 2024

Copy link


Copy Markdown

Member
this issue was "closed" but it did not get resolved
If you read the thread closely, you'll note that that "this is not going to be solved in pandas 1." The behind-the-curtain memory management of pandas 1.x doesn't offer any way to drop unused columns without consuming extra RAM. This is alleviated by moving to pandas 2... after the transition, there are now ways to get the memory benefit. This is (part of) the reason I've worked this past week to try to get us pandas-2 compatible.
@i-am-sijia
i-am-sijia
commented
Apr 1, 2024

Copy link


Copy Markdown

MemberAuthor
I have some updates for the Sharrow runs. Below are three 1-zone benchmarking runs with Sharrow. The one on the top (3/20) used the main branch at 6d817be, the middle (3/28) and bottom (3/29) ones both used this PR with the latest code changes I made last week. There is a lot to discuss here.
  1. The peak memory of the entire model did not reduce much after dropping unused variables, although there were reductions in most components. With the main branch, the memory peak was Mandatory Tour Scheduling (156 GB), and a close runner-up was Write Trip Matrices (155 GB). After dropping unused variables, Mandatory Tour Scheduling reduced to 101 GB, but Write Trip Matrices reduced to 147 GB. So the peak memory usage only reduced from 156 GB to 147 GB.
  2. The two runs after dropping unused variables showed different memory usage patterns, even though they used the same source code. The one on 3/28 showed accumulation of memory in Trip Destination, while the one on 3/29 showed accumulation of memory in Mandatory Tour Scheduling (that caused School Escorting having a high mark to start with). This looks relevant to issue Double check Sharrow memory usage fix (#751) is merged and implemented #816. We have "memory leaks" that seem to show up sometimes but not all times.
  3. The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours. This could mean that dropping unused variables might have caused longer run time, even though I did not observe longer run time in the Non Sharrow runs reported week. In the Non Sharrow runs, in addition to a 25% peak memory reduction, dropping unused variable also reduced run time by 10 mins. One possibility is that dropping variables in the Sharrow mode has implications on run time that I have not realized. Another possibility is that the machine needs a reboot as things piled up that slowed it down (although this is a virtual machine with nothing else running).
run time comp sharrow

<style>
</style>

EventWSP Sharrow On 3-20 MainWSP Sharrow On 3-28 PR 833WSP Sharrow On 3-29 PR 833
Max Memory (GB)156.1146.8150.5
mandatory_tour_scheduling156.1101.8109.9
write_trip_matrices155.0146.7147.2
trip_destination132.8146.8130.4
school_escorting124.8114.3150.5
non_mandatory_tour_scheduling116.6106.788.6
atwork_subtour_scheduling89.343.142.0
workplace_location82.872.981.2
atwork_subtour_mode_choice79.640.337.9
trip_mode_choice66.660.261.0
stop_frequency64.262.265.1
non_mandatory_tour_destination61.767.964.8
vehicle_allocation60.199.188.2
write_data_dictionary59.748.348.5
trip_scheduling58.950.350.7
school_location58.957.963.6
trip_purpose_and_destination55.750.748.3
atwork_subtour_destination54.351.048.0
track_skim_usage53.241.541.8
write_tables52.343.644.0
finalizing48.641.441.7
tour_mode_choice_simulate47.952.945.4
joint_tour_destination47.248.049.8
non_mandatory_tour_frequency42.550.741.7
trip_purpose39.438.138.4
joint_tour_scheduling36.536.033.0
atwork_subtour_frequency30.830.229.3
vehicle_type_choice28.828.727.6
joint_tour_participation28.829.928.5
cdap_simulate24.824.025.5
joint_tour_frequency24.224.324.4
free_parking21.623.323.5
joint_tour_composition21.321.221.2
compute_disaggregate_accessibility20.723.121.2
initialize_households20.421.222.3
mandatory_tour_frequency18.523.023.7
compute_accessibility12.715.516.9
auto_ownership_simulate11.112.011.8
initialize_landuse7.114.011.4
initialize_proto_population6.46.46.4
input_checker1.91.61.6
preload_injectables0.30.30.3
na0.30.30.3
@i-am-sijia
i-am-sijia
marked this pull request as ready for review
April 2, 2024 19:05
@i-am-sijia
i-am-sijia
requested a review
from jpn--April 2, 2024 19:13
i-am-sijia
i-am-sijia
commented
Apr 3, 2024
View reviewed changes
Comment threadactivitysim/core/util.py
@i-am-sijia
i-am-sijia
commented
Apr 8, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Update RE:
The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours.
This seems to be because flow.load time increased.
For example, in the 3/20 run:
18/03/202419:27:38.294-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF218/03/202419:27:38.294-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07700018/03/202419:27:38.294-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:02:27.026987school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedapply_flowin0:02:27.103988 ...
18/03/202420:09:15.654-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:09:15.995-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X18/03/202420:09:16.193-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.55200118/03/202420:09:16.193-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:03:48.747827vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedapply_flowin0:03:49.299828 ...
In the 3/29 run:
29/03/202417:41:59.160-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF229/03/202417:41:59.160-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07813929/03/202417:41:59.160-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:11:32.047284school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedapply_flowin0:11:32.125423 ...
29/03/202419:11:49.595-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:11:49.924-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X29/03/202419:11:49.955-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.39058029/03/202419:11:49.955-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:09:37.196634vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedapply_flowin0:09:37.587213 ...
Not sure if this is because of recent sharrow updates, or because of recent updates in the way unused columns are dropped.
@jpn--
jpn--
commented
Apr 11, 2024

Copy link


Copy Markdown

Member
I am not sure what is happening that has cause the code to run slower for @i-am-sijia , I am unable to replicate the problem.
I have run the full scale model on my laptop through school location choice (the first problematic model shown above) and got these results:
sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow as of 3/28)

activitysim commit e5d9878 (this PR as of 3/28)
[08:20.92] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:03.380284 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed apply_flow in 0:01:03.416361 
sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7, in use 3/20)

activitysim commit 94c4db8 (main branch as of 3/20)
[08:11.76] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:00.648809 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed apply_flow in 0:01:00.678720 
@jpn--
jpn--
commented
Apr 16, 2024

Copy link


Copy Markdown

Member
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@i-am-sijia
i-am-sijia
commented
Apr 16, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I reran the reported "3/20" and "3/29" runs, see Test 1 and Test 2 below. Last time they were run on different machines. This time they are run on the same machine. Good news is that this time I'm not seeing a huge runtime difference. Test 3 uses the same commits as one of Jeff's test, apparently the run time on Win is longer than Mac.
Test 1 (Rerunning the "3/20" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit 6d817be (main, as of 3/18)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
16/04/202413:41:49.585-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF216/04/202413:41:49.587-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.11417416/04/202413:41:49.587-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:07:21.590254school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedapply_flowin0:07:21.704428
Test 2 (Rerunning the "3/29" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
12/04/202420:47:53.356-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF212/04/202420:47:53.356-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07812812/04/202420:47:53.356-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:08:50.832949school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedapply_flowin0:08:50.911078
Test 3 (use the latest sharrow)
Sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow v2.8.2 as of 3/28)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
15/04/202413:55:03.917-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF215/04/202413:55:03.917-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.08288215/04/202413:55:03.917-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:06:49.217218school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedapply_flowin0:06:49.300100
@dhensledhensle
mentioned this pull request
Apr 16, 2024
 Closed

@i-am-sijia
i-am-sijia
commented
Apr 17, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@jpn-- , I made the key code changes needed for turning this feature on and off for individual components, I haven't pushed them. It's essentially passing a Boolean drop_unused_columns from component model settings into simple_simulate, interaction_simulate, interaction_sample etc. I modified almost all component .py (like auto_ownership.py) to feed in this setting.
Then I paused and read your latest comment on #824. Since you are generalizing the sharrow_settings to compute_settings and I checked that they are consumed at the same place where drop_unused_columns are required, it may make sense to include drop_unused_columns in compute_settings, or I can create a sibling of compute_settings to host drop_unused_columns and any other settings that might be created in the future. Thoughts?
@jpn--
jpn--
commented
Apr 18, 2024

Copy link


Copy Markdown

Member
it may make sense to include drop_unused_columns in compute_settings
I like this idea. Let's finish the review/merge of #824 and then it should be easy to update this PR and put this straight into the compute_settings structure.
@jpn--jpn--
mentioned this pull request
Apr 18, 2024
 Merged

@i-am-sijia
i-am-sijia
commented
Apr 21, 2024

Copy link


Copy Markdown

MemberAuthor
@jpn-- , I implemented drop_unused_columns in compute_settings, cherry-picked the parking location changes in #849 and the protect additional variables in https://github.com/camsys/activitysim/commit/574ee0eb444e0e97c5b2ac8c9215ed9df55bd53d, so that everything is included in this PR for efficient review and merge.
I tested setting compute_settings.drop_unused_columns to false in auto ownership, non-mandatory tour destination and scheduling, school escorting, and trip mode choice. It worked as expected.
@jpn--
jpn--
merged commit 239f415
into
ActivitySim:mainApr 21, 2024
@dhensledhensle
mentioned this pull request
Jan 28, 2025
 Merged

11 tasks
@i-am-sijia
i-am-sijia
deleted the

auto-drop-vars

branch
October 27, 2025 15:45
@jpn--jpn--
mentioned this pull request
Aug 13, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jpn--jpn--jpn-- approved these changes

Assignees

@i-am-sijiai-am-sijia

Labels


None yet


Projects

Phase 9 Work

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@i-am-sijia@jpn--

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Automatically drop unneeded columns in choosers table - #833
Merged
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Apr 21, 2024
Merged
Automatically drop unneeded columns in choosers table#833
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Conversation
@i-am-sijia
@i-am-sijiai-am-sijia
commented
Mar 18, 2024•

edited

Loading

Copy link


Copy Markdown

Member
This PR addresses #792
  •  Drop unused columns automatically based on UECs
  •  Pick up variables used/defined in source code, e.g. custom chooser
  •  Skip dropping when running household debug tracing
  •  Skip dropping when running estimation mode
  •  Pass all existing CI tests
  •  Run benchmarking runs
  •  Clean up codes, consolidate dup codes into util funtions
@i-am-sijia
i-am-sijia
commented
Mar 26, 2024

Copy link


Copy Markdown

MemberAuthor
Note that when dropping columns in pandas df, the memory actually goes up momentarily before it goes down. see discussion here pandas-dev/pandas#17092, this issue was "closed" but it did not get resolved.
Looking at my code changes last week, I realized I was also dropping the columns a bit too late for interaction simulate models. The columns should be dropped before interaction df is created. I then made those changes.
Below shows the 1-zone benchmarking runs with main branch (top) and with this PR (bottom). In the non-sharrow mode, the peak memory went down from 352 GB to 261 GB, a 25% reduction.
run time comp
Component-wise memory reduction, sorted by memory saving:
image
I'm still dealing with some crashes in the sharrow mode.
@jpn--
jpn--
commented
Mar 26, 2024

Copy link


Copy Markdown

Member
this issue was "closed" but it did not get resolved
If you read the thread closely, you'll note that that "this is not going to be solved in pandas 1." The behind-the-curtain memory management of pandas 1.x doesn't offer any way to drop unused columns without consuming extra RAM. This is alleviated by moving to pandas 2... after the transition, there are now ways to get the memory benefit. This is (part of) the reason I've worked this past week to try to get us pandas-2 compatible.
@i-am-sijia
i-am-sijia
commented
Apr 1, 2024

Copy link


Copy Markdown

MemberAuthor
I have some updates for the Sharrow runs. Below are three 1-zone benchmarking runs with Sharrow. The one on the top (3/20) used the main branch at 6d817be, the middle (3/28) and bottom (3/29) ones both used this PR with the latest code changes I made last week. There is a lot to discuss here.
  1. The peak memory of the entire model did not reduce much after dropping unused variables, although there were reductions in most components. With the main branch, the memory peak was Mandatory Tour Scheduling (156 GB), and a close runner-up was Write Trip Matrices (155 GB). After dropping unused variables, Mandatory Tour Scheduling reduced to 101 GB, but Write Trip Matrices reduced to 147 GB. So the peak memory usage only reduced from 156 GB to 147 GB.
  2. The two runs after dropping unused variables showed different memory usage patterns, even though they used the same source code. The one on 3/28 showed accumulation of memory in Trip Destination, while the one on 3/29 showed accumulation of memory in Mandatory Tour Scheduling (that caused School Escorting having a high mark to start with). This looks relevant to issue Double check Sharrow memory usage fix (#751) is merged and implemented #816. We have "memory leaks" that seem to show up sometimes but not all times.
  3. The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours. This could mean that dropping unused variables might have caused longer run time, even though I did not observe longer run time in the Non Sharrow runs reported week. In the Non Sharrow runs, in addition to a 25% peak memory reduction, dropping unused variable also reduced run time by 10 mins. One possibility is that dropping variables in the Sharrow mode has implications on run time that I have not realized. Another possibility is that the machine needs a reboot as things piled up that slowed it down (although this is a virtual machine with nothing else running).
run time comp sharrow

<style>
</style>

EventWSP Sharrow On 3-20 MainWSP Sharrow On 3-28 PR 833WSP Sharrow On 3-29 PR 833
Max Memory (GB)156.1146.8150.5
mandatory_tour_scheduling156.1101.8109.9
write_trip_matrices155.0146.7147.2
trip_destination132.8146.8130.4
school_escorting124.8114.3150.5
non_mandatory_tour_scheduling116.6106.788.6
atwork_subtour_scheduling89.343.142.0
workplace_location82.872.981.2
atwork_subtour_mode_choice79.640.337.9
trip_mode_choice66.660.261.0
stop_frequency64.262.265.1
non_mandatory_tour_destination61.767.964.8
vehicle_allocation60.199.188.2
write_data_dictionary59.748.348.5
trip_scheduling58.950.350.7
school_location58.957.963.6
trip_purpose_and_destination55.750.748.3
atwork_subtour_destination54.351.048.0
track_skim_usage53.241.541.8
write_tables52.343.644.0
finalizing48.641.441.7
tour_mode_choice_simulate47.952.945.4
joint_tour_destination47.248.049.8
non_mandatory_tour_frequency42.550.741.7
trip_purpose39.438.138.4
joint_tour_scheduling36.536.033.0
atwork_subtour_frequency30.830.229.3
vehicle_type_choice28.828.727.6
joint_tour_participation28.829.928.5
cdap_simulate24.824.025.5
joint_tour_frequency24.224.324.4
free_parking21.623.323.5
joint_tour_composition21.321.221.2
compute_disaggregate_accessibility20.723.121.2
initialize_households20.421.222.3
mandatory_tour_frequency18.523.023.7
compute_accessibility12.715.516.9
auto_ownership_simulate11.112.011.8
initialize_landuse7.114.011.4
initialize_proto_population6.46.46.4
input_checker1.91.61.6
preload_injectables0.30.30.3
na0.30.30.3
@i-am-sijia
i-am-sijia
marked this pull request as ready for review
April 2, 2024 19:05
@i-am-sijia
i-am-sijia
requested a review
from jpn--April 2, 2024 19:13
i-am-sijia
i-am-sijia
commented
Apr 3, 2024
View reviewed changes
Comment threadactivitysim/core/util.py
@i-am-sijia
i-am-sijia
commented
Apr 8, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Update RE:
The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours.
This seems to be because flow.load time increased.
For example, in the 3/20 run:
18/03/202419:27:38.294-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF218/03/202419:27:38.294-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07700018/03/202419:27:38.294-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:02:27.026987school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedapply_flowin0:02:27.103988 ...
18/03/202420:09:15.654-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:09:15.995-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X18/03/202420:09:16.193-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.55200118/03/202420:09:16.193-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:03:48.747827vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedapply_flowin0:03:49.299828 ...
In the 3/29 run:
29/03/202417:41:59.160-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF229/03/202417:41:59.160-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07813929/03/202417:41:59.160-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:11:32.047284school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedapply_flowin0:11:32.125423 ...
29/03/202419:11:49.595-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:11:49.924-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X29/03/202419:11:49.955-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.39058029/03/202419:11:49.955-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:09:37.196634vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedapply_flowin0:09:37.587213 ...
Not sure if this is because of recent sharrow updates, or because of recent updates in the way unused columns are dropped.
@jpn--
jpn--
commented
Apr 11, 2024

Copy link


Copy Markdown

Member
I am not sure what is happening that has cause the code to run slower for @i-am-sijia , I am unable to replicate the problem.
I have run the full scale model on my laptop through school location choice (the first problematic model shown above) and got these results:
sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow as of 3/28)

activitysim commit e5d9878 (this PR as of 3/28)
[08:20.92] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:03.380284 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed apply_flow in 0:01:03.416361 
sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7, in use 3/20)

activitysim commit 94c4db8 (main branch as of 3/20)
[08:11.76] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:00.648809 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed apply_flow in 0:01:00.678720 
@jpn--
jpn--
commented
Apr 16, 2024

Copy link


Copy Markdown

Member
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@i-am-sijia
i-am-sijia
commented
Apr 16, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I reran the reported "3/20" and "3/29" runs, see Test 1 and Test 2 below. Last time they were run on different machines. This time they are run on the same machine. Good news is that this time I'm not seeing a huge runtime difference. Test 3 uses the same commits as one of Jeff's test, apparently the run time on Win is longer than Mac.
Test 1 (Rerunning the "3/20" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit 6d817be (main, as of 3/18)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
16/04/202413:41:49.585-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF216/04/202413:41:49.587-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.11417416/04/202413:41:49.587-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:07:21.590254school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedapply_flowin0:07:21.704428
Test 2 (Rerunning the "3/29" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
12/04/202420:47:53.356-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF212/04/202420:47:53.356-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07812812/04/202420:47:53.356-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:08:50.832949school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedapply_flowin0:08:50.911078
Test 3 (use the latest sharrow)
Sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow v2.8.2 as of 3/28)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
15/04/202413:55:03.917-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF215/04/202413:55:03.917-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.08288215/04/202413:55:03.917-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:06:49.217218school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedapply_flowin0:06:49.300100
@dhensledhensle
mentioned this pull request
Apr 16, 2024
 Closed

@i-am-sijia
i-am-sijia
commented
Apr 17, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@jpn-- , I made the key code changes needed for turning this feature on and off for individual components, I haven't pushed them. It's essentially passing a Boolean drop_unused_columns from component model settings into simple_simulate, interaction_simulate, interaction_sample etc. I modified almost all component .py (like auto_ownership.py) to feed in this setting.
Then I paused and read your latest comment on #824. Since you are generalizing the sharrow_settings to compute_settings and I checked that they are consumed at the same place where drop_unused_columns are required, it may make sense to include drop_unused_columns in compute_settings, or I can create a sibling of compute_settings to host drop_unused_columns and any other settings that might be created in the future. Thoughts?
@jpn--
jpn--
commented
Apr 18, 2024

Copy link


Copy Markdown

Member
it may make sense to include drop_unused_columns in compute_settings
I like this idea. Let's finish the review/merge of #824 and then it should be easy to update this PR and put this straight into the compute_settings structure.
@jpn--jpn--
mentioned this pull request
Apr 18, 2024
 Merged

@i-am-sijia
i-am-sijia
commented
Apr 21, 2024

Copy link


Copy Markdown

MemberAuthor
@jpn-- , I implemented drop_unused_columns in compute_settings, cherry-picked the parking location changes in #849 and the protect additional variables in https://github.com/camsys/activitysim/commit/574ee0eb444e0e97c5b2ac8c9215ed9df55bd53d, so that everything is included in this PR for efficient review and merge.
I tested setting compute_settings.drop_unused_columns to false in auto ownership, non-mandatory tour destination and scheduling, school escorting, and trip mode choice. It worked as expected.
@jpn--
jpn--
merged commit 239f415
into
ActivitySim:mainApr 21, 2024
@dhensledhensle
mentioned this pull request
Jan 28, 2025
 Merged

11 tasks
@i-am-sijia
i-am-sijia
deleted the

auto-drop-vars

branch
October 27, 2025 15:45
@jpn--jpn--
mentioned this pull request
Aug 13, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jpn--jpn--jpn-- approved these changes

Assignees

@i-am-sijiai-am-sijia

Labels


None yet


Projects

Phase 9 Work

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@i-am-sijia@jpn--

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Automatically drop unneeded columns in choosers table - #833
Merged
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Apr 21, 2024
Merged
Automatically drop unneeded columns in choosers table#833
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Conversation
@i-am-sijia
@i-am-sijiai-am-sijia
commented
Mar 18, 2024•

edited

Loading

Copy link


Copy Markdown

Member
This PR addresses #792
  •  Drop unused columns automatically based on UECs
  •  Pick up variables used/defined in source code, e.g. custom chooser
  •  Skip dropping when running household debug tracing
  •  Skip dropping when running estimation mode
  •  Pass all existing CI tests
  •  Run benchmarking runs
  •  Clean up codes, consolidate dup codes into util funtions
@i-am-sijia
i-am-sijia
commented
Mar 26, 2024

Copy link


Copy Markdown

MemberAuthor
Note that when dropping columns in pandas df, the memory actually goes up momentarily before it goes down. see discussion here pandas-dev/pandas#17092, this issue was "closed" but it did not get resolved.
Looking at my code changes last week, I realized I was also dropping the columns a bit too late for interaction simulate models. The columns should be dropped before interaction df is created. I then made those changes.
Below shows the 1-zone benchmarking runs with main branch (top) and with this PR (bottom). In the non-sharrow mode, the peak memory went down from 352 GB to 261 GB, a 25% reduction.
run time comp
Component-wise memory reduction, sorted by memory saving:
image
I'm still dealing with some crashes in the sharrow mode.
@jpn--
jpn--
commented
Mar 26, 2024

Copy link


Copy Markdown

Member
this issue was "closed" but it did not get resolved
If you read the thread closely, you'll note that that "this is not going to be solved in pandas 1." The behind-the-curtain memory management of pandas 1.x doesn't offer any way to drop unused columns without consuming extra RAM. This is alleviated by moving to pandas 2... after the transition, there are now ways to get the memory benefit. This is (part of) the reason I've worked this past week to try to get us pandas-2 compatible.
@i-am-sijia
i-am-sijia
commented
Apr 1, 2024

Copy link


Copy Markdown

MemberAuthor
I have some updates for the Sharrow runs. Below are three 1-zone benchmarking runs with Sharrow. The one on the top (3/20) used the main branch at 6d817be, the middle (3/28) and bottom (3/29) ones both used this PR with the latest code changes I made last week. There is a lot to discuss here.
  1. The peak memory of the entire model did not reduce much after dropping unused variables, although there were reductions in most components. With the main branch, the memory peak was Mandatory Tour Scheduling (156 GB), and a close runner-up was Write Trip Matrices (155 GB). After dropping unused variables, Mandatory Tour Scheduling reduced to 101 GB, but Write Trip Matrices reduced to 147 GB. So the peak memory usage only reduced from 156 GB to 147 GB.
  2. The two runs after dropping unused variables showed different memory usage patterns, even though they used the same source code. The one on 3/28 showed accumulation of memory in Trip Destination, while the one on 3/29 showed accumulation of memory in Mandatory Tour Scheduling (that caused School Escorting having a high mark to start with). This looks relevant to issue Double check Sharrow memory usage fix (#751) is merged and implemented #816. We have "memory leaks" that seem to show up sometimes but not all times.
  3. The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours. This could mean that dropping unused variables might have caused longer run time, even though I did not observe longer run time in the Non Sharrow runs reported week. In the Non Sharrow runs, in addition to a 25% peak memory reduction, dropping unused variable also reduced run time by 10 mins. One possibility is that dropping variables in the Sharrow mode has implications on run time that I have not realized. Another possibility is that the machine needs a reboot as things piled up that slowed it down (although this is a virtual machine with nothing else running).
run time comp sharrow

<style>
</style>

EventWSP Sharrow On 3-20 MainWSP Sharrow On 3-28 PR 833WSP Sharrow On 3-29 PR 833
Max Memory (GB)156.1146.8150.5
mandatory_tour_scheduling156.1101.8109.9
write_trip_matrices155.0146.7147.2
trip_destination132.8146.8130.4
school_escorting124.8114.3150.5
non_mandatory_tour_scheduling116.6106.788.6
atwork_subtour_scheduling89.343.142.0
workplace_location82.872.981.2
atwork_subtour_mode_choice79.640.337.9
trip_mode_choice66.660.261.0
stop_frequency64.262.265.1
non_mandatory_tour_destination61.767.964.8
vehicle_allocation60.199.188.2
write_data_dictionary59.748.348.5
trip_scheduling58.950.350.7
school_location58.957.963.6
trip_purpose_and_destination55.750.748.3
atwork_subtour_destination54.351.048.0
track_skim_usage53.241.541.8
write_tables52.343.644.0
finalizing48.641.441.7
tour_mode_choice_simulate47.952.945.4
joint_tour_destination47.248.049.8
non_mandatory_tour_frequency42.550.741.7
trip_purpose39.438.138.4
joint_tour_scheduling36.536.033.0
atwork_subtour_frequency30.830.229.3
vehicle_type_choice28.828.727.6
joint_tour_participation28.829.928.5
cdap_simulate24.824.025.5
joint_tour_frequency24.224.324.4
free_parking21.623.323.5
joint_tour_composition21.321.221.2
compute_disaggregate_accessibility20.723.121.2
initialize_households20.421.222.3
mandatory_tour_frequency18.523.023.7
compute_accessibility12.715.516.9
auto_ownership_simulate11.112.011.8
initialize_landuse7.114.011.4
initialize_proto_population6.46.46.4
input_checker1.91.61.6
preload_injectables0.30.30.3
na0.30.30.3
@i-am-sijia
i-am-sijia
marked this pull request as ready for review
April 2, 2024 19:05
@i-am-sijia
i-am-sijia
requested a review
from jpn--April 2, 2024 19:13
i-am-sijia
i-am-sijia
commented
Apr 3, 2024
View reviewed changes
Comment threadactivitysim/core/util.py
@i-am-sijia
i-am-sijia
commented
Apr 8, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Update RE:
The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours.
This seems to be because flow.load time increased.
For example, in the 3/20 run:
18/03/202419:27:38.294-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF218/03/202419:27:38.294-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07700018/03/202419:27:38.294-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:02:27.026987school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedapply_flowin0:02:27.103988 ...
18/03/202420:09:15.654-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:09:15.995-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X18/03/202420:09:16.193-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.55200118/03/202420:09:16.193-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:03:48.747827vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedapply_flowin0:03:49.299828 ...
In the 3/29 run:
29/03/202417:41:59.160-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF229/03/202417:41:59.160-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07813929/03/202417:41:59.160-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:11:32.047284school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedapply_flowin0:11:32.125423 ...
29/03/202419:11:49.595-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:11:49.924-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X29/03/202419:11:49.955-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.39058029/03/202419:11:49.955-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:09:37.196634vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedapply_flowin0:09:37.587213 ...
Not sure if this is because of recent sharrow updates, or because of recent updates in the way unused columns are dropped.
@jpn--
jpn--
commented
Apr 11, 2024

Copy link


Copy Markdown

Member
I am not sure what is happening that has cause the code to run slower for @i-am-sijia , I am unable to replicate the problem.
I have run the full scale model on my laptop through school location choice (the first problematic model shown above) and got these results:
sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow as of 3/28)

activitysim commit e5d9878 (this PR as of 3/28)
[08:20.92] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:03.380284 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed apply_flow in 0:01:03.416361 
sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7, in use 3/20)

activitysim commit 94c4db8 (main branch as of 3/20)
[08:11.76] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:00.648809 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed apply_flow in 0:01:00.678720 
@jpn--
jpn--
commented
Apr 16, 2024

Copy link


Copy Markdown

Member
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@i-am-sijia
i-am-sijia
commented
Apr 16, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I reran the reported "3/20" and "3/29" runs, see Test 1 and Test 2 below. Last time they were run on different machines. This time they are run on the same machine. Good news is that this time I'm not seeing a huge runtime difference. Test 3 uses the same commits as one of Jeff's test, apparently the run time on Win is longer than Mac.
Test 1 (Rerunning the "3/20" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit 6d817be (main, as of 3/18)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
16/04/202413:41:49.585-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF216/04/202413:41:49.587-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.11417416/04/202413:41:49.587-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:07:21.590254school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedapply_flowin0:07:21.704428
Test 2 (Rerunning the "3/29" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
12/04/202420:47:53.356-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF212/04/202420:47:53.356-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07812812/04/202420:47:53.356-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:08:50.832949school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedapply_flowin0:08:50.911078
Test 3 (use the latest sharrow)
Sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow v2.8.2 as of 3/28)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
15/04/202413:55:03.917-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF215/04/202413:55:03.917-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.08288215/04/202413:55:03.917-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:06:49.217218school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedapply_flowin0:06:49.300100
@dhensledhensle
mentioned this pull request
Apr 16, 2024
 Closed

@i-am-sijia
i-am-sijia
commented
Apr 17, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@jpn-- , I made the key code changes needed for turning this feature on and off for individual components, I haven't pushed them. It's essentially passing a Boolean drop_unused_columns from component model settings into simple_simulate, interaction_simulate, interaction_sample etc. I modified almost all component .py (like auto_ownership.py) to feed in this setting.
Then I paused and read your latest comment on #824. Since you are generalizing the sharrow_settings to compute_settings and I checked that they are consumed at the same place where drop_unused_columns are required, it may make sense to include drop_unused_columns in compute_settings, or I can create a sibling of compute_settings to host drop_unused_columns and any other settings that might be created in the future. Thoughts?
@jpn--
jpn--
commented
Apr 18, 2024

Copy link


Copy Markdown

Member
it may make sense to include drop_unused_columns in compute_settings
I like this idea. Let's finish the review/merge of #824 and then it should be easy to update this PR and put this straight into the compute_settings structure.
@jpn--jpn--
mentioned this pull request
Apr 18, 2024
 Merged

@i-am-sijia
i-am-sijia
commented
Apr 21, 2024

Copy link


Copy Markdown

MemberAuthor
@jpn-- , I implemented drop_unused_columns in compute_settings, cherry-picked the parking location changes in #849 and the protect additional variables in https://github.com/camsys/activitysim/commit/574ee0eb444e0e97c5b2ac8c9215ed9df55bd53d, so that everything is included in this PR for efficient review and merge.
I tested setting compute_settings.drop_unused_columns to false in auto ownership, non-mandatory tour destination and scheduling, school escorting, and trip mode choice. It worked as expected.
@jpn--
jpn--
merged commit 239f415
into
ActivitySim:mainApr 21, 2024
@dhensledhensle
mentioned this pull request
Jan 28, 2025
 Merged

11 tasks
@i-am-sijia
i-am-sijia
deleted the

auto-drop-vars

branch
October 27, 2025 15:45
@jpn--jpn--
mentioned this pull request
Aug 13, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jpn--jpn--jpn-- approved these changes

Assignees

@i-am-sijiai-am-sijia

Labels


None yet


Projects

Phase 9 Work

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@i-am-sijia@jpn--

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Automatically drop unneeded columns in choosers table - #833
Merged
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Apr 21, 2024
Merged
Automatically drop unneeded columns in choosers table#833
jpn-- merged 19 commits into
ActivitySim:mainfrom 
wsp-sag:auto-drop-vars
Conversation
@i-am-sijia
@i-am-sijiai-am-sijia
commented
Mar 18, 2024•

edited

Loading

Copy link


Copy Markdown

Member
This PR addresses #792
  •  Drop unused columns automatically based on UECs
  •  Pick up variables used/defined in source code, e.g. custom chooser
  •  Skip dropping when running household debug tracing
  •  Skip dropping when running estimation mode
  •  Pass all existing CI tests
  •  Run benchmarking runs
  •  Clean up codes, consolidate dup codes into util funtions
@i-am-sijia
i-am-sijia
commented
Mar 26, 2024

Copy link


Copy Markdown

MemberAuthor
Note that when dropping columns in pandas df, the memory actually goes up momentarily before it goes down. see discussion here pandas-dev/pandas#17092, this issue was "closed" but it did not get resolved.
Looking at my code changes last week, I realized I was also dropping the columns a bit too late for interaction simulate models. The columns should be dropped before interaction df is created. I then made those changes.
Below shows the 1-zone benchmarking runs with main branch (top) and with this PR (bottom). In the non-sharrow mode, the peak memory went down from 352 GB to 261 GB, a 25% reduction.
run time comp
Component-wise memory reduction, sorted by memory saving:
image
I'm still dealing with some crashes in the sharrow mode.
@jpn--
jpn--
commented
Mar 26, 2024

Copy link


Copy Markdown

Member
this issue was "closed" but it did not get resolved
If you read the thread closely, you'll note that that "this is not going to be solved in pandas 1." The behind-the-curtain memory management of pandas 1.x doesn't offer any way to drop unused columns without consuming extra RAM. This is alleviated by moving to pandas 2... after the transition, there are now ways to get the memory benefit. This is (part of) the reason I've worked this past week to try to get us pandas-2 compatible.
@i-am-sijia
i-am-sijia
commented
Apr 1, 2024

Copy link


Copy Markdown

MemberAuthor
I have some updates for the Sharrow runs. Below are three 1-zone benchmarking runs with Sharrow. The one on the top (3/20) used the main branch at 6d817be, the middle (3/28) and bottom (3/29) ones both used this PR with the latest code changes I made last week. There is a lot to discuss here.
  1. The peak memory of the entire model did not reduce much after dropping unused variables, although there were reductions in most components. With the main branch, the memory peak was Mandatory Tour Scheduling (156 GB), and a close runner-up was Write Trip Matrices (155 GB). After dropping unused variables, Mandatory Tour Scheduling reduced to 101 GB, but Write Trip Matrices reduced to 147 GB. So the peak memory usage only reduced from 156 GB to 147 GB.
  2. The two runs after dropping unused variables showed different memory usage patterns, even though they used the same source code. The one on 3/28 showed accumulation of memory in Trip Destination, while the one on 3/29 showed accumulation of memory in Mandatory Tour Scheduling (that caused School Escorting having a high mark to start with). This looks relevant to issue Double check Sharrow memory usage fix (#751) is merged and implemented #816. We have "memory leaks" that seem to show up sometimes but not all times.
  3. The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours. This could mean that dropping unused variables might have caused longer run time, even though I did not observe longer run time in the Non Sharrow runs reported week. In the Non Sharrow runs, in addition to a 25% peak memory reduction, dropping unused variable also reduced run time by 10 mins. One possibility is that dropping variables in the Sharrow mode has implications on run time that I have not realized. Another possibility is that the machine needs a reboot as things piled up that slowed it down (although this is a virtual machine with nothing else running).
run time comp sharrow

<style>
</style>

EventWSP Sharrow On 3-20 MainWSP Sharrow On 3-28 PR 833WSP Sharrow On 3-29 PR 833
Max Memory (GB)156.1146.8150.5
mandatory_tour_scheduling156.1101.8109.9
write_trip_matrices155.0146.7147.2
trip_destination132.8146.8130.4
school_escorting124.8114.3150.5
non_mandatory_tour_scheduling116.6106.788.6
atwork_subtour_scheduling89.343.142.0
workplace_location82.872.981.2
atwork_subtour_mode_choice79.640.337.9
trip_mode_choice66.660.261.0
stop_frequency64.262.265.1
non_mandatory_tour_destination61.767.964.8
vehicle_allocation60.199.188.2
write_data_dictionary59.748.348.5
trip_scheduling58.950.350.7
school_location58.957.963.6
trip_purpose_and_destination55.750.748.3
atwork_subtour_destination54.351.048.0
track_skim_usage53.241.541.8
write_tables52.343.644.0
finalizing48.641.441.7
tour_mode_choice_simulate47.952.945.4
joint_tour_destination47.248.049.8
non_mandatory_tour_frequency42.550.741.7
trip_purpose39.438.138.4
joint_tour_scheduling36.536.033.0
atwork_subtour_frequency30.830.229.3
vehicle_type_choice28.828.727.6
joint_tour_participation28.829.928.5
cdap_simulate24.824.025.5
joint_tour_frequency24.224.324.4
free_parking21.623.323.5
joint_tour_composition21.321.221.2
compute_disaggregate_accessibility20.723.121.2
initialize_households20.421.222.3
mandatory_tour_frequency18.523.023.7
compute_accessibility12.715.516.9
auto_ownership_simulate11.112.011.8
initialize_landuse7.114.011.4
initialize_proto_population6.46.46.4
input_checker1.91.61.6
preload_injectables0.30.30.3
na0.30.30.3
@i-am-sijia
i-am-sijia
marked this pull request as ready for review
April 2, 2024 19:05
@i-am-sijia
i-am-sijia
requested a review
from jpn--April 2, 2024 19:13
i-am-sijia
i-am-sijia
commented
Apr 3, 2024
View reviewed changes
Comment threadactivitysim/core/util.py
@i-am-sijia
i-am-sijia
commented
Apr 8, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Update RE:
The total run time of the 3/28 and 3/29 runs are longer than the 3/20 run, 16.2 hours vs 13.4 hours.
This seems to be because flow.load time increased.
For example, in the 3/20 run:
18/03/202419:27:38.294-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF218/03/202419:27:38.294-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07700018/03/202419:27:38.294-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:02:27.026987school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums18/03/202419:30:05.321-INFO-activitysim.core.flow-completedapply_flowin0:02:27.103988 ...
18/03/202420:09:15.654-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:09:15.995-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X18/03/202420:09:16.193-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.55200118/03/202420:09:16.193-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:03:48.747827vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils18/03/202420:13:04.941-INFO-activitysim.core.flow-completedapply_flowin0:03:49.299828 ...
In the 3/29 run:
29/03/202417:41:59.160-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF229/03/202417:41:59.160-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07813929/03/202417:41:59.160-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:11:32.047284school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums29/03/202417:53:31.208-INFO-activitysim.core.flow-completedapply_flowin0:11:32.125423 ...
29/03/202419:11:49.595-INFO-activitysim.core.flow-settingupsharrowflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:11:49.924-INFO-sharrow-usingexistingflowcode736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X29/03/202419:11:49.955-INFO-activitysim.core.flow-completedsettingupflowvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utilsin0:00:00.39058029/03/202419:11:49.955-INFO-activitysim.core.flow-beginflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadvehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedflow_736GLJ6FZR4AF6ZTRFF5NB4F2D2MYM7X.loadin0:09:37.196634vehicle_type_choice.interaction_simulate.interaction_simulate.eval_interaction_utils29/03/202419:21:27.151-INFO-activitysim.core.flow-completedapply_flowin0:09:37.587213 ...
Not sure if this is because of recent sharrow updates, or because of recent updates in the way unused columns are dropped.
@jpn--
jpn--
commented
Apr 11, 2024

Copy link


Copy Markdown

Member
I am not sure what is happening that has cause the code to run slower for @i-am-sijia , I am unable to replicate the problem.
I have run the full scale model on my laptop through school location choice (the first problematic model shown above) and got these results:
sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow as of 3/28)

activitysim commit e5d9878 (this PR as of 3/28)
[08:20.92] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:03.380284 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:24.30] INFO: completed apply_flow in 0:01:03.416361 
sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7, in use 3/20)

activitysim commit 94c4db8 (main branch as of 3/20)
[08:11.76] INFO: begin flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed flow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.load in 0:01:00.648809 school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums
[09:12.41] INFO: completed apply_flow in 0:01:00.678720 
@jpn--
jpn--
commented
Apr 16, 2024

Copy link


Copy Markdown

Member
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@i-am-sijia
i-am-sijia
commented
Apr 16, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I reran the reported "3/20" and "3/29" runs, see Test 1 and Test 2 below. Last time they were run on different machines. This time they are run on the same machine. Good news is that this time I'm not seeing a huge runtime difference. Test 3 uses the same commits as one of Jeff's test, apparently the run time on Win is longer than Mac.
Test 1 (Rerunning the "3/20" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit 6d817be (main, as of 3/18)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
16/04/202413:41:49.585-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF216/04/202413:41:49.587-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.11417416/04/202413:41:49.587-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:07:21.590254school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums16/04/202413:49:11.177-INFO-activitysim.core.flow-completedapply_flowin0:07:21.704428
Test 2 (Rerunning the "3/29" run)
Sharrow commit c560b45f0e1cd5ccbb871319b5ccfbcf789debaf (sharrow v2.7)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
12/04/202420:47:53.356-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF212/04/202420:47:53.356-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.07812812/04/202420:47:53.356-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:08:50.832949school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums12/04/202420:56:44.189-INFO-activitysim.core.flow-completedapply_flowin0:08:50.911078
Test 3 (use the latest sharrow)
Sharrow commit 7fae9f060b77684b2f309c9ad2ad3d2bf3286239 (sharrow v2.8.2 as of 3/28)

ActivitySim commit e5d9878 (this PR as of 3/28)

activitysim-prototype-mtc commit 9ca05ca81a240cc74bfd764b1d859b55dbf597e0 (before school escorting sharrow changes)
15/04/202413:55:03.917-INFO-sharrow-flowexistsinlibrary: BJFBKLAOJGCDYZF4MKEWTG744CT3JZF215/04/202413:55:03.917-INFO-activitysim.core.flow-completedsettingupflowschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsumsin0:00:00.08288215/04/202413:55:03.917-INFO-activitysim.core.flow-beginflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadschool_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedflow_BJFBKLAOJGCDYZF4MKEWTG744CT3JZF2.loadin0:06:49.217218school_location.i1.logsums.gradeschool.compute_logsums.eval_nl_logsums15/04/202414:01:53.135-INFO-activitysim.core.flow-completedapply_flowin0:06:49.300100
@dhensledhensle
mentioned this pull request
Apr 16, 2024
 Closed

@i-am-sijia
i-am-sijia
commented
Apr 17, 2024•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
I recall we discussed that we would like to have a method to turn this feature off for individual components, either because it is interfering with something (e.g. tracing, estimation mode) or just because it's not working on a particular component (probably due to something weird in the spec). I do see that it turns itself off when tracing or estimation mode is used, which is OK, but I think we still want the capability to turn it off manually if desired.
@jpn-- , I made the key code changes needed for turning this feature on and off for individual components, I haven't pushed them. It's essentially passing a Boolean drop_unused_columns from component model settings into simple_simulate, interaction_simulate, interaction_sample etc. I modified almost all component .py (like auto_ownership.py) to feed in this setting.
Then I paused and read your latest comment on #824. Since you are generalizing the sharrow_settings to compute_settings and I checked that they are consumed at the same place where drop_unused_columns are required, it may make sense to include drop_unused_columns in compute_settings, or I can create a sibling of compute_settings to host drop_unused_columns and any other settings that might be created in the future. Thoughts?
@jpn--
jpn--
commented
Apr 18, 2024

Copy link


Copy Markdown

Member
it may make sense to include drop_unused_columns in compute_settings
I like this idea. Let's finish the review/merge of #824 and then it should be easy to update this PR and put this straight into the compute_settings structure.
@jpn--jpn--
mentioned this pull request
Apr 18, 2024
 Merged

@i-am-sijia
i-am-sijia
commented
Apr 21, 2024

Copy link


Copy Markdown

MemberAuthor
@jpn-- , I implemented drop_unused_columns in compute_settings, cherry-picked the parking location changes in #849 and the protect additional variables in https://github.com/camsys/activitysim/commit/574ee0eb444e0e97c5b2ac8c9215ed9df55bd53d, so that everything is included in this PR for efficient review and merge.
I tested setting compute_settings.drop_unused_columns to false in auto ownership, non-mandatory tour destination and scheduling, school escorting, and trip mode choice. It worked as expected.
@jpn--
jpn--
merged commit 239f415
into
ActivitySim:mainApr 21, 2024
@dhensledhensle
mentioned this pull request
Jan 28, 2025
 Merged

11 tasks
@i-am-sijia
i-am-sijia
deleted the

auto-drop-vars

branch
October 27, 2025 15:45
@jpn--jpn--
mentioned this pull request
Aug 13, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jpn--jpn--jpn-- approved these changes

Assignees

@i-am-sijiai-am-sijia

Labels


None yet


Projects

Phase 9 Work

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@i-am-sijia@jpn--