feat: chunkless operation

[jpn--]

It appears that I can't currently run a model with chunking just plain turned off unless it's in training mode, but training mode carries a lot of overhead monitoring memory usage. If I set to production mode without an available chunk cache history file it falls back to training mode.

This is my crack at creating a fourth chunk training mode: 'disabled'. (the first 3 are training, production, and adaptive). The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, so we can focus on performance assuming there is abundant RAM.

[jpn--]

@toliwaga it certainly would be helpful for you to review this, as you're the expert on chunk management.

[toliwaga]

I'm out of the office on vacation until the beginning of August

[coveralls]

Coverage increased (+8.5%) to 64.879% when pulling 37862e3 on jpn--:chunkless into a57efa1 on ActivitySim:master.

[jpn--]

Next steps: @esanchez01 and @stefancoe will test "disabled" mode to confirm it works correctly.

[esanchez01]

@jpn-- , After testing this on 100k households using the SANDAG 1 Zone setup (multiprocessed), there are still memory CSV files in the output/log directory as well as chunk logging across various logfiles. This does not conform with the behavior of the 'disabled' mode as described in the original post:

The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, ..

It seems like memory file creation is currently hard coded here in mem.py so changes may need to extend beyond just chunk.py. As for the chunk logging, it seems like empty logging is produced due to the placement of the 'disabled' chunk mode conditions in certain methods such as here or from functions that that don't set chunk mode conditions such as here.

You can find the configs and log files for this test run here.

[stefancoe]

I ran the model using 3 different modes using the same configuration of 20k households and mp turned off. The attached results show that the old code is fastest, 'disabled' is second but considerably slower and 'production' mode is the slowest.

chunkless_comparison.xlsx

[bstabler]

@esanchez01 to test again since this feature would be good to get working soon

[esanchez01]

I retested this feature with the new commits. There are still memory CSV files and chunk/memory logging that are produced. However, the extent of the memory tracking and logging appears to have been reduced. The run time for this new run was about 65 minutes compared to 140 minutes for the previous test, so it seems like unnecessary memory/chunk calls were further avoided.

You can find the configs and log files for the new test run here.

[bstabler]

@jpn-- please PR to develop when ready

[bstabler]

@jpn-- , actually, please move this to a branch within this repo so we can work on it here. Thanks.

[jpn--]

This is now a branch in the main repo, per @bstabler request. Closing this PR in favor of that.

[jpn--]

It appears that I can't currently run a model with chunking just plain turned off unless it's in training mode, but training mode carries a lot of overhead monitoring memory usage. If I set to production mode without an available chunk cache history file it falls back to training mode.

This is my crack at creating a fourth chunk training mode: 'disabled'. (the first 3 are training, production, and adaptive). The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, so we can focus on performance assuming there is abundant RAM.

[jpn--]

@toliwaga it certainly would be helpful for you to review this, as you're the expert on chunk management.

[toliwaga]

I'm out of the office on vacation until the beginning of August

[coveralls]

Coverage increased (+8.5%) to 64.879% when pulling 37862e3 on jpn--:chunkless into a57efa1 on ActivitySim:master.

[jpn--]

Next steps: @esanchez01 and @stefancoe will test "disabled" mode to confirm it works correctly.

[esanchez01]

@jpn-- , After testing this on 100k households using the SANDAG 1 Zone setup (multiprocessed), there are still memory CSV files in the output/log directory as well as chunk logging across various logfiles. This does not conform with the behavior of the 'disabled' mode as described in the original post:

The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, ..

It seems like memory file creation is currently hard coded here in mem.py so changes may need to extend beyond just chunk.py. As for the chunk logging, it seems like empty logging is produced due to the placement of the 'disabled' chunk mode conditions in certain methods such as here or from functions that that don't set chunk mode conditions such as here.

You can find the configs and log files for this test run here.

[stefancoe]

I ran the model using 3 different modes using the same configuration of 20k households and mp turned off. The attached results show that the old code is fastest, 'disabled' is second but considerably slower and 'production' mode is the slowest.

chunkless_comparison.xlsx

[bstabler]

@esanchez01 to test again since this feature would be good to get working soon

[esanchez01]

I retested this feature with the new commits. There are still memory CSV files and chunk/memory logging that are produced. However, the extent of the memory tracking and logging appears to have been reduced. The run time for this new run was about 65 minutes compared to 140 minutes for the previous test, so it seems like unnecessary memory/chunk calls were further avoided.

You can find the configs and log files for the new test run here.

[bstabler]

@jpn-- please PR to develop when ready

[bstabler]

@jpn-- , actually, please move this to a branch within this repo so we can work on it here. Thanks.

[jpn--]

This is now a branch in the main repo, per @bstabler request. Closing this PR in favor of that.

[jpn--]

It appears that I can't currently run a model with chunking just plain turned off unless it's in training mode, but training mode carries a lot of overhead monitoring memory usage. If I set to production mode without an available chunk cache history file it falls back to training mode.

This is my crack at creating a fourth chunk training mode: 'disabled'. (the first 3 are training, production, and adaptive). The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, so we can focus on performance assuming there is abundant RAM.

[jpn--]

@toliwaga it certainly would be helpful for you to review this, as you're the expert on chunk management.

[toliwaga]

I'm out of the office on vacation until the beginning of August

[coveralls]

Coverage increased (+8.5%) to 64.879% when pulling 37862e3 on jpn--:chunkless into a57efa1 on ActivitySim:master.

[jpn--]

Next steps: @esanchez01 and @stefancoe will test "disabled" mode to confirm it works correctly.

[esanchez01]

@jpn-- , After testing this on 100k households using the SANDAG 1 Zone setup (multiprocessed), there are still memory CSV files in the output/log directory as well as chunk logging across various logfiles. This does not conform with the behavior of the 'disabled' mode as described in the original post:

The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, ..

It seems like memory file creation is currently hard coded here in mem.py so changes may need to extend beyond just chunk.py. As for the chunk logging, it seems like empty logging is produced due to the placement of the 'disabled' chunk mode conditions in certain methods such as here or from functions that that don't set chunk mode conditions such as here.

You can find the configs and log files for this test run here.

[stefancoe]

I ran the model using 3 different modes using the same configuration of 20k households and mp turned off. The attached results show that the old code is fastest, 'disabled' is second but considerably slower and 'production' mode is the slowest.

chunkless_comparison.xlsx

[bstabler]

@esanchez01 to test again since this feature would be good to get working soon

[esanchez01]

I retested this feature with the new commits. There are still memory CSV files and chunk/memory logging that are produced. However, the extent of the memory tracking and logging appears to have been reduced. The run time for this new run was about 65 minutes compared to 140 minutes for the previous test, so it seems like unnecessary memory/chunk calls were further avoided.

You can find the configs and log files for the new test run here.

[bstabler]

@jpn-- please PR to develop when ready

[bstabler]

@jpn-- , actually, please move this to a branch within this repo so we can work on it here. Thanks.

[jpn--]

This is now a branch in the main repo, per @bstabler request. Closing this PR in favor of that.

[jpn--]

It appears that I can't currently run a model with chunking just plain turned off unless it's in training mode, but training mode carries a lot of overhead monitoring memory usage. If I set to production mode without an available chunk cache history file it falls back to training mode.

This is my crack at creating a fourth chunk training mode: 'disabled'. (the first 3 are training, production, and adaptive). The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, so we can focus on performance assuming there is abundant RAM.

[jpn--]

@toliwaga it certainly would be helpful for you to review this, as you're the expert on chunk management.

[toliwaga]

I'm out of the office on vacation until the beginning of August

[coveralls]

Coverage increased (+8.5%) to 64.879% when pulling 37862e3 on jpn--:chunkless into a57efa1 on ActivitySim:master.

[jpn--]

Next steps: @esanchez01 and @stefancoe will test "disabled" mode to confirm it works correctly.

[esanchez01]

@jpn-- , After testing this on 100k households using the SANDAG 1 Zone setup (multiprocessed), there are still memory CSV files in the output/log directory as well as chunk logging across various logfiles. This does not conform with the behavior of the 'disabled' mode as described in the original post:

The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, ..

It seems like memory file creation is currently hard coded here in mem.py so changes may need to extend beyond just chunk.py. As for the chunk logging, it seems like empty logging is produced due to the placement of the 'disabled' chunk mode conditions in certain methods such as here or from functions that that don't set chunk mode conditions such as here.

You can find the configs and log files for this test run here.

[stefancoe]

I ran the model using 3 different modes using the same configuration of 20k households and mp turned off. The attached results show that the old code is fastest, 'disabled' is second but considerably slower and 'production' mode is the slowest.

chunkless_comparison.xlsx

[bstabler]

@esanchez01 to test again since this feature would be good to get working soon

[esanchez01]

I retested this feature with the new commits. There are still memory CSV files and chunk/memory logging that are produced. However, the extent of the memory tracking and logging appears to have been reduced. The run time for this new run was about 65 minutes compared to 140 minutes for the previous test, so it seems like unnecessary memory/chunk calls were further avoided.

You can find the configs and log files for the new test run here.

[bstabler]

@jpn-- please PR to develop when ready

[bstabler]

@jpn-- , actually, please move this to a branch within this repo so we can work on it here. Thanks.

[jpn--]

This is now a branch in the main repo, per @bstabler request. Closing this PR in favor of that.

[jpn--]

It appears that I can't currently run a model with chunking just plain turned off unless it's in training mode, but training mode carries a lot of overhead monitoring memory usage. If I set to production mode without an available chunk cache history file it falls back to training mode.

This is my crack at creating a fourth chunk training mode: 'disabled'. (the first 3 are training, production, and adaptive). The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, so we can focus on performance assuming there is abundant RAM.

[jpn--]

@toliwaga it certainly would be helpful for you to review this, as you're the expert on chunk management.

[toliwaga]

I'm out of the office on vacation until the beginning of August

[coveralls]

Coverage increased (+8.5%) to 64.879% when pulling 37862e3 on jpn--:chunkless into a57efa1 on ActivitySim:master.

[jpn--]

Next steps: @esanchez01 and @stefancoe will test "disabled" mode to confirm it works correctly.

[esanchez01]

@jpn-- , After testing this on 100k households using the SANDAG 1 Zone setup (multiprocessed), there are still memory CSV files in the output/log directory as well as chunk logging across various logfiles. This does not conform with the behavior of the 'disabled' mode as described in the original post:

The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, ..

It seems like memory file creation is currently hard coded here in mem.py so changes may need to extend beyond just chunk.py. As for the chunk logging, it seems like empty logging is produced due to the placement of the 'disabled' chunk mode conditions in certain methods such as here or from functions that that don't set chunk mode conditions such as here.

You can find the configs and log files for this test run here.

[stefancoe]

I ran the model using 3 different modes using the same configuration of 20k households and mp turned off. The attached results show that the old code is fastest, 'disabled' is second but considerably slower and 'production' mode is the slowest.

chunkless_comparison.xlsx

[bstabler]

@esanchez01 to test again since this feature would be good to get working soon

[esanchez01]

I retested this feature with the new commits. There are still memory CSV files and chunk/memory logging that are produced. However, the extent of the memory tracking and logging appears to have been reduced. The run time for this new run was about 65 minutes compared to 140 minutes for the previous test, so it seems like unnecessary memory/chunk calls were further avoided.

You can find the configs and log files for the new test run here.

[bstabler]

@jpn-- please PR to develop when ready

[bstabler]

@jpn-- , actually, please move this to a branch within this repo so we can work on it here. Thanks.

[jpn--]

This is now a branch in the main repo, per @bstabler request. Closing this PR in favor of that.

[jpn--]

It appears that I can't currently run a model with chunking just plain turned off unless it's in training mode, but training mode carries a lot of overhead monitoring memory usage. If I set to production mode without an available chunk cache history file it falls back to training mode.

This is my crack at creating a fourth chunk training mode: 'disabled'. (the first 3 are training, production, and adaptive). The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, so we can focus on performance assuming there is abundant RAM.

[jpn--]

@toliwaga it certainly would be helpful for you to review this, as you're the expert on chunk management.

[toliwaga]

I'm out of the office on vacation until the beginning of August

[coveralls]

Coverage increased (+8.5%) to 64.879% when pulling 37862e3 on jpn--:chunkless into a57efa1 on ActivitySim:master.

[jpn--]

Next steps: @esanchez01 and @stefancoe will test "disabled" mode to confirm it works correctly.

[esanchez01]

@jpn-- , After testing this on 100k households using the SANDAG 1 Zone setup (multiprocessed), there are still memory CSV files in the output/log directory as well as chunk logging across various logfiles. This does not conform with the behavior of the 'disabled' mode as described in the original post:

The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, ..

It seems like memory file creation is currently hard coded here in mem.py so changes may need to extend beyond just chunk.py. As for the chunk logging, it seems like empty logging is produced due to the placement of the 'disabled' chunk mode conditions in certain methods such as here or from functions that that don't set chunk mode conditions such as here.

You can find the configs and log files for this test run here.

[stefancoe]

I ran the model using 3 different modes using the same configuration of 20k households and mp turned off. The attached results show that the old code is fastest, 'disabled' is second but considerably slower and 'production' mode is the slowest.

chunkless_comparison.xlsx

[bstabler]

@esanchez01 to test again since this feature would be good to get working soon

[esanchez01]

I retested this feature with the new commits. There are still memory CSV files and chunk/memory logging that are produced. However, the extent of the memory tracking and logging appears to have been reduced. The run time for this new run was about 65 minutes compared to 140 minutes for the previous test, so it seems like unnecessary memory/chunk calls were further avoided.

You can find the configs and log files for the new test run here.

[bstabler]

@jpn-- please PR to develop when ready

[bstabler]

@jpn-- , actually, please move this to a branch within this repo so we can work on it here. Thanks.

[jpn--]

This is now a branch in the main repo, per @bstabler request. Closing this PR in favor of that.

[jpn--]

It appears that I can't currently run a model with chunking just plain turned off unless it's in training mode, but training mode carries a lot of overhead monitoring memory usage. If I set to production mode without an available chunk cache history file it falls back to training mode.

This is my crack at creating a fourth chunk training mode: 'disabled'. (the first 3 are training, production, and adaptive). The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, so we can focus on performance assuming there is abundant RAM.

[jpn--]

@toliwaga it certainly would be helpful for you to review this, as you're the expert on chunk management.

[toliwaga]

I'm out of the office on vacation until the beginning of August

[coveralls]

Coverage increased (+8.5%) to 64.879% when pulling 37862e3 on jpn--:chunkless into a57efa1 on ActivitySim:master.

[jpn--]

Next steps: @esanchez01 and @stefancoe will test "disabled" mode to confirm it works correctly.

[esanchez01]

@jpn-- , After testing this on 100k households using the SANDAG 1 Zone setup (multiprocessed), there are still memory CSV files in the output/log directory as well as chunk logging across various logfiles. This does not conform with the behavior of the 'disabled' mode as described in the original post:

The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, ..

It seems like memory file creation is currently hard coded here in mem.py so changes may need to extend beyond just chunk.py. As for the chunk logging, it seems like empty logging is produced due to the placement of the 'disabled' chunk mode conditions in certain methods such as here or from functions that that don't set chunk mode conditions such as here.

You can find the configs and log files for this test run here.

[stefancoe]

I ran the model using 3 different modes using the same configuration of 20k households and mp turned off. The attached results show that the old code is fastest, 'disabled' is second but considerably slower and 'production' mode is the slowest.

chunkless_comparison.xlsx

[bstabler]

@esanchez01 to test again since this feature would be good to get working soon

[esanchez01]

I retested this feature with the new commits. There are still memory CSV files and chunk/memory logging that are produced. However, the extent of the memory tracking and logging appears to have been reduced. The run time for this new run was about 65 minutes compared to 140 minutes for the previous test, so it seems like unnecessary memory/chunk calls were further avoided.

You can find the configs and log files for the new test run here.

[bstabler]

@jpn-- please PR to develop when ready

[bstabler]

@jpn-- , actually, please move this to a branch within this repo so we can work on it here. Thanks.

[jpn--]

This is now a branch in the main repo, per @bstabler request. Closing this PR in favor of that.

[jpn--]

It appears that I can't currently run a model with chunking just plain turned off unless it's in training mode, but training mode carries a lot of overhead monitoring memory usage. If I set to production mode without an available chunk cache history file it falls back to training mode.

This is my crack at creating a fourth chunk training mode: 'disabled'. (the first 3 are training, production, and adaptive). The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, so we can focus on performance assuming there is abundant RAM.

[jpn--]

@toliwaga it certainly would be helpful for you to review this, as you're the expert on chunk management.

[toliwaga]

I'm out of the office on vacation until the beginning of August

[coveralls]

Coverage increased (+8.5%) to 64.879% when pulling 37862e3 on jpn--:chunkless into a57efa1 on ActivitySim:master.

[jpn--]

Next steps: @esanchez01 and @stefancoe will test "disabled" mode to confirm it works correctly.

[esanchez01]

@jpn-- , After testing this on 100k households using the SANDAG 1 Zone setup (multiprocessed), there are still memory CSV files in the output/log directory as well as chunk logging across various logfiles. This does not conform with the behavior of the 'disabled' mode as described in the original post:

The idea for disabled mode is that we don't do chunking, and also don't check or log memory usage ever, ..

It seems like memory file creation is currently hard coded here in mem.py so changes may need to extend beyond just chunk.py. As for the chunk logging, it seems like empty logging is produced due to the placement of the 'disabled' chunk mode conditions in certain methods such as here or from functions that that don't set chunk mode conditions such as here.

You can find the configs and log files for this test run here.

[stefancoe]

I ran the model using 3 different modes using the same configuration of 20k households and mp turned off. The attached results show that the old code is fastest, 'disabled' is second but considerably slower and 'production' mode is the slowest.

chunkless_comparison.xlsx

[bstabler]

@esanchez01 to test again since this feature would be good to get working soon

[esanchez01]

I retested this feature with the new commits. There are still memory CSV files and chunk/memory logging that are produced. However, the extent of the memory tracking and logging appears to have been reduced. The run time for this new run was about 65 minutes compared to 140 minutes for the previous test, so it seems like unnecessary memory/chunk calls were further avoided.

You can find the configs and log files for the new test run here.

[bstabler]

@jpn-- please PR to develop when ready

[bstabler]

@jpn-- , actually, please move this to a branch within this repo so we can work on it here. Thanks.

[jpn--]

This is now a branch in the main repo, per @bstabler request. Closing this PR in favor of that.