Poisson sampling methodology for EET

[m-richards]

In discussing with @timveitch and @janzill, we thought it would be useful to contribute back a version of the EET implemented for testing with the VITM 2 activitysim model for DTP Victoria against the rebased EET branch.

Our scope was to get EET into a state where we could run tests on the Victorian model, so the changes here are with that goal in mind, we're well aware it's not in a ready to merge state - it's not as general, well tested or flexible as you might want to integrate. That said, our approach has some substantial improvements over EET as it currently stands (though I believe some of this is addressed in the Outer Loop PR):

• Implementation of an alternate interaction sample methodology, so called poisson sampling for EET which is signficantly faster -for N alts in the model and sample set of M, monte carlo will make M uniform draws from the distribution, EET will make M x N gumble draws and poisson sampling will make N uniform draws - one for each alt. In this way, we sample alternatives without replacement and instead sample the binary "if we we to run the simulation M times, whats the probability alt n was picked at all". This changes the sample correction term, but once fed through leads to no biasing to the sample set in our testing. This provided a runtime benefit of ~50x for work location with a location sample size of 50 in the VITM 2 model with the ctramp shadow pricing.
• fixing of random numbers across shadow price iterations, so the random changes don't conflate shadow pricing convergence
• Fixing a rare bug in the way EET makes choices from the sample set when the sample set (or the ordering of the sample set) changes. As is, the index of the item in the sample choice set determines the frozen random number for that alternative. For example:
  • If alt 2 is the base case and is replaced by alt 8 in the project case, they recieve the same gumbel error term, not different ones.
  • If in the base case the sample set contains an alternative twice e.g. [1, 2, 2, 4, 5], when the final choices are made, these are de-duplicated as [1, 2, 4, 5], but then if in the project case, the sample set contains [1, 2, 6, 4, 5], this will lead to alts 4, 5 (and any others afterwards not getting a consistent error term - because alt (6) is getting the third error term in the project case, which was allocated to alt (4) in the base
  • (with the Poisson sampling, the indexing of the alternatives is not stable - so these two fringe cases became quite common)
  • I've tried to demonstrate this issue in 62ea910 if that's clearer than my description in words.
  • In solving this for the moment we've assumed that alternatives are consecutively numbered - we've been testing with the VITM 2 model, which has zones indexed from 1 which are non-consecutive, so we've been inefficiently drawing extra random numbers for the "gaps". I'm not sure what the nicest way would be to proceed in this regard.

As far as we see there are no real downsides to changing the sampling approach, with a significant runtime benefit. There's a theoretical concern that if your sample size is too small, the poisson approach might fail to generate any alternatives on the first try (which can lead to random numbers not being held constant between base and project) but for a well specified model with appropriate sampling rates this is unlikely to come up in practice. One other thing to be aware of is how the pick_count correction factor appears in the expression csvs - under this approach pick_count will always be one. That should be fine, but because this factor lives in the utility expressions and not in the code that's a bit harder to assert globally.

[jpn--]

@m-richards and VLC, we thank you for this significant and valuable contribution. This work has been included and advanced in #1064 so I'm closing this separate PR.

[m-richards]

Thanks @jpn-- for the update, no surprises for us there, happy to see this approach being taken forward in any case!

[m-richards]

In discussing with @timveitch and @janzill, we thought it would be useful to contribute back a version of the EET implemented for testing with the VITM 2 activitysim model for DTP Victoria against the rebased EET branch.

Our scope was to get EET into a state where we could run tests on the Victorian model, so the changes here are with that goal in mind, we're well aware it's not in a ready to merge state - it's not as general, well tested or flexible as you might want to integrate. That said, our approach has some substantial improvements over EET as it currently stands (though I believe some of this is addressed in the Outer Loop PR):

• Implementation of an alternate interaction sample methodology, so called poisson sampling for EET which is signficantly faster -for N alts in the model and sample set of M, monte carlo will make M uniform draws from the distribution, EET will make M x N gumble draws and poisson sampling will make N uniform draws - one for each alt. In this way, we sample alternatives without replacement and instead sample the binary "if we we to run the simulation M times, whats the probability alt n was picked at all". This changes the sample correction term, but once fed through leads to no biasing to the sample set in our testing. This provided a runtime benefit of ~50x for work location with a location sample size of 50 in the VITM 2 model with the ctramp shadow pricing.
• fixing of random numbers across shadow price iterations, so the random changes don't conflate shadow pricing convergence
• Fixing a rare bug in the way EET makes choices from the sample set when the sample set (or the ordering of the sample set) changes. As is, the index of the item in the sample choice set determines the frozen random number for that alternative. For example:
  • If alt 2 is the base case and is replaced by alt 8 in the project case, they recieve the same gumbel error term, not different ones.
  • If in the base case the sample set contains an alternative twice e.g. [1, 2, 2, 4, 5], when the final choices are made, these are de-duplicated as [1, 2, 4, 5], but then if in the project case, the sample set contains [1, 2, 6, 4, 5], this will lead to alts 4, 5 (and any others afterwards not getting a consistent error term - because alt (6) is getting the third error term in the project case, which was allocated to alt (4) in the base
  • (with the Poisson sampling, the indexing of the alternatives is not stable - so these two fringe cases became quite common)
  • I've tried to demonstrate this issue in 62ea910 if that's clearer than my description in words.
  • In solving this for the moment we've assumed that alternatives are consecutively numbered - we've been testing with the VITM 2 model, which has zones indexed from 1 which are non-consecutive, so we've been inefficiently drawing extra random numbers for the "gaps". I'm not sure what the nicest way would be to proceed in this regard.

As far as we see there are no real downsides to changing the sampling approach, with a significant runtime benefit. There's a theoretical concern that if your sample size is too small, the poisson approach might fail to generate any alternatives on the first try (which can lead to random numbers not being held constant between base and project) but for a well specified model with appropriate sampling rates this is unlikely to come up in practice. One other thing to be aware of is how the pick_count correction factor appears in the expression csvs - under this approach pick_count will always be one. That should be fine, but because this factor lives in the utility expressions and not in the code that's a bit harder to assert globally.

[jpn--]

@m-richards and VLC, we thank you for this significant and valuable contribution. This work has been included and advanced in #1064 so I'm closing this separate PR.

[m-richards]

Thanks @jpn-- for the update, no surprises for us there, happy to see this approach being taken forward in any case!

[m-richards]

In discussing with @timveitch and @janzill, we thought it would be useful to contribute back a version of the EET implemented for testing with the VITM 2 activitysim model for DTP Victoria against the rebased EET branch.

Our scope was to get EET into a state where we could run tests on the Victorian model, so the changes here are with that goal in mind, we're well aware it's not in a ready to merge state - it's not as general, well tested or flexible as you might want to integrate. That said, our approach has some substantial improvements over EET as it currently stands (though I believe some of this is addressed in the Outer Loop PR):

• Implementation of an alternate interaction sample methodology, so called poisson sampling for EET which is signficantly faster -for N alts in the model and sample set of M, monte carlo will make M uniform draws from the distribution, EET will make M x N gumble draws and poisson sampling will make N uniform draws - one for each alt. In this way, we sample alternatives without replacement and instead sample the binary "if we we to run the simulation M times, whats the probability alt n was picked at all". This changes the sample correction term, but once fed through leads to no biasing to the sample set in our testing. This provided a runtime benefit of ~50x for work location with a location sample size of 50 in the VITM 2 model with the ctramp shadow pricing.
• fixing of random numbers across shadow price iterations, so the random changes don't conflate shadow pricing convergence
• Fixing a rare bug in the way EET makes choices from the sample set when the sample set (or the ordering of the sample set) changes. As is, the index of the item in the sample choice set determines the frozen random number for that alternative. For example:
  • If alt 2 is the base case and is replaced by alt 8 in the project case, they recieve the same gumbel error term, not different ones.
  • If in the base case the sample set contains an alternative twice e.g. [1, 2, 2, 4, 5], when the final choices are made, these are de-duplicated as [1, 2, 4, 5], but then if in the project case, the sample set contains [1, 2, 6, 4, 5], this will lead to alts 4, 5 (and any others afterwards not getting a consistent error term - because alt (6) is getting the third error term in the project case, which was allocated to alt (4) in the base
  • (with the Poisson sampling, the indexing of the alternatives is not stable - so these two fringe cases became quite common)
  • I've tried to demonstrate this issue in 62ea910 if that's clearer than my description in words.
  • In solving this for the moment we've assumed that alternatives are consecutively numbered - we've been testing with the VITM 2 model, which has zones indexed from 1 which are non-consecutive, so we've been inefficiently drawing extra random numbers for the "gaps". I'm not sure what the nicest way would be to proceed in this regard.

As far as we see there are no real downsides to changing the sampling approach, with a significant runtime benefit. There's a theoretical concern that if your sample size is too small, the poisson approach might fail to generate any alternatives on the first try (which can lead to random numbers not being held constant between base and project) but for a well specified model with appropriate sampling rates this is unlikely to come up in practice. One other thing to be aware of is how the pick_count correction factor appears in the expression csvs - under this approach pick_count will always be one. That should be fine, but because this factor lives in the utility expressions and not in the code that's a bit harder to assert globally.

[jpn--]

@m-richards and VLC, we thank you for this significant and valuable contribution. This work has been included and advanced in #1064 so I'm closing this separate PR.

[m-richards]

Thanks @jpn-- for the update, no surprises for us there, happy to see this approach being taken forward in any case!

[m-richards]

In discussing with @timveitch and @janzill, we thought it would be useful to contribute back a version of the EET implemented for testing with the VITM 2 activitysim model for DTP Victoria against the rebased EET branch.

Our scope was to get EET into a state where we could run tests on the Victorian model, so the changes here are with that goal in mind, we're well aware it's not in a ready to merge state - it's not as general, well tested or flexible as you might want to integrate. That said, our approach has some substantial improvements over EET as it currently stands (though I believe some of this is addressed in the Outer Loop PR):

• Implementation of an alternate interaction sample methodology, so called poisson sampling for EET which is signficantly faster -for N alts in the model and sample set of M, monte carlo will make M uniform draws from the distribution, EET will make M x N gumble draws and poisson sampling will make N uniform draws - one for each alt. In this way, we sample alternatives without replacement and instead sample the binary "if we we to run the simulation M times, whats the probability alt n was picked at all". This changes the sample correction term, but once fed through leads to no biasing to the sample set in our testing. This provided a runtime benefit of ~50x for work location with a location sample size of 50 in the VITM 2 model with the ctramp shadow pricing.
• fixing of random numbers across shadow price iterations, so the random changes don't conflate shadow pricing convergence
• Fixing a rare bug in the way EET makes choices from the sample set when the sample set (or the ordering of the sample set) changes. As is, the index of the item in the sample choice set determines the frozen random number for that alternative. For example:
  • If alt 2 is the base case and is replaced by alt 8 in the project case, they recieve the same gumbel error term, not different ones.
  • If in the base case the sample set contains an alternative twice e.g. [1, 2, 2, 4, 5], when the final choices are made, these are de-duplicated as [1, 2, 4, 5], but then if in the project case, the sample set contains [1, 2, 6, 4, 5], this will lead to alts 4, 5 (and any others afterwards not getting a consistent error term - because alt (6) is getting the third error term in the project case, which was allocated to alt (4) in the base
  • (with the Poisson sampling, the indexing of the alternatives is not stable - so these two fringe cases became quite common)
  • I've tried to demonstrate this issue in 62ea910 if that's clearer than my description in words.
  • In solving this for the moment we've assumed that alternatives are consecutively numbered - we've been testing with the VITM 2 model, which has zones indexed from 1 which are non-consecutive, so we've been inefficiently drawing extra random numbers for the "gaps". I'm not sure what the nicest way would be to proceed in this regard.

As far as we see there are no real downsides to changing the sampling approach, with a significant runtime benefit. There's a theoretical concern that if your sample size is too small, the poisson approach might fail to generate any alternatives on the first try (which can lead to random numbers not being held constant between base and project) but for a well specified model with appropriate sampling rates this is unlikely to come up in practice. One other thing to be aware of is how the pick_count correction factor appears in the expression csvs - under this approach pick_count will always be one. That should be fine, but because this factor lives in the utility expressions and not in the code that's a bit harder to assert globally.

[jpn--]

@m-richards and VLC, we thank you for this significant and valuable contribution. This work has been included and advanced in #1064 so I'm closing this separate PR.

[m-richards]

Thanks @jpn-- for the update, no surprises for us there, happy to see this approach being taken forward in any case!

[m-richards]

In discussing with @timveitch and @janzill, we thought it would be useful to contribute back a version of the EET implemented for testing with the VITM 2 activitysim model for DTP Victoria against the rebased EET branch.

Our scope was to get EET into a state where we could run tests on the Victorian model, so the changes here are with that goal in mind, we're well aware it's not in a ready to merge state - it's not as general, well tested or flexible as you might want to integrate. That said, our approach has some substantial improvements over EET as it currently stands (though I believe some of this is addressed in the Outer Loop PR):

• Implementation of an alternate interaction sample methodology, so called poisson sampling for EET which is signficantly faster -for N alts in the model and sample set of M, monte carlo will make M uniform draws from the distribution, EET will make M x N gumble draws and poisson sampling will make N uniform draws - one for each alt. In this way, we sample alternatives without replacement and instead sample the binary "if we we to run the simulation M times, whats the probability alt n was picked at all". This changes the sample correction term, but once fed through leads to no biasing to the sample set in our testing. This provided a runtime benefit of ~50x for work location with a location sample size of 50 in the VITM 2 model with the ctramp shadow pricing.
• fixing of random numbers across shadow price iterations, so the random changes don't conflate shadow pricing convergence
• Fixing a rare bug in the way EET makes choices from the sample set when the sample set (or the ordering of the sample set) changes. As is, the index of the item in the sample choice set determines the frozen random number for that alternative. For example:
  • If alt 2 is the base case and is replaced by alt 8 in the project case, they recieve the same gumbel error term, not different ones.
  • If in the base case the sample set contains an alternative twice e.g. [1, 2, 2, 4, 5], when the final choices are made, these are de-duplicated as [1, 2, 4, 5], but then if in the project case, the sample set contains [1, 2, 6, 4, 5], this will lead to alts 4, 5 (and any others afterwards not getting a consistent error term - because alt (6) is getting the third error term in the project case, which was allocated to alt (4) in the base
  • (with the Poisson sampling, the indexing of the alternatives is not stable - so these two fringe cases became quite common)
  • I've tried to demonstrate this issue in 62ea910 if that's clearer than my description in words.
  • In solving this for the moment we've assumed that alternatives are consecutively numbered - we've been testing with the VITM 2 model, which has zones indexed from 1 which are non-consecutive, so we've been inefficiently drawing extra random numbers for the "gaps". I'm not sure what the nicest way would be to proceed in this regard.

As far as we see there are no real downsides to changing the sampling approach, with a significant runtime benefit. There's a theoretical concern that if your sample size is too small, the poisson approach might fail to generate any alternatives on the first try (which can lead to random numbers not being held constant between base and project) but for a well specified model with appropriate sampling rates this is unlikely to come up in practice. One other thing to be aware of is how the pick_count correction factor appears in the expression csvs - under this approach pick_count will always be one. That should be fine, but because this factor lives in the utility expressions and not in the code that's a bit harder to assert globally.

[jpn--]

@m-richards and VLC, we thank you for this significant and valuable contribution. This work has been included and advanced in #1064 so I'm closing this separate PR.

[m-richards]

Thanks @jpn-- for the update, no surprises for us there, happy to see this approach being taken forward in any case!

[m-richards]

In discussing with @timveitch and @janzill, we thought it would be useful to contribute back a version of the EET implemented for testing with the VITM 2 activitysim model for DTP Victoria against the rebased EET branch.

Our scope was to get EET into a state where we could run tests on the Victorian model, so the changes here are with that goal in mind, we're well aware it's not in a ready to merge state - it's not as general, well tested or flexible as you might want to integrate. That said, our approach has some substantial improvements over EET as it currently stands (though I believe some of this is addressed in the Outer Loop PR):

• Implementation of an alternate interaction sample methodology, so called poisson sampling for EET which is signficantly faster -for N alts in the model and sample set of M, monte carlo will make M uniform draws from the distribution, EET will make M x N gumble draws and poisson sampling will make N uniform draws - one for each alt. In this way, we sample alternatives without replacement and instead sample the binary "if we we to run the simulation M times, whats the probability alt n was picked at all". This changes the sample correction term, but once fed through leads to no biasing to the sample set in our testing. This provided a runtime benefit of ~50x for work location with a location sample size of 50 in the VITM 2 model with the ctramp shadow pricing.
• fixing of random numbers across shadow price iterations, so the random changes don't conflate shadow pricing convergence
• Fixing a rare bug in the way EET makes choices from the sample set when the sample set (or the ordering of the sample set) changes. As is, the index of the item in the sample choice set determines the frozen random number for that alternative. For example:
  • If alt 2 is the base case and is replaced by alt 8 in the project case, they recieve the same gumbel error term, not different ones.
  • If in the base case the sample set contains an alternative twice e.g. [1, 2, 2, 4, 5], when the final choices are made, these are de-duplicated as [1, 2, 4, 5], but then if in the project case, the sample set contains [1, 2, 6, 4, 5], this will lead to alts 4, 5 (and any others afterwards not getting a consistent error term - because alt (6) is getting the third error term in the project case, which was allocated to alt (4) in the base
  • (with the Poisson sampling, the indexing of the alternatives is not stable - so these two fringe cases became quite common)
  • I've tried to demonstrate this issue in 62ea910 if that's clearer than my description in words.
  • In solving this for the moment we've assumed that alternatives are consecutively numbered - we've been testing with the VITM 2 model, which has zones indexed from 1 which are non-consecutive, so we've been inefficiently drawing extra random numbers for the "gaps". I'm not sure what the nicest way would be to proceed in this regard.

As far as we see there are no real downsides to changing the sampling approach, with a significant runtime benefit. There's a theoretical concern that if your sample size is too small, the poisson approach might fail to generate any alternatives on the first try (which can lead to random numbers not being held constant between base and project) but for a well specified model with appropriate sampling rates this is unlikely to come up in practice. One other thing to be aware of is how the pick_count correction factor appears in the expression csvs - under this approach pick_count will always be one. That should be fine, but because this factor lives in the utility expressions and not in the code that's a bit harder to assert globally.

[jpn--]

@m-richards and VLC, we thank you for this significant and valuable contribution. This work has been included and advanced in #1064 so I'm closing this separate PR.

[m-richards]

Thanks @jpn-- for the update, no surprises for us there, happy to see this approach being taken forward in any case!

[m-richards]

In discussing with @timveitch and @janzill, we thought it would be useful to contribute back a version of the EET implemented for testing with the VITM 2 activitysim model for DTP Victoria against the rebased EET branch.

Our scope was to get EET into a state where we could run tests on the Victorian model, so the changes here are with that goal in mind, we're well aware it's not in a ready to merge state - it's not as general, well tested or flexible as you might want to integrate. That said, our approach has some substantial improvements over EET as it currently stands (though I believe some of this is addressed in the Outer Loop PR):

• Implementation of an alternate interaction sample methodology, so called poisson sampling for EET which is signficantly faster -for N alts in the model and sample set of M, monte carlo will make M uniform draws from the distribution, EET will make M x N gumble draws and poisson sampling will make N uniform draws - one for each alt. In this way, we sample alternatives without replacement and instead sample the binary "if we we to run the simulation M times, whats the probability alt n was picked at all". This changes the sample correction term, but once fed through leads to no biasing to the sample set in our testing. This provided a runtime benefit of ~50x for work location with a location sample size of 50 in the VITM 2 model with the ctramp shadow pricing.
• fixing of random numbers across shadow price iterations, so the random changes don't conflate shadow pricing convergence
• Fixing a rare bug in the way EET makes choices from the sample set when the sample set (or the ordering of the sample set) changes. As is, the index of the item in the sample choice set determines the frozen random number for that alternative. For example:
  • If alt 2 is the base case and is replaced by alt 8 in the project case, they recieve the same gumbel error term, not different ones.
  • If in the base case the sample set contains an alternative twice e.g. [1, 2, 2, 4, 5], when the final choices are made, these are de-duplicated as [1, 2, 4, 5], but then if in the project case, the sample set contains [1, 2, 6, 4, 5], this will lead to alts 4, 5 (and any others afterwards not getting a consistent error term - because alt (6) is getting the third error term in the project case, which was allocated to alt (4) in the base
  • (with the Poisson sampling, the indexing of the alternatives is not stable - so these two fringe cases became quite common)
  • I've tried to demonstrate this issue in 62ea910 if that's clearer than my description in words.
  • In solving this for the moment we've assumed that alternatives are consecutively numbered - we've been testing with the VITM 2 model, which has zones indexed from 1 which are non-consecutive, so we've been inefficiently drawing extra random numbers for the "gaps". I'm not sure what the nicest way would be to proceed in this regard.

As far as we see there are no real downsides to changing the sampling approach, with a significant runtime benefit. There's a theoretical concern that if your sample size is too small, the poisson approach might fail to generate any alternatives on the first try (which can lead to random numbers not being held constant between base and project) but for a well specified model with appropriate sampling rates this is unlikely to come up in practice. One other thing to be aware of is how the pick_count correction factor appears in the expression csvs - under this approach pick_count will always be one. That should be fine, but because this factor lives in the utility expressions and not in the code that's a bit harder to assert globally.

[jpn--]

@m-richards and VLC, we thank you for this significant and valuable contribution. This work has been included and advanced in #1064 so I'm closing this separate PR.

[m-richards]

Thanks @jpn-- for the update, no surprises for us there, happy to see this approach being taken forward in any case!

[m-richards]

In discussing with @timveitch and @janzill, we thought it would be useful to contribute back a version of the EET implemented for testing with the VITM 2 activitysim model for DTP Victoria against the rebased EET branch.

Our scope was to get EET into a state where we could run tests on the Victorian model, so the changes here are with that goal in mind, we're well aware it's not in a ready to merge state - it's not as general, well tested or flexible as you might want to integrate. That said, our approach has some substantial improvements over EET as it currently stands (though I believe some of this is addressed in the Outer Loop PR):

• Implementation of an alternate interaction sample methodology, so called poisson sampling for EET which is signficantly faster -for N alts in the model and sample set of M, monte carlo will make M uniform draws from the distribution, EET will make M x N gumble draws and poisson sampling will make N uniform draws - one for each alt. In this way, we sample alternatives without replacement and instead sample the binary "if we we to run the simulation M times, whats the probability alt n was picked at all". This changes the sample correction term, but once fed through leads to no biasing to the sample set in our testing. This provided a runtime benefit of ~50x for work location with a location sample size of 50 in the VITM 2 model with the ctramp shadow pricing.
• fixing of random numbers across shadow price iterations, so the random changes don't conflate shadow pricing convergence
• Fixing a rare bug in the way EET makes choices from the sample set when the sample set (or the ordering of the sample set) changes. As is, the index of the item in the sample choice set determines the frozen random number for that alternative. For example:
  • If alt 2 is the base case and is replaced by alt 8 in the project case, they recieve the same gumbel error term, not different ones.
  • If in the base case the sample set contains an alternative twice e.g. [1, 2, 2, 4, 5], when the final choices are made, these are de-duplicated as [1, 2, 4, 5], but then if in the project case, the sample set contains [1, 2, 6, 4, 5], this will lead to alts 4, 5 (and any others afterwards not getting a consistent error term - because alt (6) is getting the third error term in the project case, which was allocated to alt (4) in the base
  • (with the Poisson sampling, the indexing of the alternatives is not stable - so these two fringe cases became quite common)
  • I've tried to demonstrate this issue in 62ea910 if that's clearer than my description in words.
  • In solving this for the moment we've assumed that alternatives are consecutively numbered - we've been testing with the VITM 2 model, which has zones indexed from 1 which are non-consecutive, so we've been inefficiently drawing extra random numbers for the "gaps". I'm not sure what the nicest way would be to proceed in this regard.

As far as we see there are no real downsides to changing the sampling approach, with a significant runtime benefit. There's a theoretical concern that if your sample size is too small, the poisson approach might fail to generate any alternatives on the first try (which can lead to random numbers not being held constant between base and project) but for a well specified model with appropriate sampling rates this is unlikely to come up in practice. One other thing to be aware of is how the pick_count correction factor appears in the expression csvs - under this approach pick_count will always be one. That should be fine, but because this factor lives in the utility expressions and not in the code that's a bit harder to assert globally.

[jpn--]

@m-richards and VLC, we thank you for this significant and valuable contribution. This work has been included and advanced in #1064 so I'm closing this separate PR.

[m-richards]

Thanks @jpn-- for the update, no surprises for us there, happy to see this approach being taken forward in any case!