remove all bottlenecks in creating instrList with multiple sortIDs

[paciorek]

This gives huge speedups when we have multiple sortID values in a calcRange and split into individual instructions. Formerly we were creating multiple calcRanges and then multiple instr_nClass objects.

This avoids initializing all those R6 class objects. Instead the calcRange methods splits the range into a list of multiple R lists where each R list is essentially the same form as an instr_nClass (i.e., the same named fields).

Before it took 40 ms to make an instr_nClass object and 0.5 ms to make a calcRangeClass object. The time to make a simple R list is negligible by comparison (I didn't microbenchmark that). The result is that for 10000 elements (y[i] <- y[i+1]), it used to take 500 seconds to make the instruction list and now takes 0.03. So a million such elements now takes 3 seconds.

For handling the resulting R list of "R instr", for the moment I made it an S3 class "Rlist_Rinstr". We could consider making an R6 class for the overall list and an R6 class for the inner lists. At the least if we do the former, we can give it an isCompiled method and that will fit into the existing checking in calc_op and simulate. Using S3 doesn't fit within our standard style.

This feels a bit like working around the system we devised but the working around is primarily within makeInstrList, and the speedup is huge, so seems ok.

[paciorek]

Also we'll want to discuss whether anything in the setup code processing in nimble2 needs adjustment given that makeInstrList now produces an R list of lists.

[paciorek]

@perrydv I'd like to move towards merging this in. Can you take a look at what I've done in terms of makeInstrList now producing an R list of lists and how that might interact with any work you've done recently such as the support-nimble2 work?

[paciorek]

This gives huge speedups when we have multiple sortID values in a calcRange and split into individual instructions. Formerly we were creating multiple calcRanges and then multiple instr_nClass objects.

This avoids initializing all those R6 class objects. Instead the calcRange methods splits the range into a list of multiple R lists where each R list is essentially the same form as an instr_nClass (i.e., the same named fields).

Before it took 40 ms to make an instr_nClass object and 0.5 ms to make a calcRangeClass object. The time to make a simple R list is negligible by comparison (I didn't microbenchmark that). The result is that for 10000 elements (y[i] <- y[i+1]), it used to take 500 seconds to make the instruction list and now takes 0.03. So a million such elements now takes 3 seconds.

For handling the resulting R list of "R instr", for the moment I made it an S3 class "Rlist_Rinstr". We could consider making an R6 class for the overall list and an R6 class for the inner lists. At the least if we do the former, we can give it an isCompiled method and that will fit into the existing checking in calc_op and simulate. Using S3 doesn't fit within our standard style.

This feels a bit like working around the system we devised but the working around is primarily within makeInstrList, and the speedup is huge, so seems ok.

[paciorek]

Also we'll want to discuss whether anything in the setup code processing in nimble2 needs adjustment given that makeInstrList now produces an R list of lists.

[paciorek]

@perrydv I'd like to move towards merging this in. Can you take a look at what I've done in terms of makeInstrList now producing an R list of lists and how that might interact with any work you've done recently such as the support-nimble2 work?

[paciorek]

This gives huge speedups when we have multiple sortID values in a calcRange and split into individual instructions. Formerly we were creating multiple calcRanges and then multiple instr_nClass objects.

This avoids initializing all those R6 class objects. Instead the calcRange methods splits the range into a list of multiple R lists where each R list is essentially the same form as an instr_nClass (i.e., the same named fields).

Before it took 40 ms to make an instr_nClass object and 0.5 ms to make a calcRangeClass object. The time to make a simple R list is negligible by comparison (I didn't microbenchmark that). The result is that for 10000 elements (y[i] <- y[i+1]), it used to take 500 seconds to make the instruction list and now takes 0.03. So a million such elements now takes 3 seconds.

For handling the resulting R list of "R instr", for the moment I made it an S3 class "Rlist_Rinstr". We could consider making an R6 class for the overall list and an R6 class for the inner lists. At the least if we do the former, we can give it an isCompiled method and that will fit into the existing checking in calc_op and simulate. Using S3 doesn't fit within our standard style.

This feels a bit like working around the system we devised but the working around is primarily within makeInstrList, and the speedup is huge, so seems ok.

[paciorek]

Also we'll want to discuss whether anything in the setup code processing in nimble2 needs adjustment given that makeInstrList now produces an R list of lists.

[paciorek]

@perrydv I'd like to move towards merging this in. Can you take a look at what I've done in terms of makeInstrList now producing an R list of lists and how that might interact with any work you've done recently such as the support-nimble2 work?

[paciorek]

This gives huge speedups when we have multiple sortID values in a calcRange and split into individual instructions. Formerly we were creating multiple calcRanges and then multiple instr_nClass objects.

This avoids initializing all those R6 class objects. Instead the calcRange methods splits the range into a list of multiple R lists where each R list is essentially the same form as an instr_nClass (i.e., the same named fields).

Before it took 40 ms to make an instr_nClass object and 0.5 ms to make a calcRangeClass object. The time to make a simple R list is negligible by comparison (I didn't microbenchmark that). The result is that for 10000 elements (y[i] <- y[i+1]), it used to take 500 seconds to make the instruction list and now takes 0.03. So a million such elements now takes 3 seconds.

For handling the resulting R list of "R instr", for the moment I made it an S3 class "Rlist_Rinstr". We could consider making an R6 class for the overall list and an R6 class for the inner lists. At the least if we do the former, we can give it an isCompiled method and that will fit into the existing checking in calc_op and simulate. Using S3 doesn't fit within our standard style.

This feels a bit like working around the system we devised but the working around is primarily within makeInstrList, and the speedup is huge, so seems ok.

[paciorek]

Also we'll want to discuss whether anything in the setup code processing in nimble2 needs adjustment given that makeInstrList now produces an R list of lists.

[paciorek]

@perrydv I'd like to move towards merging this in. Can you take a look at what I've done in terms of makeInstrList now producing an R list of lists and how that might interact with any work you've done recently such as the support-nimble2 work?

[paciorek]

This gives huge speedups when we have multiple sortID values in a calcRange and split into individual instructions. Formerly we were creating multiple calcRanges and then multiple instr_nClass objects.

This avoids initializing all those R6 class objects. Instead the calcRange methods splits the range into a list of multiple R lists where each R list is essentially the same form as an instr_nClass (i.e., the same named fields).

Before it took 40 ms to make an instr_nClass object and 0.5 ms to make a calcRangeClass object. The time to make a simple R list is negligible by comparison (I didn't microbenchmark that). The result is that for 10000 elements (y[i] <- y[i+1]), it used to take 500 seconds to make the instruction list and now takes 0.03. So a million such elements now takes 3 seconds.

For handling the resulting R list of "R instr", for the moment I made it an S3 class "Rlist_Rinstr". We could consider making an R6 class for the overall list and an R6 class for the inner lists. At the least if we do the former, we can give it an isCompiled method and that will fit into the existing checking in calc_op and simulate. Using S3 doesn't fit within our standard style.

This feels a bit like working around the system we devised but the working around is primarily within makeInstrList, and the speedup is huge, so seems ok.

[paciorek]

Also we'll want to discuss whether anything in the setup code processing in nimble2 needs adjustment given that makeInstrList now produces an R list of lists.

[paciorek]

@perrydv I'd like to move towards merging this in. Can you take a look at what I've done in terms of makeInstrList now producing an R list of lists and how that might interact with any work you've done recently such as the support-nimble2 work?

[paciorek]

This gives huge speedups when we have multiple sortID values in a calcRange and split into individual instructions. Formerly we were creating multiple calcRanges and then multiple instr_nClass objects.

This avoids initializing all those R6 class objects. Instead the calcRange methods splits the range into a list of multiple R lists where each R list is essentially the same form as an instr_nClass (i.e., the same named fields).

Before it took 40 ms to make an instr_nClass object and 0.5 ms to make a calcRangeClass object. The time to make a simple R list is negligible by comparison (I didn't microbenchmark that). The result is that for 10000 elements (y[i] <- y[i+1]), it used to take 500 seconds to make the instruction list and now takes 0.03. So a million such elements now takes 3 seconds.

For handling the resulting R list of "R instr", for the moment I made it an S3 class "Rlist_Rinstr". We could consider making an R6 class for the overall list and an R6 class for the inner lists. At the least if we do the former, we can give it an isCompiled method and that will fit into the existing checking in calc_op and simulate. Using S3 doesn't fit within our standard style.

This feels a bit like working around the system we devised but the working around is primarily within makeInstrList, and the speedup is huge, so seems ok.

[paciorek]

Also we'll want to discuss whether anything in the setup code processing in nimble2 needs adjustment given that makeInstrList now produces an R list of lists.

[paciorek]

@perrydv I'd like to move towards merging this in. Can you take a look at what I've done in terms of makeInstrList now producing an R list of lists and how that might interact with any work you've done recently such as the support-nimble2 work?

[paciorek]

This gives huge speedups when we have multiple sortID values in a calcRange and split into individual instructions. Formerly we were creating multiple calcRanges and then multiple instr_nClass objects.

This avoids initializing all those R6 class objects. Instead the calcRange methods splits the range into a list of multiple R lists where each R list is essentially the same form as an instr_nClass (i.e., the same named fields).

Before it took 40 ms to make an instr_nClass object and 0.5 ms to make a calcRangeClass object. The time to make a simple R list is negligible by comparison (I didn't microbenchmark that). The result is that for 10000 elements (y[i] <- y[i+1]), it used to take 500 seconds to make the instruction list and now takes 0.03. So a million such elements now takes 3 seconds.

For handling the resulting R list of "R instr", for the moment I made it an S3 class "Rlist_Rinstr". We could consider making an R6 class for the overall list and an R6 class for the inner lists. At the least if we do the former, we can give it an isCompiled method and that will fit into the existing checking in calc_op and simulate. Using S3 doesn't fit within our standard style.

This feels a bit like working around the system we devised but the working around is primarily within makeInstrList, and the speedup is huge, so seems ok.

[paciorek]

Also we'll want to discuss whether anything in the setup code processing in nimble2 needs adjustment given that makeInstrList now produces an R list of lists.

[paciorek]

@perrydv I'd like to move towards merging this in. Can you take a look at what I've done in terms of makeInstrList now producing an R list of lists and how that might interact with any work you've done recently such as the support-nimble2 work?

[paciorek]

This gives huge speedups when we have multiple sortID values in a calcRange and split into individual instructions. Formerly we were creating multiple calcRanges and then multiple instr_nClass objects.

This avoids initializing all those R6 class objects. Instead the calcRange methods splits the range into a list of multiple R lists where each R list is essentially the same form as an instr_nClass (i.e., the same named fields).

Before it took 40 ms to make an instr_nClass object and 0.5 ms to make a calcRangeClass object. The time to make a simple R list is negligible by comparison (I didn't microbenchmark that). The result is that for 10000 elements (y[i] <- y[i+1]), it used to take 500 seconds to make the instruction list and now takes 0.03. So a million such elements now takes 3 seconds.

For handling the resulting R list of "R instr", for the moment I made it an S3 class "Rlist_Rinstr". We could consider making an R6 class for the overall list and an R6 class for the inner lists. At the least if we do the former, we can give it an isCompiled method and that will fit into the existing checking in calc_op and simulate. Using S3 doesn't fit within our standard style.

This feels a bit like working around the system we devised but the working around is primarily within makeInstrList, and the speedup is huge, so seems ok.

[paciorek]

Also we'll want to discuss whether anything in the setup code processing in nimble2 needs adjustment given that makeInstrList now produces an R list of lists.

[paciorek]

@perrydv I'd like to move towards merging this in. Can you take a look at what I've done in terms of makeInstrList now producing an R list of lists and how that might interact with any work you've done recently such as the support-nimble2 work?