Fix color clustering

[dy]

That fixes#2376, #2374 and #2354.
By accident pushed 41c067a to master, this should fix that.

cc @alexcjohnson@etpinard

• add baseline images

[etpinard]

can you update the package-lock file?

[dy]

Done!

[etpinard]

You forgot to git add the json mock corresponding to gl2d_line_select.png

[alexcjohnson]

This is the one that accidentally made it into master, as mentioned above

By accident pushed 41c067a to master, this should fix that.

But since I had that in master as the base of my own PR, I removed it there #2379 (comment)

[etpinard]

Can you briefly explain why this fixes #2376 ?

[dy]

That fixes #2354. That is internal regl-line2d behavior: when we do line2d.update([opts1, undefined, opts2, ...rest]), the second line in batch is ignored instead of being removed. I made it accept null, indicating that batch item should be removed.

[etpinard]

Can you add a jasmine test to 🔒 this down?

[alexcjohnson]

Playing with this branch - looking great up to 90K points!

But then something changes, kind of gradually between 90K and 92K - here's 92K, where it looks like the marker outline colors pick randomly from the marker colors?

Then go a bit farther and the marker colors also get randomized - here's 100K points.

The code I used for this is (basically copied from one of the issue codepens):

varx_array=[],color_array=[],opacity=[],size=[];for(vari=0;i<N;i++){// N = eg 90000, 92000, 100000x_array.push(i);color_array.push(Math.random());opacity.push(Math.random());size.push(Math.random()*3+3);}vartrace1={x: x_array,y: color_array,mode: 'markers',marker: {size: size,color: color_array,opacity: opacity,colorscale: [[0,'rgb(255, 0, 0)'],[0.5,'rgb(0, 255, 0)'],[1.0,'rgb(0, 0, 255)']]},type: 'scattergl'};varlayout={title: 'Scatter Plot with a Color Dimension'};Plotly.newPlot(gd,[trace1],layout);

[dy]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

[etpinard]

Why is this gone?

[etpinard]

Can we not bump mapbox-gl in this PR?

[alexcjohnson]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

Makes sense, but we need to do something reasonable in this case (which should be well within the range that scattergl is expected to support), and not have it break like this. My eyes can't distinguish 65k colors, so we should be able to find a way to discretize the colors to fit within that limit.

[dy]

@alexcjohnson fixed by d604a76

[alexcjohnson]

@alexcjohnson fixed by d604a76

Beautiful! Here's 1M points now, looks lovely.

Is there an obvious way to test this? I wouldn't want to make a mock with 100k or more points in it, is there a way to trigger the difference at a lower volume? Or perhaps draw something like this programmatically (but not random like in my test) and read a few pixels?

[alexcjohnson]

maybe I'm being paranoid, but I still don't like random in tests. Can we do something repeatable like

plotly.js/src/traces/box/plot.js

Lines 17 to 30 in ce5bc97

	varrandSeed=2000000000;
	functionseed(){
	randSeed=2000000000;
	}
	functionrand(){
	varlastVal=randSeed;
	randSeed=(69069*randSeed+1)%4294967296;
	// don't let consecutive vals be too close together
	// gets away from really trying to be random, in favor of better local uniformity
	if(Math.abs(randSeed-lastVal)<429496729)returnrand();
	returnrandSeed/4294967296;
	}

?

[etpinard]

Ha right. Good call!

[etpinard]

done via
20c3a00 and eae6da2

[alexcjohnson]

Great idea - I bet we'll find other places this should be used too.

[alexcjohnson]

By accident pushed 41c067a to master, this should fix that.

Merging master 1154589 includes the commit where I 🔪 gl2d_line_select.json, right? I don't see it generated in https://circleci.com/gh/plotly/plotly.js/7082#artifacts/containers/0. So now we need to add it back into this PR. After that this looks great to me! 💃

[etpinard]

Merging master 1154589 includes the commit where I gl2d_line_select.json, right? I

Good 👁️ with commit f8b4e52 we get:

Some weird things happened on CI during this push, so I decided to bump glslify to get clean npm i through. This partly fixes #2386

---

@dfcreative anything else you may want to add to this PR?

[dy]

@etpinard so far looks great, thanks a lot for helping out! 💃

[dy]

That fixes#2376, #2374 and #2354.
By accident pushed 41c067a to master, this should fix that.

cc @alexcjohnson@etpinard

• add baseline images

[etpinard]

can you update the package-lock file?

[dy]

Done!

[etpinard]

You forgot to git add the json mock corresponding to gl2d_line_select.png

[alexcjohnson]

This is the one that accidentally made it into master, as mentioned above

By accident pushed 41c067a to master, this should fix that.

But since I had that in master as the base of my own PR, I removed it there #2379 (comment)

[etpinard]

Can you briefly explain why this fixes #2376 ?

[dy]

That fixes #2354. That is internal regl-line2d behavior: when we do line2d.update([opts1, undefined, opts2, ...rest]), the second line in batch is ignored instead of being removed. I made it accept null, indicating that batch item should be removed.

[etpinard]

Can you add a jasmine test to 🔒 this down?

[alexcjohnson]

Playing with this branch - looking great up to 90K points!

But then something changes, kind of gradually between 90K and 92K - here's 92K, where it looks like the marker outline colors pick randomly from the marker colors?

Then go a bit farther and the marker colors also get randomized - here's 100K points.

The code I used for this is (basically copied from one of the issue codepens):

varx_array=[],color_array=[],opacity=[],size=[];for(vari=0;i<N;i++){// N = eg 90000, 92000, 100000x_array.push(i);color_array.push(Math.random());opacity.push(Math.random());size.push(Math.random()*3+3);}vartrace1={x: x_array,y: color_array,mode: 'markers',marker: {size: size,color: color_array,opacity: opacity,colorscale: [[0,'rgb(255, 0, 0)'],[0.5,'rgb(0, 255, 0)'],[1.0,'rgb(0, 0, 255)']]},type: 'scattergl'};varlayout={title: 'Scatter Plot with a Color Dimension'};Plotly.newPlot(gd,[trace1],layout);

[dy]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

[etpinard]

Why is this gone?

[etpinard]

Can we not bump mapbox-gl in this PR?

[alexcjohnson]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

Makes sense, but we need to do something reasonable in this case (which should be well within the range that scattergl is expected to support), and not have it break like this. My eyes can't distinguish 65k colors, so we should be able to find a way to discretize the colors to fit within that limit.

[dy]

@alexcjohnson fixed by d604a76

[alexcjohnson]

@alexcjohnson fixed by d604a76

Beautiful! Here's 1M points now, looks lovely.

Is there an obvious way to test this? I wouldn't want to make a mock with 100k or more points in it, is there a way to trigger the difference at a lower volume? Or perhaps draw something like this programmatically (but not random like in my test) and read a few pixels?

[alexcjohnson]

maybe I'm being paranoid, but I still don't like random in tests. Can we do something repeatable like

plotly.js/src/traces/box/plot.js

Lines 17 to 30 in ce5bc97

	varrandSeed=2000000000;
	functionseed(){
	randSeed=2000000000;
	}
	functionrand(){
	varlastVal=randSeed;
	randSeed=(69069*randSeed+1)%4294967296;
	// don't let consecutive vals be too close together
	// gets away from really trying to be random, in favor of better local uniformity
	if(Math.abs(randSeed-lastVal)<429496729)returnrand();
	returnrandSeed/4294967296;
	}

?

[etpinard]

Ha right. Good call!

[etpinard]

done via
20c3a00 and eae6da2

[alexcjohnson]

Great idea - I bet we'll find other places this should be used too.

[alexcjohnson]

By accident pushed 41c067a to master, this should fix that.

Merging master 1154589 includes the commit where I 🔪 gl2d_line_select.json, right? I don't see it generated in https://circleci.com/gh/plotly/plotly.js/7082#artifacts/containers/0. So now we need to add it back into this PR. After that this looks great to me! 💃

[etpinard]

Merging master 1154589 includes the commit where I gl2d_line_select.json, right? I

Good 👁️ with commit f8b4e52 we get:

Some weird things happened on CI during this push, so I decided to bump glslify to get clean npm i through. This partly fixes #2386

---

@dfcreative anything else you may want to add to this PR?

[dy]

@etpinard so far looks great, thanks a lot for helping out! 💃

[dy]

That fixes#2376, #2374 and #2354.
By accident pushed 41c067a to master, this should fix that.

cc @alexcjohnson@etpinard

• add baseline images

[etpinard]

can you update the package-lock file?

[dy]

Done!

[etpinard]

You forgot to git add the json mock corresponding to gl2d_line_select.png

[alexcjohnson]

This is the one that accidentally made it into master, as mentioned above

By accident pushed 41c067a to master, this should fix that.

But since I had that in master as the base of my own PR, I removed it there #2379 (comment)

[etpinard]

Can you briefly explain why this fixes #2376 ?

[dy]

That fixes #2354. That is internal regl-line2d behavior: when we do line2d.update([opts1, undefined, opts2, ...rest]), the second line in batch is ignored instead of being removed. I made it accept null, indicating that batch item should be removed.

[etpinard]

Can you add a jasmine test to 🔒 this down?

[alexcjohnson]

Playing with this branch - looking great up to 90K points!

But then something changes, kind of gradually between 90K and 92K - here's 92K, where it looks like the marker outline colors pick randomly from the marker colors?

Then go a bit farther and the marker colors also get randomized - here's 100K points.

The code I used for this is (basically copied from one of the issue codepens):

varx_array=[],color_array=[],opacity=[],size=[];for(vari=0;i<N;i++){// N = eg 90000, 92000, 100000x_array.push(i);color_array.push(Math.random());opacity.push(Math.random());size.push(Math.random()*3+3);}vartrace1={x: x_array,y: color_array,mode: 'markers',marker: {size: size,color: color_array,opacity: opacity,colorscale: [[0,'rgb(255, 0, 0)'],[0.5,'rgb(0, 255, 0)'],[1.0,'rgb(0, 0, 255)']]},type: 'scattergl'};varlayout={title: 'Scatter Plot with a Color Dimension'};Plotly.newPlot(gd,[trace1],layout);

[dy]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

[etpinard]

Why is this gone?

[etpinard]

Can we not bump mapbox-gl in this PR?

[alexcjohnson]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

Makes sense, but we need to do something reasonable in this case (which should be well within the range that scattergl is expected to support), and not have it break like this. My eyes can't distinguish 65k colors, so we should be able to find a way to discretize the colors to fit within that limit.

[dy]

@alexcjohnson fixed by d604a76

[alexcjohnson]

@alexcjohnson fixed by d604a76

Beautiful! Here's 1M points now, looks lovely.

Is there an obvious way to test this? I wouldn't want to make a mock with 100k or more points in it, is there a way to trigger the difference at a lower volume? Or perhaps draw something like this programmatically (but not random like in my test) and read a few pixels?

[alexcjohnson]

maybe I'm being paranoid, but I still don't like random in tests. Can we do something repeatable like

plotly.js/src/traces/box/plot.js

Lines 17 to 30 in ce5bc97

	varrandSeed=2000000000;
	functionseed(){
	randSeed=2000000000;
	}
	functionrand(){
	varlastVal=randSeed;
	randSeed=(69069*randSeed+1)%4294967296;
	// don't let consecutive vals be too close together
	// gets away from really trying to be random, in favor of better local uniformity
	if(Math.abs(randSeed-lastVal)<429496729)returnrand();
	returnrandSeed/4294967296;
	}

?

[etpinard]

Ha right. Good call!

[etpinard]

done via
20c3a00 and eae6da2

[alexcjohnson]

Great idea - I bet we'll find other places this should be used too.

[alexcjohnson]

By accident pushed 41c067a to master, this should fix that.

Merging master 1154589 includes the commit where I 🔪 gl2d_line_select.json, right? I don't see it generated in https://circleci.com/gh/plotly/plotly.js/7082#artifacts/containers/0. So now we need to add it back into this PR. After that this looks great to me! 💃

[etpinard]

Merging master 1154589 includes the commit where I gl2d_line_select.json, right? I

Good 👁️ with commit f8b4e52 we get:

Some weird things happened on CI during this push, so I decided to bump glslify to get clean npm i through. This partly fixes #2386

---

@dfcreative anything else you may want to add to this PR?

[dy]

@etpinard so far looks great, thanks a lot for helping out! 💃

[dy]

That fixes#2376, #2374 and #2354.
By accident pushed 41c067a to master, this should fix that.

cc @alexcjohnson@etpinard

• add baseline images

[etpinard]

can you update the package-lock file?

[dy]

Done!

[etpinard]

You forgot to git add the json mock corresponding to gl2d_line_select.png

[alexcjohnson]

This is the one that accidentally made it into master, as mentioned above

By accident pushed 41c067a to master, this should fix that.

But since I had that in master as the base of my own PR, I removed it there #2379 (comment)

[etpinard]

Can you briefly explain why this fixes #2376 ?

[dy]

That fixes #2354. That is internal regl-line2d behavior: when we do line2d.update([opts1, undefined, opts2, ...rest]), the second line in batch is ignored instead of being removed. I made it accept null, indicating that batch item should be removed.

[etpinard]

Can you add a jasmine test to 🔒 this down?

[alexcjohnson]

Playing with this branch - looking great up to 90K points!

But then something changes, kind of gradually between 90K and 92K - here's 92K, where it looks like the marker outline colors pick randomly from the marker colors?

Then go a bit farther and the marker colors also get randomized - here's 100K points.

The code I used for this is (basically copied from one of the issue codepens):

varx_array=[],color_array=[],opacity=[],size=[];for(vari=0;i<N;i++){// N = eg 90000, 92000, 100000x_array.push(i);color_array.push(Math.random());opacity.push(Math.random());size.push(Math.random()*3+3);}vartrace1={x: x_array,y: color_array,mode: 'markers',marker: {size: size,color: color_array,opacity: opacity,colorscale: [[0,'rgb(255, 0, 0)'],[0.5,'rgb(0, 255, 0)'],[1.0,'rgb(0, 0, 255)']]},type: 'scattergl'};varlayout={title: 'Scatter Plot with a Color Dimension'};Plotly.newPlot(gd,[trace1],layout);

[dy]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

[etpinard]

Why is this gone?

[etpinard]

Can we not bump mapbox-gl in this PR?

[alexcjohnson]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

Makes sense, but we need to do something reasonable in this case (which should be well within the range that scattergl is expected to support), and not have it break like this. My eyes can't distinguish 65k colors, so we should be able to find a way to discretize the colors to fit within that limit.

[dy]

@alexcjohnson fixed by d604a76

[alexcjohnson]

@alexcjohnson fixed by d604a76

Beautiful! Here's 1M points now, looks lovely.

Is there an obvious way to test this? I wouldn't want to make a mock with 100k or more points in it, is there a way to trigger the difference at a lower volume? Or perhaps draw something like this programmatically (but not random like in my test) and read a few pixels?

[alexcjohnson]

maybe I'm being paranoid, but I still don't like random in tests. Can we do something repeatable like

plotly.js/src/traces/box/plot.js

Lines 17 to 30 in ce5bc97

	varrandSeed=2000000000;
	functionseed(){
	randSeed=2000000000;
	}
	functionrand(){
	varlastVal=randSeed;
	randSeed=(69069*randSeed+1)%4294967296;
	// don't let consecutive vals be too close together
	// gets away from really trying to be random, in favor of better local uniformity
	if(Math.abs(randSeed-lastVal)<429496729)returnrand();
	returnrandSeed/4294967296;
	}

?

[etpinard]

Ha right. Good call!

[etpinard]

done via
20c3a00 and eae6da2

[alexcjohnson]

Great idea - I bet we'll find other places this should be used too.

[alexcjohnson]

By accident pushed 41c067a to master, this should fix that.

Merging master 1154589 includes the commit where I 🔪 gl2d_line_select.json, right? I don't see it generated in https://circleci.com/gh/plotly/plotly.js/7082#artifacts/containers/0. So now we need to add it back into this PR. After that this looks great to me! 💃

[etpinard]

Merging master 1154589 includes the commit where I gl2d_line_select.json, right? I

Good 👁️ with commit f8b4e52 we get:

Some weird things happened on CI during this push, so I decided to bump glslify to get clean npm i through. This partly fixes #2386

---

@dfcreative anything else you may want to add to this PR?

[dy]

@etpinard so far looks great, thanks a lot for helping out! 💃

[dy]

That fixes#2376, #2374 and #2354.
By accident pushed 41c067a to master, this should fix that.

cc @alexcjohnson@etpinard

• add baseline images

[etpinard]

can you update the package-lock file?

[dy]

Done!

[etpinard]

You forgot to git add the json mock corresponding to gl2d_line_select.png

[alexcjohnson]

This is the one that accidentally made it into master, as mentioned above

By accident pushed 41c067a to master, this should fix that.

But since I had that in master as the base of my own PR, I removed it there #2379 (comment)

[etpinard]

Can you briefly explain why this fixes #2376 ?

[dy]

That fixes #2354. That is internal regl-line2d behavior: when we do line2d.update([opts1, undefined, opts2, ...rest]), the second line in batch is ignored instead of being removed. I made it accept null, indicating that batch item should be removed.

[etpinard]

Can you add a jasmine test to 🔒 this down?

[alexcjohnson]

Playing with this branch - looking great up to 90K points!

But then something changes, kind of gradually between 90K and 92K - here's 92K, where it looks like the marker outline colors pick randomly from the marker colors?

Then go a bit farther and the marker colors also get randomized - here's 100K points.

The code I used for this is (basically copied from one of the issue codepens):

varx_array=[],color_array=[],opacity=[],size=[];for(vari=0;i<N;i++){// N = eg 90000, 92000, 100000x_array.push(i);color_array.push(Math.random());opacity.push(Math.random());size.push(Math.random()*3+3);}vartrace1={x: x_array,y: color_array,mode: 'markers',marker: {size: size,color: color_array,opacity: opacity,colorscale: [[0,'rgb(255, 0, 0)'],[0.5,'rgb(0, 255, 0)'],[1.0,'rgb(0, 0, 255)']]},type: 'scattergl'};varlayout={title: 'Scatter Plot with a Color Dimension'};Plotly.newPlot(gd,[trace1],layout);

[dy]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

[etpinard]

Why is this gone?

[etpinard]

Can we not bump mapbox-gl in this PR?

[alexcjohnson]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

Makes sense, but we need to do something reasonable in this case (which should be well within the range that scattergl is expected to support), and not have it break like this. My eyes can't distinguish 65k colors, so we should be able to find a way to discretize the colors to fit within that limit.

[dy]

@alexcjohnson fixed by d604a76

[alexcjohnson]

@alexcjohnson fixed by d604a76

Beautiful! Here's 1M points now, looks lovely.

Is there an obvious way to test this? I wouldn't want to make a mock with 100k or more points in it, is there a way to trigger the difference at a lower volume? Or perhaps draw something like this programmatically (but not random like in my test) and read a few pixels?

[alexcjohnson]

maybe I'm being paranoid, but I still don't like random in tests. Can we do something repeatable like

plotly.js/src/traces/box/plot.js

Lines 17 to 30 in ce5bc97

	varrandSeed=2000000000;
	functionseed(){
	randSeed=2000000000;
	}
	functionrand(){
	varlastVal=randSeed;
	randSeed=(69069*randSeed+1)%4294967296;
	// don't let consecutive vals be too close together
	// gets away from really trying to be random, in favor of better local uniformity
	if(Math.abs(randSeed-lastVal)<429496729)returnrand();
	returnrandSeed/4294967296;
	}

?

[etpinard]

Ha right. Good call!

[etpinard]

done via
20c3a00 and eae6da2

[alexcjohnson]

Great idea - I bet we'll find other places this should be used too.

[alexcjohnson]

By accident pushed 41c067a to master, this should fix that.

Merging master 1154589 includes the commit where I 🔪 gl2d_line_select.json, right? I don't see it generated in https://circleci.com/gh/plotly/plotly.js/7082#artifacts/containers/0. So now we need to add it back into this PR. After that this looks great to me! 💃

[etpinard]

Merging master 1154589 includes the commit where I gl2d_line_select.json, right? I

Good 👁️ with commit f8b4e52 we get:

Some weird things happened on CI during this push, so I decided to bump glslify to get clean npm i through. This partly fixes #2386

---

@dfcreative anything else you may want to add to this PR?

[dy]

@etpinard so far looks great, thanks a lot for helping out! 💃

[dy]

That fixes#2376, #2374 and #2354.
By accident pushed 41c067a to master, this should fix that.

cc @alexcjohnson@etpinard

• add baseline images

[etpinard]

can you update the package-lock file?

[dy]

Done!

[etpinard]

You forgot to git add the json mock corresponding to gl2d_line_select.png

[alexcjohnson]

This is the one that accidentally made it into master, as mentioned above

By accident pushed 41c067a to master, this should fix that.

But since I had that in master as the base of my own PR, I removed it there #2379 (comment)

[etpinard]

Can you briefly explain why this fixes #2376 ?

[dy]

That fixes #2354. That is internal regl-line2d behavior: when we do line2d.update([opts1, undefined, opts2, ...rest]), the second line in batch is ignored instead of being removed. I made it accept null, indicating that batch item should be removed.

[etpinard]

Can you add a jasmine test to 🔒 this down?

[alexcjohnson]

Playing with this branch - looking great up to 90K points!

But then something changes, kind of gradually between 90K and 92K - here's 92K, where it looks like the marker outline colors pick randomly from the marker colors?

Then go a bit farther and the marker colors also get randomized - here's 100K points.

The code I used for this is (basically copied from one of the issue codepens):

varx_array=[],color_array=[],opacity=[],size=[];for(vari=0;i<N;i++){// N = eg 90000, 92000, 100000x_array.push(i);color_array.push(Math.random());opacity.push(Math.random());size.push(Math.random()*3+3);}vartrace1={x: x_array,y: color_array,mode: 'markers',marker: {size: size,color: color_array,opacity: opacity,colorscale: [[0,'rgb(255, 0, 0)'],[0.5,'rgb(0, 255, 0)'],[1.0,'rgb(0, 0, 255)']]},type: 'scattergl'};varlayout={title: 'Scatter Plot with a Color Dimension'};Plotly.newPlot(gd,[trace1],layout);

[dy]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

[etpinard]

Why is this gone?

[etpinard]

Can we not bump mapbox-gl in this PR?

[alexcjohnson]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

Makes sense, but we need to do something reasonable in this case (which should be well within the range that scattergl is expected to support), and not have it break like this. My eyes can't distinguish 65k colors, so we should be able to find a way to discretize the colors to fit within that limit.

[dy]

@alexcjohnson fixed by d604a76

[alexcjohnson]

@alexcjohnson fixed by d604a76

Beautiful! Here's 1M points now, looks lovely.

Is there an obvious way to test this? I wouldn't want to make a mock with 100k or more points in it, is there a way to trigger the difference at a lower volume? Or perhaps draw something like this programmatically (but not random like in my test) and read a few pixels?

[alexcjohnson]

maybe I'm being paranoid, but I still don't like random in tests. Can we do something repeatable like

plotly.js/src/traces/box/plot.js

Lines 17 to 30 in ce5bc97

	varrandSeed=2000000000;
	functionseed(){
	randSeed=2000000000;
	}
	functionrand(){
	varlastVal=randSeed;
	randSeed=(69069*randSeed+1)%4294967296;
	// don't let consecutive vals be too close together
	// gets away from really trying to be random, in favor of better local uniformity
	if(Math.abs(randSeed-lastVal)<429496729)returnrand();
	returnrandSeed/4294967296;
	}

?

[etpinard]

Ha right. Good call!

[etpinard]

done via
20c3a00 and eae6da2

[alexcjohnson]

Great idea - I bet we'll find other places this should be used too.

[alexcjohnson]

By accident pushed 41c067a to master, this should fix that.

Merging master 1154589 includes the commit where I 🔪 gl2d_line_select.json, right? I don't see it generated in https://circleci.com/gh/plotly/plotly.js/7082#artifacts/containers/0. So now we need to add it back into this PR. After that this looks great to me! 💃

[etpinard]

Merging master 1154589 includes the commit where I gl2d_line_select.json, right? I

Good 👁️ with commit f8b4e52 we get:

Some weird things happened on CI during this push, so I decided to bump glslify to get clean npm i through. This partly fixes #2386

---

@dfcreative anything else you may want to add to this PR?

[dy]

@etpinard so far looks great, thanks a lot for helping out! 💃

[dy]

That fixes#2376, #2374 and #2354.
By accident pushed 41c067a to master, this should fix that.

cc @alexcjohnson@etpinard

• add baseline images

[etpinard]

can you update the package-lock file?

[dy]

Done!

[etpinard]

You forgot to git add the json mock corresponding to gl2d_line_select.png

[alexcjohnson]

This is the one that accidentally made it into master, as mentioned above

By accident pushed 41c067a to master, this should fix that.

But since I had that in master as the base of my own PR, I removed it there #2379 (comment)

[etpinard]

Can you briefly explain why this fixes #2376 ?

[dy]

That fixes #2354. That is internal regl-line2d behavior: when we do line2d.update([opts1, undefined, opts2, ...rest]), the second line in batch is ignored instead of being removed. I made it accept null, indicating that batch item should be removed.

[etpinard]

Can you add a jasmine test to 🔒 this down?

[alexcjohnson]

Playing with this branch - looking great up to 90K points!

But then something changes, kind of gradually between 90K and 92K - here's 92K, where it looks like the marker outline colors pick randomly from the marker colors?

Then go a bit farther and the marker colors also get randomized - here's 100K points.

The code I used for this is (basically copied from one of the issue codepens):

varx_array=[],color_array=[],opacity=[],size=[];for(vari=0;i<N;i++){// N = eg 90000, 92000, 100000x_array.push(i);color_array.push(Math.random());opacity.push(Math.random());size.push(Math.random()*3+3);}vartrace1={x: x_array,y: color_array,mode: 'markers',marker: {size: size,color: color_array,opacity: opacity,colorscale: [[0,'rgb(255, 0, 0)'],[0.5,'rgb(0, 255, 0)'],[1.0,'rgb(0, 0, 255)']]},type: 'scattergl'};varlayout={title: 'Scatter Plot with a Color Dimension'};Plotly.newPlot(gd,[trace1],layout);

[dy]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

[etpinard]

Why is this gone?

[etpinard]

Can we not bump mapbox-gl in this PR?

[alexcjohnson]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

Makes sense, but we need to do something reasonable in this case (which should be well within the range that scattergl is expected to support), and not have it break like this. My eyes can't distinguish 65k colors, so we should be able to find a way to discretize the colors to fit within that limit.

[dy]

@alexcjohnson fixed by d604a76

[alexcjohnson]

@alexcjohnson fixed by d604a76

Beautiful! Here's 1M points now, looks lovely.

Is there an obvious way to test this? I wouldn't want to make a mock with 100k or more points in it, is there a way to trigger the difference at a lower volume? Or perhaps draw something like this programmatically (but not random like in my test) and read a few pixels?

[alexcjohnson]

maybe I'm being paranoid, but I still don't like random in tests. Can we do something repeatable like

plotly.js/src/traces/box/plot.js

Lines 17 to 30 in ce5bc97

	varrandSeed=2000000000;
	functionseed(){
	randSeed=2000000000;
	}
	functionrand(){
	varlastVal=randSeed;
	randSeed=(69069*randSeed+1)%4294967296;
	// don't let consecutive vals be too close together
	// gets away from really trying to be random, in favor of better local uniformity
	if(Math.abs(randSeed-lastVal)<429496729)returnrand();
	returnrandSeed/4294967296;
	}

?

[etpinard]

Ha right. Good call!

[etpinard]

done via
20c3a00 and eae6da2

[alexcjohnson]

Great idea - I bet we'll find other places this should be used too.

[alexcjohnson]

By accident pushed 41c067a to master, this should fix that.

Merging master 1154589 includes the commit where I 🔪 gl2d_line_select.json, right? I don't see it generated in https://circleci.com/gh/plotly/plotly.js/7082#artifacts/containers/0. So now we need to add it back into this PR. After that this looks great to me! 💃

[etpinard]

Merging master 1154589 includes the commit where I gl2d_line_select.json, right? I

Good 👁️ with commit f8b4e52 we get:

Some weird things happened on CI during this push, so I decided to bump glslify to get clean npm i through. This partly fixes #2386

---

@dfcreative anything else you may want to add to this PR?

[dy]

@etpinard so far looks great, thanks a lot for helping out! 💃

[dy]

That fixes#2376, #2374 and #2354.
By accident pushed 41c067a to master, this should fix that.

cc @alexcjohnson@etpinard

• add baseline images

[etpinard]

can you update the package-lock file?

[dy]

Done!

[etpinard]

You forgot to git add the json mock corresponding to gl2d_line_select.png

[alexcjohnson]

This is the one that accidentally made it into master, as mentioned above

By accident pushed 41c067a to master, this should fix that.

But since I had that in master as the base of my own PR, I removed it there #2379 (comment)

[etpinard]

Can you briefly explain why this fixes #2376 ?

[dy]

That fixes #2354. That is internal regl-line2d behavior: when we do line2d.update([opts1, undefined, opts2, ...rest]), the second line in batch is ignored instead of being removed. I made it accept null, indicating that batch item should be removed.

[etpinard]

Can you add a jasmine test to 🔒 this down?

[alexcjohnson]

Playing with this branch - looking great up to 90K points!

But then something changes, kind of gradually between 90K and 92K - here's 92K, where it looks like the marker outline colors pick randomly from the marker colors?

Then go a bit farther and the marker colors also get randomized - here's 100K points.

The code I used for this is (basically copied from one of the issue codepens):

varx_array=[],color_array=[],opacity=[],size=[];for(vari=0;i<N;i++){// N = eg 90000, 92000, 100000x_array.push(i);color_array.push(Math.random());opacity.push(Math.random());size.push(Math.random()*3+3);}vartrace1={x: x_array,y: color_array,mode: 'markers',marker: {size: size,color: color_array,opacity: opacity,colorscale: [[0,'rgb(255, 0, 0)'],[0.5,'rgb(0, 255, 0)'],[1.0,'rgb(0, 0, 255)']]},type: 'scattergl'};varlayout={title: 'Scatter Plot with a Color Dimension'};Plotly.newPlot(gd,[trace1],layout);

[dy]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

[etpinard]

Why is this gone?

[etpinard]

Can we not bump mapbox-gl in this PR?

[alexcjohnson]

@alexcjohnson that is regl limitation of uint32 buffers, we can't go over 65536 various colors, that is reached at ~91k points.

Makes sense, but we need to do something reasonable in this case (which should be well within the range that scattergl is expected to support), and not have it break like this. My eyes can't distinguish 65k colors, so we should be able to find a way to discretize the colors to fit within that limit.

[dy]

@alexcjohnson fixed by d604a76

[alexcjohnson]

@alexcjohnson fixed by d604a76

Beautiful! Here's 1M points now, looks lovely.

Is there an obvious way to test this? I wouldn't want to make a mock with 100k or more points in it, is there a way to trigger the difference at a lower volume? Or perhaps draw something like this programmatically (but not random like in my test) and read a few pixels?

[alexcjohnson]

maybe I'm being paranoid, but I still don't like random in tests. Can we do something repeatable like

plotly.js/src/traces/box/plot.js

Lines 17 to 30 in ce5bc97

	varrandSeed=2000000000;
	functionseed(){
	randSeed=2000000000;
	}
	functionrand(){
	varlastVal=randSeed;
	randSeed=(69069*randSeed+1)%4294967296;
	// don't let consecutive vals be too close together
	// gets away from really trying to be random, in favor of better local uniformity
	if(Math.abs(randSeed-lastVal)<429496729)returnrand();
	returnrandSeed/4294967296;
	}

?

[etpinard]

Ha right. Good call!

[etpinard]

done via
20c3a00 and eae6da2

[alexcjohnson]

Great idea - I bet we'll find other places this should be used too.

[alexcjohnson]

By accident pushed 41c067a to master, this should fix that.

Merging master 1154589 includes the commit where I 🔪 gl2d_line_select.json, right? I don't see it generated in https://circleci.com/gh/plotly/plotly.js/7082#artifacts/containers/0. So now we need to add it back into this PR. After that this looks great to me! 💃

[etpinard]

Merging master 1154589 includes the commit where I gl2d_line_select.json, right? I

Good 👁️ with commit f8b4e52 we get:

Some weird things happened on CI during this push, so I decided to bump glslify to get clean npm i through. This partly fixes #2386

---

@dfcreative anything else you may want to add to this PR?

[dy]

@etpinard so far looks great, thanks a lot for helping out! 💃