Variable edge buf size. int64 coordinates.

[ausrasul]

The edge buffer size is too small when dealing with large images.
So I added the option to make it user defined.

Also when converting large images where the x,y coordinates are larger than the max value of int16, the coordinates become invalid/capped to the max value of int16.
It is changed to int64.

[jmon12]

@ausrasul Thank you for your quick fix, I was about to do something similar.

About the int16 issue, I didn't dive into the code yet, but why are you using int types instead of uint types? I suppose the index values must be positive. In that case, a uint32 would cover the same positive value range of int64 for two times less memory.

Is the concern about memory consumption a valid one? In the PR #19 tackling the same issue @yxdragon seems to acknowledge that.

I'll be testing your fork and give a feedback.

[jmon12]

I've been testing your implementation @ausrasul with my data and encountered an other overflow, of uint16 this time. I opened a PR on your fork.

[ausrasul]

Hi, I've added your PR and also moved buf_size to the end of the args list to prevent breaking old code that uses positional args.

Regarding why int64 instead of uint64, at line sknw.py line 46 it tries to add an int from neighbors with a uint64 which result in float64 that cannot be used as an index.
One could cast back into uint but it requires deeper understanding of the code and performance implications.

Your concern about memory is valid, but the problem not int or uint, but rather it being 64 instead of 16.
an image with coordinates reaches the range of uint32 would require 16exabytes (rough estimate).
So that is more appropriate alternative to uint16 which is too small.
uint64 as I see it now is unnecessary memory hog.

[ausrasul]

The edge buffer size is too small when dealing with large images.
So I added the option to make it user defined.

Also when converting large images where the x,y coordinates are larger than the max value of int16, the coordinates become invalid/capped to the max value of int16.
It is changed to int64.

[jmon12]

@ausrasul Thank you for your quick fix, I was about to do something similar.

About the int16 issue, I didn't dive into the code yet, but why are you using int types instead of uint types? I suppose the index values must be positive. In that case, a uint32 would cover the same positive value range of int64 for two times less memory.

Is the concern about memory consumption a valid one? In the PR #19 tackling the same issue @yxdragon seems to acknowledge that.

I'll be testing your fork and give a feedback.

[jmon12]

I've been testing your implementation @ausrasul with my data and encountered an other overflow, of uint16 this time. I opened a PR on your fork.

[ausrasul]

Hi, I've added your PR and also moved buf_size to the end of the args list to prevent breaking old code that uses positional args.

Regarding why int64 instead of uint64, at line sknw.py line 46 it tries to add an int from neighbors with a uint64 which result in float64 that cannot be used as an index.
One could cast back into uint but it requires deeper understanding of the code and performance implications.

Your concern about memory is valid, but the problem not int or uint, but rather it being 64 instead of 16.
an image with coordinates reaches the range of uint32 would require 16exabytes (rough estimate).
So that is more appropriate alternative to uint16 which is too small.
uint64 as I see it now is unnecessary memory hog.

[ausrasul]

The edge buffer size is too small when dealing with large images.
So I added the option to make it user defined.

Also when converting large images where the x,y coordinates are larger than the max value of int16, the coordinates become invalid/capped to the max value of int16.
It is changed to int64.

[jmon12]

@ausrasul Thank you for your quick fix, I was about to do something similar.

About the int16 issue, I didn't dive into the code yet, but why are you using int types instead of uint types? I suppose the index values must be positive. In that case, a uint32 would cover the same positive value range of int64 for two times less memory.

Is the concern about memory consumption a valid one? In the PR #19 tackling the same issue @yxdragon seems to acknowledge that.

I'll be testing your fork and give a feedback.

[jmon12]

I've been testing your implementation @ausrasul with my data and encountered an other overflow, of uint16 this time. I opened a PR on your fork.

[ausrasul]

Hi, I've added your PR and also moved buf_size to the end of the args list to prevent breaking old code that uses positional args.

Regarding why int64 instead of uint64, at line sknw.py line 46 it tries to add an int from neighbors with a uint64 which result in float64 that cannot be used as an index.
One could cast back into uint but it requires deeper understanding of the code and performance implications.

Your concern about memory is valid, but the problem not int or uint, but rather it being 64 instead of 16.
an image with coordinates reaches the range of uint32 would require 16exabytes (rough estimate).
So that is more appropriate alternative to uint16 which is too small.
uint64 as I see it now is unnecessary memory hog.

[ausrasul]

The edge buffer size is too small when dealing with large images.
So I added the option to make it user defined.

Also when converting large images where the x,y coordinates are larger than the max value of int16, the coordinates become invalid/capped to the max value of int16.
It is changed to int64.

[jmon12]

@ausrasul Thank you for your quick fix, I was about to do something similar.

About the int16 issue, I didn't dive into the code yet, but why are you using int types instead of uint types? I suppose the index values must be positive. In that case, a uint32 would cover the same positive value range of int64 for two times less memory.

Is the concern about memory consumption a valid one? In the PR #19 tackling the same issue @yxdragon seems to acknowledge that.

I'll be testing your fork and give a feedback.

[jmon12]

I've been testing your implementation @ausrasul with my data and encountered an other overflow, of uint16 this time. I opened a PR on your fork.

[ausrasul]

Hi, I've added your PR and also moved buf_size to the end of the args list to prevent breaking old code that uses positional args.

Regarding why int64 instead of uint64, at line sknw.py line 46 it tries to add an int from neighbors with a uint64 which result in float64 that cannot be used as an index.
One could cast back into uint but it requires deeper understanding of the code and performance implications.

Your concern about memory is valid, but the problem not int or uint, but rather it being 64 instead of 16.
an image with coordinates reaches the range of uint32 would require 16exabytes (rough estimate).
So that is more appropriate alternative to uint16 which is too small.
uint64 as I see it now is unnecessary memory hog.

[ausrasul]

The edge buffer size is too small when dealing with large images.
So I added the option to make it user defined.

Also when converting large images where the x,y coordinates are larger than the max value of int16, the coordinates become invalid/capped to the max value of int16.
It is changed to int64.

[jmon12]

@ausrasul Thank you for your quick fix, I was about to do something similar.

About the int16 issue, I didn't dive into the code yet, but why are you using int types instead of uint types? I suppose the index values must be positive. In that case, a uint32 would cover the same positive value range of int64 for two times less memory.

Is the concern about memory consumption a valid one? In the PR #19 tackling the same issue @yxdragon seems to acknowledge that.

I'll be testing your fork and give a feedback.

[jmon12]

I've been testing your implementation @ausrasul with my data and encountered an other overflow, of uint16 this time. I opened a PR on your fork.

[ausrasul]

Hi, I've added your PR and also moved buf_size to the end of the args list to prevent breaking old code that uses positional args.

Regarding why int64 instead of uint64, at line sknw.py line 46 it tries to add an int from neighbors with a uint64 which result in float64 that cannot be used as an index.
One could cast back into uint but it requires deeper understanding of the code and performance implications.

Your concern about memory is valid, but the problem not int or uint, but rather it being 64 instead of 16.
an image with coordinates reaches the range of uint32 would require 16exabytes (rough estimate).
So that is more appropriate alternative to uint16 which is too small.
uint64 as I see it now is unnecessary memory hog.

[ausrasul]

The edge buffer size is too small when dealing with large images.
So I added the option to make it user defined.

Also when converting large images where the x,y coordinates are larger than the max value of int16, the coordinates become invalid/capped to the max value of int16.
It is changed to int64.

[jmon12]

@ausrasul Thank you for your quick fix, I was about to do something similar.

About the int16 issue, I didn't dive into the code yet, but why are you using int types instead of uint types? I suppose the index values must be positive. In that case, a uint32 would cover the same positive value range of int64 for two times less memory.

Is the concern about memory consumption a valid one? In the PR #19 tackling the same issue @yxdragon seems to acknowledge that.

I'll be testing your fork and give a feedback.

[jmon12]

I've been testing your implementation @ausrasul with my data and encountered an other overflow, of uint16 this time. I opened a PR on your fork.

[ausrasul]

Hi, I've added your PR and also moved buf_size to the end of the args list to prevent breaking old code that uses positional args.

Regarding why int64 instead of uint64, at line sknw.py line 46 it tries to add an int from neighbors with a uint64 which result in float64 that cannot be used as an index.
One could cast back into uint but it requires deeper understanding of the code and performance implications.

Your concern about memory is valid, but the problem not int or uint, but rather it being 64 instead of 16.
an image with coordinates reaches the range of uint32 would require 16exabytes (rough estimate).
So that is more appropriate alternative to uint16 which is too small.
uint64 as I see it now is unnecessary memory hog.

[ausrasul]

The edge buffer size is too small when dealing with large images.
So I added the option to make it user defined.

Also when converting large images where the x,y coordinates are larger than the max value of int16, the coordinates become invalid/capped to the max value of int16.
It is changed to int64.

[jmon12]

@ausrasul Thank you for your quick fix, I was about to do something similar.

About the int16 issue, I didn't dive into the code yet, but why are you using int types instead of uint types? I suppose the index values must be positive. In that case, a uint32 would cover the same positive value range of int64 for two times less memory.

Is the concern about memory consumption a valid one? In the PR #19 tackling the same issue @yxdragon seems to acknowledge that.

I'll be testing your fork and give a feedback.

[jmon12]

I've been testing your implementation @ausrasul with my data and encountered an other overflow, of uint16 this time. I opened a PR on your fork.

[ausrasul]

Hi, I've added your PR and also moved buf_size to the end of the args list to prevent breaking old code that uses positional args.

Regarding why int64 instead of uint64, at line sknw.py line 46 it tries to add an int from neighbors with a uint64 which result in float64 that cannot be used as an index.
One could cast back into uint but it requires deeper understanding of the code and performance implications.

Your concern about memory is valid, but the problem not int or uint, but rather it being 64 instead of 16.
an image with coordinates reaches the range of uint32 would require 16exabytes (rough estimate).
So that is more appropriate alternative to uint16 which is too small.
uint64 as I see it now is unnecessary memory hog.

[ausrasul]

The edge buffer size is too small when dealing with large images.
So I added the option to make it user defined.

Also when converting large images where the x,y coordinates are larger than the max value of int16, the coordinates become invalid/capped to the max value of int16.
It is changed to int64.

[jmon12]

@ausrasul Thank you for your quick fix, I was about to do something similar.

About the int16 issue, I didn't dive into the code yet, but why are you using int types instead of uint types? I suppose the index values must be positive. In that case, a uint32 would cover the same positive value range of int64 for two times less memory.

Is the concern about memory consumption a valid one? In the PR #19 tackling the same issue @yxdragon seems to acknowledge that.

I'll be testing your fork and give a feedback.

[jmon12]

I've been testing your implementation @ausrasul with my data and encountered an other overflow, of uint16 this time. I opened a PR on your fork.

[ausrasul]

Hi, I've added your PR and also moved buf_size to the end of the args list to prevent breaking old code that uses positional args.

Regarding why int64 instead of uint64, at line sknw.py line 46 it tries to add an int from neighbors with a uint64 which result in float64 that cannot be used as an index.
One could cast back into uint but it requires deeper understanding of the code and performance implications.

Your concern about memory is valid, but the problem not int or uint, but rather it being 64 instead of 16.
an image with coordinates reaches the range of uint32 would require 16exabytes (rough estimate).
So that is more appropriate alternative to uint16 which is too small.
uint64 as I see it now is unnecessary memory hog.