Modernize Python 2 code to get ready for Python 3

[eirnym]

C

Checklist

• npm install && npm test passes
• tests are included
• documentation is changed or added
• commit message follows commit guidelines

Description of change

Continuation of #1335 and #1150. Please, review, comment, add more changes and fixes :) All changes are splitted, as some of them are massive.

UPDATE 1
Now I'm fighting with test/test-find-python.js file to allow it even to start testing in Python3-only environment.

UPDATE 2
Tests are passing, but it's possible that I'd miss something

[cclauss]

Great stuff... Just two minor change requests. Thanks.

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

extracted checks to compat module

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

There's a reason for it in this case: exception -> substitution will run much slower than knowlege of version where this name doesn't exist. This is also used in six and works well.

[eirnym]

so I'd prefer to leave this kind of check for types and for a dictionary. But I can change if you'd insist on it.

[refack]

I'm going to land @rodrigc's #1150, then I'll rebase this to see what's left.

[eirnym]

@refack it won't work as I found a lot more issues in code to be ported

[eirnym]

This doesn't mean, his work is useless.

[eirnym]

@refack@cclauss Could you have a chance to know what these variables should be? I haven't found any traces of them in functions they are and how to fix these lines.

./gyp/pylib/gyp/xcode_emulation.py:846:9: F821 undefined name 'cflags'
cflags.append('-F' + platform_root + '/Developer/Library/Frameworks/')
^
./gyp/pylib/gyp/generator/make.py:1645:50: F821 undefined name 'target'
print("WARNING: no output for", self.type, target)
^
2 F821 undefined name 'target'
2

[cclauss]

See #1333 and #1334 My sense on cflags is that it should be ldflags in this context but not 100% certain.

[refack]

it won't work as I found a lot more issues in code to be ported

As long as it passes CI, I take it as a sign it doesn't regress. I assumed that work was not done yet to get actual py3 compat.
Let's see is we can get everything done in this PR.

[eirnym]

My PR also passes CI, but not finished :D There's many things to fix

Before you try to merge it, I'd like to finish some other checks. I'll let you know

[eirnym]

Merged and I've done for today.

@cclauss#1333 is included

Unresolved questions for the next run are:

• sorted() and cmp=
• input.py: compile.ast -> ast. It's about to rewrite main part of the module.
• optparse may be dropped in a foreseeable future
• mac_tool uses veeery old version of plistlib and implies that pyobjc is installed (which may be not if we install python from MacPorts/Homebrew)
• I crashed pylint

[eirnym]

@refack I finished, including merging

[eirnym]

@cclauss for this run I used pylint --disable=E1125,E0702,E0702,E1701,E1124,C,W0311 pylib after flake8

[refack]

@eirnym I've rebased you branch. Before you continue, take a tag, and pull with --rebase, so you can compare what changed.

[eirnym]

you lost at least a half of my changes during rebasing. Merge would be always easier.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

We keep a linier git history, so we rebase. Some changes were incompatible with #1572 so I did not base them.

[eirnym]

If you'd tell me before, It would be just easier to rebase them manually. Now I repeat my work.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

Sorry. I'm juggling several branches, and reviewing a ton of code...

[eirnym]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

[refack]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

Sorry again.
There is no rush... this could wait a few more days.

Thank you for the dedication.

[eirnym]

Next possible day will be in a few days and I don't want to forget and start all things over again.

[eirnym]

I restored changes. Commit is a bit bulky, but it does it's job right.

[refack]

Thank you very much 🥇

[eirnym]

So what the current state of the PR?

[eirnym]

@refack@cclauss

[cclauss]

Conflicts.

I have approved but I am not a maintainer. Overall, I would feel more comfortable if we used the six module (or futurize) instead of inventing our own from scratch. The linter directives in gyp/pylib/gyp/compat.py are just as a result of a refusal to follow Python porting best practice use feature detection instead of version detection. The map()s and filter()s would be faster and more readable as list comprehensions.

[eirnym]

@cclauss I feel the same about six, but I'm not a maintainer to make a decision about additional dependency

Checking Python version is significantly faster and stable enough and it's used even in pip and six.

about map and filter. I changed as little as it was possible. "this is not making code clean and fast, but only a compatibility in mind". if you start make this optimisations, you'll end up with rewriting the whole code from scratch leaving only required features. This is a huge work and I'm not a person to take it in 100%.

[eirnym]

And please, remember that we have no tests. npm run test is not sufficient here.

[eirnym]

One more thing about compatibility and Python version checking: this is only about syntax and builtins checking. For Python2 you can't use async/await as well as for Python 3 you can't use print statement and very old variant of raising. With imports it's try-except impoting which is basically feautre-checking and never monkey-patching.

[eirnym]

@refack almost 2 weeks passed since I done this work. Please, tell me what else needs to be done here?

[cclauss]

Conflicts :-(

[eirnym]

@cclauss it's easy to solve code conflicts, but I don't see any intention from maintainers to merge these changes anyway.

[eirnym]

I gave up on this branch. I have a copy if anybody would be intereseted in merging it.

[eirnym]

C

Checklist

• npm install && npm test passes
• tests are included
• documentation is changed or added
• commit message follows commit guidelines

Description of change

Continuation of #1335 and #1150. Please, review, comment, add more changes and fixes :) All changes are splitted, as some of them are massive.

UPDATE 1
Now I'm fighting with test/test-find-python.js file to allow it even to start testing in Python3-only environment.

UPDATE 2
Tests are passing, but it's possible that I'd miss something

[cclauss]

Great stuff... Just two minor change requests. Thanks.

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

extracted checks to compat module

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

There's a reason for it in this case: exception -> substitution will run much slower than knowlege of version where this name doesn't exist. This is also used in six and works well.

[eirnym]

so I'd prefer to leave this kind of check for types and for a dictionary. But I can change if you'd insist on it.

[refack]

I'm going to land @rodrigc's #1150, then I'll rebase this to see what's left.

[eirnym]

@refack it won't work as I found a lot more issues in code to be ported

[eirnym]

This doesn't mean, his work is useless.

[eirnym]

@refack@cclauss Could you have a chance to know what these variables should be? I haven't found any traces of them in functions they are and how to fix these lines.

./gyp/pylib/gyp/xcode_emulation.py:846:9: F821 undefined name 'cflags'
cflags.append('-F' + platform_root + '/Developer/Library/Frameworks/')
^
./gyp/pylib/gyp/generator/make.py:1645:50: F821 undefined name 'target'
print("WARNING: no output for", self.type, target)
^
2 F821 undefined name 'target'
2

[cclauss]

See #1333 and #1334 My sense on cflags is that it should be ldflags in this context but not 100% certain.

[refack]

it won't work as I found a lot more issues in code to be ported

As long as it passes CI, I take it as a sign it doesn't regress. I assumed that work was not done yet to get actual py3 compat.
Let's see is we can get everything done in this PR.

[eirnym]

My PR also passes CI, but not finished :D There's many things to fix

Before you try to merge it, I'd like to finish some other checks. I'll let you know

[eirnym]

Merged and I've done for today.

@cclauss#1333 is included

Unresolved questions for the next run are:

• sorted() and cmp=
• input.py: compile.ast -> ast. It's about to rewrite main part of the module.
• optparse may be dropped in a foreseeable future
• mac_tool uses veeery old version of plistlib and implies that pyobjc is installed (which may be not if we install python from MacPorts/Homebrew)
• I crashed pylint

[eirnym]

@refack I finished, including merging

[eirnym]

@cclauss for this run I used pylint --disable=E1125,E0702,E0702,E1701,E1124,C,W0311 pylib after flake8

[refack]

@eirnym I've rebased you branch. Before you continue, take a tag, and pull with --rebase, so you can compare what changed.

[eirnym]

you lost at least a half of my changes during rebasing. Merge would be always easier.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

We keep a linier git history, so we rebase. Some changes were incompatible with #1572 so I did not base them.

[eirnym]

If you'd tell me before, It would be just easier to rebase them manually. Now I repeat my work.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

Sorry. I'm juggling several branches, and reviewing a ton of code...

[eirnym]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

[refack]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

Sorry again.
There is no rush... this could wait a few more days.

Thank you for the dedication.

[eirnym]

Next possible day will be in a few days and I don't want to forget and start all things over again.

[eirnym]

I restored changes. Commit is a bit bulky, but it does it's job right.

[refack]

Thank you very much 🥇

[eirnym]

So what the current state of the PR?

[eirnym]

@refack@cclauss

[cclauss]

Conflicts.

I have approved but I am not a maintainer. Overall, I would feel more comfortable if we used the six module (or futurize) instead of inventing our own from scratch. The linter directives in gyp/pylib/gyp/compat.py are just as a result of a refusal to follow Python porting best practice use feature detection instead of version detection. The map()s and filter()s would be faster and more readable as list comprehensions.

[eirnym]

@cclauss I feel the same about six, but I'm not a maintainer to make a decision about additional dependency

Checking Python version is significantly faster and stable enough and it's used even in pip and six.

about map and filter. I changed as little as it was possible. "this is not making code clean and fast, but only a compatibility in mind". if you start make this optimisations, you'll end up with rewriting the whole code from scratch leaving only required features. This is a huge work and I'm not a person to take it in 100%.

[eirnym]

And please, remember that we have no tests. npm run test is not sufficient here.

[eirnym]

One more thing about compatibility and Python version checking: this is only about syntax and builtins checking. For Python2 you can't use async/await as well as for Python 3 you can't use print statement and very old variant of raising. With imports it's try-except impoting which is basically feautre-checking and never monkey-patching.

[eirnym]

@refack almost 2 weeks passed since I done this work. Please, tell me what else needs to be done here?

[cclauss]

Conflicts :-(

[eirnym]

@cclauss it's easy to solve code conflicts, but I don't see any intention from maintainers to merge these changes anyway.

[eirnym]

I gave up on this branch. I have a copy if anybody would be intereseted in merging it.

[eirnym]

C

Checklist

• npm install && npm test passes
• tests are included
• documentation is changed or added
• commit message follows commit guidelines

Description of change

Continuation of #1335 and #1150. Please, review, comment, add more changes and fixes :) All changes are splitted, as some of them are massive.

UPDATE 1
Now I'm fighting with test/test-find-python.js file to allow it even to start testing in Python3-only environment.

UPDATE 2
Tests are passing, but it's possible that I'd miss something

[cclauss]

Great stuff... Just two minor change requests. Thanks.

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

extracted checks to compat module

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

There's a reason for it in this case: exception -> substitution will run much slower than knowlege of version where this name doesn't exist. This is also used in six and works well.

[eirnym]

so I'd prefer to leave this kind of check for types and for a dictionary. But I can change if you'd insist on it.

[refack]

I'm going to land @rodrigc's #1150, then I'll rebase this to see what's left.

[eirnym]

@refack it won't work as I found a lot more issues in code to be ported

[eirnym]

This doesn't mean, his work is useless.

[eirnym]

@refack@cclauss Could you have a chance to know what these variables should be? I haven't found any traces of them in functions they are and how to fix these lines.

./gyp/pylib/gyp/xcode_emulation.py:846:9: F821 undefined name 'cflags'
cflags.append('-F' + platform_root + '/Developer/Library/Frameworks/')
^
./gyp/pylib/gyp/generator/make.py:1645:50: F821 undefined name 'target'
print("WARNING: no output for", self.type, target)
^
2 F821 undefined name 'target'
2

[cclauss]

See #1333 and #1334 My sense on cflags is that it should be ldflags in this context but not 100% certain.

[refack]

it won't work as I found a lot more issues in code to be ported

As long as it passes CI, I take it as a sign it doesn't regress. I assumed that work was not done yet to get actual py3 compat.
Let's see is we can get everything done in this PR.

[eirnym]

My PR also passes CI, but not finished :D There's many things to fix

Before you try to merge it, I'd like to finish some other checks. I'll let you know

[eirnym]

Merged and I've done for today.

@cclauss#1333 is included

Unresolved questions for the next run are:

• sorted() and cmp=
• input.py: compile.ast -> ast. It's about to rewrite main part of the module.
• optparse may be dropped in a foreseeable future
• mac_tool uses veeery old version of plistlib and implies that pyobjc is installed (which may be not if we install python from MacPorts/Homebrew)
• I crashed pylint

[eirnym]

@refack I finished, including merging

[eirnym]

@cclauss for this run I used pylint --disable=E1125,E0702,E0702,E1701,E1124,C,W0311 pylib after flake8

[refack]

@eirnym I've rebased you branch. Before you continue, take a tag, and pull with --rebase, so you can compare what changed.

[eirnym]

you lost at least a half of my changes during rebasing. Merge would be always easier.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

We keep a linier git history, so we rebase. Some changes were incompatible with #1572 so I did not base them.

[eirnym]

If you'd tell me before, It would be just easier to rebase them manually. Now I repeat my work.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

Sorry. I'm juggling several branches, and reviewing a ton of code...

[eirnym]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

[refack]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

Sorry again.
There is no rush... this could wait a few more days.

Thank you for the dedication.

[eirnym]

Next possible day will be in a few days and I don't want to forget and start all things over again.

[eirnym]

I restored changes. Commit is a bit bulky, but it does it's job right.

[refack]

Thank you very much 🥇

[eirnym]

So what the current state of the PR?

[eirnym]

@refack@cclauss

[cclauss]

Conflicts.

I have approved but I am not a maintainer. Overall, I would feel more comfortable if we used the six module (or futurize) instead of inventing our own from scratch. The linter directives in gyp/pylib/gyp/compat.py are just as a result of a refusal to follow Python porting best practice use feature detection instead of version detection. The map()s and filter()s would be faster and more readable as list comprehensions.

[eirnym]

@cclauss I feel the same about six, but I'm not a maintainer to make a decision about additional dependency

Checking Python version is significantly faster and stable enough and it's used even in pip and six.

about map and filter. I changed as little as it was possible. "this is not making code clean and fast, but only a compatibility in mind". if you start make this optimisations, you'll end up with rewriting the whole code from scratch leaving only required features. This is a huge work and I'm not a person to take it in 100%.

[eirnym]

And please, remember that we have no tests. npm run test is not sufficient here.

[eirnym]

One more thing about compatibility and Python version checking: this is only about syntax and builtins checking. For Python2 you can't use async/await as well as for Python 3 you can't use print statement and very old variant of raising. With imports it's try-except impoting which is basically feautre-checking and never monkey-patching.

[eirnym]

@refack almost 2 weeks passed since I done this work. Please, tell me what else needs to be done here?

[cclauss]

Conflicts :-(

[eirnym]

@cclauss it's easy to solve code conflicts, but I don't see any intention from maintainers to merge these changes anyway.

[eirnym]

I gave up on this branch. I have a copy if anybody would be intereseted in merging it.

[eirnym]

C

Checklist

• npm install && npm test passes
• tests are included
• documentation is changed or added
• commit message follows commit guidelines

Description of change

Continuation of #1335 and #1150. Please, review, comment, add more changes and fixes :) All changes are splitted, as some of them are massive.

UPDATE 1
Now I'm fighting with test/test-find-python.js file to allow it even to start testing in Python3-only environment.

UPDATE 2
Tests are passing, but it's possible that I'd miss something

[cclauss]

Great stuff... Just two minor change requests. Thanks.

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

extracted checks to compat module

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

There's a reason for it in this case: exception -> substitution will run much slower than knowlege of version where this name doesn't exist. This is also used in six and works well.

[eirnym]

so I'd prefer to leave this kind of check for types and for a dictionary. But I can change if you'd insist on it.

[refack]

I'm going to land @rodrigc's #1150, then I'll rebase this to see what's left.

[eirnym]

@refack it won't work as I found a lot more issues in code to be ported

[eirnym]

This doesn't mean, his work is useless.

[eirnym]

@refack@cclauss Could you have a chance to know what these variables should be? I haven't found any traces of them in functions they are and how to fix these lines.

./gyp/pylib/gyp/xcode_emulation.py:846:9: F821 undefined name 'cflags'
cflags.append('-F' + platform_root + '/Developer/Library/Frameworks/')
^
./gyp/pylib/gyp/generator/make.py:1645:50: F821 undefined name 'target'
print("WARNING: no output for", self.type, target)
^
2 F821 undefined name 'target'
2

[cclauss]

See #1333 and #1334 My sense on cflags is that it should be ldflags in this context but not 100% certain.

[refack]

it won't work as I found a lot more issues in code to be ported

As long as it passes CI, I take it as a sign it doesn't regress. I assumed that work was not done yet to get actual py3 compat.
Let's see is we can get everything done in this PR.

[eirnym]

My PR also passes CI, but not finished :D There's many things to fix

Before you try to merge it, I'd like to finish some other checks. I'll let you know

[eirnym]

Merged and I've done for today.

@cclauss#1333 is included

Unresolved questions for the next run are:

• sorted() and cmp=
• input.py: compile.ast -> ast. It's about to rewrite main part of the module.
• optparse may be dropped in a foreseeable future
• mac_tool uses veeery old version of plistlib and implies that pyobjc is installed (which may be not if we install python from MacPorts/Homebrew)
• I crashed pylint

[eirnym]

@refack I finished, including merging

[eirnym]

@cclauss for this run I used pylint --disable=E1125,E0702,E0702,E1701,E1124,C,W0311 pylib after flake8

[refack]

@eirnym I've rebased you branch. Before you continue, take a tag, and pull with --rebase, so you can compare what changed.

[eirnym]

you lost at least a half of my changes during rebasing. Merge would be always easier.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

We keep a linier git history, so we rebase. Some changes were incompatible with #1572 so I did not base them.

[eirnym]

If you'd tell me before, It would be just easier to rebase them manually. Now I repeat my work.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

Sorry. I'm juggling several branches, and reviewing a ton of code...

[eirnym]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

[refack]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

Sorry again.
There is no rush... this could wait a few more days.

Thank you for the dedication.

[eirnym]

Next possible day will be in a few days and I don't want to forget and start all things over again.

[eirnym]

I restored changes. Commit is a bit bulky, but it does it's job right.

[refack]

Thank you very much 🥇

[eirnym]

So what the current state of the PR?

[eirnym]

@refack@cclauss

[cclauss]

Conflicts.

I have approved but I am not a maintainer. Overall, I would feel more comfortable if we used the six module (or futurize) instead of inventing our own from scratch. The linter directives in gyp/pylib/gyp/compat.py are just as a result of a refusal to follow Python porting best practice use feature detection instead of version detection. The map()s and filter()s would be faster and more readable as list comprehensions.

[eirnym]

@cclauss I feel the same about six, but I'm not a maintainer to make a decision about additional dependency

Checking Python version is significantly faster and stable enough and it's used even in pip and six.

about map and filter. I changed as little as it was possible. "this is not making code clean and fast, but only a compatibility in mind". if you start make this optimisations, you'll end up with rewriting the whole code from scratch leaving only required features. This is a huge work and I'm not a person to take it in 100%.

[eirnym]

And please, remember that we have no tests. npm run test is not sufficient here.

[eirnym]

One more thing about compatibility and Python version checking: this is only about syntax and builtins checking. For Python2 you can't use async/await as well as for Python 3 you can't use print statement and very old variant of raising. With imports it's try-except impoting which is basically feautre-checking and never monkey-patching.

[eirnym]

@refack almost 2 weeks passed since I done this work. Please, tell me what else needs to be done here?

[cclauss]

Conflicts :-(

[eirnym]

@cclauss it's easy to solve code conflicts, but I don't see any intention from maintainers to merge these changes anyway.

[eirnym]

I gave up on this branch. I have a copy if anybody would be intereseted in merging it.

[eirnym]

C

Checklist

• npm install && npm test passes
• tests are included
• documentation is changed or added
• commit message follows commit guidelines

Description of change

Continuation of #1335 and #1150. Please, review, comment, add more changes and fixes :) All changes are splitted, as some of them are massive.

UPDATE 1
Now I'm fighting with test/test-find-python.js file to allow it even to start testing in Python3-only environment.

UPDATE 2
Tests are passing, but it's possible that I'd miss something

[cclauss]

Great stuff... Just two minor change requests. Thanks.

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

extracted checks to compat module

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

There's a reason for it in this case: exception -> substitution will run much slower than knowlege of version where this name doesn't exist. This is also used in six and works well.

[eirnym]

so I'd prefer to leave this kind of check for types and for a dictionary. But I can change if you'd insist on it.

[refack]

I'm going to land @rodrigc's #1150, then I'll rebase this to see what's left.

[eirnym]

@refack it won't work as I found a lot more issues in code to be ported

[eirnym]

This doesn't mean, his work is useless.

[eirnym]

@refack@cclauss Could you have a chance to know what these variables should be? I haven't found any traces of them in functions they are and how to fix these lines.

./gyp/pylib/gyp/xcode_emulation.py:846:9: F821 undefined name 'cflags'
cflags.append('-F' + platform_root + '/Developer/Library/Frameworks/')
^
./gyp/pylib/gyp/generator/make.py:1645:50: F821 undefined name 'target'
print("WARNING: no output for", self.type, target)
^
2 F821 undefined name 'target'
2

[cclauss]

See #1333 and #1334 My sense on cflags is that it should be ldflags in this context but not 100% certain.

[refack]

it won't work as I found a lot more issues in code to be ported

As long as it passes CI, I take it as a sign it doesn't regress. I assumed that work was not done yet to get actual py3 compat.
Let's see is we can get everything done in this PR.

[eirnym]

My PR also passes CI, but not finished :D There's many things to fix

Before you try to merge it, I'd like to finish some other checks. I'll let you know

[eirnym]

Merged and I've done for today.

@cclauss#1333 is included

Unresolved questions for the next run are:

• sorted() and cmp=
• input.py: compile.ast -> ast. It's about to rewrite main part of the module.
• optparse may be dropped in a foreseeable future
• mac_tool uses veeery old version of plistlib and implies that pyobjc is installed (which may be not if we install python from MacPorts/Homebrew)
• I crashed pylint

[eirnym]

@refack I finished, including merging

[eirnym]

@cclauss for this run I used pylint --disable=E1125,E0702,E0702,E1701,E1124,C,W0311 pylib after flake8

[refack]

@eirnym I've rebased you branch. Before you continue, take a tag, and pull with --rebase, so you can compare what changed.

[eirnym]

you lost at least a half of my changes during rebasing. Merge would be always easier.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

We keep a linier git history, so we rebase. Some changes were incompatible with #1572 so I did not base them.

[eirnym]

If you'd tell me before, It would be just easier to rebase them manually. Now I repeat my work.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

Sorry. I'm juggling several branches, and reviewing a ton of code...

[eirnym]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

[refack]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

Sorry again.
There is no rush... this could wait a few more days.

Thank you for the dedication.

[eirnym]

Next possible day will be in a few days and I don't want to forget and start all things over again.

[eirnym]

I restored changes. Commit is a bit bulky, but it does it's job right.

[refack]

Thank you very much 🥇

[eirnym]

So what the current state of the PR?

[eirnym]

@refack@cclauss

[cclauss]

Conflicts.

I have approved but I am not a maintainer. Overall, I would feel more comfortable if we used the six module (or futurize) instead of inventing our own from scratch. The linter directives in gyp/pylib/gyp/compat.py are just as a result of a refusal to follow Python porting best practice use feature detection instead of version detection. The map()s and filter()s would be faster and more readable as list comprehensions.

[eirnym]

@cclauss I feel the same about six, but I'm not a maintainer to make a decision about additional dependency

Checking Python version is significantly faster and stable enough and it's used even in pip and six.

about map and filter. I changed as little as it was possible. "this is not making code clean and fast, but only a compatibility in mind". if you start make this optimisations, you'll end up with rewriting the whole code from scratch leaving only required features. This is a huge work and I'm not a person to take it in 100%.

[eirnym]

And please, remember that we have no tests. npm run test is not sufficient here.

[eirnym]

One more thing about compatibility and Python version checking: this is only about syntax and builtins checking. For Python2 you can't use async/await as well as for Python 3 you can't use print statement and very old variant of raising. With imports it's try-except impoting which is basically feautre-checking and never monkey-patching.

[eirnym]

@refack almost 2 weeks passed since I done this work. Please, tell me what else needs to be done here?

[cclauss]

Conflicts :-(

[eirnym]

@cclauss it's easy to solve code conflicts, but I don't see any intention from maintainers to merge these changes anyway.

[eirnym]

I gave up on this branch. I have a copy if anybody would be intereseted in merging it.

[eirnym]

C

Checklist

• npm install && npm test passes
• tests are included
• documentation is changed or added
• commit message follows commit guidelines

Description of change

Continuation of #1335 and #1150. Please, review, comment, add more changes and fixes :) All changes are splitted, as some of them are massive.

UPDATE 1
Now I'm fighting with test/test-find-python.js file to allow it even to start testing in Python3-only environment.

UPDATE 2
Tests are passing, but it's possible that I'd miss something

[cclauss]

Great stuff... Just two minor change requests. Thanks.

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

extracted checks to compat module

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

There's a reason for it in this case: exception -> substitution will run much slower than knowlege of version where this name doesn't exist. This is also used in six and works well.

[eirnym]

so I'd prefer to leave this kind of check for types and for a dictionary. But I can change if you'd insist on it.

[refack]

I'm going to land @rodrigc's #1150, then I'll rebase this to see what's left.

[eirnym]

@refack it won't work as I found a lot more issues in code to be ported

[eirnym]

This doesn't mean, his work is useless.

[eirnym]

@refack@cclauss Could you have a chance to know what these variables should be? I haven't found any traces of them in functions they are and how to fix these lines.

./gyp/pylib/gyp/xcode_emulation.py:846:9: F821 undefined name 'cflags'
cflags.append('-F' + platform_root + '/Developer/Library/Frameworks/')
^
./gyp/pylib/gyp/generator/make.py:1645:50: F821 undefined name 'target'
print("WARNING: no output for", self.type, target)
^
2 F821 undefined name 'target'
2

[cclauss]

See #1333 and #1334 My sense on cflags is that it should be ldflags in this context but not 100% certain.

[refack]

it won't work as I found a lot more issues in code to be ported

As long as it passes CI, I take it as a sign it doesn't regress. I assumed that work was not done yet to get actual py3 compat.
Let's see is we can get everything done in this PR.

[eirnym]

My PR also passes CI, but not finished :D There's many things to fix

Before you try to merge it, I'd like to finish some other checks. I'll let you know

[eirnym]

Merged and I've done for today.

@cclauss#1333 is included

Unresolved questions for the next run are:

• sorted() and cmp=
• input.py: compile.ast -> ast. It's about to rewrite main part of the module.
• optparse may be dropped in a foreseeable future
• mac_tool uses veeery old version of plistlib and implies that pyobjc is installed (which may be not if we install python from MacPorts/Homebrew)
• I crashed pylint

[eirnym]

@refack I finished, including merging

[eirnym]

@cclauss for this run I used pylint --disable=E1125,E0702,E0702,E1701,E1124,C,W0311 pylib after flake8

[refack]

@eirnym I've rebased you branch. Before you continue, take a tag, and pull with --rebase, so you can compare what changed.

[eirnym]

you lost at least a half of my changes during rebasing. Merge would be always easier.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

We keep a linier git history, so we rebase. Some changes were incompatible with #1572 so I did not base them.

[eirnym]

If you'd tell me before, It would be just easier to rebase them manually. Now I repeat my work.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

Sorry. I'm juggling several branches, and reviewing a ton of code...

[eirnym]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

[refack]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

Sorry again.
There is no rush... this could wait a few more days.

Thank you for the dedication.

[eirnym]

Next possible day will be in a few days and I don't want to forget and start all things over again.

[eirnym]

I restored changes. Commit is a bit bulky, but it does it's job right.

[refack]

Thank you very much 🥇

[eirnym]

So what the current state of the PR?

[eirnym]

@refack@cclauss

[cclauss]

Conflicts.

I have approved but I am not a maintainer. Overall, I would feel more comfortable if we used the six module (or futurize) instead of inventing our own from scratch. The linter directives in gyp/pylib/gyp/compat.py are just as a result of a refusal to follow Python porting best practice use feature detection instead of version detection. The map()s and filter()s would be faster and more readable as list comprehensions.

[eirnym]

@cclauss I feel the same about six, but I'm not a maintainer to make a decision about additional dependency

Checking Python version is significantly faster and stable enough and it's used even in pip and six.

about map and filter. I changed as little as it was possible. "this is not making code clean and fast, but only a compatibility in mind". if you start make this optimisations, you'll end up with rewriting the whole code from scratch leaving only required features. This is a huge work and I'm not a person to take it in 100%.

[eirnym]

And please, remember that we have no tests. npm run test is not sufficient here.

[eirnym]

One more thing about compatibility and Python version checking: this is only about syntax and builtins checking. For Python2 you can't use async/await as well as for Python 3 you can't use print statement and very old variant of raising. With imports it's try-except impoting which is basically feautre-checking and never monkey-patching.

[eirnym]

@refack almost 2 weeks passed since I done this work. Please, tell me what else needs to be done here?

[cclauss]

Conflicts :-(

[eirnym]

@cclauss it's easy to solve code conflicts, but I don't see any intention from maintainers to merge these changes anyway.

[eirnym]

I gave up on this branch. I have a copy if anybody would be intereseted in merging it.

[eirnym]

C

Checklist

• npm install && npm test passes
• tests are included
• documentation is changed or added
• commit message follows commit guidelines

Description of change

Continuation of #1335 and #1150. Please, review, comment, add more changes and fixes :) All changes are splitted, as some of them are massive.

UPDATE 1
Now I'm fighting with test/test-find-python.js file to allow it even to start testing in Python3-only environment.

UPDATE 2
Tests are passing, but it's possible that I'd miss something

[cclauss]

Great stuff... Just two minor change requests. Thanks.

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

extracted checks to compat module

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

There's a reason for it in this case: exception -> substitution will run much slower than knowlege of version where this name doesn't exist. This is also used in six and works well.

[eirnym]

so I'd prefer to leave this kind of check for types and for a dictionary. But I can change if you'd insist on it.

[refack]

I'm going to land @rodrigc's #1150, then I'll rebase this to see what's left.

[eirnym]

@refack it won't work as I found a lot more issues in code to be ported

[eirnym]

This doesn't mean, his work is useless.

[eirnym]

@refack@cclauss Could you have a chance to know what these variables should be? I haven't found any traces of them in functions they are and how to fix these lines.

./gyp/pylib/gyp/xcode_emulation.py:846:9: F821 undefined name 'cflags'
cflags.append('-F' + platform_root + '/Developer/Library/Frameworks/')
^
./gyp/pylib/gyp/generator/make.py:1645:50: F821 undefined name 'target'
print("WARNING: no output for", self.type, target)
^
2 F821 undefined name 'target'
2

[cclauss]

See #1333 and #1334 My sense on cflags is that it should be ldflags in this context but not 100% certain.

[refack]

it won't work as I found a lot more issues in code to be ported

As long as it passes CI, I take it as a sign it doesn't regress. I assumed that work was not done yet to get actual py3 compat.
Let's see is we can get everything done in this PR.

[eirnym]

My PR also passes CI, but not finished :D There's many things to fix

Before you try to merge it, I'd like to finish some other checks. I'll let you know

[eirnym]

Merged and I've done for today.

@cclauss#1333 is included

Unresolved questions for the next run are:

• sorted() and cmp=
• input.py: compile.ast -> ast. It's about to rewrite main part of the module.
• optparse may be dropped in a foreseeable future
• mac_tool uses veeery old version of plistlib and implies that pyobjc is installed (which may be not if we install python from MacPorts/Homebrew)
• I crashed pylint

[eirnym]

@refack I finished, including merging

[eirnym]

@cclauss for this run I used pylint --disable=E1125,E0702,E0702,E1701,E1124,C,W0311 pylib after flake8

[refack]

@eirnym I've rebased you branch. Before you continue, take a tag, and pull with --rebase, so you can compare what changed.

[eirnym]

you lost at least a half of my changes during rebasing. Merge would be always easier.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

We keep a linier git history, so we rebase. Some changes were incompatible with #1572 so I did not base them.

[eirnym]

If you'd tell me before, It would be just easier to rebase them manually. Now I repeat my work.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

Sorry. I'm juggling several branches, and reviewing a ton of code...

[eirnym]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

[refack]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

Sorry again.
There is no rush... this could wait a few more days.

Thank you for the dedication.

[eirnym]

Next possible day will be in a few days and I don't want to forget and start all things over again.

[eirnym]

I restored changes. Commit is a bit bulky, but it does it's job right.

[refack]

Thank you very much 🥇

[eirnym]

So what the current state of the PR?

[eirnym]

@refack@cclauss

[cclauss]

Conflicts.

I have approved but I am not a maintainer. Overall, I would feel more comfortable if we used the six module (or futurize) instead of inventing our own from scratch. The linter directives in gyp/pylib/gyp/compat.py are just as a result of a refusal to follow Python porting best practice use feature detection instead of version detection. The map()s and filter()s would be faster and more readable as list comprehensions.

[eirnym]

@cclauss I feel the same about six, but I'm not a maintainer to make a decision about additional dependency

Checking Python version is significantly faster and stable enough and it's used even in pip and six.

about map and filter. I changed as little as it was possible. "this is not making code clean and fast, but only a compatibility in mind". if you start make this optimisations, you'll end up with rewriting the whole code from scratch leaving only required features. This is a huge work and I'm not a person to take it in 100%.

[eirnym]

And please, remember that we have no tests. npm run test is not sufficient here.

[eirnym]

One more thing about compatibility and Python version checking: this is only about syntax and builtins checking. For Python2 you can't use async/await as well as for Python 3 you can't use print statement and very old variant of raising. With imports it's try-except impoting which is basically feautre-checking and never monkey-patching.

[eirnym]

@refack almost 2 weeks passed since I done this work. Please, tell me what else needs to be done here?

[cclauss]

Conflicts :-(

[eirnym]

@cclauss it's easy to solve code conflicts, but I don't see any intention from maintainers to merge these changes anyway.

[eirnym]

I gave up on this branch. I have a copy if anybody would be intereseted in merging it.

[eirnym]

C

Checklist

• npm install && npm test passes
• tests are included
• documentation is changed or added
• commit message follows commit guidelines

Description of change

Continuation of #1335 and #1150. Please, review, comment, add more changes and fixes :) All changes are splitted, as some of them are massive.

UPDATE 1
Now I'm fighting with test/test-find-python.js file to allow it even to start testing in Python3-only environment.

UPDATE 2
Tests are passing, but it's possible that I'd miss something

[cclauss]

Great stuff... Just two minor change requests. Thanks.

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

extracted checks to compat module

[cclauss]

On this one, can we please follow Python porting best practice use feature detection instead of version detection?

[eirnym]

There's a reason for it in this case: exception -> substitution will run much slower than knowlege of version where this name doesn't exist. This is also used in six and works well.

[eirnym]

so I'd prefer to leave this kind of check for types and for a dictionary. But I can change if you'd insist on it.

[refack]

I'm going to land @rodrigc's #1150, then I'll rebase this to see what's left.

[eirnym]

@refack it won't work as I found a lot more issues in code to be ported

[eirnym]

This doesn't mean, his work is useless.

[eirnym]

@refack@cclauss Could you have a chance to know what these variables should be? I haven't found any traces of them in functions they are and how to fix these lines.

./gyp/pylib/gyp/xcode_emulation.py:846:9: F821 undefined name 'cflags'
cflags.append('-F' + platform_root + '/Developer/Library/Frameworks/')
^
./gyp/pylib/gyp/generator/make.py:1645:50: F821 undefined name 'target'
print("WARNING: no output for", self.type, target)
^
2 F821 undefined name 'target'
2

[cclauss]

See #1333 and #1334 My sense on cflags is that it should be ldflags in this context but not 100% certain.

[refack]

it won't work as I found a lot more issues in code to be ported

As long as it passes CI, I take it as a sign it doesn't regress. I assumed that work was not done yet to get actual py3 compat.
Let's see is we can get everything done in this PR.

[eirnym]

My PR also passes CI, but not finished :D There's many things to fix

Before you try to merge it, I'd like to finish some other checks. I'll let you know

[eirnym]

Merged and I've done for today.

@cclauss#1333 is included

Unresolved questions for the next run are:

• sorted() and cmp=
• input.py: compile.ast -> ast. It's about to rewrite main part of the module.
• optparse may be dropped in a foreseeable future
• mac_tool uses veeery old version of plistlib and implies that pyobjc is installed (which may be not if we install python from MacPorts/Homebrew)
• I crashed pylint

[eirnym]

@refack I finished, including merging

[eirnym]

@cclauss for this run I used pylint --disable=E1125,E0702,E0702,E1701,E1124,C,W0311 pylib after flake8

[refack]

@eirnym I've rebased you branch. Before you continue, take a tag, and pull with --rebase, so you can compare what changed.

[eirnym]

you lost at least a half of my changes during rebasing. Merge would be always easier.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

We keep a linier git history, so we rebase. Some changes were incompatible with #1572 so I did not base them.

[eirnym]

If you'd tell me before, It would be just easier to rebase them manually. Now I repeat my work.

[refack]

you lost at least a half of my changes during rebasing. Merge would be always easier.

Sorry. I'm juggling several branches, and reviewing a ton of code...

[eirnym]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

[refack]

I understand, but tell about this to a developer to save their time. It's already half past midnight and I'm restoring changes for an hour by now instead going to sleep.

Sorry again.
There is no rush... this could wait a few more days.

Thank you for the dedication.

[eirnym]

Next possible day will be in a few days and I don't want to forget and start all things over again.

[eirnym]

I restored changes. Commit is a bit bulky, but it does it's job right.

[refack]

Thank you very much 🥇

[eirnym]

So what the current state of the PR?

[eirnym]

@refack@cclauss

[cclauss]

Conflicts.

I have approved but I am not a maintainer. Overall, I would feel more comfortable if we used the six module (or futurize) instead of inventing our own from scratch. The linter directives in gyp/pylib/gyp/compat.py are just as a result of a refusal to follow Python porting best practice use feature detection instead of version detection. The map()s and filter()s would be faster and more readable as list comprehensions.

[eirnym]

@cclauss I feel the same about six, but I'm not a maintainer to make a decision about additional dependency

Checking Python version is significantly faster and stable enough and it's used even in pip and six.

about map and filter. I changed as little as it was possible. "this is not making code clean and fast, but only a compatibility in mind". if you start make this optimisations, you'll end up with rewriting the whole code from scratch leaving only required features. This is a huge work and I'm not a person to take it in 100%.

[eirnym]

And please, remember that we have no tests. npm run test is not sufficient here.

[eirnym]

One more thing about compatibility and Python version checking: this is only about syntax and builtins checking. For Python2 you can't use async/await as well as for Python 3 you can't use print statement and very old variant of raising. With imports it's try-except impoting which is basically feautre-checking and never monkey-patching.

[eirnym]

@refack almost 2 weeks passed since I done this work. Please, tell me what else needs to be done here?

[cclauss]

Conflicts :-(

[eirnym]

@cclauss it's easy to solve code conflicts, but I don't see any intention from maintainers to merge these changes anyway.

[eirnym]

I gave up on this branch. I have a copy if anybody would be intereseted in merging it.