[SPARK-25270] lint-python: Add flake8 to find syntax errors and undefined names

[cclauss]

What changes were proposed in this pull request?

Add flake8 tests to find Python syntax errors and undefined names.

E901,E999,F821,F822,F823 are the "showstopper" flake8 issues that can halt the runtime with a SyntaxError, NameError, etc. Most other flake8 issues are merely "style violations" -- useful for readability but they do not effect runtime safety.

• F821: undefined name name
• F822: undefined name name in __all__
• F823: local variable name referenced before assignment
• E901: SyntaxError or IndentationError
• E999: SyntaxError -- failed to compile a file into an Abstract Syntax Tree

How was this patch tested?

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
$ flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics

Please review http://spark.apache.org/contributing.html before opening a pull request.

[cclauss]

These tests should fail until #22265 or similar is merged.

[HyukjinKwon]

Can you file a JIRA please?

[viirya]

Can you also format the PR description to follow the template?

[cclauss]

@HyukjinKwon Done.
@viirya Done.

[HyukjinKwon]

ok to test

[SparkQA]

Test build #95447 has finished for PR 22266 at commit 1291198.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #95501 has finished for PR 22266 at commit c8c0543.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

Thanks for doing the cleanup as well as the follow up PR to make sure we don't regress. Some minor questions too :)

[SparkQA]

Test build #95555 has finished for PR 22266 at commit b87c49e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #95556 has finished for PR 22266 at commit 786d0c9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

This LGTM but my only problem is we introduced another undefined name in the java_gateway in the meantime. I'll fix that today and merge this PR.

[holdenk]

Oh wait that was fixed in master I just had this branch checked out. Verified flake8 cmd against f96a8bf so we should be fine to merge to master without breaking anything. LGTM

[holdenk]

Merged to master, it's not in branch-2.4 (although if folks think in belongs there I'm open to that). I haven't closed the JIRA yet because we don't have the version tag in JIRA for the next release after 2.4.