Components Test Management

The extension provides testing for Python code through one of the supported frameworks. This involves a variety of capabilities in addition to just running tests. The following frameworks are supported:

• stdlib unittest
• pytest
• nose2

Functionality

operations

• install selected test framework if not installed
• test discovery
• run all/selected tests & suites
• debug all/selected tests & suites
• breakpoints while running tests
• show (high level) test results

Note that the active Python interpreter is used for all testing operations.

resources

• (somewhat) granular test framework config settings
• dedicated output log

commands

• "Python: Discover Unit Tests"
• "Python: Run All Unit Tests"
• "Python: Run Unit Test Method..."
• "Python: View Unit Test Output"

UI

• status bar element
  • shows test results
  • triggers quick pick for testing-related operations
• pop-up (& quick pick) to configure test framework
• code lens on tests and suites:
  • run/debug
  • results
• output panel: "Python Test Log"
• TBD: Test Explorer activity (top-level VSC element)

config settings

• "python.unitTest.cwd"
• "python.unitTest.debugPort"
• "python.unitTest.nosetestArgs"
• "python.unitTest.nosetestsEnabled"
• "python.unitTest.nosetestPath"
• "python.unitTest.promptToConfigure"
• "python.unitTest.pyTestArgs"
• "python.unitTest.pyTestEnabled"
• "python.unitTest.pyTestPath"
• "python.unitTest.unittestArgs"
• "python.unitTest.unittestEnabled"
• "python.unitTest.autoTestDiscoverOnSaveEnabled"

Extension Code

codebase structure

All test-related code is contained entirely under a single directory, src/client/unittests/:

src/client/unittest
├── codeLenses
├── common
│ ├── managers
│ ├── services
│ └── testVisitors
├── display
├── nosetest
│ └── services
├── pytest
│ └── services
└── unittest
└── services

files: external API

src/client/unittest
├── serviceRegistry.ts
└── types.ts

files: top-level code

src/client/unittest
├── common
│ ├── argumentsHelper.ts
│ ├── constants.ts
│ ├── debugLauncher.ts
│ ├── managers
│ │ ├── baseTestManager.ts
│ │ └── testConfigurationManager.ts
│ ├── runner.ts
│ ├── services
│ │ ├── configSettingService.ts
│ │ ├── storageService.ts
│ │ ├── testManagerService.ts
│ │ ├── testResultsService.ts
│ │ ├── unitTestDiagnosticService.ts
│ │ └── workspaceTestManagerService.ts
│ ├── testUtils.ts
│ ├── testVisitors
│ │ ├── flatteningVisitor.ts
│ │ ├── folderGenerationVisitor.ts
│ │ └── resultResetVisitor.ts
│ ├── types.ts
│ └── xUnitParser.ts
└── main.ts

files: configuration:

src/client/unittest
├── configurationFactory.ts
├── configuration.ts
├── nosetest
│ └── testConfigurationManager.ts
├── pytest
│ └── testConfigurationManager.ts
└── unittest
└── testConfigurationManager.ts

files: testing frameworks:

src/client/unittest
├── nosetest
│ ├── main.ts
│ ├── runner.ts
│ └── services
│ ├── argsService.ts
│ ├── discoveryService.ts
│ └── parserService.ts
├── pytest
│ ├── main.ts
│ ├── runner.ts
│ └── services
│ ├── argsService.ts
│ ├── discoveryService.ts
│ ├── parserService.ts
│ └── testMessageService.ts
└── unittest
├── helper.ts
├── main.ts
├── runner.ts
├── services
│ ├── argsService.ts
│ ├── discoveryService.ts
│ └── parserService.ts
└── socketServer.ts

files: UI features

src/client/unittest
├── codeLenses
│ ├── main.ts
│ └── testFiles.ts
└── display
├── main.ts
└── picker.ts

components

• config
• test discovery
• code lens
• picker
• runner

Adapter Script

• source: pythonFiles/testing_tools/adapter/__main__.py
• invocation:
  • [PYTHON] [SCRIPT] discover pytest [ARGS] -- [PYTEST ARGS]
  • SCRIPT (any python): pythonFiles/testing_tools/run_adapter.py
  • SCRIPT (python3-only): -m pythonFiles.testing_tools.adapter
• args:
  • --simple - simplified output for use while debugging
  • --no-hide-stdio - do not hide pytest stdout/stderr during discover (useful when debugging)
  • --pretty - format output for readability
• implicit pytest args:
  • --collect-only

Output

OUTPUT: ROOTED_TESTS[]
ROOTED_TESTS: {
# Uniquely identifies test root.
"rootid": str
# Absolute path to test root.
"root": str
# The discovered files, suites, etc. under this root.
"parents": PARENT[]
# The discovered tests under this root.
"tests": TEST[]
}
PARENT: {
# Uniquely identifies the parent.
"id": str
# The kind of parent.
"kind": "folder"|"file"|"suite"|"function"|"subtest"
# A human-friendly identifier.
"name": str
# The parent's parent (i.e. in a tree from the root.
"parentid": str
}
TEST: {
# Uniquely identifies the test.
"id": str
# A human-friendly identifier.
"name": str
# The location of the test's source code.
"source": str ("<FILENAME>:<LINENO>")
# Any supported markers associated with the test.
"markers": ("skip"|"skip-if"|"expected-failure")[]
# The ID of the test's parent.
"parentid": str
}

Notes:

• IDs may be used to identify a test or parent to the test framework (e.g. to run specific tests)
• every child of the root will have the "rootid" as its "parentid"
• supported test markers are framework-agnostic but correspond to framework-specific ones
• a test is considered a "subtest" if its parent is a function or a subtest
• a subtest is a test that is effectively a parameterization of a more general test
  • in pytest parameterized tests are the only kind of subtest
• a subtest may be a parent if there is a nested subtest in it
  • this is not supported in pytest, but is in unittest

Example:

[{
"rootid": ".",
"root": "/x/y/z",
"parents": [{
"id": "./test_spam.py",
"kind": "file",
"name": "test_spam.py",
"parentid": "."
}, {
"id": "./test_spam.py::SpamTests",
"kind": "suite",
"name": "SpamTests",
"parentid": "./test_spam.py"
},
"tests" [{
"id": "./test_spam.py::test_all",
"name": "test_all",
"source": "test_spam.py:11",
"markers": ["skip", "expected-failure"],
"parentid": "./test_spam.py"
}, {
"id": "./test_spam.py::SpamTests::test_spam1",
"name": "test_spam1",
"source": "test_spam.py:23",
"markers": ["skip"],
"parentid": "./test_spam.py::SpamTests"
}]
}]