Dump importer_yielder in favor of IMPORTER_REGISTRY and drop Etags by Hritik14 · Pull Request #600 · aboutcode-org/vulnerablecode

Hritik14 · 2022-01-26T22:57:38Z

Fixes: refactor importer_yielder.py #501
Fixes: Disable Etag #321
Fixes: Consistent naming of Importer instead of DataSource #601

The name DataSource and Importer are majorly used interchangeably. This needs to be fixed in a different PR.

pombredanne

Thanks! I have a few suggestions and ... Could you add tests?

IMPORTER_REGISTRY is neater and does not do any magical string -> object conversion. The registry looks more in sync with improvers. Fixes: aboutcode-org#501 Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Etags are meant for transient usage in browsers and are not meant for any long term usage. Fixes: aboutcode-org#321 Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

The name DataSource and Importer were majorly used interchangeably although they represent the same concept. Also, the importers now require a mandatory ``spdx_license_expression`` to run Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Same for improvers IMPROVER_REGISTRY -> IMPROVERS_REGISTRY Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Tests for: * manage.py import command * manage.py improve command * import_runner Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

We can use @classmethod and @Property together in python 3.9. Not so in 3.8 Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

pombredanne

Thanks... see a few final nits for your consideration.

pombredanne · 2022-02-07T13:27:23Z

-class GitDataSource(DataSource):
- CONFIG_CLASS = GitDataSourceConfiguration

+# TODO: Needs rewrite


May be an issue to track may work better?

This will be a part of #597

pombredanne · 2022-02-07T13:45:14Z

+ assert advisory_datas == ADVISORY_DATAS


-def test_ImportRunner_existing_package_and_new_vulnerability(db):


Could this test be kept somehow? to valid that we never will allow twice the same set of imported data?

This is done here
https://github.com/nexB/vulnerablecode/blob/c62f81f84c272686be972acebe15e2d2d6246e85/vulnerabilities/tests/test_import_runner.py#L80-L86

pombredanne · 2022-02-07T13:45:39Z

- assert vuln_refs[0].url == "https://example.com/with/more/info/CVE-2020-13371337"
-
-
-def test_ImportRunner_new_package_version_affected_by_existing_vulnerability(db):


IMHO this may be need to be kept

pombredanne · 2022-02-07T13:46:17Z

- assert impacted_package.vulnerability.vulnerability_id == "CVE-2020-13371337"
-
-
-# def test_ImportRunner_fixed_package_version_is_added(db):


Could this test have been of value? and need to be kept?

pombredanne · 2022-02-07T13:46:37Z

-# assert resolved_package.vulnerability.vulnerability_id == "CVE-2020-13371337"
-
-
-def test_ImportRunner_updated_vulnerability(db):


IMHO this may be need to be kept, moved to improvers?

All of these will be moved to improvers. Importers cannot test relationships. It will be a part of a new PR. Tracked in: #612

Signed-off-by: Hritik Vijay <hritikxx8@gmail.com> Co-authored-by: Philippe Ombredanne <pombredanne@gmail.com>

pombredanne

LGTM! 👍

Hritik14 force-pushed the migration/importer_yielder branch from 457c91f to 4a24a14Compare January 26, 2022 23:02

Hritik14 requested a review from pombredanne January 26, 2022 23:02

Hritik14 force-pushed the migration/importer_yielder branch from 4a24a14 to 7175362Compare January 26, 2022 23:05

pombredanne requested changes Jan 27, 2022
View reviewed changes

Hritik14 requested a review from pombredanne January 27, 2022 18:21

Hritik14 added this to the v30.0 milestone Feb 2, 2022

Hritik14 force-pushed the migration/importer_yielder branch 5 times, most recently from 89cdfee to 058b9a8Compare February 5, 2022 21:44

Hritik14 added 3 commits February 6, 2022 03:20

Dump importer_yielder in favor of IMPORTER_REGISTRY
7c3d7a8
IMPORTER_REGISTRY is neater and does not do any magical string -> object conversion. The registry looks more in sync with improvers. Fixes: aboutcode-org#501 Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Remove etags
12c842e
Etags are meant for transient usage in browsers and are not meant for any long term usage. Fixes: aboutcode-org#321 Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Refactor importer/improver registry
744259b
Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Hritik14 force-pushed the migration/importer_yielder branch from 058b9a8 to 0c4905eCompare February 5, 2022 21:58

Hritik14 added 6 commits February 6, 2022 03:31

Bulk rename DataSource -> Importer
f7f843a
The name DataSource and Importer were majorly used interchangeably although they represent the same concept. Also, the importers now require a mandatory ``spdx_license_expression`` to run Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Add dummy license to nginx importer
cf836e2
Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

IMPORTER_REGISTRY -> IMPORTERS_REGISTRY
5dc1281
Same for improvers IMPROVER_REGISTRY -> IMPROVERS_REGISTRY Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Improve documentation/text
b1f204a
Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Add tests
d7bc7b4
Tests for: * manage.py import command * manage.py improve command * import_runner Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Use custom classproperty for python 3.8 compatibility
c62f81f
We can use @classmethod and @Property together in python 3.9. Not so in 3.8 Signed-off-by: Hritik Vijay <hritikxx8@gmail.com>

Hritik14 force-pushed the migration/importer_yielder branch from 0c4905e to c62f81fCompare February 5, 2022 22:04

pombredanne requested changes Feb 7, 2022
View reviewed changes

Resolve a few final nits
45cdaf5
Signed-off-by: Hritik Vijay <hritikxx8@gmail.com> Co-authored-by: Philippe Ombredanne <pombredanne@gmail.com>

Hritik14 force-pushed the migration/importer_yielder branch from 61023e6 to 45cdaf5Compare February 7, 2022 14:18

pombredanne approved these changes Feb 7, 2022
View reviewed changes

pombredanne merged commit 4ebaa48 into aboutcode-org:mainFeb 7, 2022

		assert advisory_datas == ADVISORY_DATAS


		def test_ImportRunner_existing_package_and_new_vulnerability(db):

		assert vuln_refs[0].url == "https://example.com/with/more/info/CVE-2020-13371337"


		def test_ImportRunner_new_package_version_affected_by_existing_vulnerability(db):

		assert impacted_package.vulnerability.vulnerability_id == "CVE-2020-13371337"


		# def test_ImportRunner_fixed_package_version_is_added(db):

		# assert resolved_package.vulnerability.vulnerability_id == "CVE-2020-13371337"


		def test_ImportRunner_updated_vulnerability(db):

Uh oh!

Uh oh!

Conversation

Hritik14 commented Jan 26, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Hritik14 commented Jan 26, 2022•
edited
Loading