Skip to content

migrate current alpine importer to alpine importer-improver model - #623

Merged
TG1999 merged 2 commits into
aboutcode-org:mainfrom
TG1999:migration/alpine_linux
Mar 4, 2022
Merged

migrate current alpine importer to alpine importer-improver model#623
TG1999 merged 2 commits into
aboutcode-org:mainfrom
TG1999:migration/alpine_linux

Conversation

@TG1999

@TG1999TG1999 commented Feb 11, 2022

Copy link
Copy Markdown
Contributor

Signed-off-by: Tushar Goel tushar.goel.dav@gmail.com

@TG1999TG1999 changed the title migrate current alpine importer to alpine importer-improver modelmigrate current alpine importer to alpine importer-improver model #620Feb 11, 2022
@TG1999TG1999 changed the title migrate current alpine importer to alpine importer-improver model #620migrate current alpine importer to alpine importer-improver modelFeb 11, 2022
@TG1999

TG1999 commented Feb 11, 2022

Copy link
Copy Markdown
ContributorAuthor

for issue #620

@Hritik14

Hritik14 commented Feb 13, 2022

Copy link
Copy Markdown
Collaborator

@TG1999 As the development in this branch is going on and we wanted to move ahead with #476, rebasing/merging this with/from main will likely cause merge conflicts. Please accept the incoming changes for the import statements.
Alternatively, give kdiff3 a try for solving merge conflicts, it's really smart.

@TG1999
TG1999force-pushed the migration/alpine_linux branch from 7baa3d2 to cad46c2CompareFebruary 14, 2022 10:29
@TG1999

TG1999 commented Feb 14, 2022

Copy link
Copy Markdown
ContributorAuthor

@Hritik14 Resolved merge conflicts

@TG1999
TG1999force-pushed the migration/alpine_linux branch 2 times, most recently from b68f076 to 3360eb3CompareFebruary 17, 2022 12:40
@TG1999

TG1999 commented Feb 17, 2022

Copy link
Copy Markdown
ContributorAuthor

also solves issue #628 and #629

@TG1999
TG1999force-pushed the migration/alpine_linux branch 2 times, most recently from 50ba90a to 5ff44feCompareFebruary 17, 2022 12:56
@Hritik14Hritik14 mentioned this pull request Feb 17, 2022
Closed
@TG1999

TG1999 commented Feb 17, 2022

Copy link
Copy Markdown
ContributorAuthor

@pombredanne@Hritik14@sbs2001 Please check my current approach for the importer, if this looks good I will proceed with writing tests for this

pombredanne
pombredanne requested changes Feb 18, 2022
View reviewed changes

@pombredannepombredanne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are a few nits for your review!

Comment threadvulnerabilities/importer.py Outdated
Comment threadvulnerabilities/importer.py Outdated
Comment threadvulnerabilities/importer.py Outdated
Comment threadvulnerabilities/importer.py Outdated
Comment threadvulnerabilities/importer.py Outdated
def __post_init__(self):
if self.package.version:
if self.package.version or not (self.affected_version_range or self.fixed_version):
raise ValueError

@pombredannepombredanneFeb 18, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a message

Comment threadvulnerabilities/importers/alpine_linux.py Outdated
return advisories
# TODO: Handle the CVE-????-????? case
yield AdvisoryData(
summary="",

@pombredannepombredanneFeb 18, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why an empty summary? Is is mandatory?

Comment threadvulnerabilities/importers/alpine_linux.py Outdated
),
fixed_version=AlpineLinuxVersion(version),
)
for arch in archs

@pombredannepombredanneFeb 18, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if here is no archs?

Comment threadvulnerabilities/importers/alpine_linux.py Outdated
yield AdvisoryData(
summary="",
references=references,
affected_packages=[

@pombredannepombredanneFeb 18, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create this before in a separate easier to read loop

Comment threadvulnerabilities/importers/alpine_linux.py Outdated
)
for arch in archs
],
aliases=[vuln_ids[0] if is_cve(vuln_ids[0]) else ""],

@pombredannepombredanneFeb 18, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not return a list of aliases with an empty string in it.

Comment threadvulnerabilities/improvers/default.py Outdated


def get_exact_purls(affected_package: AffectedPackage) -> (List[PackageURL], PackageURL):
def get_exact_purls(affected_package: AffectedPackage) -> Tuple[List[PackageURL], PackageURL]:

@pombredannepombredanneFeb 18, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move these changes in a separate PR.

@TG1999
TG1999force-pushed the migration/alpine_linux branch from a9172e4 to 7233a37CompareFebruary 25, 2022 14:32
pombredanne
pombredanne requested changes Feb 25, 2022
View reviewed changes

@pombredannepombredanne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. LGTM... just a few cosmetic nits for your consideration. Could you also add some tests?

Comment threadvulnerabilities/importer.py Outdated
Comment threadvulnerabilities/importers/alpine_linux.py Outdated
Comment threadvulnerabilities/importers/alpine_linux.py Outdated
Comment threadvulnerabilities/importers/alpine_linux.py Outdated
Comment threadvulnerabilities/importers/alpine_linux.py Outdated
Comment threadvulnerabilities/importers/alpine_linux.py Outdated
Comment threadvulnerabilities/importers/alpine_linux.py Outdated
Comment threadvulnerabilities/importers/alpine_linux.py Outdated
Comment threadvulnerabilities/importers/alpine_linux.py
Comment threadvulnerabilities/improvers/default.py Outdated
@TG1999
TG1999force-pushed the migration/alpine_linux branch from 630e608 to cb83ceeCompareFebruary 28, 2022 15:34
@TG1999
TG1999force-pushed the migration/alpine_linux branch 2 times, most recently from 7bdea4c to 4d9b13aCompareMarch 1, 2022 16:57
@Hritik14Hritik14 linked an issue Mar 1, 2022 that may be closed by this pull request
Add license_url in Advisory Data #634
Closed
Hritik14
Hritik14 requested changes Mar 1, 2022
View reviewed changes

@Hritik14Hritik14 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I've marked a few things that I'd like you to consider.
Also, In general, there are a lot of asserts. IMO a loud logging mechanism should serve for those asserts than failing entirely and much of the asserts should make their place in the tests.

Comment threadvulnerabilities/importers/alpine_linux.py Outdated
Comment threadvulnerabilities/importer.py Outdated
Comment threadvulnerabilities/importer.py Outdated
Comment threadvulnerabilities/importers/alpine_linux.py
Comment threadvulnerabilities/importers/alpine_linux.py Outdated
Comment threadvulnerabilities/references.py Outdated
Comment threadvulnerabilities/references.py Outdated
Comment threadvulnerabilities/references.py Outdated
Comment threadvulnerabilities/references.py Outdated
Comment threadvulnerabilities/references.py Outdated
@TG1999
TG1999force-pushed the migration/alpine_linux branch from 4d9b13a to 5d1952eCompareMarch 2, 2022 15:18
TG1999
TG1999 commented Mar 3, 2022
View reviewed changes
Comment threadvulnerabilities/improver.py Outdated
TG1999
TG1999 commented Mar 3, 2022
View reviewed changes
Comment threadvulnerabilities/improver.py Outdated
@TG1999
TG1999force-pushed the migration/alpine_linux branch from 7171d02 to a3ade32CompareMarch 3, 2022 12:17
@TG1999
Migrate alpine importer to importer improver model
a69c886
Current alpine importer models need to be refactored to give AdvisoryData instead of Advisory, also add some validation to parse license expression and make affected_version_range optional
Add tests to test scraping of webpages and parsing of data
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
@TG1999
TG1999force-pushed the migration/alpine_linux branch from a3ade32 to a69c886CompareMarch 4, 2022 11:21
@TG1999
fix nix
104c760
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
@TG1999
TG1999force-pushed the migration/alpine_linux branch from 1fdd165 to 104c760CompareMarch 4, 2022 12:21
@TG1999
TG1999 merged commit 8c69661 into aboutcode-org:mainMar 4, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Hritik14Hritik14Hritik14 requested changes
@sbs2001sbs2001Awaiting requested review from sbs2001
@pombredannepombredanneAwaiting requested review from pombredanne

Assignees

No one assigned

Labels

import-improver-migration

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add license_url in Advisory Data

3 participants

@TG1999@Hritik14@pombredanne