Skip to content

Make RedHat CVE import more robust#319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sbs2001 merged 4 commits into from
Feb 24, 2021
Merged

Make RedHat CVE import more robust #319

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b263b47
Make RedHat CVE import more robust
pombredanne Jan 28, 2021
62f7348
Apply black -l 100
pombredanne Feb 9, 2021
89a5b01
Rebase with latest main
sbs2001 Feb 13, 2021
9fbfe50
Fix codestyle
sbs2001 Feb 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 36 additions & 22 deletions vulnerabilities/importers/redhat.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -20,9 +20,6 @@
# VulnerableCode is a free software code from nexB Inc. and others.
# Visit https://github.com/nexB/vulnerablecode/ for support and download.


import json

from packageurl import PackageURL
import requests

Expand All@@ -38,40 +35,56 @@ class RedhatDataSource(DataSource):
CONFIG_CLASS = DataSourceConfiguration

def __enter__(self):
self.redhat_response = fetch()

def updated_advisories(self):
processed_advisories = []
for advisory_data in self.redhat_response:
processed_advisories.append(to_advisory(advisory_data))
self.redhat_cves = fetch()

def updated_advisories(self):
processed_advisories = list(map(to_advisory, self.redhat_cves))
return self.batch_advisories(processed_advisories)


def fetch():

response = []
"""
Return a list of CVE data mappings fetched from the RedHat API.
See:
https://access.redhat.com/documentation/en-us/red_hat_security_data_api/1.0/html/red_hat_security_data_api/index
"""
cves = []
page_no = 1
url = "https://access.redhat.com/hydra/rest/securitydata/cve.json?page={}"
url_template = "https://access.redhat.com/hydra/rest/securitydata/cve.json?per_page=10000&page={}" # nopep8

cve_data = None
while True:
resp_json = requests.get(url.format(page_no)).json()
page_no += 1
if not resp_json:
current_url = url_template.format(page_no)
try:
print(f"Fetching: {current_url}")
response = requests.get(current_url)
if response.status_code != requests.codes.ok:
# TODO: log me
print(f"Failed to fetch results from {current_url}")
break
cve_data = response.json()
except Exception as e:
# TODO: log me
msg = f"Failed to fetch results from {current_url}:\n{e}"
print(msg)
break

for advisory in resp_json:
response.append(advisory)
if not cve_data:
break
cves.extend(cve_data)
page_no += 1

return response
return cves


def to_advisory(advisory_data):
affected_purls = []
if advisory_data.get("affected_packages"):
for rpm in advisory_data["affected_packages"]:
if rpm_to_purl(rpm):
affected_purls.append(rpm_to_purl(rpm))
purl = rpm_to_purl(rpm)
if purl:
affected_purls.append(purl)

references = []
bugzilla = advisory_data.get("bugzilla")
Expand DownExpand Up@@ -135,16 +148,17 @@ def to_advisory(advisory_data):
)

references.append(Reference(severities=redhat_scores, url=advisory_data["resource_url"]))

return Advisory(
summary=advisory_data["bugzilla_description"],
vulnerability_id=advisory_data["CVE"],
summary=advisory_data["bugzilla_description"],
impacted_package_urls=affected_purls,
vuln_references=references,
)


def rpm_to_purl(rpm_string):
# FIXME: there is code in scancode to handle RPM conversion AND this should

@sbs2001sbs2001Jan 29, 2021

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried nevra.py, but it's horrible here since, redhat has weird names with multiple -.

The method used here is more conservative.

@pombredannepombredanneJan 29, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried nevra.py, but it's horrible here since, redhat has weird names with multiple -.

I need to see actual real examples of real RPM package names that nevra.py cannot parse. If so this is a serious bug.

And https://github.com/nexB/vulnerablecode/blob/8efdbd190d1249e5f4bcda044f98ea50af8a86a4/vulnerabilities/tests/test_redhat_importer.py#L43 is not enough tests and foo is a not a real name: s=you should avoid to make use fake names here IMHO

We really want to have that correct and correctly done in one place only, that we can package in a library if needed.

# be all be part of the packageurl library

# Red Hat uses `-:0` instead of just `-` to separate
# package name and version
Expand All@@ -155,4 +169,4 @@ def rpm_to_purl(rpm_string):
name, version = components

if version[0].isdigit():
return PackageURL(name=name, type="rpm", version=version, namespace="redhat")
return PackageURL(namespace="redhat", name=name, type="rpm", version=version)