Uh oh!
There was an error while loading. Please reload this page.
Make RedHat CVE import more robust - #319
Conversation
| cves = [] | ||
| page_no = 1 | ||
| url = "https://access.redhat.com/hydra/rest/securitydata/cve.json?page={}" | ||
| url_template = "https://access.redhat.com/hydra/rest/securitydata/cve.json?per_page=10000&page={}" |
| def rpm_to_purl(rpm_string): | ||
| # FIXME: there is code in scancode to handle RPM conversion AND this should |
There was a problem hiding this comment.
I tried nevra.py, but it's horrible here since, redhat has weird names with multiple -.
The method used here is more conservative.
There was a problem hiding this comment.
I tried nevra.py, but it's horrible here since, redhat has weird names with multiple -.
I need to see actual real examples of real RPM package names that nevra.py cannot parse. If so this is a serious bug.
And https://github.com/nexB/vulnerablecode/blob/8efdbd190d1249e5f4bcda044f98ea50af8a86a4/vulnerabilities/tests/test_redhat_importer.py#L43 is not enough tests and foo is a not a real name: s=you should avoid to make use fake names here IMHO
We really want to have that correct and correctly done in one place only, that we can package in a library if needed.
sbs2001
commented
Jan 29, 2021
@pombredanne FWIW, #290 also makes some changes to redhat, but your changes probably won't conflict with them. |
pombredanne
commented
Jan 29, 2021
Tests are missing and need to be added here. |
sbs2001
commented
Feb 8, 2021
Could you apply black on this ? |
sbs2001
commented
Feb 13, 2021
All is working fine now, merge this as per your convenience |
This importer failed at times toward the end of the fetch step when reaching (and trying repeatidly) to re-fetch the last page. The fix consists in fetching larger batches at once (10K instead of 1k) which measn fewer API calls and less risk to be throttled and properly handling exceptions and HTTP response codes and breaking rather than retrying. This also adds some minimal logging. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
This importer failed at times toward the end of the fetch step when
reaching (and trying repeatidly) to re-fetch the last page.
The fix consists in fetching larger batches at once (10K instead of 1k)
which measn fewer API calls and less risk to be throttled and properly
handling exceptions and HTTP response codes and breaking rather than
retrying. This also adds some minimal logging.
Signed-off-by: Philippe Ombredanne pombredanne@nexb.com