CSharp and Java Queries Not Cached / Precompiled

[travisgosselin]

Starting a few weeks ago, CSharp and Java codeql analysis are taking extremely long... around 60 min, where the same analysis was taking 2 min prior. It looks as though the queries are not hitting the precompiled cache and are compiling on the fly with the following output:

[1/174] Compiled /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToGCCollect.ql.
Compiling query plan for /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.
Resolving imports for /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.
Checking QL for /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.
Optimizing /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.

Previously the output showed a cache hit when executing much faster:

[1/173] Found in cache: /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToGCCollect.ql.
Compiling query plan for /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.
Resolving imports for /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.
Compilation cache hit for /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.

I don't have any local changes nor do I see any changes to this repository in the last few weeks to this repository. Perhaps something in the weekly build process is failing to precompile all languages. Python and Node seem to be working fine on latest container showing cache hits on all queries.

[shargon]

Same here, I need to compile ql like this

RUN codeql query compile --threads=0 --additional-packs=. \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/WeakEncryption.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-611/UntrustedDataInsecureXml.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/InsufficientKeySize.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CookieWithOverlyBroadDomain.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/InadequateRSAPadding.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-011/ASPNetDebug.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-079/XSS.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-730/RegexInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-730/ReDoS.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-312/CleartextStorage.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-359/ExposureOfPrivateInformation.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-384/AbandonSession.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-248/MissingASPNETGlobalErrorHandler.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-089/SqlInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-502/UnsafeDeserializationUntrustedInput.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-502/DeserializedDelegate.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/Encryption using ECB.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-209/ExceptionInformationExposure.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-614/RequireSSL.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-119/LocalUnvalidatedArithmetic.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-601/UrlRedirect.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-022/TaintedPath.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-022/ZipSlip.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-090/LDAPInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-807/ConditionalBypass.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-201/ExposureInTransmittedData.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/HeaderCheckingDisabled.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-352/MissingAntiForgeryTokenValidation.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-114/AssemblyPathInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-643/XPathInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/InsecureRandomness.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-134/UncontrolledFormatString.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-451/MissingXFrameOptions.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-548/ASPNetDirectoryListing.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/PersistentCookie.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-091/XMLInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-117/LogForging.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-094/CodeInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CookieWithOverlyBroadPath.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-078/CommandInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Security Features/CWE-099/ResourceInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Diagnostics/DiagnosticNoExtractionErrors.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Diagnostics/CompilerMessage.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Diagnostics/ExtractorError.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Diagnostics/DiagnosticExtractionErrors.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Diagnostics/ExtractorMessage.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Diagnostics/CompilerError.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Telemetry/UnsupportedExternalAPIs.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Telemetry/SupportedExternalTaint.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Telemetry/ExternalLibraryUsage.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Telemetry/SupportedExternalSources.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Telemetry/ExtractorInformation.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Telemetry/SupportedExternalSinks.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Telemetry/SupportedExternalApis.ql" \
 "${CODEQL_HOME}/codeql-repo/csharp/ql/src/Metrics/Summaries/LinesOfCode.ql"

# It seems that there are java rules that are not pre-compiled.

RUN codeql query compile --threads=0 --additional-packs=. \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-918/RequestForgery.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-611/XXE.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-926/ImplicitlyExportedAndroidComponent.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-113/NettyResponseSplitting.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-113/ResponseSplitting.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-1104/MavenPomDependsOnBintray.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-1204/StaticInitializationVector.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-079/XSS.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-330/InsecureRandomness.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-295/ImproperWebViewCertificateValidation.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-295/InsecureTrustManager.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-489/DebuggableAttributeEnabled.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-489/WebviewDebuggingEnabled.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-730/RegexInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-730/ReDoS.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-730/PolynomialReDoS.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-297/UnsafeHostnameVerification.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-327/BrokenCryptoAlgorithm.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-312/CleartextStorageCookie.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-522/InsecureLdapAuth.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-925/ImproperIntentVerification.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-089/SqlTainted.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-502/UnsafeDeserialization.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-829/InsecureDependencyResolution.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-338/JHipsterGeneratedPRNG.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-023/PartialPathTraversalFromRemote.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-326/InsufficientKeySize.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-209/StackTraceExposure.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-614/InsecureCookie.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-601/UrlRedirect.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-347/MissingJWTSignatureCheck.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-074/JndiInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-074/XsltInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-022/TaintedPath.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-022/ZipSlip.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-441/UnsafeContentUriResolution.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-090/LdapInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-807/TaintedPermissionsCheck.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-352/SpringCSRFProtection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-643/XPathInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-134/ExternallyControlledFormatString.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-917/OgnlInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-732/ReadingFromWorldWritableFile.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-940/AndroidIntentRedirection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-681/NumericCastTainted.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-094/MvelInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-094/GroovyInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-094/SpelInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-094/JexlInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-094/TemplateInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-094/InsecureBeanValidation.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-335/PredictableSeed.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-927/ImplicitPendingIntents.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-020/OverlyLargeRange.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-470/FragmentInjectionInPreferenceActivity.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-470/FragmentInjection.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-780/RsaWithoutOaep.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-266/IntentUriPermissionManipulation.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-078/ExecTainted.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Security/CWE/CWE-078/ExecUnescaped.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Likely Bugs/Arithmetic/InformationLoss.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Diagnostics/ExtractionErrors.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Diagnostics/SuccessfullyExtractedFiles.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Diagnostics/ExtractionWarnings.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Telemetry/UnsupportedExternalAPIs.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Telemetry/SupportedExternalTaint.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Telemetry/ExternalLibraryUsage.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Telemetry/SupportedExternalSources.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Telemetry/ExtractorInformation.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Telemetry/SupportedExternalSinks.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Telemetry/SupportedExternalApis.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Metrics/Summaries/LinesOfCodeKotlin.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Metrics/Summaries/LinesOfCode.ql" \
 "${CODEQL_HOME}/codeql-repo/java/ql/src/Metrics/Summaries/LinesOfCodeJava.ql"

[shargon]

Travis Gosselin (@travisgosselin) try with https://github.com/Red4Sec/CodeQL/pkgs/container/codeql

[travisgosselin]

@travisgosselin try with https://github.com/Red4Sec/CodeQL/pkgs/container/codeql

Awesome thanks Shargon (@shargon) - this is good to know there is a well supported version out there. We ended building our own internal version and customized to some internal needs and dependencies we have.