Skip to content

SslSessionsCache growth on Linux with MTLS using many client certificates #71401

Closed
Closed
SslSessionsCache growth on Linux with MTLS using many client certificates#71401
Assignees
wfurt
Labels
area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
Milestone
7.0.0

Description

@RobinsonWM
RobinsonWM
opened on Jun 28, 2022

Description

If an application acting as a client creates many mutually-authenticated TLS connections to a server, with each connection using a different client certificate, the application's resident memory grows linearly.

If I run the same code on Windows and Linux, I only see the memory growth on Linux.

This cache appears to grow without bound:

runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs

Line 16 in dbc7d8c

privatestaticreadonlyConcurrentDictionary<SslCredKey,SafeCredentialReference>s_cachedCreds=

There are several IDisposables involved in generating a certificate and making an HTTP request, and I believe I am disposing all of them.

Reproduction Steps

https://gist.github.com/RobinsonWM/cae7244762173127d85f9b842cc3c1ba

The code in this gist reproduces the issue reliably for me. I was running it by starting a container, docker run mcr.microsoft.com/dotnet/sdk:6.0, copying in the code, and then running dotnet run.

The code starts an HTTPS server and then makes requests against it very quickly, each from a new HttpClient with a unique certificate. The test app periodically logs out the result of GC.GetTotalMemory(true) and SslSessionsCache.s_cachedCreds.Count, showing that both increase linearly on Linux but not on Windows.

This test application does muddy up the water a bit since it's running both the client and the server in the same process. I did that to make it easy to reproduce this issue. When I originally encountered this issue, it was in an application that was running only the client; the server was a separate process not written in .NET.

Expected behavior

Over time, if load is steady, the application's resident memory usage is also steady.

Actual behavior

The application works fine for a long time, but its resident memory usage grows linearly.

Regression?

I don't know if this is a regression

Known Workarounds

The same workaround for #65563 works for this issue, except for this issue it would need to be run on a timer.

privatestaticvoidClearCache(){varsslAssembly=Assembly.GetAssembly(typeof(SslStream));varsslSessionCacheClass=sslAssembly.GetType("System.Net.Security.SslSessionsCache");varcachedCredsInfo=sslSessionCacheClass.GetField("s_cachedCreds",BindingFlags.NonPublic|BindingFlags.Static);varcachedCreds=cachedCredsInfo.GetValue(null);cachedCreds.GetType().GetMethod("Clear",BindingFlags.Public|BindingFlags.Instance).Invoke(cachedCreds,null);}

Configuration

.NET 6.0 on Linux x64, with the Docker image mcr.microsoft.com/dotnet/runtime-deps:6.0
I have observed this issue on Linux x64. The same code run on Windows x64 works and does not exhibit the problem. I believe this issue is specific to Linux.

Other information

I realize this use case may sound contrived. I ran into this issue while developing a load-generation application used as part of a performance test. In order to simulate multiple users when generating load, I needed my HTTPS requests to use certificates generated on-the-fly, one for each user.

Metadata

Metadata

Assignees

Labels

area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    , 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
     blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
SslSessionsCache growth on Linux with MTLS using many client certificates · Issue #71401 · dotnet/runtime · GitHub
    Skip to content
    SslSessionsCache growth on Linux with MTLS using many client certificates #71401
    Closed
    Closed
    SslSessionsCache growth on Linux with MTLS using many client certificates#71401
    Assignees
    wfurt
    Labels
    area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
    Milestone
    7.0.0
    Description
    @RobinsonWM
    RobinsonWM
    opened on Jun 28, 2022
    Description
    If an application acting as a client creates many mutually-authenticated TLS connections to a server, with each connection using a different client certificate, the application's resident memory grows linearly.
    If I run the same code on Windows and Linux, I only see the memory growth on Linux.
    This cache appears to grow without bound: 
    runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs
    
Line 16
in
dbc7d8c
    privatestaticreadonlyConcurrentDictionary<SslCredKey,SafeCredentialReference>s_cachedCreds=
    There are several IDisposables involved in generating a certificate and making an HTTP request, and I believe I am disposing all of them.
    Reproduction Steps
    https://gist.github.com/RobinsonWM/cae7244762173127d85f9b842cc3c1ba
    The code in this gist reproduces the issue reliably for me. I was running it by starting a container, docker run mcr.microsoft.com/dotnet/sdk:6.0, copying in the code, and then running dotnet run.
    The code starts an HTTPS server and then makes requests against it very quickly, each from a new HttpClient with a unique certificate. The test app periodically logs out the result of GC.GetTotalMemory(true) and SslSessionsCache.s_cachedCreds.Count, showing that both increase linearly on Linux but not on Windows.
    This test application does muddy up the water a bit since it's running both the client and the server in the same process. I did that to make it easy to reproduce this issue. When I originally encountered this issue, it was in an application that was running only the client; the server was a separate process not written in .NET.
    Expected behavior
    Over time, if load is steady, the application's resident memory usage is also steady.
    Actual behavior
    The application works fine for a long time, but its resident memory usage grows linearly.
    Regression?
    I don't know if this is a regression
    Known Workarounds
    The same workaround for #65563 works for this issue, except for this issue it would need to be run on a timer.
    privatestaticvoidClearCache(){varsslAssembly=Assembly.GetAssembly(typeof(SslStream));varsslSessionCacheClass=sslAssembly.GetType("System.Net.Security.SslSessionsCache");varcachedCredsInfo=sslSessionCacheClass.GetField("s_cachedCreds",BindingFlags.NonPublic|BindingFlags.Static);varcachedCreds=cachedCredsInfo.GetValue(null);cachedCreds.GetType().GetMethod("Clear",BindingFlags.Public|BindingFlags.Instance).Invoke(cachedCreds,null);}
    Configuration
    .NET 6.0 on Linux x64, with the Docker image mcr.microsoft.com/dotnet/runtime-deps:6.0
    
I have observed this issue on Linux x64. The same code run on Windows x64 works and does not exhibit the problem. I believe this issue is specific to Linux.
    Other information
    I realize this use case may sound contrived. I ran into this issue while developing a load-generation application used as part of a performance test. In order to simulate multiple users when generating load, I needed my HTTPS requests to use certificates generated on-the-fly, one for each user.
    Metadata
    Metadata
    Assignees
    Labels
    area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
    Type
    No type
    Projects
    No projects
      Milestone
      Relationships
      None yet
      Development
      No branches or pull requests
      Issue actions
      
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
SslSessionsCache growth on Linux with MTLS using many client certificates · Issue #71401 · dotnet/runtime · GitHub
      Skip to content
      SslSessionsCache growth on Linux with MTLS using many client certificates #71401
      Closed
      Closed
      SslSessionsCache growth on Linux with MTLS using many client certificates#71401
      Assignees
      wfurt
      Labels
      area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
      Milestone
      7.0.0
      Description
      @RobinsonWM
      RobinsonWM
      opened on Jun 28, 2022
      Description
      If an application acting as a client creates many mutually-authenticated TLS connections to a server, with each connection using a different client certificate, the application's resident memory grows linearly.
      If I run the same code on Windows and Linux, I only see the memory growth on Linux.
      This cache appears to grow without bound: 
      runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs
      
Line 16
in
dbc7d8c
      privatestaticreadonlyConcurrentDictionary<SslCredKey,SafeCredentialReference>s_cachedCreds=
      There are several IDisposables involved in generating a certificate and making an HTTP request, and I believe I am disposing all of them.
      Reproduction Steps
      https://gist.github.com/RobinsonWM/cae7244762173127d85f9b842cc3c1ba
      The code in this gist reproduces the issue reliably for me. I was running it by starting a container, docker run mcr.microsoft.com/dotnet/sdk:6.0, copying in the code, and then running dotnet run.
      The code starts an HTTPS server and then makes requests against it very quickly, each from a new HttpClient with a unique certificate. The test app periodically logs out the result of GC.GetTotalMemory(true) and SslSessionsCache.s_cachedCreds.Count, showing that both increase linearly on Linux but not on Windows.
      This test application does muddy up the water a bit since it's running both the client and the server in the same process. I did that to make it easy to reproduce this issue. When I originally encountered this issue, it was in an application that was running only the client; the server was a separate process not written in .NET.
      Expected behavior
      Over time, if load is steady, the application's resident memory usage is also steady.
      Actual behavior
      The application works fine for a long time, but its resident memory usage grows linearly.
      Regression?
      I don't know if this is a regression
      Known Workarounds
      The same workaround for #65563 works for this issue, except for this issue it would need to be run on a timer.
      privatestaticvoidClearCache(){varsslAssembly=Assembly.GetAssembly(typeof(SslStream));varsslSessionCacheClass=sslAssembly.GetType("System.Net.Security.SslSessionsCache");varcachedCredsInfo=sslSessionCacheClass.GetField("s_cachedCreds",BindingFlags.NonPublic|BindingFlags.Static);varcachedCreds=cachedCredsInfo.GetValue(null);cachedCreds.GetType().GetMethod("Clear",BindingFlags.Public|BindingFlags.Instance).Invoke(cachedCreds,null);}
      Configuration
      .NET 6.0 on Linux x64, with the Docker image mcr.microsoft.com/dotnet/runtime-deps:6.0
      
I have observed this issue on Linux x64. The same code run on Windows x64 works and does not exhibit the problem. I believe this issue is specific to Linux.
      Other information
      I realize this use case may sound contrived. I ran into this issue while developing a load-generation application used as part of a performance test. In order to simulate multiple users when generating load, I needed my HTTPS requests to use certificates generated on-the-fly, one for each user.
      Metadata
      Metadata
      Assignees
      Labels
      area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
      Type
      No type
      Projects
      No projects
        Milestone
        Relationships
        None yet
        Development
        No branches or pull requests
        Issue actions
        
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
SslSessionsCache growth on Linux with MTLS using many client certificates · Issue #71401 · dotnet/runtime · GitHub
        Skip to content
        SslSessionsCache growth on Linux with MTLS using many client certificates #71401
        Closed
        Closed
        SslSessionsCache growth on Linux with MTLS using many client certificates#71401
        Assignees
        wfurt
        Labels
        area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
        Milestone
        7.0.0
        Description
        @RobinsonWM
        RobinsonWM
        opened on Jun 28, 2022
        Description
        If an application acting as a client creates many mutually-authenticated TLS connections to a server, with each connection using a different client certificate, the application's resident memory grows linearly.
        If I run the same code on Windows and Linux, I only see the memory growth on Linux.
        This cache appears to grow without bound: 
        runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs
        
Line 16
in
dbc7d8c
        privatestaticreadonlyConcurrentDictionary<SslCredKey,SafeCredentialReference>s_cachedCreds=
        There are several IDisposables involved in generating a certificate and making an HTTP request, and I believe I am disposing all of them.
        Reproduction Steps
        https://gist.github.com/RobinsonWM/cae7244762173127d85f9b842cc3c1ba
        The code in this gist reproduces the issue reliably for me. I was running it by starting a container, docker run mcr.microsoft.com/dotnet/sdk:6.0, copying in the code, and then running dotnet run.
        The code starts an HTTPS server and then makes requests against it very quickly, each from a new HttpClient with a unique certificate. The test app periodically logs out the result of GC.GetTotalMemory(true) and SslSessionsCache.s_cachedCreds.Count, showing that both increase linearly on Linux but not on Windows.
        This test application does muddy up the water a bit since it's running both the client and the server in the same process. I did that to make it easy to reproduce this issue. When I originally encountered this issue, it was in an application that was running only the client; the server was a separate process not written in .NET.
        Expected behavior
        Over time, if load is steady, the application's resident memory usage is also steady.
        Actual behavior
        The application works fine for a long time, but its resident memory usage grows linearly.
        Regression?
        I don't know if this is a regression
        Known Workarounds
        The same workaround for #65563 works for this issue, except for this issue it would need to be run on a timer.
        privatestaticvoidClearCache(){varsslAssembly=Assembly.GetAssembly(typeof(SslStream));varsslSessionCacheClass=sslAssembly.GetType("System.Net.Security.SslSessionsCache");varcachedCredsInfo=sslSessionCacheClass.GetField("s_cachedCreds",BindingFlags.NonPublic|BindingFlags.Static);varcachedCreds=cachedCredsInfo.GetValue(null);cachedCreds.GetType().GetMethod("Clear",BindingFlags.Public|BindingFlags.Instance).Invoke(cachedCreds,null);}
        Configuration
        .NET 6.0 on Linux x64, with the Docker image mcr.microsoft.com/dotnet/runtime-deps:6.0
        
I have observed this issue on Linux x64. The same code run on Windows x64 works and does not exhibit the problem. I believe this issue is specific to Linux.
        Other information
        I realize this use case may sound contrived. I ran into this issue while developing a load-generation application used as part of a performance test. In order to simulate multiple users when generating load, I needed my HTTPS requests to use certificates generated on-the-fly, one for each user.
        Metadata
        Metadata
        Assignees
        Labels
        area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
        Type
        No type
        Projects
        No projects
          Milestone
          Relationships
          None yet
          Development
          No branches or pull requests
          Issue actions
          
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
SslSessionsCache growth on Linux with MTLS using many client certificates · Issue #71401 · dotnet/runtime · GitHub
          Skip to content
          SslSessionsCache growth on Linux with MTLS using many client certificates #71401
          Closed
          Closed
          SslSessionsCache growth on Linux with MTLS using many client certificates#71401
          Assignees
          wfurt
          Labels
          area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
          Milestone
          7.0.0
          Description
          @RobinsonWM
          RobinsonWM
          opened on Jun 28, 2022
          Description
          If an application acting as a client creates many mutually-authenticated TLS connections to a server, with each connection using a different client certificate, the application's resident memory grows linearly.
          If I run the same code on Windows and Linux, I only see the memory growth on Linux.
          This cache appears to grow without bound: 
          runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs
          
Line 16
in
dbc7d8c
          privatestaticreadonlyConcurrentDictionary<SslCredKey,SafeCredentialReference>s_cachedCreds=
          There are several IDisposables involved in generating a certificate and making an HTTP request, and I believe I am disposing all of them.
          Reproduction Steps
          https://gist.github.com/RobinsonWM/cae7244762173127d85f9b842cc3c1ba
          The code in this gist reproduces the issue reliably for me. I was running it by starting a container, docker run mcr.microsoft.com/dotnet/sdk:6.0, copying in the code, and then running dotnet run.
          The code starts an HTTPS server and then makes requests against it very quickly, each from a new HttpClient with a unique certificate. The test app periodically logs out the result of GC.GetTotalMemory(true) and SslSessionsCache.s_cachedCreds.Count, showing that both increase linearly on Linux but not on Windows.
          This test application does muddy up the water a bit since it's running both the client and the server in the same process. I did that to make it easy to reproduce this issue. When I originally encountered this issue, it was in an application that was running only the client; the server was a separate process not written in .NET.
          Expected behavior
          Over time, if load is steady, the application's resident memory usage is also steady.
          Actual behavior
          The application works fine for a long time, but its resident memory usage grows linearly.
          Regression?
          I don't know if this is a regression
          Known Workarounds
          The same workaround for #65563 works for this issue, except for this issue it would need to be run on a timer.
          privatestaticvoidClearCache(){varsslAssembly=Assembly.GetAssembly(typeof(SslStream));varsslSessionCacheClass=sslAssembly.GetType("System.Net.Security.SslSessionsCache");varcachedCredsInfo=sslSessionCacheClass.GetField("s_cachedCreds",BindingFlags.NonPublic|BindingFlags.Static);varcachedCreds=cachedCredsInfo.GetValue(null);cachedCreds.GetType().GetMethod("Clear",BindingFlags.Public|BindingFlags.Instance).Invoke(cachedCreds,null);}
          Configuration
          .NET 6.0 on Linux x64, with the Docker image mcr.microsoft.com/dotnet/runtime-deps:6.0
          
I have observed this issue on Linux x64. The same code run on Windows x64 works and does not exhibit the problem. I believe this issue is specific to Linux.
          Other information
          I realize this use case may sound contrived. I ran into this issue while developing a load-generation application used as part of a performance test. In order to simulate multiple users when generating load, I needed my HTTPS requests to use certificates generated on-the-fly, one for each user.
          Metadata
          Metadata
          Assignees
          Labels
          area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
          Type
          No type
          Projects
          No projects
            Milestone
            Relationships
            None yet
            Development
            No branches or pull requests
            Issue actions
            
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
SslSessionsCache growth on Linux with MTLS using many client certificates · Issue #71401 · dotnet/runtime · GitHub
            Skip to content
            SslSessionsCache growth on Linux with MTLS using many client certificates #71401
            Closed
            Closed
            SslSessionsCache growth on Linux with MTLS using many client certificates#71401
            Assignees
            wfurt
            Labels
            area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
            Milestone
            7.0.0
            Description
            @RobinsonWM
            RobinsonWM
            opened on Jun 28, 2022
            Description
            If an application acting as a client creates many mutually-authenticated TLS connections to a server, with each connection using a different client certificate, the application's resident memory grows linearly.
            If I run the same code on Windows and Linux, I only see the memory growth on Linux.
            This cache appears to grow without bound: 
            runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs
            
Line 16
in
dbc7d8c
            privatestaticreadonlyConcurrentDictionary<SslCredKey,SafeCredentialReference>s_cachedCreds=
            There are several IDisposables involved in generating a certificate and making an HTTP request, and I believe I am disposing all of them.
            Reproduction Steps
            https://gist.github.com/RobinsonWM/cae7244762173127d85f9b842cc3c1ba
            The code in this gist reproduces the issue reliably for me. I was running it by starting a container, docker run mcr.microsoft.com/dotnet/sdk:6.0, copying in the code, and then running dotnet run.
            The code starts an HTTPS server and then makes requests against it very quickly, each from a new HttpClient with a unique certificate. The test app periodically logs out the result of GC.GetTotalMemory(true) and SslSessionsCache.s_cachedCreds.Count, showing that both increase linearly on Linux but not on Windows.
            This test application does muddy up the water a bit since it's running both the client and the server in the same process. I did that to make it easy to reproduce this issue. When I originally encountered this issue, it was in an application that was running only the client; the server was a separate process not written in .NET.
            Expected behavior
            Over time, if load is steady, the application's resident memory usage is also steady.
            Actual behavior
            The application works fine for a long time, but its resident memory usage grows linearly.
            Regression?
            I don't know if this is a regression
            Known Workarounds
            The same workaround for #65563 works for this issue, except for this issue it would need to be run on a timer.
            privatestaticvoidClearCache(){varsslAssembly=Assembly.GetAssembly(typeof(SslStream));varsslSessionCacheClass=sslAssembly.GetType("System.Net.Security.SslSessionsCache");varcachedCredsInfo=sslSessionCacheClass.GetField("s_cachedCreds",BindingFlags.NonPublic|BindingFlags.Static);varcachedCreds=cachedCredsInfo.GetValue(null);cachedCreds.GetType().GetMethod("Clear",BindingFlags.Public|BindingFlags.Instance).Invoke(cachedCreds,null);}
            Configuration
            .NET 6.0 on Linux x64, with the Docker image mcr.microsoft.com/dotnet/runtime-deps:6.0
            
I have observed this issue on Linux x64. The same code run on Windows x64 works and does not exhibit the problem. I believe this issue is specific to Linux.
            Other information
            I realize this use case may sound contrived. I ran into this issue while developing a load-generation application used as part of a performance test. In order to simulate multiple users when generating load, I needed my HTTPS requests to use certificates generated on-the-fly, one for each user.
            Metadata
            Metadata
            Assignees
            Labels
            area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
            Type
            No type
            Projects
            No projects
              Milestone
              Relationships
              None yet
              Development
              No branches or pull requests
              Issue actions
              
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
SslSessionsCache growth on Linux with MTLS using many client certificates · Issue #71401 · dotnet/runtime · GitHub
              Skip to content
              SslSessionsCache growth on Linux with MTLS using many client certificates #71401
              Closed
              Closed
              SslSessionsCache growth on Linux with MTLS using many client certificates#71401
              Assignees
              wfurt
              Labels
              area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
              Milestone
              7.0.0
              Description
              @RobinsonWM
              RobinsonWM
              opened on Jun 28, 2022
              Description
              If an application acting as a client creates many mutually-authenticated TLS connections to a server, with each connection using a different client certificate, the application's resident memory grows linearly.
              If I run the same code on Windows and Linux, I only see the memory growth on Linux.
              This cache appears to grow without bound: 
              runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs
              
Line 16
in
dbc7d8c
              privatestaticreadonlyConcurrentDictionary<SslCredKey,SafeCredentialReference>s_cachedCreds=
              There are several IDisposables involved in generating a certificate and making an HTTP request, and I believe I am disposing all of them.
              Reproduction Steps
              https://gist.github.com/RobinsonWM/cae7244762173127d85f9b842cc3c1ba
              The code in this gist reproduces the issue reliably for me. I was running it by starting a container, docker run mcr.microsoft.com/dotnet/sdk:6.0, copying in the code, and then running dotnet run.
              The code starts an HTTPS server and then makes requests against it very quickly, each from a new HttpClient with a unique certificate. The test app periodically logs out the result of GC.GetTotalMemory(true) and SslSessionsCache.s_cachedCreds.Count, showing that both increase linearly on Linux but not on Windows.
              This test application does muddy up the water a bit since it's running both the client and the server in the same process. I did that to make it easy to reproduce this issue. When I originally encountered this issue, it was in an application that was running only the client; the server was a separate process not written in .NET.
              Expected behavior
              Over time, if load is steady, the application's resident memory usage is also steady.
              Actual behavior
              The application works fine for a long time, but its resident memory usage grows linearly.
              Regression?
              I don't know if this is a regression
              Known Workarounds
              The same workaround for #65563 works for this issue, except for this issue it would need to be run on a timer.
              privatestaticvoidClearCache(){varsslAssembly=Assembly.GetAssembly(typeof(SslStream));varsslSessionCacheClass=sslAssembly.GetType("System.Net.Security.SslSessionsCache");varcachedCredsInfo=sslSessionCacheClass.GetField("s_cachedCreds",BindingFlags.NonPublic|BindingFlags.Static);varcachedCreds=cachedCredsInfo.GetValue(null);cachedCreds.GetType().GetMethod("Clear",BindingFlags.Public|BindingFlags.Instance).Invoke(cachedCreds,null);}
              Configuration
              .NET 6.0 on Linux x64, with the Docker image mcr.microsoft.com/dotnet/runtime-deps:6.0
              
I have observed this issue on Linux x64. The same code run on Windows x64 works and does not exhibit the problem. I believe this issue is specific to Linux.
              Other information
              I realize this use case may sound contrived. I ran into this issue while developing a load-generation application used as part of a performance test. In order to simulate multiple users when generating load, I needed my HTTPS requests to use certificates generated on-the-fly, one for each user.
              Metadata
              Metadata
              Assignees
              Labels
              area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
              Type
              No type
              Projects
              No projects
                Milestone
                Relationships
                None yet
                Development
                No branches or pull requests
                Issue actions
                
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
SslSessionsCache growth on Linux with MTLS using many client certificates · Issue #71401 · dotnet/runtime · GitHub
                Skip to content
                SslSessionsCache growth on Linux with MTLS using many client certificates #71401
                Closed
                Closed
                SslSessionsCache growth on Linux with MTLS using many client certificates#71401
                Assignees
                wfurt
                Labels
                area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
                Milestone
                7.0.0
                Description
                @RobinsonWM
                RobinsonWM
                opened on Jun 28, 2022
                Description
                If an application acting as a client creates many mutually-authenticated TLS connections to a server, with each connection using a different client certificate, the application's resident memory grows linearly.
                If I run the same code on Windows and Linux, I only see the memory growth on Linux.
                This cache appears to grow without bound: 
                runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslSessionsCache.cs
                
Line 16
in
dbc7d8c
                privatestaticreadonlyConcurrentDictionary<SslCredKey,SafeCredentialReference>s_cachedCreds=
                There are several IDisposables involved in generating a certificate and making an HTTP request, and I believe I am disposing all of them.
                Reproduction Steps
                https://gist.github.com/RobinsonWM/cae7244762173127d85f9b842cc3c1ba
                The code in this gist reproduces the issue reliably for me. I was running it by starting a container, docker run mcr.microsoft.com/dotnet/sdk:6.0, copying in the code, and then running dotnet run.
                The code starts an HTTPS server and then makes requests against it very quickly, each from a new HttpClient with a unique certificate. The test app periodically logs out the result of GC.GetTotalMemory(true) and SslSessionsCache.s_cachedCreds.Count, showing that both increase linearly on Linux but not on Windows.
                This test application does muddy up the water a bit since it's running both the client and the server in the same process. I did that to make it easy to reproduce this issue. When I originally encountered this issue, it was in an application that was running only the client; the server was a separate process not written in .NET.
                Expected behavior
                Over time, if load is steady, the application's resident memory usage is also steady.
                Actual behavior
                The application works fine for a long time, but its resident memory usage grows linearly.
                Regression?
                I don't know if this is a regression
                Known Workarounds
                The same workaround for #65563 works for this issue, except for this issue it would need to be run on a timer.
                privatestaticvoidClearCache(){varsslAssembly=Assembly.GetAssembly(typeof(SslStream));varsslSessionCacheClass=sslAssembly.GetType("System.Net.Security.SslSessionsCache");varcachedCredsInfo=sslSessionCacheClass.GetField("s_cachedCreds",BindingFlags.NonPublic|BindingFlags.Static);varcachedCreds=cachedCredsInfo.GetValue(null);cachedCreds.GetType().GetMethod("Clear",BindingFlags.Public|BindingFlags.Instance).Invoke(cachedCreds,null);}
                Configuration
                .NET 6.0 on Linux x64, with the Docker image mcr.microsoft.com/dotnet/runtime-deps:6.0
                
I have observed this issue on Linux x64. The same code run on Windows x64 works and does not exhibit the problem. I believe this issue is specific to Linux.
                Other information
                I realize this use case may sound contrived. I ran into this issue while developing a load-generation application used as part of a performance test. In order to simulate multiple users when generating load, I needed my HTTPS requests to use certificates generated on-the-fly, one for each user.
                Metadata
                Metadata
                Assignees
                Labels
                area-System.Net.Securitybugneeds-author-actionAn issue or pull request that requires more info or actions from the author.
                Type
                No type
                Projects
                No projects
                  Milestone
                  Relationships
                  None yet
                  Development
                  No branches or pull requests
                  Issue actions