Skip to content

PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM #5

Open
Open
PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM#5
Assignees
wx-guyue

Description

@wzeng96
wzeng96
opened on May 22, 2026

Hi,

I'm scalesc to run a big-scale spatial transcriptomic data using it's GPU supported. My data is ~2.7 million cells x 6,175 genes. However, when I tried to run PCA, it would always return me the CUDA memory error. More specifically, I suspect that it's due to the process when it tries to convert the sparse data to dense. Please see the detailed version and error message below. I'm wondering if the team has any insights in fixing this?

Environment

  • ScaleSC version: 0.1.0
  • CuPy version: 13.6.0
  • GPU: NVIDIA L40S (47.67GB VRAM × 2)
  • CUDA: 13.0, Driver: 580.159.03

Dataset

  • 2,702,045 cells × 6,157 genes
  • Sparse CSR float32
  • 3,000 HVGs selected via seurat_v3

Note on initialization

I'm using a custom wrapper to initialize ScaleSC due to a separate shape reader issue on large datasets:

definit_scalesc_with_fix(data_dir, **kwargs):
ssc_obj=ssc.ScaleSC(data_dir=data_dir, **kwargs)
adata=ssc_obj.reader._get_anndata_obj()
n_obs, n_vars=adata.shapessc_obj.reader.n_cell=n_obsssc_obj.reader.n_gene=n_varsssc_obj.reader.n_cell_origin=n_obsssc_obj.reader.n_gene_origin=n_varsreturnssc_objssc_obj=init_scalesc_with_fix(
data_dir=SCALESC_DIR,
preload_on_cpu=True,
preload_on_gpu=False,
gpus=[0, 1],
save_after_each_step=True,
max_cell_batch=25000
)

normalize_log1p and highly_variable_genes both complete successfully. The full pipeline also ran without issues on smaller datasets.

Problem

pca() consistently fails with:

MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /gscratch/stf/wz34/envs/ScaleSC/include/rmm/mr/device/cuda_memory_resource.hpp

I noticed

1 — allocation is always exactly ~17GB

The allocation request is consistently 17383920128B (~17GB) regardless of max_cell_batch value (tested with 100,000, 50,000, 25,000, and 1,000). This suggests the ~17GB is a fixed overhead unrelated to batch size, making max_cell_batch ineffective as a workaround.

2 — only GPU 0 is used

GPU memory monitoring confirms GPU 1 is completely unused despite gpus=[0, 1]:

GPU 0: free=30.69 GB, total=47.67 GB ← all allocations here
GPU 1: free=47.18 GB, total=47.67 GB ← completely idle

Any insights or help would be greatly appreciated. Thank you!

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    , 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
     blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM · Issue #5 · interactivereport/ScaleSC · GitHub
    Skip to content
    PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM #5
    Open
    Open
    PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM#5
    Assignees
    wx-guyue
    Description
    @wzeng96
    wzeng96
    opened on May 22, 2026
    Hi,
    I'm scalesc to run a big-scale spatial transcriptomic data using it's GPU supported. My data is ~2.7 million cells x 6,175 genes. However, when I tried to run PCA, it would always return me the CUDA memory error. More specifically, I suspect that it's due to the process when it tries to convert the sparse data to dense. Please see the detailed version and error message below. I'm wondering if the team has any insights in fixing this?
    Environment
    • ScaleSC version: 0.1.0
    • CuPy version: 13.6.0
    • GPU: NVIDIA L40S (47.67GB VRAM × 2)
    • CUDA: 13.0, Driver: 580.159.03
    Dataset
    • 2,702,045 cells × 6,157 genes
    • Sparse CSR float32
    • 3,000 HVGs selected via seurat_v3
    Note on initialization
    I'm using a custom wrapper to initialize ScaleSC due to a separate shape reader issue on large datasets:
    definit_scalesc_with_fix(data_dir, **kwargs):
ssc_obj=ssc.ScaleSC(data_dir=data_dir, **kwargs)
adata=ssc_obj.reader._get_anndata_obj()
n_obs, n_vars=adata.shapessc_obj.reader.n_cell=n_obsssc_obj.reader.n_gene=n_varsssc_obj.reader.n_cell_origin=n_obsssc_obj.reader.n_gene_origin=n_varsreturnssc_objssc_obj=init_scalesc_with_fix(
data_dir=SCALESC_DIR,
preload_on_cpu=True,
preload_on_gpu=False,
gpus=[0, 1],
save_after_each_step=True,
max_cell_batch=25000
)
    normalize_log1p and highly_variable_genes both complete successfully. The full pipeline also ran without issues on smaller datasets.
    Problem
    pca() consistently fails with:
    MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /gscratch/stf/wz34/envs/ScaleSC/include/rmm/mr/device/cuda_memory_resource.hpp

    I noticed
    1 — allocation is always exactly ~17GB
    The allocation request is consistently 17383920128B (~17GB) regardless of max_cell_batch value (tested with 100,000, 50,000, 25,000, and 1,000). This suggests the ~17GB is a fixed overhead unrelated to batch size, making max_cell_batch ineffective as a workaround.
    2 — only GPU 0 is used
    GPU memory monitoring confirms GPU 1 is completely unused despite gpus=[0, 1]:
    GPU 0: free=30.69 GB, total=47.67 GB ← all allocations here
GPU 1: free=47.18 GB, total=47.67 GB ← completely idle

    Any insights or help would be greatly appreciated. Thank you!
    Metadata
    Metadata
    Assignees
    Labels
    No labels
    No labels
    Projects
    No projects
      Milestone
      No milestone
      Relationships
      None yet
      Development
      No branches or pull requests
      Issue actions
      
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM · Issue #5 · interactivereport/ScaleSC · GitHub
      Skip to content
      PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM #5
      Open
      Open
      PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM#5
      Assignees
      wx-guyue
      Description
      @wzeng96
      wzeng96
      opened on May 22, 2026
      Hi,
      I'm scalesc to run a big-scale spatial transcriptomic data using it's GPU supported. My data is ~2.7 million cells x 6,175 genes. However, when I tried to run PCA, it would always return me the CUDA memory error. More specifically, I suspect that it's due to the process when it tries to convert the sparse data to dense. Please see the detailed version and error message below. I'm wondering if the team has any insights in fixing this?
      Environment
      • ScaleSC version: 0.1.0
      • CuPy version: 13.6.0
      • GPU: NVIDIA L40S (47.67GB VRAM × 2)
      • CUDA: 13.0, Driver: 580.159.03
      Dataset
      • 2,702,045 cells × 6,157 genes
      • Sparse CSR float32
      • 3,000 HVGs selected via seurat_v3
      Note on initialization
      I'm using a custom wrapper to initialize ScaleSC due to a separate shape reader issue on large datasets:
      definit_scalesc_with_fix(data_dir, **kwargs):
ssc_obj=ssc.ScaleSC(data_dir=data_dir, **kwargs)
adata=ssc_obj.reader._get_anndata_obj()
n_obs, n_vars=adata.shapessc_obj.reader.n_cell=n_obsssc_obj.reader.n_gene=n_varsssc_obj.reader.n_cell_origin=n_obsssc_obj.reader.n_gene_origin=n_varsreturnssc_objssc_obj=init_scalesc_with_fix(
data_dir=SCALESC_DIR,
preload_on_cpu=True,
preload_on_gpu=False,
gpus=[0, 1],
save_after_each_step=True,
max_cell_batch=25000
)
      normalize_log1p and highly_variable_genes both complete successfully. The full pipeline also ran without issues on smaller datasets.
      Problem
      pca() consistently fails with:
      MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /gscratch/stf/wz34/envs/ScaleSC/include/rmm/mr/device/cuda_memory_resource.hpp

      I noticed
      1 — allocation is always exactly ~17GB
      The allocation request is consistently 17383920128B (~17GB) regardless of max_cell_batch value (tested with 100,000, 50,000, 25,000, and 1,000). This suggests the ~17GB is a fixed overhead unrelated to batch size, making max_cell_batch ineffective as a workaround.
      2 — only GPU 0 is used
      GPU memory monitoring confirms GPU 1 is completely unused despite gpus=[0, 1]:
      GPU 0: free=30.69 GB, total=47.67 GB ← all allocations here
GPU 1: free=47.18 GB, total=47.67 GB ← completely idle

      Any insights or help would be greatly appreciated. Thank you!
      Metadata
      Metadata
      Assignees
      Labels
      No labels
      No labels
      Projects
      No projects
        Milestone
        No milestone
        Relationships
        None yet
        Development
        No branches or pull requests
        Issue actions
        
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM · Issue #5 · interactivereport/ScaleSC · GitHub
        Skip to content
        PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM #5
        Open
        Open
        PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM#5
        Assignees
        wx-guyue
        Description
        @wzeng96
        wzeng96
        opened on May 22, 2026
        Hi,
        I'm scalesc to run a big-scale spatial transcriptomic data using it's GPU supported. My data is ~2.7 million cells x 6,175 genes. However, when I tried to run PCA, it would always return me the CUDA memory error. More specifically, I suspect that it's due to the process when it tries to convert the sparse data to dense. Please see the detailed version and error message below. I'm wondering if the team has any insights in fixing this?
        Environment
        • ScaleSC version: 0.1.0
        • CuPy version: 13.6.0
        • GPU: NVIDIA L40S (47.67GB VRAM × 2)
        • CUDA: 13.0, Driver: 580.159.03
        Dataset
        • 2,702,045 cells × 6,157 genes
        • Sparse CSR float32
        • 3,000 HVGs selected via seurat_v3
        Note on initialization
        I'm using a custom wrapper to initialize ScaleSC due to a separate shape reader issue on large datasets:
        definit_scalesc_with_fix(data_dir, **kwargs):
ssc_obj=ssc.ScaleSC(data_dir=data_dir, **kwargs)
adata=ssc_obj.reader._get_anndata_obj()
n_obs, n_vars=adata.shapessc_obj.reader.n_cell=n_obsssc_obj.reader.n_gene=n_varsssc_obj.reader.n_cell_origin=n_obsssc_obj.reader.n_gene_origin=n_varsreturnssc_objssc_obj=init_scalesc_with_fix(
data_dir=SCALESC_DIR,
preload_on_cpu=True,
preload_on_gpu=False,
gpus=[0, 1],
save_after_each_step=True,
max_cell_batch=25000
)
        normalize_log1p and highly_variable_genes both complete successfully. The full pipeline also ran without issues on smaller datasets.
        Problem
        pca() consistently fails with:
        MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /gscratch/stf/wz34/envs/ScaleSC/include/rmm/mr/device/cuda_memory_resource.hpp

        I noticed
        1 — allocation is always exactly ~17GB
        The allocation request is consistently 17383920128B (~17GB) regardless of max_cell_batch value (tested with 100,000, 50,000, 25,000, and 1,000). This suggests the ~17GB is a fixed overhead unrelated to batch size, making max_cell_batch ineffective as a workaround.
        2 — only GPU 0 is used
        GPU memory monitoring confirms GPU 1 is completely unused despite gpus=[0, 1]:
        GPU 0: free=30.69 GB, total=47.67 GB ← all allocations here
GPU 1: free=47.18 GB, total=47.67 GB ← completely idle

        Any insights or help would be greatly appreciated. Thank you!
        Metadata
        Metadata
        Assignees
        Labels
        No labels
        No labels
        Projects
        No projects
          Milestone
          No milestone
          Relationships
          None yet
          Development
          No branches or pull requests
          Issue actions
          
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM · Issue #5 · interactivereport/ScaleSC · GitHub
          Skip to content
          PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM #5
          Open
          Open
          PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM#5
          Assignees
          wx-guyue
          Description
          @wzeng96
          wzeng96
          opened on May 22, 2026
          Hi,
          I'm scalesc to run a big-scale spatial transcriptomic data using it's GPU supported. My data is ~2.7 million cells x 6,175 genes. However, when I tried to run PCA, it would always return me the CUDA memory error. More specifically, I suspect that it's due to the process when it tries to convert the sparse data to dense. Please see the detailed version and error message below. I'm wondering if the team has any insights in fixing this?
          Environment
          • ScaleSC version: 0.1.0
          • CuPy version: 13.6.0
          • GPU: NVIDIA L40S (47.67GB VRAM × 2)
          • CUDA: 13.0, Driver: 580.159.03
          Dataset
          • 2,702,045 cells × 6,157 genes
          • Sparse CSR float32
          • 3,000 HVGs selected via seurat_v3
          Note on initialization
          I'm using a custom wrapper to initialize ScaleSC due to a separate shape reader issue on large datasets:
          definit_scalesc_with_fix(data_dir, **kwargs):
ssc_obj=ssc.ScaleSC(data_dir=data_dir, **kwargs)
adata=ssc_obj.reader._get_anndata_obj()
n_obs, n_vars=adata.shapessc_obj.reader.n_cell=n_obsssc_obj.reader.n_gene=n_varsssc_obj.reader.n_cell_origin=n_obsssc_obj.reader.n_gene_origin=n_varsreturnssc_objssc_obj=init_scalesc_with_fix(
data_dir=SCALESC_DIR,
preload_on_cpu=True,
preload_on_gpu=False,
gpus=[0, 1],
save_after_each_step=True,
max_cell_batch=25000
)
          normalize_log1p and highly_variable_genes both complete successfully. The full pipeline also ran without issues on smaller datasets.
          Problem
          pca() consistently fails with:
          MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /gscratch/stf/wz34/envs/ScaleSC/include/rmm/mr/device/cuda_memory_resource.hpp

          I noticed
          1 — allocation is always exactly ~17GB
          The allocation request is consistently 17383920128B (~17GB) regardless of max_cell_batch value (tested with 100,000, 50,000, 25,000, and 1,000). This suggests the ~17GB is a fixed overhead unrelated to batch size, making max_cell_batch ineffective as a workaround.
          2 — only GPU 0 is used
          GPU memory monitoring confirms GPU 1 is completely unused despite gpus=[0, 1]:
          GPU 0: free=30.69 GB, total=47.67 GB ← all allocations here
GPU 1: free=47.18 GB, total=47.67 GB ← completely idle

          Any insights or help would be greatly appreciated. Thank you!
          Metadata
          Metadata
          Assignees
          Labels
          No labels
          No labels
          Projects
          No projects
            Milestone
            No milestone
            Relationships
            None yet
            Development
            No branches or pull requests
            Issue actions
            
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM · Issue #5 · interactivereport/ScaleSC · GitHub
            Skip to content
            PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM #5
            Open
            Open
            PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM#5
            Assignees
            wx-guyue
            Description
            @wzeng96
            wzeng96
            opened on May 22, 2026
            Hi,
            I'm scalesc to run a big-scale spatial transcriptomic data using it's GPU supported. My data is ~2.7 million cells x 6,175 genes. However, when I tried to run PCA, it would always return me the CUDA memory error. More specifically, I suspect that it's due to the process when it tries to convert the sparse data to dense. Please see the detailed version and error message below. I'm wondering if the team has any insights in fixing this?
            Environment
            • ScaleSC version: 0.1.0
            • CuPy version: 13.6.0
            • GPU: NVIDIA L40S (47.67GB VRAM × 2)
            • CUDA: 13.0, Driver: 580.159.03
            Dataset
            • 2,702,045 cells × 6,157 genes
            • Sparse CSR float32
            • 3,000 HVGs selected via seurat_v3
            Note on initialization
            I'm using a custom wrapper to initialize ScaleSC due to a separate shape reader issue on large datasets:
            definit_scalesc_with_fix(data_dir, **kwargs):
ssc_obj=ssc.ScaleSC(data_dir=data_dir, **kwargs)
adata=ssc_obj.reader._get_anndata_obj()
n_obs, n_vars=adata.shapessc_obj.reader.n_cell=n_obsssc_obj.reader.n_gene=n_varsssc_obj.reader.n_cell_origin=n_obsssc_obj.reader.n_gene_origin=n_varsreturnssc_objssc_obj=init_scalesc_with_fix(
data_dir=SCALESC_DIR,
preload_on_cpu=True,
preload_on_gpu=False,
gpus=[0, 1],
save_after_each_step=True,
max_cell_batch=25000
)
            normalize_log1p and highly_variable_genes both complete successfully. The full pipeline also ran without issues on smaller datasets.
            Problem
            pca() consistently fails with:
            MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /gscratch/stf/wz34/envs/ScaleSC/include/rmm/mr/device/cuda_memory_resource.hpp

            I noticed
            1 — allocation is always exactly ~17GB
            The allocation request is consistently 17383920128B (~17GB) regardless of max_cell_batch value (tested with 100,000, 50,000, 25,000, and 1,000). This suggests the ~17GB is a fixed overhead unrelated to batch size, making max_cell_batch ineffective as a workaround.
            2 — only GPU 0 is used
            GPU memory monitoring confirms GPU 1 is completely unused despite gpus=[0, 1]:
            GPU 0: free=30.69 GB, total=47.67 GB ← all allocations here
GPU 1: free=47.18 GB, total=47.67 GB ← completely idle

            Any insights or help would be greatly appreciated. Thank you!
            Metadata
            Metadata
            Assignees
            Labels
            No labels
            No labels
            Projects
            No projects
              Milestone
              No milestone
              Relationships
              None yet
              Development
              No branches or pull requests
              Issue actions
              
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM · Issue #5 · interactivereport/ScaleSC · GitHub
              Skip to content
              PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM #5
              Open
              Open
              PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM#5
              Assignees
              wx-guyue
              Description
              @wzeng96
              wzeng96
              opened on May 22, 2026
              Hi,
              I'm scalesc to run a big-scale spatial transcriptomic data using it's GPU supported. My data is ~2.7 million cells x 6,175 genes. However, when I tried to run PCA, it would always return me the CUDA memory error. More specifically, I suspect that it's due to the process when it tries to convert the sparse data to dense. Please see the detailed version and error message below. I'm wondering if the team has any insights in fixing this?
              Environment
              • ScaleSC version: 0.1.0
              • CuPy version: 13.6.0
              • GPU: NVIDIA L40S (47.67GB VRAM × 2)
              • CUDA: 13.0, Driver: 580.159.03
              Dataset
              • 2,702,045 cells × 6,157 genes
              • Sparse CSR float32
              • 3,000 HVGs selected via seurat_v3
              Note on initialization
              I'm using a custom wrapper to initialize ScaleSC due to a separate shape reader issue on large datasets:
              definit_scalesc_with_fix(data_dir, **kwargs):
ssc_obj=ssc.ScaleSC(data_dir=data_dir, **kwargs)
adata=ssc_obj.reader._get_anndata_obj()
n_obs, n_vars=adata.shapessc_obj.reader.n_cell=n_obsssc_obj.reader.n_gene=n_varsssc_obj.reader.n_cell_origin=n_obsssc_obj.reader.n_gene_origin=n_varsreturnssc_objssc_obj=init_scalesc_with_fix(
data_dir=SCALESC_DIR,
preload_on_cpu=True,
preload_on_gpu=False,
gpus=[0, 1],
save_after_each_step=True,
max_cell_batch=25000
)
              normalize_log1p and highly_variable_genes both complete successfully. The full pipeline also ran without issues on smaller datasets.
              Problem
              pca() consistently fails with:
              MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /gscratch/stf/wz34/envs/ScaleSC/include/rmm/mr/device/cuda_memory_resource.hpp

              I noticed
              1 — allocation is always exactly ~17GB
              The allocation request is consistently 17383920128B (~17GB) regardless of max_cell_batch value (tested with 100,000, 50,000, 25,000, and 1,000). This suggests the ~17GB is a fixed overhead unrelated to batch size, making max_cell_batch ineffective as a workaround.
              2 — only GPU 0 is used
              GPU memory monitoring confirms GPU 1 is completely unused despite gpus=[0, 1]:
              GPU 0: free=30.69 GB, total=47.67 GB ← all allocations here
GPU 1: free=47.18 GB, total=47.67 GB ← completely idle

              Any insights or help would be greatly appreciated. Thank you!
              Metadata
              Metadata
              Assignees
              Labels
              No labels
              No labels
              Projects
              No projects
                Milestone
                No milestone
                Relationships
                None yet
                Development
                No branches or pull requests
                Issue actions
                
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM · Issue #5 · interactivereport/ScaleSC · GitHub
                Skip to content
                PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM #5
                Open
                Open
                PCA fails with CUDA out-of-memory on large datasets (2.7M cells) despite sufficient VRAM#5
                Assignees
                wx-guyue
                Description
                @wzeng96
                wzeng96
                opened on May 22, 2026
                Hi,
                I'm scalesc to run a big-scale spatial transcriptomic data using it's GPU supported. My data is ~2.7 million cells x 6,175 genes. However, when I tried to run PCA, it would always return me the CUDA memory error. More specifically, I suspect that it's due to the process when it tries to convert the sparse data to dense. Please see the detailed version and error message below. I'm wondering if the team has any insights in fixing this?
                Environment
                • ScaleSC version: 0.1.0
                • CuPy version: 13.6.0
                • GPU: NVIDIA L40S (47.67GB VRAM × 2)
                • CUDA: 13.0, Driver: 580.159.03
                Dataset
                • 2,702,045 cells × 6,157 genes
                • Sparse CSR float32
                • 3,000 HVGs selected via seurat_v3
                Note on initialization
                I'm using a custom wrapper to initialize ScaleSC due to a separate shape reader issue on large datasets:
                definit_scalesc_with_fix(data_dir, **kwargs):
ssc_obj=ssc.ScaleSC(data_dir=data_dir, **kwargs)
adata=ssc_obj.reader._get_anndata_obj()
n_obs, n_vars=adata.shapessc_obj.reader.n_cell=n_obsssc_obj.reader.n_gene=n_varsssc_obj.reader.n_cell_origin=n_obsssc_obj.reader.n_gene_origin=n_varsreturnssc_objssc_obj=init_scalesc_with_fix(
data_dir=SCALESC_DIR,
preload_on_cpu=True,
preload_on_gpu=False,
gpus=[0, 1],
save_after_each_step=True,
max_cell_batch=25000
)
                normalize_log1p and highly_variable_genes both complete successfully. The full pipeline also ran without issues on smaller datasets.
                Problem
                pca() consistently fails with:
                MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /gscratch/stf/wz34/envs/ScaleSC/include/rmm/mr/device/cuda_memory_resource.hpp

                I noticed
                1 — allocation is always exactly ~17GB
                The allocation request is consistently 17383920128B (~17GB) regardless of max_cell_batch value (tested with 100,000, 50,000, 25,000, and 1,000). This suggests the ~17GB is a fixed overhead unrelated to batch size, making max_cell_batch ineffective as a workaround.
                2 — only GPU 0 is used
                GPU memory monitoring confirms GPU 1 is completely unused despite gpus=[0, 1]:
                GPU 0: free=30.69 GB, total=47.67 GB ← all allocations here
GPU 1: free=47.18 GB, total=47.67 GB ← completely idle

                Any insights or help would be greatly appreciated. Thank you!
                Metadata
                Metadata
                Assignees
                Labels
                No labels
                No labels
                Projects
                No projects
                  Milestone
                  No milestone
                  Relationships
                  None yet
                  Development
                  No branches or pull requests
                  Issue actions