`partial_request` cache count keeps increasing

[earlbread]

Description

We set partial_request_cache_capacity to 256MB. However, when checking the metrics, the cache count appears to keep increasing, and as a result, the Searcher’s memory usage also keeps growing.

Upon further inspection, other caches show a pattern of growing and shrinking over time, but the partial_request cache only increases continuously. Eventually, when the cache grows large enough, the process runs out of memory (OOM).

Looking at the quickwit_cache_in_cache_num_bytes metric, the size occupied by partial_request is below 250MB. Doesn’t this metric represent the total size of the partial_request cache? If I’m misunderstanding the meaning of partial_request_cache_capacity, is there another way to limit the number of partial_request cache entries?

As a workaround, we’ve confirmed that setting partial_request_cache_capacity to 0 prevents the cache count from increasing.

I’ve attached the Quickwit configuration and metric screenshots below for reference.

Metrics

• Number Of Caches (partial_request)

• Number Of Caches (Others)

• Size Of Caches (partial_request)

• Size Of Caches (Others)

• Searcher Memory

Quickwit Node Config

Version: quickwit:qw-airmail-20250522-hotfix

{
  "cluster_id": "quickwit-quickwit-searcher",
  "node_id": "quickwit-searcher-searcher-17",
  "enabled_services": [
    "searcher"
  ],
  "gossip_listen_addr": "0.0.0.0:7282",
  "grpc_listen_addr": "0.0.0.0:7281",
  "gossip_advertise_addr": "10.128.77.87:7282",
  "grpc_advertise_addr": "10.128.77.87:7281",
  "gossip_interval": {
    "secs": 1,
    "nanos": 0
  },
  "peer_seeds": [
    "quickwit-searcher-headless"
  ],
  "data_dir_path": "/quickwit/qwdata",
  "metastore_uri": "file:///quickwit/qwdata/indexes#polling_interval=30s",
  "default_index_root_uri": "s3://prod-kr-quickwit-index/indexes",
  "rest_config": {
    "listen_addr": "0.0.0.0:7280",
    "cors_allow_origins": [],
    "extra_headers": {},
    "tls": null
  },
  "grpc_config": {
    "max_message_size": "52.4 MB",
    "tls": null
  },
  "storage_configs": {
    "s3": {
      "access_key_id": null,
      "secret_access_key": null,
      "region": "ap-northeast-2",
      "endpoint": "https://s3.ap-northeast-2.amazonaws.com",
      "force_path_style_access": false,
      "disable_multi_object_delete": false,
      "disable_multipart_upload": false
    }
  },
  "metastore_configs": {},
  "indexer_config": {
    "split_store_max_num_bytes": "107.4 GB",
    "split_store_max_num_splits": 1000,
    "max_concurrent_split_uploads": 12,
    "max_merge_write_throughput": null,
    "merge_concurrency": 2,
    "enable_otlp_endpoint": true,
    "enable_cooperative_indexing": false,
    "cpu_capacity": "4000m"
  },
  "searcher_config": {
    "aggregation_memory_limit": "500.0 MB",
    "aggregation_bucket_limit": 65000,
    "fast_field_cache_capacity": "4.0 GB",
    "split_footer_cache_capacity": "1000.0 MB",
    "partial_request_cache_capacity": "256.0 MB",
    "max_num_concurrent_split_searches": 200,
    "max_num_concurrent_split_streams": 100,
    "request_timeout_secs": 30,
    "warmup_memory_budget": "100.0 GB",
    "warmup_single_split_initial_allocation": "1000.0 MB"
  },
  "ingest_api_config": {
    "max_queue_memory_usage": "2.1 GB",
    "max_queue_disk_usage": "4.3 GB",
    "replication_factor": 1,
    "content_length_limit": "10.5 MB",
    "shard_throughput_limit": "5.2 MB",
    "shard_burst_limit": "52.4 MB",
    "shard_scale_up_factor": 1.5
  },
  "jaeger_config": {
    "enable_endpoint": true,
    "lookback_period_hours": 72,
    "max_trace_duration_secs": 3600,
    "max_fetch_spans": 10000
  }
}

[trinity-1686a]

cache size only account for the size of values, not of keys. I guess you may be somehow creating a situation where values are very small (count only query with no returned docs, queries where most splits reply with zero or close to zero documents, some form of aggregations maybe?), where ignoring the weight of keys is problematic.
Do you know the distributions of queries in your workload?

We could try accounting for the key size, which would probably solve this issue, but maybe we want to do something like computing a 128b hash of the cache key (that should be plenty to prevent collisions in any realistic scenario), which could save a fair bit of memory

[earlbread]

Thanks for the response. For now, setting partial_request_cache_capacity to 64MB has stabilized the memory.