Skip to content

Add memory usage statistics for slabs and allocation classes - #91

Merged
igchor merged 1 commit into
pmem:developfrom
igchor:more_stats
Jul 11, 2022
Merged

Add memory usage statistics for slabs and allocation classes#91
igchor merged 1 commit into
pmem:developfrom
igchor:more_stats

Conversation

@igchor

@igchorigchor commented Jul 6, 2022

Copy link
Copy Markdown

Printing those stats can be enabled by passing -report_memory_usage_stats to cachebench

Sample output:

tid 0 free slabs : 0.00%
tid 1 free slabs : 0.00%
tid 0 pid 0 cid 0 64.00B memorySize: 0.00B
tid 0 pid 0 cid 1 96.00B memorySize: 36.00MB
tid 0 pid 0 cid 2 144.00B memorySize: 124.00MB
tid 0 pid 0 cid 3 216.00B memorySize: 276.00MB
tid 0 pid 0 cid 4 328.00B memorySize: 744.00MB
tid 0 pid 0 cid 5 496.00B memorySize: 992.00MB
tid 0 pid 0 cid 6 744.00B memorySize: 508.00MB
tid 0 pid 0 cid 7 1.09KB memorySize: 416.00MB
tid 0 pid 0 cid 8 1.64KB memorySize: 212.00MB
tid 0 pid 0 cid 9 2.46KB memorySize: 132.00MB
tid 0 pid 0 cid 10 3.70KB memorySize: 152.00MB
tid 0 pid 0 cid 11 5.55KB memorySize: 136.00MB
tid 0 pid 0 cid 12 8.32KB memorySize: 36.00MB
tid 0 pid 0 cid 13 12.48KB memorySize: 20.00MB
tid 0 pid 0 cid 14 18.78KB memorySize: 20.00MB
tid 0 pid 0 cid 15 28.24KB memorySize: 32.00MB
tid 0 pid 0 cid 16 42.66KB memorySize: 200.00MB
tid 0 pid 0 cid 17 64.00KB memorySize: 36.00MB
tid 0 pid 0 cid 18 97.52KB memorySize: 12.00MB
tid 0 pid 0 cid 19 151.70KB memorySize: 4.00MB
tid 0 pid 0 cid 20 227.55KB memorySize: 4.00MB
tid 0 pid 0 cid 21 372.36KB memorySize: 0.00B
tid 0 pid 0 cid 22 585.14KB memorySize: 0.00B
tid 0 pid 0 cid 23 1.00MB memorySize: 0.00B
tid 1 pid 0 cid 0 64.00B memorySize: 0.00B
tid 1 pid 0 cid 1 96.00B memorySize: 40.00MB
tid 1 pid 0 cid 2 144.00B memorySize: 124.00MB
tid 1 pid 0 cid 3 216.00B memorySize: 272.00MB
tid 1 pid 0 cid 4 328.00B memorySize: 760.00MB
tid 1 pid 0 cid 5 496.00B memorySize: 984.00MB
tid 1 pid 0 cid 6 744.00B memorySize: 508.00MB
tid 1 pid 0 cid 7 1.09KB memorySize: 424.00MB
tid 1 pid 0 cid 8 1.64KB memorySize: 208.00MB
tid 1 pid 0 cid 9 2.46KB memorySize: 136.00MB
tid 1 pid 0 cid 10 3.70KB memorySize: 148.00MB
tid 1 pid 0 cid 11 5.55KB memorySize: 132.00MB
tid 1 pid 0 cid 12 8.32KB memorySize: 32.00MB
tid 1 pid 0 cid 13 12.48KB memorySize: 16.00MB
tid 1 pid 0 cid 14 18.78KB memorySize: 20.00MB
tid 1 pid 0 cid 15 28.24KB memorySize: 32.00MB
tid 1 pid 0 cid 16 42.66KB memorySize: 208.00MB
tid 1 pid 0 cid 17 64.00KB memorySize: 32.00MB
tid 1 pid 0 cid 18 97.52KB memorySize: 12.00MB
tid 1 pid 0 cid 19 151.70KB memorySize: 4.00MB
tid 1 pid 0 cid 20 227.55KB memorySize: 0.00B
tid 1 pid 0 cid 21 372.36KB memorySize: 0.00B
tid 1 pid 0 cid 22 585.14KB memorySize: 0.00B
tid 1 pid 0 cid 23 1.00MB memorySize: 0.00B
tid 0 pid 0 cid 0 64.00B free: 100.00%
tid 0 pid 0 cid 1 96.00B free: 0.00%
tid 0 pid 0 cid 2 144.00B free: 0.00%
tid 0 pid 0 cid 3 216.00B free: 0.00%
tid 0 pid 0 cid 4 328.00B free: 0.00%
tid 0 pid 0 cid 5 496.00B free: 0.00%
tid 0 pid 0 cid 6 744.00B free: 0.01%
tid 0 pid 0 cid 7 1.09KB free: 0.02%
tid 0 pid 0 cid 8 1.64KB free: 0.02%
tid 0 pid 0 cid 9 2.46KB free: 0.02%
tid 0 pid 0 cid 10 3.70KB free: 0.04%
tid 0 pid 0 cid 11 5.55KB free: 0.06%
tid 0 pid 0 cid 12 8.32KB free: 0.06%
tid 0 pid 0 cid 13 12.48KB free: 0.03%
tid 0 pid 0 cid 14 18.78KB free: 0.04%
tid 0 pid 0 cid 15 28.24KB free: 0.02%
tid 0 pid 0 cid 16 42.66KB free: 0.01%
tid 0 pid 0 cid 17 64.00KB free: 0.00%
tid 0 pid 0 cid 18 97.52KB free: 0.00%
tid 0 pid 0 cid 19 151.70KB free: 0.00%
tid 0 pid 0 cid 20 227.55KB free: 61.11%
tid 0 pid 0 cid 21 372.36KB free: 100.00%
tid 0 pid 0 cid 22 585.14KB free: 100.00%
tid 0 pid 0 cid 23 1.00MB free: 100.00%
tid 1 pid 0 cid 0 64.00B free: 100.00%
tid 1 pid 0 cid 1 96.00B free: 0.00%
tid 1 pid 0 cid 2 144.00B free: 0.00%
tid 1 pid 0 cid 3 216.00B free: 0.00%
tid 1 pid 0 cid 4 328.00B free: 0.00%
tid 1 pid 0 cid 5 496.00B free: 0.00%
tid 1 pid 0 cid 6 744.00B free: 0.01%
tid 1 pid 0 cid 7 1.09KB free: 0.02%
tid 1 pid 0 cid 8 1.64KB free: 0.02%
tid 1 pid 0 cid 9 2.46KB free: 0.02%
tid 1 pid 0 cid 10 3.70KB free: 0.04%
tid 1 pid 0 cid 11 5.55KB free: 0.06%
tid 1 pid 0 cid 12 8.32KB free: 0.06%
tid 1 pid 0 cid 13 12.48KB free: 0.03%
tid 1 pid 0 cid 14 18.78KB free: 0.04%
tid 1 pid 0 cid 15 28.24KB free: 0.02%
tid 1 pid 0 cid 16 42.66KB free: 0.01%
tid 1 pid 0 cid 17 64.00KB free: 0.00%
tid 1 pid 0 cid 18 97.52KB free: 0.00%
tid 1 pid 0 cid 19 151.70KB free: 0.00%
tid 1 pid 0 cid 20 227.55KB free: 100.00%
tid 1 pid 0 cid 21 372.36KB free: 100.00%
tid 1 pid 0 cid 22 585.14KB free: 100.00%
tid 1 pid 0 cid 23 1.00MB free: 100.00%

This change is Reviewable

@raema

raema commented Jul 6, 2022

Copy link
Copy Markdown
Collaborator

Hi @igchor, most of the metrics have the name and value separated by a single colon like:
'Hit Ratio : 92.07%'
That metric appears in prometheus named 'hit_ratio'

For the line below, I'm thinking how we would parse this output into 'name, value' for prometheus
tid : 0, pid : 0, cid : 0, alloc size : 64.00B, memory size : 0.00B, free : 100.00%

If we want to follow the same format as other metrics maybe we could use this format:
tid0 pid0 cid0 64.00KB size : 0.00B
tid0 pid0 cid0 64.00KB free : 100.00%

Does it make sense to change the stats output to use this format or something like this? Our existing scripts will automatically collect new metrics output in this format.

@guptask

Copy link
Copy Markdown
Collaborator

cachelib/allocator/CacheAllocator-inl.h line 2510 at r1 (raw file):

template <typename CacheTrait>
double CacheAllocator<CacheTrait>::slabsFreePercentage(TierId tid) const

should this name have approx term in it as well ?

@guptask

Copy link
Copy Markdown
Collaborator

cachelib/allocator/memory/AllocationClass.cpp line 54 at r1 (raw file):

slabAlloc_(s),
freedAllocations_{slabAlloc_.createSingleTierPtrCompressor<FreeAlloc>()} {
curAllocatedSlabs_ = allocatedSlabs_.size();

Do you want to use memory_order_seq_cst order here for atomic store ? Otherwise store() is needed if you want to use something like memory_order_release order.

@guptask

Copy link
Copy Markdown
Collaborator

cachelib/allocator/memory/AllocationClass.cpp line 126 at r1 (raw file):

 }
curAllocatedSlabs_ = allocatedSlabs_.size();

same as above comment about memory order if using assignment operator instead of atomic store().

@guptask

Copy link
Copy Markdown
Collaborator

cachelib/allocator/memory/AllocationClass.cpp line 741 at r1 (raw file):

doubleAllocationClass::approxFreePercentage() const {
if (getNumSlabs() == 0)

Cosmetic review : I suggest using curly braces even if there is one line after if statement.

@guptask

guptask commented Jul 6, 2022

Copy link
Copy Markdown
Collaborator

cachelib/allocator/tests/CacheBaseTest.cpp line 36 at r1 (raw file):

const MemoryPool& getPool(PoolId) constoverride { return memoryPool_; }
PoolStats getPoolStats(PoolId) constoverride { returnPoolStats(); }
AllocationClassBaseStat getAllocationClassStats(TierId tid, PoolId, ClassId) const {returnAllocationClassBaseStat();};

am not sure this line will pass the clang-format check. the character count is high.

@guptaskguptask left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 14 of 14 files at r1, all commit messages.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @byrnedj and @igchor)

@igchorigchor left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@raema Ok, I changed it to how you suggested. Please let me now if this is OK.

Reviewable status: 7 of 15 files reviewed, 5 unresolved discussions (waiting on @byrnedj and @guptask)


cachelib/allocator/CacheAllocator-inl.h line 2510 at r1 (raw file):

Previously, guptask (Sounak Gupta) wrote…

should this name have approx term in it as well ?

Done.


cachelib/allocator/memory/AllocationClass.cpp line 54 at r1 (raw file):

Previously, guptask (Sounak Gupta) wrote…

Do you want to use memory_order_seq_cst order here for atomic store ? Otherwise store() is needed if you want to use something like memory_order_release order.

It's just for readability, the order doesn't matter here since it's ctor.


cachelib/allocator/memory/AllocationClass.cpp line 126 at r1 (raw file):

Previously, guptask (Sounak Gupta) wrote…

same as above comment about memory order if using assignment operator instead of atomic store().

Same as above.


cachelib/allocator/memory/AllocationClass.cpp line 741 at r1 (raw file):

Previously, guptask (Sounak Gupta) wrote…

Cosmetic review : I suggest using curly braces even if there is one line after if statement.

Done.


cachelib/allocator/tests/CacheBaseTest.cpp line 36 at r1 (raw file):

Previously, guptask (Sounak Gupta) wrote…

am not sure why this line passed the clang-format check ? the character count is high.

Done.

@raema

raema commented Jul 7, 2022

Copy link
Copy Markdown
Collaborator

@raema Ok, I changed it to how you suggested. Please let me now if this is OK.

This output looks good. Thanks!

}
}

auto maxBatch = *std::max_element(batches.begin(), batches.end());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will segfault if batches.size() == 0

@byrnedjbyrnedj left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

line 52 of PromotionStrategey.h will segfault if vector is empty. did you mean to include PromotionStrategy.h on this PR?

@igchorigchor left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I commited this file by mistake.

Reviewable status: 7 of 15 files reviewed, 6 unresolved discussions (waiting on @byrnedj and @guptask)


cachelib/allocator/PromotionStrategy.h line 52 at r2 (raw file):

Previously, byrnedj (Daniel Byrne) wrote…

will segfault if batches.size() == 0

Done.

@byrnedjbyrnedj left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: 7 of 15 files reviewed, 5 unresolved discussions (waiting on @byrnedj and @guptask)

@igchor
igchor merged commit 57a3d1c into pmem:developJul 11, 2022
@igchor
igchor deleted the more_stats branch July 11, 2022 14:56
guptask added a commit to guptask/CacheLib that referenced this pull request Aug 7, 2023
Track latency of per item eviction/promotion between memory tiers
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@igchor@raema@guptask@byrnedj