Skip to content

[feature-wip] (memory tracker) (step3) Switch TLS mem tracker to separate more detailed memory usage - #8605

Merged
morningman merged 5 commits into
apache:masterfrom
xinyiZzz:switch_tls_tracker
Mar 24, 2022
Merged

[feature-wip] (memory tracker) (step3) Switch TLS mem tracker to separate more detailed memory usage#8605
morningman merged 5 commits into
apache:masterfrom
xinyiZzz:switch_tls_tracker

Conversation

@xinyiZzz

@xinyiZzzxinyiZzz commented Mar 22, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close#7196 (step 3.1/3)

Problem Summary:

In pr #8476, all memory usage of a process is recorded in the process mem tracker, and all memory usage of a query is recorded in the query mem tracker, and it is still necessary to manually call transfer to to track the cached memory size.

We hope to separate out more detailed memory usage based on Hook TCMalloc new/delete + TLS mem tracker.

In this pr, the more detailed mem tracker is switched to TLS, which automatically and accurately counts more detailed memory usage than before.

Checklist(Required)

  1. Does it affect the original behavior: (Yes)
  2. Has unit tests been added: (Yes)
  3. Has document been added or modified: (No)
  4. Does it need to update dependencies: (No)
  5. Are there any changes that cannot be rolled back: (No)

Further comments

  1. Added the method of switching mem tracker to TLS;
  • More accurate statistics of cache memory size. It is no longer necessary to manually calculate the memory size of each add/free cache, which can easily miss memory such as vector resize.
  • (TODO) Separate out the memory usage of each operator from the query mem tracker.
  1. Added callback behavior when switching TLS mem tracker consume fails;
  • Replace all methods of manually determining whether the instance mem tracker has exceeded limit in the loop, such as join node and agg node.

Statistical effect:

  • Initially (after BE restart)
    image

  • Query is executing
    image

  • After Query is over
    image

  • Load is executing
    Note: LoadChannelMgr consumption is 0, because this pr has not really called switch tls mem tracker, and the next pr will be modified in batches.
    image

  • After Load is over
    image

Note: If QueryPool and LoadPool are found to have negative values for a short time, this is normal, refer to the comments in MemTrackerTaskPool::logout_task_mem_tracker.

@morningmanmorningman added the dev/backlog waiting to be merged in future dev branch label Mar 23, 2022
yangzhg
yangzhg reviewed Mar 23, 2022
View reviewed changes
Comment threadbe/src/exec/cross_join_node.cpp Outdated
yangzhg
yangzhg reviewed Mar 23, 2022
View reviewed changes
Comment threadbe/src/exec/except_node.cpp Outdated
Comment threadbe/src/exec/intersect_node.cpp Outdated
Comment threadbe/src/runtime/thread_mem_tracker_mgr.h Outdated
Comment threadbe/src/runtime/thread_mem_tracker_mgr.h Outdated
Comment threadbe/src/vec/exec/vcross_join_node.cpp Outdated
Comment threadbe/src/vec/exec/vset_operation_node.cpp Outdated
Comment threadbe/src/runtime/thread_mem_tracker_mgr.cpp
nullptr, "In TCMalloc Hook, " + _consume_err_call_back.action_type, mem_usage, st);
if (_consume_err_call_back.call_back_func != nullptr) {
_consume_err_call_back.call_back_func();
nullptr, "In TCMalloc Hook, " + _consume_err_cb.cancel_msg, mem_usage, st);

@yangzhgyangzhgMar 23, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
nullptr, "In TCMalloc Hook, " + _consume_err_cb.cancel_msg, mem_usage, st);
nullptr, fmt::format("In TCMalloc Hook, {}", _consume_err_cb.cancel_msg), mem_usage, st);

@xinyiZzzxinyiZzzMar 23, 2022•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix~, for faster.

morningman
morningman reviewed Mar 23, 2022
View reviewed changes
Comment threadbe/src/olap/lru_cache.cpp
}

void ShardedLRUCache::release(Handle* handle) {
SCOPED_SWITCH_THREAD_LOCAL_MEM_TRACKER(_mem_tracker);

@morningmanmorningmanMar 23, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a metric counter for this SWITCH operation?
I want to know how many SWITCH may be called.

@xinyiZzzxinyiZzzMar 23, 2022•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I try to add.

@xinyiZzzxinyiZzzMar 24, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix

morningman
morningman reviewed Mar 24, 2022
View reviewed changes
Comment threadbe/src/olap/lru_cache.cpp
_shards[s] = new LRUCache(type);
_shards[s]->set_capacity(per_shard);
}
thread_local_ctx.get()->_thread_mem_tracker_mgr->clear_untracked_mems();

@morningmanmorningmanMar 24, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why clear untracked mem here?
you can add some comment.

@xinyiZzzxinyiZzzMar 24, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After the lru cache is created in the main thread, the main thread will not switch the lru cache mem tracker again.

After the non-query thread switches the mem tracker, if the thread will not switch the mem tracker again in the short term, can consider manually clear_untracked_mems.

The query thread will automatically clear_untracked_mems when detach_task.

morningman
morningman approved these changes Mar 24, 2022
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Mar 24, 2022

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 24, 2022
@github-actions

github-actionsBot commented Mar 24, 2022

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@morningman

morningman commented Mar 24, 2022

Copy link
Copy Markdown
Contributor

merge for quick test

@morningman
morningman merged commit aaaaae5 into apache:masterMar 24, 2022
@xinyiZzzxinyiZzz mentioned this pull request Mar 25, 2022
Merged
@xinyiZzzxinyiZzz changed the title [feature] (memory) Switch TLS mem tracker to separate more detailed memory usage[feature-wip] (memory) Switch TLS mem tracker to separate more detailed memory usage (Part1)Mar 30, 2022
@xinyiZzzxinyiZzz changed the title [feature-wip] (memory) Switch TLS mem tracker to separate more detailed memory usage (Part1)[feature-wip] (memory tracker) (step3) Switch TLS mem tracker to separate more detailed memory usageApr 7, 2022
morningman pushed a commit that referenced this pull request Apr 8, 2022
@xinyiZzz
[feature-wip] (memory tracker) (step4) Switch TLS mem tracker to sepa…
519305c
…rate more detailed memory usage (#8669)
Based on #8605, Separate out the memory usage of each operator from the Query/Load/StorageEngine mem tracker.
weizhengte pushed a commit to weizhengte/incubator-doris that referenced this pull request Apr 22, 2022
@xinyiZzz
[feature-wip] (memory tracker) (step4) Switch TLS mem tracker to sepa…
fa8022f
…rate more detailed memory usage (apache#8669)
Based on apache#8605, Separate out the memory usage of each operator from the Query/Load/StorageEngine mem tracker.
zhengshiJ pushed a commit to zhengshiJ/incubator-doris that referenced this pull request Apr 27, 2022
@xinyiZzzjianghaochen
[feature-wip] (memory tracker) (step4) Switch TLS mem tracker to sepa…
083fd59
…rate more detailed memory usage (apache#8669)
Based on apache#8605, Separate out the memory usage of each operator from the Query/Load/StorageEngine mem tracker.
starocean999 pushed a commit to starocean999/incubator-doris that referenced this pull request May 19, 2022
@xinyiZzz@starocean999
[feature-wip] (memory tracker) (step4) Switch TLS mem tracker to sepa…
f297031
…rate more detailed memory usage (apache#8669)
Based on apache#8605, Separate out the memory usage of each operator from the Query/Load/StorageEngine mem tracker.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yangzhgyangzhgyangzhg left review comments
@morningmanmorningmanmorningman approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.area/memory-consumptiondev/backlogwaiting to be merged in future dev branchreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature] Refactored memory statistics framework MemTracker

3 participants

@xinyiZzz@morningman@yangzhg