Skip to content

ORC-361: Remove the single thread restriction on the MemoryManagerImpl. - #433

Closed
omalley wants to merge 2 commits into
apache:masterfrom
omalley:orc-361
Closed

ORC-361: Remove the single thread restriction on the MemoryManagerImpl.#433
omalley wants to merge 2 commits into
apache:masterfrom
omalley:orc-361

Conversation

@omalley

@omalleyomalley commented Oct 4, 2019

Copy link
Copy Markdown
Contributor

The current MemoryManagerImpl assumes that only a single thread is writing ORC files. I've changed the design so that:

  • There is a single MemoryManagerImpl object for all of the threads.
  • The only expensive operations are when new writers are created or closed.
  • There is a new method (checkMemory) that is used to determine if the scale has changed. It uses an atomic long to detect changes without contention.
  • The rows between memory checks is moved into the writer.

@omalley

omalley commented Oct 4, 2019

Copy link
Copy Markdown
ContributorAuthor

I simplified the new MemoryManager implementation some more.

@omalley
ORC-361: Remove the single thread restriction on the MemoryManagerImpl.
6d4b830
Fixesapache#433
Signed-off-by: Owen O'Malley <omalley@apache.org>
omalley added a commit to omalley/orc that referenced this pull request Oct 7, 2019
@omalley
ORC-361: Remove the single thread restriction on the MemoryManagerImpl.
d1a85e0
Fixesapache#433
Signed-off-by: Owen O'Malley <omalley@apache.org>
omalley added a commit to omalley/orc that referenced this pull request Oct 8, 2019
@omalley
ORC-361: Remove the single thread restriction on the MemoryManagerImpl.
3c29fef
Fixesapache#433
Signed-off-by: Owen O'Malley <omalley@apache.org>
@omalley
ORC-361: Remove the single thread restriction on the MemoryManagerImpl.
11db9a6
Fixesapache#433
Signed-off-by: Owen O'Malley <omalley@apache.org>
omalley added a commit that referenced this pull request Oct 8, 2019
@omalley
ORC-361: Remove the single thread restriction on the MemoryManagerImpl.
20e732f
Fixes#433
Signed-off-by: Owen O'Malley <omalley@apache.org>
omalley added a commit that referenced this pull request Oct 8, 2019
@omalley
ORC-361: Remove the single thread restriction on the MemoryManagerImpl.
3aed316
Fixes#433
Signed-off-by: Owen O'Malley <omalley@apache.org>
zratkai pushed a commit to zratkai/orc that referenced this pull request Jun 20, 2024
@omalleyLaszlo Attila Toth
CDPD-29488 ORC-361: Remove the single thread restriction on the Memor…
b53166f
…yManagerImpl.
Fixesapache#433
Signed-off-by: Owen O'Malley <omalley@apache.org>
(cherry picked from commit d657ed4)
(cherry picked from commit 54900be)
Change-Id: Ie2afc5fab07bf2a4b1d1562a341665bcfc0b5c78
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@omalley@t3rmin4t0r