Support read and write lock in table level to reduce lock competition by caiconghui · Pull Request #3775 · apache/doris

caiconghui · 2020-06-04T12:22:55Z

This PR is to reduce lock competition by supporting read and write lock in table level. When we modify or read table's meta, we don't need to get db lock, just get table write or read lock. And when we get db lock, that means meta directly in db cannot be modified by other thread. Db lock only protect meta in Database class, while table lock protect meta in Table class.

caiconghui · 2020-06-04T12:23:07Z

ref #3743

morningman

Hi @caiconghui , first of all, this change is worth it.

But I don't think this is a good way to achieve it. In the previous implementation, it was not a good design to expose the database lock outside, and there have been multiple deadlocks caused by lock nesting problems.

When the lock granularity is refined to the table level, this way of exposing locks will bring more error-proneness. And so many changes to locks in one PR are difficult to review.

I suggest to refer to the implementation of kudu MetadataGroupLock to abstract the operation of the lock and then gradually modify the relevant code.

kangkaisen · 2020-06-05T14:56:39Z

I just reviewed a little, and submitted review accidentally.

kangkaisen · 2020-06-05T15:01:05Z

Hi @caiconghui , first of all, this change is worth it.
But I don't think this is a good way to achieve it. In the previous implementation, it was not a good design to expose the database lock outside, and there have been multiple deadlocks caused by lock nesting problems.
When the lock granularity is refined to the table level, this way of exposing locks will bring more error-proneness. And so many changes to locks in one PR are difficult to review.
I suggest to refer to the implementation of kudu MetadataGroupLock to abstract the operation of the lock and then gradually modify the relevant code.

+1.

caiconghui · 2020-06-16T12:22:03Z

It is hard to review the changed files at one time, so I classify the modified files into different groups. Please Pay more attention to Alter, Catalog, Load theme, which could be quite error prone. And now I don't change httpv2 code, because it is not used now, so we can modify it later.
Alter
1 Alter.java
2 AlterHandler.java
3 AlterJobV2.java
4 MaterializedViewHandler.java
5 RollupJob.java
6 RollupJobV2.java
7 SchemaChangeHandler.java
8 SchemaChangeJob.java
9 SchemaChangeJobV2.java

Stmt

10 DescribeStmt.java
11 ExportStmt.java
12 ShowDataStmt.java
13 ShowPartitionsStmt.java
14 InsertStmt.java
15 QueryStmt.java
16 SelectStmt.java
17 SetOperationStmt.java
18 WithClause.java
19 StmtExecutor.java
20 ShowExecutor.java

Backup

21 BackupHandler.java
22 BackupJob.java
23 RestoreJob.java

Catalog

24 Catalog.java
25 CatalogRecycleBin.java

Colocate

26 ColocateTableIndex.java
27 ColocateTableBalancer.java

Meta

28 Database.java
29 MetadataViewer.java
30 OlapTable.java
31 Table.java
32 InfoSchemaDb.java
33 MetaLockUtils.java

MetaManager

34 TabletStatMgr.java
35 DynamicPartitionScheduler.java
36 TabletChecker.java
37 TabletSchedCtx.java
38 TabletScheduler.java

Proc

39 EsPartitionsProcDir.java
40 EsShardProcDir.java
41 IndexInfoProcDir.java
42 IndicesProcDir.java
43 PartitionsProcDir.java
44 StatisticProcDir.java
45 TablesProcDir.java
46 TabletsProcDir.java

Check

47 CheckConsistencyJob.java
48 ConsistencyChecker.java

Rest

49 GetDdlStmtAction.java
50 MigrationAction.java
51 RowCountAction.java
52 StorageTypeCheckAction.java
53 TableQueryPlanAction.java
54 TableRowCountAction.java
55 TableSchemaAction.java
56 ShowDataAction.java
57 MetaService.java

Load
58 BrokerFileGroup.java
59 DeleteHandler.java
60 DeleteJob.java
61 ExportJob.java
62 Load.java
63 LoadChecker.java
64 BrokerLoadJob.java
65 LoadManager.java
66 KafkaRoutineLoadJob.java
67 RoutineLoadJob.java
68 BulkLoadJob.java
69 LoadJob.java
70 SparkLoadJob.java

System

71 ReportHandler.java
72 SystemInfoService.java
73 ConnectProcessor.java

Task

74 HadoopLoadPendingTask.java
75 LoadEtlTask.java
76 MiniLoadPendingTask.java
77 StreamLoadTask.java
78 SparkLoadPendingTask.java

TransactionMgr

79 DatabaseTransactionMgr.java
80 GlobalTransactionMgr.java
81 PublishVersionDaemon.java

Rpc

82 FrontendServiceImpl.java
83 MasterImpl.java

Test
84 SchemaChangeJobV2Test.java
85 RollupJobV2Test.java
86 DatabaseTest.java
87 AlterTest.java
88 DemoTest.java
89 AnotherDemoTest.java
90 MetaLockUtilsTets.java
91 InfoSchemaDbTest.java
92 TableTest.java
93 DeleteHandlerTest.java
94 StreamLoadPlannerTest.java
95 StreamLoadScanNodeTest.java
96 DatabaseTransactionMgrTest.java
97 GlobalTransactionMgrTest.java
98 StmtExecutorTest.java
99 SparkLoadJobTest.java

Looking forwards to lively discussions.

@kangkaisen @morningman @wangbo

caiconghui · 2020-11-10T03:47:33Z

Main Modification and Some Lock rule to discuss.

For meta data viewing, just simple use table read lock to replace database read lock.
For meta reading, we don't pursue strong consistency, our goal is to ensure not dead lock and thread safe, such as db.getTable do not need db read lock.
In order to escape dead lock, we keep db lock -> table lock -> other lock, for same meta level, we must sort meta by unique id before lock meta list.
Although db lock to table lock modification is large and difficult to review, but I think the heavy work is Meta modification, which in Alter, Load, Catalog theme, and logic in meta viewing is clear and less error prone.
When change property in one meta, we need to change meta lock, such as load, we only need lock table, but when we change property both in meta and its parent meta, such rename operation, we must lock both db and table
Finally. reduce unneeded lock, such as lock the same meta twice in one operation, reduce the lock scope to reduce the lock competition.

morningman · 2020-11-25T11:47:17Z

- db.writeUnlock();
- }
+ String tableName = stmt.getTableName().getTbl();
+ OlapTable olapTable = (OlapTable) db.getTableOrThrowException(tableName, TableType.OLAP);


Missing if (olapTable.getState() != OlapTableState.NORMAL) { check?

processDropMaterializedView has check table state operation

morningman · 2020-11-25T11:56:16Z

- }
- } finally {
- db.writeUnlock();
+ switch (table.getType()) {


I think we can just lock the table outside the switch, avoid call lock/unlock everywhere inside the processAlterOlapTable and processAlterExternalTable.

some function need db lock and table lock but some function only need table lock, what't more, If the nesting function is too deep， it is easy to repeat locking， if we lock small scope for modificaion， which will be more clear although we may write many lock code

morningman · 2020-11-25T12:00:52Z

- break;
- } else if (alterClause instanceof ColumnRenameClause) {
- Catalog.getCurrentCatalog().renameColumn(db, table, (ColumnRenameClause) alterClause);
+ db.writeLock();


Better put these locks inside the renameTable method.

morningman · 2020-11-25T12:01:09Z

 break;
 } else {
- Preconditions.checkState(false);
+ table.writeLock();


Put lock inside the method

morningman · 2020-11-25T12:01:34Z

 for (AlterClause alterClause : alterClauses) {
 if (alterClause instanceof TableRenameClause) {
- Catalog.getCurrentCatalog().renameTable(db, table, (TableRenameClause) alterClause);
+ db.writeLock();


Put lock inside the method

morningman · 2020-11-27T01:53:17Z

 // KIND, either express or implied. See the License for the
 // specific language governing permissions and limitations
 // under the License.
+// Licensed to the Apache Software Foundation (ASF) under one


Why this license is different?

It is an incorrect modification.

morningman · 2020-11-27T01:59:05Z

 }
-
-lock(dbs);
+ List<Table> tables = Lists.newArrayList(tableMap.values());


Are you sure this can get table list in order?
I think it is more safe and clear to change tableMap to tableList, and sort the tableList explicitly.

yes, tableMap is a treeMap which is used to for previous analysis, the treeMap will keep table id in In ascending order,

morningman · 2020-11-27T02:06:42Z

- } else {
- LOG.warn("failed to update backend report version, db {} does not exist", dbId);
- }
+ atomicLong.set(newReportVersion);


In original implementation, we use db locks to ensure mutual exclusion and order.
So here we may still need to add table locks.

atomicLong is AtomicLong,so I think it is meanless for add table lock or db lock?

morningman · 2020-11-27T02:13:07Z

- Map<String, EtlPartitionConf> etlPartitions = createEtlPartitions();
- Preconditions.checkNotNull(etlPartitions);
- taskConf.setEtlPartitions(etlPartitions);
+ EtlTaskConf taskConf = new EtlTaskConf();


Table's lock should be held to call method like createEtlPartitions()

add table lock in createEtlPartitions

morningman · 2020-11-27T02:23:49Z

+ continue;
+ }
+ OlapTable olapTable = (OlapTable) table;
+ olapTable.readLock();


Is it more safe to lock all tables at once outside the for loop?

I think the final result is same if the table is dropped before loop or in loop, we just make sure that it is thread safe, and we has already add committed txn check if user want to drop table during loading data

wangbo · 2020-12-04T12:33:05Z

 boolean skipAddPartition = false;
 OlapTable olapTable;
- db.readLock();
+ olapTable = (OlapTable) db.getTable(tableId);


When drop a table from database, a db lock will be held;
Why not get a db read lock here when get a table from database.

@wangbo
because when drop table, we need to prevent that other thread drop table or create table too. but for get table, we just need to ensure that the get table operation is thread safe, and the final result is ok, we don't purse the strict consistency here for better performance.

Another consideration is to avoid deadlock， all lock sequence is db lock -> table lock -> other lock, and if we has get table lock and sometimes need to get table again from db, there may cause dead lock, so db get table operation not get db read lock anymore

I get, you mean Performance is ahead of NPE.
For the second point about lock sequence .
I think it should become programming specifications for Doris.
1 If try to lock multiple tables or dbs, it should use getXXXInOrderMethod. 2 If try to lock different type locks, lock sequence should be guaranteed. 3 The appearance of nested locks of the same type should be avoided.

morningman

LGTM

…apache#3775) This PR is to reduce lock competition by supporting read and write lock in table level. When we modify or read table's meta, we don't need to get database lock, just get table write or read lock. And when we get database lock, that means meta directly in db cannot be modified by other thread. Database lock only protect meta in Database class, while table lock protect meta in Table class. Change-Id: Icec6f39f58708950c786059edaeb474a8aa0e324

…3763) (apache#3775)

kangkaisen reviewed Jun 5, 2020
View reviewed changes

Comment threadfe/src/main/java/org/apache/doris/alter/Alter.java Outdated

kangkaisen reviewed Jun 5, 2020
View reviewed changes

Comment threadfe/src/main/java/org/apache/doris/alter/Alter.java Outdated

kangkaisen reviewed Jun 5, 2020
View reviewed changes

Comment threadfe/src/main/java/org/apache/doris/alter/MaterializedViewHandler.java

morningman requested changes Jun 5, 2020
View reviewed changes

morningman added the api-review Categorizes an issue or PR as actively needing an API review. label Jun 5, 2020

morningman self-assigned this Jun 5, 2020

morningman added the area/catalog Issues or PRs related to catalog management label Jun 5, 2020

caiconghui force-pushed the table_lock branch from e9e9d82 to 940e992Compare June 15, 2020 00:58

caiconghui force-pushed the table_lock branch from f148915 to 81519dbCompare June 25, 2020 03:29

caiconghui force-pushed the table_lock branch 2 times, most recently from 2c247dc to 5c76212Compare September 24, 2020 14:58

caiconghui force-pushed the table_lock branch 2 times, most recently from f6a0178 to 2300dceCompare October 22, 2020 13:21

caiconghui force-pushed the table_lock branch 3 times, most recently from 0aed452 to d8f4bcdCompare November 10, 2020 03:34

caiconghui force-pushed the table_lock branch 2 times, most recently from 304942c to 45ef112Compare November 19, 2020 09:48

caiconghui force-pushed the table_lock branch from 45ef112 to 1585a01Compare November 23, 2020 02:57

morningman reviewed Nov 27, 2020
View reviewed changes

caiconghui force-pushed the table_lock branch 2 times, most recently from 6144347 to 1485ab0Compare December 2, 2020 07:41

wangbo reviewed Dec 3, 2020
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/common/util/MetaLockUtils.java

wangbo reviewed Dec 4, 2020
View reviewed changes

caiconghui force-pushed the table_lock branch from 17c0d45 to 1a02c13Compare December 28, 2020 03:19

caiconghui [蔡聪辉]and others added 19 commits January 12, 2021 15:58

Fix SparkLoadJobTest Failed
6a98354

Fix save image lock
b3b30a7

Add unit test for MetaLockUtils
619b71c

use MetadataLockUtils to lock table when finish transaction
8fb1e32

remove useless code of dropTableWithLock for Database
11e4e07

remove useless db read lock
1b089e4

remove useless db read lock and change some db read lock to table locks
841d3f8

fix
520cd27

fix
a2faf9d

fix
26f39d1

Fix dead lock bug
e518f1e

fix unit test failed for SelectStmt
7563bfb

Fix by review
7dfc2f8

Add table lock when update BackendReportVersion in SystemInfoService
6319533

change db level lock to table lock level in http2
8bb19a2

Fix broker load failed bug and add unit test for db
b4b6807

fix
b2f55d7

Add some comment for the usage of MetaLockUtils
faa6eb2

apply fix patch
56b8d31

caiconghui force-pushed the table_lock branch from 29ac965 to 56b8d31Compare January 12, 2021 08:00

morningman approved these changes Jan 12, 2021
View reviewed changes

caiconghui added the approved Indicates a PR has been approved by one committer. label Jan 12, 2021

caiconghui merged commit f773003 into apache:masterJan 13, 2021

morningman mentioned this pull request Jan 21, 2021
[Bug] Fix bug that recover table throw NPE #5279
Merged
1 task

yangzhg mentioned this pull request Feb 9, 2021
Release Notes 0.14.0 #5374
Closed

ccoffline mentioned this pull request Aug 5, 2021
[Proposal] Enforce null check at Catalog.getDb and Database.getTable #6386
Closed

caiconghui mentioned this pull request Sep 14, 2021
[Bug] The MetaObject lock design of fe would cause some problems with consistent meta when catalog do replay operation #6649
Closed
3 tasks

caiconghui deleted the table_lock branch August 23, 2023 03:37

hubgeter pushed a commit to hubgeter/doris that referenced this pull request Mar 12, 2025
[BugFix](PreparedStatement) fix stmtId overflowapache#47950(apache#…
4a3c19c
…3763) (apache#3775)

Uh oh!

Conversation

caiconghui commented Jun 4, 2020

Uh oh!

caiconghui commented Jun 4, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

kangkaisen commented Jun 5, 2020

Uh oh!

kangkaisen commented Jun 5, 2020

Uh oh!

caiconghui commented Jun 16, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

caiconghui commented Nov 10, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

caiconghuiDec 4, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

caiconghui commented Jun 4, 2020•
edited
Loading

caiconghui commented Jun 16, 2020•
edited
Loading

caiconghui commented Nov 10, 2020•
edited
Loading

caiconghuiDec 4, 2020•
edited
Loading