Skip to content
This repository was archived by the owner on May 12, 2021. It is now read-only.

TAJO-2111: Optimize Partition Table Split Computation for Amazon S3 - #994

Closed
blrunner wants to merge 219 commits into
apache:masterfrom
blrunner:s3-split-improvement
Closed

TAJO-2111: Optimize Partition Table Split Computation for Amazon S3#994
blrunner wants to merge 219 commits into
apache:masterfrom
blrunner:s3-split-improvement

Conversation

@blrunner

Copy link
Copy Markdown
Contributor

It depends on #846 and #953.

Here is my benchmark results as follows.

Configuration

  • EC2 instance type : c3.xlarge
  • Cluster: 1 master, 3 worker
  • Dataset: TPC-H (factor = 1)
  • Partition table schema
CREATE EXTERNAL TABLE lineitem_p (l_orderkey INT8, l_partkey INT8, l_suppkey INT8, l_linenumber INT8, l_quantity FLOAT8, l_extendedprice FLOAT8, l_discount FLOAT8, l_tax FLOAT8, l_returnflag TEXT, l_linestatus TEXT, l_commitdate text, l_receiptdate text, l_shipinstruct TEXT, l_shipmode TEXT, l_comment TEXT)
USING TEXT WITH ('text.delimiter'='|')
PARTITION BY COLUMN(l_shipdate text)
LOCATION 's3://Xyz';
  • Partition numbers of lineitem table: 2526 (each partitions includes just one file)

Queries

  • Q1: select * from lineitem_p limit 5;
  • Q2: select count(*) from lineitem_p;
  • Q3: select count(*) from lineitem_p where l_shipdate > '1994-09-25' and l_shipdate < '1994-10-10';

Query Execution Time

QueryNo OptimizedOptimizedImprovement
Q1573.425 sec4.228 sec135.6x
Q2653.175 sec33.444 sec19.5x
Q34.099 sec2.429 sec1.6x

Split Computation Time

QueryNo OptimizedOptimizedImprovement
Q1572921 ms2233 ms256.5x
Q2599437 ms701 ms855.1x
Q32537 ms388 ms6.5x

…into TAJO-1952
Conflicts:
tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
…into TAJO-1952
Conflicts:
tajo-plan/src/main/java/org/apache/tajo/plan/logical/PartitionedTableScanNode.java
…into TAJO-1952
Conflicts:
tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
@blrunner

Copy link
Copy Markdown
ContributorAuthor

I'll reopen this PR after finishing #1020 and #1024.

Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@blrunner