Skip to content

STORM-2242: Persisting Trident state honors configured batch.size.rows - #1824

Closed
sakanaou wants to merge 1 commit into
apache:1.0.x-branchfrom
sakanaou:STORM-2242
Closed

STORM-2242: Persisting Trident state honors configured batch.size.rows#1824
sakanaou wants to merge 1 commit into
apache:1.0.x-branchfrom
sakanaou:STORM-2242

Conversation

@sakanaou

Copy link
Copy Markdown
Contributor

Persisting the Trident state in org.apache.storm.cassandra.trident.state.CassandraState with batching enabled does not honor the configuration for cassandra.batch.size.rows.

This results in a warning at least:

10:33:33.720 [SharedPool-Worker-16] WARN o.a.c.cql3.statements.BatchStatement - Batch of prepared statements for [gin.ngram_count] is of size 5200, exceeding specified threshold of 5120 by 80.

An exception like this is also possible:

10:30:54.287 [SharedPool-Worker-1] ERROR o.a.c.cql3.statements.BatchStatement - Batch of prepared statements for [gin.df] is of size 103428, exceeding specified threshold of 51200 by 52228. (see batch_size_fail_threshold_in_kb)
10:30:54.295 [Thread-29-b-1-executor[7 7]] WARN o.a.s.c.trident.state.CassandraState - Batch write operation is failed.
10:30:54.297 [Thread-29-b-1-executor[7 7]] ERROR org.apache.storm.daemon.executor -
com.datastax.driver.core.exceptions.InvalidQueryException: Batch too large
at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50) ~[cassandra-driver-core-3.1.0.jar:na]
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) ~[cassandra-driver-core-3.1.0.jar:na]
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245) ~[cassandra-driver-core-3.1.0.jar:na]
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64) ~[cassandra-driver-core-3.1.0.jar:na]
at org.apache.storm.cassandra.trident.state.CassandraState.updateState(CassandraState.java:159) ~[storm-cassandra-1.0.2.IQSER_20161212.jar:1.0.2.IQSER_20161212]
at org.apache.storm.cassandra.trident.state.CassandraStateUpdater.updateState(CassandraStateUpdater.java:34) [storm-cassandra-1.0.2.IQSER_20161212.jar:1.0.2.IQSER_20161212]
at org.apache.storm.cassandra.trident.state.CassandraStateUpdater.updateState(CassandraStateUpdater.java:30) [storm-cassandra-1.0.2.IQSER_20161212.jar:1.0.2.IQSER_20161212]
at org.apache.storm.trident.planner.processor.PartitionPersistProcessor.finishBatch(PartitionPersistProcessor.java:98) [storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.trident.planner.SubtopologyBolt.finishBatch(SubtopologyBolt.java:151) [storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.trident.topology.TridentBoltExecutor.finishBatch(TridentBoltExecutor.java:266) [storm-core-1.0.2.jar:1.0.2]

This effectivly disables the usage of batching.

d2r pushed a commit to d2r/storm that referenced this pull request Oct 16, 2018
We are closing stale Pull Requests to make the list more manageable.
Please re-open any Pull Request that has been closed in error.
Closesapache#608Closesapache#639Closesapache#640Closesapache#648Closesapache#662Closesapache#668Closesapache#692Closesapache#705Closesapache#724Closesapache#728Closesapache#730Closesapache#753Closesapache#803Closesapache#854Closesapache#922Closesapache#986Closesapache#992Closesapache#1019Closesapache#1040Closesapache#1041Closesapache#1043Closesapache#1046Closesapache#1051Closesapache#1078Closesapache#1146Closesapache#1164Closesapache#1165Closesapache#1178Closesapache#1213Closesapache#1225Closesapache#1258Closesapache#1259Closesapache#1268Closesapache#1272Closesapache#1277Closesapache#1278Closesapache#1288Closesapache#1296Closesapache#1328Closesapache#1342Closesapache#1353Closesapache#1370Closesapache#1376Closesapache#1391Closesapache#1395Closesapache#1399Closesapache#1406Closesapache#1410Closesapache#1422Closesapache#1427Closesapache#1443Closesapache#1462Closesapache#1468Closesapache#1483Closesapache#1506Closesapache#1509Closesapache#1515Closesapache#1520Closesapache#1521Closesapache#1525Closesapache#1527Closesapache#1544Closesapache#1550Closesapache#1566Closesapache#1569Closesapache#1570Closesapache#1575Closesapache#1580Closesapache#1584Closesapache#1591Closesapache#1600Closesapache#1611Closesapache#1613Closesapache#1639Closesapache#1703Closesapache#1711Closesapache#1719Closesapache#1737Closesapache#1760Closesapache#1767Closesapache#1768Closesapache#1785Closesapache#1799Closesapache#1822Closesapache#1824Closesapache#1844Closesapache#1874Closesapache#1918Closesapache#1928Closesapache#1937Closesapache#1942Closesapache#1951Closesapache#1957Closesapache#1963Closesapache#1964Closesapache#1965Closesapache#1967Closesapache#1968Closesapache#1971Closesapache#1985Closesapache#1986Closesapache#1998Closesapache#2031Closesapache#2032Closesapache#2071Closesapache#2076Closesapache#2108Closesapache#2119Closesapache#2128Closesapache#2142Closesapache#2174Closesapache#2206Closesapache#2297Closesapache#2322Closesapache#2332Closesapache#2341Closesapache#2377Closesapache#2414Closesapache#2469
d2r pushed a commit to d2r/storm that referenced this pull request Oct 16, 2018
We are closing stale Pull Requests to make the list more manageable.
Please re-open any Pull Request that has been closed in error.
Closesapache#608Closesapache#639Closesapache#640Closesapache#648Closesapache#662Closesapache#668Closesapache#692Closesapache#705Closesapache#724Closesapache#728Closesapache#730Closesapache#753Closesapache#803Closesapache#854Closesapache#922Closesapache#986Closesapache#992Closesapache#1019Closesapache#1040Closesapache#1041Closesapache#1043Closesapache#1046Closesapache#1051Closesapache#1078Closesapache#1146Closesapache#1164Closesapache#1165Closesapache#1178Closesapache#1213Closesapache#1225Closesapache#1258Closesapache#1259Closesapache#1268Closesapache#1272Closesapache#1277Closesapache#1278Closesapache#1288Closesapache#1296Closesapache#1328Closesapache#1342Closesapache#1353Closesapache#1370Closesapache#1376Closesapache#1391Closesapache#1395Closesapache#1399Closesapache#1406Closesapache#1410Closesapache#1422Closesapache#1427Closesapache#1443Closesapache#1462Closesapache#1468Closesapache#1483Closesapache#1506Closesapache#1509Closesapache#1515Closesapache#1520Closesapache#1521Closesapache#1525Closesapache#1527Closesapache#1544Closesapache#1550Closesapache#1566Closesapache#1569Closesapache#1570Closesapache#1575Closesapache#1580Closesapache#1584Closesapache#1591Closesapache#1600Closesapache#1611Closesapache#1613Closesapache#1639Closesapache#1703Closesapache#1711Closesapache#1719Closesapache#1737Closesapache#1760Closesapache#1767Closesapache#1768Closesapache#1785Closesapache#1799Closesapache#1822Closesapache#1824Closesapache#1844Closesapache#1874Closesapache#1918Closesapache#1928Closesapache#1937Closesapache#1942Closesapache#1951Closesapache#1957Closesapache#1963Closesapache#1964Closesapache#1965Closesapache#1967Closesapache#1968Closesapache#1971Closesapache#1985Closesapache#1986Closesapache#1998Closesapache#2031Closesapache#2032Closesapache#2071Closesapache#2076Closesapache#2108Closesapache#2119Closesapache#2128Closesapache#2142Closesapache#2174Closesapache#2206Closesapache#2297Closesapache#2322Closesapache#2332Closesapache#2341Closesapache#2377Closesapache#2414Closesapache#2469
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@sakanaou