Skip to content

STORM-2355: Storm-HDFS: inotify support - #1937

Closed
tibkiss wants to merge 3 commits into
apache:1.x-branchfrom
tibkiss:feature/inotify_in_hdfsspout-1.x-branch
Closed

STORM-2355: Storm-HDFS: inotify support#1937
tibkiss wants to merge 3 commits into
apache:1.x-branchfrom
tibkiss:feature/inotify_in_hdfsspout-1.x-branch

Conversation

@tibkiss

@tibkisstibkiss commented Feb 12, 2017

Copy link
Copy Markdown
Contributor

No description provided.

@tibkiss
STORM-2355: Storm-HDFS: inotify support (initial version)
dc9aec2
Change-Id: I5e8a81738ff59afb8073a75c604bde2c1ef5b8d1
@tibkiss
Handle leftover files in HdfsSpout's input directory
3a6ef55
Change-Id: I0e2a8b0a15c090629cc5b220bb5891cdcabd7659
harshach
harshach reviewed Feb 25, 2017
View reviewed changes
Comment threadexternal/storm-hdfs/src/main/java/org/apache/storm/hdfs/common/HdfsDirectoryMonitor.java Outdated
import org.slf4j.LoggerFactory;

import java.io.IOException;
import java.util.*;

@harshachharshachFeb 25, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avoid wild-card imports.

harshach
harshach reviewed Feb 25, 2017
View reviewed changes
Comment threadexternal/storm-hdfs/src/test/java/org/apache/storm/hdfs/spout/TestHdfsSpout.java Outdated
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.*;

@harshachharshachFeb 25, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here.

harshach
harshach reviewed Feb 25, 2017
View reviewed changes
Comment threadexternal/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java Outdated
return null;
}
for (Path file : hdfsDirectoryMonitor) {
if (Path.getPathWithoutSchemeAndAuthority(file).toString().startsWith(lockDirPath.toString())) {

@harshachharshachFeb 25, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can merge these two if blocks into single if block.

harshach
harshach reviewed Feb 25, 2017
View reviewed changes
Comment threadpom.xml
<disruptor.version>3.3.2</disruptor.version>
<jgrapht.version>0.9.0</jgrapht.version>
<guava.version>16.0.1</guava.version>
<guava.version>14.0.1</guava.version>

@harshachharshachFeb 25, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason to revert to a older version of guava

@tibkisstibkissFeb 25, 2017•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hdfs dependency hell story: hadoop depends on 14.0.1 guava. If guava 16 is used & inotify calls are made the following exception occurs:

org.apache.hadoop.ipc.RemoteException:
com.google.common.collect.ComparisonChain.compare(ZZ)Lcom/google/common/collect/ComparisonChain;
at org.apache.hadoop.hdfs.server.namenode.JournalSet$1.compare(JournalSet.java:67)
at org.apache.hadoop.hdfs.server.namenode.JournalSet$1.compare(JournalSet.java:62)
at java.util.TimSort.countRunAndMakeAscending(TimSort.java:324)
at java.util.TimSort.sort(TimSort.java:189)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.chainAndMakeRedundantStreams(JournalSet.java:333)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:284)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1508)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1532)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1997)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1496)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)

 at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy37.getEditsFromTxid(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolTranslatorPB.java:1511)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy38.getEditsFromTxid(Unknown Source)
at org.apache.hadoop.hdfs.DFSInotifyEventInputStream.poll(DFSInotifyEventInputStream.java:111)
at org.apache.storm.hdfs.common.HdfsUtils.listNewFiles(HdfsUtils.java:94)
at org.apache.storm.hdfs.spout.TestHdfsSpout.testListNewFiles(TestHdfsSpout.java:549)

@tibkiss

tibkiss commented Feb 25, 2017

Copy link
Copy Markdown
ContributorAuthor

Thanks for the comments. I'll be OoO next week, will address them beginning of March.

@roshannaik

roshannaik commented Feb 27, 2017

Copy link
Copy Markdown
Contributor

@tibkiss on the JIRA I had a few questions to get clarity around this.
That would help get a perspective to evaluate this. Could you please respond to them ?

@tibkiss

tibkiss commented Mar 6, 2017•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@roshannaik : I've replied to your questions on the JIRA by extending the issue description. Please check it out and let me know if you have further questions. Thanks!

@tibkiss
Addressing reviewer's comments about coding style
b1a13f4
Change-Id: I932f16f267459b2447ca611d10304cbea7d879d4
@tibkiss

tibkiss commented Mar 6, 2017

Copy link
Copy Markdown
ContributorAuthor

@harshach : Updated this PR based on your review.

@tibkiss

tibkiss commented Mar 8, 2017

Copy link
Copy Markdown
ContributorAuthor

bump

@roshannaik

roshannaik commented Mar 14, 2017

Copy link
Copy Markdown
Contributor

@tibkiss thanks for adding the details on the jira. that is useful. i have started looking into it.

d2r pushed a commit to d2r/storm that referenced this pull request Oct 16, 2018
Derek Dagit
Closes Pull Requests unchanged in 2018
fcda81c
We are closing stale Pull Requests to make the list more manageable.
Please re-open any Pull Request that has been closed in error.
Closesapache#608Closesapache#639Closesapache#640Closesapache#648Closesapache#662Closesapache#668Closesapache#692Closesapache#705Closesapache#724Closesapache#728Closesapache#730Closesapache#753Closesapache#803Closesapache#854Closesapache#922Closesapache#986Closesapache#992Closesapache#1019Closesapache#1040Closesapache#1041Closesapache#1043Closesapache#1046Closesapache#1051Closesapache#1078Closesapache#1146Closesapache#1164Closesapache#1165Closesapache#1178Closesapache#1213Closesapache#1225Closesapache#1258Closesapache#1259Closesapache#1268Closesapache#1272Closesapache#1277Closesapache#1278Closesapache#1288Closesapache#1296Closesapache#1328Closesapache#1342Closesapache#1353Closesapache#1370Closesapache#1376Closesapache#1391Closesapache#1395Closesapache#1399Closesapache#1406Closesapache#1410Closesapache#1422Closesapache#1427Closesapache#1443Closesapache#1462Closesapache#1468Closesapache#1483Closesapache#1506Closesapache#1509Closesapache#1515Closesapache#1520Closesapache#1521Closesapache#1525Closesapache#1527Closesapache#1544Closesapache#1550Closesapache#1566Closesapache#1569Closesapache#1570Closesapache#1575Closesapache#1580Closesapache#1584Closesapache#1591Closesapache#1600Closesapache#1611Closesapache#1613Closesapache#1639Closesapache#1703Closesapache#1711Closesapache#1719Closesapache#1737Closesapache#1760Closesapache#1767Closesapache#1768Closesapache#1785Closesapache#1799Closesapache#1822Closesapache#1824Closesapache#1844Closesapache#1874Closesapache#1918Closesapache#1928Closesapache#1937Closesapache#1942Closesapache#1951Closesapache#1957Closesapache#1963Closesapache#1964Closesapache#1965Closesapache#1967Closesapache#1968Closesapache#1971Closesapache#1985Closesapache#1986Closesapache#1998Closesapache#2031Closesapache#2032Closesapache#2071Closesapache#2076Closesapache#2108Closesapache#2119Closesapache#2128Closesapache#2142Closesapache#2174Closesapache#2206Closesapache#2297Closesapache#2322Closesapache#2332Closesapache#2341Closesapache#2377Closesapache#2414Closesapache#2469
d2r pushed a commit to d2r/storm that referenced this pull request Oct 16, 2018
Derek Dagit
Closes Pull Requests unchanged in 2018
fdb95d0
We are closing stale Pull Requests to make the list more manageable.
Please re-open any Pull Request that has been closed in error.
Closesapache#608Closesapache#639Closesapache#640Closesapache#648Closesapache#662Closesapache#668Closesapache#692Closesapache#705Closesapache#724Closesapache#728Closesapache#730Closesapache#753Closesapache#803Closesapache#854Closesapache#922Closesapache#986Closesapache#992Closesapache#1019Closesapache#1040Closesapache#1041Closesapache#1043Closesapache#1046Closesapache#1051Closesapache#1078Closesapache#1146Closesapache#1164Closesapache#1165Closesapache#1178Closesapache#1213Closesapache#1225Closesapache#1258Closesapache#1259Closesapache#1268Closesapache#1272Closesapache#1277Closesapache#1278Closesapache#1288Closesapache#1296Closesapache#1328Closesapache#1342Closesapache#1353Closesapache#1370Closesapache#1376Closesapache#1391Closesapache#1395Closesapache#1399Closesapache#1406Closesapache#1410Closesapache#1422Closesapache#1427Closesapache#1443Closesapache#1462Closesapache#1468Closesapache#1483Closesapache#1506Closesapache#1509Closesapache#1515Closesapache#1520Closesapache#1521Closesapache#1525Closesapache#1527Closesapache#1544Closesapache#1550Closesapache#1566Closesapache#1569Closesapache#1570Closesapache#1575Closesapache#1580Closesapache#1584Closesapache#1591Closesapache#1600Closesapache#1611Closesapache#1613Closesapache#1639Closesapache#1703Closesapache#1711Closesapache#1719Closesapache#1737Closesapache#1760Closesapache#1767Closesapache#1768Closesapache#1785Closesapache#1799Closesapache#1822Closesapache#1824Closesapache#1844Closesapache#1874Closesapache#1918Closesapache#1928Closesapache#1937Closesapache#1942Closesapache#1951Closesapache#1957Closesapache#1963Closesapache#1964Closesapache#1965Closesapache#1967Closesapache#1968Closesapache#1971Closesapache#1985Closesapache#1986Closesapache#1998Closesapache#2031Closesapache#2032Closesapache#2071Closesapache#2076Closesapache#2108Closesapache#2119Closesapache#2128Closesapache#2142Closesapache#2174Closesapache#2206Closesapache#2297Closesapache#2322Closesapache#2332Closesapache#2341Closesapache#2377Closesapache#2414Closesapache#2469
@d2rd2r mentioned this pull request Oct 16, 2018
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@harshachharshachharshach left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tibkiss@roshannaik@harshach