Skip to content

Create ZippedTextFileReader.java - #1928

Closed
megha10 wants to merge 5 commits into
apache:masterfrom
megha10:patch-1
Closed

Create ZippedTextFileReader.java#1928
megha10 wants to merge 5 commits into
apache:masterfrom
megha10:patch-1

Conversation

@megha10

@megha10megha10 commented Feb 7, 2017

Copy link
Copy Markdown

This is to read from a zip file

@megha10
Create ZippedTextFileReader.java
01d7b28
This is to read from a zip file
@megha10megha10 closed this Feb 7, 2017
@megha10megha10 reopened this Feb 7, 2017
@tibkiss

tibkiss commented Feb 12, 2017

Copy link
Copy Markdown
Contributor

Thanks for contributing, nice idea!

Seems like the build has failed due to missing symbols in ZippedTextFileReader.java:
https://travis-ci.org/apache/storm/jobs/199168284

Unit (and possibly integration) tests would be really beneficial so that we can ensure quality in a long run.

It is a protocol that first a JIRA issue is created for the feature/fix, then a PR is posted including the JIRA Issue ID in the commit message.

Could you please address these issues?

pasalkarsachin1
pasalkarsachin1 suggested changes Feb 13, 2017
View reviewed changes

@pasalkarsachin1pasalkarsachin1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also don't see the implementation incase you want to read it from where it left using offset. Can you add that too?

Comment threadexternal/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/ZippedTextFileReader.java Outdated
zipFilename = path.getName();

LOG.info("zipFilename: ============:" + zipFilename);
System.out.println("zipFilename: ============:" + zipFilename);

@pasalkarsachin1pasalkarsachin1Feb 13, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove syso

roshannaik
roshannaik requested changes Feb 15, 2017
View reviewed changes

@roshannaikroshannaik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to add a short alias for this reader, like say 'zip' .. in org.apache.storm.hdfs.spout.Configs class

and update checkValidReader() and createFileReader() methods in HdfsSpout accordingly

roshannaik
roshannaik requested changes Feb 15, 2017
View reviewed changes

@roshannaikroshannaik left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs unit tests covering these cases:

  • Basic reading of a zip file to completion
  • Read a file partially with one reader instance, commit progress, abandon the first reader, then resume reading the remainder of same file with another reader.

@megha10

megha10 commented Feb 15, 2017

Copy link
Copy Markdown
Author

Sure will do that :)

d2r pushed a commit to d2r/storm that referenced this pull request Oct 16, 2018
Derek Dagit
Closes Pull Requests unchanged in 2018
fcda81c
We are closing stale Pull Requests to make the list more manageable.
Please re-open any Pull Request that has been closed in error.
Closesapache#608Closesapache#639Closesapache#640Closesapache#648Closesapache#662Closesapache#668Closesapache#692Closesapache#705Closesapache#724Closesapache#728Closesapache#730Closesapache#753Closesapache#803Closesapache#854Closesapache#922Closesapache#986Closesapache#992Closesapache#1019Closesapache#1040Closesapache#1041Closesapache#1043Closesapache#1046Closesapache#1051Closesapache#1078Closesapache#1146Closesapache#1164Closesapache#1165Closesapache#1178Closesapache#1213Closesapache#1225Closesapache#1258Closesapache#1259Closesapache#1268Closesapache#1272Closesapache#1277Closesapache#1278Closesapache#1288Closesapache#1296Closesapache#1328Closesapache#1342Closesapache#1353Closesapache#1370Closesapache#1376Closesapache#1391Closesapache#1395Closesapache#1399Closesapache#1406Closesapache#1410Closesapache#1422Closesapache#1427Closesapache#1443Closesapache#1462Closesapache#1468Closesapache#1483Closesapache#1506Closesapache#1509Closesapache#1515Closesapache#1520Closesapache#1521Closesapache#1525Closesapache#1527Closesapache#1544Closesapache#1550Closesapache#1566Closesapache#1569Closesapache#1570Closesapache#1575Closesapache#1580Closesapache#1584Closesapache#1591Closesapache#1600Closesapache#1611Closesapache#1613Closesapache#1639Closesapache#1703Closesapache#1711Closesapache#1719Closesapache#1737Closesapache#1760Closesapache#1767Closesapache#1768Closesapache#1785Closesapache#1799Closesapache#1822Closesapache#1824Closesapache#1844Closesapache#1874Closesapache#1918Closesapache#1928Closesapache#1937Closesapache#1942Closesapache#1951Closesapache#1957Closesapache#1963Closesapache#1964Closesapache#1965Closesapache#1967Closesapache#1968Closesapache#1971Closesapache#1985Closesapache#1986Closesapache#1998Closesapache#2031Closesapache#2032Closesapache#2071Closesapache#2076Closesapache#2108Closesapache#2119Closesapache#2128Closesapache#2142Closesapache#2174Closesapache#2206Closesapache#2297Closesapache#2322Closesapache#2332Closesapache#2341Closesapache#2377Closesapache#2414Closesapache#2469
d2r pushed a commit to d2r/storm that referenced this pull request Oct 16, 2018
Derek Dagit
Closes Pull Requests unchanged in 2018
fdb95d0
We are closing stale Pull Requests to make the list more manageable.
Please re-open any Pull Request that has been closed in error.
Closesapache#608Closesapache#639Closesapache#640Closesapache#648Closesapache#662Closesapache#668Closesapache#692Closesapache#705Closesapache#724Closesapache#728Closesapache#730Closesapache#753Closesapache#803Closesapache#854Closesapache#922Closesapache#986Closesapache#992Closesapache#1019Closesapache#1040Closesapache#1041Closesapache#1043Closesapache#1046Closesapache#1051Closesapache#1078Closesapache#1146Closesapache#1164Closesapache#1165Closesapache#1178Closesapache#1213Closesapache#1225Closesapache#1258Closesapache#1259Closesapache#1268Closesapache#1272Closesapache#1277Closesapache#1278Closesapache#1288Closesapache#1296Closesapache#1328Closesapache#1342Closesapache#1353Closesapache#1370Closesapache#1376Closesapache#1391Closesapache#1395Closesapache#1399Closesapache#1406Closesapache#1410Closesapache#1422Closesapache#1427Closesapache#1443Closesapache#1462Closesapache#1468Closesapache#1483Closesapache#1506Closesapache#1509Closesapache#1515Closesapache#1520Closesapache#1521Closesapache#1525Closesapache#1527Closesapache#1544Closesapache#1550Closesapache#1566Closesapache#1569Closesapache#1570Closesapache#1575Closesapache#1580Closesapache#1584Closesapache#1591Closesapache#1600Closesapache#1611Closesapache#1613Closesapache#1639Closesapache#1703Closesapache#1711Closesapache#1719Closesapache#1737Closesapache#1760Closesapache#1767Closesapache#1768Closesapache#1785Closesapache#1799Closesapache#1822Closesapache#1824Closesapache#1844Closesapache#1874Closesapache#1918Closesapache#1928Closesapache#1937Closesapache#1942Closesapache#1951Closesapache#1957Closesapache#1963Closesapache#1964Closesapache#1965Closesapache#1967Closesapache#1968Closesapache#1971Closesapache#1985Closesapache#1986Closesapache#1998Closesapache#2031Closesapache#2032Closesapache#2071Closesapache#2076Closesapache#2108Closesapache#2119Closesapache#2128Closesapache#2142Closesapache#2174Closesapache#2206Closesapache#2297Closesapache#2322Closesapache#2332Closesapache#2341Closesapache#2377Closesapache#2414Closesapache#2469
@d2rd2r mentioned this pull request Oct 16, 2018
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@roshannaikroshannaikroshannaik requested changes
+1 more reviewer
@pasalkarsachin1pasalkarsachin1pasalkarsachin1 requested changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@megha10@tibkiss@roshannaik@pasalkarsachin1