diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4133561f06..dfbd3e876b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -27,4 +27,4 @@ Here are a few tips to get your contribution in: 4. Make sure that your code passes the unit tests. You can run the tests with `mvn test` in the root directory. 5. Add new unit tests for your code. -If you’d like to report a bug but don’t have time to fix it, you can still post it to our [issue tracker](https://issues.apache.org/jira/browse/PARQUET), or email the mailing list (dev@parquet.incubator.apache.org). +If you’d like to report a bug but don’t have time to fix it, you can still post it to our [issue tracker](https://issues.apache.org/jira/browse/PARQUET), or email the mailing list (dev@parquet.apache.org). diff --git a/NOTICE b/NOTICE index c6e3bf2525..a0c2011915 100644 --- a/NOTICE +++ b/NOTICE @@ -1,5 +1,5 @@ -Apache Parquet MR (Incubating) +Apache Parquet MR Copyright 2014 The Apache Software Foundation This product includes software developed at diff --git a/dev/README.md b/dev/README.md index 4a4b18e446..25dd33692c 100644 --- a/dev/README.md +++ b/dev/README.md @@ -26,8 +26,8 @@ Merging a pull request requires being a committer on the project. * How to merge a Pull request: have an apache and apache-github remote setup ``` -git remote add apache-github git@github.com:apache/incubator-parquet-mr.git -git remote add apache https://git-wip-us.apache.org/repos/asf/incubator-parquet-mr.git +git remote add apache-github git@github.com:apache/parquet-mr.git +git remote add apache https://git-wip-us.apache.org/repos/asf/parquet-mr.git ``` run the following command ``` @@ -42,19 +42,19 @@ example output: ``` Which pull request would you like to merge? (e.g. 34): ``` -Type the pull request number (from https://github.com/apache/incubator-parquet-mr/pulls) and hit enter. +Type the pull request number (from https://github.com/apache/parquet-mr/pulls) and hit enter. ``` === Pull Request #X === title Blah Blah Blah source repo/branch target master -url https://api.github.com/repos/apache/incubator-parquet-mr/pulls/X +url https://api.github.com/repos/apache/parquet-mr/pulls/X Proceed with merging pull request #3? (y/n): ``` If this looks good, type y and hit enter. ``` -From git-wip-us.apache.org:/repos/asf/incubator-parquet-mr.git +From git-wip-us.apache.org:/repos/asf/parquet-mr.git * [new branch] master -> PR_TOOL_MERGE_PR_3_MASTER Switched to branch 'PR_TOOL_MERGE_PR_3_MASTER' @@ -68,7 +68,7 @@ Delta compression using up to 4 threads. Compressing objects: 100% (26/26), done. Writing objects: 100% (36/36), 5.32 KiB, done. Total 36 (delta 17), reused 0 (delta 0) -To git-wip-us.apache.org:/repos/asf/incubator-parquet-mr.git +To git-wip-us.apache.org:/repos/asf/parquet-mr.git b767ac4..485658a PR_TOOL_MERGE_PR_X_MASTER -> master Restoring head pointer to b767ac4e Note: checking out 'b767ac4e'. diff --git a/dev/source-release.sh b/dev/source-release.sh index a61a6f3237..a8f5b75b25 100644 --- a/dev/source-release.sh +++ b/dev/source-release.sh @@ -28,7 +28,7 @@ if [ -z "$2" ]; then exit fi -version=$1-incubating +version=$1 rc=$2 if [ -d tmp/ ]; then @@ -68,7 +68,7 @@ svn co --depth=empty https://dist.apache.org/repos/dist/dev/incubator/parquet tm mkdir -p tmp/$tagrc cp ${tarball}* tmp/$tagrc svn add tmp/$tagrc -echo "svn ci -m 'Apache Parquet MR (Incubating) $version RC${rc}' tmp/$tagrc" +echo "svn ci -m 'Apache Parquet MR $version RC${rc}' tmp/$tagrc" # clean up rm -rf tmp diff --git a/parquet-avro/pom.xml b/parquet-avro/pom.xml index fea929dc11..6a5288bf9b 100644 --- a/parquet-avro/pom.xml +++ b/parquet-avro/pom.xml @@ -29,8 +29,8 @@ parquet-avro jar - Apache Parquet Avro (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Avro + https://parquet.apache.org 1.7.6 diff --git a/parquet-cascading/pom.xml b/parquet-cascading/pom.xml index 4141b8f81e..103c41147b 100644 --- a/parquet-cascading/pom.xml +++ b/parquet-cascading/pom.xml @@ -29,8 +29,8 @@ parquet-cascading jar - Apache Parquet Cascading (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Cascading + https://parquet.apache.org diff --git a/parquet-column/pom.xml b/parquet-column/pom.xml index 5297d09fd3..b9dea059e8 100644 --- a/parquet-column/pom.xml +++ b/parquet-column/pom.xml @@ -29,8 +29,8 @@ parquet-column jar - Apache Parquet Column (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Column + https://parquet.apache.org diff --git a/parquet-common/pom.xml b/parquet-common/pom.xml index 065e3769d4..613138d6be 100644 --- a/parquet-common/pom.xml +++ b/parquet-common/pom.xml @@ -29,8 +29,8 @@ parquet-common jar - Apache Parquet Common (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Common + https://parquet.apache.org diff --git a/parquet-encoding/pom.xml b/parquet-encoding/pom.xml index 7cd6aecd0f..8950e3825e 100644 --- a/parquet-encoding/pom.xml +++ b/parquet-encoding/pom.xml @@ -29,8 +29,8 @@ parquet-encoding jar - Apache Parquet Encodings (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Encodings + https://parquet.apache.org diff --git a/parquet-generator/pom.xml b/parquet-generator/pom.xml index eff0191e47..c5004a4fa7 100644 --- a/parquet-generator/pom.xml +++ b/parquet-generator/pom.xml @@ -29,8 +29,8 @@ parquet-generator jar - Apache Parquet Generator (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Generator + https://parquet.apache.org diff --git a/parquet-hadoop-bundle/pom.xml b/parquet-hadoop-bundle/pom.xml index b95374e57d..c22aa852e5 100644 --- a/parquet-hadoop-bundle/pom.xml +++ b/parquet-hadoop-bundle/pom.xml @@ -29,8 +29,8 @@ parquet-hadoop-bundle jar - Apache Parquet Hadoop Bundle (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Hadoop Bundle + https://parquet.apache.org diff --git a/parquet-hadoop/pom.xml b/parquet-hadoop/pom.xml index e4ea052934..9a11af5f32 100644 --- a/parquet-hadoop/pom.xml +++ b/parquet-hadoop/pom.xml @@ -29,8 +29,8 @@ parquet-hadoop jar - Apache Parquet Hadoop (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Hadoop + https://parquet.apache.org diff --git a/parquet-hive-bundle/pom.xml b/parquet-hive-bundle/pom.xml index 6e889ea64b..d1a46a1be4 100644 --- a/parquet-hive-bundle/pom.xml +++ b/parquet-hive-bundle/pom.xml @@ -29,8 +29,8 @@ parquet-hive-bundle jar - Apache Parquet Hive Bundle (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Hive Bundle + https://parquet.apache.org diff --git a/parquet-hive/parquet-hive-binding/parquet-hive-0.10-binding/pom.xml b/parquet-hive/parquet-hive-binding/parquet-hive-0.10-binding/pom.xml index 3732b2d4ae..dd85f21361 100644 --- a/parquet-hive/parquet-hive-binding/parquet-hive-0.10-binding/pom.xml +++ b/parquet-hive/parquet-hive-binding/parquet-hive-0.10-binding/pom.xml @@ -27,7 +27,7 @@ 4.0.0 parquet-hive-0.10-binding - Apache Parquet Hive 0.10 Binding (Incubating) + Apache Parquet Hive 0.10 Binding jar diff --git a/parquet-hive/parquet-hive-binding/parquet-hive-0.12-binding/pom.xml b/parquet-hive/parquet-hive-binding/parquet-hive-0.12-binding/pom.xml index ae2d7b3029..d41730b34f 100644 --- a/parquet-hive/parquet-hive-binding/parquet-hive-0.12-binding/pom.xml +++ b/parquet-hive/parquet-hive-binding/parquet-hive-0.12-binding/pom.xml @@ -27,7 +27,7 @@ 4.0.0 parquet-hive-0.12-binding - Apache Parquet Hive 0.12 Binding (Incubating) + Apache Parquet Hive 0.12 Binding jar diff --git a/parquet-hive/parquet-hive-binding/parquet-hive-binding-bundle/pom.xml b/parquet-hive/parquet-hive-binding/parquet-hive-binding-bundle/pom.xml index 27d6842532..fef13cf8dd 100644 --- a/parquet-hive/parquet-hive-binding/parquet-hive-binding-bundle/pom.xml +++ b/parquet-hive/parquet-hive-binding/parquet-hive-binding-bundle/pom.xml @@ -29,7 +29,7 @@ 4.0.0 parquet-hive-binding-bundle - Apache Parquet Hive Binding Bundle (Incubating) + Apache Parquet Hive Binding Bundle jar diff --git a/parquet-hive/parquet-hive-binding/parquet-hive-binding-factory/pom.xml b/parquet-hive/parquet-hive-binding/parquet-hive-binding-factory/pom.xml index cae2f8e1da..68351ebf45 100644 --- a/parquet-hive/parquet-hive-binding/parquet-hive-binding-factory/pom.xml +++ b/parquet-hive/parquet-hive-binding/parquet-hive-binding-factory/pom.xml @@ -27,7 +27,7 @@ 4.0.0 parquet-hive-binding-factory - Apache Parquet Hive Binding Factory (Incubating) + Apache Parquet Hive Binding Factory jar diff --git a/parquet-hive/parquet-hive-binding/parquet-hive-binding-interface/pom.xml b/parquet-hive/parquet-hive-binding/parquet-hive-binding-interface/pom.xml index 7120319b2c..580be1c618 100644 --- a/parquet-hive/parquet-hive-binding/parquet-hive-binding-interface/pom.xml +++ b/parquet-hive/parquet-hive-binding/parquet-hive-binding-interface/pom.xml @@ -27,7 +27,7 @@ 4.0.0 parquet-hive-binding-interface - Apache Parquet Hive Binding Interface (Incubating) + Apache Parquet Hive Binding Interface jar diff --git a/parquet-hive/parquet-hive-binding/pom.xml b/parquet-hive/parquet-hive-binding/pom.xml index 39f6845c6c..eaad2cfbf1 100644 --- a/parquet-hive/parquet-hive-binding/pom.xml +++ b/parquet-hive/parquet-hive-binding/pom.xml @@ -27,7 +27,7 @@ 4.0.0 parquet-hive-binding - Apache Parquet Hive Binding Parent (Incubating) + Apache Parquet Hive Binding Parent pom diff --git a/parquet-hive/pom.xml b/parquet-hive/pom.xml index 586496b656..f9e00a7953 100644 --- a/parquet-hive/pom.xml +++ b/parquet-hive/pom.xml @@ -29,8 +29,8 @@ parquet-hive pom - Apache Parquet Hive (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Hive + https://parquet.apache.org diff --git a/parquet-jackson/pom.xml b/parquet-jackson/pom.xml index f4825a7f56..beaf211592 100644 --- a/parquet-jackson/pom.xml +++ b/parquet-jackson/pom.xml @@ -29,8 +29,8 @@ parquet-jackson jar - Apache Parquet Jackson (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Jackson + https://parquet.apache.org diff --git a/parquet-pig-bundle/pom.xml b/parquet-pig-bundle/pom.xml index aa8e66716a..942c3a13d2 100644 --- a/parquet-pig-bundle/pom.xml +++ b/parquet-pig-bundle/pom.xml @@ -29,8 +29,8 @@ parquet-pig-bundle jar - Apache Parquet Pig Bundle (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Pig Bundle + https://parquet.apache.org diff --git a/parquet-pig/pom.xml b/parquet-pig/pom.xml index 102d604ca8..10419dd096 100644 --- a/parquet-pig/pom.xml +++ b/parquet-pig/pom.xml @@ -29,8 +29,8 @@ parquet-pig jar - Apache Parquet Pig (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Pig + https://parquet.apache.org diff --git a/parquet-protobuf/pom.xml b/parquet-protobuf/pom.xml index 309a52d732..6cd5ecb5d8 100644 --- a/parquet-protobuf/pom.xml +++ b/parquet-protobuf/pom.xml @@ -35,8 +35,8 @@ - Apache Parquet Protobuf (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Protobuf + https://parquet.apache.org diff --git a/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoReadSupport.java b/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoReadSupport.java index bc7f4d2990..e6921dbce1 100644 --- a/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoReadSupport.java +++ b/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoReadSupport.java @@ -46,6 +46,16 @@ public static void setRequestedProjection(Configuration configuration, String re configuration.set(PB_REQUESTED_PROJECTION, requestedProjection); } + /** + * Set name of protobuf class to be used for reading data. + * If no class is set, value from file header is used. + * Note that the value in header is present only if the file was written + * using parquet-protobuf project, it will fail otherwise. + * */ + public static void setProtobufClass(Configuration configuration, String protobufClass) { + configuration.set(PB_CLASS, protobufClass); + } + @Override public ReadContext init(InitContext context) { String requestedProjectionString = context.getConfiguration().get(PB_REQUESTED_PROJECTION); @@ -63,16 +73,22 @@ public ReadContext init(InitContext context) { @Override public RecordMaterializer prepareForRead(Configuration configuration, Map keyValueMetaData, MessageType fileSchema, ReadContext readContext) { - String strProtoClass = keyValueMetaData.get(PB_CLASS); + String headerProtoClass = keyValueMetaData.get(PB_CLASS); + String configuredProtoClass = configuration.get(PB_CLASS); + + if (configuredProtoClass != null) { + LOG.debug("Replacing class " + headerProtoClass + " by " + configuredProtoClass); + headerProtoClass = configuredProtoClass; + } - if (strProtoClass == null) { + if (headerProtoClass == null) { throw new RuntimeException("I Need parameter " + PB_CLASS + " with Protocol Buffer class"); } - LOG.debug("Reading data with Protocol Buffer class" + strProtoClass); + LOG.debug("Reading data with Protocol Buffer class " + headerProtoClass); MessageType requestedSchema = readContext.getRequestedSchema(); - Class protobufClass = Protobufs.getProtobufClass(strProtoClass); + Class protobufClass = Protobufs.getProtobufClass(headerProtoClass); return new ProtoRecordMaterializer(requestedSchema, protobufClass); } diff --git a/parquet-protobuf/src/main/resources/META-INF/NOTICE b/parquet-protobuf/src/main/resources/META-INF/NOTICE index 215e2c48de..73f887c3ac 100644 --- a/parquet-protobuf/src/main/resources/META-INF/NOTICE +++ b/parquet-protobuf/src/main/resources/META-INF/NOTICE @@ -1,5 +1,5 @@ -Apache Parquet MR (Incubating) +Apache Parquet MR Copyright 2014 The Apache Software Foundation This product includes software developed at diff --git a/parquet-protobuf/src/test/java/org/apache/parquet/proto/ProtoInputOutputFormatTest.java b/parquet-protobuf/src/test/java/org/apache/parquet/proto/ProtoInputOutputFormatTest.java index 51927f25c7..5c6ebcaf2b 100644 --- a/parquet-protobuf/src/test/java/org/apache/parquet/proto/ProtoInputOutputFormatTest.java +++ b/parquet-protobuf/src/test/java/org/apache/parquet/proto/ProtoInputOutputFormatTest.java @@ -19,24 +19,21 @@ package org.apache.parquet.proto; import com.google.protobuf.Message; -import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; -import org.junit.Test; -import org.apache.parquet.Log; +import org.apache.parquet.proto.test.TestProtobuf; +import org.apache.parquet.proto.test.TestProtobuf.FirstCustomClassMessage; +import org.apache.parquet.proto.test.TestProtobuf.SecondCustomClassMessage; import org.apache.parquet.proto.utils.ReadUsingMR; import org.apache.parquet.proto.utils.WriteUsingMR; -import org.apache.parquet.proto.test.TestProtobuf; +import org.junit.Test; import java.util.List; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; +import static org.junit.Assert.*; public class ProtoInputOutputFormatTest { - private static final Log LOG = Log.getLog(ProtoInputOutputFormatTest.class); - /** * Writes Protocol Buffer using first MR job, reads written file using * second job and compares input and output. @@ -95,6 +92,34 @@ public void testProjection() throws Exception { assertTrue("Found data outside projection.", readDocument.getNameCount() == 0); } + /** + * When user specified protobuffer class in configuration, + * It should replace class specified in header. + * */ + @Test + public void testCustomProtoClass() throws Exception { + FirstCustomClassMessage.Builder inputMessage; + inputMessage = FirstCustomClassMessage.newBuilder(); + inputMessage.setString("writtenString"); + + Path outputPath = new WriteUsingMR().write(new Message[]{inputMessage.build()}); + ReadUsingMR readUsingMR = new ReadUsingMR(); + String customClass = SecondCustomClassMessage.class.getName(); + ProtoReadSupport.setProtobufClass(readUsingMR.getConfiguration(), customClass); + List result = readUsingMR.read(outputPath); + + assertEquals(1, result.size()); + Message msg = result.get(0); + assertFalse("Class from header returned.", + msg instanceof FirstCustomClassMessage); + assertTrue("Custom class was not used", + msg instanceof SecondCustomClassMessage); + + String stringValue; + stringValue = ((SecondCustomClassMessage) msg).getString(); + assertEquals("writtenString", stringValue); + } + /** * Runs job that writes input to file and then job reading data back. */ diff --git a/parquet-protobuf/src/test/resources/TestProtobuf.proto b/parquet-protobuf/src/test/resources/TestProtobuf.proto index d0e8845393..afa0f63833 100644 --- a/parquet-protobuf/src/test/resources/TestProtobuf.proto +++ b/parquet-protobuf/src/test/resources/TestProtobuf.proto @@ -126,4 +126,14 @@ message HighIndexMessage { repeated int32 repeatedInt = 50000; } +//custom proto class - ProtoInputOutputFormatTest + +message FirstCustomClassMessage { + optional string string = 11; +} + +message SecondCustomClassMessage { + optional string string = 11; +} + //please place your unit test Protocol Buffer definitions here. diff --git a/parquet-scala/pom.xml b/parquet-scala/pom.xml index ab1872670c..fe423c5b26 100644 --- a/parquet-scala/pom.xml +++ b/parquet-scala/pom.xml @@ -29,8 +29,8 @@ parquet-scala_${scala.binary.version} jar - Apache Parquet Scala (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Scala + https://parquet.apache.org diff --git a/parquet-scrooge/pom.xml b/parquet-scrooge/pom.xml index 42d2656c38..ea567c8d61 100644 --- a/parquet-scrooge/pom.xml +++ b/parquet-scrooge/pom.xml @@ -29,8 +29,8 @@ parquet-scrooge_${scala.binary.version} jar - Apache Parquet Scrooge (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Scrooge + https://parquet.apache.org diff --git a/parquet-thrift/pom.xml b/parquet-thrift/pom.xml index c19d2a186b..358124eca3 100644 --- a/parquet-thrift/pom.xml +++ b/parquet-thrift/pom.xml @@ -29,8 +29,8 @@ parquet-thrift jar - Apache Parquet Thrift (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Thrift + https://parquet.apache.org 4.4 diff --git a/parquet-tools/pom.xml b/parquet-tools/pom.xml index 5811b70298..5a036c4a41 100644 --- a/parquet-tools/pom.xml +++ b/parquet-tools/pom.xml @@ -29,8 +29,8 @@ parquet-tools jar - Apache Parquet Tools (Incubating) - https://parquet.incubator.apache.org + Apache Parquet Tools + https://parquet.apache.org provided diff --git a/parquet-tools/src/main/resources/META-INF/NOTICE b/parquet-tools/src/main/resources/META-INF/NOTICE index 2350449762..7066d54f51 100644 --- a/parquet-tools/src/main/resources/META-INF/NOTICE +++ b/parquet-tools/src/main/resources/META-INF/NOTICE @@ -1,5 +1,5 @@ -Apache Parquet MR (Incubating) +Apache Parquet MR Copyright 2014 The Apache Software Foundation This product includes software developed at diff --git a/pom.xml b/pom.xml index 880191d799..51e2e7f63c 100644 --- a/pom.xml +++ b/pom.xml @@ -12,14 +12,14 @@ 1.8.0-SNAPSHOT pom - Apache Parquet MR (Incubating) - https://parquet.incubator.apache.org + Apache Parquet MR + https://parquet.apache.org Parquet is a columnar storage format that supports nested data. This provides the java implementation. - scm:git:git@github.com:apache/incubator-parquet-mr.git - scm:git:git@github.com:apache/incubator-parquet-mr.git - scm:git:https://git-wip-us.apache.org/repos/asf/incubator-parquet-mr.git + scm:git:git@github.com:apache/parquet-mr.git + scm:git:git@github.com:apache/parquet-mr.git + scm:git:https://git-wip-us.apache.org/repos/asf/parquet-mr.git @@ -37,15 +37,15 @@ Dev Mailing List - dev@parquet.incubator.apache.org - dev-subscribe@parquet.incubator.apache.org - dev-unsubscribe@parquet.incubator.apache.org + dev@parquet.apache.org + dev-subscribe@parquet.apache.org + dev-unsubscribe@parquet.apache.org Commits Mailing List - commits@parquet.incubator.apache.org - commits-subscribe@parquet.incubator.apache.org - commits-unsubscribe@parquet.incubator.apache.org + commits@parquet.apache.org + commits-subscribe@parquet.apache.org + commits-unsubscribe@parquet.apache.org