Problem with Maven repo specifications in POMs of dependencies in some development environments

[asfimport]

Running mvn verify based on the instructions in the README results in this error

Could not resolve dependencies for project org.apache.parquet:parquet-thrift:jar:1.11.0: Could not find artifact com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16

As a workaround, the local ~/.m2/settings.xml file can be modified to include the twitter maven repo:

<repository>
  <id>twitter</id>
  <name>twitter</name>
  <url>http://maven.twttr.com</url>
</repository>

After adding this, mvn verify works. This should not be necessary though, since the artifact is a transitive dependency and the POM of the direct dependency (elephant-bird) contains the repo specification, which works in most environments.

Reporter: Andy Grove / @andygrove
Assignee: Andy Grove / @andygrove

Related issues:

• Build fails due to missing hadoop-lzo (duplicates)

PRs and other links:

• GitHub Pull Request #639

_{Note: This issue was originally created as PARQUET-1556. Please see the migration documentation for further details.}

[asfimport]

Nándor Kollár / @nandorKollar:
This additional repository should be in the POM instead of adding it to the settings. Looks strange for me, I couldn't reproduce the failure. My settings file doesn't have this additional Twitter repository, and I couldn't see it in the output of mvn help:effective-pom either.

[asfimport]

Zoltan Ivanfi / @zivanfi:
Now that is strange. If I issue this command:

mvn dependency:get -Dartifact=com.hadoop.gplcompression:hadoop-lzo:0.4.16

I get the same error. But it is still able to download the artifact somehow when I run the following:

mvn -Dmaven.repo.local=/tmp/fresh-clean-empty-local-repo clean install -DskipTests | grep hadoop-lzo
Downloading from jitpack.io: https://jitpack.io/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
Downloading from central: https://repo.maven.apache.org/maven2/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
Downloading from twitter: http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
Downloaded from twitter: http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom (1.3 kB at 1.3 kB/s)
Downloading from jitpack.io: https://jitpack.io/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.jar
Downloading from central: https://repo.maven.apache.org/maven2/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.jar
Downloading from twitter: http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.jar
Downloaded from twitter: http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.jar (63 kB at 86 kB/s)

[asfimport]

Andy Grove / @andygrove:
I'm also curious to understand the implications of adding a library that depends on GPL licensed codec. Is this dependency just used for tests?

Since LZO is GPL licensed it doesn't come bundled with Hadoop installation. You will have to install it separately.

[asfimport]

Zoltan Ivanfi / @zivanfi:
That's a very good point, thanks for raising it. We don't use Hadoop-LZO ourselves. Running mvn dependency:tree shows that this is a compile-time transitive dependecy:

[INFO] org.apache.parquet:parquet-thrift:jar:1.12.0-SNAPSHOT
[INFO] +- com.twitter.elephantbird:elephant-bird-core:jar:4.4:compile
[INFO] |  \- com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16:compile

Before adding it to the POM we have to understand:

• Why it can be downloaded for most people even without a corresponding repo entry.
• Why it fails for others.
• What it would mean to add the repo to the POM (would it lead to shipping a GPL dependency).
• Can we avoid pulling this in all together?

[asfimport]

Zoltan Ivanfi / @zivanfi:
I came to the conclusion that the only possible source of the twitter repo is the POM of the elephant-bird dependecy. However, I have no idea why this doesn't work for you, @andygrove. I have tried it with both Maven 3.5.2 and 3.6.0 and both are able to download the transitive dependecy. What version do you use?