Uh oh!
There was an error while loading. Please reload this page.
Simplify classpath. - #65
Conversation
bzz
commented
May 10, 2015
I tried spark-local and md interpreters both from project root (dev mode) and from packaged distribution artchive - both worked well for me. @Leemoonsoo do you know why on CI we have RAT check failure? Do not see any changes in licences in this PR... @khalidhuseynov can you please also try these changes with the latest code for thift-tajo-server? |
Leemoonsoo
commented
May 10, 2015
@bzz it's strange. I'm trying to figure out why RAT check is failing. |
bzz
commented
May 10, 2015
khalidhuseynov
commented
May 11, 2015
Yes, it solved the problem with CLASSPATH length and path duplication. Tajo thrift interpreter works fine in my local build now. Looks good to me. |
Leemoonsoo
commented
May 13, 2015
RAT check looks like flaky. I made CI to print rat.txt when it fails. so next time, we'll know what causes RAT check fail. Ready to merge! |
bzz
commented
May 13, 2015
Looks great to me! Thank you for printing RAT, will check it next time |
RPCMoritz
commented
May 13, 2015
While this was a decent effort (it actually made my classpath-related error almost readable, where previously it was three screens worth of text), there is still an issue with classpath's lack of simplicity. In partcular, (in combination with yarn) it pulls in (without any obvious documentation) many different classpaths, some of which may depend on environment variables. I'm currently (Zeppelin master, HDP 2.2w/ spark tech preview on yarn) getting "bad substitution" errors from launch_container.sh:27, an issue similar to Ambari-8028 (https://issues.apache.org/jira/browse/AMBARI-8028) and which I suspect is down to remaining references to ${hdp.version} (and potentially $PWD, $HADOOP_CONF_DIR), which I'm chasing down now. I think it might be a good idea to move the classpath over to zeppelin-site.xml in any case, since over the long term managing the classpath via shell scripts is going to be painful and not user-friendly. Especially the CLASSPATH+= makes identifying the issue I encountered very difficult. |
When Zeppelin constructs JVM CLASSPATH, it adds path of every single jar. That creates really long CLASSPATH and it may cause some problem like https://issues.apache.org/jira/browse/ZEPPELIN-68.
This PR solves it by constructing CLASSPATH with wildcard.
Another problem is, Zeppelin constructs CLASSPATH with duplicated entry or not related entry.
This PR solves it by constructing CLASSPATH not in common.sh but in each script zeppelin-daemon.sh, interpreter.sh, zeppelin.sh while they need different entry in CLASSPATH.