Uh oh!
There was an error while loading. Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork 4.8k
HIVE-29419: Provide a Hive-specific docker image for Tez AM#6435
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Uh oh!
There was an error while loading. Please reload this page.
Changes from all commits
File filter
Filter by extension
Conversations
Uh oh!
There was an error while loading. Please reload this page.
Jump to
Uh oh!
There was an error while loading. Please reload this page.
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -88,4 +88,25 @@ | ||
| <name>hive.query.results.cache.directory</name> | ||
| <value>${HIVE_QUERY_RESULTS_CACHE_DIRECTORY}</value> | ||
| </property> | ||
| <property> | ||
| <name>hive.server2.tez.initialize.default.sessions</name> | ||
| <value>false</value> | ||
| </property> | ||
| <property> | ||
| <name>hive.server2.tez.use.external.sessions</name> | ||
| <value>${HIVE_SERVER2_TEZ_USE_EXTERNAL_SESSIONS}</value> | ||
| </property> | ||
| <!-- | ||
| A registry namespace prefix is a hardcoded prefix for Tez external sessions. | ||
| The actual tez.am.registry.namespace value is appended to this prefix. | ||
| Once hive can use the registry client that Tez provides (ZkAMRegistryClient), this property will be removed. | ||
Member There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is there a ticket for that? needs Tez upgrade?
| ||
| --> | ||
| <property> | ||
| <name>hive.server2.tez.external.sessions.namespace</name> | ||
| <value>/tez-external-sessions${TEZ_AM_REGISTRY_NAMESPACE}</value> | ||
| </property> | ||
| <property> | ||
| <name>hive.server2.tez.external.sessions.registry.class</name> | ||
| <value>org.apache.hadoop.hive.ql.exec.tez.ZookeeperExternalSessionsRegistryClient</value> | ||
| </property> | ||
| </configuration> | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| { | ||
| "taskSchedulerDescriptors": [ | ||
| { | ||
| "className": "org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService", | ||
| "entityName": "LLAP" | ||
| } | ||
| ], | ||
| "containerLauncherDescriptors": [ | ||
| { | ||
| "className": "org.apache.hadoop.hive.llap.tezplugins.LlapContainerLauncher", | ||
| "entityName": "LLAP" | ||
| } | ||
| ], | ||
| "taskCommunicatorDescriptors": [ | ||
| { | ||
| "className": "org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator", | ||
| "entityName": "LLAP" | ||
| } | ||
| ], | ||
| "enableContainers": false, | ||
| "enableUber": false | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| log4j.rootLogger=INFO, console | ||
Member There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. maybe, simply | ||
| log4j.appender.console=org.apache.log4j.ConsoleAppender | ||
| log4j.appender.console.Target=System.err | ||
| log4j.appender.console.layout=org.apache.log4j.PatternLayout | ||
| log4j.appender.console.layout.ConversionPattern=%d{ISO8601} %-5p %c{1} - %m%n | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| <?xml version="1.0"?> | ||
| <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> | ||
| <!-- | ||
| Licensed to the Apache Software Foundation (ASF) under one or more | ||
| contributor license agreements. See the NOTICE file distributed with | ||
| this work for additional information regarding copyright ownership. | ||
| The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| --> | ||
| <configuration> | ||
| <property> | ||
| <name>tez.am.mode.session</name> | ||
| <value>true</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.am.framework.mode</name> | ||
| <value>${TEZ_FRAMEWORK_MODE}</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.am.zookeeper.quorum</name> | ||
| <value>${TEZ_AM_ZOOKEEPER_QUORUM}</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.am.registry.namespace</name> | ||
| <value>${TEZ_AM_REGISTRY_NAMESPACE}</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.local.mode</name> | ||
| <value>false</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.am.tez-ui.webservice.enable</name> | ||
| <value>false</value> | ||
| </property> | ||
| <!-- Tez AM should not timeout in ZK Mode --> | ||
| <property> | ||
| <name>tez.session.am.dag.submit.timeout.secs</name> | ||
| <value>-1</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.ignore.lib.uris</name> | ||
| <value>true</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.am.disable.client-version-check</name> | ||
| <value>true</value> | ||
| </property> | ||
| <!-- | ||
| In standalone AM mode with LLAP, the task scheduler (LlapTaskSchedulerService) and task | ||
| communicator (LlapTaskCommunicator) are instantiated at AM startup from service_plugins_descriptor.json | ||
| on the classpath — not from the DAG payload as in YARN-submitted mode. These plugins read their | ||
| configuration from TezConfiguration, which is built exclusively from tez-site.xml on the classpath. | ||
| Hive-specific properties must therefore appear here so the AM can find LLAP daemons and connect | ||
| to them, even though they are conceptually Hive settings. | ||
| --> | ||
| <property> | ||
| <name>hive.zookeeper.quorum</name> | ||
| <value>${HIVE_ZOOKEEPER_QUORUM}</value> | ||
| </property> | ||
| <property> | ||
| <name>hive.llap.daemon.service.hosts</name> | ||
| <value>${HIVE_LLAP_DAEMON_SERVICE_HOSTS}</value> | ||
| </property> | ||
| </configuration> |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -132,14 +132,35 @@ services: | ||
| - zookeeper_datalog:/datalog | ||
| - zookeeper_logs:/logs | ||
| #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly) | ||
| # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682 | ||
| # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon | ||
| # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM | ||
| # Prerequisites: | ||
| # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released | ||
| # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions | ||
| # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume | ||
| tezam: | ||
| profiles: | ||
| - llap | ||
| ||
| image: apache/hive:${HIVE_VERSION} | ||
| ||
| container_name: tezam | ||
| hostname: tezam | ||
| depends_on: | ||
| - zookeeper | ||
| restart: on-failure:3 | ||
| environment: | ||
| USER: hive | ||
| SERVICE_NAME: 'tezam' | ||
| TEZ_FRAMEWORK_MODE: STANDALONE_ZOOKEEPER | ||
| TEZ_AM_ZOOKEEPER_QUORUM: zookeeper:2181 | ||
| # LLAP daemon discovery | ||
| HIVE_ZOOKEEPER_QUORUM: zookeeper:2181 | ||
| HIVE_LLAP_DAEMON_SERVICE_HOSTS: '@llap0' | ||
| # Directories shared between HiveServer2 and LLAP daemon | ||
| HIVE_SCRATCH_DIR: /opt/hive/scratch | ||
| HIVE_QUERY_RESULTS_CACHE_DIRECTORY: /opt/hive/scratch/_resultscache_ | ||
| volumes: | ||
| - warehouse:/opt/hive/data/warehouse | ||
| - scratch:/opt/hive/scratch | ||
| networks: | ||
| - hive | ||
| llapdaemon: | ||
| profiles: | ||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not initialize ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's a yarn-based thing that HS2 initializes a few sessions on startup
in the standalone mode, HS2 has no control over them, just discovers them via ZK