yarn-site.xml (Hadoop 2.6)

yarn.ipc.client.factory.class Factory to create client IPC classes.
Default:
yarn.ipc.server.factory.class Factory to create server IPC classes.
Default:
yarn.ipc.record.factory.class Factory to create serializeable records.
Default:
yarn.ipc.rpc.class RPC class implementation
Default:org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
yarn.resourcemanager.hostname The hostname of the RM.
Default:0.0.0.0
yarn.resourcemanager.address The address of the applications manager interface in the RM.
Default:${yarn.resourcemanager.hostname}:8032
yarn.resourcemanager.bind-host The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.resourcemanager.address and yarn.resourcemanager.webapp.address, respectively. This is most useful for making RM listen to all interfaces by setting to 0.0.0.0.
Default:
yarn.resourcemanager.client.thread-count The number of threads used to handle applications manager requests.
Default:50
yarn.am.liveness-monitor.expiry-interval-ms The expiry interval for application master reporting.
Default:600000
yarn.resourcemanager.principal The Kerberos principal for the resource manager.
Default:
yarn.resourcemanager.scheduler.address The address of the scheduler interface.
Default:${yarn.resourcemanager.hostname}:8030
yarn.resourcemanager.scheduler.client.thread-count Number of threads to handle scheduler interface.
Default:50
yarn.http.policy This configures the HTTP endpoint for Yarn Daemons.The following values are supported: – HTTP_ONLY : Service is provided only on http – HTTPS_ONLY : Service is provided only on https
Default:HTTP_ONLY
yarn.resourcemanager.webapp.address The http address of the RM web application.
Default:${yarn.resourcemanager.hostname}:8088
yarn.resourcemanager.webapp.https.address The https adddress of the RM web application.
Default:${yarn.resourcemanager.hostname}:8090
yarn.resourcemanager.resource-tracker.address
Default:${yarn.resourcemanager.hostname}:8031
yarn.acl.enable Are acls enabled.
Default:false
yarn.admin.acl ACL of who can be admin of the YARN cluster.
Default:*
yarn.resourcemanager.admin.address The address of the RM admin interface.
Default:${yarn.resourcemanager.hostname}:8033
yarn.resourcemanager.admin.client.thread-count Number of threads used to handle RM admin interface.
Default:1
yarn.resourcemanager.connect.max-wait.ms Maximum time to wait to establish connection to ResourceManager.
Default:900000
yarn.resourcemanager.connect.retry-interval.ms How often to try connecting to the ResourceManager.
Default:30000
yarn.resourcemanager.am.max-attempts The maximum number of application attempts. It’s a global setting for all application masters. Each application master can specify its individual maximum number of application attempts via the API, but the individual number cannot be more than the global upper bound. If it is, the resourcemanager will override it. The default number is set to 2, to allow at least one retry for AM.
Default:2
yarn.resourcemanager.container.liveness-monitor.interval-ms How often to check that containers are still alive.
Default:600000
yarn.resourcemanager.keytab The keytab for the resource manager.
Default:/etc/krb5.keytab
yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled Flag to enable override of the default kerberos authentication filter with the RM authentication filter to allow authentication using delegation tokens(fallback to kerberos if the tokens are missing). Only applicable when the http authentication type is kerberos.
Default:true
yarn.nm.liveness-monitor.expiry-interval-ms How long to wait until a node manager is considered dead.
Default:600000
yarn.resourcemanager.nodes.include-path Path to file with nodes to include.
Default:
yarn.resourcemanager.nodes.exclude-path Path to file with nodes to exclude.
Default:
yarn.resourcemanager.resource-tracker.client.thread-count Number of threads to handle resource tracker calls.
Default:50
yarn.resourcemanager.scheduler.class The class to use as the resource scheduler.
Default:org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
yarn.scheduler.minimum-allocation-mb The minimum allocation for every container request at the RM, in MBs. Memory requests lower than this won’t take effect, and the specified value will get allocated at minimum.
Default:1024
yarn.scheduler.maximum-allocation-mb The maximum allocation for every container request at the RM, in MBs. Memory requests higher than this won’t take effect, and will get capped to this value.
Default:8192
yarn.scheduler.minimum-allocation-vcores The minimum allocation for every container request at the RM, in terms of virtual CPU cores. Requests lower than this won’t take effect, and the specified value will get allocated the minimum.
Default:1
yarn.scheduler.maximum-allocation-vcores The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won’t take effect, and will get capped to this value.
Default:32
yarn.resourcemanager.recovery.enabled Enable RM to recover state after starting. If true, then yarn.resourcemanager.store.class must be specified.
Default:false
yarn.resourcemanager.work-preserving-recovery.enabled Enable RM work preserving recovery. This configuration is private to YARN for experimenting the feature.
Default:false
yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms Set the amount of time RM waits before allocating new containers on work-preserving-recovery. Such wait period gives RM a chance to settle down resyncing with NMs in the cluster on recovery, before assigning new containers to applications.
Default:10000
yarn.resourcemanager.store.class The class to use as the persistent store. If org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore is used, the store is implicitly fenced; meaning a single ResourceManager is able to use the store at any point in time. More details on this implicit fencing, along with setting up appropriate ACLs is discussed under yarn.resourcemanager.zk-state-store.root-node.acl.
Default:org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
yarn.resourcemanager.state-store.max-completed-applications The maximum number of completed applications RM state store keeps, less than or equals to ${yarn.resourcemanager.max-completed-applications}. By default, it equals to ${yarn.resourcemanager.max-completed-applications}. This ensures that the applications kept in the state store are consistent with the applications remembered in RM memory. Any values larger than ${yarn.resourcemanager.max-completed-applications} will be reset to ${yarn.resourcemanager.max-completed-applications}. Note that this value impacts the RM recovery performance.Typically, a smaller value indicates better performance on RM recovery.
Default:${yarn.resourcemanager.max-completed-applications}
yarn.resourcemanager.zk-address Host:Port of the ZooKeeper server to be used by the RM. This must be supplied when using the ZooKeeper based implementation of the RM state store and/or embedded automatic failover in a HA setting.
Default:
yarn.resourcemanager.zk-num-retries Number of times RM tries to connect to ZooKeeper.
Default:1000
yarn.resourcemanager.zk-retry-interval-ms Retry interval in milliseconds when connecting to ZooKeeper. When HA is enabled, the value here is NOT used. It is generated automatically from yarn.resourcemanager.zk-timeout-ms and yarn.resourcemanager.zk-num-retries.
Default:1000
yarn.resourcemanager.zk-state-store.parent-path Full path of the ZooKeeper znode where RM state will be stored. This must be supplied when using org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore as the value for yarn.resourcemanager.store.class
Default:/rmstore
yarn.resourcemanager.zk-timeout-ms ZooKeeper session timeout in milliseconds. Session expiration is managed by the ZooKeeper cluster itself, not by the client. This value is used by the cluster to determine when the client’s session expires. Expirations happens when the cluster does not hear from the client within the specified session timeout period (i.e. no heartbeat).
Default:10000
yarn.resourcemanager.zk-acl ACL’s to be used for ZooKeeper znodes.
Default:world:anyone:rwcda
yarn.resourcemanager.zk-state-store.root-node.acl ACLs to be used for the root znode when using ZKRMStateStore in a HA scenario for fencing. ZKRMStateStore supports implicit fencing to allow a single ResourceManager write-access to the store. For fencing, the ResourceManagers in the cluster share read-write-admin privileges on the root node, but the Active ResourceManager claims exclusive create-delete permissions. By default, when this property is not set, we use the ACLs from yarn.resourcemanager.zk-acl for shared admin access and rm-address:random-number for username-based exclusive create-delete access. This property allows users to set ACLs of their choice instead of using the default mechanism. For fencing to work, the ACLs should be carefully set differently on each ResourceManger such that all the ResourceManagers have shared admin access and the Active ResourceManger takes over (exclusively) the create-delete access.
Default:
yarn.resourcemanager.zk-auth Specify the auths to be used for the ACL’s specified in both the yarn.resourcemanager.zk-acl and yarn.resourcemanager.zk-state-store.root-node.acl properties. This takes a comma-separated list of authentication mechanisms, each of the form ‘scheme:auth’ (the same syntax used for the ‘addAuth’ command in the ZK CLI).
Default:
yarn.resourcemanager.fs.state-store.uri URI pointing to the location of the FileSystem path where RM state will be stored. This must be supplied when using org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore as the value for yarn.resourcemanager.store.class
Default:${hadoop.tmp.dir}/yarn/system/rmstore
yarn.resourcemanager.fs.state-store.retry-policy-spec hdfs client retry policy specification. hdfs client retry is always enabled. Specified in pairs of sleep-time and number-of-retries and (t0, n0), (t1, n1), …, the first n0 retries sleep t0 milliseconds on average, the following n1 retries sleep t1 milliseconds on average, and so on.
Default:2000, 500
yarn.resourcemanager.ha.enabled Enable RM high-availability. When enabled, (1) The RM starts in the Standby mode by default, and transitions to the Active mode when prompted to. (2) The nodes in the RM ensemble are listed in yarn.resourcemanager.ha.rm-ids (3) The id of each RM either comes from yarn.resourcemanager.ha.id if yarn.resourcemanager.ha.id is explicitly specified or can be figured out by matching yarn.resourcemanager.address.{id} with local address (4) The actual physical addresses come from the configs of the pattern – {rpc-config}.{id}
Default:false
yarn.resourcemanager.ha.automatic-failover.enabled Enable automatic failover. By default, it is enabled only when HA is enabled
Default:true
yarn.resourcemanager.ha.automatic-failover.embedded Enable embedded automatic failover. By default, it is enabled only when HA is enabled. The embedded elector relies on the RM state store to handle fencing, and is primarily intended to be used in conjunction with ZKRMStateStore.
Default:true
yarn.resourcemanager.ha.automatic-failover.zk-base-path The base znode path to use for storing leader information, when using ZooKeeper based leader election.
Default:/yarn-leader-election
yarn.resourcemanager.cluster-id Name of the cluster. In a HA setting, this is used to ensure the RM participates in leader election for this cluster and ensures it does not affect other clusters
Default:
yarn.resourcemanager.ha.rm-ids The list of RM nodes in the cluster when HA is enabled. See description of yarn.resourcemanager.ha .enabled for full details on how this is used.
Default:
yarn.resourcemanager.ha.id The id (string) of the current RM. When HA is enabled, this is an optional config. The id of current RM can be set by explicitly specifying yarn.resourcemanager.ha.id or figured out by matching yarn.resourcemanager.address.{id} with local address See description of yarn.resourcemanager.ha.enabled for full details on how this is used.
Default:
yarn.client.failover-proxy-provider When HA is enabled, the class to be used by Clients, AMs and NMs to failover to the Active RM. It should extend org.apache.hadoop.yarn.client.RMFailoverProxyProvider
Default:org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider
yarn.client.failover-max-attempts When HA is enabled, the max number of times FailoverProxyProvider should attempt failover. When set, this overrides the yarn.resourcemanager.connect.max-wait.ms. When not set, this is inferred from yarn.resourcemanager.connect.max-wait.ms.
Default:
yarn.client.failover-sleep-base-ms When HA is enabled, the sleep base (in milliseconds) to be used for calculating the exponential delay between failovers. When set, this overrides the yarn.resourcemanager.connect.* settings. When not set, yarn.resourcemanager.connect.retry-interval.ms is used instead.
Default:
yarn.client.failover-sleep-max-ms When HA is enabled, the maximum sleep time (in milliseconds) between failovers. When set, this overrides the yarn.resourcemanager.connect.* settings. When not set, yarn.resourcemanager.connect.retry-interval.ms is used instead.
Default:
yarn.client.failover-retries When HA is enabled, the number of retries per attempt to connect to a ResourceManager. In other words, it is the ipc.client.connect.max.retries to be used during failover attempts
Default:0
yarn.client.failover-retries-on-socket-timeouts When HA is enabled, the number of retries per attempt to connect to a ResourceManager on socket timeouts. In other words, it is the ipc.client.connect.max.retries.on.timeouts to be used during failover attempts
Default:0
yarn.resourcemanager.max-completed-applications The maximum number of completed applications RM keeps.
Default:10000
yarn.resourcemanager.delayed.delegation-token.removal-interval-ms Interval at which the delayed token removal thread runs
Default:30000
yarn.resourcemanager.proxy-user-privileges.enabled If true, ResourceManager will have proxy-user privileges. Use case: In a secure cluster, YARN requires the user hdfs delegation-tokens to do localization and log-aggregation on behalf of the user. If this is set to true, ResourceManager is able to request new hdfs delegation tokens on behalf of the user. This is needed by long-running-service, because the hdfs tokens will eventually expire and YARN requires new valid tokens to do localization and log-aggregation. Note that to enable this use case, the corresponding HDFS NameNode has to configure ResourceManager as the proxy-user so that ResourceManager can itself ask for new tokens on behalf of the user when tokens are past their max-life-time.
Default:false
yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs Interval for the roll over for the master key used to generate application tokens
Default:86400
yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs Interval for the roll over for the master key used to generate container tokens. It is expected to be much greater than yarn.nm.liveness-monitor.expiry-interval-ms and yarn.rm.container-allocation.expiry-interval-ms. Otherwise the behavior is undefined.
Default:86400
yarn.resourcemanager.nodemanagers.heartbeat-interval-ms The heart-beat interval in milliseconds for every NodeManager in the cluster.
Default:1000
yarn.resourcemanager.nodemanager.minimum.version The minimum allowed version of a connecting nodemanager. The valid values are NONE (no version checking), EqualToRM (the nodemanager’s version is equal to or greater than the RM version), or a Version String.
Default:NONE
yarn.resourcemanager.scheduler.monitor.enable Enable a set of periodic monitors (specified in yarn.resourcemanager.scheduler.monitor.policies) that affect the scheduler.
Default:false
yarn.resourcemanager.scheduler.monitor.policies The list of SchedulingEditPolicy classes that interact with the scheduler. A particular module may be incompatible with the scheduler, other policies, or a configuration of either.
Default:org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy
yarn.resourcemanager.configuration.provider-class The class to use as the configuration provider. If org.apache.hadoop.yarn.LocalConfigurationProvider is used, the local configuration will be loaded. If org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider is used, the configuration which will be loaded should be uploaded to remote File system first.
Default:org.apache.hadoop.yarn.LocalConfigurationProvider
yarn.resourcemanager.system-metrics-publisher.enabled The setting that controls whether yarn system metrics is published on the timeline server or not by RM.
Default:false
yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size Number of worker threads that send the yarn system metrics data.
Default:10
yarn.nodemanager.hostname The hostname of the NM.
Default:0.0.0.0
yarn.nodemanager.address The address of the container manager in the NM.
Default:${yarn.nodemanager.hostname}:0
yarn.nodemanager.bind-host The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.nodemanager.address and yarn.nodemanager.webapp.address, respectively. This is most useful for making NM listen to all interfaces by setting to 0.0.0.0.
Default:
yarn.nodemanager.admin-env Environment variables that should be forwarded from the NodeManager’s environment to the container’s.
Default:MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX
yarn.nodemanager.env-whitelist Environment variables that containers may override rather than use NodeManager’s default.
Default:JAVA_HOME, HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, HADOOP_CONF_DIR, HADOOP_YARN_HOME
yarn.nodemanager.container-executor.class who will execute(launch) the containers.
Default:org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor
yarn.nodemanager.container-manager.thread-count Number of threads container manager uses.
Default:20
yarn.nodemanager.delete.thread-count Number of threads used in cleanup.
Default:4
yarn.nodemanager.delete.debug-delay-sec Number of seconds after an application finishes before the nodemanager’s DeletionService will delete the application’s localized file directory and log directory. To diagnose Yarn application problems, set this property’s value large enough (for example, to 600 = 10 minutes) to permit examination of these directories. After changing the property’s value, you must restart the nodemanager in order for it to have an effect. The roots of Yarn applications’ work directories is configurable with the yarn.nodemanager.local-dirs property (see below), and the roots of the Yarn applications’ log directories is configurable with the yarn.nodemanager.log-dirs property (see also below).
Default:0
yarn.nodemanager.keytab Keytab for NM.
Default:/etc/krb5.keytab
yarn.nodemanager.local-dirs List of directories to store localized files in. An application’s localized file directory will be found in: ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers’ work directories, called container_${contid}, will be subdirectories of this.
Default:${hadoop.tmp.dir}/nm-local-dir
yarn.nodemanager.local-cache.max-files-per-directory It limits the maximum number of files which will be localized in a single local directory. If the limit is reached then sub-directories will be created and new files will be localized in them. If it is set to a value less than or equal to 36 [which are sub-directories (0-9 and then a-z)] then NodeManager will fail to start. For example; [for public cache] if this is configured with a value of 40 ( 4 files + 36 sub-directories) and the local-dir is “/tmp/local-dir1″ then it will allow 4 files to be created directly inside “/tmp/local-dir1/filecache”. For files that are localized further it will create a sub-directory “0” inside “/tmp/local-dir1/filecache” and will localize files inside it until it becomes full. If a file is removed from a sub-directory that is marked full, then that sub-directory will be used back again to localize files.
Default:8192
yarn.nodemanager.localizer.address Address where the localizer IPC is.
Default:${yarn.nodemanager.hostname}:8040
yarn.nodemanager.localizer.cache.cleanup.interval-ms Interval in between cache cleanups.
Default:600000
yarn.nodemanager.localizer.cache.target-size-mb Target size of localizer cache in MB, per nodemanager. It is a target retention size that only includes resources with PUBLIC and PRIVATE visibility and excludes resources with APPLICATION visibility
Default:10240
yarn.nodemanager.localizer.client.thread-count Number of threads to handle localization requests.
Default:5
yarn.nodemanager.localizer.fetch.thread-count Number of threads to use for localization fetching.
Default:4
yarn.nodemanager.log-dirs Where to store container logs. An application’s localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers’ log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container.
Default:${yarn.log.dir}/userlogs
yarn.log-aggregation-enable Whether to enable log aggregation. Log aggregation collects each container’s logs and moves these logs onto a file-system, for e.g. HDFS, after the application completes. Users can configure the “yarn.nodemanager.remote-app-log-dir” and “yarn.nodemanager.remote-app-log-dir-suffix” properties to determine where these logs are moved to. Users can access the logs via the Application Timeline Server.
Default:false
yarn.log-aggregation.retain-seconds How long to keep aggregation logs before deleting them. -1 disables. Be careful set this too small and you will spam the name node.
Default:-1
yarn.log-aggregation.retain-check-interval-seconds How long to wait between aggregated log retention checks. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful set this too small and you will spam the name node.
Default:-1
yarn.nodemanager.log.retain-seconds Time in seconds to retain user logs. Only applicable if log aggregation is disabled
Default:10800
yarn.nodemanager.remote-app-log-dir Where to aggregate logs to.
Default:/tmp/logs
yarn.nodemanager.remote-app-log-dir-suffix The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}
Default:logs
yarn.nodemanager.resource.memory-mb Amount of physical memory, in MB, that can be allocated for containers.
Default:8192
yarn.nodemanager.pmem-check-enabled Whether physical memory limits will be enforced for containers.
Default:true
yarn.nodemanager.vmem-check-enabled Whether virtual memory limits will be enforced for containers.
Default:true
yarn.nodemanager.vmem-pmem-ratio Ratio between virtual memory to physical memory when setting memory limits for containers. Container allocations are expressed in terms of physical memory, and virtual memory usage is allowed to exceed this allocation by this ratio.
Default:2.1
yarn.nodemanager.resource.cpu-vcores Number of vcores that can be allocated for containers. This is used by the RM scheduler when allocating resources for containers. This is not used to limit the number of physical cores used by YARN containers.
Default:8
yarn.nodemanager.resource.percentage-physical-cpu-limit Percentage of CPU that can be allocated for containers. This setting allows users to limit the amount of CPU that YARN containers use. Currently functional only on Linux using cgroups. The default is to use 100% of CPU.
Default:100
yarn.nodemanager.webapp.address NM Webapp address.
Default:${yarn.nodemanager.hostname}:8042
yarn.nodemanager.container-monitor.interval-ms How often to monitor containers.
Default:3000
yarn.nodemanager.container-monitor.resource-calculator.class Class that calculates containers current resource utilization.
Default:
yarn.nodemanager.health-checker.interval-ms Frequency of running node health script.
Default:600000
yarn.nodemanager.health-checker.script.timeout-ms Script time out period.
Default:1200000
yarn.nodemanager.health-checker.script.path The health check script to run.
Default:
yarn.nodemanager.health-checker.script.opts The arguments to pass to the health check script.
Default:
yarn.nodemanager.disk-health-checker.interval-ms Frequency of running disk health checker code.
Default:120000
yarn.nodemanager.disk-health-checker.min-healthy-disks The minimum fraction of number of disks to be healthy for the nodemanager to launch new containers. This correspond to both yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs. i.e. If there are less number of healthy local-dirs (or log-dirs) available, then new containers will not be launched on this node.
Default:0.25
yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage The maximum percentage of disk space utilization allowed after which a disk is marked as bad. Values can range from 0.0 to 100.0. If the value is greater than or equal to 100, the nodemanager will check for full disk. This applies to yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs.
Default:90.0
yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb The minimum space that must be available on a disk for it to be used. This applies to yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs.
Default:0
yarn.nodemanager.linux-container-executor.path The path to the Linux container executor.
Default:
yarn.nodemanager.linux-container-executor.resources-handler.class The class which should help the LCE handle resources.
Default:org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler
yarn.nodemanager.linux-container-executor.cgroups.hierarchy The cgroups hierarchy under which to place YARN proccesses (cannot contain commas). If yarn.nodemanager.linux-container-executor.cgroups.mount is false (that is, if cgroups have been pre-configured), then this cgroups hierarchy must already exist and be writable by the NodeManager user, otherwise the NodeManager may fail. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler.
Default:/hadoop-yarn
yarn.nodemanager.linux-container-executor.cgroups.mount Whether the LCE should attempt to mount cgroups if not found. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler.
Default:false
yarn.nodemanager.linux-container-executor.cgroups.mount-path Where the LCE should attempt to mount cgroups if not found. Common locations include /sys/fs/cgroup and /cgroup; the default location can vary depending on the Linux distribution in use. This path must exist before the NodeManager is launched. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler, and yarn.nodemanager.linux-container-executor.cgroups.mount is true.
Default:
yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users This determines which of the two modes that LCE should use on a non-secure cluster. If this value is set to true, then all containers will be launched as the user specified in yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user. If this value is set to false, then containers will run as the user who submitted the application.
Default:true
yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user The UNIX user that containers will run as when Linux-container-executor is used in nonsecure mode (a use case for this is using cgroups) if the yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users is set to true.
Default:nobody
yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern The allowed pattern for UNIX user names enforced by Linux-container-executor when used in nonsecure mode (use case for this is using cgroups). The default value is taken from /usr/sbin/adduser
Default:^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$
yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage This flag determines whether apps should run with strict resource limits or be allowed to consume spare resources if they need them. For example, turning the flag on will restrict apps to use only their share of CPU, even if the node has spare CPU cycles. The default value is false i.e. use available resources. Please note that turning this flag on may reduce job throughput on the cluster.
Default:false
yarn.nodemanager.log-aggregation.compression-type T-file compression types used to compress aggregated logs.
Default:none
yarn.nodemanager.principal The kerberos principal for the node manager.
Default:
yarn.nodemanager.aux-services the valid service name should only contain a-zA-Z0-9_ and can not start with numbers
Default:
yarn.nodemanager.sleep-delay-before-sigkill.ms No. of ms to wait between sending a SIGTERM and SIGKILL to a container
Default:250
yarn.nodemanager.process-kill-wait.ms Max time to wait for a process to come up when trying to cleanup a container
Default:2000
yarn.nodemanager.resourcemanager.minimum.version The minimum allowed version of a resourcemanager that a nodemanager will connect to. The valid values are NONE (no version checking), EqualToNM (the resourcemanager’s version is equal to or greater than the NM version), or a Version String.
Default:NONE
yarn.client.nodemanager-client-async.thread-pool-max-size Max number of threads in NMClientAsync to process container management events
Default:500
yarn.client.nodemanager-connect.max-wait-ms Max time to wait to establish a connection to NM
Default:900000
yarn.client.nodemanager-connect.retry-interval-ms Time interval between each attempt to connect to NM
Default:10000
yarn.client.max-cached-nodemanagers-proxies Maximum number of proxy connections to cache for node managers. If set to a value greater than zero then the cache is enabled and the NMClient and MRAppMaster will cache the specified number of node manager proxies. There will be at max one proxy per node manager. Ex. configuring it to a value of 5 will make sure that client will at max have 5 proxies cached with 5 different node managers. These connections for these proxies will be timed out if idle for more than the system wide idle timeout period. Note that this could cause issues on large clusters as many connections could linger simultaneously and lead to a large number of connection threads. The token used for authentication will be used only at connection creation time. If a new token is received then the earlier connection should be closed in order to use the new token. This and (yarn.client.nodemanager-client-async.thread-pool-max-size) are related and should be in sync (no need for them to be equal). If the value of this property is zero then the connection cache is disabled and connections will use a zero idle timeout to prevent too many connection threads on large clusters.
Default:0
yarn.nodemanager.recovery.enabled Enable the node manager to recover after starting
Default:false
yarn.nodemanager.recovery.dir The local filesystem directory in which the node manager will store state when recovery is enabled.
Default:${hadoop.tmp.dir}/yarn-nm-recovery
yarn.nodemanager.docker-container-executor.exec-name Name or path to the Docker client.
Default:/usr/bin/docker
yarn.nodemanager.aux-services.mapreduce_shuffle.class
Default:org.apache.hadoop.mapred.ShuffleHandler
mapreduce.job.jar
Default:
mapreduce.job.hdfs-servers
Default:${fs.defaultFS}
yarn.web-proxy.principal The kerberos principal for the proxy, if the proxy is not running as part of the RM.
Default:
yarn.web-proxy.keytab Keytab for WebAppProxy, if the proxy is not running as part of the RM.
Default:
yarn.web-proxy.address The address for the web proxy as HOST:PORT, if this is not given then the proxy will run as part of the RM
Default:
yarn.application.classpath CLASSPATH for YARN applications. A comma-separated list of CLASSPATH entries. When this value is empty, the following default CLASSPATH for YARN applications would be used. For Linux: $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*, $HADOOP_COMMON_HOME/share/hadoop/common/lib/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*, $HADOOP_YARN_HOME/share/hadoop/yarn/*, $HADOOP_YARN_HOME/share/hadoop/yarn/lib/* For Windows: %HADOOP_CONF_DIR%, %HADOOP_COMMON_HOME%/share/hadoop/common/*, %HADOOP_COMMON_HOME%/share/hadoop/common/lib/*, %HADOOP_HDFS_HOME%/share/hadoop/hdfs/*, %HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*, %HADOOP_YARN_HOME%/share/hadoop/yarn/*, %HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*
Default:
yarn.timeline-service.enabled Indicate to clients whether timeline service is enabled or not. If enabled, clients will put entities and events to the timeline server.
Default:false
yarn.timeline-service.hostname The hostname of the timeline service web application.
Default:0.0.0.0
yarn.timeline-service.address This is default address for the timeline server to start the RPC server.
Default:${yarn.timeline-service.hostname}:10200
yarn.timeline-service.webapp.address The http address of the timeline service web application.
Default:${yarn.timeline-service.hostname}:8188
yarn.timeline-service.webapp.https.address The https address of the timeline service web application.
Default:${yarn.timeline-service.hostname}:8190
yarn.timeline-service.bind-host The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.timeline-service.address and yarn.timeline-service.webapp.address, respectively. This is most useful for making the service listen to all interfaces by setting to 0.0.0.0.
Default:
yarn.timeline-service.store-class Store class name for timeline store.
Default:org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore
yarn.timeline-service.ttl-enable Enable age off of timeline store data.
Default:true
yarn.timeline-service.ttl-ms Time to live for timeline store data in milliseconds.
Default:604800000
yarn.timeline-service.leveldb-timeline-store.path Store file name for leveldb timeline store.
Default:${hadoop.tmp.dir}/yarn/timeline
yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms Length of time to wait between deletion cycles of leveldb timeline store in milliseconds.
Default:300000
yarn.timeline-service.leveldb-timeline-store.read-cache-size Size of read cache for uncompressed blocks for leveldb timeline store in bytes.
Default:104857600
yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size Size of cache for recently read entity start times for leveldb timeline store in number of entities.
Default:10000
yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size Size of cache for recently written entity start times for leveldb timeline store in number of entities.
Default:10000
yarn.timeline-service.handler-thread-count Handler thread count to serve the client RPC requests.
Default:10
yarn.timeline-service.http-authentication.type Defines authentication used for the timeline server HTTP endpoint. Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#
Default:simple
yarn.timeline-service.http-authentication.simple.anonymous.allowed Indicates if anonymous requests are allowed by the timeline server when using ‘simple’ authentication.
Default:true
yarn.timeline-service.principal The Kerberos principal for the timeline server.
Default:
yarn.timeline-service.keytab The Kerberos keytab for the timeline server.
Default:/etc/krb5.keytab
yarn.timeline-service.client.max-retries Default maximum number of retires for timeline servive client.
Default:30
yarn.timeline-service.client.retry-interval-ms Default retry time interval for timeline servive client.
Default:1000
yarn.client.application-client-protocol.poll-interval-ms The interval that the yarn client library uses to poll the completion status of the asynchronous API of application client protocol.
Default:200
yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled RSS usage of a process computed via /proc/pid/stat is not very accurate as it includes shared pages of a process. /proc/pid/smaps provides useful information like Private_Dirty, Private_Clean, Shared_Dirty, Shared_Clean which can be used for computing more accurate RSS. When this flag is enabled, RSS is computed as Min(Shared_Dirty, Pss) + Private_Clean + Private_Dirty. It excludes read-only shared mappings in RSS computation.
Default:false
hadoop.registry.rm.enabled Is the registry enabled: does the RM start it up, create the user and system paths, and purge service records when containers, application attempts and applications complete
Default:false
hadoop.registry.zk.root
Default:/registry
hadoop.registry.zk.session.timeout.ms Zookeeper session timeout in milliseconds
Default:60000
hadoop.registry.zk.connection.timeout.ms Zookeeper session timeout in milliseconds
Default:15000
hadoop.registry.zk.retry.times Zookeeper connection retry count before failing
Default:5
hadoop.registry.zk.retry.interval.ms
Default:1000
hadoop.registry.zk.retry.ceiling.ms Zookeeper retry limit in milliseconds, during exponential backoff: {@value} This places a limit even if the retry times and interval limit, combined with the backoff policy, result in a long retry period
Default:60000
hadoop.registry.zk.quorum List of hostname:port pairs defining the zookeeper quorum binding for the registry
Default:localhost:2181
hadoop.registry.secure Key to set if the registry is secure. Turning it on changes the permissions policy from “open access” to restrictions on kerberos with the option of a user adding one or more auth key pairs down their own tree.
Default:false
hadoop.registry.system.acls A comma separated list of Zookeeper ACL identifiers with system access to the registry in a secure cluster. These are given full access to all entries. If there is an “@” at the end of a SASL entry it instructs the registry client to append the default kerberos domain.
Default:sasl:yarn@, sasl:mapred@, sasl:mapred@hdfs@
hadoop.registry.kerberos.realm The kerberos realm: used to set the realm of system principals which do not declare their realm, and any other accounts that need the value. If empty, the default realm of the running process is used. If neither are known and the realm is needed, then the registry service/client will fail.
Default:
hadoop.registry.jaas.context Key to define the JAAS context. Used in secure mode
Default:Client
yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds Defines how often NMs wake up to upload log files. The default value is -1. By default, the logs will be uploaded when the application is finished. By setting this configure, logs can be uploaded periodically when the application is running. The minimum rolling-interval-seconds can be set is 3600.
Default:-1

Leave a comment