hdfs-site.xml (Hadoop 2.6)

hadoop.hdfs.configuration.version version of this configuration file
Default:1
dfs.namenode.logging.level The logging level for dfs namenode. Other values are “dir” (trace namespace mutations), “block” (trace block under/over replications and block creations/deletions), or “all”.
Default:info
dfs.namenode.rpc-address RPC address that handles all clients requests. In the case of HA/Federation where multiple namenodes exist, the name service id is added to the name e.g. dfs.namenode.rpc-address.ns1 dfs.namenode.rpc-address.EXAMPLENAMESERVICE The value of this property will take the form of nn-host1:rpc-port.
Default:
dfs.namenode.rpc-bind-host The actual address the RPC server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.rpc-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node listen on all interfaces by setting it to 0.0.0.0.
Default:
dfs.namenode.servicerpc-address RPC address for HDFS Services communication. BackupNode, Datanodes and all other services should be connecting to this address if it is configured. In the case of HA/Federation where multiple namenodes exist, the name service id is added to the name e.g. dfs.namenode.servicerpc-address.ns1 dfs.namenode.rpc-address.EXAMPLENAMESERVICE The value of this property will take the form of nn-host1:rpc-port. If the value of this property is unset the value of dfs.namenode.rpc-address will be used as the default.
Default:
dfs.namenode.servicerpc-bind-host The actual address the service RPC server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.servicerpc-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node listen on all interfaces by setting it to 0.0.0.0.
Default:
dfs.namenode.secondary.http-address The secondary namenode http server address and port.
Default:0.0.0.0:50090
dfs.namenode.secondary.https-address The secondary namenode HTTPS server address and port.
Default:0.0.0.0:50091
dfs.datanode.address The datanode server address and port for data transfer.
Default:0.0.0.0:50010
dfs.datanode.http.address The datanode http server address and port.
Default:0.0.0.0:50075
dfs.datanode.ipc.address The datanode ipc server address and port.
Default:0.0.0.0:50020
dfs.datanode.handler.count The number of server threads for the datanode.
Default:10
dfs.namenode.http-address The address and the base port where the dfs namenode web ui will listen on.
Default:0.0.0.0:50070
dfs.namenode.http-bind-host The actual adress the HTTP server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.http-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node HTTP server listen on all interfaces by setting it to 0.0.0.0.
Default:
dfs.https.enable Deprecated. Use “dfs.http.policy” instead.
Default:false
dfs.http.policy Decide if HTTPS(SSL) is supported on HDFS This configures the HTTP endpoint for HDFS daemons: The following values are supported: – HTTP_ONLY : Service is provided only on http – HTTPS_ONLY : Service is provided only on https – HTTP_AND_HTTPS : Service is provided both on http and https
Default:HTTP_ONLY
dfs.client.https.need-auth Whether SSL client certificate authentication is required
Default:false
dfs.client.cached.conn.retry The number of times the HDFS client will pull a socket from the cache. Once this number is exceeded, the client will try to create a new socket.
Default:3
dfs.https.server.keystore.resource Resource file from which ssl server keystore information will be extracted
Default:ssl-server.xml
dfs.client.https.keystore.resource Resource file from which ssl client keystore information will be extracted
Default:ssl-client.xml
dfs.datanode.https.address The datanode secure http server address and port.
Default:0.0.0.0:50475
dfs.namenode.https-address The namenode secure http server address and port.
Default:0.0.0.0:50470
dfs.namenode.https-bind-host The actual adress the HTTPS server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.https-address. It can also be specified per name node or name service for HA/Federation. This is useful for making the name node HTTPS server listen on all interfaces by setting it to 0.0.0.0.
Default:
dfs.datanode.dns.interface The name of the Network Interface from which a data node should report its IP address.
Default:default
dfs.datanode.dns.nameserver The host name or IP address of the name server (DNS) which a DataNode should use to determine the host name used by the NameNode for communication and display purposes.
Default:default
dfs.namenode.backup.address The backup node server address and port. If the port is 0 then the server will start on a free port.
Default:0.0.0.0:50100
dfs.namenode.backup.http-address The backup node http server address and port. If the port is 0 then the server will start on a free port.
Default:0.0.0.0:50105
dfs.namenode.replication.considerLoad Decide if chooseTarget considers the target’s load or not
Default:true
dfs.default.chunk.view.size The number of bytes to view for a file on the browser.
Default:32768
dfs.datanode.du.reserved Reserved space in bytes per volume. Always leave this much space free for non dfs use.
Default:0
dfs.namenode.name.dir Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy.
Default:file://${hadoop.tmp.dir}/dfs/name
dfs.namenode.name.dir.restore Set to true to enable NameNode to attempt recovering a previously failed dfs.namenode.name.dir. When enabled, a recovery of any failed directory is attempted during checkpoint.
Default:false
dfs.namenode.fs-limits.max-component-length Defines the maximum number of bytes in UTF-8 encoding in each component of a path. A value of 0 will disable the check.
Default:255
dfs.namenode.fs-limits.max-directory-items Defines the maximum number of items that a directory may contain. A value of 0 will disable the check.
Default:1048576
dfs.namenode.fs-limits.min-block-size Minimum block size in bytes, enforced by the Namenode at create time. This prevents the accidental creation of files with tiny block sizes (and thus many blocks), which can degrade performance.
Default:1048576
dfs.namenode.fs-limits.max-blocks-per-file Maximum number of blocks per file, enforced by the Namenode on write. This prevents the creation of extremely large files which can degrade performance.
Default:1048576
dfs.namenode.edits.dir Determines where on the local filesystem the DFS name node should store the transaction (edits) file. If this is a comma-delimited list of directories then the transaction file is replicated in all of the directories, for redundancy. Default value is same as dfs.namenode.name.dir
Default:${dfs.namenode.name.dir}
dfs.namenode.shared.edits.dir A directory on shared storage between the multiple namenodes in an HA cluster. This directory will be written by the active and read by the standby in order to keep the namespaces synchronized. This directory does not need to be listed in dfs.namenode.edits.dir above. It should be left empty in a non-HA cluster.
Default:
dfs.namenode.edits.journal-plugin.qjournal
Default:org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager
dfs.permissions.enabled If “true”, enable permission checking in HDFS. If “false”, permission checking is turned off, but all other behavior is unchanged. Switching from one parameter value to the other does not change the mode, owner or group of files or directories.
Default:true
dfs.permissions.superusergroup The name of the group of super-users.
Default:supergroup
dfs.namenode.acls.enabled Set to true to enable support for HDFS ACLs (Access Control Lists). By default, ACLs are disabled. When ACLs are disabled, the NameNode rejects all RPCs related to setting or getting ACLs.
Default:false
dfs.namenode.lazypersist.file.scrub.interval.sec The NameNode periodically scans the namespace for LazyPersist files with missing blocks and unlinks them from the namespace. This configuration key controls the interval between successive scans. Set it to a negative value to disable this behavior.
Default:300
dfs.block.access.token.enable If “true”, access tokens are used as capabilities for accessing datanodes. If “false”, no access tokens are checked on accessing datanodes.
Default:false
dfs.block.access.key.update.interval Interval in minutes at which namenode updates its access keys.
Default:600
dfs.block.access.token.lifetime The lifetime of access tokens in minutes.
Default:600
dfs.datanode.data.dir Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored.
Default:file://${hadoop.tmp.dir}/dfs/data
dfs.datanode.data.dir.perm Permissions for the directories on on the local filesystem where the DFS data node store its blocks. The permissions can either be octal or symbolic.
Default:700
dfs.replication Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.
Default:3
dfs.replication.max Maximal block replication.
Default:512
dfs.namenode.replication.min Minimal block replication.
Default:1
dfs.blocksize The default block size for new files, in bytes. You can use the following suffix (case insensitive): k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.), Or provide complete size in bytes (such as 134217728 for 128 MB).
Default:134217728
dfs.client.block.write.retries The number of retries for writing blocks to the data nodes, before we signal failure to the application.
Default:3
dfs.client.block.write.replace-datanode-on-failure.enable If there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes. As a result, the number of datanodes in the pipeline is decreased. The feature is to add new datanodes to the pipeline. This is a site-wide property to enable/disable the feature. When the cluster size is extremely small, e.g. 3 nodes or less, cluster administrators may want to set the policy to NEVER in the default configuration file or disable this feature. Otherwise, users may experience an unusually high rate of pipeline failures since it is impossible to find new datanodes for replacement. See also dfs.client.block.write.replace-datanode-on-failure.policy
Default:true
dfs.client.block.write.replace-datanode-on-failure.policy This property is used only if the value of dfs.client.block.write.replace-datanode-on-failure.enable is true. ALWAYS: always add a new datanode when an existing datanode is removed. NEVER: never add a new datanode. DEFAULT: Let r be the replication number. Let n be the number of existing datanodes. Add a new datanode only if r is greater than or equal to 3 and either (1) floor(r/2) is greater than or equal to n; or (2) r is greater than n and the block is hflushed/appended.
Default:DEFAULT
dfs.client.block.write.replace-datanode-on-failure.best-effort This property is used only if the value of dfs.client.block.write.replace-datanode-on-failure.enable is true. Best effort means that the client will try to replace a failed datanode in write pipeline (provided that the policy is satisfied), however, it continues the write operation in case that the datanode replacement also fails. Suppose the datanode replacement fails. false: An exception should be thrown so that the write will fail. true : The write should be resumed with the remaining datandoes. Note that setting this property to true allows writing to a pipeline with a smaller number of datanodes. As a result, it increases the probability of data loss.
Default:false
dfs.blockreport.intervalMsec Determines block reporting interval in milliseconds.
Default:21600000
dfs.blockreport.initialDelay Delay for first block report in seconds.
Default:0
dfs.blockreport.split.threshold If the number of blocks on the DataNode is below this threshold then it will send block reports for all Storage Directories in a single message. If the number of blocks exceeds this threshold then the DataNode will send block reports for each Storage Directory in separate messages. Set to zero to always split.
Default:1000000
dfs.datanode.directoryscan.interval Interval in seconds for Datanode to scan data directories and reconcile the difference between blocks in memory and on the disk.
Default:21600
dfs.datanode.directoryscan.threads How many threads should the threadpool used to compile reports for volumes in parallel have.
Default:1
dfs.heartbeat.interval Determines datanode heartbeat interval in seconds.
Default:3
dfs.namenode.handler.count The number of server threads for the namenode.
Default:10
dfs.namenode.safemode.threshold-pct Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.namenode.replication.min. Values less than or equal to 0 mean not to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safe mode permanent.
Default:0.999f
dfs.namenode.safemode.min.datanodes Specifies the number of datanodes that must be considered alive before the name node exits safemode. Values less than or equal to 0 mean not to take the number of live datanodes into account when deciding whether to remain in safe mode during startup. Values greater than the number of datanodes in the cluster will make safe mode permanent.
Default:0
dfs.namenode.safemode.extension Determines extension of safe mode in milliseconds after the threshold level is reached.
Default:30000
dfs.namenode.resource.check.interval The interval in milliseconds at which the NameNode resource checker runs. The checker calculates the number of the NameNode storage volumes whose available spaces are more than dfs.namenode.resource.du.reserved, and enters safemode if the number becomes lower than the minimum value specified by dfs.namenode.resource.checked.volumes.minimum.
Default:5000
dfs.namenode.resource.du.reserved The amount of space to reserve/require for a NameNode storage directory in bytes. The default is 100MB.
Default:104857600
dfs.namenode.resource.checked.volumes A list of local directories for the NameNode resource checker to check in addition to the local edits directories.
Default:
dfs.namenode.resource.checked.volumes.minimum The minimum number of redundant NameNode storage volumes required.
Default:1
dfs.datanode.balance.bandwidthPerSec Specifies the maximum amount of bandwidth that each datanode can utilize for the balancing purpose in term of the number of bytes per second.
Default:1048576
dfs.hosts Names a file that contains a list of hosts that are permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, all hosts are permitted.
Default:
dfs.hosts.exclude Names a file that contains a list of hosts that are not permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, no hosts are excluded.
Default:
dfs.namenode.max.objects The maximum number of files, directories and blocks dfs supports. A value of zero indicates no limit to the number of objects that dfs supports.
Default:0
dfs.namenode.datanode.registration.ip-hostname-check If true (the default), then the namenode requires that a connecting datanode’s address must be resolved to a hostname. If necessary, a reverse DNS lookup is performed. All attempts to register a datanode from an unresolvable address are rejected. It is recommended that this setting be left on to prevent accidental registration of datanodes listed by hostname in the excludes file during a DNS outage. Only set this to false in environments where there is no infrastructure to support reverse DNS lookup.
Default:true
dfs.namenode.decommission.interval Namenode periodicity in seconds to check if decommission is complete.
Default:30
dfs.namenode.decommission.nodes.per.interval The number of nodes namenode checks if decommission is complete in each dfs.namenode.decommission.interval.
Default:5
dfs.namenode.replication.interval The periodicity in seconds with which the namenode computes repliaction work for datanodes.
Default:3
dfs.namenode.accesstime.precision The access time for HDFS file is precise upto this value. The default value is 1 hour. Setting a value of 0 disables access times for HDFS.
Default:3600000
dfs.datanode.plugins Comma-separated list of datanode plug-ins to be activated.
Default:
dfs.namenode.plugins Comma-separated list of namenode plug-ins to be activated.
Default:
dfs.stream-buffer-size The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations.
Default:4096
dfs.bytes-per-checksum The number of bytes per checksum. Must not be larger than dfs.stream-buffer-size
Default:512
dfs.client-write-packet-size Packet size for clients to write
Default:65536
dfs.client.write.exclude.nodes.cache.expiry.interval.millis The maximum period to keep a DN in the excluded nodes list at a client. After this period, in milliseconds, the previously excluded node(s) will be removed automatically from the cache and will be considered good for block allocations again. Useful to lower or raise in situations where you keep a file open for very long periods (such as a Write-Ahead-Log (WAL) file) to make the writer tolerant to cluster maintenance restarts. Defaults to 10 minutes.
Default:600000
dfs.namenode.checkpoint.dir Determines where on the local filesystem the DFS secondary name node should store the temporary images to merge. If this is a comma-delimited list of directories then the image is replicated in all of the directories for redundancy.
Default:file://${hadoop.tmp.dir}/dfs/namesecondary
dfs.namenode.checkpoint.edits.dir Determines where on the local filesystem the DFS secondary name node should store the temporary edits to merge. If this is a comma-delimited list of directoires then teh edits is replicated in all of the directoires for redundancy. Default value is same as dfs.namenode.checkpoint.dir
Default:${dfs.namenode.checkpoint.dir}
dfs.namenode.checkpoint.period The number of seconds between two periodic checkpoints.
Default:3600
dfs.namenode.checkpoint.txns The Secondary NameNode or CheckpointNode will create a checkpoint of the namespace every ‘dfs.namenode.checkpoint.txns’ transactions, regardless of whether ‘dfs.namenode.checkpoint.period’ has expired.
Default:1000000
dfs.namenode.checkpoint.check.period The SecondaryNameNode and CheckpointNode will poll the NameNode every ‘dfs.namenode.checkpoint.check.period’ seconds to query the number of uncheckpointed transactions.
Default:60
dfs.namenode.checkpoint.max-retries The SecondaryNameNode retries failed checkpointing. If the failure occurs while loading fsimage or replaying edits, the number of retries is limited by this variable.
Default:3
dfs.namenode.num.checkpoints.retained The number of image checkpoint files that will be retained by the NameNode and Secondary NameNode in their storage directories. All edit logs necessary to recover an up-to-date namespace from the oldest retained checkpoint will also be retained.
Default:2
dfs.namenode.num.extra.edits.retained The number of extra transactions which should be retained beyond what is minimally necessary for a NN restart. This can be useful for audit purposes or for an HA setup where a remote Standby Node may have been offline for some time and need to have a longer backlog of retained edits in order to start again. Typically each edit is on the order of a few hundred bytes, so the default of 1 million edits should be on the order of hundreds of MBs or low GBs. NOTE: Fewer extra edits may be retained than value specified for this setting if doing so would mean that more segments would be retained than the number configured by dfs.namenode.max.extra.edits.segments.retained.
Default:1000000
dfs.namenode.max.extra.edits.segments.retained The maximum number of extra edit log segments which should be retained beyond what is minimally necessary for a NN restart. When used in conjunction with dfs.namenode.num.extra.edits.retained, this configuration property serves to cap the number of extra edits files to a reasonable value.
Default:10000
dfs.namenode.delegation.key.update-interval The update interval for master key for delegation tokens in the namenode in milliseconds.
Default:86400000
dfs.namenode.delegation.token.max-lifetime The maximum lifetime in milliseconds for which a delegation token is valid.
Default:604800000
dfs.namenode.delegation.token.renew-interval The renewal interval for delegation token in milliseconds.
Default:86400000
dfs.datanode.failed.volumes.tolerated The number of volumes that are allowed to fail before a datanode stops offering service. By default any volume failure will cause a datanode to shutdown.
Default:0
dfs.image.compress Should the dfs image be compressed?
Default:false
dfs.image.compression.codec If the dfs image is compressed, how should they be compressed? This has to be a codec defined in io.compression.codecs.
Default:org.apache.hadoop.io.compress.DefaultCodec
dfs.image.transfer.timeout Socket timeout for image transfer in milliseconds. This timeout and the related dfs.image.transfer.bandwidthPerSec parameter should be configured such that normal image transfer can complete successfully. This timeout prevents client hangs when the sender fails during image transfer. This is socket timeout during image tranfer.
Default:60000
dfs.image.transfer.bandwidthPerSec Maximum bandwidth used for image transfer in bytes per second. This can help keep normal namenode operations responsive during checkpointing. The maximum bandwidth and timeout in dfs.image.transfer.timeout should be set such that normal image transfers can complete successfully. A default value of 0 indicates that throttling is disabled.
Default:0
dfs.image.transfer.chunksize Chunksize in bytes to upload the checkpoint. Chunked streaming is used to avoid internal buffering of contents of image file of huge size.
Default:65536
dfs.namenode.support.allow.format Does HDFS namenode allow itself to be formatted? You may consider setting this to false for any production cluster, to avoid any possibility of formatting a running DFS.
Default:true
dfs.datanode.max.transfer.threads Specifies the maximum number of threads to use for transferring data in and out of the DN.
Default:4096
dfs.datanode.readahead.bytes While reading block files, if the Hadoop native libraries are available, the datanode can use the posix_fadvise system call to explicitly page data into the operating system buffer cache ahead of the current reader’s position. This can improve performance especially when disks are highly contended. This configuration specifies the number of bytes ahead of the current read position which the datanode will attempt to read ahead. This feature may be disabled by configuring this property to 0. If the native libraries are not available, this configuration has no effect.
Default:4193404
dfs.datanode.drop.cache.behind.reads In some workloads, the data read from HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is delivered to the client. This behavior is automatically disabled for workloads which read only short sections of a block (e.g HBase random-IO workloads). This may improve performance for some workloads by freeing buffer cache spage usage for more cacheable data. If the Hadoop native libraries are not available, this configuration has no effect.
Default:false
dfs.datanode.drop.cache.behind.writes In some workloads, the data written to HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is written to disk. This may improve performance for some workloads by freeing buffer cache spage usage for more cacheable data. If the Hadoop native libraries are not available, this configuration has no effect.
Default:false
dfs.datanode.sync.behind.writes If this configuration is enabled, the datanode will instruct the operating system to enqueue all written data to the disk immediately after it is written. This differs from the usual OS policy which may wait for up to 30 seconds before triggering writeback. This may improve performance for some workloads by smoothing the IO profile for data written to disk. If the Hadoop native libraries are not available, this configuration has no effect.
Default:false
dfs.client.failover.max.attempts Expert only. The number of client failover attempts that should be made before the failover is considered failed.
Default:15
dfs.client.failover.sleep.base.millis Expert only. The time to wait, in milliseconds, between failover attempts increases exponentially as a function of the number of attempts made so far, with a random factor of +/- 50%. This option specifies the base value used in the failover calculation. The first failover will retry immediately. The 2nd failover attempt will delay at least dfs.client.failover.sleep.base.millis milliseconds. And so on.
Default:500
dfs.client.failover.sleep.max.millis Expert only. The time to wait, in milliseconds, between failover attempts increases exponentially as a function of the number of attempts made so far, with a random factor of +/- 50%. This option specifies the maximum value to wait between failovers. Specifically, the time between two failover attempts will not exceed +/- 50% of dfs.client.failover.sleep.max.millis milliseconds.
Default:15000
dfs.client.failover.connection.retries Expert only. Indicates the number of retries a failover IPC client will make to establish a server connection.
Default:0
dfs.client.failover.connection.retries.on.timeouts Expert only. The number of retry attempts a failover IPC client will make on socket timeout when establishing a server connection.
Default:0
dfs.client.datanode-restart.timeout Expert only. The time to wait, in seconds, from reception of an datanode shutdown notification for quick restart, until declaring the datanode dead and invoking the normal recovery mechanisms. The notification is sent by a datanode when it is being shutdown using the shutdownDatanode admin command with the upgrade option.
Default:30
dfs.nameservices Comma-separated list of nameservices.
Default:
dfs.nameservice.id The ID of this nameservice. If the nameservice ID is not configured or more than one nameservice is configured for dfs.nameservices it is determined automatically by matching the local node’s address with the configured address.
Default:
dfs.internal.nameservices Comma-separated list of nameservices that belong to this cluster. Datanode will report to all the nameservices in this list. By default this is set to the value of dfs.nameservices.
Default:
dfs.ha.namenodes.EXAMPLENAMESERVICE The prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).
Default:
dfs.ha.namenode.id The ID of this namenode. If the namenode ID is not configured it is determined automatically by matching the local node’s address with the configured address.
Default:
dfs.ha.log-roll.period How often, in seconds, the StandbyNode should ask the active to roll edit logs. Since the StandbyNode only reads from finalized log segments, the StandbyNode will only be as up-to-date as how often the logs are rolled. Note that failover triggers a log roll so the StandbyNode will be up to date before it becomes active.
Default:120
dfs.ha.tail-edits.period How often, in seconds, the StandbyNode should check for new finalized log segments in the shared edits log.
Default:60
dfs.ha.automatic-failover.enabled Whether automatic failover is enabled. See the HDFS High Availability documentation for details on automatic HA configuration.
Default:false
dfs.support.append Does HDFS allow appends to files?
Default:true
dfs.client.use.datanode.hostname Whether clients should use datanode hostnames when connecting to datanodes.
Default:false
dfs.datanode.use.datanode.hostname Whether datanodes should use datanode hostnames when connecting to other datanodes for data transfer.
Default:false
dfs.client.local.interfaces A comma separated list of network interface names to use for data transfer between the client and datanodes. When creating a connection to read from or write to a datanode, the client chooses one of the specified interfaces at random and binds its socket to the IP of that interface. Individual names may be specified as either an interface name (eg “eth0″), a subinterface name (eg “eth0:0″), or an IP address (which may be specified using CIDR notation to match a range of IPs).
Default:
dfs.datanode.shared.file.descriptor.paths A comma-separated list of paths to use when creating file descriptors that will be shared between the DataNode and the DFSClient. Typically we use /dev/shm, so that the file descriptors will not be written to disk. Systems that don’t have /dev/shm will fall back to /tmp by default.
Default:/dev/shm,/tmp
dfs.short.circuit.shared.memory.watcher.interrupt.check.ms The length of time in milliseconds that the short-circuit shared memory watcher will go between checking for java interruptions sent from other threads. This is provided mainly for unit tests.
Default:60000
dfs.namenode.kerberos.internal.spnego.principal
Default:${dfs.web.authentication.kerberos.principal}
dfs.secondary.namenode.kerberos.internal.spnego.principal
Default:${dfs.web.authentication.kerberos.principal}
dfs.namenode.avoid.read.stale.datanode Indicate whether or not to avoid reading from “stale” datanodes whose heartbeat messages have not been received by the namenode for more than a specified time interval. Stale datanodes will be moved to the end of the node list returned for reading. See dfs.namenode.avoid.write.stale.datanode for a similar setting for writes.
Default:false
dfs.namenode.avoid.write.stale.datanode Indicate whether or not to avoid writing to “stale” datanodes whose heartbeat messages have not been received by the namenode for more than a specified time interval. Writes will avoid using stale datanodes unless more than a configured ratio (dfs.namenode.write.stale.datanode.ratio) of datanodes are marked as stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting for reads.
Default:false
dfs.namenode.stale.datanode.interval Default time interval for marking a datanode as “stale”, i.e., if the namenode has not received heartbeat msg from a datanode for more than this time interval, the datanode will be marked and treated as “stale” by default. The stale interval cannot be too small since otherwise this may cause too frequent change of stale states. We thus set a minimum stale interval value (the default value is 3 times of heartbeat interval) and guarantee that the stale interval cannot be less than the minimum value. A stale data node is avoided during lease/block recovery. It can be conditionally avoided for reads (see dfs.namenode.avoid.read.stale.datanode) and for writes (see dfs.namenode.avoid.write.stale.datanode).
Default:30000
dfs.namenode.write.stale.datanode.ratio When the ratio of number stale datanodes to total datanodes marked is greater than this ratio, stop avoiding writing to stale nodes so as to prevent causing hotspots.
Default:0.5f
dfs.namenode.invalidate.work.pct.per.iteration *Note*: Advanced property. Change with caution. This determines the percentage amount of block invalidations (deletes) to do over a single DN heartbeat deletion command. The final deletion count is determined by applying this percentage to the number of live nodes in the system. The resultant number is the number of blocks from the deletion list chosen for proper invalidation over a single heartbeat of a single DN. Value should be a positive, non-zero percentage in float notation (X.Yf), with 1.0f meaning 100%.
Default:0.32f
dfs.namenode.replication.work.multiplier.per.iteration *Note*: Advanced property. Change with caution. This determines the total amount of block transfers to begin in parallel at a DN, for replication, when such a command list is being sent over a DN heartbeat by the NN. The actual number is obtained by multiplying this multiplier with the total number of live nodes in the cluster. The result number is the number of blocks to begin transfers immediately for, per DN heartbeat. This number can be any positive, non-zero integer.
Default:2
nfs.server.port Specify the port number used by Hadoop NFS.
Default:2049
nfs.mountd.port Specify the port number used by Hadoop mount daemon.
Default:4242
nfs.dump.dir This directory is used to temporarily save out-of-order writes before writing to HDFS. For each file, the out-of-order writes are dumped after they are accumulated to exceed certain threshold (e.g., 1MB) in memory. One needs to make sure the directory has enough space.
Default:/tmp/.hdfs-nfs
nfs.rtmax This is the maximum size in bytes of a READ request supported by the NFS gateway. If you change this, make sure you also update the nfs mount’s rsize(add rsize= # of bytes to the mount directive).
Default:1048576
nfs.wtmax This is the maximum size in bytes of a WRITE request supported by the NFS gateway. If you change this, make sure you also update the nfs mount’s wsize(add wsize= # of bytes to the mount directive).
Default:1048576
nfs.keytab.file *Note*: Advanced property. Change with caution. This is the path to the keytab file for the hdfs-nfs gateway. This is required when the cluster is kerberized.
Default:
nfs.kerberos.principal *Note*: Advanced property. Change with caution. This is the name of the kerberos principal. This is required when the cluster is kerberized.It must be of this format: nfs-gateway-user/nfs-gateway-host@kerberos-realm
Default:
nfs.allow.insecure.ports When set to false, client connections originating from unprivileged ports (those above 1023) will be rejected. This is to ensure that clients connecting to this NFS Gateway must have had root privilege on the machine where they’re connecting from.
Default:true
dfs.webhdfs.enabled Enable WebHDFS (REST API) in Namenodes and Datanodes.
Default:true
hadoop.fuse.connection.timeout The minimum number of seconds that we’ll cache libhdfs connection objects in fuse_dfs. Lower values will result in lower memory consumption; higher values may speed up access by avoiding the overhead of creating new connection objects.
Default:300
hadoop.fuse.timer.period The number of seconds between cache expiry checks in fuse_dfs. Lower values will result in fuse_dfs noticing changes to Kerberos ticket caches more quickly.
Default:5
dfs.metrics.percentiles.intervals Comma-delimited set of integers denoting the desired rollover intervals (in seconds) for percentile latency metrics on the Namenode and Datanode. By default, percentile latency metrics are disabled.
Default:
dfs.encrypt.data.transfer Whether or not actual block data that is read/written from/to HDFS should be encrypted on the wire. This only needs to be set on the NN and DNs, clients will deduce this automatically. It is possible to override this setting per connection by specifying custom logic via dfs.trustedchannel.resolver.class.
Default:false
dfs.encrypt.data.transfer.algorithm This value may be set to either “3des” or “rc4″. If nothing is set, then the configured JCE default on the system is used (usually 3DES.) It is widely believed that 3DES is more cryptographically secure, but RC4 is substantially faster. Note that if AES is supported by both the client and server then this encryption algorithm will only be used to initially transfer keys for AES. (See dfs.encrypt.data.transfer.cipher.suites.)
Default:
dfs.encrypt.data.transfer.cipher.suites This value may be either undefined or AES/CTR/NoPadding. If defined, then dfs.encrypt.data.transfer uses the specified cipher suite for data encryption. If not defined, then only the algorithm specified in dfs.encrypt.data.transfer.algorithm is used. By default, the property is not defined.
Default:
dfs.encrypt.data.transfer.cipher.key.bitlength The key bitlength negotiated by dfsclient and datanode for encryption. This value may be set to either 128, 192 or 256.
Default:128
dfs.trustedchannel.resolver.class TrustedChannelResolver is used to determine whether a channel is trusted for plain data transfer. The TrustedChannelResolver is invoked on both client and server side. If the resolver indicates that the channel is trusted, then the data transfer will not be encrypted even if dfs.encrypt.data.transfer is set to true. The default implementation returns false indicating that the channel is not trusted.
Default:
dfs.data.transfer.protection A comma-separated list of SASL protection values used for secured connections to the DataNode when reading or writing block data. Possible values are authentication, integrity and privacy. authentication means authentication only and no integrity or privacy; integrity implies authentication and integrity are enabled; and privacy implies all of authentication, integrity and privacy are enabled. If dfs.encrypt.data.transfer is set to true, then it supersedes the setting for dfs.data.transfer.protection and enforces that all connections must use a specialized encrypted SASL handshake. This property is ignored for connections to a DataNode listening on a privileged port. In this case, it is assumed that the use of a privileged port establishes sufficient trust.
Default:
dfs.data.transfer.saslproperties.resolver.class SaslPropertiesResolver used to resolve the QOP used for a connection to the DataNode when reading or writing block data. If not specified, the value of hadoop.security.saslproperties.resolver.class is used as the default value.
Default:
dfs.datanode.hdfs-blocks-metadata.enabled Boolean which enables backend datanode-side support for the experimental DistributedFileSystem#getFileVBlockStorageLocations API.
Default:false
dfs.client.file-block-storage-locations.num-threads Number of threads used for making parallel RPCs in DistributedFileSystem#getFileBlockStorageLocations().
Default:10
dfs.client.file-block-storage-locations.timeout.millis Timeout (in milliseconds) for the parallel RPCs made in DistributedFileSystem#getFileBlockStorageLocations().
Default:1000
dfs.journalnode.rpc-address The JournalNode RPC server address and port.
Default:0.0.0.0:8485
dfs.journalnode.http-address The address and port the JournalNode HTTP server listens on. If the port is 0 then the server will start on a free port.
Default:0.0.0.0:8480
dfs.journalnode.https-address The address and port the JournalNode HTTPS server listens on. If the port is 0 then the server will start on a free port.
Default:0.0.0.0:8481
dfs.namenode.audit.loggers List of classes implementing audit loggers that will receive audit events. These should be implementations of org.apache.hadoop.hdfs.server.namenode.AuditLogger. The special value “default” can be used to reference the default audit logger, which uses the configured log system. Installing custom audit loggers may affect the performance and stability of the NameNode. Refer to the custom logger’s documentation for more details.
Default:default
dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy. This setting controls how much DN volumes are allowed to differ in terms of bytes of free disk space before they are considered imbalanced. If the free space of all the volumes are within this range of each other, the volumes will be considered balanced and block assignments will be done on a pure round robin basis.
Default:10737418240
dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy. This setting controls what percentage of new block allocations will be sent to volumes with more available disk space than others. This setting should be in the range 0.0 – 1.0, though in practice 0.5 – 1.0, since there should be no reason to prefer that volumes with less available disk space receive more block allocations.
Default:0.75f
dfs.namenode.edits.noeditlogchannelflush Specifies whether to flush edit log file channel. When set, expensive FileChannel#force calls are skipped and synchronous disk writes are enabled instead by opening the edit log file with RandomAccessFile(“rws”) flags. This can significantly improve the performance of edit log writes on the Windows platform. Note that the behavior of the “rws” flags is platform and hardware specific and might not provide the same level of guarantees as FileChannel#force. For example, the write will skip the disk-cache on SAS and SCSI devices while it might not on SATA devices. This is an expert level setting, change with caution.
Default:false
dfs.client.cache.drop.behind.writes Just like dfs.datanode.drop.cache.behind.writes, this setting causes the page cache to be dropped behind HDFS writes, potentially freeing up more memory for other uses. Unlike dfs.datanode.drop.cache.behind.writes, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. If the native libraries are not available to the DataNode, this configuration has no effect.
Default:
dfs.client.cache.drop.behind.reads Just like dfs.datanode.drop.cache.behind.reads, this setting causes the page cache to be dropped behind HDFS reads, potentially freeing up more memory for other uses. Unlike dfs.datanode.drop.cache.behind.reads, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. If the native libraries are not available to the DataNode, this configuration has no effect.
Default:
dfs.client.cache.readahead When using remote reads, this setting causes the datanode to read ahead in the block file using posix_fadvise, potentially decreasing I/O wait times. Unlike dfs.datanode.readahead.bytes, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. When using local reads, this setting determines how much readahead we do in BlockReaderLocal. If the native libraries are not available to the DataNode, this configuration has no effect.
Default:
dfs.namenode.enable.retrycache This enables the retry cache on the namenode. Namenode tracks for non-idempotent requests the corresponding response. If a client retries the request, the response from the retry cache is sent. Such operations are tagged with annotation @AtMostOnce in namenode protocols. It is recommended that this flag be set to true. Setting it to false, will result in clients getting failure responses to retried request. This flag must be enabled in HA setup for transparent fail-overs. The entries in the cache have expiration time configurable using dfs.namenode.retrycache.expirytime.millis.
Default:true
dfs.namenode.retrycache.expirytime.millis The time for which retry cache entries are retained.
Default:600000
dfs.namenode.retrycache.heap.percent This parameter configures the heap size allocated for retry cache (excluding the response cached). This corresponds to approximately 4096 entries for every 64MB of namenode process java heap size. Assuming retry cache entry expiration time (configured using dfs.namenode.retrycache.expirytime.millis) of 10 minutes, this enables retry cache to support 7 operations per second sustained for 10 minutes. As the heap size is increased, the operation rate linearly increases.
Default:0.03f
dfs.client.mmap.enabled If this is set to false, the client won’t attempt to perform memory-mapped reads.
Default:true
dfs.client.mmap.cache.size When zero-copy reads are used, the DFSClient keeps a cache of recently used memory mapped regions. This parameter controls the maximum number of entries that we will keep in that cache. The larger this number is, the more file descriptors we will potentially use for memory-mapped files. mmaped files also use virtual address space. You may need to increase your ulimit virtual address space limits before increasing the client mmap cache size. Note that you can still do zero-copy reads when this size is set to 0.
Default:256
dfs.client.mmap.cache.timeout.ms The minimum length of time that we will keep an mmap entry in the cache between uses. If an entry is in the cache longer than this, and nobody uses it, it will be removed by a background thread.
Default:3600000
dfs.client.mmap.retry.timeout.ms The minimum amount of time that we will wait before retrying a failed mmap operation.
Default:300000
dfs.client.short.circuit.replica.stale.threshold.ms The maximum amount of time that we will consider a short-circuit replica to be valid, if there is no communication from the DataNode. After this time has elapsed, we will re-fetch the short-circuit replica even if it is in the cache.
Default:1800000
dfs.namenode.path.based.cache.block.map.allocation.percent The percentage of the Java heap which we will allocate to the cached blocks map. The cached blocks map is a hash map which uses chained hashing. Smaller maps may be accessed more slowly if the number of cached blocks is large; larger maps will consume more memory.
Default:0.25
dfs.datanode.max.locked.memory The amount of memory in bytes to use for caching of block replicas in memory on the datanode. The datanode’s maximum locked memory soft ulimit (RLIMIT_MEMLOCK) must be set to at least this value, else the datanode will abort on startup. By default, this parameter is set to 0, which disables in-memory caching. If the native libraries are not available to the DataNode, this configuration has no effect.
Default:0
dfs.namenode.list.cache.directives.num.responses This value controls the number of cache directives that the NameNode will send over the wire in response to a listDirectives RPC.
Default:100
dfs.namenode.list.cache.pools.num.responses This value controls the number of cache pools that the NameNode will send over the wire in response to a listPools RPC.
Default:100
dfs.namenode.path.based.cache.refresh.interval.ms The amount of milliseconds between subsequent path cache rescans. Path cache rescans are when we calculate which blocks should be cached, and on what datanodes. By default, this parameter is set to 30 seconds.
Default:30000
dfs.namenode.path.based.cache.retry.interval.ms When the NameNode needs to uncache something that is cached, or cache something that is not cached, it must direct the DataNodes to do so by sending a DNA_CACHE or DNA_UNCACHE command in response to a DataNode heartbeat. This parameter controls how frequently the NameNode will resend these commands.
Default:30000
dfs.datanode.fsdatasetcache.max.threads.per.volume The maximum number of threads per volume to use for caching new data on the datanode. These threads consume both I/O and CPU. This can affect normal datanode operations.
Default:4
dfs.cachereport.intervalMsec Determines cache reporting interval in milliseconds. After this amount of time, the DataNode sends a full report of its cache state to the NameNode. The NameNode uses the cache report to update its map of cached blocks to DataNode locations. This configuration has no effect if in-memory caching has been disabled by setting dfs.datanode.max.locked.memory to 0 (which is the default). If the native libraries are not available to the DataNode, this configuration has no effect.
Default:10000
dfs.namenode.edit.log.autoroll.multiplier.threshold Determines when an active namenode will roll its own edit log. The actual threshold (in number of edits) is determined by multiplying this value by dfs.namenode.checkpoint.txns. This prevents extremely large edit files from accumulating on the active namenode, which can cause timeouts during namenode startup and pose an administrative hassle. This behavior is intended as a failsafe for when the standby or secondary namenode fail to roll the edit log by the normal checkpoint threshold.
Default:2.0
dfs.namenode.edit.log.autoroll.check.interval.ms How often an active namenode will check if it needs to roll its edit log, in milliseconds.
Default:300000
dfs.webhdfs.user.provider.user.pattern Valid pattern for user and group names for webhdfs, it must be a valid java regex.
Default:^[A-Za-z_][A-Za-z0-9._-]*[$]?$
dfs.client.context The name of the DFSClient context that we should use. Clients that share a context share a socket cache and short-circuit cache, among other things. You should only change this if you don’t want to share with another set of threads.
Default:default
dfs.client.read.shortcircuit This configuration parameter turns on short-circuit local reads.
Default:false
dfs.domain.socket.path Optional. This is a path to a UNIX domain socket that will be used for communication between the DataNode and local HDFS clients. If the string “_PORT” is present in this path, it will be replaced by the TCP port of the DataNode.
Default:
dfs.client.read.shortcircuit.skip.checksum If this configuration parameter is set, short-circuit local reads will skip checksums. This is normally not recommended, but it may be useful for special setups. You might consider using this if you are doing your own checksumming outside of HDFS.
Default:false
dfs.client.read.shortcircuit.streams.cache.size The DFSClient maintains a cache of recently opened file descriptors. This parameter controls the size of that cache. Setting this higher will use more file descriptors, but potentially provide better performance on workloads involving lots of seeks.
Default:256
dfs.client.read.shortcircuit.streams.cache.expiry.ms This controls the minimum amount of time file descriptors need to sit in the client cache context before they can be closed for being inactive for too long.
Default:300000
dfs.datanode.shared.file.descriptor.paths Comma separated paths to the directory on which shared memory segments are created. The client and the DataNode exchange information via this shared memory segment. It tries paths in order until creation of shared memory segment succeeds.
Default:/dev/shm,/tmp
dfs.client.use.legacy.blockreader.local Legacy short-circuit reader implementation based on HDFS-2246 is used if this configuration parameter is true. This is for the platforms other than Linux where the new implementation based on HDFS-347 is not available.
Default:false
dfs.block.local-path-access.user Comma separated list of the users allowd to open block files on legacy short-circuit local read.
Default:
dfs.client.domain.socket.data.traffic This control whether we will try to pass normal data traffic over UNIX domain socket rather than over TCP socket on node-local data transfer. This is currently experimental and turned off by default.
Default:false
dfs.namenode.reject-unresolved-dn-topology-mapping If the value is set to true, then namenode will reject datanode registration if the topology mapping for a datanode is not resolved and NULL is returned (script defined by net.topology.script.file.name fails to execute). Otherwise, datanode will be registered and the default rack will be assigned as the topology path. Topology paths are important for data resiliency, since they define fault domains. Thus it may be unwanted behavior to allow datanode registration with the default rack if the resolving topology failed.
Default:false
dfs.client.slow.io.warning.threshold.ms The threshold in milliseconds at which we will log a slow io warning in a dfsclient. By default, this parameter is set to 30000 milliseconds (30 seconds).
Default:30000
dfs.datanode.slow.io.warning.threshold.ms The threshold in milliseconds at which we will log a slow io warning in a datanode. By default, this parameter is set to 300 milliseconds.
Default:300
dfs.namenode.xattrs.enabled Whether support for extended attributes is enabled on the NameNode.
Default:true
dfs.namenode.fs-limits.max-xattrs-per-inode Maximum number of extended attributes per inode.
Default:32
dfs.namenode.fs-limits.max-xattr-size The maximum combined size of the name and value of an extended attribute in bytes.
Default:16384
dfs.namenode.startup.delay.block.deletion.sec The delay in seconds at which we will pause the blocks deletion after Namenode startup. By default it’s disabled. In the case a directory has large number of directories and files are deleted, suggested delay is one hour to give the administrator enough time to notice large number of pending deletion blocks and take corrective action.
Default:0
dfs.namenode.list.encryption.zones.num.responses When listing encryption zones, the maximum number of zones that will be returned in a batch. Fetching the list incrementally in batches improves namenode performance.
Default:100
dfs.namenode.inotify.max.events.per.rpc Maximum number of events that will be sent to an inotify client in a single RPC response. The default value attempts to amortize away the overhead for this RPC while avoiding huge memory requirements for the client and NameNode (1000 events should consume no more than 1 MB.)
Default:1000
dfs.user.home.dir.prefix The directory to prepend to user name to get the user’s home direcotry.
Default:/user
dfs.datanode.cache.revocation.timeout.ms When the DFSClient reads from a block file which the DataNode is caching, the DFSClient can skip verifying checksums. The DataNode will keep the block file in cache until the client is done. If the client takes an unusually long time, though, the DataNode may need to evict the block file from the cache anyway. This value controls how long the DataNode will wait for the client to release a replica that it is reading without checksums.
Default:900000
dfs.datanode.cache.revocation.polling.ms How often the DataNode should poll to see if the clients have stopped using a replica that the DataNode wants to uncache.
Default:500
dfs.datanode.block.id.layout.upgrade.threads The number of threads to use when creating hard links from current to previous blocks during upgrade of a DataNode to block ID-based block layout (see HDFS-6482 for details on the layout).
Default:12
dfs.encryption.key.provider.uri The KeyProvider to use when interacting with encryption keys used when reading and writing to an encryption zone.
Default:
dfs.storage.policy.enabled Allow users to change the storage policy on files and directories.
Default:true
dfs.namenode.legacy-oiv-image.dir Determines where to save the namespace in the old fsimage format during checkpointing by standby NameNode or SecondaryNameNode. Users can dump the contents of the old format fsimage by oiv_legacy command. If the value is not specified, old format fsimage will not be saved in checkpoint.
Default:

Leave a comment