Yarn Memory and CPU allocation

1) Add below lines in yarn-site.xml

Note :- Calculate RAM per node. follow below links

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html    OR

http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_ig_yarn_tuning.html


[hadoop@adp032 hadoop]$ cat /usr/local/hadoop/hadoop/etc/hadoop/yarn-site.xml

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>3072</value>
</property>


<property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>256</value>
</property>

<property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>3072</value>
</property>


<property>
        <name>yarn.app.mapreduce.am.resource.mb</name>
        <value>512</value>
</property>

<property>
        <name>yarn.app.mapreduce.am.command-opts</name>
        <value>-Xmx409m</value>
</property>

2) Add below mapred-site.xml

[hadoop@adp032 hadoop]$ cat /usr/local/hadoop/hadoop/etc/hadoop/mapred-site.xml

<property>
        <name>mapreduce.map.memory.mb</name>
                <value>256</value>
                </property>

<property>
        <name>mapreduce.map.java.opts</name>
                <value>-Xmx201m</value>
                </property>

<property>
        <name>mapreduce.reduce.memory.mb</name>
                <value>512</value>
                </property>


<property>
        <name>mapreduce.reduce.java.opts</name>
                <value>-Xmx410m</value>
</property>

3) Restart resourcemanager rand nodemanager on all nodes.

[hadoop@adp032 hadoop]$  yarn-daemon.sh stop resourcemanager;yarn-daemon.sh start resourcemanager

[hadoop@adp032 hadoop]$ yarn-daemon.sh stop nodemanager;yarn-daemon.sh start nodemanager

CPU Allocations :

4) Add below line in yarn-site.xml

[hadoop@adp032 hadoop]$ cat /usr/local/hadoop/hadoop/etc/hadoop/yarn-site.xml

<property>

        <name>yarn.nodemanager.resource.cpu-vcores</name>

        <value>1</value>

</property>

<property>

        <name>mapreduce.map.cpu.vcores</name>

        <value>1</value>

</property>

<property>

         <name>mapreduce.reduce.cpu.vcores</name>

         <value>1</value>

</property>

5)  Perform step 3 . Restart demons.

6) Verify changes on resourcemanager webui. 

7)  You may face below error

Diagnostics: Container [pid=24068,containerID=container_1445068083523_0001_02_000001] is running beyond virtual memory limits. Current usage: 106.7 MB of 512 MB physical memory used; 1.1 GB of 1.0 GB virtual memory used. Killing container.

This is happening on Centos/RHEL 6 due to its aggressive allocation of virtual memory.


Add following property in yarn-site.xml

<property>
   <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

Ref : http://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits