HDFS TO AWS S3

Step 1 :-

Login to Cloudera manager

Go to services and Click on hdfs

Go to configuration and click on view and edit roles

Click on service-wide configuration

Click on advance

Step 2 :-

Add below details in Cluster-wide Configuration Safety Value for core-site.xml

<property>

<name>fs.s3n.awsAccessKeyId</name>

<value>XXXXXXXXXXXXXXXXXXX</value>

</property>

<property>

<name>fs.s3n.awsSecretAccessKey</name>

<value>XXXXXXXXXXXXXXXXXXXXXXX</value>

</property>

Step 3 :-

Save configuration. 

Click on Action and Deploy Configuration 

Restart hdfs once 

Step 4:-

[nitin@nitin-ubuntu ~]# sudo -u hdfs hadoop distcp s3n://big-store/getting_price.sh hdfs://<CLSUTER1>:8020/user/nitin/

[nitin@nitin-ubuntu ~]# sudo -u hdfs hadoop distcp s3n://big-store/getting_price.sh hdfs://10.10.10.216:8020/user/nitin/