Over a million developers have joined DZone.

Edge Testing your Hadoop Environment

DZone's Guide to

Edge Testing your Hadoop Environment

An in-depth guide to securing your Hadoop environment by testing from the edge.

· Security Zone ·
Free Resource

Learning by doing is more effective than learning by watching - that’s why Codebashing offers a hands-on interactive training platform in 10 major programming languages. Learn more about AppSec training for enterprise developers.

You can run the attack library for OSX or Linux from an edge node or from outside the cluster. I ran against mine from my OSX laptop against my cluster that I had network access to. You should try to scan from inside your network, from an edge node and from a remote site on the Internet.

You will need Python 2.7 or Python 3.x installed first:

 git clone git@github.com:CERT-W/hadoop-attack-library.git

pip install requests lxml 

You may need root or sudo access to install on your machine. One of the scanners hits the WebHDFS link that you may have seen a warning about.

 python hdfsbrowser.py timscluster 

Beginning to test services accessibility using default ports ...

Testing service WebHDFS

[+] Service WebHDFS is available

Testing service HttpFS

[-] Exception during requesting the service

[+] Sucessfully retrieved 1 services

drwxrwxrwx hdfs:hdfs 2017-01-15T05:50:27+0000 /

drwxrwxrwx yarn:hadoop 2017-01-11T19:25:26+0000 app-logs /app-logs

drwxrwxrwx hdfs:hdfs 2016-12-21T23:12:40+0000 apps /apps

drwxrwxrwx yarn:hadoop 2016-09-15T21:02:30+0000 ats /ats

drwxrwxrwx root:hdfs 2016-12-21T23:08:34+0000 avroresults /avroresults

drwxrwxrwx hdfs:hdfs 2016-12-13T03:42:55+0000 banking /banking

To see how available your Hadoop configurations are, you can use Hadoop Snooper. This is under: Tools\Techniques\and\Procedures\Getting\the\target\environment\configuration.

python hadoopsnooper.py timscluster -o test

Specified destination path does not exist, do you want to create it ? [y/N]y

[+] Creating configuration directory

[+] core-site.xml successfully created

[+] mapred-site.xml successfully created

[+] yarn-site.xml successfully created

This downloads all those configuration files to a directory named test.

These were not the full configuration files, but they pointed to correct internal servers and give an attacker more information.

Another scan worth running is SQLMap. This tool will let you check various SQL tools in the system. SQLMap requires Python 2.6 or 2.7.

➜ projects git clone https://github.com/sqlmapproject/sqlmap.git sqlmap-dev

Cloning into 'sqlmap-dev'...

remote: Counting objects: 55560, done.

remote: Compressing objects: 100% (41/41), done.

remote: Total 55560 (delta 22), reused 0 (delta 0), pack-reused 55519

Receiving objects: 100% (55560/55560), 47.25 MiB | 2.28 MiB/s, done.

Resolving deltas: 100% (42960/42960), done.

Checking connectivity... done.

➜ projects python sqlmap.py --update

➜ projects cd sqlmap-dev

➜ sqlmap-dev git:(master) python sqlmap.py --update



___ ___[.]_____ ___ ___ {}

|_ -| . [)] | .'| . |

|___|_ [']_|_|_|__,| _|

|_|V |_| http://sqlmap.org

[!] legal disclaimer: Usage of sqlmap for attacking targets without prior mutual consent is illegal. It is the end user's responsibility to obey all applicable local, state and federal laws. Developers assume no liability and are not responsible for any misuse or damage caused by this program

[*] starting at 16:49:13

[16:49:13] [INFO] updating sqlmap to the latest development version from the GitHub repository

[16:49:13] [INFO] update in progress .

[16:49:14] [INFO] already at the latest revision 'f542e82'

[*] shutting down at 16:49:14

SQLMap is awesome for testing any RDBMS, the more you know about it the better to stress it.  MySQL and PostgreSQL are often used to host metadata for Hadoop and can be tested with SQLMap. Remember Bobby Tables.

Find out how CxSAST can help you scan uncompiled and unbuilt code while identifying hundreds of security vulnerabilities in the most prevalent coding languages.

hadoop ,security ,sqlmap ,sql ,nosql

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}