Over a million developers have joined DZone.

An S3 File Bucket Downloader Written in Ruby

DZone's Guide to

An S3 File Bucket Downloader Written in Ruby

· Cloud Zone ·
Free Resource

Discover a centralized approach to monitor your virtual infrastructure, on-premise IT environment, and cloud infrastructure – all on a single platform.

Today I wanted to download files from a website that I happened to find out that stored all files in S3. By accessing the website root, I realized that it was just the response of a S3 ListBucket API call. For instance:

    <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">  

In order to download all files more quickly, I wrote the following Ruby program that downloads all files from this website, and I hope it can be useful for others:

    require 'net/http'  
    require 'rexml/document'  
    baseurl = 'foo.com'  
    # get the XML data as a string  
    xml_data = Net::HTTP.get_response(URI.parse("http://" + baseurl)).body  
    # extract event information  
    doc = REXML::Document.new(xml_data)  
    titles = []  
    links = []  
    Net::HTTP.start(baseurl) do |http|  
      doc.elements.each('ListBucketResult/Contents/Key') do |ele|  
        puts "Downloading " + ele.text  
        resp = http.get("/" + ele.text)  
        open("images/" + ele.text.gsub("/", "_") + ".jpg", "wb") { |file|  
    puts "Done"  

Learn how to auto-discover your containers and monitor their performance, capture Docker host and container metrics to allocate host resources, and provision containers.


Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}