Custom HTTP/HTTPS GET/POST Queries In Ruby

Nice to see the quality information presented in an easy and understanding manner. This is very nice to see this blog and it's really informative for the readers.
Thank you.

UK dissertations

Well, my example is now very old -- I had almost forgotten about it, since I wrote it about 3 years ago. But the reason why I URL-encoded the login and password is because all non-alphanumeric data should be URL encoded to ensure it is sent properly. For example, what if your password has a & in it? That could throw off the whole request. URL encoding has nothing to do with security. If you look at the URL for a Google Search request, next to q= you could -- on some browsers -- see your request URL encoded. It's the type of stuff like %24 for &, or %3D for = or %2F for /. Space is URL-encoded to + or %20, so + is actually %2B. That sort of thing. So since I can't predict the content of people's logins, or especially, their passwords, we should URL encode them to ensure they are sent correctly and our request is well-formed. Also, if people are doing web scraping or want to mirror as much as possible a request that works in a web browser, I highly recommend Charles Proxy from www.charlesproxy.com - written in fast Java code, it runs on Mac, Linux or Windows and works equally well for HTTP and HTTPS requests, regardless of browser. It even works with applications that don't support proxy servers, though the use of reverse proxying and either host files or some other form of DNS redirection. For instance, it can be configured to listen on localhost:81 and pass requests on to example.org, and as long as your hosts file says 127.0.0.1 example.org, your application will be none the wiser. It's also great for normal web development, as you can see a timeline of your requests, with or without bandwidth throttling, and inspect cookies, headers, and AJAX responses. It even has excellent support for Flash inspection. Plus you can create your own requests or craft them by editing existing ones. An amazing tool, and I'm constantly finding new ways to use it. Perfect for all web projects that requires code like this. (And no, I'm not getting paid by the developer to recommend it. It's just that good. And did I mention its Mac-compatibility?)

I build a dead-simple post method from parts of your snippet, hope you can use it ;) http://pragmatig.wordpress.com/2009/01/23/simple-http-post-without-nethttp-complexity/

Ahh, I just spent the last ... oh I don't know how long - lets say three hours - trying to get this code to work. I finally did, but the problem was I needed to specify a user agent! So here's my code snippet in full - and get2/post2 is interchangable with get/post, the difference is that the 2 is safer, it won't toss any Net:: Errors. Also notice that I URL encoded the username and password. require "net/http" require "net/https" require "erb" require "singleton" PANEL = 'panel.dreamhost.com' PATH = '/index.cgi' USERAGENT = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1' class Panel include Singleton def Panel.login(webid, password) unless @headers.nil? raise ScriptError, "You should already be logged in!" end @http = Net::HTTP.new(PANEL, 443) @http.use_ssl = true # GET request -> so the host can set cookies resp, data = @http.get2(PATH, {'User-Agent' => USERAGENT}) cookie = resp.response['set-cookie'].split('; ')[0] # POST request -> logging in data = "Nscmd=Nlogin&username=#{ERB::Util.url_encode(webid)}&password=#{ERB::Util.url_encode(password)}&submit=Log%20In" @headers = { 'Cookie' => cookie, 'Referer' => 'https://'+PANEL+PATH, 'Content-Type' => 'application/x-www-form-urlencoded', 'User-Agent' => USERAGENT } puts "headers = " + @headers.inspect resp, data = @http.post2(PATH, data, @headers) # Output on the screen -> we should get either a 302 redirect (after a successful login) or an error page puts 'Code = ' + resp.code puts 'Message = ' + resp.message resp.each {|key, val| puts key + ' = ' + val} puts data end end If you curious, I'm scraping my own API for DreamHost's Control Panel, so I don't have to labourously open my web browser every time I need to add or change something. (Yeah, I'm that lazy.) Anyway, its unfinished, I'll be refactoring the code next - the Panel.login method was just for testing.