WebRequest, or rather HttpWebRequest has the annoying behaviour or throwing a WebException when the server returns 404 ‘not found’ status, or in fact any unexpected status number. It would be much better if it didn’t do this and simply allowed the application to decide what it should do on different status codes. At the very least there should be some way of turning this behaviour on or off. In fact it would be nice if the whole WebRequest class wasn’t a monolithic black box, but a toolbox of components that allowed you to tailor an HTTP client to your requirements. I was a little surprised when I did some Googling earlier and couldn’t find a nice open source alternative to WebRequest; it’s the sort of thing that the community is usually quite good at coding around. Oh well, I’ll add that to my ever growing list of potential future GitHub projects (that will never happen).

My quick and dirty fix for this problem was an extension method that catches WebException, checks if the type of the exception is a protocol exception – this seems to be the status for status code exceptions, and then returns the response from the exception’s Response property. It’s horrible, but it seems to work:

public static HttpWebResponse GetHttpResponse(this HttpWebRequest request)
{
    HttpWebResponse response = null;

    try
    {
        response = (HttpWebResponse)request.GetResponse();
    }
    catch (WebException exception)
    {
        if (exception.Status == WebExceptionStatus.ProtocolError)
        {
            response = (HttpWebResponse) exception.Response;
        }
        else
        {
            throw;
        }
    }

    return response;
}

It conveniently returns an HttpWebResponse instead of a WebResponse.

You could use it like this …

var response = request.GetHttpResponse();
if (response.StatusCode != HttpStatusCode.NotFound)
{
    // handle appropriately
}

Of course, if you want to handle the response asynchronously, you’ll have to write an extension method for EndGetResponse as well.

… or, if you’re using .NET 4.5, you could use HttpClient. It wraps WebRequest internally, but it does return status codes rather than throwing it seems.