Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Race Condition When Modifying the HttpRequestMessageProperty in an Outgoing WCF Message

DZone's Guide to

Race Condition When Modifying the HttpRequestMessageProperty in an Outgoing WCF Message

·
Free Resource

I’ve stumbled upon a fairly obscure race condition in WCF’s implementation of the HTTP transport, and just wanted to quickly share it with you should you ever encounter it.

I was using a simple publish/subscribe router similar to the one I’ve blogged about two years ago to automatically publish notifications to a list of subscribers. The router is a WCF service that implements a generic interface that takes a Message and creates multiple copies of it to send to all registered subscribers.

Here’s my original take on the dispatch code: [exception-handling removed for clarity]

foreach (string subscriber in subscribers)
{
Message toSend = buffer.CreateMessage();
IGenericOneWayContract proxy =
ChannelFactory<IGenericOneWayContract>.CreateChannel(
Program.Binding, new EndpointAddress(subscriber));
proxy.Action(toSend);
((ICommunicationObject)proxy).Close();
toSend.Close();
}

…now, seeing that the messages are copied and the send operation is stateless, there’s no reason why I shouldn’t be able to parallelize these operations and make sure subscribers get the notification messages more promptly:

Parallel.ForEach(
subscribers,
subscriber =>
{
Message toSend = buffer.CreateMessage();
IGenericOneWayContract proxy =
ChannelFactory<IGenericOneWayContract>.CreateChannel(
Program.Binding, new EndpointAddress(subscriber));
proxy.Action(toSend);
((ICommunicationObject) proxy).Close();
toSend.Close();
});

But unfortunately, the latter version of the code fails with the following exception, that only occurs once in several hundreds of messages on my machine: [removed parameters from stack trace for clarity]

System.NullReferenceException: Object reference not set to an instance of an object.

Server stack trace:
at System.ServiceModel.Channels.HttpRequestMessageProperty.get_Headers()
at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.PrepareHttpSend…
at System.ServiceModel.Channels.HttpOutput.Send…
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest…
at System.ServiceModel.Channels.RequestChannel.Request…
at System.ServiceModel.Dispatcher.RequestChannelBinder.Send…
at System.ServiceModel.Channels.ServiceChannel.Call…
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService…
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke…

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage…
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke…
at WCFHttpMessageProperties.IGenericOneWayContract.Action…
at WCFHttpMessageProperties.PubSubRouter.<>c__DisplayClass1.<Action>b__0…
at WCFHttpMessageProperties.Parallel.<>c__DisplayClass3`1.<ForEach>b__0…

After looking at the code with Reflector, it seems that during the PrepareHttpSend method that we see in the stack trace, there’s a phase where the message properties (including the HttpRequestMessageProperty) are refreshed, e.g. with HTTP header information. Why, however, should there be a problem with the parallel version of the code if I’m delivering separate copies of the message to my subscribers?

Well, from looking at the implementation of message cloning with Reflector, it appears that message properties are not cloned unless they implement the IMessageProperty interface (see MessageProperties.CreateCopyOfPropertyValue). The HttpRequestMessageProperty class does not implement this interface, and therefore “cloning” it consists of simply returning the original value.

Now the problem should be evident – we have multiple threads that are trying to modify the same HttpRequestMessageProperty instance, with its HTTP header collection and whatnot. There’s a fairly rare race condition because most of the time the operations on the HTTP headers don’t collide, and happen to leave the message property in a consistent state. When they do collide, we get the above exception.

There’s actually another variety of this problem that I’ve seen in the wild, but haven’t been able to reproduce on my machine – it’s when the message is eventually sent, but contains duplicate HTTP headers! This is obviously rather difficult to diagnose because the message is successfully sent with the duplicate headers but the subscriber refuses to receive it! The same fix was sufficient for both of the issues.

So, what’s the fix? In this case, short of writing a manual serialization mechanism for message properties, I didn’t see many options other than a workaround. The workaround I found is to remove the HttpRequestMessageProperty from the message before creating copies – this will cause the transport channel (HttpOutput) to create a new instance of the property and attach this fresh copy to the message.

In other words, the complete and working version of the code is something along the following lines:

message.Properties.Remove(HttpRequestMessageProperty.Name);
MessageBuffer buffer = message.CreateBufferedCopy(int.MaxValue);
string[] subscribers = …;
Parallel.ForEach(
subscribers,
subscriber =>
{
Message toSend = buffer.CreateMessage();
IGenericOneWayContract proxy =
ChannelFactory<IGenericOneWayContract>.CreateChannel(
Program.Binding, new EndpointAddress(subscriber));
proxy.Action(toSend);
((ICommunicationObject) proxy).Close();
toSend.Close();
});
buffer.Close();

I thought it would be interesting to test this behavior with WCF 4.0. On Visual Studio 2010 Beta 2, I was able to reproduce the duplicate HTTP headers problem, with the following exception, although the problem is admittedly rare:

System.ServiceModel.ProtocolException: Content Type application/soap+xml; charset=utf-8,application/soap+xml; charset=utf-8 was not supported by service http://localhost:8993/Client_6.  The client and service bindings may be mismatched. ---> System.Net.WebException: The remote server returned an error: (415) Cannot process the message because the content type 'application/soap+xml; charset=utf-8,application/soap+xml; charset=utf-8' was not the expected type 'application/soap+xml; charset=utf-8'..
at System.Net.HttpWebRequest.GetResponse…
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply…
--- End of inner exception stack trace ---

Server stack trace:
at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException…
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply… at System.ServiceModel.Channels.RequestChannel.Request…
at System.ServiceModel.Dispatcher.RequestChannelBinder.Send…
at System.ServiceModel.Channels.ServiceChannel.Call…
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService…
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke…

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage…
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke…
at WCFHttpMessageProperties.IGenericOneWayContract.Action…
at WCFHttpMessageProperties.PubSubRouter.<>c__DisplayClass2.<Action>b__1…
at WCFHttpMessageProperties.Parallel.<>c__DisplayClass4`1.<ForEach>b__1…

Finally, in another test run I encountered a third variety of this exception, that I wasn’t able to see with WCF 3.0. It’s due to an internal implementation change, but the underlying problem is still the same race condition as before: [edited for clarity]

System.ArgumentException: Item has already been added. Key in dictionary: 'Content-Type'  Key being added: 'Content-Type'

Server stack trace:
at System.Collections.Hashtable.Insert…
at System.Collections.Hashtable.Add…
at System.Collections.Specialized.NameObjectCollectionBase.BaseAdd…
at System.Collections.Specialized.NameValueCollection.Add…
at System.Net.WebHeaderCollection.Add…
at System.Collections.Specialized.NameValueCollection.Add…
at System.ServiceModel.Channels.HttpRequestContext.ListenerHttpContext.System.ServiceModel.Channels.HttpRequestMessageProperty.IHttpHeaderProvider.CopyHeaders…
at System.ServiceModel.Channels.HttpRequestMessageProperty.get_Headers…
at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.PrepareHttpSend…
at System.ServiceModel.Channels.HttpOutput.Send…
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest…
at System.ServiceModel.Channels.RequestChannel.Request…
at System.ServiceModel.Dispatcher.RequestChannelBinder.Send…
at System.ServiceModel.Channels.ServiceChannel.Call…
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService…
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke…

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage…
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke…
at WCFHttpMessageProperties.IGenericOneWayContract.Action…
at WCFHttpMessageProperties.PubSubRouter.<>c__DisplayClass2.<Action>b__1…
at WCFHttpMessageProperties.Parallel.<>c__DisplayClass4`1.<ForEach>b__1…

This last exception even seems to hint that someone at Microsoft introduced an additional sanity check so that messages with duplicate headers do not even leave the HTTP transport, but this sanity check doesn’t work (see the previous exception) and doesn’t fix the root cause of this issue, which is the fact that WCF message properties are not properly cloned when the message is cloned.

Fortunately, the same workaround fixes the problem in WCF 4.0 as well.

 

Topics:

Published at DZone with permission of Sasha Goldshtein, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}