Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Invisible Race Conditions: The Cache Has Poisoned Us

DZone's Guide to

Invisible Race Conditions: The Cache Has Poisoned Us

Some race conditions may seem to be invisible. In this post, we take a look at a poisoned cache that caused some headaches in a location that was previously debugged.

· Web Dev Zone
Free Resource

Get the senior executive’s handbook of important trends, tips, and strategies to compete and win in the digital economy.

We got a memory corruption error one day that was quite interesting. It was in a place where we had previously fixed a memory corruption error and was, at a glance, quite impossible.

The code would checkout an item from the cache and increment its ref count, which will keep it alive for as long as we were using it. But something made it fail, and quite horribly, too. We finally tracked the code down to this piece of code, which is run when we update the cache:

_items.AddOrUpdate(url, httpCacheItem, (s, oldItem) =>
{
    oldItem.ReleaseRef();
    return httpCacheItem;
});

When the ref count goes to zero, we’ll release the memory, and _items is a Concurrent Dictionary.

Do you see the error?

The AddOrUpdate method will call updateValueFactory when it needs to update a value, but it makes no promises with regards to its atomicity. In other words, if you have two threads calling this method, the update lambda will be called twice with the same item, resulting in the early release of the value and hence memory corruption.

This can be seen here:

while (true)
{
    TValue oldValue;
    if (TryGetValueInternal(key, hashcode, out oldValue))
    {
        // key exists, try to update
        TValue newValue = updateValueFactory(key, oldValue, factoryArgument);
        if (TryUpdateInternal(key, hashcode, newValue, oldValue))
        {
            return newValue;
        }
    }
    else
    {
        // key doesn't exist, try to add
        TValue resultingValue;
        if (TryAddInternal(key, hashcode, addValueFactory(key, factoryArgument), false, true, out resultingValue))
        {
            return resultingValue;
        }
  }
}

As you can see, we are looking at a loop that may be executed several times, as such updateValueFactory can be called several times, and the only guarantee we have is that after the method has returned, the last value we were called with was the value that was in the cache and we replaced.

Here is the fix:

HttpCacheItem old = null;
_items.AddOrUpdate(url, httpCacheItem, (s, oldItem) =>
{
    old = oldItem;
    return httpCacheItem;
});
old?.ReleaseRef();

That was quite hard to figure out because, at a glance, this looks just fine.

Read this guide to learn everything you need to know about RPA, and how it can help you manage and automate your processes.

Topics:
cache ,race conditions ,web dev

Published at DZone with permission of Oren Eini, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}