I thought that I found a bug in the TPL, but it looks like it's working (more or less) by design. Basically, when a task is completed, all awaiting tasks will be notified on that, which is pretty much what you would expect. What isn’t usually expected is that those tasks can interfere with one another. Consider the following code:

public static async Task LetUserKnow(Task parent)
{
    await parent;
    Console.WriteLine("Task Completed, press Enter key to start new task");
    Console.ReadLine();
    SpawnNewTask();
}

public static async Task WarnOnTimeout(Task parent)
{
    var delay = Task.Delay(2500);
    if (await Task.WhenAny(delay, parent) == delay)
    {
        Console.WriteLine("Timed out!");
    }
}

We have two tasks, which accept a parent task and do something with it. What do you think will happen when we run the following code?

var task = Task.Run(() => Thread.Sleep(1000));
LetUserKnow(task);
WarnOnTimeout(task);

Unless you are very quick on the draw, running this code will result in a timeout message, but how? We know that we have a much shorter duration for the task than the timeout, so what is going on?

Well, effectively, what is going on is that the parent task has a list of children that it will notify, and by default, it will do so synchronously and sequentially. If a child task blocks for whatever reason (for example, it might be processing a lot of work), the other children of the parent task will not be notified.

If there is a timeout setup, it will be triggered, even though the parent task was already completed. It took us a lot of time to figure out the repro in this issue, and we were certain that this is some sort of race condition in the TPL. I had a blog post talking all about it, but the Microsoft team is fast enough that they were able to literally answer my issue before I had the time to complete my blog post. That is really impressive.

I should note that the suggestion, using RunContinuationsAsynchronously, works quite well for creating a new Task or using TaskCompletionSource, but there is no way to specify that when you are using Task.Run. What is worse for us is that since this is not the default (for perfectly good performance reasons, by the way), this means that any code that we call into might trigger this. I would have much rather be able to specify than when waiting for the task, rather than when creating it.