Managing Distributed System Locks With Azure Storage

Distributed systems have been there for a while now and there are well-known patterns already established when designing them. Today, we will discuss one of the popular patterns: "locks."

Simply put, locks are how processes gain exclusive access to a resource to perform a certain action. For example, imagine there are a bunch of Blobs in a storage account, and you need one instance of your service to process each blob to avoid duplicate processing. The way to do it would be to acquire a lock on the blob, complete processing, and release it.  However, a potential issue arises if a process fails before releasing the lock, either because the process died or due to a network partition, leaving the resource locked indefinitely. This can lead to deadlocks and resource contention. 

To prevent deadlocks, one strategy that can be employed is to use timeouts or leased-based locks.

Timeout Lock

• In this case, there is a predefined timeout the process requests the lock for. If the lock is not released before the timeout, the system ensures the lock is eventually released.

Lease Lock

• For lease-based locks, a renew lease API is provided alongside the timeout mechanism. The process holding the lock must call this API before the lease expires to maintain exclusive access to the resource. If the process fails to renew the lease in time, the lock is automatically released, allowing other processes to acquire it.

Pros and Cons of Timeout and Lease-Based Locks

Both the above strategies are a way to quickly recover from process failures or network partitions in distributed systems. 

Lease Lock Strategy With Azure Storage

Let's look at how to use the Lease Lock strategy with Azure Storage. This also covers the Timeout lock strategy.

Step 1: Import the Storage Blob Nuget

"12.23.0" is the latest version at the time of authoring this article. The latest versions can be found at Azure Storage Blobs.

<ItemGroup>
  <PackageReference Include="Azure.Storage.Blobs" Version="12.23.0" />
</ItemGroup>

Step 2: Acquire the Lease

Below is the code to acquire the lease.

    public async Task<string> TryAcquireLeaseAsync(string blobName, TimeSpan durationInSeconds, string leaseId = default)
    {
      	BlobContainerClient blobContainerClient = new BlobContainerClient(new Uri($"https://{storageName}.blob.core.windows.net/processors"), tokenCredential, blobClientOptions);
        BlobLeaseClient blobLeaseClient = blobContainerClient.GetBlobClient(blobName).GetBlobLeaseClient(leaseId);

        try
        {
            BlobLease lease = await blobLeaseClient.AcquireAsync(durationInSeconds).ConfigureAwait(false);
            return lease.LeaseId;
        }
        catch (RequestFailedException ex) when (ex.Status == 409)
        {
            return default;
        }
    }

1. First, we create a Blob Container Client and retrieve the Blob Client for the specific blob we want to acquire a lease on.
2. Second, the "Acquire Async" method tries to acquire the lease for a specific duration. If the acquisition was successful, a lease Id is returned if not a 409 (Status code for conflict) is thrown.
3. The "Acquire Async" is the key method here. The rest of the code can be tailored/edited as per your needs.

Step 3: Renew the Lease

• "Renew Async" is the method in the Storage .NET SDK used for renewing the lease.
• If the renewal is unsuccessful an exception is thrown along with the reason for the cause of failure.

public async Task ReleaseLeaseAsync(string blobName, string leaseId)
{
  BlobLeaseClient blobLeaseClient = this.blobContainerClient.GetBlobClient(blobName).GetBlobLeaseClient(leaseId);

  await blobLeaseClient.RenewAsync().ConfigureAwait(false);
}

Step 4: Orchestrate the Acquire and Renew Lease Methods

• Initially, we call the "Try Acquire Lease Async" to fetch the lease identifier from Step 2. Once it is successful, a background task is kicked off that calls the "Renew Lease Async" from Step 3 every X seconds. Just make sure there is enough time between the timeout and when the renew lease method is called.

string leaseId = await this.blobReadProcessor.TryAcquireLeaseAsync(blobName, TimeSpan.FromSeconds(60)).ConfigureAwait(false);

Task leaseRenwerTask = this.taskFactory.StartNew(
        async () =>
        {
            while (leaseId != default && !cancellationToken.IsCancellationRequested)
            {
                await Task.Delay(renewLeaseMillis).ConfigureAwait(false);
                await this.blobReadProcessor.RenewLeaseAsync(blobName, leaseId).ConfigureAwait(false);
            }
        },
        CancellationToken.None,
        TaskCreationOptions.LongRunning,
        TaskScheduler.Default);

• The cancellation token is used to gracefully stop the lease renewal task when it's no longer needed.

Step 5: Cancel the Lease Renewal

• When the "Cancel Async" method is called, the "IsCancellationRequested" in Step 4 becomes true, because of which we no longer enter the while loop and request for lease renewal.

await cancellationTokenSource.CancelAsync().ConfigureAwait(false);
await leaseRenwerTask.WaitAsync(Timeout.InfiniteTimeSpan).ConfigureAwait(false);

Step 6: Release the Lease

Finally, to release the lease just call the "Release Async" method.

public async Task ReleaseLeaseAsync(string blobName, string leaseId)
{
  BlobLeaseClient blobLeaseClient = this.blobContainerClient.GetBlobClient(blobName).GetBlobLeaseClient(leaseId);

  await blobLeaseClient.ReleaseAsync().ConfigureAwait(false);
}

Conclusion

Locks are among the fundamental patterns in distributed systems to gain exclusive access to resources. It is necessary to keep the pitfalls in mind while dealing with them for the smooth running of operations. By using Azure Storage, we can implement these efficient locking mechanisms that can prevent indefinite blocking and, at the same time, provide elasticity in how the locks are maintained.