ASP.NET Core Data Protection for Service Fabric with Kestrel and WebListener

In ASP.NET 1.x - 4.x, if you deployed your application to a Web farm, you had to ensure that the configuration files on each server shared the same value for validationKey and decryptionKey, which were used for hashing and decryption respectively. In ASP.NET Core this is accomplished via the data protection stack which was designed to address many of the shortcomings of the old cryptographic stack. The new API provides a simple, easy to use mechanism for data encryption, decryption, key management, and rotation. The data protection system ships with several in-box key storage providers: File system, Registry, AzureStorage, and Redis.

Since we are working with low-latency microservices at massive scale via Azure Service Fabric, in this article we’ll describe an approach to create a custom ASP.NET Core data protection key repository using Service Fabric’s built in Reliable Collections, which are Replicated, Persisted, Asynchronous, and Transactional.

Previous readers will note we’ve covered how to integrate ASP.Net Core and Kestrel into Service Fabric, moreover how to create Service Fabric microservices in the new .Net Core xproj structure (soon to be superseded with VS 2017), therefore we'll jump straight into building the AspNetCore.DataProtection.ServiceFabric microservice. To test everything out, we'll create a sample ASP.Net Core Web API microservice and finally for completeness integrate WebListener, a Windows-only web server.

To begin, we create a new stateful Service Fabric microservice called DataProtectionService:

using Microsoft.ServiceFabric.Data;
using Microsoft.ServiceFabric.Data.Collections;
using Microsoft.ServiceFabric.Services.Communication.Runtime;
using Microsoft.ServiceFabric.Services.Remoting.Runtime;
using Microsoft.ServiceFabric.Services.Runtime;
using System;
using System.Collections.Generic;
using System.Fabric;
using System.Threading;
using System.Threading.Tasks;
using System.Xml.Linq;

namespace AspNetCore.DataProtection.ServiceFabric
{
    internal sealed class DataProtectionService : StatefulService, IDataProtectionService
    {
        public DataProtectionService(StatefulServiceContext context, IReliableStateManager stateManager) : base(context, stateManager as IReliableStateManagerReplica)
        {

        }

        protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
        {
            return new[]
            {
                new ServiceReplicaListener(context => this.CreateServiceRemotingListener(context))
            };
        }

        public async Task<List<XElement>> GetAllDataProtectionElements()
        {
            var elements = new List<XElement>();

            var dictionary = await this.StateManager.GetOrAddAsync<IReliableDictionary<Guid, XElement>>("AspNetCore.DataProtection");
            using (var tx = this.StateManager.CreateTransaction())
            {
                var enumerable = await dictionary.CreateEnumerableAsync(tx);
                var enumerator = enumerable.GetAsyncEnumerator();
                var token = new CancellationToken();

                while (await enumerator.MoveNextAsync(token))
                {
                    elements.Add(enumerator.Current.Value);
                }
            }

            return elements;
        }

        public async Task<XElement> AddDataProtectionElement(XElement element)
        {
            Guid id = Guid.Parse(element.Attribute("id").Value);

            var dictionary = await this.StateManager.GetOrAddAsync<IReliableDictionary<Guid, XElement>>("AspNetCore.DataProtection");
            using (var tx = this.StateManager.CreateTransaction())
            {
                var result = await dictionary.GetOrAddAsync(tx, id, element);
                await tx.CommitAsync();

                return result;
            }
        }
    }
}

Congratulations you’ve just implemented a custom key storage provider using a Service Fabric Reliable Dictionary! To integrate with ASP.Net Core Data Protection API, we need to also create a ServiceFabricXmlRepository class which implements an IXmlRepository. In a new stateless microservice called ServiceFabric.DataProtection.web create our ServiceFabricXmlRepository:

using AspNetCore.DataProtection.ServiceFabric;
using Microsoft.AspNetCore.DataProtection.Repositories;
using Microsoft.ServiceFabric.Services.Client;
using Microsoft.ServiceFabric.Services.Remoting.Client;
using System;
using System.Collections.Generic;
using System.Xml.Linq;

namespace ServiceFabric.DataProtection.Web
{
    public class ServiceFabricXmlRepository : IXmlRepository
    {
        public IReadOnlyCollection<XElement> GetAllElements()
        {
            var proxy = ServiceProxy.Create<IDataProtectionService>(new Uri("fabric:/ServiceFabric.DataProtection/DataProtectionService"), new ServicePartitionKey());
            return proxy.GetAllDataProtectionElements().Result.AsReadOnly();
        }

        public void StoreElement(XElement element, string friendlyName)
        {
            if (element == null)
            {
                throw new ArgumentNullException(nameof(element));
            }

            var proxy = ServiceProxy.Create<IDataProtectionService>(new Uri("fabric:/ServiceFabric.DataProtection/DataProtectionService"), new ServicePartitionKey());
            proxy.AddDataProtectionElement(element).Wait();
        }
    }
}

To easily bootstrap our custom ServiceFabricXmlRepository into ASP.Net Core on start-up, create the following DataProtectionBuilderExtensions class:

using Microsoft.AspNetCore.DataProtection;
using Microsoft.AspNetCore.DataProtection.Repositories;
using Microsoft.Extensions.DependencyInjection;
using System;

namespace ServiceFabric.DataProtection.Web
{
    public static class DataProtectionBuilderExtensions
    {
        public static IDataProtectionBuilder PersistKeysToServiceFabric(this IDataProtectionBuilder builder)
        {
            if (builder == null)
            {
                throw new ArgumentNullException(nameof(builder));
            }

            return builder.Use(ServiceDescriptor.Singleton<IXmlRepository>(services => new ServiceFabricXmlRepository()));
        }

        public static IDataProtectionBuilder Use(this IDataProtectionBuilder builder, ServiceDescriptor descriptor)
        {
            if (builder == null)
            {
                throw new ArgumentNullException(nameof(builder));
            }

            if (descriptor == null)
            {
                throw new ArgumentNullException(nameof(descriptor));
            }

            for (int i = builder.Services.Count - 1; i >= 0; i--)
            {
                if (builder.Services[i]?.ServiceType == descriptor.ServiceType)
                {
                    builder.Services.RemoveAt(i);
                }
            }

            builder.Services.Add(descriptor);

            return builder;
        }
    }
}

Building upon previous articles that detailed how to integrate Kestrel and Service Fabric, we extend WebHostBuilderHelper to also support the WebListener webserver:

using Microsoft.AspNetCore.Hosting;
using Microsoft.Net.Http.Server;
using System.Fabric;
using System.IO;

namespace ServiceFabric.DataProtection.Web
{
    internal static class WebHostBuilderHelper
    {
        public static IWebHost GetServiceFabricWebHost(ServerType serverType)
        {
            var endpoint = FabricRuntime.GetActivationContext().GetEndpoint("ServiceEndpoint");
            string serverUrl = $"{endpoint.Protocol}://{FabricRuntime.GetNodeContext().IPAddressOrFQDN}:{endpoint.Port}";

            return GetWebHost(endpoint.Protocol.ToString(), endpoint.Port.ToString(), serverType);
        }

        public static IWebHost GetWebHost(string protocol, string port, ServerType serverType)
        {
            switch (serverType)
            {
                case ServerType.WebListener:
                    {
                        IWebHostBuilder webHostBuilder = new WebHostBuilder()
                            .UseWebListener(options =>
                            {
                                options.ListenerSettings.Authentication.Schemes = AuthenticationSchemes.None;
                                options.ListenerSettings.Authentication.AllowAnonymous = true;
                            });

                        return ConfigureWebHostBuilder(webHostBuilder, protocol, port);
                    }
                case ServerType.Kestrel:
                    {
                        IWebHostBuilder webHostBuilder = new WebHostBuilder();
                        webHostBuilder.UseKestrel();

                        return ConfigureWebHostBuilder(webHostBuilder, protocol, port);
                    }
                default:
                    return null;
            }
        }

        static IWebHost ConfigureWebHostBuilder(IWebHostBuilder webHostBuilder, string protocol, string port)
        {
            return webHostBuilder
                .UseContentRoot(Directory.GetCurrentDirectory())
                .UseWebRoot(Path.Combine(Directory.GetCurrentDirectory(), "wwwroot"))
                .UseStartup<Startup>()
                .UseUrls($"{protocol}://+:{port}")
                .Build();
        }
    }

    enum ServerType
    {
        Kestrel,
        WebListener
    }
}

Your Web microservice should look something like:

using Microsoft.ServiceFabric.Services.Communication.AspNetCore;
using Microsoft.ServiceFabric.Services.Communication.Runtime;
using Microsoft.ServiceFabric.Services.Runtime;
using System.Collections.Generic;
using System.Fabric;

namespace ServiceFabric.DataProtection.Web
{
    internal sealed class WebService : StatelessService
    {
        ServerType _serverType;

        public WebService(StatelessServiceContext context, ServerType serverType)
            : base(context)
        {
            _serverType = serverType;
        }

        protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
        {
            return new ServiceInstanceListener[]
            {
                new ServiceInstanceListener(serviceContext =>
                {
                    switch (_serverType)
                    {
                        case ServerType.WebListener :
                            {
                                return new WebListenerCommunicationListener(serviceContext, "ServiceEndpoint", url =>
                                {
                                    return WebHostBuilderHelper.GetServiceFabricWebHost(_serverType);
                                });
                            }
                        case ServerType.Kestrel:
                            {
                                return new KestrelCommunicationListener(serviceContext, "ServiceEndpoint", url =>
                                {
                                    return WebHostBuilderHelper.GetServiceFabricWebHost(_serverType);
                                });
                            }
                        default:
                            return null;
                    }
                })                  
            };
        }
    }
}

Next, modify Program.cs with below code:

using CommandLine;
using Microsoft.AspNetCore.Hosting;
using Microsoft.ServiceFabric.Services.Runtime;
using System;
using System.Threading;

namespace ServiceFabric.DataProtection.Web
{
    internal static class Program
    {
        public static void Main(string[] args)
        {
            var parser = new Parser(with => 
            {
                with.EnableDashDash = true;
                with.HelpWriter = Console.Out;
            });

            var result = parser.ParseArguments<Options>(args);

            result.MapResult(options =>
            {
                switch (options.Host.ToLower())
                {
                    case "servicefabric-weblistener":
                        {
                            ServiceRuntime.RegisterServiceAsync("WebServiceType", context => new WebService(context, ServerType.WebListener)).GetAwaiter().GetResult();
                            Thread.Sleep(Timeout.Infinite);
                            break;
                        }
                    case "servicefabric-kestrel":
                        {
                            ServiceRuntime.RegisterServiceAsync("WebServiceType", context => new WebService(context, ServerType.Kestrel)).GetAwaiter().GetResult();
                            Thread.Sleep(Timeout.Infinite);
                            break;
                        }
                    case "weblistener":
                        {
                            using (var host = WebHostBuilderHelper.GetWebHost(options.Protocol, options.Port, ServerType.WebListener))
                            {
                                host.Run();
                            }
                            break;
                        }
                    case "kestrel":
                        {
                            using (var host = WebHostBuilderHelper.GetWebHost(options.Protocol, options.Port, ServerType.Kestrel))
                            {
                                host.Run();
                            }
                            break;
                        }
                    default:
                        break;
                }

                return 0;
            },
            errors =>
            {
                return 1;
            });
        }
    }

    internal sealed class Options
    {
        [Option(Default = "weblistener", HelpText = "Host - Options [weblistener] or [kestrel] or [servicefabric-weblistener] or [servicefabric-kestrel]")]
        public string Host { get; set; }

        [Option(Default = "http", HelpText = "Protocol - Options [http] or [https]")]
        public string Protocol { get; set; }

        [Option(Default = "localhost", HelpText = "IP Address or Uri - Example [localhost] or [127.0.0.1]")]
        public string IpAddressOrFQDN { get; set; }

        [Option(Default = "5000", HelpText = "Port - Example [80] or [5000]")]
        public string Port { get; set; }
    }
}

And finally, PersistKeysToServiceFabric needs to be added to Startup.cs as this will instruct the ASP.NET Core data protection stack to use our custom AspNetCore.DataProtection.ServiceFabric key repository:

using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.DataProtection;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Swashbuckle.AspNetCore.Swagger;

namespace ServiceFabric.DataProtection.Web
{
    public class Startup
    {
        public Startup(IHostingEnvironment env)
        {
            var builder = new ConfigurationBuilder()
                .SetBasePath(env.ContentRootPath)
                .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
                .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
                .AddEnvironmentVariables();
            Configuration = builder.Build();
        }

        public IConfigurationRoot Configuration { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            // Add framework services.
            services.AddMvc();

            // Add Service Fabric DataProtection
            services.AddDataProtection()
                    .SetApplicationName("ServiceFabric-DataProtection-Web")
                    .PersistKeysToServiceFabric();

            services.AddSwaggerGen(c =>
            {
                c.SwaggerDoc("v1", new Info { Title = "AspNetCore.DataProtection.ServiceFabric API", Version = "v1" });
            });
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
        {
            app.UseMvc();
            app.UseSwaggerUi(c =>
            {
                c.SwaggerEndpoint("/swagger/v1/swagger.json", "AspNetCore.DataProtection.ServiceFabric API v1");
            });
            app.UseSwagger();
        }
    }
}

All that is now left to do within your .Net Core Web Application PackageRoot is to edit the ServiceManifest.xml CodePackage so that we tell Web.exe to “host” within Service Fabric using WebListener:

 <CodePackage Name="Code" Version="1.0.0">
    <EntryPoint>
      <ExeHost>
        <Program>ServiceFabric.DataProtection.Web.exe</Program>
        <Arguments>--host servicefabric-weblistener</Arguments>
        <WorkingFolder>CodePackage</WorkingFolder>
        <ConsoleRedirection FileRetentionCount="5" FileMaxSizeInKb="2048" />
      </ExeHost>
    </EntryPoint>
  </CodePackage>

At an administrative command prompt you'll need to issue the below command to create the correct URL ACL for port 80 (please refer to the WebListener references section below for detailed instructions):

 netsh http add urlacl url=http://+:80/ user=Users 

Upon successful deployment to a multi-node cluster, use Swagger and the Protect/Unprotect APIs to test that all nodes have access to the same data protection keys:

Note, as we've created a custom ASP.NET Core data protection key repository, the data protection system will deregister the default key encryption at the rest mechanism that the heuristic provided, so keys will no longer be encrypted at rest. It is strongly recommended that you additionally specify an explicit key encryption mechanism for production applications.