Implementing the Ignite.NET Plugin: Distributed Semaphore

Apache Ignite.NET 2.0 has introduced a plugin system. Plugins can be .NET-only or .NET + Java. Let’s see how to implement the latter.

Why Would I Need a Plugin?

Ignite.NET is built on top of Ignite, which is written in Java. JVM is started within a .NET process, and the .NET part talks to the Java part and reuses existing Ignite functionality where possible.

The plugin system exposes this platform interaction mechanism to third parties. One of the main use cases is making Ignite and third party Java APIs available in .NET.

A good example of such an API is IgniteSemaphore, which is not yet available in Ignite.NET.

All source code for this post is available on GitHub.

Distributed Semaphore API

Ignite Semaphore is very similar to System.Threading.Semaphore (MSDN), but the effect is cluster-wide: limit the number of threads executing a given piece of code across all Ignite nodes.

It should be used in C# code like this:

IIgnite ignite = Ignition.GetIgnite();
ISemaphore semaphore = ignite.GetOrCreateSemaphore(name: "foo", count: 3);

semaphore.WaitOne();  // Enter the semaphore (may block)
// Do work
semaphore.Release();

Looks simple enough, and it's quite useful; the same API as built-in .NET Semaphore. Obviously, we can’t change the IIgnite interface, so GetOrCreateSemaphore is an extension method. Now onto the implementation!

Java Plugin

Let’s start with Java side of things. We need a way to call the Ignite.semaphore() method there and provide access to the resulting instance to the .NET platform.

Create a Java project and reference Ignite 2.0 from Maven (detailed instructions can be found in this Building Multi-Platform Ignite Cluster post).

Every plugin starts with PluginConfiguration. Our plugin does not need any configuration properties, but the class must exist, so just make a simple one:

public class IgniteNetSemaphorePluginConfiguration implements PluginConfiguration {} 

Then comes the plugin entry point: PluginProvider<PluginConfiguration>. This interface has lots of methods, but most of them can be left empty (name and version must not be null, so put something in there). We are interested only in the initExtensions method, which allows us to provide a cross-platform interoperation entry point. This is done by registering the PlatformPluginExtension implementation:

public class IgniteNetSemaphorePluginProvider implements PluginProvider<IgniteNetSemaphorePluginConfiguration> {
    public String name() { return "DotNetSemaphore"; }
    public String version() { return "1.0"; }

    public void initExtensions(PluginContext pluginContext, ExtensionRegistry extensionRegistry) 
            throws IgniteCheckedException {
        extensionRegistry.registerExtension(PlatformPluginExtension.class,
                new IgniteNetSemaphorePluginExtension(pluginContext.grid()));
    }
...
}

PlatformPluginExtension has a unique id to retrieve it from the .NET side and a PlatformTarget createTarget() method to create an object that can be invoked from .NET.

PlatformTarget interface in Java mirrors IPlatformTarget interface in .NET. When you call IPlatformTarget.InLongOutLong in .NET, PlatformTarget.processInLongOutLong is called in Java on your implementation. There are a number of other methods that allow exchanging of primitives, serialized data, and objects. Each method has a type parameter which specifies an operation code, in case when there are many different methods on your plugin.

We are going to need two PlatformTarget classes: one that represents our plugin as a whole and has the getOrCreateSemaphore method, and another one to represent each particular semaphore. The first one should take a string name and int count and return an object, so we need to implement PlatformTarget.processInStreamOutObject. The other methods are not needed and can be left blank:

public class IgniteNetPluginTarget implements PlatformTarget {
    private final Ignite ignite;

    public IgniteNetPluginTarget(Ignite ignite) {
        this.ignite = ignite;
    }

    public PlatformTarget processInStreamOutObject(int i, BinaryRawReaderEx binaryRawReaderEx) throws IgniteCheckedException {
        String name = binaryRawReaderEx.readString();
        int count = binaryRawReaderEx.readInt();

        IgniteSemaphore semaphore = ignite.semaphore(name, count, true, true);

        return new IgniteNetSemaphore(semaphore);
    }
...
}

For each ISemaphore object in .NET, there will be one IgniteNetSemaphore in Java, which is also a PlatformTarget. This object will handle WaitOne and Release methods and delegate them to an underlying IgniteSemaphore object. Since both of these methods are void and parameterless, the simplest PlatformTarget method will work:

public long processInLongOutLong(int i, long l) throws IgniteCheckedException {
    if (i == 0) semaphore.acquire();
    else semaphore.release();

    return 0;
}

That’s it, the Java part is implemented! We just need to make our IgniteNetSemaphorePluginProvider class available to the Java service loader by creating a resources\META-INF.services\org.apache.ignite.plugin.PluginProvider file with a single line containing the class name. Package the project with Maven (mvn package in the console, or use IDEA UI). There should be a IgniteNetSemaphorePlugin-1.0-SNAPSHOT.jar file in the target directory. We can move on to the .NET part now.

.NET Plugin

First, let’s make sure our Java code gets picked up by Ignite. Create a console project, install the Ignite NuGet package, and start Ignite with the path to the jar file that we just created:

var cfg = new IgniteConfiguration
{
    JvmClasspath = @"..\..\..\..\Java\target\IgniteNetSemaphorePlugin-1.0-SNAPSHOT.jar"
};

Ignition.Start(cfg);

The Ignite node starts up and we should see our plugin name in the log:

[16:02:38] Configured plugins:
[16:02:38]   ^-- DotNetSemaphore 1.0

Great! For the .NET part, we’ll take an API-first approach: implement the extension method first and continue from there.

public static class IgniteExtensions
{
    public static Semaphore GetOrCreateSemaphore(this IIgnite ignite, string name, int count)
    {
        return ignite.GetPlugin<SemaphorePlugin>("semaphorePlugin").GetOrCreateSemaphore(name, count);
    }
}

For the GetPlugin method to work, the IgniteConfiguration.PluginConfigurations property should be set. It takes a collection of IPluginConfiguration implementations, and each implementation must, in turn, link to an IPluginProvider implementation with an attribute:

[PluginProviderType(typeof(SemaphorePluginProvider))]
class SemaphorePluginConfiguration : IPluginConfiguration  {...}

On node startup, Ignite.NET iterates through plugin configurations, instantiates plugin providers, and calls the Start(IPluginContext<SemaphorePluginConfiguration> context) method on them. IIgnite.GetPlugin calls are then delegated to IPluginProvider.GetPlugin of the provider with the specified name.

class SemaphorePluginProvider : IPluginProvider<SemaphorePluginConfiguration>
{
    private SemaphorePlugin _plugin;

    public T GetPlugin<T>() where T : class
    {
        return _plugin as T;
    }

    public void Start(IPluginContext<SemaphorePluginConfiguration> context)
    {
        _plugin = new SemaphorePlugin(context);
    }

    ...

}

IPluginContext provides access to the Ignite instance, Ignite and plugin configurations, and has theGetExtension method, which delegates to PlatformPluginExtension.createTarget() in Java. This way we “establish connection” between the two platforms. IPlatformTarget in .NET gets linked to PlatformTarget in Java; they can call each other, and the lifetime of the Java object is tied to the lifetime of the .NET object. Once the .NET object is reclaimed by the garbage collector, finalizer releases the Java object reference, and it will also be garbage collected.

The remaining implementation is simple - just call the appropriate IPlatformTarget methods:

class SemaphorePlugin
{
    private readonly IPlatformTarget _target;  // Refers to IgniteNetPluginTarget in Java

    public SemaphorePlugin(IPluginContext<SemaphorePluginConfiguration> context)
    {
        _target = context.GetExtension(100);
    }

    public Semaphore GetOrCreateSemaphore(string name, int count)
    {
        var semaphoreTarget = _target.InStreamOutObject(0, w =>
        {
            w.WriteString(name);
            w.WriteInt(count);
        });

        return new Semaphore(semaphoreTarget);
    }
}

class Semaphore
{
    private readonly IPlatformTarget _target;  // Refers to IgniteNetSemaphore in Java

    public Semaphore(IPlatformTarget target)
    {
        _target = target;
    }

    public void WaitOne()
    {
        _target.InLongOutLong(0, 0);
    }

    public void Release()
    {
        _target.InLongOutLong(1, 0);
    }
}

We are done! Quite a bit of boilerplate code, but adding more logic to the existing plugin is easy, just implement a pair of methods on both sides. Ignite uses JNI and unmanaged memory to exchange data between .NET and Java platforms within a single process, which is simple and efficient.

Testing

To demonstrate the distributed nature of our Semaphore, we can run multiple Ignite nodes where each of them calls WaitOne(). We’ll see that only two nodes at a time are able to acquire the semaphore:

var ignite = Ignition.Start(cfg);
var sem = ignite.GetOrCreateSemaphore("foo", 2);

Console.WriteLine("Trying to acquire semaphore...");

sem.WaitOne();

Console.WriteLine("Semaphore acquired. Press any key to release.");
Console.ReadKey();

Download the full project from GitHub.