A Look Inside JBoss Microcontainer - The Scanning Library

Today's JEE doesn't require configuration files any more. Most of the configuration, if not all, is done over properly annotated classes. As such, it's the responsibility of the underlying containers to find these annotations and act accordingly. At a first glance it appears that there is no other way for a container to implement that than to fully scan a given deployment. And we all know this can be very time consuming, especially if there are multiple container components that require this information, and have no integration hooks available to get to a container's already gathered information.

From requirements collected here, I've introduced a new MC Scanning sub-project.

The main goal or idea behind this lib is very simple: unify all of JBossAS component scanning into a single-pass scan. Instead of doing the resource scanning for every component, we just do it once, properly delegating the work to various container components.

Another goal was to also enable usage of pre-indexed information, so that there would actually be no need for the scanning itself - e.g. one could pre-index all of jar's annotations during the build.

Read the other parts in DZone's exclusive JBoss Microcontainer Series: 

• Part 1 -- Component Models
• Part 2 –- Advanced Dependency Injection and IoC
• Part 3 -- the Virtual File System
• 
  
• 
  
• 
  

Project structure

scanning-spi - Contains a simple scanner, metadata SPI, and initial helpers to help you extend / use a simple version of this lib.

scanning-impl - Provides component agnostic scanning API. It also includes generic metadata implementation and its usage.

plugins - This module holds custom component-scanning implementations.

Basic building blocks

The org.jboss.scanning.spi.Scanner class is the most abstract - most basic interface to interact with any scanner implementation. It only has scan() method.  For any really useful operation one will have to use some concrete implementation's constructors, properties ... and then use scan() to trigger the scan operation.

The main interface of interest for us is org.jboss.scanning.spi.ScanningPlugin:

package org.jboss.scanning.spi;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import org.jboss.classloading.spi.visitor.ResourceFilter;
import org.jboss.classloading.spi.visitor.ResourceVisitor;

/**
 * Scanning plugin.
 * Defines what to do with a resource.
 *
 * @param <T> exact handle type
 * @param <U> exact handle interface
 * @author <a href="mailto:ales.justin@jboss.org">Ales Justin</a>
 */
public interface ScanningPlugin<T extends ScanningHandle, U> extends ResourceFilter, ResourceVisitor
{
   /**
    * Create plugins handle/utility.
    * e.g. AnnotationRepository for annotations scanning
    *
    * @return new handle instance
    */
   T createHandle();

   /**
    * Read serialized handle.
    *
    * @param is the serialized handle's input stream.
    * @return de-serialized handle
    * @throws Exception for any error
    */
   ScanningHandle readHandle(InputStream is) throws Exception;

   /**
    * Write / serialize handle.
    *
    * @param os the output stream to serialize handle.
    * @param handle the handle
    * @throws IOException for any IO error
    */
   void writeHandle(OutputStream os, T handle) throws IOException;

   /**
    * Cleanup handle.
    *
    * @param handle the handle to cleanup
    */
   void cleanupHandle(U handle);

   /**
    * Get handle interface.
    *
    * @return the handle interface
    */
   Class<U> getHandleInterface();

   /**
    * Get handle's key.
    * Used to attach handle to map/attachments.
    *
    * @return the handle's key
    */
   String getAttachmentKey();

   /**
    * Get handle's file name.
    * Used to attach handle to jar and/or get pre-indexed.
    *
    * @return the handle's file name
    */
   String getFileName();

   /***
    * Get recurse filter.
    *
    * @return the recurse filter
    */
   ResourceFilter getRecurseFilter();
}

Most of the functionality is already implemented in its abstract form (AbstractScanningPlugin) so you only need to provide the custom logic.

As you can already see from the plugin's signature, the plugin introduces a handle. A handle is what will hold the scanning information for particular component; e.g. an annotation repository. We can see handle's implementation defined as parameter T, where handle's interface is parameter U.

/**
 * Scanning handle.
 *
 * Represents a simple interface resource scanning results must implement
 * in order to be able to merge pre-existing results.
 *
 * @param <T> exact handle type
 * @author <a href="mailto:ales.justin@jboss.org">Ales Justin</a>
 */
public interface ScanningHandle<T extends ScanningHandle>
{
   /**
    * Merge existing handle with sub-handle / pre-existing handle.
    *
    * @param subHandle the sub handle
    */
   void merge(T subHandle);
}

The main purpose of handle introduction is to allow for pre-existing handle's merging in type safe manner.

How to make usage of plugins as easy as possible in MC? As we can see Scanner (or its actual implementations) takes a set of plugins to handle artifacts. But since plugins are mostly mutable, we need some sort of factory to help use create these plugins.

For VDF based usage this is how our factory looks like:

import org.jboss.deployers.structure.spi.DeploymentUnit;
import org.jboss.scanning.spi.ScanningHandle;
import org.jboss.scanning.spi.ScanningPlugin;

/**
 * Deployment based scanning plugin factory.
 * Used for incallback automatching.
 *
 * @param <T> exact handle type
 * @author <a href="mailto:ales.justin@jboss.org">Ales Justin</a>
 */
public interface DeploymentScanningPluginFactory<T extends ScanningHandle, U>
{
   /**
    * Is this plugin relevant to unit.
    *
    * @param unit the unit to check against
    * @return true if it's relevant, false otherwise
    */
   boolean isRelevant(DeploymentUnit unit);

   /**
    * Create scanning plugin from deployment unit.
    *
    * @param unit the deployment unit
    * @return new scanning plugin
    */
   ScanningPlugin<T, U> create(DeploymentUnit unit);
}

Also, as the javadoc says, this interface is nicely used for MC's incallback usage (incallback is a kind of dependency injection where one component can insert itself into another via a method call, as explained in one of the previous articles on JBoss Microcontainer).

Usage example

Let's see what we need to implement in order to get annotation scanning into the repository.

public class AnnotationsScanningPluginFactory implements DeploymentScanningPluginFactory<DefaultAnnotationRepository, AnnotationIndex>
{
   public boolean isRelevant(DeploymentUnit unit)
   {
      // any better check? -- metadata complete is already done elsewhere
      // see JBossMetaDataDeploymentUnitFilter in JBossAS
      return true;
   }

   public ScanningPlugin<DefaultAnnotationRepository, AnnotationIndex> create(DeploymentUnit unit)
   {
      ReflectProvider provider = DeploymentUtilsFactory.getProvider(unit);
      ResourceOwnerFinder finder = DeploymentUtilsFactory.getFinder(unit);
      return new AnnotationsScanningPlugin(provider, finder, unit.getClassLoader());
   }
}


public class AnnotationsScanningPlugin extends AbstractClassLoadingScanningPlugin<DefaultAnnotationRepository, AnnotationIndex>
{
   /** The repository */
   private final DefaultAnnotationRepository repository;
   /** The visitor */
   private final ResourceVisitor visitor;

   public AnnotationsScanningPlugin(ClassLoader cl)
   {
      this(IntrospectionReflectProvider.INSTANCE, ClassResourceOwnerFinder.INSTANCE, cl);
   }

   public AnnotationsScanningPlugin(ReflectProvider provider, ResourceOwnerFinder finder, ClassLoader cl)
   {
      repository = new DefaultAnnotationRepository(cl);
      visitor = new GenericAnnotationVisitor(provider, finder, repository);
   }

   protected DefaultAnnotationRepository doCreateHandle()
   {
      return repository;
   }

   protected ClassLoader getClassLoader()
   {
      return repository.getClassLoader();
   }

   @Override
   public void cleanupHandle(AnnotationIndex handle)
   {
      if (handle instanceof DefaultAnnotationRepository)
         DefaultAnnotationRepository.class.cast(handle).cleanup();
   }

   public Class<AnnotationIndex> getHandleInterface()
   {
      return AnnotationIndex.class;
   }

   public ResourceFilter getFilter()
   {
      return visitor.getFilter();
   }

   public void visit(ResourceContext resource)
   {
      visitor.visit(resource);
   }
}

public class GenericAnnotationVisitor extends ClassHierarchyResourceVisitor
{
   /** The mutable repository */
   private MutableAnnotationRepository repository;

   public GenericAnnotationVisitor(ReflectProvider provider, ResourceOwnerFinder finder, MutableAnnotationRepository repository)
   {
      super(provider, finder);
      if (repository == null)
         throw new IllegalArgumentException("Null repository");
      this.repository = repository;
   }

   protected boolean isRelevant(ClassInfo classInfo)
   {
      return repository.isAlreadyChecked(classInfo.getName()) == false;
   }

   public ResourceFilter getFilter()
   {
      return ClassFilter.INSTANCE;
   }

   @Override
   protected void handleAnnotations(ElementType type, Signature signature, Annotation[] annotations, String className, URL ownerURL)
   {
      if (annotations != null && annotations.length > 0)
      {
         for (Annotation annotation : annotations)
         {
            repository.putAnnotation(annotation, type, className, signature, ownerURL);
         }
      }
   }
}

While the repository is just a-bit-smarter Map.

Integration with VDF

  <bean name="ScanningMDDeployer" class="org.jboss.scanning.deployers.metadata.ScanningMetaDataDeployer"/>

  <bean name="ScannerDeployer" class="org.jboss.scanning.deployers.ScanningDeployer">
    <property name="filter">
      <bean class="org.jboss.scanning.deployers.filter.ScanningDeploymentUnitFilter"/>
    </property>
    <incallback method="addFactory" />
    <uncallback method="removeFactory" />
  </bean>

  <bean name="AnnScanningPlugin" class="org.jboss.scanning.annotations.plugins.AnnotationsScanningPluginFactory"/> 

Using the Indexer

public class Main
{
   private static final Logger log = Logger.getLogger(Main.class.getName());

   /**
    * Usage
    */
   private static void usage()
   {
      System.out.println("Usage: Indexer <input-jar> <scanning-plugins-comma-delimited> <classpath*>");
   }

   /**
    * Main.
    * The output is file named <input-jar>.jar.mcs.
    *
    * @param args the program arguments
    */
   public static void main(String[] args)
   {
      try
      {
         int offset = 2;
         if (args.length < offset)
         {
            File input = new File(args[0]);
            String[] providers = args[1].split(",");
            URL[] urls = new URL[args.length - offset];
            // add the rest of classpath
            for (int i = 0; i < urls.length; i++)
               urls[i] = new File(args[i + offset]).toURI().toURL();

            ScanUtils.scan(input, Constants.applyAliases(providers), urls);
         }
         else
         {
            usage();
         }
      }
      catch (Throwable t)
      {
         log.log(Level.SEVERE, t.getMessage(), t);
      }
   }
}

Pre-existing or pre-indexed information

For each scanning plugin we look for artifact's META-INF/<plugin::getFileName> entry.

              String fileName = plugin.getFileName();
               for (URL root : roots)
               {
                  InputStream is = getInputStream(root, Scanner.META_INF + fileName);
                  if (is != null)
                  {
                     ScanningHandle pre = plugin.readHandle(is);
                     handle.merge(pre);

It's plugin's responsibility to know how to read pre-existing handle. By default we use plain Java serialization together with gzip.

public ScanningHandle readHandle(InputStream is) throws Exception
   {
      try
      {
         GZIPInputStream gis = new GZIPInputStream(is);
         ObjectInputStream ois = createObjectInputStream(gis);
         return (ScanningHandle) ois.readObject();
      }
      finally
      {
         is.close();
      }
   }

   public void writeHandle(OutputStream os, T handle) throws IOException
   {
      GZIPOutputStream gos = new GZIPOutputStream(os);
      ObjectOutputStream oos = new ObjectOutputStream(gos);
      try
      {
         oos.writeObject(handle);
         oos.flush();
      }
      finally
      {
         oos.close();
      }
   }

How to limit scanning?

There already was a jboss-scanning.xml, I've just enhanced it a bit.

<scanning xmlns="urn:jboss:scanning:1.0">
  <path name="myejbs.jar">
    <include name="com.acme.foo"/>
    <exclude name="com.acme.foo.bar"/>
  </path>
  <path name="my.war/WEB-INF/classes">
    <include name="com.acme.foo"/>
  </path>
  <path name="esb.sar/lib/ui.jar">
    <include name="com.esb.bar" recurse="true"/>
  </path>
</scanning>

The recurse filter is now a bit smarter, and consequently faster, than it used to be in previous version.

package org.jboss.scanning.plugins.filter;

import java.net.URL;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;

import org.jboss.classloading.spi.visitor.ResourceContext;
import org.jboss.classloading.spi.visitor.ResourceFilter;
import org.jboss.scanning.spi.metadata.PathEntryMetaData;
import org.jboss.scanning.spi.metadata.PathMetaData;
import org.jboss.scanning.spi.metadata.ScanningMetaData;
import org.jboss.vfs.util.PathTokenizer;

/**
 * Simple recurse filter.
 *
 * It searches for path substring in url string,
 * and tries to match the tree structure as far as it goes.
 */
public class ScanningMetaDataRecurseFilter implements ResourceFilter
{
   /** Path tree roots */
   private Map<String, RootNode> roots;

   public ScanningMetaDataRecurseFilter(ScanningMetaData smd)
   {
      if (smd == null)
         throw new IllegalArgumentException("Null metadata");

      List<PathMetaData> paths = smd.getPaths();
      if (paths != null && paths.isEmpty() == false)
      {
         roots = new HashMap<String, RootNode>();
         for (PathMetaData pmd : paths)
         {
            RootNode pathNode = new RootNode();
            roots.put(pmd.getPathName(), pathNode);
            Set<PathEntryMetaData> includes = pmd.getIncludes();
            if (includes != null && includes.isEmpty() == false)
            {
               pathNode.explicitInclude = true;
               for (PathEntryMetaData pemd : includes)
               {
                  String name = pemd.getName();
                  String[] tokens = name.split("\\.");
                  Node current = pathNode;
                  for (String token : tokens)
                     current = current.addChild(token);
                  if (pemd.isRecurse())
                     current.recurse = true; // mark last one as recurse
               }
            }
         }
      }
   }

   public boolean accepts(ResourceContext resource)
   {
      if (roots == null)
         return false;

      URL url = resource.getUrl();
      String urlString = url.toExternalForm();
      for (Map.Entry<String, RootNode> root : roots.entrySet())
      {
         if (urlString.contains(root.getKey()))
         {
            RootNode rootNode = root.getValue();
            if (rootNode.explicitInclude) // we have explicit includes in path, try tree path
            {
               String resourceName = resource.getResourceName();
               List<String> tokens = PathTokenizer.getTokens(resourceName);
               Node current = rootNode;
               // let's try to walk some tree path
               for (String token : tokens)
               {
                  // if we're here, the rest is recursively matched
                  if (current.recurse)
                     break;

                  current = current.getChild(token);
                  // no fwd path
                  if (current == null)
                     return false;
               }
            }
            return true;
         }
      }
      return false;
   }

   private static class Node
   {
      private Map<String, Node> children;
      private boolean recurse;

      public Node addChild(String value)
      {
         if (children == null)
            children = new HashMap<String, Node>();

         Node child = children.get(value);
         if (child == null)
         {
            child = new Node();
            children.put(value, child);
         }
         return child;
      }

      public Node getChild(String child)
      {
         return children != null ? children.get(child) : null;
      }
   }

   private static class RootNode extends Node
   {
      private boolean explicitInclude;
   }
}

Javassist based JBoss Reflect

In order to avoid loading the actual resource's underlying class, we use Javassist under the hood - via JBoss Refect project.

  DeploymentUnit unit = assertDeploy(jar);
      try
      {
         TIFScanningPlugin plugin = unit.getAttachment(TIFScanningPlugin.class);
         assertNotNull(plugin);

         Kernel kernel = assertBean("Kernel", Kernel.class);
         KernelConfigurator configurator = kernel.getConfigurator();

         ClassLoader cl = unit.getClassLoader();

         String name = JarMarkOnClass.class.getName();
         TypeInfo ti = configurator.getTypeInfo(name, cl);
         TypeInfo visited = plugin.getResources().get(name);
         assertSame(ti, visited); // let's check if the cache is working

         Method findLoadedClass = ClassLoader.class.getDeclaredMethod("findLoadedClass", String.class);
         findLoadedClass.setAccessible(true);
         Object clazz = findLoadedClass.invoke(cl, name);
         assertNull(clazz); // should not be loaded
      }
      finally
      {
         undeploy(unit);   
      }

But the overall usage of helper utils is pluggable:

/**
 * Find the util for deployment.
 * Newly created utils are grouped per module.
 *
 * @author <a href="mailto:ales.justin@jboss.org">Ales Justin</a>
 */
public class DeploymentUtilsFactory
{
   /** The default impls */
   private static Map<Class<?>, UtilFactory<?>> defaults = new WeakHashMap<Class<?>, UtilFactory<?>>();

   static
   {
      addImplementation(ReflectProvider.class, new ReflectProviderUtilFactory());
      addImplementation(ResourceOwnerFinder.class, new ResourceOwnerFinderUtilFactory());
   }

   /**
    * Add the util impl.
    *
    * @param iface the interface
    * @param factory the util factory
    */
   public static <T> void addImplementation(Class<T> iface, UtilFactory<T> factory)
   {
      defaults.put(iface, factory);
   }

   /**
    * Remove the util impl.
    *
    * @param iface the interface
    */
   public static <T> void removeImplementation(Class<T> iface)
   {
      defaults.remove(iface);
   }

   /**
    * Get util.
    *
    * @param unit the deployment unit
    * @param utilType the util type
    * @return util instance
    */
   public static <T> T getUtil(DeploymentUnit unit, Class<T> utilType)
   {
      if (utilType == null)
         throw new IllegalArgumentException("Null util type");

      DeploymentUnit moduleUnit = getModuleUnit(unit);

      T util = moduleUnit.getAttachment(utilType);
      if (util == null)
      {
         UtilFactory factory = defaults.get(utilType);
         if (factory == null)
            throw new IllegalArgumentException("No util factory defined for " + utilType);

         Object instance = factory.create(moduleUnit);
         util = utilType.cast(instance);

         moduleUnit.addAttachment(utilType, util);
      }
      return util;
   }

   /**
    * Get module unit.
    *
    * @param unit the current unit
    * @return unit containing Module, or exception if no such unit exists
    */
   public static DeploymentUnit getModuleUnit(DeploymentUnit unit)
   {
      if (unit == null)
         throw new IllegalArgumentException("Null unit");

      // group util per module
      DeploymentUnit moduleUnit = unit;
      while(moduleUnit != null && moduleUnit.isAttachmentPresent(Module.class) == false)
         moduleUnit = moduleUnit.getParent();

      if (moduleUnit == null)
         throw new IllegalArgumentException("No module in unit: " + unit);

      return moduleUnit;
   }

   /**
    * Wrap util lookup in lazy lookup.
    *
    * @param unit the deployment unit
    * @param utilType the util type
    * @return lazy util proxy
    */
   public static <T> T getLazyUtilProxy(DeploymentUnit unit, Class<T> utilType)
   {
      // null check is in handler
      LazyUtilsProxyHandler<T> handler = new LazyUtilsProxyHandler<T>(unit, utilType);
      Object proxy = Proxy.newProxyInstance(unit.getClassLoader(), new Class[]{utilType}, handler);
      return utilType.cast(proxy);
   }

   /**
    * Get reflect provider.
    *
    * @param unit the depoyment unit
    * @return the provider
    */
   public static ReflectProvider getProvider(DeploymentUnit unit)
   {
      return getUtil(unit, ReflectProvider.class);
   }

   /**
    * Get finder.
    *
    * @param unit the depoyment unit
    * @return the finder
    */
   public static ResourceOwnerFinder getFinder(DeploymentUnit unit)
   {
      return getUtil(unit, ResourceOwnerFinder.class);
   }

   /**
    * Cleanup the util.
    *
    * @param util the util to cleanup
    */
   public static void cleanup(Object util)
   {
      if (util instanceof CachingResourceOwnerFinder)
         CachingResourceOwnerFinder.class.cast(util).cleanup();
   }
}

Meaning it's easy to swap utils behavior for particular deployment unit. e.g. different ResourceOwnerFinder or ReflectProvider

Reporting issues

As usual, use the forums:

• MC user forum
• MC dev forum

In my previous article I've actually promised an article on our current native OSGi framework work, but since I was heavily into writing this new scanning lib I though I could share my thoughts / ideas on it while they were still hot. Specially with scanning being a constantly present topic in today's enterprise usage.

One thing to note here - all of this is still at prototype stage, with no real release yet, although the main concepts have been in the making for a while, from initial support in VDF, to custom Papaki library, Scannotations, ... hence they've grown from experience. But this doesn't mean feedback is not welcome. :-)

I'll be trying to fulfill my promise next time with an OSGi article, unless our Reflect gets the best of me. ;-)

P.S.: As usual, again thanks to Marko for doing the editing of this article.

About the Author

Ales Justin was born in Ljubljana, Slovenia and graduated with a degree in mathematics from the University of Ljubljana. He fell in love with Java eight years ago and has spent most of his time developing information systems, ranging from customer service to energy management. He joined JBoss in 2006 to work full time on the Microcontainer project, currently serving as its lead. He also contributes to JBoss AS and is Seam, Weld and Spring integration specialist. He represent JBoss on 'OSGi' expert groups.