Java Annotations and Reflection: Powerful Building Blocks for a DBMS Interface

Annotations are a Java feature introduced in JDK5 (a similar mechanism, attributes, also exists in .NET). The main idea of annotations is to give the programmer the ability to specify all necessary information in the same place. Consider the "classical" approach: there are source files with program code; separate files with API documentation that describes the behavior of this code; and additional files in SQL or some other format providing some related information (needed, for example, for a database interface).

The problem with this approach lies in the difficulty collecting and managing (programmatically as well as mentally) all the necessary information. When looking at the source of a method, the developer may want to understand the meaning of its parameters. But these are explained in the documentation file, which then must be opened and searched.

This is inconvenient, but what’s worse is that the information in separate places soon becomes inconsistent: for example, the program sources are updated but not the documentation. And while a discrepancy between implementation and documentation is bad but not critical, inconsistency in other areas can be fatal, such as when a configuration file explains the format of a class, and then a new field is added to this class without an update to the configuration file. With annotations, this information can be accessed in one location for learning, updates, etc.

While not yet widely recognized, annotations have a highly useful role in developing an interface to a database management system. By centralizing information about object formats, annotations simplify the programmer’s job, and eliminate the risk of errors from discrepancies in the way objects (classes) are described.  This is especially beneficial in a database interface because object descriptions are necessarily complex: a database needs more information than is available in a normal Java class definition. For example, a string field can be stored in a database using different encodings. Also, to query the database efficiently, indexes must be created, and there should be a way for the programmer to mark primary and foreign keys within the Java class.

Annotations are especially powerful as the basis for a DBMS interface when combined with Java’s reflection mechanism, which allows an application to get information about class format at runtime. Since Java works with objects, one requirement is to specify the format of objects (classes) to the database engine. In languages such as C/C++ there are two common ways to do this:

• Use a specialized compiler or preprocessor that analyzes application source and extracts class information, or
• Require the programmer to describe class formats — in which case this work may actually be done twice: once when declaring the classes in the application, and a second time, when describing their format for the DBMS

But in Java, reflection can enable the DBMS to get format information about application classes at runtime, eliminating the need for either extra software (the specialized compiler or preprocessor) or extra work.

Presenting an Annotations-Based DBMS Interface in Java

The remainder of this article illustrates how Java’s annotations and reflection can be harnessed to provide a database API that is both easier to use, and less error-prone, than alternative approaches. As an example it uses the Java Native interface (JNI) for the eXtremeDB embedded database http://www.mcobject.com/extremedb-jni, which provides a means for this database system written in C to be used from within “pure” Java applications. Complete information about annotation syntax and semantics in Java can be found at http://www.developer.com/article.php/3556176. For this article it is enough to know that

• Annotations are a special kind of Java interface
• Annotations are represented with syntax in which the annotation’s name is preceded by the @ character
• Annotations can be associated with types, fields, methods, parameters and local variable declarations
• Annotations’ parameters can be specified using keyword notation, and if a parameter is not specified, then a default value is used. The default "value" parameter can be set without specifying the name of the parameter
• Annotations can be obtained using the reflection API at runtime

The first step is to define a database via class declarations in a Java application. The eXtremeDB Java API consists of nine annotations, but this relatively simple example requires just four: @Index, @Indexable, @Indexes and @Key. These annotations’ properties are as follows:

@Index

The @Index annotation specifies an index for the class. Actually, there is a more convenient way to define a single field index: using the @Indexable annotation for the particular field (described in more detail below). However, the need for compound indexes in our example is addressed by this separate class-level annotation. The parameters below demonstrate @Index with a B-Tree index, but an R-Tree, Patricia trie, KD Tree or hash index could also be used.

Parameters:

String name(): name of the index
boolean unique() default false: whether index is unique or allows duplicates
int initSize() default 1000: initial size for hash table
Database.IndexType type() default Database.IndexType.BTree: type of the index
Key[] keys(): list of index keys

Target: class

Example:

       @Index(name="byName", unique=true, keys={@Key("lastName"), @Key("firstName")})       class Person {           String firstName;           String lastName;

@Indexable 

This annotation marks a field as indexable and provides the most convenient way to define a single-field index. Because Java allows only one annotation of a particular type to be associated with a concrete declaration, a field can be included in only one index using the @Indexable annotation. If the application requires defining more than one index for a field, the @Index annotation is used instead.

Parameters:

boolean unique() default false: whether index is unique or allows duplicates
boolean descending() default false: whether objects are placed in the index in descending or ascending  order of the key
int initSize() default 1000: initial size for hash table
Database.IndexType type() default Database.IndexType.BTree: type of the index

Target: field

Example:

       @Indexable       int age;

@Indexes

This annotation specifies a set of class indexes. As mentioned above, Java doesn't allow duplicate annotations, so if multiple indexes are required, their definitions should be “wrapped” in the @Indexes annotation.

Parameters:

Index[] value(): list of index descriptors

Target: class

Example:

       @Indexes({@Index(name="byName", keys={@Key("lastName"), @Key("firstName")}),                 @Index(name="byAddress", keys={@Key("address.city"), @Key("address.street")})}       class Employee {           String firstName;           String lastName;           Address address;           ...       }

@Key

The @key annotation specifies an index key. Its purpose is to define a pair of key properties: the name of the indexed field and its order (such as “LastName, FirstName” or “FirstName, Lastname”) in the index.

Parameters:

String value(): name of the indexed field
boolean descending() default false:  where objects are placed in the index in descending or ascending order of the key

Target: class (parameter of @Index annotation)

Example: see example of @Index annotation

Defining a Database

Using the annotations listed above, a database can be created within a Java application by defining the following classes:

        class Record {@Indexablepublic int id; // record identifierpublic String str; // some string value        }  @Indexes({@Index(name="byName", keys={@Key("lastName"), @Key("firstName")}),              @Index(name=”byFullName”,keys={@Key(“fullName”)})})        class Person        {              public String firstName;              public String lastName;              public String fullName;              public int age;              public float weight;        }

When the application containing this definition starts up, the JNI uses reflection to examine the Java classes (and their annotations). It uses this information to build the database dictionary, or centralized repository for meta-data such as data types, relationships, indexes and more. Reflection enables the database run-time to “know” how much space to allocate for an object, what fields an object has (and their types), fields that participate in different indexes, and more.

For developers who are used to working in Java, this approach provides considerable convenience. The alternative – whether based on a proprietary JNI provided for a DBMS, or a standards-based interface such as Java Database Connectivity (JDBC) – consists of learning a data definition language, or DDL, that is different from Java; manually creating a schema document in this language; processing this schema using the DBMS’s language processor or interpreter; and ensuring that the resulting database dictionary file is accessible to application processes that need it.

Accessing the Database

With eXtremeDB JNI, annotations are used only to define the database. In the procedural code, these annotations have no role, and the developer works with database objects as if they are Plain Old Java Objects (POJOs), which is exactly what most Java aficionados prefer. The following examples show how the database defined above is accessed in Java application code:

Creating a database instance:

        Database db = new Database();        Database.Parameters params = new Database.Parameters();        params.memPageSize = PAGE_SIZE;        params.classes = new Class[] { Record.class };        db.open("operations-db", params, DATABASE_SIZE);

Opening a connection to the database:

       Connection con = new Connection(db);

Inserting data into the database:

       // start Read-Write transaction       con.startTransaction(Database.TransactionType.ReadWrite);       Record  rec = new Record(); // create Java object       // fill data       rec.id = 1;       rec.str = "Some value";       con.insert(rec); // insert object into eXtremeDB database       con.commitTransaction(); // commit changes

Querying the database:

      // open read-only transaction      con.startTransaction(Database.TransactionType.ReadOnly);      // Open cursor by "id" index      Cursor<Record> cursor = new Cursor<Record>(con, Record.class, "id");      for (Record rec : cursor) { // print out all objects          System.out.println("id=" + rec.id + ", str=\"" + rec.str + "\"");      }      cursor.close(); // close cursor      con.commitTransaction(); // end transaction

Updating an object:

      con.startTransaction(Database.TransactionType.ReadWrite);      // Perform simple index search: locate Record by id      cursor = new Cursor<Record>(con, Record.class, "id");      // find record to update       rec = cursor.find(2);      // update object      rec.str = "Updated string";      cursor.update(); // update current object (pointed by the cursor) in database      cursor.close(); //release cursor      con.commitTransaction(); // commit changes

Deleting an object:

      con.startTransaction(Database.TransactionType.ReadWrite);      // Perform simple index search: locate Record by id      cursor = new Cursor<Record>(con, Record.class, "id");      // find record with id == 3      rec = cursor.find(3);      cursor.remove(); // remove current object (pointed by cursor) from database      cursor.close(); //release cursor      con.commitTransaction(); // commit changes

Deleting all objects:

      con.startTransaction(Database.TransactionType.ReadWrite);      con.removeAll(Record.class);      con.commitTransaction();      // cleanup      con.disconnect();      db.close();

Annotations-Based API vs. Java Database Connectivity (JDBC)

In several places, the code above illustrates a major advantage of the annotations-based JNI over approaches such as Java Database Connectivity (JDBC): with the JNI, there is no need for the programmer to implement pack/unpack functions.  Instead, a line of code such as

                   Record rec = cursor.find("Some-Key");

is all that is needed to get a fully-populated instance of ‘rec’.  The JNI will retrieve all of rec’s field values, with no explicit unpacking required.

Another code comparison illustrates an advantage described earlier in this article, namely, that annotations provide a way to keep relevant information about application functions in a central location, eliminating the possibility of errors from discrepancies in the way data is described.

With JDBC, selecting an object from the database would look something like this:

    List<Person> findPersons(Connection con, String name)     {        Statement stmt = con.prepareStatement("select * from Person where full_name=?");        stmt.setString(1, name);        ResultSet cursor = stmt.executeQuery();        ArrayList<Person> persons = new ArrayList<Person>();        while (cursor.next()) {             Person person = new Person();            person.firstName = cursor.getString("first_name")            person.lastName = cursor.getString("last_name")            person.fullName = cursor.getString("full_name")            person.age = cursor.getInt("age")            person.weight = cursor.getFloat("weight")            persons.add(person);        }        cursor.close();        stmt.close();        return persons;    }

However, the database JNI based on annotations and reflection allows the same function to be coded as follows:

    List<Person> findPersons(Connection con, String name)     {        con.startTransaction(Database.TransactionType.ReadOnly); // open read-only transaction        Cursor<Person> cursor = new Cursor<Person(con, Person.class, "byFullName");        ArrayList<Person> persons = new ArrayList<Person>();        if (cursor.search(Cursor.Operation.Equals, name)) {            for (Person person : cursor) {                persons.add(person);            }        }        cursor.close(); // close cursor        con.commitTransaction(); // end transaction        return list;    }

Note that the JDBC code must extract (copy) each of the object’s fields from the database cursor (i.e. “map” the relational DB row to the Java class, as in object-relational mapping).  If the Java class fields are not defined as having the same type, or one or more fields are left out, a discrepancy is introduced between the Java class and the RDBMS table. For example, a variable might be defined as a floating point number in the RDBMS and as an integer in the Java class. This loss of fidelity/precision would likely crop up as a bug before the software is released and add to the overall development time. If the bug somehow slipped through QA and made it into production code, loss of precision could be more or less severe depending on the function's and the application's purpose. For example, a high degree of accuracy would be desired in contexts such as a bank balance or a calculation of “distance to empty” or “time to target.” Using the Java language (with annotations) as the data definition language completely eliminates this risk.

Annotations for Specialized Database Features

The example above demonstrates how annotations can be used to define indexes and keys that will be used for organizing, sorting and searching the objects contained in a Java class. The approach can be extended to allow definition of virtually any database characteristic within the Java class declaration. The following annotations, again from McObject’s eXtremeDB JNI, illustrate how an annotations-based API addresses more specialized database challenges.

Challenge 1:  Supporting different national encoding schemes
A Java string consists of two-byte characters (UTF-16 encoding). eXtremeDB is able to store strings either as sequences of chars (bytes) or as wide-character (two-bytes) strings. By default, eXtremeDB JNI uses UTF-8 encoding to convert Java strings into sequences of bytes. With the @Encoding annotation, the developer can specify any national encoding:

@Encoding parameters:

String value(): string encoding, like "UTF-8", "ASCII", "Windows-1251", ...

Target: field

Example:

       @Encoding("UTF-8")       String name;

Specifying "UTF-16" encoding in the annotation causes the string to be stored in eXtremeDB without any conversion, as a sequence of wide characters.

This approach to annotation also allows strings in certain languages to be stored in the most efficient way – for example, if I store strings in Russian in UTF-8, then each Cyrillic character will be stored as two bytes. But if I use "windows-1251" encoding, then each character consumes one byte in the database.

Challenge 2: Handling large data objects
eXtremeDB arrays and strings are limited to 65535 elements. To store large volumes of data, BLOBs are used. Although it would be possible to define a BLOB type in the eXtremeDB Java API, it is more convenient to allow the programmer to use the standard Java array or string types within the class definition and to use the @Blob annotation to inform eXtremeDB that this field should be stored as a BLOB in the database.

@Blob parameters: none

Target: field

Example:

       @Blob       byte body[]; 

This approach stores a field’s value in the database as a BLOB

Challenge 3: Managing in-memory, on-disk and combined (hybrid) data storage
A traditional DBMS caches data temporarily in memory, but eventually writes updated records through to disk. eXtremeDB was developed as an in-memory database system (IMDS) that stores records in main memory, where they can be accessed directly by the application. IMDSs never go to disk, with  the goal (and result) of accelerating performance by eliminating various types of overhead including disk and file I/O, cache processing, and data transfer.

McObject later introduced eXtremeDB Fusion, a hybrid DBMS that supports in-memory and on-disk storage in the same database instance. This data storage flexibility enables developers to make tradeoffs that can affect performance, persistence, hardware cost and even form factor. Hybrid storage is also useful when a product line includes some units with hard disks, and others without. In this scenario (common in digital TV set-top boxes, for example) a hybrid DBMS can make it much easier to port software code across the product line.

With eXtremeDB Fusion's native database definition language, specifying one set of data as transient (managed in memory), while choosing on-disk storage for other record types, requires a simple schema declaration. The Java Native Interface accomplishes the same thing – and more – with the @Persistent annotation, which enables the developer to “toggle” between on-disk and in-memory storage, a large or compact data layout, and to implement other settings that override the default settings for storage.

@Persistent parameters:

boolean list() default false: create a list index for this class
boolean autoid() default false: assign an AUTOID to the instances of this class
boolean disk() default false: this class is stored on disk
boolean inmemory() default false: this class is in-memory only
boolean compact() default false: use the compact layout for this class (the size of an instance of this class cannot be larger than 64Kb)
boolean large() default false: use the standard layout for this class (the size of instance of this class can be larger than 64Kb)

Note:  inmemory()/ondisk() and compact()/large() serve to override the Database.Parameters class values for compactClassesByDefault and diskClassesByDefault.

Target: class

Example:

       @Persistent(disk=true, list=true)       class MyClass {           ...       }

Conclusion

Together, Java’s reflection and annotations provide powerful building blocks for a database interface. In the case of eXtremeDB JNI, a relatively small set of annotations (there are nine in all) leverages virtually all the DBMS's features. Though the database is written in C and accessed principally, until now, from C and C++ applications, the JNI enables the Java developer to stay within the Java environment and eliminates the need to create and process an external database schema file, or to write serialization code (with its attendant risk of introducing errors). Meanwhile, the Java application gains a distinct performance advantage from database commands that execute with the speed of compiled C code rather than interpreted Java.

The benefits are compelling enough to look for other database access challenges where this general approach might apply. One answer lies in providing access to a DBMS such as eXtremeDB from .NET, which supports reflection and, as mentioned above, offers an analog to annotations, called attributes. Unlike Java, which has its special Java Native API for interaction between Java virtual machine and native code, .NET allows almost any external function to be called. This might seem to simplify creation of such an interface, but the work would also likely involve a trip outside the "safe" realm of .NET managed code, and into unmanaged code’s pointer arithmetic, direct access to memory, and other "riskier" aspects. Building the interface would require familiarity with .NET's rules of method declaration and invocation, yet significant amounts of the JNI code could likely be re-used. Given the large and growing number of developers focusing on .NET and C#, this seems a promising direction for a DBMS seeking to increase its presence in Windows applications.