SymmetricDS 2.1 Synchronization Tool Released
Retailers depend on SymmetricDS for synchronization between stores and corporate office; Health Care organizations depend on it to support branch locations; and Telecommunication companies use it to enable provisioning products.
The 2.1 release of SymmetricDS includes two new database dialects, 20+ bug fixes and 40+ improvements or new features. Highlighted below are some of the new features.
The newly supported databases are Informix 11 and HSQLDB 2.0.
There have been long awaited improvements to job scheduling. They include using a pool of threads for jobs and allowing jobs to be scheduled via cron expressions. All jobs are now registered as JMX beans and can be started and stopped via JMX calls.
The initial load has been improved by preventing all real time synchronization events from flowing until all reload batches have been processed. In the past, synchronization events could sneak in between reload batches. Optionally, reload batches can be inserted on the channel associated with the table versus using the traditional ‘reload’ channel. There has also been an enhancement when using the initial.load.delete.first parameter; table purging during an initial load can now be customized via a property that allows the purge SQL to be specified. Prior to this version all purging happened via a hard coded delete statement.
A new feature that also helps both initial loads and large batches is the addition of the transport.max.bytes.to.sync property. This is a threshold which, when passed, will end a synchronization at the next batch boundary. This helps free resources for other nodes and also allows an ACK to occur as soon as possible after a big batch has been loaded.
Routing performance has been improved by changing the algorithm that detects data_id gaps to use a new table called SYM_DATA_GAP. Gaps occur when database transactions that capture data in SYM_DATA for synchronization are rolled back. This new table records all gaps that are found in data_ids. The introduction of this table means that the query to select data to route can now select data by data_id without joining to sym_data_event. It also means that data that has already been routed will no longer be selected if an unexpired gap is found in data_id. This feature is not on by default in 2.1. It can be turned by setting routing.data.reader.type to gap.
There have also been improvements to the SYM_OUTGOING_BATCH table. A new column has been introduced that can be used to reliably tell if an ER has occurred. Previously, the only mechanism to discover batches that are in ER was to select batches based on the status. Because the status flips back to NE when a batch is retried this could be unreliable.
Another improvement to SYM_OUTGOING_BATCH is the introduction of new statuses. Previously, the possible statuses were: NE, SE, OK, ER. In 2.1 we have introduced QY and LD. The life cycle of a successfully transferred batch now goes from NE, QY, SE, LD, to OK. QY means that a batch is currently being queried from the source database. SE means that a batch is being sent from the source to the target. And finally, LD means that a batch is being loaded into the target database. Another change in 2.1 was to move a batch all the way from source to target before starting on a new batch. Prior to 2.1 we performed each step for ALL batches that were going to processed during a single synchronization.
One more small improvement to SYM_OUTGOING_BATCH is to not overwrite a batch whose status was manually set to OK with an ER. This could happen in the past to a continuously failing batch that a user wants to mark as OK to get it out of the way. If the sync was in progress, and the batch was marked as OK, it would be overwritten with the actual results of the synchronization. This is no longer the case.
The SYM_STATISTIC and SYM_STATISTIC_ALERT tables are no longer used. Instead, a denormalized view of statistics are written to SYM_NODE_HOST_CHANNEL_STATS. This table captures periodic statistics about data flow by channel and by SymmetricDS instance. This capturing of performance data is still a beta feature. In the future there are plans for SYM_NODE_HOST_JOB_STATS and SYM_NODE_HOST_STATS which will capture additional performance data and statistics jobs and the SymmetricDS instance. All statistics tables have been designed so that they could be trickled back to a central node for reporting.
Infrastructure improvements to the standalone SymmetricDS install include an upgrade of the embedded Jetty web server to version 7 and an upgrade of the Spring Framework to version 3.0.4.
SymmetricDS command lines options have been improved to give more control over logging. A debug mode has been added, and options have been provided to allow console and file logging to be turned off. A command line option has also been added to simplify the creation of a war file for deployment to another application server.
A road map for future SymmetricDS releases is available via our JIRA issue tracker. We encourage the community to log feature requests and issues. Support for SymmetricDS is available via the product forums or from JumpMind, Inc.