Scott Leberknight

Software Architect at Fortitude Technologies

Reston, US

Joined Mar 2005

About

Scott is a software architect at Fortitude Technologies. He has been developing software for over 20 years using many different technologies. He started a long time ago in a galaxy far away with FORTRAN and C on a VAX and now uses ostensibly more modern ones like Java, Ruby, Groovy, and Python. He dabbles in functional languages like Scala and Clojure, is interested in way too many different technologies to list, and likes solving challenging problems in (hopefully somewhat) pragmatic ways. Scott holds a B.S. in Engineering Science and Mechanics from Virginia Tech, and a Masters in Systems Engineering from the University of Maryland. He has spoken at the No Fluff Just Stuff conferences, various user groups, and conferences like DevIgnition. Scott enjoys spending time with his wife and three children. He also plays and coaches soccer, snowboards, and mountain bikes whenever he can. Despite all that, he (unfortunately) will still not be ready when the zombie apocalypse occurs...

Stats

Reputation:	478
Pageviews:	704.0K
Articles:	10
Comments:	19

Articles
Comments

Articles

Handling Big Data with HBase Part 5: Data Modeling (or, Life Without SQL)

This is the fifth of a series of blogs introducing Apache HBase. In the fourth part, we saw the basics of using the Java API to interact with HBase to create tables, retrieve data by row key, and do table scans. This part will discuss how to design schemas in HBase. HBase has nothing similar to a rich query capability like SQL from relational databases. Instead, it forgoes this capability and others like relationships, joins, etc. to instead focus on providing scalability with good performance and fault-tolerance. So when working with HBase you need to design the row keys and table structure in terms of rows and column families to match the data access patterns of your application. This is completely opposite what you do with relational databases where you start out with a normalized database schema, separate tables, and then you use SQL to perform joins to combine data in the ways you need. With HBase you design your tables specific to how they will be accessed by applications, so you need to think much more up-front about how data is accessed. You are much closer to the bare metal with HBase than with relational databases which abstract implementation details and storage mechanisms. However, for applications needing to store massive amounts of data and have inherent scalability, performance characteristics and tolerance to server failures, the potential benefits can far outweigh the costs. In the last part on the Java API, I mentioned that when scanning data in HBase, the row key is critical since it is the primary means to restrict the rows scanned; there is nothing like a rich query like SQL as in relational databases. Typically you create a scan using start and stop row keys and optionally add filters to further restrict the rows and columns data returned. In order to have some flexibility when scanning, the row key should be designed to contain the information you need to find specific subsets of data. In the blog and people examples we've seen so far, the row keys were designed to allow scanning via the most common data access patterns. For the blogs, the row keys were simply the posting date. This would permit scans in ascending order of blog entries, which is probably not the most common way to view blogs; you'd rather see the most recent blogs first. So a better row key design would be to use a reverse order timestamp, which you can get using the formula (Long.MAX_VALUE - timestamp), so scans return the most recent blog posts first. This makes it easy to scan specific time ranges, for example to show all blogs in the past week or month, which is a typical way to navigate blog entries in web applications. For the people table examples, we used a composite row key composed of last name, first name, middle initial, and a (unique) person identifier to distinguish people with the exact same name, separated by dashes. For example, Brian M. Smith with identifier 12345 would have row key smith-brian-m-12345. Scans for the people table can then be composed using start and end rows designed to retrieve people with specific last names, last names starting with specific letter combinations, or people with the same last name and first name initial. For example, if you wanted to find people whose first name begins with B and last name is Smith you could use the start row key smith-b and stop row key smith-c (the start row key is inclusive while the stop row key is exclusive, so the stop key smith-c ensures all Smiths with first name starting with the letter "B" are included). You can see that HBase supports the notion of partial keys, meaning you do not need to know the exact key, to provide more flexibility creating appropriate scans. You can combine partial key scans with filters to retrieve only the specific data needed, thus optimizing data retrieval for the data access patterns specific to your application. So far the examples have involved only single tables containing one type of information and no related information. HBase does not have foreign key relationships like in relational databases, but because it supports rows having up to millions of columns, one way to design tables in HBase is to encapsulate related information in the same row - a "wide" table design. It is called a "wide" design since you are storing all information related to a row together in as many columns as there are data items. In our blog example, you might want to store comments for each blog. The "wide" way to design this would be to include a column family named comments and then add columns to the comment family where the qualifiers are the comment timestamp; the comment columns would look like comments:20130704142510 and comments:20130707163045. Even better, when HBase retrieves columns it returns them in sorted order, just like row keys. So in order to display a blog entry and its comments, you can retrieve all the data from one row by asking for the content, info, and comments column families. You could also add a filter to retrieve only a specific number of comments, adding pagination to them. The people table column families could also be redesigned to store contact information such as separate addresses, phone numbers, and email addresses in column families allowing all of a person's information to be stored in one row. This kind of design can work well if the number of columns is relatively modest, as blog comments and a person's contact information would be. If instead you are modeling something like an email inbox, financial transactions, or massive amounts of automatically collected sensor data, you might choose instead to spread a user's emails, transactions, or sensor readings across multiple rows (a "tall" design) and design the row keys to allow efficient scanning and pagination. For an inbox the row key might look like - which would permit easily scanning and paginating a user's inbox, while for financial transactions the row key might be -. This kind of design can be called "tall" since you are spreading information about the same thing (e.g. readings from the same sensor, transactions in an account) across multiple rows, and is something to consider if there will be an ever-expanding amount of information, as would be the case in a scenario involving data collection from a huge network of sensors. Designing row keys and table structures in HBase is a key part of working with HBase, and will continue to be given the fundamental architecture of HBase. There are other things you can do to add alternative schemes for data access within HBase. For example, you could implement full-text searching via Apache Lucene either within rows or external to HBase (search Google for HBASE-3529). You can also create (and maintain) secondary indexes to permit alternate row key schemes for tables; for example in our people table the composite row key consists of the name and a unique identifier. But if we desire to access people by their birth date, telephone area code, email address, or any other number of ways, we could add secondary indexes to enable that form of interaction. Note, however, that adding secondary indexes is not something to be taken lightly; every time you write to the "main" table (e.g. people) you will need to also update all the secondary indexes! (Yes, this is something that relational databases do very well, but remember that HBase is designed to accomodate a lot more data than traditional RDBMSs were.) Conclusion to Part 5 In this part of the series, we got an introduction to schema design in HBase (without relations or SQL). Even though HBase is missing some of the features found in traditional RDBMS systems such as foreign keys and referential integrity, multi-row transactions, multiple indexes, and son on, many applications that need inherent HBase benefits like scaling can benefit from using HBase. As with anything complex, there are tradeoffs to be made. In the case of HBase, you are giving up some richness in schema design and query flexibility, but you gain the ability to scale to massive amounts of data by (more or less) simply adding additional servers to your cluster. In the next and last part of this series, we'll wrap up and mention a few (of the many) things we didn't cover in these introductory blogs. References HBase web site, http://hbase.apache.org/ HBase wiki, http://wiki.apache.org/hadoop/Hbase HBase Reference Guide http://hbase.apache.org/book/book.html HBase: The Definitive Guide, http://bit.ly/hbase-definitive-guide Google Bigtable Paper, http://labs.google.com/papers/bigtable.html Hadoop web site, http://hadoop.apache.org/ Hadoop: The Definitive Guide, http://bit.ly/hadoop-definitive-guide Fallacies of Distributed Computing, http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing HBase lightning talk slides, http://www.slideshare.net/scottleber/hbase-lightningtalk Sample code, https://github.com/sleberknight/basic-hbase-examples

Updated October 11, 2022

· 19,345 Views · 3 Likes

How to Test HTTP Clients Using the Spark Micro-Framework

Spinning up heavyweight web or application servers adds complexity and slows tests down, but using the Spark micro-framework can make things easier and speed things up.

December 9, 2016

· 11,090 Views · 1 Like

Towards More Functional Java Using Lambdas as Predicates

This series on Java 8 functional features and functional style reviews the use of lambdas and the importance of separation of concerns.

September 18, 2016

· 17,973 Views · 29 Likes

Handling Big Data with HBase Part 4: The Java API

Editor's note: Be sure to check out part 2 as well. This is the fourth of an introductory series of blogs on Apache HBase. In the third part, we saw a high level view of HBase architecture . In this part, we'll use the HBase Java API to create tables, insert new data, and retrieve data by row key. We'll also see how to setup a basic table scan which restricts the columns retrieved and also uses a filter to page the results. Having just learned about HBase high-level architecture, now let's look at the Java client API since it is the way your applications interact with HBase. As mentioned earlier you can also interact with HBase via several flavors of RPC technologies like Apache Thrift plus a REST gateway, but we're going to concentrate on the native Java API. The client APIs provide both DDL (data definition language) and DML (data manipulation language) semantics very much like what you find in SQL for relational databases. Suppose we are going to store information about people in HBase, and we want to start by creating a new table. The following listing shows how to create a new table using the HBaseAdmin class. Configuration conf = HBaseConfiguration.create(); HBaseAdmin admin = new HBaseAdmin(conf); HTableDescriptor tableDescriptor = new HTableDescriptor(TableName.valueOf("people")); tableDescriptor.addFamily(new HColumnDescriptor("personal")); tableDescriptor.addFamily(new HColumnDescriptor("contactinfo")); tableDescriptor.addFamily(new HColumnDescriptor("creditcard")); admin.createTable(tableDescriptor); The people table defined in preceding listing contains three column families: personal, contactinfo, and creditcard. To create a table you create an HTableDescriptor and add one or more column families by adding HColumnDescriptor objects. You then call createTable to create the table. Now we have a table, so let's add some data. The next listing shows how to use the Put class to insert data on John Doe, specifically his name and email address (omitting proper error handling for brevity). Configuration conf = HBaseConfiguration.create(); HTable table = new HTable(conf, "people"); Put put = new Put(Bytes.toBytes("doe-john-m-12345")); put.add(Bytes.toBytes("personal"), Bytes.toBytes("givenName"), Bytes.toBytes("John")); put.add(Bytes.toBytes("personal"), Bytes.toBytes("mi"), Bytes.toBytes("M")); put.add(Bytes.toBytes("personal"), Bytes.toBytes("surame"), Bytes.toBytes("Doe")); put.add(Bytes.toBytes("contactinfo"), Bytes.toBytes("email"), Bytes.toBytes("john.m.doe@gmail.com")); table.put(put); table.flushCommits(); table.close(); In the above listing we instantiate a Put providing the unique row key to the constructor. We then add values, which must include the column family, column qualifier, and the value all as byte arrays. As you probably noticed, the HBase API's utility Bytes class is used a lot; it provides methods to convert to and from byte[] for primitive types and strings. (Adding a static import for the toBytes() method would cut out a lot of boilerplate code.) We then put the data into the table, flush the commits to ensure locally buffered changes take effect, and finally close the table. Updating data is also done via the Put class in exactly the same manner as just shown in the prior listing. Unlike relational databases in which updates must update entire rows even if only one column changed, if you only need to update a single column then that's all you specify in the Put and HBase will only update that column. There is also a checkAndPut operation which is essentially a form of optimistic concurrency control - the operation will only put the new data if the current values are what the client says they should be. Retrieving the row we just created is accomplished using the Get class, as shown in the next listing. (From this point forward, listings will omit the boilerplate code to create a configuration, instantiate the HTable, and the flush and close calls.) Get get = new Get(Bytes.toBytes("doe-john-m-12345")); get.addFamily(Bytes.toBytes("personal")); get.setMaxVersions(3); Result result = table.get(get); The code in the previous listing instantiates a Get instance supplying the row key we want to find. Next we use addFamily to instruct HBase that we only need data from the personal column family, which also cuts down the amount of work HBase must do when reading information from disk. We also specify that we'd like up to three versions of each column in our result, perhaps so we can list historical values of each column. Finally, calling get returns a Result instance which can then be used to inspect all the column values returned. In many cases you need to find more than one row. HBase lets you do this by scanning rows, as shown in the second part which showed using a scan in the HBase shell session. The corresponding class is the Scan class. You can specify various options, such as the start and ending row key to scan, which columns and column families to include and the maximum versions to retrieve. You can also add filters, which allow you to implement custom filtering logic to further restrict which rows and columns are returned. A common use case for filters is pagination. For example, we might want to scan through all people whose last name is Smith one page (e.g. 25 people) at a time. The next listing shows how to perform a basic scan. Scan scan = new Scan(Bytes.toBytes("smith-")); scan.addColumn(Bytes.toBytes("personal"), Bytes.toBytes("givenName")); scan.addColumn(Bytes.toBytes("contactinfo"), Bytes.toBytes("email")); scan.setFilter(new PageFilter(25)); ResultScanner scanner = table.getScanner(scan); for (Result result : scanner) { // ... } In the above listing we create a new Scan that starts from the row key smith- and we then use addColumn to restrict the columns returned (thus reducing the amount of disk transfer HBase must perform) to personal:givenName and contactinfo:email. A PageFilter is set on the scan to limit the number of rows scanned to 25. (An alternative to using the page filter would be to specify a stop row key when constructing the Scan.) We then get a ResultScanner for the Scan just created, and loop through the results performing whatever actions are necessary. Since the only method in HBase to retrieve multiple rows of data is scanning by sorted row keys, how you design the row key values is very important. We'll come back to this topic later. You can also delete data in HBase using the Delete class, analogous to the Put class to delete all columns in a row (thus deleting the row itself), delete column families, delete columns, or some combination of those. Connection Handling In the above examples not much attention was paid to connection handling and RPCs (remote procedure calls). HBase provides the HConnection class which provides functionality similar to connection pool classes to share connections, for example you use the getTable() method to get a reference to an HTable instance. There is also an HConnectionManager class which is how you get instances of HConnection. Similar to avoiding network round trips in web applications, effectively managing the number of RPCs and amount of data returned when using HBase is important, and something to consider when writing HBase applications. Conclusion to Part 4 In this part we used the HBase Java API to create a people table, insert a new person, and find the newly inserted person information. We also used the Scan class to scan the people table for people with last name "Smith" and showed how to restrict the data retrieved and finally how to use a filter to limit the number of results. In the next part, we'll learn how to deal with the absence of SQL and relations when modeling schemas in HBase. References HBase web site, http://hbase.apache.org/ HBase wiki, http://wiki.apache.org/hadoop/Hbase HBase Reference Guide http://hbase.apache.org/book/book.html HBase: The Definitive Guide, http://bit.ly/hbase-definitive-guide Google Bigtable Paper, http://labs.google.com/papers/bigtable.html Hadoop web site, http://hadoop.apache.org/ Hadoop: The Definitive Guide, http://bit.ly/hadoop-definitive-guide Fallacies of Distributed Computing, http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing HBase lightning talk slides, http://www.slideshare.net/scottleber/hbase-lightningtalk Sample code, https://github.com/sleberknight/basic-hbase-examples

December 18, 2013

· 56,261 Views · 3 Likes

Capture a Signature on iOS

Originally authored by Jason Harwig The Square Engineering Blog has a great article on Smoother Signatures for Android, but I didn't find anything specifically about iOS. So, what is the best way to capture a users signature on an iOS device? Although I didn't find any articles on signature capture, there are good implementations on the App Store. My target user experience was the iPad application Paper by 53, a drawing application with beautiful and responsive brushes. All code is available in the Github repository: SignatureDemo. Connecting the Dots The simplest approach is to capture the touches and connect them with straight lines. In the initializer of a UIView subclass, create the path and gesture recognizer to capture touch events. // Create a path to connect lines path = [UIBezierPath bezierPath]; // Capture touches UIPanGestureRecognizer *pan = [[UIPanGestureRecognizer alloc] initWithTarget:self action:@selector(pan:)]; pan.maximumNumberOfTouches = pan.minimumNumberOfTouches = 1; [self addGestureRecognizer:pan]; Capture the pan events into a bézier path by connecting the points with lines. - (void)pan:(UIPanGestureRecognizer *)pan { CGPoint currentPoint = [pan locationInView:self]; if (pan.state == UIGestureRecognizerStateBegan) { [path moveToPoint:currentPoint]; } else if (pan.state == UIGestureRecognizerStateChanged) [path addLineToPoint:currentPoint]; [self setNeedsDisplay]; } Stroke the path - (void)drawRect:(CGRect)rect { [[UIColor blackColor] setStroke]; [path stroke]; } An example "J" character rendered using this technique reveals some issues. At slow velocities iOS captures enough touch resolution that the lines aren't noticeable, but faster movement shows large gaps between touches that accentuates the lines. The 2012 Apple Developer Conference included a session Building Advanced Gesture Recognizers that addresses this issue using math. Quadratic Bézier Curves Instead of connected lines between the touch points, quadratic bézier curves connect the points using the technique discussed in the aforementioned WWDC session (Seek to 42:15.) Connect the touch points with a quadratic curve using the touch points as the control points and the mid points as start and end. Adding quadratic curves to the previous code requires the storing the previous touch point, so add an instance variable for that. CGPoint previousPoint; Create a function to calculate the midpoint of two points. static CGPoint midpoint(CGPoint p0, CGPoint p1) { return (CGPoint) { (p0.x + p1.x) / 2.0, (p0.y + p1.y) / 2.0 }; } Update the pan gesture handler to add quadratic curves instead of straight lines - (void)pan:(UIPanGestureRecognizer *)pan { CGPoint currentPoint = [pan locationInView:self]; CGPoint midPoint = midpoint(previousPoint, currentPoint); if (pan.state == UIGestureRecognizerStateBegan) { [path moveToPoint:currentPoint]; } else if (pan.state == UIGestureRecognizerStateChanged) { [path addQuadCurveToPoint:midPoint controlPoint:previousPoint]; } previousPoint = currentPoint; [self setNeedsDisplay]; } Not much code and already we see a big difference. The touch points are no longer visible, but it looks a little bland when drawing a signature. Every curve is the same width, which doesn't match the physics of a real pen. Variable Stroke Width The width can be varied based on the touch velocity to create a more natural stroke. The UIPanGestureRecognizer already includes a method called velocityInView: that returns the current touch velocity as a CGPoint. To render a stroke of varying width, I switched to OpenGL ES and a technique called tesselation to convert the stroke into triangles – specifically, triangle strips (OpenGL has support for drawing lines, but iOS doesn't support variable line widths with smoothing.) The quadratic points along a curve also need to be calculated, but is beyond the scope of this article. Check the source on github for details. Given two points, a perpendicular vector is calculated and its magnitude set to half the current thickness. Given the nature of GL_TRIANGLE_STRIP only two points are needed to create the next rectangle segment with two triangles. Here is an example of the final output using quadratic bézier curves, and velocity based stroke thickness creating a visually appealing and natural signature.

April 8, 2013

· 19,964 Views

Reading Hive Tables from MapReduce

This article is by Stephen Mouring Jr, appearing courtesy of Scott Leberknight. This is part two of a two part blog series on how to read/write Apache Hive data from MapReduce jos. Part one (Writing Hive Tables from MapReduce) is here. So just as sometimes you need to write data to Hive with a custom MapReduce job, sometimes you need to read that data back from Hive with a custom MapReduce job. As covered in part one, Hive is a layer that sits on HDFS and imposes a standard convention on the structure of the files so it can interpret them as columns and rows. Reading data out of Hive is just a matter of parsing the files correctly. Recall that files processed by MapReduce (and by extension, Hive) are output as key value pairs. Hive ignores the keys (read as a BytesWritable with a value of null) and reads/writes the values as Text objects. The value of the Text object for each row is the concatenation of all the column values delimited by the delimiter of the table (which Hive defaults to the "char 1" ASCII character). Seems like a simple problem, so my first thought was to just using String.split() in the map() method of the MapReduce job. String SEPARATOR_FIELD = new String(new char[] {1}); String[] rowColumns = new String (rowTextObject.getBytes()).split(SEPARATOR_FIELD); In theory this should have worked perfectly, but unfortunately I have found that String.split() actually consumes repeated delimiters. This is a problem if any of the values in the row are blank, since split() will shift the positions of your columns and you will be unable to match up what values belong with which columns. An alternative would be to create a String from the Text object and iterate through it using indexOf(). This approach however requires extra object creation and depending on the scale of your MapReduce job and the size of your rows, may slow you down needlessly. So an alternative is to use the Text object's find() method. String SEPARATOR_FIELD = new String(new char[] {1}); String[] rowColumns = new String[NUMBER_OF_COLUMNS_IN_YOUR_HIVE_TABLE]; int start = 0; int end = 0; for (int i = 0; i < rowColumns.length; ++i) { end = rowTextObject.find(SEPARATOR_FIELD, start); if (end == -1) { end = rowString.getLength(); } rowColumns[i] = new String(rowTextObject.getBytes(), start, end-start); start = end + 1; } This will parse out each value into the appropriately index of the rowColumns array. Blank values will also be handled correctly and result in blank strings being inserted into the rowColumns array.

January 11, 2013

· 6,178 Views · 1 Like

Distributed Lock using Zookeeper

This article is by Stephen Mouring, Jr. On my project we have a number of software components that run concurrently, some on a cron, and some as part of our build process. Many of these components need to mutate data in our data store and have the possibility of conflicting with one another. What is worse is that many of these processes run on separate machines making language level or even file system level synchronization impossible. Zookeeper is a natural solution to the problem. It is a distributed system for, among other things, managing coordination across a cluster of machines. Zookeeper manages information as a hierarchical system of "nodes" (much like a file system). Each node can contain data or can contain child nodes. Zookeeper supports several types of nodes. A node can be either "ephemeral" or "persistent" meaning it is either deleted when the process that created it ends or it remains until manually deleted. A node can also be "sequential" meaning each time a node is created with a given name, a sequence number is postfixed to that name. This allows you to create a series of nodes with the same name that are ordered in the same order they were created. To solved our problem we need to have a locking mechanism that works across processes and across machines that allows one holder of the lock to execute at a given time. Below is the Java code we wrote to solve the problem. I will go through it step by step. public class DistributedLock { private final ZooKeeper zk; private final String lockBasePath; private final String lockName; private String lockPath; public DistributedLock(ZooKeeper zk, String lockBasePath, String lockName) { this.zk = zk; this.lockBasePath = lockBasePath; this.lockName = lockName; } public void lock() throws IOException { try { // lockPath will be different than (lockBasePath + "/" + lockName) becuase of the sequence number ZooKeeper appends lockPath = zk.create(lockBasePath + "/" + lockName, null, Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL); final Object lock = new Object(); synchronized(lock) { while(true) { List nodes = zk.getChildren(lockBasePath, new Watch() { @Override public void process(WatchedEvent event) { synchronized (lock) { lock.notifyAll(); } } }); Collections.sort(nodes); // ZooKeeper node names can be sorted lexographically if (lockPath.endsWith(nodes.get(0)) { return; } else { lock.wait(); } } } } catch (KeeperException e) { throw new IOException (e); } catch (InterruptedException e) { throw new IOException (e); } } public void unlock() throws IOException { try { zk.delete(lockPath, -1); lockPath = null; } catch (KeeperException e) { throw new IOException (e); } catch (InterruptedException e) { throw new IOException (e); } } } (Disclaimer: Credit for this code goes to Aaron McCurry for developing the core mechanism of this lock as well as the design for using ZooKeeper. Kudos to Aaron!) Each process that wants to use the lock should instantiate an object of the DistributedLock class. The DistributedLock constructor takes three parameters. The first parameter is a reference to the ZooKeeper client. The second parameter is the "base path" where you want your lock nodes to reside in. Remember that ZooKeeper stores its nodes like a file system, so think of this base path as the directory you want your lock nodes created in. The third parameter is the name of the lock to use. Note you should use the same lock name for every process that you want to share the same lock. The lock name is the common reference that multiple processes lock on. Note: This class can support multiple locks if you use a different lock name for each lock you want to create. Say you have two data stores (A and B). You have several processes that need mutate A and B. You could use two different lock names (say LockA and LockB) to represent the locks for each data store. Any process that needs to mutate data store A could create a DistributedLock with a lockname of LockA. Likewise, any process that needs to mutate data store B could create a DistributedLock with a lockname of LockB. A proces that needs to mutate both datastores would create two DistributedLock objects (one with lock name of LockA and one with a lock name of LockB). Once your process has created a DistributedLock object it can then call the lock() method to attempt to acquire the lock. The lock() method will block until the lock is acquired. // lockPath will be different than (lockBasePath + "/" + lockName) becuase of the sequence number ZooKeeper appends lockPath = zk.create(lockBasePath + "/" + lockName, null, Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL); First of all, the lock() method creates a node in ZooKeeper to represent its "position in line" waiting for the lock. The node created is EPHEMERAL which means if our process dies for some reason, its lock or request for the lock with automatically disappear thanks to ZooKeeper's node management, so we do not have worry about timing out nodes or cleaning up stale nodes. final Object lock = new Object(); synchronized(lock) { while(true) { List nodes = zk.getChildren(lockBasePath, new Watch() { @Override public void process(WatchedEvent event) { synchronized (lock) { lock.notifyAll(); } } }); // Sequential ZooKeeper node names can be sorted lexographically! Collections.sort(nodes); // Are we the "topmost" node? (The node with the lowest sequence number that is.) if (lockPath.endsWith(nodes.get(0)) { return; } else { lock.wait(); } } } To understand the code above you need to understand how ZooKeeper works. ZooKeeper operates through a system of callbacks. When you call getChildren() you can pass in a "watcher" that will get called anytime the list of children changes. The gist of what we are doing here is this. We are creating an ordered list of nodes (sharing the same name). Whenever the list changes, every process that has registered a node is notified. Since the nodes are ordered, one node will be "on top" or in other words have the lowest sequence number. That node is the node that owns the lock. When a process detects that its node is the top most node, it proceeds to execute. When it is finished, it deletes its node, triggering a notification to all other processes who then determine who the next node is who has the lock. The tricky part of the code from a Java perspective is the use of nested synchronized blocks. The nested synchronization structure is used to ensure that the DistributedLock is able to process every update it gets from ZooKeeper and does not "lose" an update if two or more updates come from ZooKeeper in quick succession. The inner synchronized block in the Watcher method is called from an outside thread whenever ZooKeeper reports a change to its children. Since the Watcher callback is in a synchronized block keyed to the same Java lock object as the outer synchronized block, it means that the update from ZooKeeper cannot be processed until the contents of the outer synchronized block is finished. In other words, when an update comes in from ZooKeeper, it fires a notifyAll() which wakes up the loop in the lock() method. That lock method gets the updated children and sets a new Watcher. (Watchers have to be reset once they fire as they are not a perpetual callback. They fire once and then disappear.) If the newly reset Watcher fires before the rest of the loop executes, it will block because it is synchronized on the same Java lock object as the loop. The loop finishes its pass, and if it has not acquired the distrubted lock, it waits on the Java lock object. This frees the Watcher to execute whenever a new update comes, repeating the cycle. Once the lock() method returns, it means your process has the dsitributed lock and can continue to execute its business logic. Once it is complete it can release the lock by calling the unlock() method. public void unlock() throws IOException { try { zk.delete(lockPath, -1); lockPath = null; } catch (KeeperException e) { throw new IOException (e); } catch (InterruptedException e) { throw new IOException (e); } } All unlock() does is explictly delete this process's node which notifies all the other waiting processes and allows the next one in line to go. Because the nodes are EPHEMERAL, the process can exit without unlocking and ZooKeeper will eventually reap its node allowing the next process to execute. This is a good thing because it means if your process ends prematurely without you having a chance to call unlock() it will not block the remaining processes. Note that it is best to explicitly call unlock() if you can, because it is much faster than waiting for ZooKeeper to reap your node. You will delay the other processes less if you explicity unlock.

January 8, 2013

· 60,370 Views · 5 Likes

Groovy, Sometimes You Still Need a Semicolon.

Like Javascript, semicolons are optional in Groovy except for when they aren't optional. These examples are both pretty contrived, though I found both because they're actually something that I've written, and could both be written better. That's not really the point I'm making though. When something doesn't compile when it looks like it clearly should sometimes it's hard to track down why, and it's surprising to learn that it's because you need a semicolon. Example the first: Generics at the end of a line: def list = [1,2,3] as List println list If you try to compile this in Groovy it will give you the error message: 'unexpected token: println', however this: def list = [1,2,3] as List; println list Gives the expected output. Example the second: Ambiguous Closures {-> assert GroovyClosureTest == owner.getClass() }() {-> assert GroovyClosureTest == delegate.getClass() }() I don't think you'd really ever need to do something like this, but a closure can be defined and called on a single line. Because of Groovy's special closure parameter syntax (e.g. list.each() {} being synonomous with list.each({})) the compiler thinks I'm passing the second closure into the first as an argument. Again a semicolon is needed to seperate the two lines: {-> assert GroovyClosureTest == owner.getClass() }(); {-> assert GroovyClosureTest == delegate.getClass() }() From

November 6, 2009

· 30,549 Views · 1 Like

Sorting Collections in Hibernate Using SQL in @OrderBy

When you have collections of associated objects in domain objects, you generally want to specify some kind of default sort order. For example, suppose I have domain objects Timeline and Event: @Entity class Timeline { @Required String description @OneToMany(mappedBy = "timeline") @javax.persistence.OrderBy("startYear, endYear") Set events } @Entity class Event { @Required Integer startYear Integer endYear @Required String description @ManyToOne Timeline timeline } In the above example I've used the standard JPA (Java Persistence API) @OrderBy annotation which allows you to specify the order of a collection of objects via object properties, in this example a @OneToMany association . I'm ordering first by startYear in ascending order and then by endYear, also in ascending order. This is all well and good, but note that I've specified that only the start year is required. (The @Required annotation is a custom Hibernate Validator annotation which does exactly what you would expect.) How are the events ordered when you have several events that start in the same year but some of them have no end year? The answer is that it depends on how your database sorts null values by default. Under Oracle 10g nulls will come last. For example if two events both start in 2001 and one of them has no end year, here is how they are ordered: 2001 2002 Some event 2001 2003 Other event 2001 Event with no end year What if you want to control how null values are ordered so they come first rather than last? In Hibernate there are several ways you could do this. First, you could use the Hibernate-specific @Sort annotation to perform in-memory (i.e. not in the database) sorting, using natural sorting or sorting using a Comparator you supply. For example, assume I have an EventComparator helper class that implements Comparator. I could change Timeline's collection of events to look like this: @OneToMany(mappedBy = "timeline") @org.hibernate.annotations.Sort(type = SortType.COMPARATOR, comparator = EventCompator) Set events Using @Sort will perform sorting in-memory once the collection has been retrieved from the database. While you can certainly do this and implement arbitrarily complex sorting logic, it's probably better to sort in the database when you can. So we now need to turn to Hibernate's @OrderBy annotation, which lets you specify a SQL fragment describing how to perform the sort. For example, you can change the events mapping to : @OneToMany(mappedBy = "timeline") @org.hibernate.annotations.OrderBy("start_year, end_year") Set events This sort order is the same as using the JPA @OrderBy with "startYear, endYear" sort order. But since you write actual SQL in Hibernate's @OrderBy you can take advantage of whatever features your database has, at the possible expense of portability across databases. As an example, Oracle 10g supports using a syntax like "order by start_year, end_year nulls first" to order null end years before non-null end years. You could also say "order by start_year, end year nulls last" which sorts null end years last as you would expect. This syntax is probably not portable, so another trick you can use is the NVL function, which is supported in a bunch of databases. You can rewrite Timeline's collection of events like so: @OneToMany(mappedBy = "timeline") @org.hibernate.annotations.OrderBy("start_year, nvl(end_year , start_year)") Set events The expression "nvl(end_year , start_year)" simply says to use end_year as the sort value if it is not null, and start_year if it is null. So for sorting purposes you end up treating end_year as the same as the start_year if end_year is null. In the contrived example earlier, applying the nvl-based sort using Hibernate's @OrderBy to specify SQL sorting criteria, you now end with the events sorted like this: 2001 Event with no end year 2001 2002 Some event 2001 2003 Other event Which is what you wanted in the first place. So if you need more complex sorting logic than what you can get out of the standard JPA @javax.persistence.OrderBy, try one of the Hibernate sorting options, either @org.hibernate.annotations.Sort or @org.hibernate.annotations.OrderBy. Adding a SQL fragment into your domain class isn't necessarily the most elegant thing in the world, but it might be the most pragmatic thing.

September 16, 2009

· 101,414 Views

Using a Hibernate Interceptor To Set Audit Trail Properties

In almost every application I've done, the database tables have some kind of audit trail fields. Sometimes this is a separate "audit log" table where all inserts, updates, deletes, and possibly even queries are logged. Other times there are the four typical audit trail fields in each table, for example you might have created_by, created_on, updated_by, and updated_on fields in each table. The goal in the latter case is to update those four fields with the appropriate information as to who created or updated a record and when they did it. Using a simple Hibernate Interceptor this can be accomplished with no changes to your application code (with several assumptions which I'll detail next). In other words, you won't need to and definitely should not be manually setting those audit properties littered around your application code. The basic assumptions I'll make for this simple audit interceptor are that: (1) model objects contain the four audit properties mentioned above, and (2) there is an easy way to obtain the current user's information from anywhere in the code. The first assumption is needed since you need some way to identify which properties constitute the audit trail properties. The second assumption is required because you need some way to obtain the credentials of the person making the change in order to set the createdBy or updatedBy property in your Hibernate Interceptor class. So, for reference purposes, assume you have a (Groovy) base entity like this with the four audit properties: @MappedSuperclassclass BaseEntity implements Serializable { String createdBy Date createdOn String updatedBy Date updatedOn} I'm using the Hibernate ImprovedNamingStrategy so that camel case names are translated to underscored names, e.g. "createdBy" becomes "created_by". Next assume there is a BlogEntry entity class that extends BaseEntity and inherits the audit trail properties: @Entityclass BlogEntry extends BaseEntity { @Id @GeneratedValue (strategy = GenerationType.IDENTITY) Long id @Version Long version String title @Column (name = "entry_text") String text @Temporal (TemporalType.TIMESTAMP) Date publishedOn} To implement the interceptor, we need to implement the aforementioned Interceptor interface. We could do this directly, but it is better to extend EmptyInterceptor so we need only implement the methods we actually care about. Without further ado, here's the implementation (excluding package declaration and imports): class AuditTrailInterceptor extends EmptyInterceptor { boolean onFlushDirty(Object entity, Serializable id, Object[] currentState, Object[] previousState, String[] propertyNames, Type[] types) { setValue(currentState, propertyNames, "updatedBy", UserUtils.getCurrentUsername()) setValue(currentState, propertyNames, "updatedOn", new Date()) true } boolean onSave(Object entity, Serializable id, Object[] state, String[] propertyNames, Type[] types) { setValue(state, propertyNames, "createdBy", UserUtils.getCurrentUsername()) setValue(state, propertyNames, "createdOn", new Date()) true } private void setValue(Object[] currentState, String[] propertyNames, String propertyToSet, Object value) { def index = propertyNames.toList().indexOf(propertyToSet) if (index >= 0) { currentState[index] = value } } So what did we do? First, we implemented the onFlushDirty and onSave methods because they are called for SQL updates and inserts, respectively. For example, when a new entity is first saved, the onSave method is called, at which point we want to set the createdBy and properties. And if an existing entity is updated, onFlushDirty is called and we set the updatedBy and updatedOn. Second, we are using the setValue helper method to do the real work. Specfically, the only way to modify the state in a Hibernate Interceptor (that I am aware of anyway) is to dig into the currentState array and change the appropriate value. In order to do that, you first need to trawl through the propertyNames array to find the index of the property you are trying to set. For example, if you are updating a blog entry you need to set the updatedBy and updatedOn properties within the currentState array. For a BlogEntry object, the currentState array might look like this before the update (the updated by and on propertes are both null in this case because the entity was created by Bob but has not been updated yet): { "Bob", 2008-08-27 10:57:19.0, null, null, 2008-08-27 10:57:19.0, "Lorem ipsum...", "My First Blog Entry", 0} You then need to look at the propertyNames array to provide context for what the above data represents: { "createdBy", "createdOn", "updatedBy", "updatedOn", "publishedOn", "text", "title", "version"} So in the above updatedBy is at index 2 and updatedOn is located at index 3. setValue() works by finding the index of the property it needs to set, e.g. "updatedBy," and if the property was found, it changes the value at that index in the currentState array. So for updatedBy at index 2, the following is the equivalent code if we had actually hardcoded the implementation to always expect the audit fields as the first four properties (which is obviously not a great idea): // Equivalent hard-coded code to change "updatedBy" in above example// Don't use in production!currentState[2] = UserUtils.getCurrentUsername() To actually make your interceptor do something, you need to enable it on the Hibernate Session. You can do this in one of several ways. If you are using plain Hibernate (i.e. not with Spring or another framework) you can set the interceptor globally on the SessionFactory, or you can enable it for each Session as in the following example code: // Configure interceptor globally (applies to all Sessions)sessionFactory = new AnnotationConfiguration() .configure() .setNamingStrategy(ImprovedNamingStrategy.INSTANCE) .setInterceptor(new AuditTrailInterceptor()) .buildSessionFactory()// Enable per SessionSession session = getSessionFactory().openSession(new AuditTrailInterceptor()) If you enable the interceptor globally, it must be thread-safe. If you are using Spring you can easily configure a global interceptor on your session factory bean: On the other hand, if you would rather enable the interceptor per session, you either need to use the openSession(Interceptor) method to open your sessions or alternatively implement your own version of CurrentSessionContext to use the getCurrentSession() method in order to set the interceptor. Using getCurrentSession() is preferable anyway since it allows several different classes (e.g. DAOs) to use the same session without needing to explicitly pass the Session object around to each object that needs it. At this point we're done. But, if you know about the Hibernate eventing system (e.g. you can listen for events such as inserts and updates and define event listener classes to respond to those events), you might be wondering why I didn't use that mechanism rather than the Interceptor. The reason is that, to the best of my current knowledge, you cannot alter state of objects in event listeners. So for example you would not be able to change an entity's state in a PreInsertEventListener implementation class. If anyone knows this is incorrect or has implemented it, I'd love to hear about it. Until next time, happy auditing! Originally posted on Scott Leberknight's blog

August 27, 2008

· 102,651 Views · 2 Likes

Comments

Redirecting HTTP Requests With Zuul in Spring Boot

Mar 28, 2019 · Jesus J. Puente

Gotcha, thanks for the update. Only reason I posted is that in recent years I have come across Java developers who literally believe that you must use Spring when using Java. Not. Even. Kidding. And while maybe that's great for Spring, I've seen simple projects where Spring was used to wire together 5 or 6 small classes, and nothing else. The junior developer in one situation gave me a blank stare when I said you don't need Spring just to instantiate a few classes and to just manually construct the objects and pass in dependencies via constructors.

Redirecting HTTP Requests With Zuul in Spring Boot

Mar 28, 2019 · Jesus J. Puente

Zuul isn't part of Spring Cloud, it is a standalone Netflix project that Spring Cloud wraps/integrates/etc. and itself has nothing to do with Spring. On GitHub at https://github.com/Netflix/zuul

The Levenshtein Distance Algorithm

Oct 12, 2018 · Harsh Binani

I think this comment is for a different article, not for the Levenshtein Algorithm??? There's nothing in here about microservices...

Java Collections Are Evolving

Jun 06, 2018 · Mike Gates

Wow I never realized the list returned from Arrays.asList() was not immutable...now that I went back and read the JavaDoc it does actually say it returns a "fixed-size list backed by the specified array" and that "changes to the returned list 'write-through' to the array". Learn something every day...thanks for the article!

Distributed Lock using Zookeeper

Dec 16, 2017 · Eric Gregory

I'd suggest taking a look at Apache Curator, which has many "recipes" built in including Leader Election and a Leader Latch. See here: https://curator.apache.org/curator-recipes/index.html

REST - Using Apache Wink (Part 01) [Starting JAX-RS Web Services - 01]

Jan 24, 2017 · Sumith Puri

Also according to a thread in the wink user mailing list from December 2016, it looks like they might simply retire the project ( see http://mail-archives.apache.org/mod_mbox/wink-user/201612.mbox/browser ). So between that and the fact that it supports the now very old JAX-RS 1.1, I'd suggest avoiding this and use Jersey or something else.

REST - Using Apache Wink (Part 01) [Starting JAX-RS Web Services - 01]

Jan 24, 2017 · Sumith Puri

From what I can tell Apacke Wink hasn't had any releases since 2013 and it only implements JAX-RS 1.1, and the official website's last update was in September 2013.

In contrast JAX-RS 2.0 came out around May 2013 and, for example, the current Jersey version 2.25.1 (as of 2017-01-23) supports JAX-RS 2.0. So I'm not sure why anyone would choose to use this now in 2017.

Creating Maps With Named Lambdas

Dec 09, 2016 · Per-Åke Minborg

It is your opinion that it is not very nice, not fact. I don't agree. Plus, I did split the lines but the comment formatter apparently didn't preserve the line breaks. Anyway, for creating small maps, especially in unit tests, it works out pretty nicely and I don't think I've ever heard anyone say it was difficult to read or modify later. It would be better if Java adopted a literal map intitalization syntax like many other modern (or even non-modern) languages do. e.g. in Ruby:

options = { font_size: 10, font_family: "Arial" }

Creating Maps With Named Lambdas

Dec 08, 2016 · Per-Åke Minborg

Pretty cool. For years I've always just created utiliies with a method having the signature:

public static <K, V> Map<K, V> newHashMap(Object... items)

That way I can get whatever key and value type I want, and while not as nice, you can still do something like:

Map<String, Integer> blue = newHashMap( "red", 66, "green", 134, "blue", 244);

Not awesome, but not too horrible either. And I add variants like newLinkedHashMap, newConcurrentMap, newTreeMap, and so on.

Maybe Java will someday get literal map/list syntax..

ReentrantLock Cheat Sheet

Oct 03, 2016 · Thomas Krieger

I do not see an "anotify" method in the Condition interface. Did you maybe mean the signal or signalAll methods? Or am I just missing something, b/c I'm pretty sure there's no anotify method in Condition...

Towards More Functional Java Using Lambdas as Predicates

Sep 19, 2016 · Dave Fecak

Yup that would be another way to do it, and as mentioned in the above comments would be more efficient for large maps. In fact you could make it a method reference and just do:

deleteKeys.forEach(dataCache::remove);

Towards More Functional Java Using Lambdas as Predicates

Sep 19, 2016 · Dave Fecak

Thanks Stefan and Ivan for the good insights, and you are definitely right that you cannot blindly refactor to streams. Ivan is also correct that the second example which only traverses the keys to be deleted is not equivalent from a performance perspective especially in very large maps...thanks for catching that!

For some context, the cache generally contains on the order of a hundred items, and generally more like several dozen, so the actual data set in this case is very small. And, the set of keys to delete is generally just a few, and generally a lot less than the number of items in the cache.

So in this specific case, refactoring to streams should not cause any real, noticeable degradation in performance, and I am pretty sure the new removeIf code is at least as performant as the original code I found (since both the original code and removeIf use an Iterator over the entire map and call remove() on each matching item). But if this were a map containing a million entries then this might not be such a good thing to do...maybe I need to go explore that further and write a follow-up.

Optional Parameters Handling Strategy in Java

Aug 29, 2016 · Shamik Mitra

Lombok can generate builders for you, which makes it trivial to add builders to complex objects. (Of course, some people don't like Lombok, but that's a different conversation.) See https://projectlombok.org/features/Builder.html

OOP is dead

Sep 17, 2009 · Patrick Hunlock

Rob,

Good point about nulls being a royal PITA in general. In the specific case I had, the data was actually not used like that, i.e. it was not searched by date. I changed the actual class names from my real project but they were similar in concept, and the "Events" containing startYear and endYear were really just for reference purposes and were basically metadata for the parent "Timeline" domain object, sort of like a resume where you use dates to delineate previous jobs but you probably don't go around searching resumes by years (or at least I don't).

So to answer your question, since we don't need to search by those dates, using nulls wasn't an issue in that regard. And thus we don't end up having to use nvl or anything else throughout the app b/c this is localized to one domain object. The only logic is that when reports are generated the users wanted the year ranges to look like "2001" when there's no end date and "2001-02" when there is an end date. The display logic was actually perhaps a bit easier with nulls in this case since we don't need to check if the endYear is the same as startYear.

- Scott

Sorting Collections in Hibernate Using SQL in @OrderBy

Sep 17, 2009 · Nate Bergman

Rob,

- Scott

Sorting Collections in Hibernate Using SQL in @OrderBy

Sep 17, 2009 · Nate Bergman

Rob,

- Scott

Sorting Collections in Hibernate Using SQL in @OrderBy

Sep 17, 2009 · Nate Bergman

Rob,

- Scott

OOP is dead

Sep 17, 2009 · Patrick Hunlock

Rob,

- Scott

OOP is dead

Sep 17, 2009 · Patrick Hunlock

Rob,

- Scott