NewSQL for a Cause

it started with an idea.

nearly three years ago, walid darwish and some friends and colleagues (who happened to be fellow developers) noticed something among themselves.

"we all did work for not-for-profit organizations," he says.

they all had causes that they donated time and money to in their daily lives.

but being developers, it wasn't enough to just volunteer on the weekends. they wanted to put their skills to work, too. so, after taking about a year and a half to consolidate their thoughts, they banded together under a new banner — causesquare , available on google play and amazon and coming soon to the app store.

the premise

"you can look at it as a marketplace," darwish says. "if you have an interest in certain causes, such as human rights or animal rights, and you want to find people who actually do work — actual work — to promote that cause, who do advocacy work for it, you go to causesquare."

charitable organizations can register on the app and put up news, videos, campaign updates, all that good stuff. meanwhile, users can search those organizations to find those that interest them. with that done, they can donate their money or time (through volunteering) via the app itself. if they want, they can also share their causes' updates through social media to help spread the word.

"you can start engaging on your own time," darwish says.

it's a win-win.

a year and a half after co-founding the company, darwish proudly remains cto of causesquare. but anyone who's worked with data can imagine the sheer amount of information getting thrown around.

the tools

"when you talk about marketplaces, the amount of data is huge," he says. "you can't use traditional databases."

the idea of one database, one machine, just plain doesn't work in this situation. knowing that the amount of data would skyrocket by the day, the team knew their options were being limited for them.

after all, interacting with the network they were planning on putting together was a mammoth task. "are you streaming? how much are you streaming? what device are you using?" darwish says, reiterating the questions his team had to consider.

fortunately, in the world of november 2015, there was an answer: distributed databases.

and there was certainly no shortage of options. sure, traditional databases were out, but there were still plenty of alternatives. mongodb , couchbase , and apache cassandra got looks, but in the end, the team wanted to stay within the sql realm.

a standard sql interface was important for a couple of reasons. first, it's familiar. anyone who has worked with databases probably already knows that. and while nosql has certainly seen a resurgence in the past decade, the skillsets are different, and it would have required a completely different development process. then there was the eventual consistency so many databases are using. given that the team was hoping charities would be using this for donations and scheduling, eventual consistency wasn't something they could wait for. overall, the nosql outlets added up to more specialized knowledge and higher costs, which the team wanted to avoid. meanwhile, as mentioned, the database had to be capable of dealing with big data underpinning.

"so, we decided to narrow our search to these types of databases," darwish says.

in the end, there were two contenders: nuodb and hp vertica .

nuodb ( docs ) was founded in 2008 as nimbusdb. as its old name implied, it was designed to play nicely in the cloud. meanwhile, hp vertica ( docs ) is a column store that meshes nicely with hadoop and other aspects of big data architecture.

both databases were promising, but much like in highlander, there could be only one.

of course, the sql interface was a big hit, and both of those had it. but what else was the team looking for? well, scaling, for one. both databases were built to handle big datasets, but darwish says nuodb's cloud-ready infrastructure just fit better. the built-in horizontal scaling ability was a big hit with causesquare's developers, as was its transactional system.

darwish had also been keeping his eye on nuodb for some time. it was a contender for another project he had been working on several years before causesquare, but it hadn't quite matured to where his company needed it to be.

this time, though, he and the team were ready to pull the trigger. "nuodb saved us a lot of development," he says.

but of course, making the decision is just part one. then, there's the actual implementation.

fortunately, their homework paid off. newsql was not only familiar and accessible to the team, but nuodb, specifically, seemed to play well with everything causesquare threw at it.

"when we built our application, we all came from different backgrounds, but everything fit very nicely," darwish says. "we never had a problem where we said, 'you know, nuodb just can't do that.'"

one of the most helpful aspects is the database's variety of access. the team uses a hybrid node from database aspects, and they need to access it with both jdbc and jpa — using eclipselink. fortunately for them, there have been no problems with that.

the future

of course, the golden rule of databases is that no one database is perfect for every solution. in this case, newsql was the way one company opted to go — while considering a variety of other potential, perfectly workable databases.

but while oracle, among others, might rule the db roost, darwish is convinced the trend is moving toward a more distributed environment.

this is gartner's magic quadrant for operational database management systems from last year. as you can see, nuodb is firmly within the visionary quadrant, as is memsql, another distributed, newsql solution. voltdb, yet another, is creeping toward that same quadrant. and that's not even considering the variety of nosql solutions up for consideration.

darwish says this is to be expected, positing about nuodb in particular, "they’re a visionary now, but i wouldn’t be surprised that in a very few years, they’ll be a market leader."

the conclusion

darwish probably says it best himself. "it’s building up. we need distributed databases. a single machine doesn’t work anymore."

of course, he's referring to larger, more data-intense projects, but the point stands. nosql's recent resurgence has helped fill gaps left by the sql ecosystem. meanwhile, newsql has risen to help put a familiar face on nosql's far-reaching changes, while simultaneously keeping its better characteristics, particularly the ease of scale, around.

in this case, nuodb provided a sql interface, horizontal scaling, the ability to easily integrate their tools and connections, and acid compliance (with a focus on immediate consistency). that was a winning combination for their stated priorities.

and if you want to see what it all went toward, by all means, check out causesquare. there's no shortage of worthy causes for consideration, if you've got some spare cash or time on your hands. after all, what's a bit of big data without big caring to go with it?