Warning – People know me as a cloud guy, it’s unusual for me to get excited by hardware, but there’s something in this story that goes beyond simple hardware but instead delivers on Marc Andreeson’s now famous proclamation that software is eating the world. Its software differentiated hardware, and it’s a good example of why hardware isn’t completely dead yet.
At HP Discover (disclosure, HP contributed to my T&E for the event), the company introduced StoreAll, a highly scalable storage platform for unstructured data. Now I generally leave storage commentary to others more enamored with the field, but what excited me about the announcement was how it so well showed the smart use of hardware technologies, the great integration of acquired technologies, and the application of pure research.
So what is StoreAll and why is it a seeming awesome trifecta? From the release:
HP StoreAll Storage provides a highly scalable platform for unstructured data. It offers converged object and file storage services in a hyperscalable platform which grows to accommodate up to 16 petabytes (PB) of data. This new system supports billions of objects and files in a single name space, providing clients with a platform to manage big data retention and cloud storage needs without extra administrators or additional hardware. The platform includes HP StoreAll Express Query, a breakthrough embedded metadata database technology created by HP Labs, the company’s central research arm, that allows organizations to locate files and perform file system analytics up to 100,000 times faster than previous file system search methods. In addition, HP StoreAll Express Query IDOL connector integrates with HP Autonomy Intelligent Data Operating Layer (IDOL) to streamline the processing of dynamic content across large data sets. HP StoreAll Express Query’s accelerated file namespace scan delivers inline updates to IDOL-based applications, making it possible to rapidly process new data changes and deliver up to date analytics so that decisions are based on realtime vs. outdated information, while using considerably less compute resources than conventional storage technologies.
OK, so that’s a mouthful, and storage is boring right? So why was it exciting for me?
Let’s look at what the big macro trends evident in the world. The amount of data that is available to organizations is increasing rapidly. At the same time there are economic drivers to mine this data and derive commercial benefit from it. But organizations don’t have the appetite, or the resourcing to bring together the individual parts needed to run these sort of data mining operations. What excited me about this announcement is that it combines the disparate requirements for analysis, rapid assessment, data retention, and physical storage – all in one product.
Let’s look at what went together into this package. Express Query is the result of some research by Alistair Veitch of HP Labs. They started looking at regular databases but found that they didn’t scale and were limiting. What they came up with was a custom system that builds indices and incrementally updates data. Effective algorithms to compress data, storage techniques and the ability to process incoming data without affecting ongoing query work. It’s optimized to write efficiency and this efficiency means it is 3-5 times better on search queries than Cassandra. StoreAll also comes with a connector to Autonomy that streamlines data processing further and enables it to deliver analytics and streamlined processing and review for business processes like eDiscovery and regulatory compliance.
It’s not the hardware that gets me excited about StoreAll, rather it’s the application of smart software that allows the hardware to essentially be “tuned” to specific applications. It’s a way to actually deliver more value from IT kit and, I suspect, it’s the flavor of products that we’re going to see fro combined hardware and software vendors like HP going forward – which most people agree that raw hardware itself is becoming commoditized, the addition of smart software on top of that hardware, gives hardware vendors the ability to further defend against the coming democratization of storage.
Of course there are limits to how much software can really do – and there’s always a tendency to just slap a “software improved” label on anything in an attempt to up-sell it – it seems to be the “with added fiber” of traditional hardware vendors. Paul Miller riffs on this topic, in particular as it relates to HP’s “Autonomy improved” mantra.
The bottom line is that smart software can absolutely improve traditional hardware solutions – vendors just need to make sure they’re selling substantive solutions and not just buzzwords.