If you follow just about any tech site that discusses storage, you’d be forgiven for thinking that pretty much everyone in the universe has moved to cloud storage. The reality is many of the fortune 500 companies are still housing their own hardware in their own data centers, although nearly all of them have a plan or desire to adopt cloud infrastructure.
Regardless of what you think ‘cloud’ or ‘software defined’ means (which could be an entire post in itself), it is clear that there’s a significant amount of money being spent on “not cloud” storage: big refrigerator-sized boxes that hold data.
Check this out:
Source: IDC Worldwide Quarterly Enterprise Storage Systems Tracker, March 11, 2016
Clearly $10B in a quarter, even if declining, is nothing to shake a stick at. Why is all this investment still going into traditional external storage?
Usually when this question comes up one or more of the following are offered as an explanation (with some associated words):
Reliability: “bullet proof”.. “data integrity”.. “end to end checking”
Availability: “Five.. Six.. Seven 9’s up-time”. “No SPOFs!”
Performance: SPC-1, SPC2, TPC-c, TPC-e, “one million IOPS” (or more)
Serviceability: “Zero downtime maintenance” “Hotswap components”
Disaster Recovery: “Sync/Async replication” “Site Failover Capability”
Just about every storage box sold in the upper price bands of the IDC external storage tracker has a significant amount of engineering towards features in these categories. Similarly, bloggers and analysts usually cite the security concerns of the cloud, or the budgetary challenges associated with moving from cap-ex to op-ex infrastructure costs.
All of this is of course accurate. But in my opinion, the first reason businesses used external storage appliances (i.e. “disks in a box outside of the servers”) and continue to rely on them today: Server and Application scale-out. In the earlier days of “enterprise storage”, the first big change was that the “disk drives” moved into a box outside the server, and applications began to scale across multiple servers. Databases, web server clusters, virtual machine hyper-visor clusters were made possible by the fact that from any server, you could access the storage in one location and know it was the latest up to date copy of that data. As information systems grew to be more complex, and applications flourished in this environment, the “Enterprise Storage” characteristics mentioned above became prevalent out of necessity. Basically: once your data was all in one basket, that basket evolved via continual improvement in reliability, availability, performance, serviceability, and — because the data was becoming so important to the business — disaster recovery capabilities.
Cloud infrastructure promises to disrupt this paradigm, but when I hear about “moving” to the cloud, I always keep in mind that many of the applications with the highest value data today still depend on the hub-and-spoke idea for storage and clustering.
Of course cloud infrastructure providers can offer the same storage access semantics, but typically they don’t. If you look at what you can buy on AWS, you can get all kinds of slow and fast block storage, but it just attaches to your instance. If you want a shared pool of storage, that is usually object based or archive. That kind of data has totally different access characteristics and rules. It doesn’t support the bread-and-butter database workloads that still pull in the billions of revenue in on premises storage. So, cloud storage isn’t a slam dunk (yet).
What will change that? Probably applications and data access methods will have to change before cloud service providers offer competing storage with the kind of characteristics for a traditional OLTP workload. Indeed, the stuff that was ‘born’ on the cloud or has moved there has been non-traditional apps with different requirements. For example, posting a photo to your wall, nobody cares if it doesn’t show up in their feed for 20 seconds. That’s a different situation than say, credit card fraud detection.
There is a *huge* amount of opportunity in this space, and it’s an area I watch closely.