Recently, I had the opportunity to attend an Amazon Web Services (AWS) Big Data Solutions conference. While I have done some work in AWS using S3 and Redshift, it was a good idea for me to get a more rounded understanding of all the products/services that AWS offers. To get to the end right away, I will say that I continue to be very impressed with the quality, scalability, and affordability of AWS. In my humble opinion, AWS is years ahead of any other cloud solution and companies can get immediate cost savings from moving to AWS. However, let me provide a little more background also.
One of my most memorable conversations wasn’t with any of the AWS gurus (sorry Amazon) but rather with a VP of a Silicon Valley company. The reason it was memorable wasn’t just because this lady was a VP in Silicon Valley; but rather, the conversation itself was very timely and telling. In just a 10 minute conversation, she confirmed for me the vast “tech divide” that exists in this country even with very experienced tech executives. In a nutshell, we discussed the almost binary mindset about moving to the cloud. On one hand, there is a group of people who believe that running your own data center and maintaining physical infrastructure is outdated and unnecessary. On the other hand, there is a group of people who believe that moving infrastructure to the cloud is unsafe and untested to the point of being irresponsible. When I asked the VP how she overcame this very large divide, she said, “…with a lot of education…”. When I pressed her for details, it came to light that at least one of the cloud deals was many months in the making…not just a few, but many. I’ve left out the details for obvious reasons.
As successful as AWS is currently, clearly there is a long way to go. While it’s understandable that IT executives would question the security of a third-party, the hesitancy to go to the cloud sometimes seems disproportionate. It’s almost as if the benefits of – say, not having to worry about backups – is perceived as loss of control. Therefore, some of the benefits of AWS is currently acting as a hindrance. My speculation is that since IT infrastructure has historically been hard work and expensive, AWS seems too good to be true. However, that will fade over time. It is simply that this is a new age in IT and that AWS (or other cloud vendors) really will take care of most of the “grunt work” associated with today’s data centers…and do it better…and much cheaper! That’s why we all got into this business in the first place, right? …to make business cheaper and better at a lower cost. I think it’s just that AWS’ business model is so cheap and easy, that many find it difficult to accept….but please do. As an AWS customer, the speed, ease, and savings are real…and they have much better security than many company-managed data centers (certifications like FERPA, HIPAA, FedRAMP and so on).
In an anecdotal example, one company processes a very large amount of data (see image below) for only $3 per hour (or around only $26K per year!) for the infrastructure.
AWS has an amazingly comprehensive list of products and services and many other products available through the AWS Marketplace. The idea is that AWS can be your organization’s entire infrastructure that can be controlled all through a console. Now, that doesn’t mean that everything is “dumbed down” or super easy. All the traditional IT concepts still apply in this environment and reading their documentation is really required to understand how it all fits together. However, with a little investment in time, secure and robust environments can be created quickly and easily. Additionally, if the IT organization needs a bit more process or security to it, going to AWS is a perfect opportunity to re-do roles, permissions and process. So some planning is recommended but AWS enables all the typical practices that you’d expect. For example, when creating a user account, AWS requires that user to be part of a security group…and that security group is granted rights and/or roles.
Starting with the basics, AWS enables data collection and storage through a number of services starting with the Simple Storage Service (S3). From there, DynamoDB is their default recommendation for a database (although they have many others to choose from). From there, data processing options are available like Kenesis for real-time processing. Going “down-stream”, AWS offers a specific data warehouse database option with Redshift that “plays well” with all the traditional business intelligence and data visualization tools like Tableau.
So where this is going is that AWS provides a very end-to-end solution for your infrastructure and data needs. Given Amazon’s scale, they are in a great position to provide it and they are very committed to being the “one-stop-shop” for all your needs. I was skeptical too, at first. However, when making the important decision to move to the cloud, AWS is the best choice right now…and your business will appreciate the speed and cost-effectiveness!