ARCHITECHT Daily: As Amazon's cloud grows, the data center market consolidates

Scale drives almost everything in the era of cloud computing, and it's a big reason why customer choi
ARCHITECHT Daily: As Amazon's cloud grows, the data center market consolidates
By ARCHITECHT • Issue #93
Scale drives almost everything in the era of cloud computing, and it’s a big reason why customer choice is not always the name of the game when it comes to infrastructure. 
Globally, there are really three major cloud providers (four if you count Alibaba’s growing cloud business, and five if you count IBM) and now a decreasing number of companies from which to lease data center space, as well. On Friday, it was announced that Digital Realty, the world’s largest wholesale data center provider, is buying rival DuPont Fabros for about $7.6 billion. According to estimates by market research firm Structure Research, the deal will give Digital Realty more than 26 percent of global data center market share.
Meanwhile, Amazon Web Services announced a new campus near its U.S. East cloud region in northern Virginia, which might or might not include new data center capacity. AWS also announced on Thursday that is in the midst of a major migration to its cloud. According to ZDNet, “the company has moved 8 petabytes of data and 6 petabytes of images already to Amazon, and is in the process of moving 550 databases and 500 services.”
The economics of data centers seem to demand that the big get bigger, and everybody else get out of the way. While large data center and colocation providers like Digital Realty and Equinix continue to see revenues increase, even in the face of competition from the public cloud, part of that is due to rabid expansion via acquisition. For better or worse, companies with serious computing and storage needs are seeing their pool of providers—and some of their bargaining power—shrink before their eyes.

Sponsor: Cloudera
Listen to the latest ARCHITECHT Show podcast
Chef co-founder and CTO Adam Jacob talks about the evolution of his company and its product as user demands have evolved (or not) over the past decade. He tackles the benefits and shortcomings of technologies such as containers and cloud computing, as well as the difficulty of doing security right and nailing the open source business model.
Artificial intelligence
It was a little strange when Google bought these companies, but selling them off is a little strange, too, given Google’s, er Alphabet’s, focus on artificial intelligence. Maybe Alphabet doesn’t need independent robotics businesses, but there are a lot of people who think AI needs a physical form to reach its pinnacle.
Baidu is not messing around with its autonomous car ambitions, having also partnered recently with Bosch and Continental. 
This is really cool work. Essentially, it’s an approach to reinforcement learning that allows multiple systems to act independently and then learn from each other, thus creating a centralized intelligence.
This is a pretty classic use case for the technique Uber is using (LSTM), which is ideal for time-series data. The real question is whether better predictions will lead to customer-friendly pricing—my $99+ estimate for a trip from UW to Seattle’s airport yesterday morning was a little steep.
This is a pretty cool idea, building 3-D recreations of cities and layering in real traffic data to simulate real life. Synthetic training data is a big deal in AI right now; we’ll see how well it does for an application like driving with life-and-death consequences.
DeepMind’s training dataset of 300,000 YouTube clips finds AI struggles to recognize Homer Simpson actions
Speaking of DeepMind … Not only does Demis Hassabis seem like a genuinely nice person, but he has some great insights on raising genius children, and some interesting “desert island” music picks.
The Salk Institute has discovered new insights into how human eyes and brains talk to each other to parse visual information, and they’re already applying it to computer vision.
This research suggests that robots and AI systems need reasonable confidence that what they’re doing is useful (or not harmful), without veering too far in either direction. It could have repercussions in human-robot interaction.
When you’re dealing with as many images as Facebook is, time and performance are of the essence. This is an area we should expect to see the company continue to lead in.
Sponsor: DigitalOcean
Cloud and infrastructure
This is a consumer thing, and I don’t think it would ever try something comparable with AWS, but it’s one of those business decisions that makes people nervous about the cloud.
I’ve seen some other coverage of this 10-year deal, but this piece gets at some of the thornier issues with it. For example, it’s a “private cloud” deal that involves a lot of employee movement and outsourcing.
This is a good writeup on the promise of FPGAs for workloads like deep learning and edge computing, leaning heavily on Microsoft’s vision for exposing them to Azure users.
I don’t think anybody would deny that the x86 architecture has been immensely important and successful, and that it will be around for decades to come even in the face of more competition. But the last three paragraphs read like an ode to patent litigation.
CockroachDB is doing some impressive work around building a distributed SQL database. If you’re looking for some insights into where databases are headed, check this out.
Switch isn’t as large as Digital Realty or Equinix, but it’s very proud of its data center designs and expanding across the globe with some large facilities. Now it’s trying to push the industry to adopt a newer method for rating data center efficiency.
Listen the the ARCHITECHT Show podcast. New episodes every Thursday!
All things data
I honestly hadn’t heard of Algolia, but it’s the search engine behind many popular sites, including Medium and Twitch. With Google Site Search winding down, there’s a big opportunity in search on the horizon.
If you saw Cloudera’s first quarter results and got hung up on the large loss, analyst Tony Baer points out here that much of that had to do with one-time stock payouts. Minus that, revenue grew 57 percent and losses shrunk by 26 percent.
That’s what the CEO is supposed to say, although he didn’t provide any date or any real numbers to give us a sense of how close MapR is to being profitable. As I’ve said before, this space will get much more interesting once MapR is public, too.
This is a nice humblebrag from the folks at CB Insights about how they finally got data flowing in so they could use it to improve their analysis. The lesson, if you’re relying on other people for data, is to appeal to their self interest. Facebook and Amazon mastered this.
Some advice for companies trying to make the most of machine learning, by cleaning up the data feeding the models.
As part of an NSF program, Microsoft contributed millions of dollars worth of Azure resources to scientists across the country. Here are some of the projects they’re working on, which are actually pretty interesting.
Did you enjoy this issue?
The most interesting news, analysis, blog posts and research in cloud computing, artificial intelligence and software engineering. Delivered daily to your inbox. Curated by Derrick Harris. Check out the Architecht site at
Carefully curated by ARCHITECHT with Revue. If you were forwarded this newsletter and you like it, you can subscribe here. If you don't want these updates anymore, please unsubscribe here.