First things first

Two years ago, Apple acquired a distributed database startup called FoundationDB. This week, Apple open sourced the technology to quite a bit of fanfare and speculation. Whatever the reasons Apple acquired FoundationDB in the first place -- presumably using the technology and the team to improve performance of key applications -- I think its reasons for open sourcing the technology are at least as interesting.

We've seen over the past few years that Apple is embracing open source technologies internally -- it's a famously heavy user of Apache Cassandra and Apache Mesos, for example -- and even starting to talk about its usage publicly. This probably has less to do with Apple wanting to change its well-documented culture of silence, and more do to with Apple wanting to hire good distributed systems engineers. We're talking about a group of people with valuable and scarce talent, and who typically like to talk about what they're working on.

So why open source FoundationDB rather than just continue working on it internally? My assumption is Apple realized that the reason the other technologies it uses are so, well, useful, is that they have a community of other large-scale companies using and contributing to them. Companies can compete tooth-and-nail at the business level, but it's typically a good thing to have Netflix, Uber and other engineering-centric organizations using and advancing open source projects. If FoundationDB is as good a key-value store as its supporters claim it is, and if Apple manages the community effectively, the technology will mature and Apple can reap the rewards without doing all the heavy lifting itself.

It's open source 101, and Apple is finally following the leads of Google, Facebook, Netflix, Yahoo and many other companies before it. We'll see how this all plays out, but Apple does have at least one user in its camp already -- hot data-warehousing startup Snowflake, which is using FoundationDB as its metadata store and vows to contribute back to the new open source project. Here's how Snowflake explains its requirements, in a nutshell, in a blog post:

The read and write patterns of our metadata are more akin to online transaction processing (OLTP) than usage patterns of an analytic data warehouse or online analytical processing system (OLAP) that is Snowflake. For these purposes our metadata store requires:

  • Very high frequency of tiny reads and writes at sub-millisecond latency
  • Support for metadata storage that significantly varies in access patterns, volume, and size
  • Good performance for reading small data ranges
  • Running Snowflake as a data warehouse-as-a-service requires high availability even during software upgrades. As a result, multiple versions of the service can be deployed at the same time. And services accessing metadata must be able to handle multiple version of metadata objects.

I have to say, I'm really interested to see how successful Apple is in growing the FoundationDB community and, beyond that, how much more active it becomes in the open source world overall. At its size and with the scale of its businesses, secrecy starts to have diminishing returns.

Facebook and Alibaba are building their own AI chips

Here are the highlights:

There's actually nothing too surprising about Alibaba doing this, if you look at the spaces in which it operates. It's competing locally against companies like Baidu and Tencent, and globally against companies like Amazon, Apple and Google, in spaces ranging from e-commerce to cloud computing to smart devices to health care. Like the rest of them, it probably views specialized AI chips as a means to improving its data center efficiency and AI training, and also probably to power a lineup of consumer devices. If we assume the software and algorithm side of AI is actually fairly democratized (as I do, although some companies definitely have a data advantage), then hardware cost and performance might be big factors in the consumer device space.

The Facebook news is a little more mysterious, although my gut tells me it has more to do with devices than with data centers. Facebook seems pretty content working with Nvidia and even Intel on data center processors, but it might run the risk of getting shut out of smart-home market thanks to the walled gardens of Apple, Amazon and Google. Also, it has the Oculus VR hardware business to think about.

I'll admit my theory on the smart-home market is complete speculation, but I think it makes some sense -- especially if you consider how Facebook came to be so big in the first place. In a web- and mobile-centric world, it made your social network the jumping-off point for interacting online. If devices like the Amazon Echo and Google Home (and their associated platforms) become the new focal point of digital interaction, then where does that leave Facebook? Possibly out in the cold, unless it has its own platform to push that tries to make Facebook a player in this world, as well ...

Read and share this issue online here.

Sponsor: MongoDB

AI and machine learning

Sponsor: Replicated

Cloud and infrastructure

Data and analytics