Apparently, just like Google, Amazon realized that silly things like transactions don't scale, neither do auto-incrementing counters, nor table joins... so they made some kind of wacky database called Amazon Dynamo that doesn't require them. The entire system is essentially blobs of binary data, accessible by unique keys. Each key is mapped to a unique URL, which makes the ReST fanboys drool... its fast and highly available, but it doesn't do a majority of what a typical database does.

In other words, they've replaced their database with a freakin' file system! Or, more precisely, its a highly distributable cache that is freaky fast, freaky scalable, and ignores silly things like security:

Other Assumptions: Dynamo is used only by Amazon’s internal services. Its operation environment is assumed to be non-hostile and there are no security related requirements such as authentication and authorization.

erm... that's a pretty big assumption! Hopefully they don't put credit card numbers in that thing...

They aren't alone... many of the biggest sites on the web use a similar technology, called memcached. There are other radical database replacements out there, such as Hadoop, and Mnesia, both are Sam Ruby's Long Bets... and who knows what exactly the guys over at Freebase are doing behind the scenes... but the general feeling is that fast databases in 3 years will look nothing like they do today.

I just keep getting this weird feeling that we've been here before... Its almost like people have given up on the idea that you can ever truly capture data.

For example: what's on the internet right... NOW? OK, how about... NOW? How about NOW? Firstly, its an unanswerable question... and even if we could know it, things like proper ACID transactions certainly would never help. By the time the transaction finished, the data's no longer valid.

As organizations place more data on easily accessed internal systems, the question turns to "what' on MY network right... NOW?" At present, we can answer this. But what about the future? Financial and government people will demand all kinds of horrendously complex solutions to ensure zero transaction failures and 100% accurate data for auditors... but that doesn't stop one clumsy employee from typing the wrong number. Isn't there some wasted energy there?

Bruce Schneier says a good security plan has three sections: Protection, Detection, and Reaction. The math is pure, but the weak link is the human operator. On very large and very complex systems, failure is the norm, not the exception. You can't focus all your energies on protection from failure, because then you'll never develop the tools or skills needed to find and fix inevitable problems.

The same may hold true for databases... for sufficiently complex systems, a database only represents a vague approximation of the state of your organization. This problem will only get worse when people implement SOA across the enterprise... which will enable complex interactions and dependencies across business units that were never before possible... Empowering? Yes. Important? Yes. Scary? Yes.

As bandwidth gets cheaper, many people will be testing the truth of that statement... But, since fortune favors the prepared mind, I think we're safe ;-)