The many faces of Android fragmentation

[Android fragmentation is only getting started. Research Director Andreas Constantinou breaks down the 3 dimensions of Android fragmentation and argues how Android will become a victim of its own success]
The article is also available in Chinese.

There’s been plenty of talk of Android fragmentation, but little analysis of its meaning and impacts.

As far as definitions go, the best way to look at fragmentation is not from an API viewpoint, but from an application viewpoint; if you take the top-10,000  (free and paid) apps on Android, how many of these run on all the Android-powered phones?

For Google’s Android team, fragmentation is what keeps them up at night. Fragmentation reduces the addressable market of applications, increases the cost of development and could ultimately break the developer story around Android as we ‘ll see.

Google’s CTS (compatibility test spec) is predicated on ensuring that Market apps run on every Android phone. Android handsets have to pass CTS in order to get access to private codelines, the Market or the Android trademark as we covered in our earlier analysis of Google’s 8 control points – and yes, Google controls what partners do with Android, contrary to the Engadget story.

The 3 dimensions of Android fragmentation
Many observers would point to fragmentation arising as a result of the open source (APL2) license attached to the Android public source code. Reality however is much more complex. There are 3 dimensions of Android fragmentation:

1. Codebase fragmentation. Very few companies have taken the approach of forking the public Android codebase, as permitted under the APL2 license; Google innovates so fast (5 major versions in 12 months) that once you fork, the costs of keeping up-to-date with Google’s tip-of-tree are increasing prohibitively over time (Nokia found out the hard way by forking WebKit and then regretting it).

The main fork of the Android codebase is by China Mobile (the world’s biggest operator with over 500M subscribers) who has outsourced Android development to software company Borqs. China Mobile cares less about keeping up-to-date with the latest Android features as the China market operates as an island where cheap, fake (Shanzai) handsets are predominant. Mediatek, a leading vendor of chipsets shipping in 200-300 million handsets per year plans to make Android available, which could mean another major fork. Cyanogen and GeeksPhone also fork the Android public codeline, but they are designed for a niche of tech-savvy Android fans.

2. Release fragmentation. Google has released 5 major updates to Android in 12 months (1.5, 1.6, 2.0, 2.1 and recently 2.2), all of which introduce major features and often API breaks. You may notice how accessing the Android Market from a 1.6 versus a 2.1 handset gives you a different set of apps. So much for forward compatibility. AndroidFragmentation.com (a community project) has documented several cases of release fragmentation arising from releases which break APIs (e.g. 2.0 SDK breaks older contact apps) or from inconsistent OEM implementations (e.g. receiving multicast messages over WiFi is disabled for most HTC devices).

Release fragmentation is the victim of Google’s own speed of innovation – and Andy Rubin has hinted there’s more major releases coming out in the next 6 months. It’s clearly a sign of how young, agile Internet companies know how to develop software much better that companies with a mobile legacy; major Symbian versions take 12-18 months to release.

Release fragmentation is particularly acute due to the lack limited availability of an automatic update mechanism much like that found on the iPhone. We call the phenomenon ‘runtime aging’ and it is directly responsible for increasing the cost of developing applications. Tier-1 network operators see handsets in their installed base with browsers which are 1-6 years old – that’s how hairy it can get for mobile content (and software) development companies. [update: we understand that certain Android handsets come with a firmware update (FOTA) solution available from Google and other FOTA vendors, but it is installed reactively (i.e. to avoid handset recalls) rather than proactively (i.e. to update all handsets to the latest OS flavour)].

Google itself reports that the Android installed base is split between devices running 1.5, 1.6 and 2.1 versions (or at least for those devices accessing the Android Market). The detailed breakdown as of mid May 2010 is as follows:

Release fragmentation is also arises out of Google’s elitist treatment of its OEM partners. Google will pick and choose which private codeline is available to which OEM based on commercial criteria (contrary to Michael Gartenberg’s story). Take for example how Sony Ericsson’s X10 (running on Android 1.6) came to market after the Nexus One (running on Android 2.1). Ironically, both handsets were made by HTC. [correction: the X10 was developed by Sony Ericsson Japan]

3. Profile fragmentation. Android was designed for volume smartphones. But it arrived at an opportune time – just after the iPhone launch and just as consumer electronics manufacturers were looking at how to develop connected devices. This resulted in two effects that Google hadn’t planned for:

– Android was taken up by all tier-1 (and many tier-2) operators/carriers hoping to develop iPhone-like devices at cheaper prices (i.e. lower subsidies) and greater differentiation. That meant that while operators funded Android’s adolescent years (2008-2010), they niched Android handsets to high-end features and smartphone price points.

– Android is now being taken up by 10s of consumer electronics manufacturers, from car displays and set-top boxes to tablets, DECT phones and picture frames. The Archos internet tablet was just the beginning. Each of these devices has very different requirements and therefore results in different platform profiles.

The timing of Android’s entry into the market has therefore resulted in two implications related to fragmentation.

Firstly, Android’s official codebase isn’t suited for mass-market handsets (think ARM9 or ARM11, 200-500MHz). To get to really large volumes (100M+ annually), Google will need to sanction a second Android profile for mass-market devices. This is a Catch-22, as a second profile is needed to hit large volumes, but it would also break the Android developer story.

Secondly, every new platform profile designed for different form factors (in-car, set-top box, tablet, etc) will create API variations that will be hard to manage. That’s one of the key reasons behind the Google TV initiative and the Open Embedded Software Foundation. However even Google can’t move fast enough to coordinate (manage?) the 10s of use cases and form factors emerging for Android.

All in all, Android fragmentation is going to get far worse, as Android becomes a victim of its own success.But hey, would you expect to have a single app (and a single codebase) that runs on your TV, phone and car?

And there the opportunity lies for tools vendors to provide app porting tools, compatibility test tools and SDKs to help bridge the gap across the eventual jungle of Android fragmentation. And for those looking to better understand the Android commercials we offer a half-day training course on the commercial dynamics behind Android.

What do readers think? Do you have any fragmentation stories to share?

– Andreas
you should follow me on twitter: @andreascon

Is Android Evil?

[Is Android really open? Research Director Andreas Constantinou uncovers the many control points behind Android and explains why Android might be the most closed system in the history of open source].
The article is also available in Chinese and
Greek.

You thought Android was open? The Android governance model consists of an elaborate set of control points that allows Google to bundle its own services and control the exact software and hardware make-up on every handset. All this while touting the openness rhetoric that is founded on the Apache permissive license used in the Android SDK.

[updated in response to reader comments]: Whereas Android is completely open for the software developer ecosystem, it’s completely closed for the handset OEM (pre-load) ecosystem. There is no other platform which is so asymmetrical in terms of its governance structures.

Indeed, Google’s mobile platform is the smartest implementation of open source designed for driving commercial agendas. But before we dig into why, it’s worth discussing why Android’s success has very little to do with open source.

What makes Android tick
Despite early skepticism, Google’s Android operating system has been unequivocally supported by the mobile industry, including more network operators and handset manufacturers than one can count – with the stubborn exception of Nokia. Android managed to ramp from 1 handset model in 2008 to 50+ models announced for 2010 launch, leaving most industry observers in awe.

The Android success has nothing to do with open source; it’s owed to three key factors:

Apple. As strange as it might seem, Android owes much of its success to one of its arch-rivals. Let me explain. With the unprecedented success of the iPhone and the take-it-or-leave-it terms dictated by Apple to network operators, the carriers have been eagerly looking for cheaper alternatives; as such the tier-1 operators have been embarking on Android projects to produce iPhones for people who can’t afford the iPhone and more importantly, without forking out the 300EUR+ subsidy needed to remain competitive in an iPhone market.

Network operators/carriers around the world are eager to differentiate. Android provides the allure of a unified software platform supporting operator differentiation at a low cost (3 months instead of 12+ months offered by SavaJe, which was also aimed at the MNO customisation market). For larger operators with a software strategy, Android also presents a safe investment, as the mainstream option for bringing down the cost of smartphones. That’s why most Android handset projects are backed by a commercial bipoles of operator + OEM deals, with purchase commitments and NRE fees coming from the operator.

Qualcomm. The $10B chipset vendor has been paramount to Android’s ramp up; manufacturers can take Qualcomm’s hardware reference design which is pre-integrated with Android and can go to market within an estimated 9-12 months (down from 16 months for the Motorola Cliq handset and 24+ months for the HTC G1). Besides Qualcomm we should also mention TI’s OMAP3 platform (on which Moto Droid is based) and ST Ericsson and Broadcom who are ramping up to offer chipsets with out-of-the-box support for Android.

In other words, in an Android handset, most of the OEM budget goes into differentiation; compare that to Symbian where most of the OEM budget goes into baseporting (radio and functional integration of hardware) due to historical choices made by Symbian in 2001. All-in-all, Android allows OEMs to reduce their R&D budgets and invest in differentiation, which is mana from heaven to manufacturers.

We should also not forget the ‘free factor’ (technically zero per-unit royalties for the public SDK) which stirred the emotional hype around Android handsets.

All in all, the ‘open source’ marketing moniker has been very successful at triggering major industry disruption – incl. Nokia ‘s acquisition of Symbian and the derailment of Windows Mobile. Perhaps more importantly, the openness rhetoric and the Google aura has attracted thousands of developers on the platform, at a time when the money equation is sub-par; consider that – compared to the Apple devices – Android handsets are around 9x less in volume and paid-for apps are available in 6x fewer countries.

Behind the Open Source facade
What’s even more fascinating is how closed Android is, despite Google’s old do-no-evil don’t be evil mantra and the permissive Apache 2 license which Android SDK source code is under. Paraphrasing a famous line from Henry Ford’s book on the Model-T, anyone can have Android in their own colour as long as it’s black. Android is the best example of how a company can use open source to build up interest and community participation, while running a very tight commercial model. [updated in response to reader comments:] Again I ‘ll emphasize that the closed aspects of Android apply to the handset OEM (pre-load) ecosystem, not the software developer (post-load) ecosystem (see the comments section for a deep dive into pre-load vs post-load].

How does Google control what services, software and hardware ships in Android handsets? The search giant has built an elaborate system of control points around Android handsets.

To dig deeper we spent two months talking to industry sources close to Android commercials – and the reality has been startling. From a high level, Google uses 8 control points to manage the make-up of Android handsets:

1. Private branches. There are multiple, private codelines available to selected partners (typically the OEM working on an Android project) on a need-to-know basis only. The private codelines are an estimated 6+ months ahead of the public SDK and therefore essential for an OEM to stay competitive. The main motivation for the public SDK and source code is to introduce the latest features (those stemming from private branches) into third party apps.

2.  Closed review process. All code reviewers work for Google, meaning that Google is the only authority that can accept or reject a code submission from the community. There is also a rampant NIH (not invented here) culture inside Google that assumes code written by Googlers is second to none. Ask anyone who’s tried to contribute a patch to Android and you hear the same story: very few contributions get in and often no reason is offered on rejection.

3. Speed of evolution. Google innovates the Android platform at a speed that’s unprecedented for the mobile industry, releasing 4 major updates (1.6  to 2.1) in 18 months. OEMs wanting to build on Android have no choice but to stay close to Google so as not to lose on new features/bug fixes released. The Nexus One, Motorola Droid, HTC G1 and other Experience handsets serve the purpose of innovation testbeds for Google.

4. Incomplete software. The public SDK source code is by no means sufficient to build a handset. Key building blocks missing are radio integration, international language packs, operator packs – and of course Google’s closed source apps like Market, Gmail and GTalk. There are a few custom ROM builders with a full Android stack like the Cyanogen distribution, but these use binaries that are not licensed for distribution in commercial handsets.

5. Gated developer community. Android Market is the exclusive distribution and discovery channel for the 40,000+ apps created by developers; and is available to phone manufacturers on separate agreement. This is one of the strongest control points as no OEM would dare produce a handset that doesn’t tap into the Android Market (perhaps with the exception of DECT phones, picture frames, in-car terminals or other exotic uses of Android). However, one should acknowledge that Android’s acceptance process for Market apps is liberal as it gets – and the complete antithesis of the Apple vetting process for apps.

6. Anti-fragmentation agreement. Little is known about the anti-fragmentation agreement signed by OHA members but we understand it’s a commitment to not release handsets which are not CTS compliant (more on CTS later).

7. Private roadmap. The visibility offered into Android’s roadmap is pathetic. At the time of writing, the roadmap published publicly is a year out of date (Q1 2009). To get a sneak peak into the private roadmap you need Google’s blessing.

8. Android trademark. Google holds the trademark to the Android name; as a manufacturer you can only leverage on the Android branding with approval from Google, much like how you need Sun’s approval to claim your handset is Java-powered.

In short, it’s either the Google way or the highway. If you want to branch off Android you ‘re completely on your own and you need resources of the size of China Mobile (see their OMS effort) to make it viable (hint: China Mobile is the biggest network operator bar none).

The Open Handset Alliance is another myth; since Google managed to attract sufficient industry interest in 2008, the OHA is simply a set of signatures with membership serving only as a VIP Club badge.

Another big chapter in the Android saga is the CTS (compatibility test suite) which is the formal testing process by which a handset passes Google requirements. According to our sources, CTS extends significantly beyond API compliance, and into performance testing, hardware features, device design, UI specs and bundled services. CTS is based on the principle of ensuring baseline compliance, so it’s ok to add features, but it’s not ok to detract; compare this with Apple’s no-Flash policy. Note that beyond CTS compliance, there are additional commercial licensing agreements that OEMs have to sign for Google services and private line access.

CTS hampers Android’s progress as well, as it precludes OEMs from creating stripped-down versions of Android that would fit on mass-market phones – those shipping in the 10s of millions. CTS – and forward compatibility to the pool of 40,000+ apps – is Google’s main challenge for hitting a 2-digit market share in the smartphone market. These restrictions – and frienemy relationship between Google and its OEM partners – have stirred up discussions of an ‘Android foundation‘ within OEM circles

The Google Endgame
With Android, Google aims to deliver a consistent platform to its own revenue-generating services. For now, this is the ad business. But in the future, Google is aiming at voice (reaching the billions who don’t have a data connection) and Checkout (i.e. becoming the Visa of mobile).

Yet whatever the endgame, it’s worth realising that [from the manufacturer perspective] Android is no more open – and no less closed – than [licensable operating systems like] Windows Mobile, Apple OSX or PalmOS, Symbian and BREW; it’s the smartest implementation of open source aimed at driving commercial agendas. Android is much less about the do-no-evil rhetoric that the PR spinners in Mountain View would like us to think.

[Updated in response to readers’ comments:] so, is Android evil? No, it isn’t. It has done no harm – quite the contrary, Android has boosted the level of innovation on mobile software. The point of the article is not to vilify Google or concoct visions of Darth Vader; but to balance the level of openness hysteria with a reality check on the commercial dynamics of mobile open source.

– Andreas
you should follow me on twitter: @andreascon

[we are running on-site business workshops for companies who want to understand the commercials behind Android and OHA. Contact us if you ‘re interested. Or, if you are a mobile developer, voice out your views on Android and other mobile platforms in the biggest mobile developer survey to date. Join in at visionmobile.com/developers]