RIM: a leap ahead in user experience, but can it execute?

[RIM’s acquisition of UI firm TAT marked the largest mobile software M&A of 2010. Research Director Andreas Constantinou explains why the acquisition places RIM a leap ahead of the top-10 OEMs in terms of UI capabilities and asks – can RIM execute on the promise?]

RIM: a leap ahead in user experience, but can it execute?

In December, RIM surprised industry observers by buying TAT (The Astonishing Tribe), a 200-strong UI technology and design firm based out of Malmö, Sweden. At nearly $130 million, RIM’s move marked the largest mobile software M&A transaction of 2010 globally and an impressive 5.5x multiplier over TAT’s 2009 revenues of 170 million SEK. It follows a string of RIM acquisitions since 2009, namely QNX (operating system), Cellmania (content billing and distribution), Dash (two-way navigation), DataViz (document viewer), Torch Mobile (WebKit experts) and Viigo (software house).

More importantly, TAT’s acquisition places RIM a leap ahead in the league of top-10 handset manufacturers in terms of own UI capabilities. Here’s why.

A leap ahead of the competition
TAT was founded in 2002 by 6 games engineers and designers out of university (here’s their story) but has come a very long way. TAT is not just another technology company. It has seen its Kastor 2D/3D graphics framework deployed in over 500 million phones across 5 out of the top-7 OEMs. More important to the RIM story is TAT’s Cascades product, a UI framework that allows OEMs to design their phones not in terms of applications, but in terms of screens, allowing what can be termed ‘rapid variant management’ (more about that later).

TAT has also been clearly ahead of the UI technology vendor pack – vendors like Ikivo, Digital Aria, Acrodea, Bluestreak, YouILabs and Scalado – thanks to its design skills. When other vendors have banked on technology marketing, standards implementation or operator deals, TAT has used its design skills to get into the door of both OEMs and operators/carriers (check out this video on the ‘future of screens’). The marriage of design skills and technology licensing allowed TAT to build momentum and cash-flow when OEMs were cutting budgets post-2005. These same skills were what got TAT the deal to design Google’s Android 1.0 UI. TAT’s strength lies in the combination of UI framework technology and the first-class design skills – both of which are now with RIM.

So what does RIM get?

TAT’s acquisition is far more encompassing than many would have thought – it puts RIM a leap ahead of the pack in the league of top-10 handset manufacturers in six ways:

1. Match the iPhone
With the Cascades technology, RIM can now match and even exceed the sophistication of the iPhone UI (see this and this video demos). Long term this means RIM has a chance to contain the exodues of enterprise customers opting for replacing their RIM with iPhones due to the outdated UI and usability on the Blackberry OS 6. Heck, it would be even easy for RIM to offer ‘deep skins’ for BlackBerry handsets where the navigation and core apps closely resemble the iPhone apps.

 

2. Rapid variant management
TAT’s Cascades is a departure from how OEMs build handsets today, by allowing the UI to be designed in terms of screens and not applications.

The downside is that Cascades-enabling an existing software stack means that legacy ‘spaghetti’ applications have to be ported one by one on top of TAT’s framework, which takes 9-12 months for the complete UI (it’s 10s of millions of lines of code that have to be ported). This is what has historically limited Cascades to only tactical wins for specific applications on Motorola, Samsung and Asus handsets.

The upside is that with Cascades RIM gets rapid variant management; creating 100+ operator variants from a single vanilla UI is just a button (and an XML file) away. Designing in screens rather than apps means that RIM can keep its investment into messaging, graphics and enterprise middleware but radically change the UI look and feel. This allows RIM’s carrier customers more differentiation and exclusivity opportunities, all without delaying the time to market – and therefore securing the carrier multi-million subsidy and marketing carrier budgets.

Rapid variant management is today one of the few domains where Android suffers and Nokia’s Symbian still excels, so a very important differentiator for RIM once the integration work is out of the way.

3. Consumer and enterprise personas
We covered earlier how RIM needs to escape its dual personality disorder by designing separate consumer and enterprise product lines. However, designing a different set of apps for enterprise and consumers is complex – not to mention managing many more device models and variants in the field.  With TAT, RIM buys the ability to have enterprise AND consumer UI personas ship in the same phone – not only that, but in a way that can be easily switched by the user at the flick of a button. Switching between enterprise and consumer personas is also much cheaper to do at the UI level rather than the bare metal level with what’s called ‘mobile virtualization‘.

This implies that with TAT’s technology, RIM can allow users to switch between consumer and enterprise UI personas; a consumer UI when you want to browse on Facebook and check out Flickr and an enterprise UI when you want to check the email attachment for your next meeting. Note that Nokia and HTC Sense have also implemented basic switching between work and personal skins.

4. Enterprise UI customization
Besides the runtime technology, TAT develops Motion Lab, a tool that a designer can use to define UI screens and UI flows through a drag-n-drop environment. For RIM, this means that enterprises can customize the phone’s navigation to focus on the few key applications that are used most of the time. It also offers RIM a level of enterprise customization beyond what other OEMs can achieve out of the box.

5. UI personalities
With the erosion of the market of downloadable ringtones and wallpapers, the industry has turned to apps as the next premium content market. Yet, there are still new revenue opportunities in downloadable content. In Japan, DoCoMo has led the market of downloadable UIs in the form of “standby screens” (programmable home screens), and which Acrodea has extended to the dialer and menu apps.  This has created a small market of downloadable UIs for both DoCoMo and KDDI.

With TAT, RIM can extend that market to the world, and across more embedded applications – creating what can be called the market of downloadable UI personalities. Whether RIM can turn this capability into a new ‘market’ is questionable, but it certainly presents a unique point of differentiation and an opportunity for a new revenue stream for RIM.

6. Connected experiences
With the acquisition of Dash, a 2-way car navigation company, RIM has its sights set beyond phones and tablets into the automotive segment. To deliver a consistent UI across these varied form factors a new OS (QNX) is far from adequate. It needs a portable UI technology that allows RIM to reuse its UI assets with minimum maintenance overhead across different form factors, from phones to cars. TAT’s Cascades is exactly this technology and as TAT has shown, it can be extended to connected screens in the living room, in the street, in the car, and in the hands.

Filling in the gaps that TAT left
With TAT out of the picture, how can other OEMs catch up to the level of UI technology sophistication and design skills? There’s a variety of UI technology vendors out there (see below for an extract from our Mobile Industry Atlas), but none really combine the UI ‘screens’ framework or the design skills of TAT.

Many companies claim to have “UI frameworks”, but they all invariably mean a combination of SVG engines, 2D and 3D graphics toolkits or compositing engines – which address UI development as an application, not a screen paradigm. Historically there have only been three companies who have developed screen-based UI frameworks; TAT, Digital Airways and Next Device. Digital Airways was behind the UI of the Vodafone Simply series of five handsets launched between 2005 and 2007 and the Porsche P9522 handset introduced by Sagem in late 2008; the company has since transitioned into UI services in mobile, embedded, automotive and aerospace – however the company ceased trading sometime in 2010. Next Device was acquired by Mentor Graphics (makers of the Nucleus RTOS), who didn’t manage to leverage the technology asset as the licensing model was markedly different to Nucleus’ site-licensing. The gap that TAT left creates an opportunity for other UI middleware vendors (e.g. Ikivo, Acrodea, Digital Aria, Sasken,) to maneuver into this technology space.

Another way to deliver ‘screen-based’ phone design and variant management is via development tools; much like how OpenPlug (now Alcatel Lucent) uses the Adobe Flash IDE to create mobile apps. However this is still virgin territory and we ‘re not aware of any sufficiently advanced UI tools vendors in the mobile domain.

Can RIM execute?
All in all, TAT can deliver Apple-class user experience that offers RIM a strategic advantage compared to OEMs leveraging 3rd party Windows Phone and Android platforms.  This all sounds great on paper of course, but it’s all a question of execution.

Can RIM’s corporate monoculture adapt to the creative minds of TAT? Will the TATers get the mandate and budgets to innovate deep into RIM’s product lines? How long will RIM take to integrate the TAT technology on top of the QNX platform and where will the competition be at that point?

Ladies and gentlemen, place your bets.

– Andreas
you should following me on Twitter: @andreascon

[Andreas Constantinou is Research Director at VisionMobile, and oversees the research, strategy and industry mapping projects at VisionMobile. Andreas also served on TAT’s advisory board during 2008-9]

Breaking the 500 million barrier of mobile software

[Which are the most ubiquitous mobile software products out there? Marketing Manager Matos Kapetanakis opens up our 5th edition of the 100 Million Club, the watchlist of embedded software products and talks about the really big numbers of mobile software.]

Welcome to the H2 2009 edition of the 100 Million Club, the semi-annual watchlist of mobile software products that have been embedded in more than 100 million mobile devices since their release. Despite the apparent opportunity in the one-billion-a-year handset market, very few software companies have managed to overcome the commercial and technical challenges inherent in the mobile industry.

Key highlights in this H2 2009 edition:

– “The cumulative number of shipments of all the 100 Million Club software products up to the end of 2009 is 24.6 billion – an 11% increase since the previous half”

– “The estimated 250 million cumulative shipments for Apple’s WebKit show that it is fast becoming a de facto browser platform.”

– “BlackBerry is the next smartphone platform, after Symbian, that will break through the 100 million shipments barrier.”

What’s new in H2 2009?
So, what major changes have we seen since our previous update?

First off we’re happy to welcome three new entrants to the Club: ARM, Mimer and Numonyx have joined, adding three new middleware products to our watchlist. Mimer has just broken the 100 million barrier with its SQL database engine, while ARM brings us Mali-JSR184, a 3D graphics engine for wireless devices. The Flash Data Integrator by Numonyx is already ahead of the game, having been shipped in more than 900 million devices.

We have also had to remove three software products that have long been part of the Club. For different reasons, Mobile BAE by Beatnik and Picsel’s File Viewer are no longer part of the 100 Million Club, while Nokia’s Series 60 OS has been incorporated in the Symbian OS.

(click to download)

Growth in the 100 Million Club
The H2 2009 edition of the 100 Million Club is comprised of 30 software products by 26 companies. The total number of shipments of all 30 products, up to the end of 2009, comes to 24.6 billion – an 11% increase since the previous half.

In the previous edition, the Club featured 15 software products that exceeded 500 million shipments, 6 of which had also broken through the 1 billion barrier. The H2 2009 edition features 17 products with more than 500 million sales, 7 of which have surpassed 1 billion shipments. In other words, for the first time the majority of the products featured in the 100 Million Club have over 500 million shipments.

In the second half of 2009, CAPS by Scalado and OKL4 by Open Kernel Labs managed to break through the 500 million barrier, while Myriad Group’s messaging client and Nokia’s Series 40 OS now have more than 1 billion shipments each.

Category leaders: apps, browsers, middleware and operating systems
Quickoffice wins by default in the embedded applications category, since it’s the only embedded application featured in the 100 Million Club.

Adobe is still number one in the application environments category, with Flash/Flash Lite having been embedded in more than 1.3 billion devices up to the end of 2009. The growth of Flash Lite has decelerated significantly from 43% (1H09) to 15% (2H09) as share of devices sold with the software embedded; however the pace should be picking up pace again with Flash shipments later in 2010.

Myriad Group, whose browser has almost twice as many shipments as the other category products combined, dominates the browser market.

In the middleware category things are not that clear, due to the diversity of products. In absolute numbers, the messaging client by Myriad Group has the most shipments (1.2B) and vRapid Mobile by Red Bend shows the highest of growth over the second half of 2009. UI software is also highly penetrated within mobile devices, led by graphics engines by Ikivo, Scalado and The Astonishing Tribe which are at or around the 500 million mark.

The operating system market features 6 products that have been embedded in more than 1 billion devices. It’s worth noting that mass-appeal operating systems like OSE, Nucleus and recently Series 40 have cumulative shipments numbering in the billions, while BREW has just broken past the 500 million mark. In contrast, most major smartphone platforms – Android, OSX, Windows Mobile, BlackBerry – apart from Symbian have yet to reach 100 million shipments.

Finally, the input engines category features two products, both by Nuance inherited from the past acquisitions of Tegic and Zi Corp. As is evident in the chart, T9/XT9 is by far the most prominent, having been embedded in a staggering 4.8 billion mobile devices up to the end of the second half of 2009.

100 Million Club facts and trends

Two companies account for 38% of shipments: Only two companies have multiple software products included in the 100 Million Club, each company featuring three products. The cumulative number of shipments of these two companies is 9.5 billion, representing 38% of all 100 Million Club products’ shipments up to the end of H2 2009. The software products are Myriad Group’s Browser, messaging client and Jbed and Nuance’s T9/XT9, eZiText and VSuite.

WebKit on the rise: We estimate that up to the end of 2009 WebKit, the open source browser engine, has been embedded in more than 250 million devices. WebKit owes most of its market penetration to Nokia (Symbian shipments with the Series 40 contribution picking up), while its recent adoption by RIM can only accelerate its market penetration.

Top revenue models: In this edition, we asked the 100 Million Club members to provide us with the top two revenue models for their products. The responses revealed that the most common revenue models for embedded software are per-unit royalties,followed by NRE (non-recurring engineering fees) for product integration or customisation. Despite the tight profit margins, handset OEMs and network operators are still paying for software on a per-unit basis, with the ‘paradigm shift’ to per-active user revenue models taking longer than most would have expected.

What’s in stock for the 100 Million Club
Our watchlist continues to grow, as more products make it past 100 million shipments. Blackberry should be entering the Club in the next edition (H1 2010), with OSX, Windows Mobile and the much younger Android lagging a further 6-18 months behind.

The bigger picture of mobile software is very different than the industry hype would have us think.

– Matos

UI Technologies are trendy…but what are they really good for?

[UI development flow and actors: Graphical Designer, Interaction Designer, Software Engineer, classical technologies: GTK, Qt, next generation: Flex, Silverlight, WPF, TAT, XUL, SVG… guest blogger Thomas Menguy describes what are the main concepts behind all the UI technologies, what the new generation ones have in common, what those modern approaches are bringing to the product development flow…and what is missing for the mobile space].

A good UI is nothing without talented graphical designers and interaction designers: How the plethora of new UI technologies are helping unleashing their creativity? What are the main concepts behind those technologies?Let’s try to find out!

UI is trendy… thank you MacOS X, Vista and iPhone!

image

image

image

UIQ

S60

iPhone

Put the designers in the application development driver seat!

Here is a little slide about the actors involved in UI design

image

UI flow actors and their expertize

What does it mean?

Different actors, different knowledge …. So different technologies and different tools!

Those three roles can be clearly separated only if the UI technology allows it. This is clearly not the case in today mainstream UI technologies where the software engineer is in charge of implementing the UI and the service part, most of the time in C/C++ , based on specifications (word document, Photoshop images, sometime adobe flash prototypes), that are subject to interpretation.

  • The technologies used by the designers have nothing in common with the one used to do the actual UI.

  • The technologies that allow UI implementation…require an heavy engineering knowledge.

  • Big consequence: the software engineer decides at the end!

The picture is different for web technologies where it has been crucial and mandatory to keep strongly uncorrelated the service backend from its representation : Web browsers have different API and behavior, backend have to be accessed by many other way than web representation…and above all data is remote and presentation is “half local/half remote”.

Separating representation, interaction and data has been the holly grail of applications and services development for years. It has been formalized through a well known pattern (or even paradigm in that case) : MVC (Model View Controller)

image

MVC pattern / source: wikipedia
From wikipedia: http://en.wikipedia.org/wiki/Model-view-controller
Model
The domain-specific representation of the information on which the application operates. Domain logic adds meaning to raw data (e.g., calculating if today is the user’s birthday, or the totals, taxes, and shipping charges for shopping cart items).
Many applications use a persistent storage mechanism (such as a database) to store data. MVC does not specifically mention the data access layer because it is understood to be underneath or encapsulated by the Model.
View
Renders the model into a form suitable for interaction, typically a user interface element. Multiple views can exist for a single model for different purposes.
Controller
Processes and responds to events, typically user actions, and may invoke changes on the model.

All the UI technologies are offering a way to handle those 3 aspects and, as a consequence, are providing a programming model defining how information and events flow is handled through the MVC.

See below a simple schema I’ve made describing a GTK application: when you look at an application screen, it is made of graphical elements like buttons, lists, images, text labels, called widgets (or controls) .

Rmk: the term “widget” is used with its literal meaning : “window-gadget”, this term is now used a lot in web 2.0 marketing terminology and by Yahoo/Google/MS to represent a “mini application” that can be put on a web page or run through an engine on a desktop PC or a mobile phone, to avoid confusion I prefer the term of “control” over widget for the UI technologies, but will continue using “widget” in the rest of the GTK example as it is the term used by GTK itself.
Widgets are organized hierarchically in a tree, meaning that a widget can contain other widgets, for example a list can contain images or text labels. In the example below the “root” widget is called a “Window”, it contains a kind of canvas which itself contains a status bar, a title bar, a list and a softbutton bar. Then the list contains items, the title bar has a Label, the softbutton bar contains some buttons and so on.

A widget is responsible for

  • Its own drawing using a low level rendering engine, called GDK in the GTK case (GDK offers API like draw_image, draw_text, etc).
  • Computing its size according to its own nature (like the size of the text that will be displayed for example) and the size of its sons.
  • Reacting to some events and emiting some specific ones: the button will emit a “press event” when it is pressed with the touchscreen or when its associated keypad key is pressed.

The widget tree will propagate system events (keypad/touchscreen, etc) and internal events (redraw, size change, etc) through the widgets. The developer will register callbacks (in fact functions, piece of code implementing a functionality) that will be called when widgets will fire events (like the “press event”) .

image

GTK Widget tree structure: a phone screen example

The major GTK/gLib formalism is how those events/callback are handled: through what is called a “gloop” where all events are posted in the loop queue, dequeued one by one and “executed” in this loop, meaning their associated user callbacks will be called. This loop is running in one thread. This is what we call a programming model. In nearly all the UI technologies such a loop exists with various formalisms for the queue handling, event representation, etc.

To finish with the above schema the user callback will then access to the middleware services, the various databases and so on.

There is no clear MVC formalism in that case, the controller is mixed with the view …and even the model that is mixed … with the widgets! (so with the view)

Qt Model is really identical to the this one.

One last point very relevant for application development and design: the notion of states. Each application is in fact a state machine displaying screens linked by transitions, like in the example below where in the state 1 the user decides to write an SMS, it will open an SMS editor screen and clicking send will go to a selection of phone numbers.

image

Application State Machine: write sms example

Here is an attempt to formalize a modern UI framework with Data binding (for Model abstraction).

image

UI engines formalization
Control: equivalent to a widget but where the MVC model is fully split. A Data Model as to be associated alongside with a Renderer to make it usable.
Control Tree: equivalent to the widget tree: aggregation of Controls, association of the controls with a Renderer and a Data Model. Possibly specification of Event Handlers.
Data Model: Object defining (and containing when instantiated) a set of strongly defined and typed data that can be associated with a Control instance.
Data Binding: Service used to populate a Data Model.
Control Renderer: Object that is able to graphically represent a Control associated with a Data Model, using services from a Rendering Library.
Rendering Library: Set of graphical primitives, animations, etc.
Event Handling (and Event Handler): code (any language) reacting to events and modifying the current state machine, the Control Tree, etc.
Standardized Services: Interfaces defined to access middleware directly from the event handling code.
Server Abstraction: Possibility to transparently use Data Binding or any service call locally or remotely.

Ok if you are still there, and your brain is still functional, here is what’s happening today in this area….

In traditional UI frameworks like GTK, Qt, win32, etc the control tree description is done with a C/C++ description … a little niche technology have paved another way: it is called HTML: after all an HTML web page description is defining a tree of controls, W3C use a pedantic term for it : the DOM tree. JavaScript callbacks are then attached to those widget to allow user interaction. It is why all the new UI technologies are based on an XML description for this tree, it is muuuuuch more easier to use, and allow a quicker description of the controls, and above all it allows nice design tools to manipulate the UI….Apart from this XML representation the majority of the UI technologies are coming with:

  • An animation model, allowing smooth transitions, popularized by the iphone UI, but it was already there in MXML (Adobe Flex Format), XAML (MS format), SVG, TAT offer….
  • Modern rendering engines (Flash for Flex, MS has one, TAT Kastor).
  • Nice UI tools for quick implementation: Adobe Flex Builder, MS Expression line, TAT Cascades, Digital Airways Kide, Ikivo SVG…
  • In many case : a runtime, to be able to run a scripting language.

Here are some quick tables, really not complete, of some of the most relevant UI technologies in the PC and mobile phone space.

Just to explain the columns:

  • RIA : Rich Internet Application, delivered through a browser plugin
  • RDA : Rich Desktop Application: delivered through a desktop runtime
  • Runtime: ok, galvoded name here, just a name to represent the piece of technology that allows the UI to run
  • UI: Technology to describe the control tree (you know what it means now!)
  • Event Handling: the dynamic UI part, and how to code it (which languages)
  • Tools: UI tools

 

image

RIA&RDA Chart

 

image

Embedded rich UI technologies

So its time to answer the main point of this post: How those technologies are helping unleashing designers creativity? By defining a new development flow, allowing each actors to have a different role.

Here is an Adobe Flex “standard development flow:

image

Adobe Flex&Air tools flow

In the next schema I try to depict a more complete Adobe Flex flow, adapted to the mobile world, where, for me, a central piece is missing today: it is not possible now to expand “natively” the adobe air engine, this is mandatory for mobile platform with very specific hardware, middleware, form factors.

So I take the adobe flow more as an example to demonstrate how it should work, than as the paradigm of the best UI flow because this is not the case today (same remarks for MS flow, less true for TAT for example)

 

 

image

An Adobe UI design flow for embedded

This shows clearly that the “creativity” phases are clearly uncorrelated: different tools are used between the designers, they can do a lot of iterations together, without any need of the software engineer. This one can focus on implementing the services needed by the UI, optimizing its platform, adding middleware features.

  1. Interaction Designer defines the application high level views and rough flow
  2. Graphical Designer “draws those first screens”
  3. Interaction Designer import it through Thermo
  4. Graphical Designer designs all the Application graphical Assets
  5. Interaction Designer rationalizes and formalize what kind of data, events and high level services the application needs
  6. Interaction Designer & Software Engineer are working together on the above aspects HINT: A FORMALIZM IS MISSING HERE once done:
  7. Software Engineer prepares all the event, data services, test it unitarily, in brief: prepare the native platform
  8. Interaction Designer continues working on the application flows and events, trying new stuffs, experimenting with the Graphic Designer based on the formalism agreed with the Software Engineer.
  9. Once done … application is delivered to the Software Engineer that will perform target integration, optimization…and perhaps (hum certainly) some round trip with the other actors 🙂

So this is it! this flows really focus on giving power to the designers…taking it from the engineer hands. Some technologies are also missing to really offer a full Mobile Phone solution:

  • All the PC technologies are about building ONE application and not a whole system with strong interaction between application. With today technologies, the designers are missing this part….leaving it to the engineer: How to cleanly do animation between applications?
  • Strong theming and customization needed for:
    • product variant management: operator variants, product variants (with different screen sizes and button layouts for example), language variant (in many phones 60 languages has to be supported, but in separated language packs).
    • A not well known one: Factory line fast flashing of those variants. It is very long to flash the whole software of a mobile phone while on the factory line … so if you are able to have a big common part and a “customization” part as little as possible but with the full UI…you gain productivity…and big money 🙂
  • Adapted preset of widget or controls (try to do a phone with WPF or Flex…all the mandatory widgets are missing)

Anyway an UI technology is only the way to interact with a user…to offer him a service. Most of the technologies presented above are about service delivery and not only UI…My next post will be about this notion of service delivery platforms.

[Update] Replaced the term “ergonomics specialist” by “Interaction Designer”, thanks Barbara, see comments below.

Thomas Menguy