RIM: a leap ahead in user experience, but can it execute?

[RIM’s acquisition of UI firm TAT marked the largest mobile software M&A of 2010. Research Director Andreas Constantinou explains why the acquisition places RIM a leap ahead of the top-10 OEMs in terms of UI capabilities and asks – can RIM execute on the promise?]

RIM: a leap ahead in user experience, but can it execute?

In December, RIM surprised industry observers by buying TAT (The Astonishing Tribe), a 200-strong UI technology and design firm based out of Malmö, Sweden. At nearly $130 million, RIM’s move marked the largest mobile software M&A transaction of 2010 globally and an impressive 5.5x multiplier over TAT’s 2009 revenues of 170 million SEK. It follows a string of RIM acquisitions since 2009, namely QNX (operating system), Cellmania (content billing and distribution), Dash (two-way navigation), DataViz (document viewer), Torch Mobile (WebKit experts) and Viigo (software house).

More importantly, TAT’s acquisition places RIM a leap ahead in the league of top-10 handset manufacturers in terms of own UI capabilities. Here’s why.

A leap ahead of the competition
TAT was founded in 2002 by 6 games engineers and designers out of university (here’s their story) but has come a very long way. TAT is not just another technology company. It has seen its Kastor 2D/3D graphics framework deployed in over 500 million phones across 5 out of the top-7 OEMs. More important to the RIM story is TAT’s Cascades product, a UI framework that allows OEMs to design their phones not in terms of applications, but in terms of screens, allowing what can be termed ‘rapid variant management’ (more about that later).

TAT has also been clearly ahead of the UI technology vendor pack – vendors like Ikivo, Digital Aria, Acrodea, Bluestreak, YouILabs and Scalado – thanks to its design skills. When other vendors have banked on technology marketing, standards implementation or operator deals, TAT has used its design skills to get into the door of both OEMs and operators/carriers (check out this video on the ‘future of screens’). The marriage of design skills and technology licensing allowed TAT to build momentum and cash-flow when OEMs were cutting budgets post-2005. These same skills were what got TAT the deal to design Google’s Android 1.0 UI. TAT’s strength lies in the combination of UI framework technology and the first-class design skills – both of which are now with RIM.

So what does RIM get?

TAT’s acquisition is far more encompassing than many would have thought – it puts RIM a leap ahead of the pack in the league of top-10 handset manufacturers in six ways:

1. Match the iPhone
With the Cascades technology, RIM can now match and even exceed the sophistication of the iPhone UI (see this and this video demos). Long term this means RIM has a chance to contain the exodues of enterprise customers opting for replacing their RIM with iPhones due to the outdated UI and usability on the Blackberry OS 6. Heck, it would be even easy for RIM to offer ‘deep skins’ for BlackBerry handsets where the navigation and core apps closely resemble the iPhone apps.

 

2. Rapid variant management
TAT’s Cascades is a departure from how OEMs build handsets today, by allowing the UI to be designed in terms of screens and not applications.

The downside is that Cascades-enabling an existing software stack means that legacy ‘spaghetti’ applications have to be ported one by one on top of TAT’s framework, which takes 9-12 months for the complete UI (it’s 10s of millions of lines of code that have to be ported). This is what has historically limited Cascades to only tactical wins for specific applications on Motorola, Samsung and Asus handsets.

The upside is that with Cascades RIM gets rapid variant management; creating 100+ operator variants from a single vanilla UI is just a button (and an XML file) away. Designing in screens rather than apps means that RIM can keep its investment into messaging, graphics and enterprise middleware but radically change the UI look and feel. This allows RIM’s carrier customers more differentiation and exclusivity opportunities, all without delaying the time to market – and therefore securing the carrier multi-million subsidy and marketing carrier budgets.

Rapid variant management is today one of the few domains where Android suffers and Nokia’s Symbian still excels, so a very important differentiator for RIM once the integration work is out of the way.

3. Consumer and enterprise personas
We covered earlier how RIM needs to escape its dual personality disorder by designing separate consumer and enterprise product lines. However, designing a different set of apps for enterprise and consumers is complex – not to mention managing many more device models and variants in the field.  With TAT, RIM buys the ability to have enterprise AND consumer UI personas ship in the same phone – not only that, but in a way that can be easily switched by the user at the flick of a button. Switching between enterprise and consumer personas is also much cheaper to do at the UI level rather than the bare metal level with what’s called ‘mobile virtualization‘.

This implies that with TAT’s technology, RIM can allow users to switch between consumer and enterprise UI personas; a consumer UI when you want to browse on Facebook and check out Flickr and an enterprise UI when you want to check the email attachment for your next meeting. Note that Nokia and HTC Sense have also implemented basic switching between work and personal skins.

4. Enterprise UI customization
Besides the runtime technology, TAT develops Motion Lab, a tool that a designer can use to define UI screens and UI flows through a drag-n-drop environment. For RIM, this means that enterprises can customize the phone’s navigation to focus on the few key applications that are used most of the time. It also offers RIM a level of enterprise customization beyond what other OEMs can achieve out of the box.

5. UI personalities
With the erosion of the market of downloadable ringtones and wallpapers, the industry has turned to apps as the next premium content market. Yet, there are still new revenue opportunities in downloadable content. In Japan, DoCoMo has led the market of downloadable UIs in the form of “standby screens” (programmable home screens), and which Acrodea has extended to the dialer and menu apps.  This has created a small market of downloadable UIs for both DoCoMo and KDDI.

With TAT, RIM can extend that market to the world, and across more embedded applications – creating what can be called the market of downloadable UI personalities. Whether RIM can turn this capability into a new ‘market’ is questionable, but it certainly presents a unique point of differentiation and an opportunity for a new revenue stream for RIM.

6. Connected experiences
With the acquisition of Dash, a 2-way car navigation company, RIM has its sights set beyond phones and tablets into the automotive segment. To deliver a consistent UI across these varied form factors a new OS (QNX) is far from adequate. It needs a portable UI technology that allows RIM to reuse its UI assets with minimum maintenance overhead across different form factors, from phones to cars. TAT’s Cascades is exactly this technology and as TAT has shown, it can be extended to connected screens in the living room, in the street, in the car, and in the hands.

Filling in the gaps that TAT left
With TAT out of the picture, how can other OEMs catch up to the level of UI technology sophistication and design skills? There’s a variety of UI technology vendors out there (see below for an extract from our Mobile Industry Atlas), but none really combine the UI ‘screens’ framework or the design skills of TAT.

Many companies claim to have “UI frameworks”, but they all invariably mean a combination of SVG engines, 2D and 3D graphics toolkits or compositing engines – which address UI development as an application, not a screen paradigm. Historically there have only been three companies who have developed screen-based UI frameworks; TAT, Digital Airways and Next Device. Digital Airways was behind the UI of the Vodafone Simply series of five handsets launched between 2005 and 2007 and the Porsche P9522 handset introduced by Sagem in late 2008; the company has since transitioned into UI services in mobile, embedded, automotive and aerospace – however the company ceased trading sometime in 2010. Next Device was acquired by Mentor Graphics (makers of the Nucleus RTOS), who didn’t manage to leverage the technology asset as the licensing model was markedly different to Nucleus’ site-licensing. The gap that TAT left creates an opportunity for other UI middleware vendors (e.g. Ikivo, Acrodea, Digital Aria, Sasken,) to maneuver into this technology space.

Another way to deliver ‘screen-based’ phone design and variant management is via development tools; much like how OpenPlug (now Alcatel Lucent) uses the Adobe Flash IDE to create mobile apps. However this is still virgin territory and we ‘re not aware of any sufficiently advanced UI tools vendors in the mobile domain.

Can RIM execute?
All in all, TAT can deliver Apple-class user experience that offers RIM a strategic advantage compared to OEMs leveraging 3rd party Windows Phone and Android platforms.  This all sounds great on paper of course, but it’s all a question of execution.

Can RIM’s corporate monoculture adapt to the creative minds of TAT? Will the TATers get the mandate and budgets to innovate deep into RIM’s product lines? How long will RIM take to integrate the TAT technology on top of the QNX platform and where will the competition be at that point?

Ladies and gentlemen, place your bets.

– Andreas
you should following me on Twitter: @andreascon

[Andreas Constantinou is Research Director at VisionMobile, and oversees the research, strategy and industry mapping projects at VisionMobile. Andreas also served on TAT’s advisory board during 2008-9]

The Android UI Dilemma: Unify or Differentiate?

[The UI of Android mobile devices is at the epicenter of a conflict between Google and the OEM struggle for differentiation. Guest author Ben Hookway analyses why Google’s UI strategy will be paramount to its proliferation as Android moves to multiple screens]

The Android UI dilemma - unify or differentiate?

The topic of User Interfaces always solicits strong views. It’s a bit like TV – everyone is an expert on it because everyone uses it. Those who have been in the mobile industry a while have seen the tide of UI control flow in and out.

In 2002 operators are demanding custom UIs from handset OEMs in the form of Vodafone Live and Orange SPV. Naturally, most OEMs resist, trying to capture consumer loyalty to the handset, not the network.

[poll id=5]

Three years on, and OEMs are opting to customizing Windows Mobile and Symbian powered handsets rather than creating all-out new UIs. At the same time, network operators are seeing poor returns from UI customization and are dissolving their teams.

With the iPhone big bang in 2007, the UI is back to being the hottest topic in mobile. Post iPhone, almost all tier-1 OEMs are developing their own UI layers, namely HTC Sense, Motorola Blur, Sony Ericsson Rachael, Samsung TouchWiz and LG S-Class. In parallel, network operators are building bigger teams to attempt more control over the UI again; Vodafone, Orange and T-Mobile have 100+ person teams working on ‘signature’ applications and UI definition, while the trend seems to have spilled over to the other side of the Atlantic with Verizon and AT&T opening up multi-million software development centers. Behind the scenes there is also a good deal of demand for UI technology and expertise such as TAT’s Cascades and Mentor’s Inflexion products.

The latest development in the UI saga is the rumoured tighter control being exerted by Google on the Gingerbread release of Android. The aim of this appears to be to reduce API fragmentation issues caused by custom OEM UIs and deliver a more consistent UI brand across different manufacturer devices.

This is not going to go down well with Google’s partners. There is a core commercial conflict on how Google wants to take Android forward. Google wants an Apple-like level of control over the device appearance with their Android handset compliance definition encompassing hardware features, software performance, and service bundling (see the recently published CTS and CDD documents).

However the Apple and Google business models could not be more different.

–       Apple controls the semiconductor, hardware, and software make-up of the iPhone all the way to ad services, branding and retail pricing, whereas Google only controls the software.

–       Apple makes its own devices (at a rate of 1 new model per year), whereas Google relies on partner OEMs to produce 100s of new models per year.

–       Apple spends big advertising dollars in communicating a consistent brand experience across products, while Google co-markets its Experience handsets but puts no money (or effort) in partner handsets.

Most importantly, unlike Apple, Google relies on OEM partners to bring these devices to market.

In the OEM world of survival of the fittest and thinning margins, there are two differentiating factors: price and UI. Yet, both of these factors are being constrained; price is continually declining (standing at 100 GBP unsubsidized for Android handsets) thanks to ODMs willing to sacrifice margins; and the UI is apparently being locked down by Google in the Gingerbread release.

The economic model of handset OEMs necessitates UI differentiation and Google is taking that away. For Google to expect Apple-like control on a fundamentally different business model is just unrealistic.

And it’s only getting worse.

Battling across 4 screens
The next battle (if we are not already in it) is going to be about platforms for your whole life – not just your mobile, TV, PC in isolation, but as one joined-up world; Experience Ecosystems made up of multiple screens where experience can easily roam from one screen to the next.

Browsers are already bridging the gap across laptops, phones, tablets TVs and cars, while the ‘app’ paradigm is taking this further.

To have mobile, TV and PC seamlessly join up requires consistency of the user experience. Apple is the obvious role model here. The Mac, iPhone, iPad, all use similar gestures and interactions, come with similar application design guidelines and are connected to the same centralized service cloud of iTunes and MobileMe. Apple is again the role model in creating the first Experience Ecosystem.

In the Android camp, Google recently announced Google TV. A consistent user experience across mobile and TV is going to be not just important but paramount. Don’t be surprised to see mobile handset OEMs to extend use of Android to other consumer electronics from picture frames and DECT phones to set-top boxes and hi-fis.

But how is Google going to achieve this consistency without an Apple-like hardware control? Hardware control gives you complete user experience consistency in terms of UI responsiveness, screen quality and more. In the next release of Android (Gingerbread) the UI is apparently going to be more locked down; an attempt by Google to gain more control without resorting to hardware manufacturing.

Imagine browsing content on your BrandX Android based tablet and then synching it to your BrandX TV set for viewing over the air. Consistency of experience between the 2 devices will be key. The web browser provides this consistency of interaction and is the best lowest-common denominator right now. But the app phenomenon is outperforming the web by leveraging on location, micropayments, personal user information and intuitve discovery. As one of the main engines behind the app phenomenon, Android could well be powering the battle of the smart living room.

The industry tension over the UI customization of Android is not going to go away anytime soon – rather its going to amplify as more and more manufacturers leverage Android in creating smart, connected and differentiated consumer electronics devices.

OEMs need to plan for their differentiated UI to span multiple devices. Having a familiar experience across devices can be a key driver of brand loyalty and is strategically important to each OEM in creating their own Experience Ecosystem. Competitive pressures make this a key pillar of differentiation that cannot be wasted. It cannot be done half-heartedly. There need to be clear benefits to consumers and clear continuity across as many devices as possible. But the benefits of the larger Android community also need to be maintained, for example having unrestricted access to the Android Market and consistent consumer marketing as to the differentiation offered by the OS itself.

Google needs to accept that UI differentiation is a strategic requirement for its partners, offer alternative differentiation strategies or fundamentally change how it brings Android to market.

Question is, does Google see this as a challenge to the proliferation of Android, and if so, what will they do about it?

– Ben

[Ben Hookway is the CEO of Vidiactive, a company bringing web video to TV, using an open and multi-device approach. He consults on user experience technology and trends, having been founder and CEO of Next Device, which was acquired by Mentor Graphics. Get in touch with Ben: ben.hookway (at) vidiactive.com]

UI Technologies are trendy…but what are they really good for?

[UI development flow and actors: Graphical Designer, Interaction Designer, Software Engineer, classical technologies: GTK, Qt, next generation: Flex, Silverlight, WPF, TAT, XUL, SVG… guest blogger Thomas Menguy describes what are the main concepts behind all the UI technologies, what the new generation ones have in common, what those modern approaches are bringing to the product development flow…and what is missing for the mobile space].

A good UI is nothing without talented graphical designers and interaction designers: How the plethora of new UI technologies are helping unleashing their creativity? What are the main concepts behind those technologies?Let’s try to find out!

UI is trendy… thank you MacOS X, Vista and iPhone!

image

image

image

UIQ

S60

iPhone

Put the designers in the application development driver seat!

Here is a little slide about the actors involved in UI design

image

UI flow actors and their expertize

What does it mean?

Different actors, different knowledge …. So different technologies and different tools!

Those three roles can be clearly separated only if the UI technology allows it. This is clearly not the case in today mainstream UI technologies where the software engineer is in charge of implementing the UI and the service part, most of the time in C/C++ , based on specifications (word document, Photoshop images, sometime adobe flash prototypes), that are subject to interpretation.

  • The technologies used by the designers have nothing in common with the one used to do the actual UI.

  • The technologies that allow UI implementation…require an heavy engineering knowledge.

  • Big consequence: the software engineer decides at the end!

The picture is different for web technologies where it has been crucial and mandatory to keep strongly uncorrelated the service backend from its representation : Web browsers have different API and behavior, backend have to be accessed by many other way than web representation…and above all data is remote and presentation is “half local/half remote”.

Separating representation, interaction and data has been the holly grail of applications and services development for years. It has been formalized through a well known pattern (or even paradigm in that case) : MVC (Model View Controller)

image

MVC pattern / source: wikipedia
From wikipedia: http://en.wikipedia.org/wiki/Model-view-controller
Model
The domain-specific representation of the information on which the application operates. Domain logic adds meaning to raw data (e.g., calculating if today is the user’s birthday, or the totals, taxes, and shipping charges for shopping cart items).
Many applications use a persistent storage mechanism (such as a database) to store data. MVC does not specifically mention the data access layer because it is understood to be underneath or encapsulated by the Model.
View
Renders the model into a form suitable for interaction, typically a user interface element. Multiple views can exist for a single model for different purposes.
Controller
Processes and responds to events, typically user actions, and may invoke changes on the model.

All the UI technologies are offering a way to handle those 3 aspects and, as a consequence, are providing a programming model defining how information and events flow is handled through the MVC.

See below a simple schema I’ve made describing a GTK application: when you look at an application screen, it is made of graphical elements like buttons, lists, images, text labels, called widgets (or controls) .

Rmk: the term “widget” is used with its literal meaning : “window-gadget”, this term is now used a lot in web 2.0 marketing terminology and by Yahoo/Google/MS to represent a “mini application” that can be put on a web page or run through an engine on a desktop PC or a mobile phone, to avoid confusion I prefer the term of “control” over widget for the UI technologies, but will continue using “widget” in the rest of the GTK example as it is the term used by GTK itself.
Widgets are organized hierarchically in a tree, meaning that a widget can contain other widgets, for example a list can contain images or text labels. In the example below the “root” widget is called a “Window”, it contains a kind of canvas which itself contains a status bar, a title bar, a list and a softbutton bar. Then the list contains items, the title bar has a Label, the softbutton bar contains some buttons and so on.

A widget is responsible for

  • Its own drawing using a low level rendering engine, called GDK in the GTK case (GDK offers API like draw_image, draw_text, etc).
  • Computing its size according to its own nature (like the size of the text that will be displayed for example) and the size of its sons.
  • Reacting to some events and emiting some specific ones: the button will emit a “press event” when it is pressed with the touchscreen or when its associated keypad key is pressed.

The widget tree will propagate system events (keypad/touchscreen, etc) and internal events (redraw, size change, etc) through the widgets. The developer will register callbacks (in fact functions, piece of code implementing a functionality) that will be called when widgets will fire events (like the “press event”) .

image

GTK Widget tree structure: a phone screen example

The major GTK/gLib formalism is how those events/callback are handled: through what is called a “gloop” where all events are posted in the loop queue, dequeued one by one and “executed” in this loop, meaning their associated user callbacks will be called. This loop is running in one thread. This is what we call a programming model. In nearly all the UI technologies such a loop exists with various formalisms for the queue handling, event representation, etc.

To finish with the above schema the user callback will then access to the middleware services, the various databases and so on.

There is no clear MVC formalism in that case, the controller is mixed with the view …and even the model that is mixed … with the widgets! (so with the view)

Qt Model is really identical to the this one.

One last point very relevant for application development and design: the notion of states. Each application is in fact a state machine displaying screens linked by transitions, like in the example below where in the state 1 the user decides to write an SMS, it will open an SMS editor screen and clicking send will go to a selection of phone numbers.

image

Application State Machine: write sms example

Here is an attempt to formalize a modern UI framework with Data binding (for Model abstraction).

image

UI engines formalization
Control: equivalent to a widget but where the MVC model is fully split. A Data Model as to be associated alongside with a Renderer to make it usable.
Control Tree: equivalent to the widget tree: aggregation of Controls, association of the controls with a Renderer and a Data Model. Possibly specification of Event Handlers.
Data Model: Object defining (and containing when instantiated) a set of strongly defined and typed data that can be associated with a Control instance.
Data Binding: Service used to populate a Data Model.
Control Renderer: Object that is able to graphically represent a Control associated with a Data Model, using services from a Rendering Library.
Rendering Library: Set of graphical primitives, animations, etc.
Event Handling (and Event Handler): code (any language) reacting to events and modifying the current state machine, the Control Tree, etc.
Standardized Services: Interfaces defined to access middleware directly from the event handling code.
Server Abstraction: Possibility to transparently use Data Binding or any service call locally or remotely.

Ok if you are still there, and your brain is still functional, here is what’s happening today in this area….

In traditional UI frameworks like GTK, Qt, win32, etc the control tree description is done with a C/C++ description … a little niche technology have paved another way: it is called HTML: after all an HTML web page description is defining a tree of controls, W3C use a pedantic term for it : the DOM tree. JavaScript callbacks are then attached to those widget to allow user interaction. It is why all the new UI technologies are based on an XML description for this tree, it is muuuuuch more easier to use, and allow a quicker description of the controls, and above all it allows nice design tools to manipulate the UI….Apart from this XML representation the majority of the UI technologies are coming with:

  • An animation model, allowing smooth transitions, popularized by the iphone UI, but it was already there in MXML (Adobe Flex Format), XAML (MS format), SVG, TAT offer….
  • Modern rendering engines (Flash for Flex, MS has one, TAT Kastor).
  • Nice UI tools for quick implementation: Adobe Flex Builder, MS Expression line, TAT Cascades, Digital Airways Kide, Ikivo SVG…
  • In many case : a runtime, to be able to run a scripting language.

Here are some quick tables, really not complete, of some of the most relevant UI technologies in the PC and mobile phone space.

Just to explain the columns:

  • RIA : Rich Internet Application, delivered through a browser plugin
  • RDA : Rich Desktop Application: delivered through a desktop runtime
  • Runtime: ok, galvoded name here, just a name to represent the piece of technology that allows the UI to run
  • UI: Technology to describe the control tree (you know what it means now!)
  • Event Handling: the dynamic UI part, and how to code it (which languages)
  • Tools: UI tools

 

image

RIA&RDA Chart

 

image

Embedded rich UI technologies

So its time to answer the main point of this post: How those technologies are helping unleashing designers creativity? By defining a new development flow, allowing each actors to have a different role.

Here is an Adobe Flex “standard development flow:

image

Adobe Flex&Air tools flow

In the next schema I try to depict a more complete Adobe Flex flow, adapted to the mobile world, where, for me, a central piece is missing today: it is not possible now to expand “natively” the adobe air engine, this is mandatory for mobile platform with very specific hardware, middleware, form factors.

So I take the adobe flow more as an example to demonstrate how it should work, than as the paradigm of the best UI flow because this is not the case today (same remarks for MS flow, less true for TAT for example)

 

 

image

An Adobe UI design flow for embedded

This shows clearly that the “creativity” phases are clearly uncorrelated: different tools are used between the designers, they can do a lot of iterations together, without any need of the software engineer. This one can focus on implementing the services needed by the UI, optimizing its platform, adding middleware features.

  1. Interaction Designer defines the application high level views and rough flow
  2. Graphical Designer “draws those first screens”
  3. Interaction Designer import it through Thermo
  4. Graphical Designer designs all the Application graphical Assets
  5. Interaction Designer rationalizes and formalize what kind of data, events and high level services the application needs
  6. Interaction Designer & Software Engineer are working together on the above aspects HINT: A FORMALIZM IS MISSING HERE once done:
  7. Software Engineer prepares all the event, data services, test it unitarily, in brief: prepare the native platform
  8. Interaction Designer continues working on the application flows and events, trying new stuffs, experimenting with the Graphic Designer based on the formalism agreed with the Software Engineer.
  9. Once done … application is delivered to the Software Engineer that will perform target integration, optimization…and perhaps (hum certainly) some round trip with the other actors 🙂

So this is it! this flows really focus on giving power to the designers…taking it from the engineer hands. Some technologies are also missing to really offer a full Mobile Phone solution:

  • All the PC technologies are about building ONE application and not a whole system with strong interaction between application. With today technologies, the designers are missing this part….leaving it to the engineer: How to cleanly do animation between applications?
  • Strong theming and customization needed for:
    • product variant management: operator variants, product variants (with different screen sizes and button layouts for example), language variant (in many phones 60 languages has to be supported, but in separated language packs).
    • A not well known one: Factory line fast flashing of those variants. It is very long to flash the whole software of a mobile phone while on the factory line … so if you are able to have a big common part and a “customization” part as little as possible but with the full UI…you gain productivity…and big money 🙂
  • Adapted preset of widget or controls (try to do a phone with WPF or Flex…all the mandatory widgets are missing)

Anyway an UI technology is only the way to interact with a user…to offer him a service. Most of the technologies presented above are about service delivery and not only UI…My next post will be about this notion of service delivery platforms.

[Update] Replaced the term “ergonomics specialist” by “Interaction Designer”, thanks Barbara, see comments below.

Thomas Menguy