Developer Program Metrics: How Intuit Measures Developer Success

How do you grow a successful developer community around a financial product? Intuit’s Quickbooks is a well-known small business accounting software that has 100s of accounting applications built on top of the platform, from tax planners to CRM and HR apps. We spoke to Intuit to better understand the mechanisms the company uses to make sure its developer ecosystem succeeds.

Originally a desktop accounting software, QuickBooks has moved to a Software-as-a-Service model with a clear vision of the core product features it can enable through its service model. But they also know that customers often need additional functionality, so they have created APIs to let third party providers build those features.

QuickBooks found that by offering a range of app integrations with their main product, customers spent less time moving data from one system to another and saved time, increased their satisfaction with the product, and ended up looking for more ways to use QuickBooks in their daily business systems.

In 2014, when QuickBooks API program was in its infancy, they offered up to 50 third-party apps, and began to see the impact this was having on their customer lifetime value. They decided to invest more heavily in their API program and now they have over 450 published apps (and over 1,200 in total, including private integrations) and thousands of developers building on the platform.

The network of app developers has provided a wide choice of high quality app integrations to small businesses using QuickBooks, solving important problems in its user base. This has helped QuickBooks strengthen relationships with its existing customer base and also grow the base.

The QuickBooks Online platform is now growing a two-sided “network effects“ platform by delivering benefits to both developers and small businesses.

We spoke to Ketan Kittur, Director of Product Management and Partner Integrations at Intuit to understand how Intuit ensures that its app developers are successful. “The two-sided value of our platform is critical. We want to be serious about measuring retention and quality, so that we can give our customers the best experience and outcomes for running their business. We also want to understand the value these integrations are bringing to our product,” Kittur says.

VisionMobile: How do you first identify which developers need to be targeted in a developer program?

Ketan Kittur: We look at what problems our small business customers are facing that are not part of our product roadmap. Our developer team then scans the developer community – local and abroad- for potential solutions to the problems. The identified developers may already have solutions or they may include developers who are interested in building applications to address these problems. Our outreach team then works closely with the developers and guides them through the integration process, so their app integrations are of high quality delivering value to the customers.

Through these discussions, we have found that there are two main things that matter for developers:

  1. They are also entrepreneurs. They are serious business people building their own products and customer base.
  2. They do not want to waste a lot of time. Their development effort needs to return value.

VM: How do you align building a developer program with overall business goals?

KK: Our goal is to help our small businesses be successful using QuickBooks. For problems that are not solved by QuickBooks, we need a thriving a developer community delivering high quality app integrations so we have a highly engaged user base of QuickBooks users. Hence, the goal of both our developer team and developers are aligned- solve critical problems for small businesses using the QuickBooks platform and grow our businesses. When your company has a global footprint like Intuit, we become an attractive platform for developers, allowing them to grow beyond their geographic reach and inspire them to solve for problems that may exist in certain parts of the world.
With a customer-centric focus on everything we deliver, quality becomes an extremely important consideration. We use specific metrics to measure the apps in our app store, examining the number of users/connections the app has, the average app rating, customer reviews as well as churn metrics. These metrics help us figure out which apps are doing well and how we can help developers become even more successful by solving issues with their integrations.

VM: How do you measure the success that developers are able to achieve by being part of your developer program?

KK: There are two key things we track to measure success:

  1. How we can make it a lucrative option for developers, and help them grow their business.
  2. How can we help developers build (quality) apps more quickly, efficiently

 First, let’s talk about how we are helping our developers be successful on QuickBooks and grow their business. To do this, we offer developers a channel to our customers via our personalized Apps Store and through relevant in-product discovery. For example, if you are a small business owner and you’re in the expenses component of the software, all of the apps relevant for expense tables can be discovered through in-product discovery In doing so, we are enabling apps to be “found” based on need and necessity. As a result, we can help developers retain –and gain – their customers.

We strive hard to increase the number of relevant leads to developers from QuickBooks. We are working on implementing analytics that help us monitor this metric. Please note that while we can deliver leads to developers, the actual conversion depends on the specific developer application. While the reasons are beyond our control, it is important to understand how we can ensure that the leads become actual customers – this is how we can help our developers succeed.

 Lately, we have been working with a handful of developers in efforts to share insights we are learning from customers and identify opportunities for developers. In doing this, we are educating developers about our customer pain points, while also sharing an analysis of where the gaps are, and how we are looking to better integrate their apps into our workflows so it becomes seamless.

 Now let’s talk about how we can help developers build quality apps quickly and efficiently., For this,  we look at it from the perspective of the overall developer experience – i.e. easy on-boarding of APIs with SDKs and documentation, API stability dashboards etc . If you go to our website we have details of how our APIs work, access to sample code, SDKs, etc. The website includes everything needed to build an app – and we also have a developer hub where to post questions.

 Then our focus shifts to helping developers publish an app. Each developer will work closely with a member of our team to get through the entire process. We conduct an annual security review and a technical review to make sure that the app works the way it is supposed to. Our experts will give developers feedback and guidance. We also conduct a marketing review, which entails looking at the app store page description, the introductory video, and how the integration works. We use a portion of our marketing budget to help developers position their app to our customers so they can be successful.

There are several other programs and opportunities to help developers find new customers, market their apps and grow their business. For example, last year Intuit hosted the ‘App Showdown’ contest, where almost 100 developers competed over the course of several months for the chance to be selected as one of ten finalists. The winner received a $100,000 prize that can be used to get their business to the next level. Finalists were chosen based on several metrics including quality of integration to QuickBooks, innovation, market impact, and number of users on QuickBooks. These are all metrics we are already looking at to ensure the success of apps on our platform.

Building an API program is about a business accepting its new market positioning as a platform. Third party developers become a key asset in how well the platform can grow. Not only do they encourage new customers to your business, but they also create greater lifetime value for existing customers by giving them more reasons to use your products and engage with your business in multiple ways.

Network effects of a platform is a core strength of the platform model, but requires an investment and recognition that there is no longer a go-it-alone mindset required. Now it is a raise-all-ships mentality that supports network partners to grow their businesses successfully as well. A platform that recognises that API developers are businesses and sets in place metrics to help those partners grow is a platform that has a greater chance at long term success. Intuit are building that into their DNA by focusing on the metrics of platform network effects.

VisionMobile measures developer satisfaction twice per year, across the industry’s 20+ leading developer programs on 20+ program features and services. If  you wish to know more about our Developer Program Benchmarking research drop us a line here: sales@visionmobile.com.

Disclaimer: Intuit is a customer of VisionMobile, but there was no financial motivation behind this article.

 

Developer Program Metrics: How Intel Measures Developer Satisfaction.

Many modern competitive battles have been won by attracting developers – whether it’s Apple in the case of mobile platforms, Amazon in cloud, or Salesforce in CRM platforms.

Companies are now investing 10s of millions of dollars in attracting and engaging developers to build extensions, products and partnerships around their platform. As these developer platforms and programs are becoming commonplace, we wanted to investigate their best practices; what makes them tick.

Scott Apeland oversees developer relations at Intel and has recently launched a whole new developer program aimed at encouraging developers to build new products with their artificial intelligence and machine learning tools. We spoke to Scott about how he identified which developers to target and how he is tracking the effectiveness of their developer program.

Intel hosts one of the largest developer programs in the world, with over 20 million active developers visiting their platform and making use of their APIs in projects and products.

With a central focus on developers as customers, Intel has been able to introduce more commonly understood metrics to measure customer satisfaction. One of these metrics is the Net Promoter Score (NPS), which asks: “How likely is it that you would recommend Intel Developer Zone to a friend or colleague?”

Intel survey their developers every three months and aggregate the responses. Developers rate their likelihood to recommend out of 10 Intel conducts the NPS surveys via monthly emails, anonymous feedback, through their Innovators and Managers Programs, and at roadshows and workshops.

In the first half of 2016, Intel received over 2,500 responses and 2,600 comments on developers experience with Intel. Measuring their results by business unit and overall, Intel has recorded a jump from an average NPS score of 19.5 in 2015 to 27 in 2016. The Internet of Things and Robotics developer tools jumped the greatest in terms of improvements in developer satisfaction during this time, while other developer programs such as mobile app development and 3D cameras also increased levels of developer satisfaction.

VisionMobile: How do you first identify which developers need to be targeted in a developer program?

Scott Apeland: Because Intel’s developer program is one of the most diverse and broadest, it is growing quickly. We had 16 million developers in 2015, and now we have reached 20 million in the Intel Developer Zone. This means we have to be really good at understanding different types of developer needs.

So we profile the developers in each audience segment and then create developer personas. We also use VisionMobile’s developer segmentation model to understand the different motivations of each segment and that’s very helpful for us too.

For example, in our data center developer program, we are working with HPC (High Performance Computing) developers with advanced C++ skills. They are really proficient at optimizing software to get the most out of the hardware. These developers are trying to get the most performance they can through techniques such as parallelization and vectorization. We help them by providing have the tools, training and support at the right level for these advanced developers who have years of experience.

But also in the data center program, we have developers who are trying to take advantage of brand new tech like network function virtualization. These developers need a different set of tools, and have different partners in the ecosystem and skillsets to deal with. That’s just in the data center world.

In the laptop space, we have a game developer program where we are helping devs take advantage of the latest platforms and create new experiences with 3D graphics, virtual reality, and 3D video. This is a whole different type of developer. And in the IoT space, you are often dealing with a “maker” looking at what problems they can solve using a combination of h/w and software together in a new innovative way. They are more likely to be individuals innovating by themselves but value sharing their knowledge and expertise in a community.

VM: How do you align building a developer program with overall business goals?

SA: Each business unit comes to us with their objective. For example, in the data center business unit, they have an objective to accelerate the growth of artificial intelligence (AI) and machine  learning technology. To achieve this we must have the developer community on board and partner with them on delivering open source frameworks and tools.

So we just rolled out our AI developer program on November 17. The program was front and center in Intel’s announcement and is focused on upstreaming tools, technology and training, making them available to all types of developers.

We’ve been working with the developer community for many years but each time we put together a new initiative we start by interviewing real developers and software experts both inside and outside of Intel. Internally we spoke with developers optimizing the AI frameworks and creating the software tools. Externally we spoke with the top developers in our black belt program. We have a close relationship with them and can discuss what’s important to them and how to make them successful. They provided us with valuable insight to refine the program and focus it squarely on real developer needs. At AI day we rolled out a new AI Developer Zone with frameworks, tools, tutorials, videos, sample code, a partnership with Coursera, a contest with Kaggle to solve early detection of cancer, and a brand new student developer program. Academia plays an important role in advancing AI, so we put a strong focus on that as well. All of those elements came together and culminated in our entrance to the market.

VM: How do you measure the success of your developer program?

SA: We use a number of metrics, including the number of developers using our technology and the impact their applications have in the market. But one of the most important success metrics is developer satisfaction because without that we can not maintain developer mindshare and support long term.  To measure satisfaction we use the Net Promoter Score (NPS). It gives us a top level benchmark: would developers recommend us to their colleagues? It is an indicator that management can use to assess how we are doing in general.

In addition to this score, what is even more valuable is the comments we get back from the survey. We get thousands of comments back (2,656 last quarter) around how we can improve things and that’s where we hear about the need for more additional tutorials and code samples, for example, or how the forums are tough to use online. We get continued feedback so we can really make this work for the developer side.

What I do is I make the NPS a top level goal in our division. The goal is to improve our score year over year, and every quarter, my staff will review our NPS results. Everyone will read all the comments and then we get together and decide on common themes that are emerging. For example, we may notice the need for more tutorials in Chinese or that a particular developer tool is difficult to use, and we take that feedback and we form an action team. That team is empowered to dive in and address the issue. The team includes a dev evangelist, a content author, a community manager, a web experience expert and a geo-rep for global input. They go and look at each comment in greater detail and put together a plan to improve that area. Then the team reports back every couple of weeks on their progress. Since we’ve started this approach we’ve been able to move the needle on NPS with a jump from 1.6 two year ago when we started to 27 today – you don’t see that kind of jump in the NPS world. NPS has really helped us in driving a customer-centric culture in our organization..

For any business opening up APIs to external developers, it is possible to measure the number of visits to the developer portal page or the number of developers requesting an API key. But in the same way that ‘likes’ in social media are often just a vanity metric, developer portal visits and API signups tell you very little about whether a developer program is successful or not.

Intel have found that by using the Net Promoter Score, they can engage regularly with their developers as first-class customers and support them to become champions of Intel’s products. It is not just a satisfaction metric that Intel have introduced, it is a key customer engagement strategy. They use the NPS and associated feedback as a continuous improvement mechanism to identify what developer resources they can next create and how to respond to new developer needs. The proof is in the results: each year, more and more developers are comfortable with recommending Intel to their colleagues.

Do you have best practices you ‘d like to share from your developer program? Drop us a line here: hello@visionmobile.com.

 

Disclaimer: Intel is a customer of VisionMobile, but there was no financial motivation behind this article.