Return on Developer Investment

My most fun job ever was as a C++ developer. Ok, I don’t have much grey hair yet, but I fondly remember the late 90s and the challenges of writing a background synchronisation application on a Compaq iPaq. And reverse engineering Mozilla’s Navigator into an XSLT parser.

My second most fun job ever has been building a company that helps the world understand developers, with research. We’ve come a long way – and a few pivots – from surveying the pulse of 400 developers in 2009 to 30,000 developers annually in 2016. That’s a lot of data – in fact more than our analyst team can chew.

It’s a privilege to be working with some of the biggest names in tech – I ‘ve learned a lot the past 2 years. Earlier this month, Amazon, Microsoft, Facebook, Adobe, Intel, Oracle and many more joined our first Future Developer Summit, and shared some of their best practices in how they work with developers. I ‘d like to share some the learnings here.

Return on Developer Investment.

You would think that with billions of dollars spent every year on building tools for developers, running hackathons, loyalty programs, tutorials and how-tos, evangelist and MVP programs – the platform leaders would have figured it all out. Yet, with so much money being spent on developer tools and marketing there is no standard for measuring the Return on Developer Investment.

Most companies represented at the Future Developer Summit shared how they measure success. At their inception, developer-facing orgs measure success by number of developers touched – but that’s a meaningless metric, a dinosaur from the age of print marketing. Some platforms are using NPS (net promoter score), polling their active developers once a year for how likely they are to recommend the platform. Many are informing product decisions based on developer comments (“will you ever fix that”?) – you’ll be surprised how many decisions are taken based on “the devs that I spoke to said..”.

Other developer relations teams are measuring success through the number of apps in the store, and the number of apps using signature APIs. In the case of open source projects, a popular metric is GitHub stars, forks and commits over time. The more sophisticated platforms track the Return on Developer Investment funnel from SDK downloads to app download and use. But there isn’t a consistent way to measure how the investments in hackathons, tutorials, how-tos, loaner devices, evangelism programs and some many more developer-facing activities are paying off for the likes of Google, Amazon and Facebook.

Quality of apps, not quantity.

Another theme of the Future Developer Summit was the need for quality, not quantity of applications at the start of an ecosystem. B2B ecosystems like Slack and Intuit prioritise quality; Poorly written messaging apps can damage not just the perception of Slack, but also the perception of chatbots in general. Similarly, a poorly written app for the QuickBooks platform can wreak havoc to sensitive financial data for thousands of small businesses. As a result both Slack and Intuit have very stringent app review processes, including weeks of testing, usability and security reviews. To improve quality for bots, Slack has pioneered a “Botness” program, bringing together bot platforms and leading bot developers; the aim is to “make bots suck less” i.e. improve the bot user experience and avert a long-term damage to the reputation of chat bots. There are already 250 members signed up and the next event is on November 4 in NYC .

The next Future Developer Summit will focus on best practices for developer relations. If you ‘d like to be part of the invite-only audience of platform leaders, register your interest at


Chatbots: From hype to “figuring it out”

Chatbots and conversational UI are among the most hyped technologies of 2016. Chatbots are computer programs that can maintain automated text conversations with users through use of artificial intelligence and natural language processing.

Early excitement about chatbots has given way to the realization that many challenges remain in both technology and understanding of what users are looking for when chatting with computers. In the words of Fred Wilson, a prominent venture capitalist who invested in Kik, a popular messaging platform:

“New user behaviors take time to develop and sometimes require a breakthrough app to get things started. That’s where we are with chatbots. The hype phase is over and we are now into the figuring it out phase. That’s usually when interesting stuff starts to happen.”

According to Comscore, smartphone apps now account for half the time Americans spend online. At the same time, the mobile app market has become saturated. It is extremely difficult and expensive right now to succeed with a consumer focused mobile app. The mobile app gold rush is over for all but large deep-pocket publishers. Mobile app developers are looking for the next big thing. 

While many challenges remain, chatbots promise to create a new channel for reaching mobile users alongside mobile apps. That’s why there is no shortage of entrepreneurs and developers taking their shots at “figuring it out” with chatbots.

All popular messaging services, including Facebook Messenger, WhatsApp, Skype, Telegram, Slack and Kik, are opening their platforms to innovation by 3rd party developers. In July 2016 Microsoft announced that 30,000 developers have signed up for the Microsoft Bot Framework. In the same month Facebook reported that developers created 11,000 bots for its Facebook Messenger platform. In August 2016 Kik, a smaller messaging app popular with teens in North America, reported that more than 20,000 bots have been built for Kik, and Kik users have exchanged almost 2 billion messages with chatbots since the company has opened its app to developers. The chatbot group on Facebook has close to 13,000 members, and a new chatbot is being announced almost every day.

The outcomes of the “figuring it out” phase in chatbots will be decided by developers. This is why we included questions about chatbot development in our 11th Developer Economics survey. 8,464 developers answered questions about chatbots and we published full results in the Chatbot Developer Landscape 2016 report.

The data shows that the efforts of Facebook, Microsoft and Slack to promote chatbot development achieved early results. An absolute majority of developers worldwide are aware of the opportunities in chatbots. At the same time, the chatbot ecosystem is still in its early stages and lots of work remains to attract masses of developers to the chatbot idea. Less than quarter of developers who are aware of chatbots are convinced of the chatbot appeal.

It’s notable that Apple and Google, the undisputed leaders of the mobile app world, fell far behind Facebook, Microsoft, and even smaller messaging apps in seizing the chatbot opportunity. Facebook is the undisputed global leader in developer mindshare with over 40% of developers interested in developing for the Facebook Messenger platform.

If you’d like to know more about what developers think about chatbots, take a look at our Chatbot Developer Landscape 2016 report, which answers the following questions:

  • What does the chatbot landscape look like?
  • How popular is the idea of chatbots with developers?
  • How mature is the chatbot developer ecosystem?
  • Which developers are more attracted to the chatbot promise?
  • Which messaging platforms are popular with chatbot developers?