How Okta is Broadening Their Developer Network with SlashData’s Developer Program Benchmarking Report

SlashData’s Developer Program Benchmarking report provides annual insights into developer behavior and trends that help tech companies make better, data-driven decisions.

Many Big Tech companies have come to trust our unique data insights in helping them understand developers better and shape their strategies.

Here’s how one of them –Okta– is using SlashData’s Developer Program Benchmarking to stand out and unlock more developer opportunities.

The Gap.

Optimize Developer Strategies that Boost Okta’s Developer Community.

Every tech company needs to have a better understanding of their developer-user base — so they can make more informed decisions. Plus, having a clearer view of the strengths and weaknesses of your developer programs will expose how you measure up against competitors and how to stand out.

To achieve this, it becomes necessary to:

  • Track your program adoption rate –anywhere in the world
  • Stay on top of developer satisfaction with your programs
  • See how competitor developer programs are scoring – what they’re doing right and the gaps you can capitalize on
  •  Measure the performance of your specific offerings based on what developers value

…and so on.

Tech companies like Okta are using SlashData’s Developer Program Benchmarking to do just that.

A Must-Have Resource for Developer Relations”

“Over the years, I’ve used SlashData’s reports to understand and prioritize what matters to developers. I’ve found the reports quite helpful in terms of identifying areas for improvement and the impact of our investments.

For example, when I was at Mozilla, we noticed our satisfaction score for blogs and newsletters wasn’t where we’d like it to be. So, we invested in creating a developer newsletter and improving the blog. 

And in 18 months, we saw our satisfaction score for this area rise to the top 3. This satisfaction score and other standard metrics (reach, opens, etc.) helped us determine how successful our program was.

Now, at Okta, I’m again using the report to identify areas for improvement and will invest in our documentation, blog and website. SlashData reports give us a trendline that shows, over time, if and how these investments pay off, and allow us to make changes if we are not seeing results.”

Ali Spivak

Sr. Director of Developer Relations, Okta [ex. Mozilla]

The Results.

Instead of second-guessing developers’ interests and neglecting areas of investments that count, Director of Developer Relations – Ali Spivak – could use SlashData’s reports to:

  • Help determine areas of investment and to see –over time — how those investments impact developer adoption and satisfaction level
  • Provide invaluable insights on how their program was performing and where their opportunities for growth lie
  • Shape and augment the user research they were already conducting and focus more on specific areas that required attention
  • Track developers’ satisfaction scores with their program and specific offerings
  • Benchmark how Okta’s developer programs stack up against other developer programs to provide them with a model of improvement
  • Help stakeholders understand what developers are experiencing and prioritizing, their needs, and also justify the investments made into developer programs

Here’s How We Can Work Together to Build a Thriving Developer Ecosystem…

In today’s diverse and complex developer ecosystem, there’s real power in knowing where to focus your efforts to help build a thriving developer community.

That’s what SlashData’s Developer Program Benchmarking helps you achieve.

With SlashData, you get:

Data-Driven Insights for Effective Decision-making

Data is the fuel that drives the engine of decision-making – as a data-driven company, we’re always aware of its immense value.

So, when you come to us and share your current challenges, we work to help you see and understand how much difference data-backed decision-making can make.

We’re always ready to engage and walk you through the data –so you can uncover and bridge the current gaps you’re experiencing.

Client-Tailored Global Developer Survey

SlashData’s Developer Program Benchmarking report is a syndicated research study available to tech companies who are subscribed to our program. It is based on data collected in our Global Developer Nation survey that’s fielded twice per year.

If your developer program hasn’t been benchmarked, we will include it in the upcoming editions of our in-depth research study.

There are many hidden potentials for your growth in engaging developers in a way that targets their needs and  improves their experiences.

That’s why we survey about 20 developer programs every 6 months, as the list of programs benchmarked is ever-changing. This helps us learn more about what boosts satisfaction and adoption rates for developers.

Full Access to Deliverables for Close Monitoring

We provide full access to the agreed deliverables as soon as they get completed.

These usually include a market landscape report, insights report specific to your company, and an interactive data dashboard that allows filtering and zeroing in on a particular developer segment.

Our clients have found these reports quite useful in helping them shape and even augment the user research they are already conducting on their own.

These deliverables will help you pinpoint the specific places you should be focusing on for in-depth user research.

Plus, it provides areas of opportunities where you can exploit advantages over competition to stand out.

Beyond ready to build stronger relationships with your developer community?

Get in touch and we will dive in together in the solutions that will win developers’ hearts.

How developers’ support needs change with experience

Developers have a wide variety of support and learning needs that evolve as they progress through their careers. Here, we’ll look at some of the best ways to help developers build on their skills by answering their technical questions, creating a valuable community that they can integrate with, and providing professional certifications as proof of learning. In a highly competitive job market, vendors can demonstrate value to developers by helping them to build on their skills and get an advantage in the job market.

Here, we take a look at data from two of our most recent Developer Nation surveys. In our Q1 2021 survey, we asked developers, amongst many other topics, how they prefer to communicate with vendors about technical topics. In our Q3 2021 survey, we took a deeper dive into developers’ views on what makes great technical certifications and what are the key features of a successful community. The data here is only a small sample of what we collect, so if this sparks some interesting questions for you, then please get in touch.

It’s a matter of experience

Data from our Q3 2021 survey, which was fielded between June and August 2021, shows that overall, there are more early-career developers (those with 0-2 years of experience) than highly-experienced developers (those with 11 or more years of experience). Developers with different levels of experience undoubtedly have different support needs (and we’ll come to this later), but taking a global perspective on experience levels risks missing some interesting regional variations.

South Asia and Western Europe sit at opposite ends of the experience spectrum – South Asia has the largest proportion of inexperienced developers, and Western Europe has the smallest. This means that when creating a regional strategy, not only should you think about the cultural and economic differences that exist between regions, but also, due to their experience levels, developers will have very different support needs.Developers in Western Europe are more experienced than average and far more experienced than those in South AsiaTechnically correct is the best kind of correct

We see here how developers’ support needs evolve as they gain experience. In fact, communicating with vendors about technical questions becomes more important as developers gain experience – more experienced developers are very likely working on more challenging projects and, as such, more often require expert support. What’s interesting is which communication channels become more important.

Email is consistently the most important, regardless of experience level. It seems that the power of direct, asynchronous communication is clear to all developers, though it does become more important to more experienced developers, as well as to older developers (and age is, of course, correlated with experience). On the other hand, other direct but synchronous communication methods such as online chat retain their importance to developers of all experience levels (but fall in importance for the oldest), whilst live interactive coding sessions only fall out of favour amongst the most experienced. Not every communication method is created equally, and neither is every technical question. Irrespective of their experience levels, developers want to engage directly to have their technical questions answered and are happy to do synchronously or asynchronously.

Issue trackers and code repositories nearly quadruple in importance for the most experienced developers when compared with the least experienced. Here, you have experienced developers asking their technical questions through established open-source channels that may feel inaccessible to less-experienced developers. There’s definitely scope to widen participation amongst inexperienced developers in this fundamental pillar of software development. We also see that Q&A sites steadily increase in importance as developers become more experienced. That’s not to say that inexperienced developers aren’t going to StackOverflow – they’re still using such sites to get information; it’s just that they are more likely to simply consume rather than ask technical questions of vendors.

Direct communication via email or chat is most important to developers at all experience levels

A sense of community

Interacting with vendors or peers through a code repository or on a Q&A site is one of the many ways in which developers interact with their community. Community support is a powerful facilitator of learning and development for many developers and is as much a source of inspiration as it is camaraderie. We see that developers of differing experience levels have very different ideas about what they want from a community, but collaboration and support are two of the most stable and important features to developers of all experience levels.

But experienced and inexperienced developers lean on their community support network in different ways. A knowledgeable community becomes more important to developers as they gain experience – here, these most experienced developers likely find more value in a community that can help them answer complex questions. On the other hand, inexperienced developers are more likely to look for strong leadership in a community – they are likely looking to more experienced members for guidance and learning opportunities.

Strong leadership and interactivity are less important aspects of a community to experienced developers

Certifiably important

Vendor support and community are just two of the myriad ways that developers build their skills throughout their careers, but in an increasingly competitive professional environment, many developers study for professional certifications to get an edge. Such certifications are important to developers at different stages of their professional life – early-career developers are likely looking to distinguish themselves from the masses, whilst seasoned professionals may want to protect their lucrative career or even switch specialisation. Regardless, because of certifications’ wide appeal, developers at all experience levels similarly agree on the importance of certifications being suitable for a variety of learning styles.

On the other hand, industry recognition, online availability, and affordability are three of the most important features of a professional certification program, and they become more important as developers gain experience. This demonstrates that as developers mature, they become more focused on the core aspects of professional certifications. We also see how their job-seeking habits change. The importance of recognition on job boards rises steadily from zero to five years of experience before falling sharply afterwards. This suggests that after around five years in the industry, developers have built their professional network and are less reliant on job boards, though the professional credibility of a certification is still paramount.

Developers at all experience levels recognise that many learning styles should be catered for

What does this all mean?

Here, we’ve seen that there is great variation in the experience levels of developers across the world, as well as between different geographical regions. We’ve also learnt that developers of different experience levels have very different views about the type of support they want to receive from vendors and from their communities, whether they are asking technical questions or becoming certified. Therefore, you should look at the experience levels of your user base and use this to figure out how best to support them. However, experience isn’t the whole story; our extensive research shows that a plethora of factors influence developers’ needs and decisions. Developers’ roles, level of decision-making seniority, industry, and technology choices all impact their needs for support. Understanding developers’ needs and behaviour requires not only a rich set of data but also extensive experience and knowledge to build the personas that inform a robust strategy.

Don’t know where to start? Well, at SlashData we have a wealth of experience in understanding developer behaviour through our twice-yearly global survey, as well as through numerous custom research projects with our clients and partners. We also have a deep and detailed body of research on developers through our Developer Program Benchmarking research. Get in touch to find out more.

You can also go through a case study that shows how Okta and Mozilla used the Developer Program Benchmarking to bring their developer program among the Top 3 in terms of developer satisfaction.

Google has the leading developer program, but Amazon is catching up

Developers. Decision-makers. Kingmakers?
For several years now, at SlashData we have been helping our clients – some of the biggest names in tech – to understand how their developer programs measure against the competition. Twice a year, we run an extensive and wide-ranging global survey to understand who developers are, what tools and resources they use, and where they are going. Developers share with us their experiences with vendors’ resources – which ones they use, how often they use them, and how happy they are with the experience. We also dig a little deeper into what developers value in vendor support, resources, and communities.

Our research shows that developers are becoming increasingly involved in all stages of the decision-making process. Not only are they writing specifications for vendors and tooling choices, but they are also influencing decision-makers and budget holders. If software is eating the world, then developers are writing the menu. 

To attract developers, many tech companies are actively investing in Developer Relations (DevRel) teams and developer marketing activities. They are creating an abundance of resources, training programs, technical support, events, and community activities. It’s not always clear which activities should be priorities and how resources should be allocated to achieve long-term strategic goals. We are here to help.

Our Developer Program Benchmarking research tracks 20+ of the leading developer programs, and captures developer sentiment across more than twenty developer program attributes, ranging from documentation and sample code to mentoring programs and access to experts. In so doing, it helps DevRel and developer marketing practitioners understand how their developer program compares against the rest.

Here, we give you a snapshot of the state of play for these developer programs. We use three KPIs to create a 360° overview of how each developer program performs:

  1. Adoption – How many developers use a vendor’s resources
  2. Engagement – How frequently developers engage with the resources
  3. Satisfaction – How developers rate their experience using the resources

bubble chart showing how developer perceive the leading developer programs

We can see that the Market Leaders; Google, Microsoft, and Amazon highly engage and satisfy developers. Their market share – or adoption rate, shown by the size of the bubble – reinforces their market-leading position. In fact, when we take a longer-term view of this data, it becomes clear that Google and Microsoft have long been the market leaders, staying at or near the top of the table for all three KPIs. 

Recently however, Amazon has made considerable progress. In fact, Amazon’s developer program has been growing faster than the global developer population, which is currently 24.3M (you can explore more in our developer population calculator), while Google and Microsoft’s share has dropped slightly. When you take into account the large increase in Amazon’s satisfaction score and their aggressive growth strategy, the top table positions don’t seem so assured.

Our data also uncovers the Satisfying Specialists – these developer programs are often small and focused. Unity, Red Hat and DigitalOcean sit firmly in this space. Developers don’t need to engage frequently with these vendors’ resources, but when they do, they have an excellent experience. For these vendors, low engagement is not a cause for concern, though it does come with its own challenges – when developers have fewer touchpoints there are fewer opportunities to speak to them or to influence their behaviour. For these (and other) vendors with low engagement, messaging becomes vital. 

The Under-realised Value segment contains developer programs that, although having high engagement amongst developers, are being held back by their low satisfaction ratings. These programs are often (though not always) small, and the vendors here have a clear imperative to improve their developers’ experience. Thankfully, with developers engaging frequently with the resources there are ample opportunities to effect positive change.

But what, exactly, to change? 

This brings us to the true power of our Developer Program Benchmarking research. Not only do we understand how developers engage with vendors’ resources, but we also know which resources are important to developers, and how satisfied they are with the resources that companies provide. 

Though developers’ preferences change and evolve, some things stay constant. Of the twenty-plus resources that we ask about, documentation & sample code, tutorials & how-to videos, and development tools, integrations & libraries have consistently been rated as the most important resources that companies should offer. This shows that developers are focused not only on getting things done, using documentation and development tools to speed up the development process, but they also highly value having the opportunity to learn. We can see this repeated further down the list – training courses & hands-on labs provide the learning opportunities, whilst technical support allows them to lean on experts when they need to.

Table showing the 5 resources: documentation, tutorials, development tools, training courses and technical support

In this way, we can tell which resources developers value, and how their experience matches their expectations. This information, when combined with our wealth of survey data on demographics, firmographics, technology choices, motivations, skills, and much more, becomes incredibly powerful for informing strategic planning. We help some of the leading tech companies in the world to understand precisely which resources need improvement, and which developers will benefit most from such improvements. Have you ever wanted to know how to tailor your tutorials to the right level of complexity? Have you ever tried to decide how to localise your content? What about marketing to enterprise developers, what do they care about? 

We also go a level deeper. For many developer programs, we specifically ask developers how they use resources relating to different products or disciplines. For example, we help developer programs to understand whether or not they are vulnerable in the cloud compute market, or what are the specific preferences of developers using IoT resources. Once again, coupled with the rest of our rich and diverse data, this information allows you to create a finely tuned strategy that allocates resources efficiently and effectively.

With developers having such power in the decision-making process, this is a win-win for everyone involved. By understanding what developers value, you can tailor your offering to suit their needs, increasing retention, growing your audience, and ultimately, adding to your bottom line. SlashData are the analysts of the developer nation, and we can help you understand developers.

You can download a preview of the latest Developer Programs Benchmarking here.

The Winners Are Here – Developers’ Choice Awards 2018 Digital Edition

Google, Mozilla, Unreal, Unity, Amazon, and Microsoft Win SlashData’s Developers’ Choice Awards 2018 Digital Edition

Continue reading The Winners Are Here – Developers’ Choice Awards 2018 Digital Edition

How Satisfied are Software Developers with Google, Amazon, and Facebook?

The absolute leader in developer satisfaction is Unity; according to SlashData’s recently released Developer Program Benchmarking report. This proves that not only the companies with the most traction and the biggest budgets can create excellent developer support programs.

Every six months SlashData benchmarks the top developer programs against each other, in the largest study of its kind. Last week we released the Developer Program Benchmarking report where over 21,000 respondents from 150 countries across mobile, desktop, IoT, cloud, web, games, AR/VR and machine learning and data science were asked which development resources they use and how satisfied they are with them.

Developer Program Benchmarking H2 2017 Satisfaction Chart

Google, Microsoft, and Mozilla are not only among the largest developer programs; they lead the pack in terms of developer satisfaction and engagement. Other major developer companies like Amazon, Facebook, Oracle, and Apple follow at some distance.

Unity and Unreal, unlike Microsoft, Google, Amazon, and others, cater to the specific niche of game developers. Unity has the highest developer satisfaction of all programs in our list; Unreal is number three in the list. Tencent, the producer of WeChat who mostly addresses a geographical developer segment in China, has a level of engagement that’s on par with giants like Facebook and Amazon (its Western counterparts in some sense), despite being one of the smallest programs in our survey. Other companies like NVIDIA and Cisco may have moderate overall performances, but lead the way in important attributes such as training or access to devices.

To reach these results, first we measure what developers value in resources and activities, in all its diversity across several segments of the developer population. Second, by measuring each program’s impact in terms of adoption, engagement (frequency of use), and developer satisfaction. Third, by highlighting the best practice leaders: those vendors that are doing an excellent job in specific aspects of developer programs, to whom you can look for inspiration and insights on how to improve. There is no single leader across all of the 20 activities we measure – everyone can improve somewhere.

“In this third edition of Developer Program Benchmarking, we matched for the first time how budget allocation in major programs aligns with the expectations of developers. We now have the data to back up our long-time suspicion that events and conferences may not be the best value for money. Also for the first time, we show how expectations are shifting, i.e. which activities are becoming more important to developers over time. We now have a more complete picture of how to get the best return on investment for developer-facing activities.” notes Stijn Schuermans, Senior Business Analyst at SlashData.

To access the full study drop us a note at or download the brochure.

SlashData Announces the Developer Satisfaction Awards 2017

Menlo Park, 10 October 2017. SlashData is excited to announce that nine organisations from the software industry have been unveiled as leaders for developer satisfaction at the 2017 Future Developer Summit 2017 (, an invite-only, Director-level event held at Menlo Park, California this week.

Companies including Amazon, Apple, Cisco, Epic Games, Google, Microsoft, Mozilla, Oracle and Unity were awarded for striving to place developers at the heart of their business.

The Developer Satisfaction Awards recognise the software products and brands that developers are most satisfied with. Results are based on the independent and unbiased opinions of over 40,000 developers surveyed annually, from around the globe, combined with SlashData’s rigid research methodology. SlashData (formerly known as VisionMobile) is an independent research firm helping the top-100 software companies measure how satisfied developers are with their products.

Andreas Constantinou CEO of SlashData says “The software industry has lacked an objective metric for measuring developer satisfaction, engagement and awareness – until now. The Developer Satisfaction Awards reveal the real developer experience – not an analyst opinion – on how the leading software companies are performing in winning the hearts and minds of developers around the world.”

The Developer Satisfaction Awards were presented in front of an impressive audience of 35 Director-level attendees representing 35 leading software companies, from Adobe to Salesforce.

SlashData would like to thank Richard Hurring (Founder and CEO of Catchy Agency) and Bhavesh Patel (President & CEO of for presenting the Developer Satisfaction awards, respectively, at the Future Developer Summit this week.

Full list of winners and runners-up

1.Developer Satisfaction with Database as a Service

WINNER: Microsoft Azure SQL Database

1st RUNNER-UP: Amazon RDS

2ND RUNNER-UP: Amazon Aurora

2.Developer Satisfaction with Game engines


1st RUNNER-UP: Epic Games’ Unreal

2ND RUNNER-UP: Apple’s Spritekit /Scenekit

3.Developer satisfaction with developer programs


1st RUNNER-UP: Google

2ND RUNNER-UP: Epic Games’ Unreal

4.Developer satisfaction with training in developer programs


1st RUNNER-UP: Microsoft


5.Developer satisfaction with documentation in developer programs

WINNER: Mozilla

1st RUNNER-UP: Unity


6.Developer engagement with developer programs

WINNER: Google

1st RUNNER-UP: Microsoft

2ND RUNNER-UP: Mozilla

For a full list of the finalists for each category as well as more information about the awards you can visit the Developer Satisfaction Awards 2017 website.

Unity leads the way in developer satisfaction

As software continues to eat the world (to paraphrase Marc Andreessen), software developers fulfill an ever more critical role in the progress of technology and, by extension, society. Supporting developer productivity is good for business. Those developers then become innovators – co-creators – that give a boost to your core business.

It’s also challenging. Developer programs consist of a myriad of activities, ranging from simple providing sample code and developer education, to tooling, to in-person events and online communication. It’s hard to be great at everything, and it’s hard to allocate effort and money effectively for maximum impact.

Every six months we benchmark top developer programs against each other. First, by measuring what developers value in those resources and activities, in all its diversity across several segments of the developer population. Second, by highlighting the best practice leaders: those vendors that are doing an excellent job in specific aspects of developer programs, to whom you can look for inspiration and insights on how to improve. There is no single leader across all of the 20 activities we measure – everyone can improve somewhere.

unity leads developer satisfaction

The top spot in terms of developer satisfaction is taken by Unity, with an overall developer satisfaction score of 75 out of 100. Unity shows exceptional performance on several attributes: tutorials, how-to videos & webinars, and official forums. This may be skewed by the fact that their products cater to a specific subset of developers (game developers) who might score attributes differently than others.

Google, Microsoft, and Mozilla are not only among the largest developer programs; they lead the pack in terms of developer satisfaction and engagement. Other major developer companies like Amazon, Facebook, Oracle, and Apple follow at some distance.

This doesn’t imply, however, that only the companies with the most traction and the biggest budgets can create excellent developer support programs. The living proof of that are Unity and Tencent. As we said, Unity has the highest developer satisfaction of all programs in our list. Tencent, the producer of WeChat who mostly addresses a geographical developer segment in China, has a developer satisfaction on par with Facebook and well beyond Twitter’s, and one of the highest levels of engagement in our survey. Other companies like Intel and Cisco may have moderate overall performances, but lead the way in important attributes such as training, technical support, or access to devices.

The study above shows data from the 12th edition SlashData Developer Economics survey. Over 21,200 respondents were asked which developer programs they used and how satisfied they are with them. These respondents came from 162 countries around the world and span mobile, desktop, IoT, cloud, AR/VR and machine learning developers and data scientists. The results were collected by SlashData over a period of six weeks between November and December 2016.

To access the full study drop us a note at or download the brochure

How Orange doubled the size of its developer community in 18 months.

Orange is one of the world’s leading operators with 265 million customers in 29 countries, offering fixed and mobile connectivity services. Orange launched its Orange Partner program for developers several years ago and now offers identity, payment, communication, cloud, IoT and proximity APIs. We spoke to Thierry Gaillet, developer advocate and API evangelist for Orange, who shared some of the practices that the telco used to double the size of its developer community in 18 months.

In addition to marketing, community activities and events, Orange Partner runs two to three hackathons a year, each of which includes a meticulous developer recruitment, selection and preparation process, and a closely-monitored competition and follow-up phase. Orange’s network has grown to 3,000+ registered developers, while developers get the opportunity to create customer-tested solutions and take them to market.

VisionMobile: You recently ran two large-scale hackathons in France. Could you share some of the best practices you followed?

Thierry: What differentiates the hackathons we organize, is that we partner with industry partners to understand and address the challenges they face. We carefully plan and supervise every phase of the ‘hackathon cycle’, which aims to cultivate long-term relationships between developers and industry partners rather than to competition for competition’s sake.

We give participants a lot of time to prepare and study the submitted themes, use-cases and challenges, we make sure our people are available to assist and listen to developers, and also invest in their future by mentoring winning teams.

Initially, we promote the event to gather potential participants in our network. There are two types of teams we take on during the qualification process; one is high-potential, less experienced developers. The second type is more mature IT professionals, with a lot more experience under their belt, often in the form of ready-shaped teams working in enterprises.

Once we have a shortlist of candidates, we organize a networking kick-off, mixing teams (including developers, designers and marketers) as well as experts from our partners and Orange and make relevant technical resources, documentation and supporting material available to them, in order to train them for the actual competition.

I’ve been a participant in hackathons myself in the past and remember distinctly how difficult it is to be introduced to tools and hardware right when the clock starts ticking for you to build a prototype. We avoid this by investing in a long preparation process, making sure candidates – including developers and other professionals – can focus on what they are developing, during the competition.

This is what we call the “animation” phase, which lasts from ten days to a month. We monitor discussion channels for Q&A regarding the hackathon, share video tutorials and host webinars about the IoT hardware and all the resources which will be available during the competition. This way, when developers come to compete, we know they will be comfortable taking the challenges based on detailed use-cases, they will understand the communication protocols for example, or how to use the hardware kits they are given to work with.

The two-day hackathon starts with participants coding straightaway and building tangible prototypes to solve real-life problems, which is appealing as it carries the aspect of usefulness and usability. As an example, one of the teams created a smart safety helmet for factory workers, which would check and communicate whether workers are properly equipped in factories or building sites, at the right time and the right place.

The team members met during the event kick-off of the Industry 4.0 hackathon and received €4,000 as a prize, sponsored by EDF,  a leader in low-carbon energies and Air Liquide, the leader in gases, technologies and services for Industry and Health. This team then launched their own start-up: e-novact and industrialized their connected helmet solution which is now available to large industries.

VisionMobile: What experiences can you share from your most recent hackathon?

Thierry: The most recent event we organized was the LoRa IoT challenge, in collaboration with Objenious, a subsidiary to Bouygues Telecom, which was held between December 2016 and the 18th of January 2017. This was appealing for us because as competitors we have contrasting agendas, but also a shared interest to promote LoRa®, the Low Power Wide Area Network (LPWAN) technology. As members of the LoRa Alliance, we share the same IoT goals and want to demonstrate to the French market that the two leading companies are joining forces to promote the same technology.

The 25 participating teams of developers and other professionals were called to solve real-life challenges presented by three of our partners:

  • Groupama, a leader in the insurance industry, covering 70% of French agricultural businesses, which was looking to mitigate risks of agricultural activities,
  • Schneider Electric, global specialist in energy management and automation, who were looking to optimize electricity distribution, and
  • Colas, a leader in the construction and maintenance of transport infrastructure, interested in embedding LoRa sensors in bridges, carparks and more locations, to monitor traffic and improve driver experience.

Together, these teams have received a total of €30,000, comprised of six prizes.

From the IoT LoRa Challenge and previous hackathons, we had various teams applying for the Orange Fab accelerator start-up program – from which over 240 start-ups have “graduated” so far. We often invite start-ups on our booths to large events, as Viva Technology, to let them showcase their products/services. We also often present start-ups to our Go-Ignite alliance counterparts (Deutsche Telekom, Telefonicá and SingTel) to widen the scope with a total addressable market of more than a billion customers worldwide.

VisionMobile: What is the greatest benefit for Orange, from these hackathons?

Thierry: The placement of a hackathon-borne solution on the market is not a requirement but part of our goal to promote innovation. Part of this is ensuring participants are serious about joining our hackathon and spending quality time, ideally to fully develop a solution, which is the primary reason we run our hackathons on weekdays.

The second reason, is making our experts available during this time. The Orange Partner experts are present at the hackathon to both make sure participants are comfortable using the resources available, but also receive first-hand feedback from developers. This feedback is key to our product development. It informs us of what technologies and tools developers in our community are using and like working with.

We get the most valuable feedback as we speak to our developers at the time they’re solving a problem, when we can get to the core of how they work. Participants come to hack with their own tools, they know what our competition is doing so this is an opportunity for us to understand why they are using each tool, what process and platforms our competitors use and through these understand how to improve our products based on this feedback.

Looking at our developer program, these hackathons are an important way for us to engage developers in our community, in addition to our digital marketing and participation to key events and exhibitions. In the past 18 months, we have doubled the number of developers in our network, as well as other professionals such as business developers, marketers and designers – not just with the hackathons themselves, of course, but with various meet-ups and promotion campaigns targeting fast growing market segments, such as West African countries and Egypt, with many innovative startups and small businesses using our SMS, USSD and payment APIs, for instance.

We’ve also managed to better qualify our customer or developer base from the feedback that we have during various events. It’s very important that we offer something tangible for the community, not just announcing a new product or a competition, while these hackathons are also a way for us to make sure we have a more direct contact with our community.

VisionMobile: What are you looking to improve in the next hackathons?

Thierry: One thing which needs to be improved is the right mix of participants’ profiles, in these events. We aim to grow our developer network and also make sure participants are serious about what they do. In similar events there are always the “hackathon professionals” who are well prepared and pitch-winners of earlier, similar events. We try to limit the number of participating pros, to make sure we allow for fresh talent to surface.

Another challenge we have faced is the simplification of the legal framework of the overall event, like the need to provide contracts protecting participants’ intellectual property and solutions. As those contracts involved multiple partners, we can now rely on a solid foundation for planning the next hackathons.

On another note, we plan to arrange hackathons in other French cities. We know we’re missing some very interesting teams who cannot attend the Paris events, and we don’t want to restrict these projects inside the capital city.

Finally, we’re working on ways to improve even further what we do. For example, we’d like to be able to give away the resources we make available at the event. In our last event this included kits by Microchip Technology and Sagemcom, and our own devices, such as Pops (embedded computers with BLE/SMS gateways), beacons or other systems.

Did you know that VisionMobile measures developer program satisfaction twice per year? We survey 40,000 developers annually on their attitudes and satisfaction across the industry’s 20 leading developer programs.

Disclaimer: Orange is a customer of VisionMobile, but there was no financial motivation behind this article.

Developer Program Metrics: How Intuit Measures Developer Success

How do you grow a successful developer community around a financial product? Intuit’s Quickbooks is a well-known small business accounting software that has 100s of accounting applications built on top of the platform, from tax planners to CRM and HR apps. We spoke to Intuit to better understand the mechanisms the company uses to make sure its developer ecosystem succeeds.

Originally a desktop accounting software, QuickBooks has moved to a Software-as-a-Service model with a clear vision of the core product features it can enable through its service model. But they also know that customers often need additional functionality, so they have created APIs to let third party providers build those features.

QuickBooks found that by offering a range of app integrations with their main product, customers spent less time moving data from one system to another and saved time, increased their satisfaction with the product, and ended up looking for more ways to use QuickBooks in their daily business systems.

In 2014, when QuickBooks API program was in its infancy, they offered up to 50 third-party apps, and began to see the impact this was having on their customer lifetime value. They decided to invest more heavily in their API program and now they have over 450 published apps (and over 1,200 in total, including private integrations) and thousands of developers building on the platform.

The network of app developers has provided a wide choice of high quality app integrations to small businesses using QuickBooks, solving important problems in its user base. This has helped QuickBooks strengthen relationships with its existing customer base and also grow the base.

The QuickBooks Online platform is now growing a two-sided “network effects“ platform by delivering benefits to both developers and small businesses.

We spoke to Ketan Kittur, Director of Product Management and Partner Integrations at Intuit to understand how Intuit ensures that its app developers are successful. “The two-sided value of our platform is critical. We want to be serious about measuring retention and quality, so that we can give our customers the best experience and outcomes for running their business. We also want to understand the value these integrations are bringing to our product,” Kittur says.

VisionMobile: How do you first identify which developers need to be targeted in a developer program?

Ketan Kittur: We look at what problems our small business customers are facing that are not part of our product roadmap. Our developer team then scans the developer community – local and abroad- for potential solutions to the problems. The identified developers may already have solutions or they may include developers who are interested in building applications to address these problems. Our outreach team then works closely with the developers and guides them through the integration process, so their app integrations are of high quality delivering value to the customers.

Through these discussions, we have found that there are two main things that matter for developers:

  1. They are also entrepreneurs. They are serious business people building their own products and customer base.
  2. They do not want to waste a lot of time. Their development effort needs to return value.

VM: How do you align building a developer program with overall business goals?

KK: Our goal is to help our small businesses be successful using QuickBooks. For problems that are not solved by QuickBooks, we need a thriving a developer community delivering high quality app integrations so we have a highly engaged user base of QuickBooks users. Hence, the goal of both our developer team and developers are aligned- solve critical problems for small businesses using the QuickBooks platform and grow our businesses. When your company has a global footprint like Intuit, we become an attractive platform for developers, allowing them to grow beyond their geographic reach and inspire them to solve for problems that may exist in certain parts of the world.
With a customer-centric focus on everything we deliver, quality becomes an extremely important consideration. We use specific metrics to measure the apps in our app store, examining the number of users/connections the app has, the average app rating, customer reviews as well as churn metrics. These metrics help us figure out which apps are doing well and how we can help developers become even more successful by solving issues with their integrations.

VM: How do you measure the success that developers are able to achieve by being part of your developer program?

KK: There are two key things we track to measure success:

  1. How we can make it a lucrative option for developers, and help them grow their business.
  2. How can we help developers build (quality) apps more quickly, efficiently

 First, let’s talk about how we are helping our developers be successful on QuickBooks and grow their business. To do this, we offer developers a channel to our customers via our personalized Apps Store and through relevant in-product discovery. For example, if you are a small business owner and you’re in the expenses component of the software, all of the apps relevant for expense tables can be discovered through in-product discovery In doing so, we are enabling apps to be “found” based on need and necessity. As a result, we can help developers retain –and gain – their customers.

We strive hard to increase the number of relevant leads to developers from QuickBooks. We are working on implementing analytics that help us monitor this metric. Please note that while we can deliver leads to developers, the actual conversion depends on the specific developer application. While the reasons are beyond our control, it is important to understand how we can ensure that the leads become actual customers – this is how we can help our developers succeed.

 Lately, we have been working with a handful of developers in efforts to share insights we are learning from customers and identify opportunities for developers. In doing this, we are educating developers about our customer pain points, while also sharing an analysis of where the gaps are, and how we are looking to better integrate their apps into our workflows so it becomes seamless.

 Now let’s talk about how we can help developers build quality apps quickly and efficiently., For this,  we look at it from the perspective of the overall developer experience – i.e. easy on-boarding of APIs with SDKs and documentation, API stability dashboards etc . If you go to our website we have details of how our APIs work, access to sample code, SDKs, etc. The website includes everything needed to build an app – and we also have a developer hub where to post questions.

 Then our focus shifts to helping developers publish an app. Each developer will work closely with a member of our team to get through the entire process. We conduct an annual security review and a technical review to make sure that the app works the way it is supposed to. Our experts will give developers feedback and guidance. We also conduct a marketing review, which entails looking at the app store page description, the introductory video, and how the integration works. We use a portion of our marketing budget to help developers position their app to our customers so they can be successful.

There are several other programs and opportunities to help developers find new customers, market their apps and grow their business. For example, last year Intuit hosted the ‘App Showdown’ contest, where almost 100 developers competed over the course of several months for the chance to be selected as one of ten finalists. The winner received a $100,000 prize that can be used to get their business to the next level. Finalists were chosen based on several metrics including quality of integration to QuickBooks, innovation, market impact, and number of users on QuickBooks. These are all metrics we are already looking at to ensure the success of apps on our platform.

Building an API program is about a business accepting its new market positioning as a platform. Third party developers become a key asset in how well the platform can grow. Not only do they encourage new customers to your business, but they also create greater lifetime value for existing customers by giving them more reasons to use your products and engage with your business in multiple ways.

Network effects of a platform is a core strength of the platform model, but requires an investment and recognition that there is no longer a go-it-alone mindset required. Now it is a raise-all-ships mentality that supports network partners to grow their businesses successfully as well. A platform that recognises that API developers are businesses and sets in place metrics to help those partners grow is a platform that has a greater chance at long term success. Intuit are building that into their DNA by focusing on the metrics of platform network effects.

VisionMobile measures developer satisfaction twice per year, across the industry’s 20+ leading developer programs on 20+ program features and services. If  you wish to know more about our Developer Program Benchmarking research drop us a line here:

Disclaimer: Intuit is a customer of VisionMobile, but there was no financial motivation behind this article.


Return on Developer Investment

My most fun job ever was as a C++ developer. Ok, I don’t have much grey hair yet, but I fondly remember the late 90s and the challenges of writing a background synchronisation application on a Compaq iPaq. And reverse engineering Mozilla’s Navigator into an XSLT parser.

My second most fun job ever has been building a company that helps the world understand developers, with research. We’ve come a long way – and a few pivots – from surveying the pulse of 400 developers in 2009 to 30,000 developers annually in 2016. That’s a lot of data – in fact more than our analyst team can chew.

It’s a privilege to be working with some of the biggest names in tech – I ‘ve learned a lot the past 2 years. Earlier this month, Amazon, Microsoft, Facebook, Adobe, Intel, Oracle and many more joined our first Future Developer Summit, and shared some of their best practices in how they work with developers. I ‘d like to share some the learnings here.

Return on Developer Investment.

You would think that with billions of dollars spent every year on building tools for developers, running hackathons, loyalty programs, tutorials and how-tos, evangelist and MVP programs – the platform leaders would have figured it all out. Yet, with so much money being spent on developer tools and marketing there is no standard for measuring the Return on Developer Investment.

Most companies represented at the Future Developer Summit shared how they measure success. At their inception, developer-facing orgs measure success by number of developers touched – but that’s a meaningless metric, a dinosaur from the age of print marketing. Some platforms are using NPS (net promoter score), polling their active developers once a year for how likely they are to recommend the platform. Many are informing product decisions based on developer comments (“will you ever fix that”?) – you’ll be surprised how many decisions are taken based on “the devs that I spoke to said..”.

Other developer relations teams are measuring success through the number of apps in the store, and the number of apps using signature APIs. In the case of open source projects, a popular metric is GitHub stars, forks and commits over time. The more sophisticated platforms track the Return on Developer Investment funnel from SDK downloads to app download and use. But there isn’t a consistent way to measure how the investments in hackathons, tutorials, how-tos, loaner devices, evangelism programs and some many more developer-facing activities are paying off for the likes of Google, Amazon and Facebook.

Quality of apps, not quantity.

Another theme of the Future Developer Summit was the need for quality, not quantity of applications at the start of an ecosystem. B2B ecosystems like Slack and Intuit prioritise quality; Poorly written messaging apps can damage not just the perception of Slack, but also the perception of chatbots in general. Similarly, a poorly written app for the QuickBooks platform can wreak havoc to sensitive financial data for thousands of small businesses. As a result both Slack and Intuit have very stringent app review processes, including weeks of testing, usability and security reviews. To improve quality for bots, Slack has pioneered a “Botness” program, bringing together bot platforms and leading bot developers; the aim is to “make bots suck less” i.e. improve the bot user experience and avert a long-term damage to the reputation of chat bots. There are already 250 members signed up and the next event is on November 4 in NYC .

The next Future Developer Summit will focus on best practices for developer relations. If you ‘d like to be part of the invite-only audience of platform leaders, register your interest at