Artificial intelligence is revolutionizing the business landscape, promising to introduce efficiencies and improvements across functions. One such area is software development and coding: Generative AI (GenAI)-based models can create code snippets, suggest optimizations or even automate entire coding tasks — with the goal of accelerating development and reducing repetitive work.
As the field continues to evolve, AI for programmers, especially GenAI for coding, is poised to address long-standing challenges, such as reducing bugs, improving documentation, expediting and enhancing code, and streamlining collaboration among team members. The question is: How well is GenAI working? Are organizations actually realizing improvements in these areas today?
What the Data Shows
Various studies and surveys highlight the promise of GenAI in these areas. For example:
- According to a Stack Overflow survey, one-third of developers (33%) see increasing productivity as the most valuable benefit of integrating AI coding tools into their workflows.
- 7 in 10 (70%) users of Microsoft’s GitHub Copilot — a GenAI-based coding assistant and developer tool — say they are more productive with Copilot, and 85% feel more confident in their code quality, according to GitHub studies.
These benefits and use cases are exciting, as engineering teams seek to most effectively harness GenAI’s potential. However, perceived productivity gains and self-reported results don’t always match up with real-world data collected in the field. As engineering teams employ GenAI-based developer tools like Copilot, let’s look at what a new study is reporting:
Can You ‘Uplevel’ Your Software Development Team and Productivity with GenAI?
Recent research from engineering intelligence platform Uplevel indicates that some GenAI-based developer tools may not consistently increase coding efficiency and could even increase bug rates.
Uplevel’s study analyzed data from a sample of 800 software developers — looking at engineering metrics pre- and post-implementation of Copilot. The developers were segmented into a test group (those with Copilot access) and control group (those without access). Here’s what the research found:
- A 41% increase in bug rate post-implementation of Copilot.
- No increase to efficiency metrics — including no significant change in coding speed or pull request (PR) cycle time (that is, the time to merge code into a repository) with Copilot use.
- Copilot did not effectively alleviate burnout — with a more substantial reduction in burnout indicators observed among those not using Copilot at all.
Uplevel representatives note that this data doesn’t mean engineering teams should abandon their use of GenAI for coding, though.
“These are powerful tools that are emerging,” says Matt Hoffman, product manager and data analyst, Uplevel. “Although we didn’t see a positive effect on productivity in this research, technology is moving super fast. We want to keep following the space and doing these kinds of studies. We also think it’s important that organizations approach their implementations with an experimental mindset. Look at things like: ‘What are the blockers for your teams?’ ‘What’s slowing you down?’ Then measure whether something like Copilot moves the needle in those areas.”
To that end, in its report on the study’s findings, Uplevel encourages organizations to:
- Set specific goals for the use of GenAI-based coding tools
- Offer training to their teams
- Continue to experiment with the technology in a variety of areas and use cases
- Monitor the effectiveness metrics tools like Copilot may impact
As Uplevel CEO Joe Levy further emphasizes: “Engineering teams today seek to allocate their time to the highest value work, complete that work as effectively as possible, and do so without burning out. They look at data to drive decision-making, and right now, the data doesn’t show appreciable gains in these specific areas through generative AI. But we’re not suggesting that developers ignore GenAI-based tools like Copilot, Gemini or CodeWhisperer. These tools are all new, there is a learning curve, and most teams have yet to land on the most effective use cases that improve productivity.”
The Bottom Line
AI for developers isn’t going away and stands to have a profound impact on transforming the software development landscape. However, as with any transformative technology, it comes with its own set of challenges and limitations that developers and organizations need to address.
By following best practices, measuring productivity, and staying apprised of timely and relevant GenAI data, developers can best harness the power of AI tools, and know when and how to complement them with human expertise.
As change barrels ahead, engineering teams must continue to monitor and adapt their AI tool usage, ensuring that it aligns with their broader goals and drives meaningful improvements in key software development processes.