Researchers, who claim to have conducted the largest-scale study of gender bias to date have found that code written by female programmers is rated more highly than code written by men. However, they also found that this higher rating — based on code acceptance from other coders — only proved true when their peers didn’t realize the code had been written by a woman. Acceptance of their contributions then fell below the acceptance level of code written by men.

To examine the prevalence of gender bias within the world of open source programming, researchers from California Polytechnic State University and North Carolina State University analyzed user behavior on one of the largest open-source software code repositories in the world, known as GitHub. The community consists of more than 12 million users, and the gender is apparent in 1.4 million of these profiles.

“Our results suggest that although women on GitHub may be more competent overall, bias against them exists nonetheless,” the authors explain.

The researchers took a thorough look at approximately three million pull requests, or suggested changes to code, submitted on GitHub on April 1, 2015, and found that code written by women was approved at a rate of 78.6 percent versus 74.6 percent for code written by men. Not only that, but 25 percent of women had almost 100 percent of their pull requests accepted, while only about 13.5 percent of men reached a 100-percent acceptance rate.

“Women’s acceptance rates dominate over men’s for every programming language in the top 10, to various degrees,” the researchers found.

In order to determine what could be causing this statistic, the researchers looked at a number of different factors, including whether they were seeing the outcome of “survivorship bias,” where individuals who last longer tend to make more contributions. In other words, were the women outpacing men because they stuck it out? What they found was that women contributed more often than men no matter how long they had been involved in open source. They also looked at whether women were making smaller changes to code and found that they were not, and at whether women were outperforming men in only certain kinds of code. Once again, they were not.

They checked other factors as well, such as whether women were contributing “more valuable” pull requests in response to known issues. But women actually responded less to known issues or bugs than men did.

The researchers then examined whether women were benefiting from reverse bias — the desire of developers to promote the work of women in a field where they are such a small minority. To answer this, the authors differentiated between women whose profiles made it clear that they were female, and women developers whose profiles were gender-neutral. It was here that they discovered the women’s work to be more likely to be accepted than men’s unless “their gender is identifiable,” in which case the acceptance rate was below men’s.

Finally, the researchers found that familiarity plays a major role in showing the bias. When contributions from “insiders” who were known and trusted within a project were analyzed, gender differences disappeared. It was when analyzing “outsider” pull requests from individuals not well-known to the project that the disparity between the genders became truly apparent.

“While our big data study does not definitely prove that differences between gendered interactions are caused by bias among individuals, the trends observed in this paper are troubling,” the paper concludes.

The researchers suggest that women contributing code to GitHub may be more competent than their male counterparts due to higher attrition rates for women in the lower levels of STEM careers, which result in higher levels of average training and experience.

A pre-print of the research is available online. It has not yet been peer-reviewed, but it offers a deep look at their methodology and appears to reveal a serious trend that should be examined further.

• View article: https://peerj.com/preprints/1733.pdf?

R&D 100 AWARD ENTRIES NOW OPEN:

Establish your company as a technology leader! For more than 50 years, the R&D 100 Awards have showcased new products of technological significance. You can join this exclusive community! Learn more.