Change Description Difference Analysis between Human and Code Differencing Techniques

Moojun Kim, Beomcheol Kim, Jindae Kim

http://doi.org/10.5626/JOK.2023.50.2.150

This study investigated the difference between descriptions of code changes made by source code differencing tools and humans. We applied two popular source code differencing techniques to collected code changes. We found that these tools often generated different descriptions for the same changes, and only 3% of the changes have the same descriptions from both tools. On the other hand, human participants agree on change descriptions for 50% of the given changes. Furthermore, many of the different descriptions were caused by simple mistakes. If we ignore differences caused by these mistakes, human participants described 71% of the changes similarly. We also compared change type and entity type of edit scripts generated by human and the source code differencing techniques for the same changes. We found that the techniques generated the same description as humans for only 8.20～35.65% of the changes, which indicates that these techniques require significant improvement to provide descriptions similar to human’s.