{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,3]],"date-time":"2026-05-03T03:17:27Z","timestamp":1777778247431,"version":"3.51.4"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,7,22]],"date-time":"2023-07-22T00:00:00Z","timestamp":1689984000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Science Foundation","award":["1915196, and 1915198"],"award-info":[{"award-number":["1915196, and 1915198"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2023,9,30]]},"abstract":"<jats:p>\n            Assessing team software development projects is notoriously difficult and typically based on subjective metrics. To help make assessments more rigorous, we conducted an empirical study to explore relationships between subjective metrics based on peer and instructor assessments, and objective metrics based on GitHub and chat data. We studied 23 undergraduate software teams (\n            <jats:italic>n<\/jats:italic>\n            = 117 students) from two undergraduate computing courses at two North American research universities. We collected data on teams\u2019 (a) commits and issues from their GitHub code repositories, (b) chat messages from their Slack and Microsoft Teams channels, (c) peer evaluation ratings from the CATME peer evaluation system, and (d) individual assignment grades from the courses. We derived metrics from (a) and (b) to measure both individual team members\u2019\n            <jats:italic>contributions<\/jats:italic>\n            to the team, and the\n            <jats:italic>equality<\/jats:italic>\n            of team members\u2019 contributions. We then performed Pearson analyses to identify correlations among the metrics, peer evaluation ratings, and individual grades. We found significant positive correlations between team members\u2019 GitHub contributions, chat contributions, and peer evaluation ratings. In addition, the equality of teams\u2019 GitHub contributions was positively correlated with teams\u2019 average peer evaluation ratings and negatively correlated with the variance in those ratings. However, no such positive correlations were detected between the equality of teams\u2019 chat contributions and their peer evaluation ratings. Our study extends previous research results by providing evidence that (a) team members\u2019 chat contributions, like their GitHub contributions, are positively correlated with their peer evaluation ratings; (b) team members\u2019 chat contributions are positively correlated with their GitHub contributions; and (c) the equality of team\u2019 GitHub contributions is positively correlated with their peer evaluation ratings. These results lend further support to the idea that combining objective and subjective metrics can make the assessment of team software projects more comprehensive and rigorous.\n          <\/jats:p>","DOI":"10.1145\/3593592","type":"journal-article","created":{"date-parts":[[2023,5,2]],"date-time":"2023-05-02T12:37:47Z","timestamp":1683031067000},"page":"1-23","update-policy":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Combining GitHub, Chat, and Peer Evaluation Data to Assess Individual Contributions to Team Software Development Projects"],"prefix":"10.1145","volume":"23","author":[{"ORCID":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/orcid.org\/0000-0001-5128-905X","authenticated-orcid":false,"given":"Christopher","family":"Hundhausen","sequence":"first","affiliation":[{"name":"Oregon State University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/orcid.org\/0000-0003-3676-7775","authenticated-orcid":false,"given":"Phill","family":"Conrad","sequence":"additional","affiliation":[{"name":"University of California, Santa Barbara"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/orcid.org\/0000-0003-0620-500X","authenticated-orcid":false,"given":"Olusola","family":"Adesope","sequence":"additional","affiliation":[{"name":"Washington State University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/orcid.org\/0000-0003-3050-0391","authenticated-orcid":false,"given":"Ahsun","family":"Tariq","sequence":"additional","affiliation":[{"name":"Oregon State University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,7,22]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3440994"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tjem.2018.08.001"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2017.43"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/1404520.1404522"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/1352135.1352218"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/2732155"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366948"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/0149-7189(79)90048-X"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/1953163.1953312"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2019.00075"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1093\/oxfordjournals.oep.a041563"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/1595453.1595468"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.5555\/3199572.3199591"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPC.2017.2656626"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/CHASE.2012.6223016"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3381836"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.2307\/1924845"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/2538862.2538970"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/2325296.2325339"},{"key":"e_1_3_1_21_2","unstructured":"Stephanie Glen. 2022. Correlation Coefficient: Simple Definition Formula Easy Steps. Retrieved May 11 2023 from https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/www.statisticshowto.com\/probability-and-statistics\/correlation-coefficient-formula\/."},{"key":"e_1_3_1_22_2","first-page":"74","article-title":"Evidence for the pareto principle in open source software activity","author":"Goeminne Mathieu","year":"2011","unstructured":"Mathieu Goeminne and Tom Mens. 2011. Evidence for the pareto principle in open source software activity. In Proceedings of the 2011 Workshop on Software Quality and Maintainability (SQM\u201911).74\u201382.","journal-title":"Proceedings of the 2011 Workshop on Software Quality and Maintainability (SQM\u201911)."},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/CLEI52000.2020.00068"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/1985441.1985476"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3159450.3159584"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734359"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.2307\/1927875"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545945.3569790"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2022.2071543"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSEET.2008.24"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/FLOSS.2007.12"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.2307\/2230396"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387940.3392224"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.18260\/1-2--2355"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2810295"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/2818052.2869117"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/1062455.1062566"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.5465\/amle.2016.0163"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/1999747.1999823"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0165-1765(97)00101-8"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.2307\/1926398"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3017680.3017682"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2018.8658908"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICS51289.2020.00099"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/1953163.1953209"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.5465\/amle.2010.0177"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/CHASE.2013.6614747"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2018.8658983"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/2445196.2445351"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/WIIAT50758.2020.00027"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3145477"},{"key":"e_1_3_1_52_2","volume-title":"Applied Social Psychology: Understanding and Addressing Social and Practical Problems","author":"Schneider Frank W.","year":"2011","unstructured":"Frank W. Schneider, Jamie A. Gruman, and Larry M. Coutts. 2011. Applied Social Psychology: Understanding and Addressing Social and Practical Problems. SAGE, Thousand Oaks, CA."},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2010.5609637"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3229882"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/2538862.2538932"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.4018\/jossp.2009010105"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110717"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/2676723.2677223"},{"key":"e_1_3_1_59_2","unstructured":"Purdue University. 2023. CATME Five Teamwork Dimensions. Retrieved May 11 2023 from https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/info.catme.org\/features\/catme-five-dimensions\/."},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2017.11.021"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2009.5306322"},{"key":"e_1_3_1_62_2","first-page":"171","volume-title":"Proceedings of the 10th Conference on Australasian Computing Education\u2014Volume 78 (ACE\u201908)","author":"Konsky Brian R. von","year":"2008","unstructured":"Brian R. von Konsky and Jim Ivins. 2008. Assessing the capability and maturity of capstone software engineering projects. In Proceedings of the 10th Conference on Australasian Computing Education\u2014Volume 78 (ACE\u201908). 171\u2013180."},{"key":"e_1_3_1_63_2","first-page":"367","volume-title":"Proceedings of the 12th Working Conference on Mining Software Repositories (MSR\u201915)","author":"Yu Yue","year":"2015","unstructured":"Yue Yu, Huaimin Wang, Vladimir Filkov, Premkumar Devanbu, and Bogdan Vasilescu. 2015. GitHub. In Proceedings of the 12th Working Conference on Mining Software Repositories (MSR\u201915). IEEE, Los Alamitos, CA, 367\u2013371."}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/dl.acm.org\/doi\/10.1145\/3593592","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/dl.acm.org\/doi\/pdf\/10.1145\/3593592","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:47:47Z","timestamp":1750178867000},"score":1,"resource":{"primary":{"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/dl.acm.org\/doi\/10.1145\/3593592"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,22]]},"references-count":62,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,9,30]]}},"alternative-id":["10.1145\/3593592"],"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/doi.org\/10.1145\/3593592","relation":{},"ISSN":["1946-6226"],"issn-type":[{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,22]]},"assertion":[{"value":"2022-10-31","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-03","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}