King's College London

Generative AI (GenAI) represents a profound transformation in the dynamics of teaching and learning, setting new expectations for students and educators alike and ushering in both challenges and opportunities. This project aims to contribute to a deeper understanding of the latter within the realm of assessment and feedback. Grounded in the belief that students and educators should collaboratively shape the new teaching and learning paradigms demanded by the GenAI revolution, our project emphasises the principle of co-creation at every stage, from the project’s inception to its implementation, and ultimately, to the delivery of tangible outcomes.

We focus on the realm of essay writing and the subsequent evaluation of these essays in the context of Political Science - where the project leads’ academic expertise lies - and with an overarching framework inspired by the PAIR framework developed by KCL Professor Oguz Acar.  

The project seeks to achieve the following objectives: 

• Explore the integration of GenAI into the essay marking process and, specifically, investigate the application of the PAIR framework within the context of essay writing, focusing on the Political Science discipline.
• Investigate the incorporation of GenAI into the marking process and assess students’ attitudes towards GenAI-generated feedback in comparison to feedback generated by humans.
• Enhance comprehension of inclusivity challenges and enablers related to GenAI applications in assessment and feedback.
• Gain insight into the impact of GenAI on student-teacher trust.

We will achieve the above objectives through the implementation of the following activities:  

• Phase 1: Two full-day workshops will be conducted in March 2024, involving 20 students from King’s SSPP faculty, with 10 participants in each session. During these workshops, students will be composing a short Political Science essay using the PAIR framework. The essay-writing process will employ a cognitive group interview approach to monitor students' engagement with, and perceptions of, the PAIR-informed assignment brief, as well as of their use of GenAI. The cognitive interviews method uses targeted questions to evaluate participants' comprehension of a particular task while this is executed, thus capturing authentic insight. After the session, the Principal Investigator will use these insights to refine the PAIR-informed essay brief for integration into the module she convenes in the 2024/25 academic year.
• Phase 2: Following the essay-writing workshop, the resulting essays will be assessed by both human markers and Generative AI. The feedback from both sources will be provided to students without disclosing whether it was written by humans or the machine.
• Pre-Phase 1 and Post-Phase 2: In addition to the above, all 20 students will complete a preliminary survey to capture their perceptions regarding GenAI in assessment, with a particular focus on trust and inclusivity. Furthermore, a post-workshop survey will be administered to assess changes in perceptions related to GenAI in assessment and to gauge their views on the feedback provided by both GenAI and human assessors on their essay outputs.

The project team will disseminate the research findings through a research paper and at Teaching and Learning conferences.

The project team also includes:

• Camille Bacha De Carvalho (2^nd year Politics student)
• Duru Karadeniz (2^nd year PPE student)
• Ziti Lei (3^rd year IR student)