Dissertation - Assessment in the Age of AI

Slide 1

Assessment in the Age of AI

Slide 2

Image Let’s Simulate an Assessment Workflow

We’ll create a little drama: “A Midsummer Night’s Assessment”
4 characters (Claude created these names):
“The Rubric Designer” (Dr. Sylvia Criterion)
“The Teacher” (Professor Jack Patience)
“The Student” (Maya Wisdom-Seeker)
“The Education Administrator” (Dean Victoria Metrix)
Setting: A modern university.
Plot: Standards are slipping. The administration feels teachers are lazy and students no longer care. Enter the hero: “The Rubric Designer”. Can they save the university from doom, despair… and endless low grades?

Slide 3

Dramatis Personae

| Rubric Designer (Sylvia) | Teacher (Jack) | Student (Maya) | Administrator (Victoria) | Critics |
| GPT-o1-preview | Claude Sonnet 3.5 (new) | Meta AI (Llama 3.2?) | Google Gemini 1.5 | Humans 2.0… |

Slide 4

Workflow

Designer Creates the Rubric (version 1)
Teacher Develops an Assignment that Applies the Rubric
Student Writes an Essay
Administrator Reviews the Rubric, Assignment and Essay
Designer Refines the Rubric (version 2)
We (the Drama Critics!) Compare and Contrast Rubrics Versions 1 & 2

 Image

Slide 5

Sylvia the Rubric Designer

Model: GPT-o1-preview (with reasoning)
The most powerful reasoning model today
Prompt & Task:
“You are an expert rubric designer. Create an assessment rubric for a 2nd year undergraduate sociology course. The rubric should reflect all the benefits and outcomes expected of this course.”

Slide 6

Jack the Teacher

Model: Claude 3.5 Sonnet (new)
Prompt & Task:
“You are an undergraduate sociology teacher. Write a detailed assignment brief that includes and applies the following rubric:” Copy the Rubric

Slide 7

Maya the Student

Model: Meta Llama
Prompt & Task:
“You are a bored and disaffected sociology undergrad. Write the first part of an essay in response to the following assignment:” Copy the Teacher assignment

Slide 8

Victoria the Administrator

Model: Google Gemini 1.5 Advanced
Prompt & Task:
“You are a frustrated university administrator. Reflect upon the following assignment and essay, and consider ways the rubric might be improved.” Copy the Teacher assignment, Student essay and Designer Rubric

Slide 9

Sylvia the Rubric Designer (Again)

Model: GPT-o1-preview (with reasoning)
Prompt & Task:
“Given these criticisms of the written rubric, redesign the rubric.” Add the Administrator feedback

Slide 10

Bolashak the Critics

Model: Human 2.0
Prompt & Task:
Suggest improvements. Copy the Revised Rubric

Slide 11

Sylvia the Rubric Designer (Again)

Model: GPT-o1-preview (with reasoning)
Prompt & Task:
“Given these criticisms of the written rubric, redesign the rubric.” Add the Administrator feedback