It is Higher Ed Week here at this substack, so I had better comment on the NYMag’s “AI cheating” article. If you haven’t read it then let me summarise it for you: students are using AI to do all their assessment and there is nothing anyone can do about it?
Well, they can do some things, including:
Actively ignore it
Fret
Remind everyone that students have always tried to cheat
Or related, “I was able to pick up AI written stuff last semester”
Claim “the students are only harming themselves”
Let’s just raise the bar of what it means to get an A.
Ask the “administration” to set new guidelines
Argue “well if AI can write the assignment, is it really an assignment?”
Propose writing assignments that require your own special sauce to answer
Calculate the number of years before your retirement.
Pronounce, “if everyone is doing it, maybe we shouldn’t call it ‘cheating’?”
Extoll the new reduced anxiety and stress numbers amongst students
Pontificate: “we should take a multifaceted approach and delve into the context students are facing.”
(That last one could have been written by AI and probably was. All the others, and I mean all of them, appeared in the article.1) If you have an image of a coyote over a cliff before realising gravity is a thing, then I think you have the right image of a faculty meeting on this subject.
That said, I recently wondered if we could “pollute the prompt” by inserting things into the assignment task. Apparently, I wasn’t the only one.
Some professors have resorted to deploying so-called Trojan horses, sticking strange phrases, in small white text, in between the paragraphs of an essay prompt. (The idea is that this would theoretically prompt ChatGPT to insert a non sequitur into the essay.) Students at Santa Clara recently found the word broccoli hidden in a professor’s assignment. Last fall, a professor at the University of Oklahoma sneaked the phrases “mention Finland” and “mention Dua Lipa” in his. A student discovered his trap and warned her classmates about it on TikTok. “It does work sometimes,” said Jollimore, the Cal State Chico professor. “I’ve used ‘How would Aristotle answer this?’ when we hadn’t read Aristotle. But I’ve also used absurd ones and they didn’t notice that there was this crazy thing in their paper, meaning these are people who not only didn’t write the paper but also didn’t read their own paper before submitting it.”
This might work for a time, but it isn’t any kind of long-term solution.
Interestingly, however, I wonder if the race is the point.
There are, of course, plenty of simple ways to fool both professors and detectors. After using AI to produce an essay, students can always rewrite it in their own voice or add typos. Or they can ask AI to do that for them: One student on TikTok said her preferred prompt is “Write it as a college freshman who is a li’l dumb.” Students can also launder AI-generated paragraphs through other AIs, some of which advertise the “authenticity” of their outputs or allow students to upload their past essays to train the AI in their voice. “They’re really good at manipulating the systems. You put a prompt in ChatGPT, then put the output into another AI system, then put it into another AI system. At that point, if you put it into an AI-detection system, it decreases the percentage of AI used every time,” said Eric, a sophomore at Stanford.
Aside from anything else, if this student takes these skills to the workplace, they are going to translate into a more effective job. Put simply, all this stuff we call cheating may be training students to work with AI in ways a University has no idea how to do.
What do grades mean?
There is no professor sitting down to mark dozens of exams who hasn’t thought about that question at a philosophical level. (And then mostly conclude that they mean whatever is enough to minimise student complaints about grading.) This is, for the most part, because they don’t like grading and realise that they only have to do it because they don’t trust students to care enough about learning, and they are performing an HR recruiting function for their future employers. (Yes, some faculty may like learning about how much the students have learned, but they are as numerous as the set of feel-good movies about such teachers.) This is just another way of saying that if they could use AI to cheat on grading, they would do so in a heartbeat.
Anyhow, on this score, this particular quote pushed a number of buttons:
“College is just how well I can use ChatGPT at this point,” a student in Utah recently captioned a video of herself copy-and-pasting a chapter from her Genocide and Mass Atrocity textbook into ChatGPT.
It is both true and will send shivers down the spines of textbook publishing execs; although, so long as the student buys the thing they are pasting from, is that an issue?
But the point is that grades are a ChatGPT test right now.
More to the point, grades are a Turing Test: how well can a student with AI imitate a student without AI?
The question is: is that a problem?
From a future workplace standpoint, yes and no. It is a problem if would be employers are looking at grades and using them to screen job applicants. But that isn’t happening so much. There are numerous interviews and other screening devices that employers now use because they lost faith in the fidelity of grades some years ago. And for the rest they are using … ahem … AI to sort through applications.
And no, it isn’t a problem because, if many are to be believed, it will be essential to be able to use AI in the workplace, and so if that is what Universities are certifying now, maybe that is well and good.
By 2027, we are going to find out. That is a year after the ChatGPT 4 generation will graduate. By then, employers will know whether they are basically selecting people at random rather than on whether they will be good at the job.
In the meantime, what are we doing here? I have to admit that I am partial to the idea of giving up on assessment completely and honestly letting employers work it all out. At one level, it is simply dishonest at the moment to tell them we are certifying someone for Econ 101 if we aren’t. At a minimum, we should report on how much of a grade was certifying to be AI-free via in-class exams or some other supervised assessment.
But long-term, I think we have to face facts. Assessment and certification were always both very expensive and often quite imperfect. Universities have struggled with it, and my sense is that it has only been getting worse, not better. Alice Evans has called for an end to any assessment that is unsupervised (such as take-home exams). I’m quite sensitive to that, as I was not sure that in-class assessment was producing an accurate evaluation for some students, and going back to that seems wrong. I haven’t had any in-class assessment beyond presentations and participation for two decades now and frankly, I always felt it aligned more with the way people might have to do these things when eventually at work. In-person stuff is more artificial.
The problem, however, is cost. Supervision requires people. That makes it hard. It is even hard these days when people write on computers so there is discussion about making stuff written again. That is even costlier and distorts the signal further. Moreover, isn’t it just wrong to get someone to take a derivative without assistance on paper? It is literally a huge step back.
Of course, AI might come to the rescue here, too. It might make supervised assessment such as oral exams virtually costless — at least to the instructor.
In the end, I would love to start thinking about “just giving up” more seriously. Let’s replace the usual assessment with more Khan Academy-like self-tests that encourage mastery. And then let’s see the employers work out what they want following that. AI may be the push we need to change the entire apparatus for the better.
That said, one person seems to have conflated two of them: ““Cheating correlates with mental health, well-being, sleep exhaustion, anxiety, depression, belonging,” said Denise Pope, a senior lecturer at Stanford and one of the world’s leading student-engagement researchers.” That wasn’t what the students in the article were feeling.