Skip to main content

Remember seeing something about GPT-4 doing well on standardized tests? It turns out it may have memorized the answers.
aisnakeoil.substack.com/p/gpt-…
#gpt4 #AIHype #ThisIsWhyWeDontTestOnTheTrainingData
To benchmark GPT-4's coding ability, OpenAl evaluated it on problems from Codeforces, a website that hosts coding competitions. Surprisingly, Horace He pointed out that GPT-4 solved 10/10 pre-2021 problems and 0/10 recent problems in the easy category. The training data cutoff for GPT-4 is September 2021. This strongly suggests that the model is able to memorize solutions from its training set — or at least partly memorize them, enough that it can fill in what it can't recall.

GPT-4 and professional benchmarks: the wrong answer to the wrong question

OpenAI may have tested on the training data. Besides, human benchmarks are meaningless for bots.
Arvind Narayanan (AI Snake Oil)
#AIhype #GPT4 #thisiswhywedonttestonthetrainingdata
⇧