Search | fedi.ml

Search

Items tagged with: thisiswhywedonttestonthetrainingdata

Link to source

Janelle Shane

2 years ago

Link to source

Janelle Shane
2 years ago

Remember seeing something about GPT-4 doing well on standardized tests? It turns out it may have memorized the answers.
aisnakeoil.substack.com/p/gpt-…
#gpt4 #AIHype #ThisIsWhyWeDontTestOnTheTrainingData

To benchmark GPT-4's coding ability, OpenAl evaluated it on problems from Codeforces, a website that hosts coding competitions. Surprisingly, Horace He pointed out that GPT-4 solved 10/10 pre-2021 problems and 0/10 recent problems in the easy category. The training data cutoff for GPT-4 is September 2021. This strongly suggests that the model is able to memorize solutions from its training set — or at least partly memorize them, enough that it can fill in what it can't recall.

GPT-4 and professional benchmarks: the wrong answer to the wrong question

OpenAI may have tested on the training data. Besides, human benchmarks are meaningless for bots.

^{Arvind Narayanan (AI Snake Oil)}

#AIhype #GPT4 #thisiswhywedonttestonthetrainingdata

Please wait

View in context

⇧