Machine Learning Street Talk
How Researchers Test AI for Hidden Goals — Apollo Research
“It's possible that the model might get really good results in its training distribution, but it might actually be learning the hospital where the test was taken, not the actual test as well. So it doesn't generalize. But why is this uniquely bad with language models and reinfo...”
Open episode and seek to 07:20ConceptResearch
Report this moment“I guess the interesting thing for me is that when we think of reinforcement learning algorithms like AlphaGo Zero, it makes sense that they are reward-seeking because there is this structured inference process. this is a language model. It's doing greedy sampling of tokens. So...”
Open episode and seek to 13:41ConceptResearch
Report this moment“I think when you're trying to get a high reward, it's really useful to think about how are they going to score me? How are they going to be evaluating me? How are they checking my answers? And then modifying your behavior to optimize for those. And things that are really usefu...”
Open episode and seek to 15:56ConceptResearch
Report this moment