“Frontier Labs buy their reinforcement learning environments from a cottage industry of small vendors. Almost nobody audits them. This week, someone who worked inside one spoke up.”

ConceptResearch

“Prakash argued this is an old supplier quality problem. Quarantine the new vendor. Sample grade. I separated the defect rate from a second variable, how far the labs are scaling RL on top of that signal.”

ConceptResearch

“Obviously, we've seen examples recently of how when the RL signal is not particularly clean, we can get all kinds of crazy downstream behaviours. So I've got a few questions on this, but I guess the first one is simply, How confident are you in the reward signal that you are a...”

ConceptResearch

“can actually help you do better, build better diffusion models”

ConceptResearch

“It's more fundamental than that. In fact, the book goes into variational autoencoders. It goes into MCMC methods, Markov chain Monte Carlo methods, which are used to sample uh, sort of sample from particular distributions. It goes into free energy estimation. It goes— there's ...”

ConceptResearch

“it's not only accurate, it's almost as close to what the traditional weather models can do accurately, but also tens of thousands of times faster.”

ConceptResearch

“that's what neural operators enable because they model inputs and outputs as continuous functions that can be infinitely resolved”

ConceptResearch

“that's how we can ensure that these physics-informed neural operators can work at higher fidelity and higher resolution than even the training data that was available.”

ConceptResearch

“one of the big benefits though is that when you can get the model explicit examples of the model's reasoning about these things, I think it's just, it's often very easy to lose track of how much worse off we would be if we did not have this.”

ConceptResearch

“I'm sure that the upper bound for how long, like whatever those unreleased models are on like a well-structured task is probably just incredibly high”

ConceptResearch

“You'll also see in the prisoner's dilemma case, there's some where it's like, wait, but if we defect, then another ChatGPT playing will get punished for the defection. Do we care? No. We get reward and then we vanish. Great. You're like, wait, did you just do the full reward a...”

ConceptResearch

“I have been enamored with this story for a minute now because, again, to your point, this is where companies go to test their models out. Usually there's enough breadcrumbs, so there's a delicious little trail and you can find your way back to Meta or OpenAI or Anthropic, what...”

ConceptResearch

“Right. And one of the other tests, Gavin, one of the other tests is to ask about Tiananmen Square or to ask about Taiwan. Yes. Yes. Because you can usually tell if a model is being hosted by a Chinese company or has been distilled from a Chinese model because in the thinking t...”

ConceptResearch

“This model sometimes answers very plainly about whether Taiwan is a country or not. Right. And other times it refuses to. It thinks Tiananmen Square is like a, like a Red Hot Chili Peppers demo album from the late '80s or something.”

ConceptResearch

“We can— if you're good at evals, if you're good at assessing quality, You can build that data by running various scenarios, collect a bunch of this data, and then you can train models to—”

ConceptResearch

“For us, somebody ships a better model, they have now unlocked 4 more use cases where we can be valuable.”

ConceptResearch