“I'm just always amazed when I think back to the kind of toy models of superposition is only like 3 years, right, ago now. And we have come so far.”

ConceptResearch

“where the geometry components come in is that the features aren't like, sort of like naive, maybe like SAE build take on things is like, you could imagine that the model is encoding like a bunch of totally orthogonal concepts to each other”

ConceptResearch

“I'm guessing it's like most, like the, the different days of the week, I would guess have a very high inner product in the sense, which would to me suggest that they all, the model kind of knows like this is a day of the week sort of thing.”

ConceptResearch

“The models have now understood that they have better capability. It's almost like a human being, right? Once you realize that you have acquired a new capability, you become more confident. And same thing with these models. They tend to have become more confident with disagreei...”

ConceptResearch

“I think the idea that the models can improve their own training if the leading scientists working on this believe it, and it's an extension of what we are already seeing in code and math, of course you should believe it, right?”

ConceptResearch

“it obviously also reflects a major weakness in the models where they have this knowledge cutoff and they don't really know What's going on?”

ConceptResearch

“The new paper is on some pretty small open models, so it needs replication.”

ConceptResearch

“if you create GPT-3 or even GPT-4 or even GPT-5 via a different method, it's not going to be competitive”

ConceptResearch

“This is not about language alone. Yes, you can recognize popcorn. But everything else is about understanding how the world operates. And if it had to go over obstacles or found logic in how things operate, then world models matter.”

ConceptResearch

“There's so many of them. But, you know, that, you know, potentially using some of that information for training, I imagine systems like this, you know, on different environments and stuff. So, I mean, it should— the video showed the, the, the humanoid had a Ziploc and was dext...”

ConceptResearch

“The first time anyone was able to generate reasonable-looking proteins was the advent of diffusion models. So it was pretty crazy. There were a bunch of generative modeling approaches that would kind of work if you had a bunch of data. So people got these working for images. T...”

ConceptResearch

“the whole research process and identifying these types of scaling directions becomes a lot simpler”

ConceptResearch

“the whole research process and identifying these types of scaling directions becomes a lot simpler”

ConceptResearch