Trends over time

“Tell me more about, it's like running into a wall at their sprint and get smashed into pieces. Clearly they are not training them like that on an ongoing basis because they'd run out of physical robots to do the training on awfully quickly. So from that alone, I was like, okay, clearly they must be doing a lot of training in simulation. Tell me more about why these kinds of tasks are amenable to training in simulation and what kinds of tasks are still difficult to train in simulation?”

ConceptResearch

“The co— yeah. Contacts are basically, the physics is harder to model in simulation. So where the physics is easier to model, simulation gets in.”

ConceptResearch

“Now, these models are also very subject to kind of memorization in some sense.”

ConceptResearch

“Well, today we have a model coming out about every uh 2 weeks.”

ConceptResearch

“Opus 5.5 really does feel like we have— uh it feels like a next-level model”

ConceptResearch

“Yes, you can iterate and make these better. I wanna make sure I, that, that Soll, it can get there, but it just shows you that, that visually how different the level is of these 2 models. And I think that's something really important. And like, honestly, like I would be slightly worried if I was at OpenAI that this is the thing we have going against Opus 5.5.”

ConceptResearch

“benchmarks that I never trust anymore unless I can use the model”

ConceptResearch

“Not much more to say on it. It's interesting that there is headroom on price optimization. while keeping sort of frontier-ish capabilities. I do wonder if the labs haven't sort of optimized the models hard or as hard as they could be for cost and speed. And they might be adopting some of the tricks that the open source models have uh showcased to work.”

ConceptResearch

“This is their own build eval. That's like, we want the model to build a workflow. So it needs to create a correct compute graph. in code. And yeah, the one thing they showcase is like, oh, we built this eval to showcase what this model is good for.”

ConceptResearch

“He's former OpenAI. So, you know, should know, should know what he's doing here. And nominally it's a new architecture as well, not just a new training method. We don't really know more, more than that, right? It's a, it's a car race. What do you want? I mean, a car chase.”

ConceptResearch

“The nice thing about GPU Mode is that it is also a community in the sense that like a lot of the lectures are very easy enough for a beginner to follow and ask questions and things like that. And like the competitions are like, somewhat not secondary, but like you can participate in them to learn. I think with a lot of like ML Sys, ML Perf kind of benchmarks, like for the most part, like only serious labs and companies participate.”

ConceptResearch

“Any, any interesting takes for people that want to get into it? So I think one of the biggest news is GPT-5.6 wrote more efficient kernels so Terra and, uh, Luna could be 80% cheaper. And then we've seen other competitions where people are like setting records and they're like, we're doing some auto research loop. And these are people that don't have a background in any kernel writing, right?”

ConceptResearch

“There's basically this— I think people don't understand how m much benchmarks come from the same group at Princeton. Yeah.”

ConceptResearch

“collapsing it is the first step. And then the initial model needs to be good enough so you can continue to iterate”

ConceptResearch

“training a neural model end to end, you can, you can cover a lot of those failure modes”

ConceptResearch

“Very cool. And I wonder, does the model currently take in pure waveform data, or does it take in featurized data? I just remember I was doing audio stuff for my PhD, and back in those days, it would have been unimaginable to put in waveform data. And now it's simply a case of if the model is big enough and you train for long enough, you can.”

ConceptResearch