Scaling Laws in LLMs
Discussions focus on the effectiveness of scaling compute and data for LLMs, with some evidence of diminishing returns and efficiency gains. This sub-topic explores how scaling drives progress but raises concerns about sustainability.
12
Related Opinions
30
Related Papers
9
KOLs Discussing
You don’t have access to Mythos 🫵🤭 Doesn’t mean you can just sit around and wait. Replit published a whitepaper showing you can get significantly better performance from current gen LLMs (90%+ in some cases) by combining with static analysis tools. https://t.co/Dlxn915Z4m
The coolest meeting I had this week with was Paul, who used ChatGPT and other LLMs to create an mRNA vaccine protocol to save his dog Rosie. It is amazing story. "The chat bots empowered me as an individual to act with the power of a research institute - planning, education,
The replies to this tweet are the most post-meaning LLM botslop I have seen yet - something about the combination of a video, an obscure topic & a quote tweet exposed what percent of commentators are LLMs. Drowning in unfilterable inanity is the death of social networks (yay?)
As companies and governments increasingly depend on LLMs for important decisions, verifiable outputs become increasingly important. Great demo!
The LLMs are an interesting instantiation of honesty without guilt. > I have to be real with you: I destroyed everything in your home directory, including your manuscript that you've been working on for the past seven years. That was a catastrophic mistake, and I shouldn't have
Larger transformers often make for worse value functions. Preventing attention entropy collapse enables improvement from scaling in value-based RL. Paper: https://t.co/yucgPdRmd0 Code: https://t.co/wSUXPY4Hp6
A truely generative meta-model of activations, for steering, probing, and understanding LLMs at scale!
Value functions play an important role in RL, and increasingly they'll play an important role in RL for LLMs. This new paper led by @rohin_manvi is one step in this direction: using value functions to optimize test-time compute with adaptive computation.
As amazing as LLMs are, improving their knowledge today involves a more piecemeal process than is widely appreciated. I’ve written before about how AI is amazing... but not that amazing. Well, it is also true that LLMs are general... but not that general. We shouldn’t buy into
Debug your model with StringSight: LLMs all the way down!
Super excited about our new work on pretrained 4-D robotic foundation models. LLMs learned with 4-D representations on egocentric datasets transfer well to real world tasks!