logoalt Hacker News

rhdunn • today at 8:28 PM • 0 replies • view on HN

Do we know that that is the case?

We know the architectures of the open weight models as they include architecture diagrams, support in projects like llama.cpp, etc.

Companies like Anthropic and OpenAI don't release information about their models (aside from general things like context window size, whether they support reasoning, etc.). A lot of what we know is speculation. From press releases Anthropic have stated that they are getting Claude to assist with designing new iterations of the models, but no specific details.

The open weight models and other arXiv papers/research are iterating on and experimenting with the base/core transformer architecture:

- RoPE/NoPE/etc. for context window scaling;

- dense vs mixture of experts models;

- Mamba-based State Space Models (falcon models, etc.);

- reasoning, combining transformers with a residual stream, Attention Residuals (Kimi), etc.

- activation functions: ReLU; SwiGLU; etc.

- etc.