alt Hacker News> New Causal Encoder–Decoder architecture: just 8B active parameters for input, 16B for output.
Oh interesting, I can assume what the benefits is for including the Encoder, but whats the downside? I’m thinking GPT (which is decoder only) ruled out Encoder for a reason?
Enc-decs are usually harder to train at frontier scale. Not 100% sure what DeepSeek has done differently here initial read seems to be something related to layer reuse but I just skimmed things so far.