> Sharing matrices substantially reduces the weights that must be stored and moved while keeping the expert count unchanged. Each group of 64 experts shares two large matrices, while each expert retains a small set of parameters that captures its differences.

145B parameters with only peak 29GB memory usage - really impressive, truly something you can actually run on a decent spec MBP