They are both somewhat important. The compute speed impacts the tokens per second, and memory impacts the size of the model/capability.

IMO memory size is the most important since you can always wait longer for it to compute while a blazing fast but dumb model is much less useful.