For me the biggest problem was never memorizing the suffixes but determining whether the verb belongs to group 1 or 2. While you can tell immediately that some verbs belong to group 1, the others (e.g. kaeru mentioned in the post) are not so easy (as far as I know there's no algorithm for that and you just need to memorize with every verb to which group it belongs).