> I'd say GC is always the fastest to free objects within the main code path. Literally zero instructions

True, but reference counting or free need not be far behind. They can append the pointer being freed to a per-thread list (⇒ no locking needed) that a separate thread that does the actual freeing periodically claims and then iterates over to actually free the objects.

Disadvantage is that memory usage goes up a bit because the actual freeing is delayed, but that (likely) is less so than with a garbage collector.