Both Apple and NeXT had machines prototyped around it, but it was initially very expensive I believe, and I think Apple was easily convinced to go with PowerPC ... and rather than evolve it and push it further Motorola dropped it in favour of going in on PowerPC.
The sad thing is Intel showed there was still life left in CISC, and Motorola themselves ended up circling back on 68k in the form of ColdFire which proved you could do for 68k what Intel did w/ the Pentium. But by then all their 68k customers had moved on from the 68k ISA.
68k was much harder to optimize than x86, being way more CISC-y
68k like VAX was seen as dead avenue not only compared to RISC