Apologies for hijacking an advice thread, but I'm another dad who's had to deal with this incompatibility nightmare, and I really really want to know what the hell happened within Microsoft & Mojang to end up with this mess.
No, seriously, I'm fascinated by the story. It's been over ten years. I'm happy to assume that most of the people involved are smart and good with decent intentions. I also bet there were many kinds of commercial and organisational pressures, especially because it's (a) Microsoft and (b) one of the biggest gaming acquisitions ever.
Does anyone here have any insider knowledge they could share?
As a couple of others have mentioned, the original split between the original Java Edition and the now Bedrock Edition happened in 2011. This is several years prior to the 2014 acquisition of Mojang by Microsoft. So while Microsoft may have had motivations in Mojang promoting or expanding Bedrock adoption, that train had already left the station by the time Microsoft entered the picture.
Bedrock started as "Pocket Edition". Probably the best place to get the story is at: https://minecraft.wiki/w/Pocket_Edition
I like Minecraft quite a lot and am not concerned with interoperability between Java and Bedrock... but will only play the Java Edition because of game-play differences between the two; differences in which Java Edition comes out as being materially better than Bedrock... at least from my perspective.
No insider knowledge but as far as I know, bedrock is written C++ not Java. They did that for porting to other devices and consoles, as well as preformance. It also allows them to sell skins, server hosting, more micro transactions since it's more locked down. So I think it's just a thing that happened, and it has been going on for a while and if they did anything to either the Java edition or the bedrock edition then it would make a lot of people really mad.
The developers are used to Java, and as someone who's worked with both Java and C++, Java is simply better for the kind of iterative development minecraft seems to do. From what I've seen (which admittedly isn't much more than the public facing history and the minecraft live/minecon events), there isn't really a strict roadmap and they seem to value messing with features a lot. Also, the game was only rewritten in C++ to support multiplatform ports, and bedrock isn't even the first time they did it (though, the other C++ versions, also known as legacy console, weren't handled in house). The existence of a windows version is a strange curiosity stemming from the microsoft buyout and their ability to do so since that codebase is insanely portable.
Also, mojang (not microsoft) really appreciates modders, which java edition is strictly superior for
Consoles hate having a jit vm so java is a non-starter for them.
The real reason is you want to run your game on the latest greatest WiiStation(tm). if it is written in C/C++ you sit down with a copy of their development library and change your code to fit. If it is written in java you port the jvm to the device. And I don't really know how hard that is(having never ported the jvm) but most people sort of balk at that step.
Microsoft thought it was easier/better to rewrite minecraft in C++ than port the jvm to the xbox. There was a big enough customer base attached to the original java version they did not want to discard it(backlash would be too huge and messy).
Here is something you can consider: https://pumpkinmc.org/ it's Minecraft rewritten in Rust. It's very fast and you can make your own mods.
Bedrock has much better performance so it was used for consoles, but there were already hundreds of thousands of mods for Java edition, thus the split.
Minecraft started on Java, a massive mod and server community created around it. Consoles won't run JIT/JVM/Java, so the Bedrock C++ version was created. People don't want to give up the massive mod community, so OG pc players stay on Java, while console players and newer PC players go to bedrock (with some of those PC players jumping over to Java after servers or mods are discovered).
They almost have feature parity, but there are some small differences around the edges. There exists modded server options for enabling bedrock and java users to connect to the same server.