The obsession with triangles comes from Microsoft, who around 1995 chose to support only triangles in DirectX.
As a consequence of that, the GPU vendors designed then GPUs supporting only triangles in hardware, abandoning earlier attempts to also support other figures.
That has been inherited until today, so any program that wants to use not only the general-purpose computing features of a GPU, but also the dedicated graphics hardware from it, must convert any other kinds of graphics primitives into triangles, before rendering on the GPU.
I don't think that is quite right. I had a book about computer graphics using OpenGL from before then, and I remember it saying (something like) that various polygons were supported at the API layer, but the GL implementation would decompose them into triangles at some point, and this could have surprising results for texture mapping, so it was typically a good idea to triangulate shapes explicitly.
One major technical reason that triangles are nice is that every triangle determines a plane, and rendering points onto a bitmap is easier if you know the surface is planar.