I don't think IBM's influence was that large. IBM was a world on its own. The rest had hardware from many manufacturers. If there's a single point in time where the byte got its definitive shape, I'd say it was Intel's 4004, or the 8008.
The calculator chip 4004 never had any relevance for computers, and it did not have "bytes".
Intel 8008 did not have anything original in its architecture, it was just a monolithic PMOS re-implementation of the CPU of the embedded computer designed for the serial terminal Datapoint 2200, which had been designed with TTL integrated circuits. All the decisions about sizes, e.g. 8-bit data and 14-bit addresses, had been done by Datapoint in 1970, not by Intel. Datapoint had chosen 8-bit bytes in order to support the recently standardized ASCII 7-bit character set (and the 8-bit IBM EBCDIC character set, if necessary), i.e. the character sets used by the computers to which such a serial terminal could be connected.
At the time when the first microprocessors were designed, during the first half of the seventies, the most important architectural influence on any new computer designs were the DEC PDP-11 minicomputers.
DEC PDP-11 used 8-bit bytes, which was a significant change from the previous DEC computers, most of which used word sizes that were a multiple of 6, like 12-bit, 18-bit or 36-bit.
DEC PDP-11 had transitioned to 8-bit bytes (in 1970) mainly due to the influence of IBM System/360. The standardization of the 7-bit ASCII code for characters, which could no longer fit inside 6-bit bytes, has contributed to this decision, but the standardization of ASCII was itself possible only because many computer vendors had already transitioned or decided to transition to 8-bit bytes, so they could store the new ASCII characters in their bytes.
After 1967, when ASCII was standardized in a form close to the present form, after which it was also taken into international standards by ISO and CCITT, all new computer instruction-set architectures were designed with 8-bit bytes.
S/360 is what pushed 8 bit bytes to worldwide prominence.
IBM was ridiculously huge, both by actual IBM hardware, but also by clones, and manufacture of S/360-compatible systems. Even many wildly different computers often had third party interfaces to hook up S/360 channel devices, or controllers for the actual devices (it was common for channel devices to be linked like this: S/360 -> channel processor -> controller -> device-specific connections -> actual device).
Even the fact that we format code usually to 72 characters is related to 80-character standard S/360 punched card, where the remaining 8 characters were used for sorting/comment code, and why professional terminals (as opposed to things like 40 column mode on home computers) had 80 column displays.
Also, had ASCII been ready earlier, S/360 would have used ASCII as default encoding - IIRC S/360 team decided they can't wait for ASCII to be finalized and they needed to start design and making of various devices that would have to be encoding-aware, thus EBCDIC was born.