Because either you have:

1. An E2E system where the provider has de facto access to the encrypted data, or

2. You shift key management to the users and let them risk data loss.

Either way:

a. The provider can release an app version at any time that accesses the data on the client side, and

b. Most of your users cannot differentiate between E2EE and SSL/TLS, nor are they interested in doing so, nor they care about it.