Normally, this should not be happening if your application is under 100mb and you rely on a good network connection.
In order to assist you, I would kindly ask you to gather a Fiddler logs about the failing build and send it over in a private support ticket. To get the log, you can follow the next steps:
1. Download Fiddler from here and install it.
2. Run Fiddler and go to Tools -> Fiddler Options.
3. On the HTTPS page, verify that Capture HTTPS Connects is enabled.
4. Verify that Decrypt HTTPS traffic is enabled with the ...from all processes option and minimize Fiddler to tray.
5. Try to build your project in the preferred AppBuilder client and when it fails, continue to the next step.
6. In Fiddler, go to File -> Save -> All Sessions, save the archive to disk. This will produce a SAZ file.
7. Zip the .saz file and attach it in the ticket
I hope this is doable for you and I am looking forward to your reply.