The real concern is less about desktop users, but those who are on mobile devices.
A person using a desktop probably has a fast connection.
Those people using a mobile phone, on the other hand, will more than likely have a slower connection.
They will hit the back button if your page doesn’t load quickly enough.
If that happens, then Google will see it and think your site is of low quality.
You may have great content, but that’s not the way Google will see it.
They know if people are quickly leaving your site and you want to do anything in your power to make sure users stay as long as possible.