I think I have identified the problem.
The maximum file size appears to be a signed integer only capable of holding ~2GB, when you go past that it overflows to -2GB and goes up from there. So when I had 3GB set as the max, the max was really -1.1GB, the server took anything below 1.1GB fine, but would not accept larger file sizes.
This is on 5.0.4, I am upgrading later today to a newer version and plan to check this specifically at least.