Can not upload large files (>2GB)

[AlexBeaudoin]

I am running the current version of OpenKM (OpenKM-5.1.9), but this was a problem since at least 5.1.8.

I've been trying futilely to increase the maximum file size from 25Megs to something over 2GB. Talking to our users, I would ideally like to be able to manage files in the 40-100Gig range.

I have set max.file.size to 1000000 (~100GB), but the behaviour is similar for any max.file.size > 2GB.

When I try to upload a large file (2882406644 bytes) (~2.8GB)

Code: Select all

15:30:12,307 ERROR [DirectDocumentModule] Uploaded file size: 2.0 GB (2147483647), Max file size: 1.7 GB (1778384896)
15:30:12,307 WARN  [FileUploadServlet] 2147483647
com.openkm.core.FileSizeExceededException: 2147483647

I've even tried to use the webdav interface - the file copies correctly, but then truncates to 2GB after the CHECKIN_DOCUMENT phase.

I read http://forum.openkm.com/viewtopic.php?f=4&t=5142 and restarting OpenKM does not seem to change anything. I have deployed it on a 64bit OS (Linux Mint 12 Lisa, x84_64).

Thanks,
Alex

[pavila]

Have you modified the configuration property from Administration > Config? OpenKM.cfg no longer is used to configure many of this configuration properties: only for Hibernate related config.

[AlexBeaudoin]

Yes I have.

[pavila]

The problem is the InputStream.available() method which returns an Integer, and "2882406644" is out of this range (see Java Primitive Data Types). So you should not upload files bigger than 2GB.

The only option I see is disabling the check, and this would be if "max.file.size" is 0.

I have created this ticket http://issues.openkm.com/view.php?id=2047, please try the nighbuild and set max.file.size to 0 in order to disable the file size check.

[AlexBeaudoin]

I downloaded the nightly build and set the max.file.size to 0.

I no longer get a "FileSizeExceededException" but if I download the file, I still get the truncated 2gigabyte version.

However, when I go directly to the file ($root$/repository/repository/datastore/__/__/__/sha1sum) the file is there in it's complete form.

Alex

[pavila]

Yes, there is a problem with the current way of calculating the file size in OpenKM for files bigger than 2GB. I have making some improvement in OpenKM 6.0 (actually not released) to be able to handle files bigger than 2GB. I expect to release a beta next week.

I have also noticed a upload file size limitation with several browsers. For example, Frefox and IE can't upload files bigger than 2GB but Chrome can. Which browser do you use?

[AlexBeaudoin]

Been working under both Chrome, and Firefox.

I was uploading the file using Chrome - and the complete file exists in the repository after changing the max.file.size to 0 (which is an improvement). I'm pretty sure the download issue is not Chrome-dependant.

Good to hear that the problem is getting worked on!

Alex

[AlexBeaudoin]

Hmm.. looks like you are right. Using Firefox, I can't upload a >2GB document at all.

Not much you can do - I'll be sure to tell my users to use Chrome or the WebDAV interface when uploading large files.

[pavila]

I have made some modifications to handle uploads bigged than 2GB, but only works with Chrome. These modifications are too many to be included in OpenKM 5.1.10 so it will be included in OpenKM 6.0. These modification ensure you keep the right document size in OpenKM, so the download also works without corrupting the file.

By default, the Java Servlet spec does not allow uploads bigger than 2GB because store the size in a integer. So, I'm not sure if webdav would work as expected. If you try with WebDAV, please share your experiences with us.