5.1.10 Snapshot Unstable These Days

[kevinchang]

Hi jilort and pavila,

I kept upgrading snapshot daily build to get the latest fixes.
However, snapshot builds become unstable in my environment, users will encounter communication error when checkout document.
I myself even spotted lots of error messages in console. Please check the attachment.
Because my installation cannot keep working normally for even one day, I fallback to build:7460, but the error log still here. Really lots of error log, I need your help badly.

[kevinchang]

I finally fallback to release build 5.1.9, and it become stable now.
This afternoon is really bad for me, because the nightly build 5.1.10 crash twice this afternoon.
I would like to provide more information and I can repeat the problem any time.

[pavila]

I recommend to stop JBoss, and remove $JBOSS_HOME/server/default/tmp & $JBOSS_HOME/server/default/work when upgrading because sometimes JBoss has problems after several hot deploys.

[kevinchang]

Hi pavila

I ran clean.sh before each update, but the unstable problem still exist.

The release build 5.1.9 works stable these days.

[pavila]

I have seen a lot this error in you log file:

Code: Select all

java.net.SocketException: Too many open files

And I don't it is related to OpenKM-5.1.10. It would help if you can post the steps to reproduce this error.

[kevinchang]

Hi pavila,

I'm using Mysql + LDAP.
It's quite simple to repeat this problem in my environment. Upgrade to a 5.1.10 daily build, user start to login, then click any folder or file will cause the "too many open file" error log.

[pavila]

Can you reproduce this error in a OpenKM 5.1.10 clean installation? I can't see this error. There are many concurrent users? Many documents in the repository?

[kevinchang]

Hi pavila,

I will try a clean installation next Monday.
I notice the "too many open files" error even just after I restart openkm instance, only I am click a single file.
Yes, there are over 10 thousand files in my repository.
Clean installation means jboss environment with 5.1.0 war package, and the same configuarion,right?

[jllort]

I'm not sure if 5.1.10 has minimal change and you should remove all tree hibernate libraries under /server/default/lib this is the only difference that you could find, but I'm not totally sure about it on community version.

[kevinchang]

Hi jilort,

Where can I get a jboss package with 5.1.10?

[pavila]

Actually you can't. Download the JBoss + OpenKM 5.1.9 bundle and replace the WAR with the last night build from OpenKM 5.1.10

[Alexires]

April 29 and April 30 build are halting with error

Code: Select all

[WARNING] *** CHECKSUM FAILED - Checksum failed on download: local = 'b1a8b2056692b57d2ac4ddb23991ad798eb59e8b'; remote = '

I was hoping to resolve some problems by moving to the 5.1.10 build, but it sounds like it is rather unstable. What timeframe are you guys looking at for releasing it globally?

Also, I receive the "Too Many Files Open" error when I attempt to import a folder that has already been imported, if it has a lot of files in it (30,000).

[Alexires]

I think I might have worked out where that IO Too Many Open Files stems from. Apparently Linux is configured to limit the number of total open files in one session. Increasing this value may prevent IO errors (I haven't tried it yet, I'm not home). Check out this link: http://www.cyberciti.biz/faq/linux-incr ... pen-files/

[Alexires]

I've increased ulimit -Hn to 1000000, however, it still seems that OpenKM chokes on 1024 (the default). I'm currently looking at the run.sh file which might not be reading the ulimit properly, hence setting the java ulimit to 1024 (thus giving IO errors). I'll have a play and see if it works.

EDIT: SOLVED! Changed the value for FD_MAX in run.conf manually from "maximum" to a value just lower than the value given in ulimit -Hn (in ubuntu linux). It seems like OpenKM is using (at a max) 2050ish of the value given in ulimit -Hn (calculated by looking at htop and selecting "java -Dprogram...." then hitting the l key to show open descriptors. Scrolling to the bottom gives about 2050 (excluding mem descriptors)) so I'd imagine a value for FD_MAX of 10000 to be fine.

jllort or pavila, perhaps have a look at the run.sh script and see whether it is getting the correct value for ulimit from the system. I don't know a lot about coding, and what I read was fine, but once I changed it, I managed to import much much more of my 30,000 file directory via Import Repository than I could by leaving run.conf as it is...

[pavila]

Is quite odd you have this error in OpenKM 5.1.10 SNAPHSOT I don't understand why this OpenKM release is taking too mucho file descriptors. Which Ubuntu version are you using? Is 32 or 64 bits?