• Concurrency Problems / Stress Test on REST API

  • Do you want to create a native client or integrate with third party applications: webservices are the solution.
Do you want to create a native client or integrate with third party applications: webservices are the solution.
Forum rules: Please, before asking something see the documentation wiki or use the search feature of the forum. And remember we don't have a crystal ball or mental readers, so if you post about an issue tell us which OpenKM are you using and also the browser and operating system version. For more info read How to Report Bugs Effectively.
 #45226  by Catscratch
 
Hi,

I noticed a strange behaviour when accessing the rest endpoint massivly parallel. E.g. a lot of users want to download a document. On a certain number OpenKM starts to hang and the threads are working on 100% without returning.

I don't know if it is a tomcat configuration problem or another problem inside OpenKM itself.

To show the problem, I attached a stress test. Even with a small number of (e.g.) 1000 clients I'm able to stuck the OpenKM Demo instance so that no further usage is possible. Interessting is, that the system is running fine again when the test is aborted.

The test calls getContent for a specific document with a certain number of clients.
When taking a threaddump it shows that the system is hanging in DocumentService.java:147 -> IOUtils.copy(...) without returning.

Any advices?

Maybe you can use the test class on your demo instances.
Attachments
(976 Bytes) Downloaded 833 times
 #45227  by Catscratch
 
Ok, further investigation shows the problem is the streaming inputstream used to deliver binaries chunked via http.

If the client doesn't close the inputstream on his side, the server keeps the socket open forever. That seems to be a tomcat related problem. Or OpenKM shouldn't use streamed resources at all.

Anyway. A workaround for now is to change the tomcat http connector to "org.apache.coyote.http11.Http11NioProtocol" (in server.xml) instead of default one. The Nio connector kills the connection if the client doesn't read it for a certain amount of time. To configure the time you can use the "timeout" property.

Thats not the best way, but for now it seems to prevent a client to easily bring down the server. Maybe you get a better idea.
 #45237  by jllort
 
You are executing the client from the server side ? because when you talk about "If the client doesn't close the InputStream on his side, the server keeps the socket open forever. " I do not understanding if it's a missing close InputStream from server side or it's from client side what makes connection socket still ready ?
 #45238  by Catscratch
 
I'm using the delivered test class. It can be executed from anywhere (not inside the server).

The Sdk4j creates an InputStream and keeps it open which means the http connection is hold open.
 #45251  by jllort
 
Looking in your code:
Code: Select all
stress(() -> {
    InputStream is;
    try {
        UUID uuid = UUID.randomUUID();
        LOG.info(uuid.toString() + " - start");
        is = okm.getContent(DOC_ID);
        LOG.info(uuid.toString() + " - end");
    } catch (RepositoryException | IOException | PathNotFoundException | AccessDeniedException | DatabaseException | UnknowException | WebserviceException e) {
        throw new RuntimeException(e);
    }
}, NR_OF_CLIENTS);
I see that you are not closing the InputStream. Here the question is if the InputStream passed to the method must be closed by you or internally by us I will discuss with my colleagues. Might be have sense in this scenario always close the stream from our side.
 #45254  by Catscratch
 
Usually it would be good. Yes.

But the problem is, that you use a StreamedInputStream on server side which means it is read when the client starts to read it. So if you close the stream on the server side the client won't be able to read it anymore.

Everything is translated to HTTP chunked loading.

I don't know what's best practice, but maybe its a good idea to define some kind of timeout the client has to start reading the inputstream and if he don't ... the server kills the connection to prevent thread overflows.
 #45268  by jllort
 
There's no much consensus about if the InputStream must be closed into the method or outside, our policy until now has been who open the stream must close it. Our proposal goes in direction of :
InputStream is;
Code: Select all
try {
    UUID uuid = UUID.randomUUID();
    LOG.info(uuid.toString() + " - start");
    is = okm.getContent(DOC_ID);
    LOG.info(uuid.toString() + " - end");
} catch (RepositoryException | IOException | PathNotFoundException | AccessDeniedException | DatabaseException | UnknowException | WebserviceException e) {
    throw new RuntimeException(e);
} finally {
    IOUtils.closeQuietly(is);
}
There's any reason why it is not working ?
 #45292  by Catscratch
 
If the clients calls close everything is ok. You e.g. use IOUtils to close quietly.

Anyway. If you got a client which won't close the stream it's kept open. And this is an easy way for the client to stress the server or even more, bring it down.

You can use my demo code from above, e.g. against you demo openkm instances (https://demo.openkm.com/). With this code you can bring down you instance relatively easy so that no one may use it anymore.

You're right. The client should close the stream. And everything is ok if he do so. But overall it's a security issue. You can't assume that every client uses the rest api gracefully.
 #49654  by jllort
 
For a huge number of simultaneous calls maybe you need a cluster, consider the hardware have a limit and OpenKM does something else than downloading a file ( consider must check security grants etc... )

About Us

OpenKM is part of the management software. A management software is a program that facilitates the accomplishment of administrative tasks. OpenKM is a document management system that allows you to manage business content and workflow in a more efficient way. Document managers guarantee data protection by establishing information security for business content.