Concurrency Problems / Stress Test on REST API

Do you want to create a native client or integrate with third party applications: webservices are the solution.
Forum rules
Please, before asking something see the documentation wiki or use the search feature of the forum. And remember we don't have a crystal ball or mental readers, so if you post about an issue tell us which OpenKM are you using and also the browser and operating system version. For more info read How to Report Bugs Effectively.
Post Reply
Catscratch
Platinum Boarder
Platinum Boarder
Posts: 334
Joined: Wed Feb 16, 2011 10:35 am

Concurrency Problems / Stress Test on REST API

Post by Catscratch » Wed Jan 24, 2018 1:05 pm

Hi,

I noticed a strange behaviour when accessing the rest endpoint massivly parallel. E.g. a lot of users want to download a document. On a certain number OpenKM starts to hang and the threads are working on 100% without returning.

I don't know if it is a tomcat configuration problem or another problem inside OpenKM itself.

To show the problem, I attached a stress test. Even with a small number of (e.g.) 1000 clients I'm able to stuck the OpenKM Demo instance so that no further usage is possible. Interessting is, that the system is running fine again when the test is aborted.

The test calls getContent for a specific document with a certain number of clients.
When taking a threaddump it shows that the system is hanging in DocumentService.java:147 -> IOUtils.copy(...) without returning.

Any advices?

Maybe you can use the test class on your demo instances.
Attachments
OKMStressGetContentTest.zip
(976 Bytes) Downloaded 49 times
My OpenKM installation:
- OpenKM Community 6.3.1 on Tomcat 7, Debian x64, 8GB RAM, Postgresql, LDAP, Only https

Catscratch
Platinum Boarder
Platinum Boarder
Posts: 334
Joined: Wed Feb 16, 2011 10:35 am

Re: Concurrency Problems / Stress Test on REST API

Post by Catscratch » Wed Jan 24, 2018 2:55 pm

Ok, further investigation shows the problem is the streaming inputstream used to deliver binaries chunked via http.

If the client doesn't close the inputstream on his side, the server keeps the socket open forever. That seems to be a tomcat related problem. Or OpenKM shouldn't use streamed resources at all.

Anyway. A workaround for now is to change the tomcat http connector to "org.apache.coyote.http11.Http11NioProtocol" (in server.xml) instead of default one. The Nio connector kills the connection if the client doesn't read it for a certain amount of time. To configure the time you can use the "timeout" property.

Thats not the best way, but for now it seems to prevent a client to easily bring down the server. Maybe you get a better idea.
My OpenKM installation:
- OpenKM Community 6.3.1 on Tomcat 7, Debian x64, 8GB RAM, Postgresql, LDAP, Only https

jllort
Moderator
Moderator
Posts: 10707
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: Concurrency Problems / Stress Test on REST API

Post by jllort » Thu Jan 25, 2018 8:36 am

You are executing the client from the server side ? because when you talk about "If the client doesn't close the InputStream on his side, the server keeps the socket open forever. " I do not understanding if it's a missing close InputStream from server side or it's from client side what makes connection socket still ready ?

Catscratch
Platinum Boarder
Platinum Boarder
Posts: 334
Joined: Wed Feb 16, 2011 10:35 am

Re: Concurrency Problems / Stress Test on REST API

Post by Catscratch » Thu Jan 25, 2018 2:29 pm

I'm using the delivered test class. It can be executed from anywhere (not inside the server).

The Sdk4j creates an InputStream and keeps it open which means the http connection is hold open.
My OpenKM installation:
- OpenKM Community 6.3.1 on Tomcat 7, Debian x64, 8GB RAM, Postgresql, LDAP, Only https

jllort
Moderator
Moderator
Posts: 10707
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: Concurrency Problems / Stress Test on REST API

Post by jllort » Sat Jan 27, 2018 4:50 pm

Looking in your code:

Code: Select all

stress(() -> {
            InputStream is;
            try {
                UUID uuid = UUID.randomUUID();
                LOG.info(uuid.toString() + " - start");
                is = okm.getContent(DOC_ID);
                LOG.info(uuid.toString() + " - end");
            } catch (RepositoryException | IOException | PathNotFoundException | AccessDeniedException | DatabaseException | UnknowException | WebserviceException e) {
                throw new RuntimeException(e);
            }
        }, NR_OF_CLIENTS);
I see that you are not closing the InputStream. Here the question is if the InputStream passed to the method must be closed by you or internally by us I will discuss with my colleagues. Might be have sense in this scenario always close the stream from our side.

Catscratch
Platinum Boarder
Platinum Boarder
Posts: 334
Joined: Wed Feb 16, 2011 10:35 am

Re: Concurrency Problems / Stress Test on REST API

Post by Catscratch » Mon Jan 29, 2018 3:07 pm

Usually it would be good. Yes.

But the problem is, that you use a StreamedInputStream on server side which means it is read when the client starts to read it. So if you close the stream on the server side the client won't be able to read it anymore.

Everything is translated to HTTP chunked loading.

I don't know what's best practice, but maybe its a good idea to define some kind of timeout the client has to start reading the inputstream and if he don't ... the server kills the connection to prevent thread overflows.
My OpenKM installation:
- OpenKM Community 6.3.1 on Tomcat 7, Debian x64, 8GB RAM, Postgresql, LDAP, Only https

jllort
Moderator
Moderator
Posts: 10707
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: Concurrency Problems / Stress Test on REST API

Post by jllort » Thu Feb 01, 2018 2:59 pm

There's no much consensus about if the InputStream must be closed into the method or outside, our policy until now has been who open the stream must close it. Our proposal goes in direction of :
InputStream is;

Code: Select all

            try {
                UUID uuid = UUID.randomUUID();
                LOG.info(uuid.toString() + " - start");
                is = okm.getContent(DOC_ID);
                LOG.info(uuid.toString() + " - end");
            } catch (RepositoryException | IOException | PathNotFoundException | AccessDeniedException | DatabaseException | UnknowException | WebserviceException e) {
                throw new RuntimeException(e);
            } finally {
               IOUtils.closeQuietly(is);
            }
There's any reason why it is not working ?

Catscratch
Platinum Boarder
Platinum Boarder
Posts: 334
Joined: Wed Feb 16, 2011 10:35 am

Re: Concurrency Problems / Stress Test on REST API

Post by Catscratch » Tue Feb 06, 2018 5:13 pm

If the clients calls close everything is ok. You e.g. use IOUtils to close quietly.

Anyway. If you got a client which won't close the stream it's kept open. And this is an easy way for the client to stress the server or even more, bring it down.

You can use my demo code from above, e.g. against you demo openkm instances (https://demo.openkm.com/). With this code you can bring down you instance relatively easy so that no one may use it anymore.

You're right. The client should close the stream. And everything is ok if he do so. But overall it's a security issue. You can't assume that every client uses the rest api gracefully.
My OpenKM installation:
- OpenKM Community 6.3.1 on Tomcat 7, Debian x64, 8GB RAM, Postgresql, LDAP, Only https

Post Reply