Not sure if pagination works as expected

He we will discuss about how to make customization and improvement to the OpenKM source code.
Forum rules
Please, before asking something see the documentation wiki or use the search feature of the forum. And remember we don't have a crystal ball or mental readers, so if you post about an issue tell us which OpenKM are you using and also the browser and operating system version. For more info read How to Report Bugs Effectively.
Post Reply
HugoHiasl
Fresh Boarder
Fresh Boarder
Posts: 5
Joined: Wed Jun 17, 2020 4:35 pm

Not sure if pagination works as expected

Post by HugoHiasl »

Hello,

as stated in the user forum I am new to openkm and reviving my j2ee skills from tens of years ago.

I was now uploading about 1700 files to a single folder which is meant to be the "Unhandled" folder for REST uploads.

When I try to view the documents it takes a long time until they appear. A lot of the time might get lost due to DEBUG mode and file logging.
But when I digged a bit deeper I realized that even when I have enabled pagination and only show 10 entries per folder, thousands of SQLs are executed for the short period of time to show the first page of this folder and then immediately go to second page.

In that (lets say 1 minute) there are 21.970 result set processing entries in the hibernate trace file.
For example:

Code: Select all

/* load collection com.openkm.dao.bean.NodeBase.rolePermissions */
select rolepermis0_.NRP_NODE as NRP1_83_0_, rolepermis0_.NRP_PERMISSION as NRP2_0_, rolepermis0_.NRP_ROLE as NRP3_0_ from OKM_NODE_ROLE_PERMISSION rolepermis0_ where rolepermis0_.NRP_NODE=?
This is there once per document in folder. Not only for document to show in pagination. Same for properties, subscriptions etc.

Maybe my approach is wrong. But probably there is a limitation missing if pagination is enabled. Can you please take a look?

For the attached file I started tomcat in DEBUG. Then did wait until it finished to start up. Pagination in openkm is already switched on.

1. I clicked on "_New" folder with approx. 1700 documents.
2. I clicked on little right arrow to go to next page
3. Stopped tomcat right after displaying the results.

After that I cut off all the lines until the first statement to get the node_document entries.

Thanks a lot.

EDIT: The forum says: failed to move uploaded file. So I cannot upload it. I think the description to reproduce it is sufficient. If not let me know and I can send via email or other option.

jllort
Moderator
Moderator
Posts: 11151
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: Not sure if pagination works as expected

Post by jllort »

There're several problems what can not be easily solved in the community edition. Let's make some history. Current CE comes from 6.2 professional branch. I'm talking about 4-6 years ago. In that period we were optimist and we use hibernate with a bad paradigm "get everything in the UI although you do not need". In professional version 6.4 we make some improvement to change this scenario but although we get a better performance in comparison with CE also there's some lock. Finally, in version 7 we have to rebuild everything thinking in terms of "get only what you need", "minimize a control de queries", focus in performance and large list ( for example we have done tests with 100K file into the same parent and in works reasonable well ).

In version 6.4 we introduced new pagination what works better but still have not been shared in the community edition. Anyway, it only solves part of the problems in the UI. In version 7 we have thought everything again.

Why this large dissertation? Because in the community edition should be necessary a deep and currently we are not in this stage. The only approach what will work for you is split data into several folders, reducing the total number of elements by a parent you will minimize the unwished defect.

You might success working with CE for 1-2 million docs, but if you are thinking in a huge amount 10, 30 million an upper the CE is not for you.

Post Reply