Page 1 of 1

Import document to point different index

PostPosted:Wed Dec 20, 2017 7:02 am
by openkm_user
Greetings,

We already have millions of documents in our repository so the index size is about 10GB, importing new documents now has reduced import speed considerably.

So we would like to point OpenKM to different index only for import (Administration -> Import) but the default index location can remain same for search.

Basically,
Administration -> Import will point to C:\okmrepo\index
Default index will point to C:\openkm\tomcat\repository\index

Please let us know how this can be achieved.

Thanks!

Re: Import document to point different index

PostPosted:Thu Dec 21, 2017 7:35 am
by jllort
Remember do this action with openkm stopped. Take a look here https://docs.openkm.com/kcenter/view/ok ... -home.html . You can move whole repository or only index ( what seems is your case ). Remember after moving that the grants might be changed, the user who's starting openkm also must have grants for accessing there.

In professional edition we have an additional parameter for huge repositories "lucene works asynchronous" that means the new documents are not processed in real time while are imported what cause some delay when importing documents or change metadata and it goes into background queue. That means after few seconds you are able to search from search engine ( I'm talking about search by name or metadata, not search by contents what is another kind of feature ) and the performance of importing data is not affected by lucene search indexing process. Unfortunately is still not present in community edition. I talk about it, because you have milions of docs, I suppose you are using OpenKM from API scenario. Also in huge repositories I suggest you control the growing of OKM_ACTIVITY ( or disable if you really do not need it ) and also consider cleaning OKM_DASHBOARD and OKM_DASHBOARD_ACTIVITY, for huge repositories arrived some point you will need some optimization.

Re: Import document to point different index

PostPosted:Thu Dec 21, 2017 11:24 am
by openkm_user
Hi,

Changing index path is not an issue, but we need to give 2 different index path for one OpenKM instance.

Importing pointing to,
C:\okmrepo\index

Default search in OpenKM pointing to,
C:\openkm\tomcat\repository\index

So the application (searching already imported old data) will continue working and the new import will not have any speed issues as well.

Re: Import document to point different index

PostPosted:Sat Dec 23, 2017 10:40 am
by jllort
You can not split lucene index, the solution is configure lucene index as async ( are needed changes in source code for it , if you want to some improvement in this way you are welcome, the source code is at https://github.com/openkm/document-management-system)

Re: Import document to point different index

PostPosted:Thu Jan 04, 2018 1:36 pm
by openkm_user
How can we stop indexing itself while importing new documents,
Administration -> Import

So after importing we will re-index whole repository. Any help is appreciated.

Thanks in advance!

Re: Import document to point different index

PostPosted:Fri Jan 05, 2018 2:57 pm
by jllort
If you stop the crotab task name "text extraction queue" will not index the content, what is not possible in community version is stop the indexing for document name, author etc.. when the document is created. In professional version exist a configuration parameter for disabling lucene index, but this parameter is still not present in community version.