unable to reopen IndexReader when using SDK4J find method

[Catscratch]

Hi,

I try to find documents by using the SDK.
SDK 1.0
OpenKM 6.3.0 Community

Code: Select all

OKMWebservices okm = OKMWebservicesFactory.newInstance("http://localhost:8080/OpenKM", "okmAdmin", "admin");
QueryParams p = new QueryParams();
p.setDomain(QueryParams.DOCUMENT);
List<QueryResult> list = okm.find(p);

On the last line (find) I get an UnknownException.

Code: Select all

com.openkm.sdk4j.exception.UnknowException:  Unable to reopen IndexReader
	at com.openkm.sdk4j.impl.SearchImpl.find(SearchImpl.java:283)
	at com.openkm.sdk4j.OKMWebservice10.find(OKMWebservice10.java:829)
	at TestOkmSdk.test(TestOkmSdk.java:57)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

After a fresh restart of OpenKM I get results, but not all documents. Only a few. But after some time I only get the exception above. So it seems so be a problem with the lucene indexer I think. What can I do to prevent this?

Thanks!

[jllort]

First, this is a quite strange search, because you're not filtering for any field, that is not good idea. If you wan to get all documents you got in repository this is not the way for doing it. If you for example do a query with name "*.doc" you got the same type of error ?

[Catscratch]

Yes.

I added an additional filter param:

Code: Select all

p.setName("*.png");

That should return only 1 document. Because there is only 1 png.
But the same exception appears.

In the server log there is only a warning:

Code: Select all

2014-08-28 10:12:11,189 [http-apr-0.0.0.0-8080-exec-3] WARN  org.apache.cxf.jaxrs.impl.WebApplicationExceptionMapper- WebApplicationException has been caught : Unable to reopen IndexReader

But on the server side I also got a lot of exceptions after rebooting:

Code: Select all

java.io.FileNotFoundException: /opt/openkm63/repository/index/com.openkm.dao.bean.NodeBase/_e.fnm (Datei oder Verzeichnis nicht gefunden)
	at java.io.RandomAccessFile.open(Native Method)
	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
	at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:69)
	at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:90)
	at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:91)
	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78)
	at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:353)
	at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:68)
	at org.apache.lucene.index.IndexWriter.getFieldInfos(IndexWriter.java:1197)
	at org.apache.lucene.index.IndexWriter.getCurrentFieldInfos(IndexWriter.java:1213)
	at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1149)
	at org.hibernate.search.backend.Workspace.createNewIndexWriter(Workspace.java:202)
	at org.hibernate.search.backend.Workspace.getIndexWriter(Workspace.java:180)
	at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:103)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
2014-08-26 20:30:04,139 [Hibernate Search: Directory writer-1] ERROR org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor- Unexpected error in Lucene Backend: 
org.hibernate.search.SearchException: Unable to remove class com.openkm.dao.bean.NodeDocument#8c14d400-cdda-4b85-aee2-44f54e0f432a from index.
	at org.hibernate.search.backend.impl.lucene.works.DeleteWorkDelegate.performWork(DeleteWorkDelegate.java:91)
	at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:106)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
	at org.hibernate.search.backend.impl.lucene.works.DeleteWorkDelegate.performWork(DeleteWorkDelegate.java:87)
	... 6 more

and

Code: Select all

2014-08-26 20:30:04,140 [Hibernate Search: Directory writer-1] ERROR org.hibernate.search.exception.impl.LogErrorHandler- Exception occurred org.hibernate.search.SearchException: Unable to remove class com.openkm.dao.bean.NodeDocument#8c14d400-cdda-4b85-aee2-44f54e0f432a from index.
Primary Failure:
	Entity com.openkm.dao.bean.NodeDocument  Id 8c14d400-cdda-4b85-aee2-44f54e0f432a  Work Type  org.hibernate.search.backend.DeleteLuceneWork
Subsequent failures:
	Entity com.openkm.dao.bean.NodeDocument  Id 8c14d400-cdda-4b85-aee2-44f54e0f432a  Work Type  org.hibernate.search.backend.AddLuceneWork
	Entity com.openkm.dao.bean.NodeDocument  Id 8c14d400-cdda-4b85-aee2-44f54e0f432a  Work Type  org.hibernate.search.backend.AddLuceneWork

org.hibernate.search.SearchException: Unable to remove class com.openkm.dao.bean.NodeDocument#8c14d400-cdda-4b85-aee2-44f54e0f432a from index.
	at org.hibernate.search.backend.impl.lucene.works.DeleteWorkDelegate.performWork(DeleteWorkDelegate.java:91)
	at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:106)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
	at org.hibernate.search.backend.impl.lucene.works.DeleteWorkDelegate.performWork(DeleteWorkDelegate.java:87)
	... 6 more
2014-08-26 20:30:04,140 [Hibernate Search: Directory writer-1] WARN  org.hibernate.search.backend.Workspace- going to force release of the IndexWriter lock

That's why I removed the "index" folder from the "repository" folder to recreate the index. But after some time I run in the same exceptions above.

[jllort]

Seems you got corruption in lucene, probably the computer or the service has shut down on past and from there you got it. First of all I suggest, always got backup and use MySQL or PostgreSQL database if you use default database HSQL ( embeded ) and being corrupted in 90% of cases is not possible recover ( take it in mind ).

In case of lucene repository corruption simply, stop application. Remove folder opt/openkm63/repository/index/. Start application again. Automatically the lucene index will be rebuild ( depending the number of files etc... you'll got top cpu use during some minutes or hours ).

[Catscratch]

I tried removing the index folder a few times now. After restarting openkm, lucene starts building the index again, which works for some time. But the on a certain point (after some time) I got the exceptions above.

I don't know why. Maybe to many documents? (about 25000)

[pavila]

25K documents are very few documents and Lucene index corruption is quite rare, unless you have a problem with the hard disk. I recommend to try a clean OpenKM installation in another machine, and import here all these documents.

[Catscratch]

Ok, I'll give it a try. Thanks so far.

[jllort]

As pavila suggest, could be some hardware problem ( disk sector fails ) what will be the cause of this corruption.