Text Extractor giving mysql exception for all documents

[JavaDev]

Whenever Text Extractor is executed it gives the following exceptions for all documents[pdf, doc, txt, excel etc]

Code: Select all

2015-04-09 17:26:30,427 [http-bio-80-exec-60] WARN  org.hibernate.util.JDBCExceptionReporter - SQL Error: 1366, SQLState: HY000
2015-04-09 17:26:30,428 [http-bio-80-exec-60] ERROR org.hibernate.util.JDBCExceptionReporter - Incorrect string value: '\xE2\x89\xA520 ...' for column 'NDC_TEXT' at row 1
2015-04-09 17:26:30,428 [http-bio-80-exec-60] WARN  com.openkm.extractor.TextExtractorWorker - could not update: [com.openkm.dao.bean.NodeDocument#6402a1fa-783b-456d-b486-8d7977759412]
com.openkm.core.DatabaseException: could not update: [com.openkm.dao.bean.NodeDocument#6402a1fa-783b-456d-b486-8d7977759412]
        at com.openkm.dao.NodeDocumentDAO.textExtractorHelper(NodeDocumentDAO.java:1344)
        at com.openkm.extractor.TextExtractorWorker.processSerial(TextExtractorWorker.java:138)
        at com.openkm.extractor.TextExtractorWorker.processQueue(TextExtractorWorker.java:125)
        at com.openkm.extractor.TextExtractorWorker.run(TextExtractorWorker.java:80)
        at sun.reflect.GeneratedMethodAccessor222.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at bsh.Reflect.invokeOnMethod(Unknown Source)
        at bsh.Reflect.invokeObjectMethod(Unknown Source)
        at bsh.BSHPrimarySuffix.doName(Unknown Source)
        at bsh.BSHPrimarySuffix.doSuffix(Unknown Source)
        at bsh.BSHPrimaryExpression.eval(Unknown Source)
        at bsh.BSHPrimaryExpression.eval(Unknown Source)
        at bsh.Interpreter.eval(Unknown Source)
        at bsh.Interpreter.eval(Unknown Source)
        at bsh.Interpreter.eval(Unknown Source)
        at com.openkm.util.ExecutionUtils.runScript(ExecutionUtils.java:112)
        at com.openkm.core.Cron$RunnerBsh.run(Cron.java:103)
        at com.openkm.servlet.admin.CronTabServlet.execute(CronTabServlet.java:227)
        at com.openkm.servlet.admin.CronTabServlet.doGet(CronTabServlet.java:115)
        at com.openkm.servlet.admin.CronTabServlet.service(CronTabServlet.java:71)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.hibernate.exception.GenericJDBCException: could not update: [com.openkm.dao.bean.NodeDocument#6402a1fa-783b-456d-b486-8d7977759412]
        at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
        at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128)
        at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2613)
        at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2495)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2822)
        at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:113)
        at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273)
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265)
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185)
        at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
        at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)
        at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)
        at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:383)
        at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:133)
        at com.openkm.dao.HibernateUtil.commit(HibernateUtil.java:325)
        at com.openkm.dao.NodeDocumentDAO.textExtractorHelper(NodeDocumentDAO.java:1340)
        ... 63 more
Caused by: java.sql.SQLException: Incorrect string value: '\xE2\x89\xA520 ...' for column 'NDC_TEXT' at row 1
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3562)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3494)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1960)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2114)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2696)
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2105)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2398)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2316)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2301)
        at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
        at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2595)
        ... 76 more

This is affecting the search.

The encoding of the OpenKM Database is set to utf8-utf8_bin. Also when we tried deleting one of the documents for which the above exception occurred its entry remained in the table OKM_NODE_BASE even after deletion from the trash.

[jllort]

It seems a codification problem. Which OpenKM version are you using and what MySQL version.
Does your document have chinese or similar language ... characters ?

[JavaDev]

The Mysql version is 5.1 and OpenKM version is 6.2.5.
Our application would be supporting UTF-8 with 255 characters. Hence the documents consist of the UTF8 characters.
No Chinese or similar lagnguage characters are supported.

[pavila]

I suggest upgrading to a night build and try to reproduce the problem. Take a look at http://wiki.openkm.com/index.php/Migrat ... 3_to_6.3.1

[JavaDev]

We have made few application specific changes to the OpenKM API's and also it is not possible for us to upgrade the version at this point of time.
Is there any other alternative solution for this issue?

[pavila]

The fix has come source code changes. You would need to apply these changes to your own source code. Take a look at subversion changes.