Page 1 of 2

Respository export/import fails

PostPosted:Wed Nov 02, 2011 1:57 pm
by Catscratch
Ok, as requested I start a new topic for this problem.

I tried to export and than import the repository to change the versioning configuration from derby to mysql.

Ok, here is what I did.

I wrote my own migration guide:
  • Shutdown OpenKM
  • Backup complete MySQL
  • Specific Backup of tables: OKM_CRON_TAB, OKM_CONFIG
  • Backup whole repository folder (OKM_ROOT/repository)
  • Download and install OpenKM Integration WAR
  • Startup OpenKM
  • Go to ADMIN->Repository export->check Metadata and export the repository (REP_EXP_HOME)
  • Shutdown OpenKM
  • edit the OKM_ROOT/repository.xml and reconfigure for MySQL versioning (see below)
  • clear database
    DROP DATABASE okm_app;
    CREATE DATABAE okm_app;
    DROP DATABASE okm_repo;
    CREATE DATABAE okm_repo;
  • edit OKM_ROOT/openkm.cfg and set hibernate.ddl to create
  • clear the the OKM_ROOT/repository folder
    rm -r repository
    mkdir repository
  • Startup OpenKM (to create database tables)
  • Shutdown OpenKM
  • edit OKM_ROOT/openkm.cfg and set hibernate.ddl to none
  • Import the database tables: OKM_CRON_TAB, OKM_CONFIG
  • Startup OpenKM
  • Go to ADMIN->Repository Import, check Metadata and import the repository from REP_EXP_HOME
For the reconfiguration of the versioning storage I removed this:
Code: Select all
<Versioning rootPath="${rep.home}/version">
     ...
    <PersistenceManager class="org.apache.jackrabbit.core.persistence.bundle.DerbyPersistenceManager">
          <param name="url" value="jdbc:derby:${rep.home}/version/db;create=true"/>
          <param name="schemaObjectPrefix" value="version_"/>
     </PersistenceManager>
</Versioning>
And inserted this:
Code: Select all
<Versioning rootPath="${rep.home}/version">
     ...
    <PersistenceManager class="org.apache.jackrabbit.core.persistence.bundle.MySqlPersistenceManager">
          <param name="driver" value="com.mysql.jdbc.Driver"/>
          <param name="url" value="jdbc:mysql://localhost:3306/okm_repo?autoReconnect=true"/>
          <param name="schema" value="mysql"/>
          <param name="user" value="openkm"/>
          <param name="password" value="******"/>

          <param name="schemaObjectPrefix" value="version_"/>
          <param name="externalBLOBs" value="false"/>
     </PersistenceManager-->
</Versioning>
After that I got the following problem.
The repository is imported, but for each originally file there a two files now.
FILENAME.EXT
FILENAME.EXT.JSON

Both are although in the REP_EXP_HOME, but this seems correct, because of the stored metadata. But the JSON-files should not be imported. Instead of importing them, the metadata should be read and set in the database.

Ok, some more information you could need:

OKM Version for export and import
-> Version: 5.1.8-SNAPSHOT (build: 7350)

Errors on importing:
-> Only a lot of warning that no text extraction could be done on the json files

Sample JSON file for an image (1.jpg):
Code: Select all
{"path":"/okm:root/Testfolder/1.jpg","language":"","author":"rf2","created":{"year":2011,"month":9,"dayOfMonth":14,"hourOfDay":15,"minute":58,"second":51},"lastModified":{"year":2011,"month":9,"dayOfMonth":14,"hourOfDay":15,"minute":58,"second":51},"mimeType":"image/jpeg","locked":false,"checkedOut":false,"actualVersion":{"name":"1.0","created":{"year":2011,"month":9,"dayOfMonth":14,"hourOfDay":15,"minute":58,"second":51},"size":11418,"author":"rf2","actual":true,"comment":""},"permissions":15,"uuid":"7663ce8b-0ce4-4482-8cdc-2ec555cf2579","subscribed":false,"convertibleToPdf":true,"convertibleToSwf":true,"convertibleToDxf":false,"subscriptors":[],"keywords":[],"categories":[],"notes":[]}
The imagefile (1.jpg) itself is correct in the REP_EXP_HOME folder.

Hopefully you may help.
Thanks and best regards.

Re: Respository export/import fails

PostPosted:Wed Nov 02, 2011 9:45 pm
by jllort
What's strange to me is that uuid should be the first item and you've got in the middle

could you try with this 1.jpg.okm
Code: Select all
{"uuid":"7663ce8b-0ce4-4482-8cdc-2ec555cf2579","author":"rf2","name":"1.jpg","path":"/okm:root/test/1.jpg","created":{"year":2011,"month":9,"dayOfMonth":14,"hourOfDay":15,"minute":58,"second":51},"lastModified":{"year":2011,"month":10,"dayOfMonth":2,"hourOfDay":22,"minute":41,"second":51},"language":"","keywords":[],"categories":[],"subscriptors":[],"notes":[],"propertyGroups":[],"version":{"author":"okmAdmin","name":"1.0","created":{"year":2011,"month":10,"dayOfMonth":2,"hourOfDay":22,"minute":41,"second":51},"size":59261,"mimeType":"image/jpeg","comment":"Imported by okmAdmin"},"grantedUsers":{"system":15},"grantedRoles":{"UserRole":15}}
And then changing from your 1.jpg.okm the uuid entry to starting point

Re: Respository export/import fails

PostPosted:Thu Nov 03, 2011 5:40 pm
by pavila
First of all, please try with a more recent OpenKM 5.1.8 snapshot. Second, you need also to change the PersistentManager database configuration inside the <Workspace name="${wsp.name}"> tag.

Re: Respository export/import fails

PostPosted:Mon Nov 14, 2011 10:26 am
by Catscratch
pavila wrote:First of all, please try with a more recent OpenKM 5.1.8 snapshot. Second, you need also to change the PersistentManager database configuration inside the <Workspace name="${wsp.name}"> tag.
What should I change there?
Here is the actual repository.xml.
Code: Select all
 <Workspace name="${wsp.name}">
  ...		
	<!-- MySQL -->
	<PersistenceManager class="org.apache.jackrabbit.core.persistence.bundle.MySqlPersistenceManager">
          <param name="driver" value="com.mysql.jdbc.Driver"/>
          <param name="url" value="jdbc:mysql://localhost:3306/okm_repo?autoReconnect=true"/>
          <param name="schema" value="mysql"/>
          <param name="user" value="openkm"/>
          <param name="password" value="********"/>

          <param name="schemaObjectPrefix" value="${wsp.name}_"/>
          <param name="externalBLOBs" value="false"/>
        </PersistenceManager>
   ...

Re: Respository export/import fails

PostPosted:Wed Nov 16, 2011 3:46 pm
by pavila
Can you attach the whole repository.xml file?

Re: Respository export/import fails

PostPosted:Sun Nov 20, 2011 2:59 pm
by Catscratch
Ok.
Code: Select all
<?xml version="1.0"?>
<!DOCTYPE Repository PUBLIC "-//The Apache Software Foundation//DTD Jackrabbit 1.6//EN"
                            "http://jackrabbit.apache.org/dtd/repository-1.6.dtd">
<Repository>
    <!-- virtual file system where the repository stores global state
        (e.g. registered namespaces, custom node types, etc.) -->
    <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
        <param name="path" value="${rep.home}/repository"/>
    </FileSystem>

    <!-- Security configuration -->
    <Security appName="OpenKM">
        <!-- Security manager: FQN of class implementing the JackrabbitSecurityManager interface -->
        <!--<SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager" workspaceName="security">-->
            <!-- workspace access: FQN of class implementing the WorkspaceAccessManager interface -->
            <!-- <WorkspaceAccessManager class="..."/> -->
            <!-- <param name="config" value="${rep.home}/security.xml"/> -->
        <!--</SecurityManager>-->

        <!-- Access manager: FQN of class implementing the AccessManager interface -->
        <AccessManager class="com.openkm.core.OKMAccessManager"/>
        <!-- <AccessManager class="org.apache.jackrabbit.core.security.SimpleAccessManager"/> -->
        <!-- <AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager"> -->
            <!-- <param name="config" value="${rep.home}/access.xml"/> -->
        <!-- </AccessManager> -->

        <!-- <LoginModule class="org.apache.jackrabbit.core.security.simple.SimpleLoginModule"> -->
        <!-- <LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule"> -->
           <!-- Anonymous user name ('anonymous' is the default value) -->
           <!-- <param name="anonymousId" value="anonymous"/> -->
           <!-- Administrator user id (default value if param is missing is 'admin') -->
           <!-- <param name="adminId" value="admin"/> -->
           <!-- <param name="principalProvider" value="..."/> -->
        <!--</LoginModule>-->
    </Security>

    <!-- Location of workspaces root directory and name of default workspace -->
    <Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>

    <!-- Workspace configuration template:
         used to create the initial workspace if there's no workspace yet -->
    <Workspace name="${wsp.name}">
        <!-- Virtual file system of the workspace:
             class: FQN of class implementing the FileSystem interface -->
        <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
            <param name="path" value="${wsp.home}"/>
        </FileSystem>

        <!-- Persistence manager of the workspace:
             class: FQN of class implementing the PersistenceManager interface -->
		<!-- STANDARD -->
		<!--PersistenceManager class="org.apache.jackrabbit.core.persistence.bundle.DerbyPersistenceManager">
          <param name="url" value="jdbc:derby:${wsp.home}/db;create=true"/>
          <param name="schemaObjectPrefix" value="${wsp.name}_"/>
        </PersistenceManager-->
		
		<!-- MySQL -->
		<PersistenceManager class="org.apache.jackrabbit.core.persistence.bundle.MySqlPersistenceManager">
          <param name="driver" value="com.mysql.jdbc.Driver"/>
          <param name="url" value="jdbc:mysql://localhost:3306/okm_repo?autoReconnect=true"/>
          <param name="schema" value="mysql"/>
          <param name="user" value="openkm"/>
          <param name="password" value="********"/>

          <param name="schemaObjectPrefix" value="${wsp.name}_"/>
          <param name="externalBLOBs" value="false"/>
        </PersistenceManager>

        <!-- Search index and the file system it uses.
             class: FQN of class implementing the QueryHandler interface -->
        <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
            <param name="path" value="${wsp.home}/index"/>
            <param name="textFilterClasses" value="
            org.apache.jackrabbit.extractor.PlainTextExtractor,
            org.apache.jackrabbit.extractor.MsWordTextExtractor,
            org.apache.jackrabbit.extractor.MsExcelTextExtractor,
            org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,
            org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,
            org.apache.jackrabbit.extractor.RTFTextExtractor,
            org.apache.jackrabbit.extractor.HTMLTextExtractor,
            org.apache.jackrabbit.extractor.XMLTextExtractor,
            org.apache.jackrabbit.extractor.PngTextExtractor,
            org.apache.jackrabbit.extractor.MsOutlookTextExtractor,
            com.openkm.extractor.PdfTextExtractor,
            com.openkm.extractor.AudioTextExtractor,
            com.openkm.extractor.ExifTextExtractor,
            com.openkm.extractor.Tesseract3TextExtractor,
            com.openkm.extractor.SourceCodeTextExtractor,
            com.openkm.extractor.MsOffice2007TextExtractor"/>
            <param name="extractorPoolSize" value="2"/>
            <param name="supportHighlighting" value="false"/>
            <param name="indexingConfiguration" value="${wsp.home}/../../../indexing_configuration.xml"/>
        </SearchIndex>
    </Workspace>

    <!-- Configures the versioning -->
    <Versioning rootPath="${rep.home}/version">
        <!-- Configures the filesystem to use for versioning for the respective
             persistence manager -->
        <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
            <param name="path" value="${rep.home}/version" />
        </FileSystem>

        <!-- Configures the persistence manager to be used for persisting version state.
             Please note that the current versioning implementation is based on
             a 'normal' persistence manager, but this could change in future
             implementations. -->
        <PersistenceManager class="org.apache.jackrabbit.core.persistence.bundle.DerbyPersistenceManager">
          <param name="url" value="jdbc:derby:${rep.home}/version/db;create=true"/>
          <param name="schemaObjectPrefix" value="version_"/>
        </PersistenceManager>
		
		<!-- MySQL -->
		<!--PersistenceManager class="org.apache.jackrabbit.core.persistence.bundle.MySqlPersistenceManager">
          <param name="driver" value="com.mysql.jdbc.Driver"/>
          <param name="url" value="jdbc:mysql://localhost:3306/okm_repo?autoReconnect=true"/>
          <param name="schema" value="mysql"/>
          <param name="user" value="openkm"/>
          <param name="password" value="********"/>

          <param name="schemaObjectPrefix" value="version_"/>
          <param name="externalBLOBs" value="false"/>
        </PersistenceManager-->
		
    </Versioning>

    <!-- Search index for content that is shared repository wide
         (/jcr:system tree, contains mainly versions) -->
    <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
        <param name="path" value="${rep.home}/repository/index"/>
        <param name="textFilterClasses" value=""/>
        <param name="extractorPoolSize" value="2"/>
        <param name="supportHighlighting" value="false"/>
    </SearchIndex>

    <!-- DataStore improve file handling performance -->
    <DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
        <param name="path" value="${rep.home}/repository/datastore"/>
        <param name="minRecordLength" value="100"/>
    </DataStore>
</Repository>

Re: Respository export/import fails

PostPosted:Mon Dec 05, 2011 7:48 pm
by Catscratch
No idea? :-)

Re: Respository export/import fails

PostPosted:Wed Dec 14, 2011 11:24 am
by pavila
Take a look to the PersistenceManager section whithin Versioning: the MySQL configuration is commented. Also try with the released OpenKM 5.1.8 (final not the nightbuild)

Re: Respository export/import fails

PostPosted:Wed Dec 21, 2011 3:55 pm
by Catscratch
I tried it. It's still the old problem. On the import, all files AND json files are imported. The import should insert the file and read the metadata from the json file. But it imports both...

Re: Respository export/import fails

PostPosted:Wed Jan 04, 2012 4:15 pm
by Catscratch
Any idea?

Re: Respository export/import fails

PostPosted:Mon Jan 09, 2012 9:20 pm
by pavila
The import metadata feature is developed in trunk branch and has been partially backported to OpenKM 5.1.x so this is why you can't import metadata: is not fully implemented on OpenKM 5.1.8.

I have made a backport from trunk to OpenKM 5.1.9-SNAPSHOT so you can test it.

Re: Respository export/import fails

PostPosted:Fri Feb 17, 2012 10:10 pm
by elsteve
It appears that this is still broken in the recently released 5.1.9.

I have a 5.1.8-2 system running, exported the repository with meta-data and imported into a brand new 5.1.9 install. It loaded the .JSON file as an actual file instead of applying the meta-data correctly.

Any thoughts?

Thank you!

Re: Respository export/import fails

PostPosted:Fri Feb 17, 2012 10:45 pm
by pavila
Please, can you attach an zip archive with a few exported documents with metadata? I would like to reproduce the issue in my local installation.

Re: Respository export/import fails

PostPosted:Sun Feb 19, 2012 6:20 pm
by elsteve
pavila wrote:Please, can you attach an zip archive with a few exported documents with metadata? I would like to reproduce the issue in my local installation.
Sure - files were exported from a 5.1.8-2 system.

Thanks!

Re: Respository export/import fails

PostPosted:Mon Feb 20, 2012 9:36 am
by pavila
Ok, I see the problem: Repository export metadata files extension has been changed from .json ( OpenKM 5.1.8 ) to .okm ( OpenKM 5.1.9 ). Rename them to the new extension.