Administration multiple imports

[openkm_user]

Hi,

Can we do multiple imports through Administration -> Import? Like same user logging in through different browsers or multiple tabs of same browser and importing at the same time?

Thanks!

[jllort]

Might be. If imports does not conflict one each others, I think should not be a problem. What you have in mind, because you might be interested in plan it into a crontab task or script ( periodic execution ).

[openkm_user]

We are importing couple terabytes of documents and doing it through Administration -> Import and this is done using one user (one instance). I am not sure if we can do like, import document1, document2,...document10 in a browser and import document11, document12,...documentn using same user in different browser. Will this create any problems?

Thanks so much for all the answers .

[jllort]

For a huge import I suggest using Administration > Import feature. I have another question ? This import is a periodical import or you are only making an initial one ? did you changed to MySQL database or another one rather DBSQL ( what comes by default ) ?

[openkm_user]

Yes, we are doing through Administration -> Import only, but can we do this in 2 or more instances? Like 2 or more users login to OpenKM (using same authentication or different) and import at the same time.

Currently import is done by only one user in chunks (few GBs at a time), after this current 2TB documents there will be more GBs added in future. And yes we have changed to MySQL as per your instruction for production.

[jllort]

I would like more detailed information about how you are importing. The source folder in the server and destination folder into OpenKM are always the same or they changes for each import process ? for example you always import from /home/openkm/import folder to /okm:root/imported into OpenKM or it changes.

I'm trying to think if there's some way to automatize the process, and not depending on user login ( basically copy in the server and get some automatic process what makes it for you ). But before suggesting this solution I would like to understanding if it will going right in your scenario.

[openkm_user]

A user userA will import from /home/openkm/import/folderA into /okm:root/imported/folderA and another user userB will do /home/openkm/import/folderB into /okm:root/imported/folderB. So basically both are importing into /okm:root/imported but different files are imported into different folders.

[jllort]

Always into the same folders, confirm this point.

For example I suggest this script, create a crontab task for it.

Code: Select all

import com.openkm.module.db.stuff.DbSessionManager;
import com.openkm.api.*;
import com.openkm.util.impexp.HTMLInfoDecorator;
import com.openkm.util.impexp.RepositoryImporter;
import com.openkm.util.impexp.ImpExpStats;
import com.openkm.util.FormatUtil;
import java.io.*;

PrintWriter out = new PrintWriter("/home/openkm/out.html", "UTF-8");
boolean metadata = false;
boolean history = false;
boolean uuid = false;
String systemToken = DbSessionManager.getInstance().getSystemToken();
File dir = new File("/home/openkm/import/folderA");
long begin = System.currentTimeMillis();
ImpExpStats stats = RepositoryImporter.importDocuments(systemToken, dir, "/okm:root/imported/folderA", metadata, history, uuid, out, new HTMLInfoDecorator(files));
long end = System.currentTimeMillis();
out.println("<hr/>");
out.println("<div class=\"ok\">Filesystem '"+new File(fsPath).getAbsolutePath()+"' imported into '"+repoPath+"'</div>");
out.println("<br/>");
out.println("<b>Documents:</b> "+stats.getDocuments()+"<br/>");
out.println("<b>Folders:</b> "+stats.getFolders()+"<br/>");
out.println("<b>Mails:</b> "+stats.getMails()+"<br/>");
out.println("<b>Size:</b> "+FormatUtil.formatSize(stats.getSize())+"<br/>");
out.println("<b>Time:</b> "+FormatUtil.formatSeconds(end - begin)+"<br/>");

// TODO 
// After importing should clean folderA and folderB contents !!!
// The same with the other folder
// Also could send html contents into html with MailUtils class

However I suggest a crontab based in jar ( you will be able to debug it easily ). If you do not have OpenKM portable edtion take a look here https://sourceforge.net/projects/openkmportabledev/

You should be interested in
repository_import.jsp file
RepositoryImporter.java class
https://wiki.openkm.com/index.php/Crontab
https://wiki.openkm.com/index.php/Cront ... e_importer ( another way to solve it )