Page 1 of 1
Administration multiple imports
PostPosted:Tue Dec 20, 2016 9:11 am
by openkm_user
Hi,
Can we do multiple imports through Administration -> Import? Like same user logging in through different browsers or multiple tabs of same browser and importing at the same time?
Thanks!
Re: Administration multiple imports
PostPosted:Wed Dec 21, 2016 4:43 pm
by jllort
Might be. If imports does not conflict one each others, I think should not be a problem. What you have in mind, because you might be interested in plan it into a crontab task or script ( periodic execution ).
Re: Administration multiple imports
PostPosted:Mon Dec 26, 2016 12:19 pm
by openkm_user
We are importing couple terabytes of documents and doing it through
Administration ->
Import and this is done using one user (one instance). I am not sure if we can do like, import document1, document2,...document10 in a browser and import document11, document12,...documentn using same user in different browser. Will this create any problems?
Thanks so much for all the answers
.
Re: Administration multiple imports
PostPosted:Tue Dec 27, 2016 2:39 pm
by jllort
For a huge import I suggest using Administration > Import feature. I have another question ? This import is a periodical import or you are only making an initial one ? did you changed to MySQL database or another one rather DBSQL ( what comes by default ) ?
Re: Administration multiple imports
PostPosted:Tue Dec 27, 2016 3:19 pm
by openkm_user
Yes, we are doing through Administration -> Import only, but can we do this in 2 or more instances? Like 2 or more users login to OpenKM (using same authentication or different) and import at the same time.
Currently import is done by only one user in chunks (few GBs at a time), after this current 2TB documents there will be more GBs added in future. And yes we have changed to MySQL as per your instruction for production.
Re: Administration multiple imports
PostPosted:Wed Dec 28, 2016 6:06 pm
by jllort
I would like more detailed information about how you are importing. The source folder in the server and destination folder into OpenKM are always the same or they changes for each import process ? for example you always import from /home/openkm/import folder to /okm:root/imported into OpenKM or it changes.
I'm trying to think if there's some way to automatize the process, and not depending on user login ( basically copy in the server and get some automatic process what makes it for you ). But before suggesting this solution I would like to understanding if it will going right in your scenario.
Re: Administration multiple imports
PostPosted:Tue Jan 03, 2017 9:26 am
by openkm_user
A user userA will import from /home/openkm/import/folderA into /okm:root/imported/folderA and another user userB will do /home/openkm/import/folderB into /okm:root/imported/folderB. So basically both are importing into /okm:root/imported but different files are imported into different folders.
Re: Administration multiple imports
PostPosted:Wed Jan 04, 2017 10:49 am
by jllort
Always into the same folders, confirm this point.
For example I suggest this script, create a crontab task for it.
Code: Select allimport com.openkm.module.db.stuff.DbSessionManager;
import com.openkm.api.*;
import com.openkm.util.impexp.HTMLInfoDecorator;
import com.openkm.util.impexp.RepositoryImporter;
import com.openkm.util.impexp.ImpExpStats;
import com.openkm.util.FormatUtil;
import java.io.*;
PrintWriter out = new PrintWriter("/home/openkm/out.html", "UTF-8");
boolean metadata = false;
boolean history = false;
boolean uuid = false;
String systemToken = DbSessionManager.getInstance().getSystemToken();
File dir = new File("/home/openkm/import/folderA");
long begin = System.currentTimeMillis();
ImpExpStats stats = RepositoryImporter.importDocuments(systemToken, dir, "/okm:root/imported/folderA", metadata, history, uuid, out, new HTMLInfoDecorator(files));
long end = System.currentTimeMillis();
out.println("<hr/>");
out.println("<div class=\"ok\">Filesystem '"+new File(fsPath).getAbsolutePath()+"' imported into '"+repoPath+"'</div>");
out.println("<br/>");
out.println("<b>Documents:</b> "+stats.getDocuments()+"<br/>");
out.println("<b>Folders:</b> "+stats.getFolders()+"<br/>");
out.println("<b>Mails:</b> "+stats.getMails()+"<br/>");
out.println("<b>Size:</b> "+FormatUtil.formatSize(stats.getSize())+"<br/>");
out.println("<b>Time:</b> "+FormatUtil.formatSeconds(end - begin)+"<br/>");
// TODO
// After importing should clean folderA and folderB contents !!!
// The same with the other folder
// Also could send html contents into html with MailUtils class
However I suggest a crontab based in jar ( you will be able to debug it easily ). If you do not have OpenKM portable edtion take a look here
https://sourceforge.net/projects/openkmportabledev/
You should be interested in
repository_import.jsp file
RepositoryImporter.java class
https://wiki.openkm.com/index.php/Crontab
https://wiki.openkm.com/index.php/Cront ... e_importer ( another way to solve it )