Metadata Importer unable to locate file in OpenKM

OpenKM has many interesting features, but requires some configuration process to show its full potential.
Forum rules
Please, before asking something see the documentation wiki or use the search feature of the forum. And remember we don't have a crystal ball or mental readers, so if you post about an issue tell us which OpenKM are you using and also the browser and operating system version. For more info read How to Report Bugs Effectively.
Post Reply
jorseng
Junior Boarder
Junior Boarder
Posts: 23
Joined: Tue Jan 16, 2018 2:22 pm

Metadata Importer unable to locate file in OpenKM

Post by jorseng » Wed Jul 11, 2018 3:37 am

Dear Jllort,

I was testing out the file importer and index importer script.
However, it cannot locate files in OpenKM although i have checked that the file path and name is the same.
Notice the image below, filepath is the same as the one in the importer but it is not able to locate the file for adding metadata values.

not_found_error.png

Code: Select all

import com.openkm.core.*;
import com.openkm.api.*;
import com.openkm.module.db.stuff.DbSessionManager;

import java.text.*;
import java.io.*;
import java.nio.file.Files;
import java.nio.file;
import java.nio.file.Path;
import java.util.*;
import java.util.Iterator;
import java.util.List;
import java.util.Collection;
import java.util.concurrent;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.FilenameUtils;

import com.googlecode.jcsv.CSVStrategy;
import com.googlecode.jcsv.reader.CSVReader;
import com.googlecode.jcsv.reader.internal.CSVReaderBuilder;
import com.googlecode.jcsv.reader.internal.DefaultCSVEntryParser;

import com.openkm.dao.bean.QueryParams;
import com.openkm.bean.QueryResult;
import com.openkm.util.FileLogger;
import com.openkm.util.PathUtils;
import com.openkm.bean.ContentInfo;

//******************************PARAMETERS********************************************

OKMDocument okmdocument = OKMDocument.getInstance();
OKMFolder okmfolder = OKMFolder.getInstance();
String token = DbSessionManager.getInstance().getSystemToken();
String okm_path = "/okm:root/scanner/MR/";
String upload_path = "/home/ksklsu/batchupload/MR/";
String backup_path = "/home/ksklsu/backup/backup_batchupload/MR/";

//******************************PARAMETERS********************************************

//************************* AUTOIMPORT FILE START ****************************************

public void autoImport(String okmPath, File fldpath){

  try {
	
	print("Scanning " + fldpath.getName() + "<br>");
	for (File folder : fldpath.listFiles()) {
		print("Importing " + folder.getName() + "<br>");
	
	try {
	  if (folder.isDirectory()) {
		try {
		  
		// To Create folder in OKM
		try {
		  okmfolder.createSimple(token, okmPath + folder.getName());
		} catch (ItemExistsException ie) {
		  print("ksklsu-folder already exists<br>");
		}
		
		// To create system backup folder
		File dir = new File (backup_path + folder.getName());
		try {
			if(dir.exists()){
				print( dir.getName() + " Folder Exist <br>");
			} else {
				dir.mkdir();
				dir.setReadable(true,false);
				dir.setWritable(true,false);
				print("System- " + dir.getName() +" Folder created <br>");
			}
		} catch (Exception e) {
			print ("Exception " + e + "<br>");
		}

		// Add files in the folder
			for (File file : folder.listFiles()) {
				// Check if file is still writing
				String newFileName = dir + "/" + file.getName();
				Path target_path = new File(newFileName).toPath();
				
				String file_ext = FilenameUtils.getExtension(file.getPath());
				String ext = "csv";
				print (file_ext+ "<br>");

				// Only import non csv because metadata.csv not needed
				if(!file_ext.equals(ext)) {
				
					long length = file.length();
					Thread.sleep(1000);
					if (file.length() > length) continue;  // Skip file this time
					try {
					String x_file = okmPath + folder.getName() + "/" + file.getName();
					okmdocument.createSimple(token, x_file, new FileInputStream(file));
					print("Created " + okmPath + file.getName() + "<br>");
					} catch (Exception e) {
						print("Exception: " + e + "<br>");
						continue;
					}
					// Move files
					try {
						Files.move(file.toPath(), target_path);
						print("Moved to backup " + file.getName());
					} catch (Exception e) {
						print ("Exception: " + e);
						continue;
					}
				} 
			}
			
			
		} catch (Exception e) {
			print ("Exception: " + e + "<br>");
		}
	  }
	} catch (Exception e) {
	  print ("Exception:" + e + "<br>");
	  // Something bad happened to prevent import. Skip to next file.
	 continue;
	}
	}
  } catch (Exception e) {
	print("Exception: " + e + "<br>");
  }

}

autoImport(okm_path, new File(upload_path)); // 1. Check folder path for upload



//************************* AUTOIMPORT FILE END ****************************************/

//************************* AUTOINDEXING START **************************************
// Start of metadata csv file grabbing, auto indexing based on csv
// CHECK CSV FORMAT, SAVING IN LATER VERSION CAUSES INCOMPATIBILITY

String 	FILE_LOG_NAME = "CSVLOG";
String 	Metadata_path = upload_path; 
File	META_FILE_PATH= new File(Metadata_path);
List<String> BATCH = new ArrayList<String>();

String 	META_FILE_NAME = "/metadata.csv";	

String 	grpName  = "okg:medicalrecord";	
String	grpLabel1= "okp:medicalrecord.pt";
String	grpLabel2= "okp:medicalrecord.mrn";
String	grpLabel3= "okp:medicalrecord.date";
String 	grpLabel4= "okp:medicalrecord.name";
String	grpLabel5= "okp:medicalrecord.ic";
String	grpLabel6= "okp:medicalrecord.case";
String	grpLabel7= "okp:medicalrecord.epi";

int uniqueFileName = 1;
int column1 = 2; 
int column2 = 3; 
int column3 = 4;
int column4 = 5;
int column5 = 6;
int column6 = 7;
int	column7	= 8;

try {
    for (File folder : META_FILE_PATH.listFiles()) {
	  try {
		  if (folder.isDirectory()) {
			  BATCH.add(folder.getName());
		  }
	  } catch (Exception e) {
		  print("Exception: " + e + "</br>");
	  }
	}
} catch (Exception e) {
	print("Exception: " + e + "</br>");
}

for (String batch : BATCH) {
	String mtdata_file = Metadata_path + batch + META_FILE_NAME;
	print("Importing metadata " + batch + "</br>");
	// Format defintion
	char delimiter = ',';
	char quoteCharacter = '"';
	char commentIndicator = '#';
	boolean skipHeader = true;
	boolean ignoreEmptyLines = true;
	CSVStrategy strategy = new CSVStrategy(delimiter, quoteCharacter, commentIndicator, skipHeader, ignoreEmptyLines);
	// File reader
	Reader reader = new FileReader(mtdata_file);
	// CSV reader
	CSVReader csvParser = new CSVReaderBuilder(reader).strategy(strategy).entryParser(new DefaultCSVEntryParser()).build();
	List data = csvParser.readAll();
	int count = 1;
	int countFound = 0;
	int countNotDocument = 0;
	int moreThanOneDocumentFound = 0;
	int notFound = 0;
	int noName = 0;

	for (Iterator it = data.listIterator(); it.hasNext();) {
		String[] row = (String[]) it.next();
		String docPath = row[uniqueFileName];
		print(count + ">>>> " + docPath);

		if (docPath != null && !docPath.equals("")) {
			QueryParams queryParams = new QueryParams();
			queryParams.setDomain(QueryParams.DOCUMENT);
			queryParams.setName(PathUtils.getName(docPath));
			Collection results = OKMSearch.getInstance().find(token, queryParams);

			if (results.size() == 1) {
				QueryResult queryResult = (QueryResult) results.iterator().next();
				if (queryResult.getNode() != null) { 
					print("found");
					countFound++;

					Map mtdata = new HashMap();
					mtdata.put(grpLabel1,row[column1]);
					mtdata.put(grpLabel2,row[column2]);
					mtdata.put(grpLabel3,row[column3]);
					mtdata.put(grpLabel4,row[column4]);
					mtdata.put(grpLabel5,row[column5]);
					mtdata.put(grpLabel6,row[column6]);
					mtdata.put(grpLabel7,row[column7]);
					
					// ADD GROUP
					OKMPropertyGroup.getInstance().addGroup(token, docPath, grpName);
					// Set property value
					OKMPropertyGroup.getInstance().setPropertiesSimple(token, docPath, grpName, mtdata);
			
				} else {
					print("error is not document");
					countNotDocument++;
				}
			} else if (results.size() > 1) {
				print("error more than one document found can not decide");
				moreThanOneDocumentFound++;
			} else {
				print("not found");
				notFound++;
			}
		} else {
			print("error document has no name");
			noName++;
		}
		print("</br>");

		//FileLogger.info(FILE_LOG_NAME, "Document name ''{0}'' to ''{1}''", row[0], row[posDocRevNo]);
		count++;
	}
	
	print("Total:" + count + "</br>");
	print("Found:" + countFound + "</br>");
	print("Error not document:" + countNotDocument + "</br>");
	print("Error more then one document found:" + moreThanOneDocumentFound + "</br>");
	print("Error not found:" + notFound + "</br>");
	print("Error name empty:" + notFound + "</br>");

}

//************************* AUTOINDEXING END**************************************/

//********************* AUTOFOLDER MEDICAL RECORD START **************************


print("Auto-folder starting..."+ "</br>");

String medicalrecordpath = "/okm:root/MR_temp/";
String propertyGroup =  "okg:medicalrecord";
String scanFolder = "/okm:root/scanner/MR/"; 
int    countMove = 0;
ArrayList doclist = OKMDocument.getInstance().getChildren(token,scanFolder);// ADDED TOKEN


for (i=0;i<doclist.size();i++){
	
	if (OKMPropertyGroup.getInstance().hasGroup(token,doclist.get(i).getUuid(),propertyGroup))  { // ADDED TOKEN
		
		OKMDocument.getInstance().move(token,doclist.get(i).getUuid(),medicalrecordpath); // ADDED TOKEN
		countMove++;
	}

}


print("Moved " + countMove + " files to " + medicalrecordpath + "</br>");
print("Auto-folder end.");
//********************* AUTOFOLDER MEDICAL RECORD END ******************************/

The text below is the metadata.csv file.

Code: Select all

Name,DP_uniqueFileName,DP_PatientType,DP_MRN,DP_AdmDate,DP_Name,DP_IC,DP_CaseNo,DP_Episode
"DayPatient 53","/okm:root/scanner/MR/Batch 1/00546598 - 20180628.pdf","dp","00546598","20180628120000","CHIN YOONG (FOLDER)","420401065018","201800007217",""
"InPatient 54","/okm:root/scanner/MR/Batch 1/00554001 - 20180627.pdf","ip","00554001","20180627120000","MOHANA SUN DRAM A/L PARAMUCHUWA","800107065847","201800007119",""

I don't get why system is unable to locate the files. Please help.

jllort
Moderator
Moderator
Posts: 10267
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: Metadata Importer unable to locate file in OpenKM

Post by jllort » Wed Jul 11, 2018 2:58 pm

I do not like scripting an specially the big scripts because you can not debug on it. I suggest convert as a crontab https://docs.openkm.com/kcenter/view/ok ... b-job.html

In the output of the script are shown errors caused by document already exists, this should not affect.

You should verify the property group code:
Check if it works the code below:

Code: Select all

String docPath = "/okm:root/scanner/MR/Batch 1/00546598 - 20180628.pdf";
OKMRepository.getInstance().hasNode(null,docPath);
if it works then check:

Code: Select all

Map mtdata = new HashMap();
mtdata.put(grpLabel1,row[column1]);
mtdata.put(grpLabel2,row[column2]);
mtdata.put(grpLabel3,row[column3]);
mtdata.put(grpLabel4,row[column4]);
mtdata.put(grpLabel5,row[column5]);
mtdata.put(grpLabel6,row[column6]);
mtdata.put(grpLabel7,row[column7]);
					
// ADD GROUP
OKMPropertyGroup.getInstance().addGroup(token, docPath, grpName);
// Set property value
OKMPropertyGroup.getInstance().setPropertiesSimple(token, docPath, grpName, mtdata);

jorseng
Junior Boarder
Junior Boarder
Posts: 23
Joined: Tue Jan 16, 2018 2:22 pm

Re: Metadata Importer unable to locate file in OpenKM

Post by jorseng » Thu Jul 12, 2018 4:14 am

Dear Jllort,

I have checked as you have suggested, it returned true.

I will try to convert it into a crontab job, meanwhile i have tested with a newly created csv file.
It seems to be working fine with the newly create csv file.
Funny thing is, the wording and separator is exactly the same. Thus I am quite certain the issue lies with the csv file.

Is there any behavior of csv files which i should be aware of?

jorseng
Junior Boarder
Junior Boarder
Posts: 23
Joined: Tue Jan 16, 2018 2:22 pm

Re: Metadata Importer unable to locate file in OpenKM

Post by jorseng » Thu Jul 12, 2018 4:31 am

Dear Jllort,

I have found the issue, it is the unicode used for the CSV file, i have no idea why the older file was using unicode(UTF-16).
It is working once i saved it as UTF-8.

jllort
Moderator
Moderator
Posts: 10267
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: Metadata Importer unable to locate file in OpenKM

Post by jllort » Fri Jul 13, 2018 3:51 pm

Did you download the CSV file from our documentation or you have created ?
If you have downloaded, tell me from where and I will review it.

jorseng
Junior Boarder
Junior Boarder
Posts: 23
Joined: Tue Jan 16, 2018 2:22 pm

Re: Metadata Importer unable to locate file in OpenKM

Post by jorseng » Mon Jul 16, 2018 1:50 am

No, it is not from your documentation. :)

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 3 guests