• Metadata Importer unable to locate file in OpenKM

  • OpenKM has many interesting features, but requires some configuration process to show its full potential.
OpenKM has many interesting features, but requires some configuration process to show its full potential.
Forum rules: Please, before asking something see the documentation wiki or use the search feature of the forum. And remember we don't have a crystal ball or mental readers, so if you post about an issue tell us which OpenKM are you using and also the browser and operating system version. For more info read How to Report Bugs Effectively.
 #46301  by jorseng
 
Dear Jllort,

I was testing out the file importer and index importer script.
However, it cannot locate files in OpenKM although i have checked that the file path and name is the same.
Notice the image below, filepath is the same as the one in the importer but it is not able to locate the file for adding metadata values.

not_found_error.png
not_found_error.png (33.17 KiB) Viewed 2879 times
Code: Select all
import com.openkm.core.*;
import com.openkm.api.*;
import com.openkm.module.db.stuff.DbSessionManager;

import java.text.*;
import java.io.*;
import java.nio.file.Files;
import java.nio.file;
import java.nio.file.Path;
import java.util.*;
import java.util.Iterator;
import java.util.List;
import java.util.Collection;
import java.util.concurrent;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.FilenameUtils;

import com.googlecode.jcsv.CSVStrategy;
import com.googlecode.jcsv.reader.CSVReader;
import com.googlecode.jcsv.reader.internal.CSVReaderBuilder;
import com.googlecode.jcsv.reader.internal.DefaultCSVEntryParser;

import com.openkm.dao.bean.QueryParams;
import com.openkm.bean.QueryResult;
import com.openkm.util.FileLogger;
import com.openkm.util.PathUtils;
import com.openkm.bean.ContentInfo;

//******************************PARAMETERS********************************************

OKMDocument okmdocument = OKMDocument.getInstance();
OKMFolder okmfolder = OKMFolder.getInstance();
String token = DbSessionManager.getInstance().getSystemToken();
String okm_path = "/okm:root/scanner/MR/";
String upload_path = "/home/ksklsu/batchupload/MR/";
String backup_path = "/home/ksklsu/backup/backup_batchupload/MR/";

//******************************PARAMETERS********************************************

//************************* AUTOIMPORT FILE START ****************************************

public void autoImport(String okmPath, File fldpath){

  try {
	
	print("Scanning " + fldpath.getName() + "<br>");
	for (File folder : fldpath.listFiles()) {
		print("Importing " + folder.getName() + "<br>");
	
	try {
	  if (folder.isDirectory()) {
		try {
		  
		// To Create folder in OKM
		try {
		  okmfolder.createSimple(token, okmPath + folder.getName());
		} catch (ItemExistsException ie) {
		  print("ksklsu-folder already exists<br>");
		}
		
		// To create system backup folder
		File dir = new File (backup_path + folder.getName());
		try {
			if(dir.exists()){
				print( dir.getName() + " Folder Exist <br>");
			} else {
				dir.mkdir();
				dir.setReadable(true,false);
				dir.setWritable(true,false);
				print("System- " + dir.getName() +" Folder created <br>");
			}
		} catch (Exception e) {
			print ("Exception " + e + "<br>");
		}

		// Add files in the folder
			for (File file : folder.listFiles()) {
				// Check if file is still writing
				String newFileName = dir + "/" + file.getName();
				Path target_path = new File(newFileName).toPath();
				
				String file_ext = FilenameUtils.getExtension(file.getPath());
				String ext = "csv";
				print (file_ext+ "<br>");

				// Only import non csv because metadata.csv not needed
				if(!file_ext.equals(ext)) {
				
					long length = file.length();
					Thread.sleep(1000);
					if (file.length() > length) continue;  // Skip file this time
					try {
					String x_file = okmPath + folder.getName() + "/" + file.getName();
					okmdocument.createSimple(token, x_file, new FileInputStream(file));
					print("Created " + okmPath + file.getName() + "<br>");
					} catch (Exception e) {
						print("Exception: " + e + "<br>");
						continue;
					}
					// Move files
					try {
						Files.move(file.toPath(), target_path);
						print("Moved to backup " + file.getName());
					} catch (Exception e) {
						print ("Exception: " + e);
						continue;
					}
				} 
			}
			
			
		} catch (Exception e) {
			print ("Exception: " + e + "<br>");
		}
	  }
	} catch (Exception e) {
	  print ("Exception:" + e + "<br>");
	  // Something bad happened to prevent import. Skip to next file.
	 continue;
	}
	}
  } catch (Exception e) {
	print("Exception: " + e + "<br>");
  }

}

autoImport(okm_path, new File(upload_path)); // 1. Check folder path for upload



//************************* AUTOIMPORT FILE END ****************************************/

//************************* AUTOINDEXING START **************************************
// Start of metadata csv file grabbing, auto indexing based on csv
// CHECK CSV FORMAT, SAVING IN LATER VERSION CAUSES INCOMPATIBILITY

String 	FILE_LOG_NAME = "CSVLOG";
String 	Metadata_path = upload_path; 
File	META_FILE_PATH= new File(Metadata_path);
List<String> BATCH = new ArrayList<String>();

String 	META_FILE_NAME = "/metadata.csv";	

String 	grpName  = "okg:medicalrecord";	
String	grpLabel1= "okp:medicalrecord.pt";
String	grpLabel2= "okp:medicalrecord.mrn";
String	grpLabel3= "okp:medicalrecord.date";
String 	grpLabel4= "okp:medicalrecord.name";
String	grpLabel5= "okp:medicalrecord.ic";
String	grpLabel6= "okp:medicalrecord.case";
String	grpLabel7= "okp:medicalrecord.epi";

int uniqueFileName = 1;
int column1 = 2; 
int column2 = 3; 
int column3 = 4;
int column4 = 5;
int column5 = 6;
int column6 = 7;
int	column7	= 8;

try {
    for (File folder : META_FILE_PATH.listFiles()) {
	  try {
		  if (folder.isDirectory()) {
			  BATCH.add(folder.getName());
		  }
	  } catch (Exception e) {
		  print("Exception: " + e + "</br>");
	  }
	}
} catch (Exception e) {
	print("Exception: " + e + "</br>");
}

for (String batch : BATCH) {
	String mtdata_file = Metadata_path + batch + META_FILE_NAME;
	print("Importing metadata " + batch + "</br>");
	// Format defintion
	char delimiter = ',';
	char quoteCharacter = '"';
	char commentIndicator = '#';
	boolean skipHeader = true;
	boolean ignoreEmptyLines = true;
	CSVStrategy strategy = new CSVStrategy(delimiter, quoteCharacter, commentIndicator, skipHeader, ignoreEmptyLines);
	// File reader
	Reader reader = new FileReader(mtdata_file);
	// CSV reader
	CSVReader csvParser = new CSVReaderBuilder(reader).strategy(strategy).entryParser(new DefaultCSVEntryParser()).build();
	List data = csvParser.readAll();
	int count = 1;
	int countFound = 0;
	int countNotDocument = 0;
	int moreThanOneDocumentFound = 0;
	int notFound = 0;
	int noName = 0;

	for (Iterator it = data.listIterator(); it.hasNext();) {
		String[] row = (String[]) it.next();
		String docPath = row[uniqueFileName];
		print(count + ">>>> " + docPath);

		if (docPath != null && !docPath.equals("")) {
			QueryParams queryParams = new QueryParams();
			queryParams.setDomain(QueryParams.DOCUMENT);
			queryParams.setName(PathUtils.getName(docPath));
			Collection results = OKMSearch.getInstance().find(token, queryParams);

			if (results.size() == 1) {
				QueryResult queryResult = (QueryResult) results.iterator().next();
				if (queryResult.getNode() != null) { 
					print("found");
					countFound++;

					Map mtdata = new HashMap();
					mtdata.put(grpLabel1,row[column1]);
					mtdata.put(grpLabel2,row[column2]);
					mtdata.put(grpLabel3,row[column3]);
					mtdata.put(grpLabel4,row[column4]);
					mtdata.put(grpLabel5,row[column5]);
					mtdata.put(grpLabel6,row[column6]);
					mtdata.put(grpLabel7,row[column7]);
					
					// ADD GROUP
					OKMPropertyGroup.getInstance().addGroup(token, docPath, grpName);
					// Set property value
					OKMPropertyGroup.getInstance().setPropertiesSimple(token, docPath, grpName, mtdata);
			
				} else {
					print("error is not document");
					countNotDocument++;
				}
			} else if (results.size() > 1) {
				print("error more than one document found can not decide");
				moreThanOneDocumentFound++;
			} else {
				print("not found");
				notFound++;
			}
		} else {
			print("error document has no name");
			noName++;
		}
		print("</br>");

		//FileLogger.info(FILE_LOG_NAME, "Document name ''{0}'' to ''{1}''", row[0], row[posDocRevNo]);
		count++;
	}
	
	print("Total:" + count + "</br>");
	print("Found:" + countFound + "</br>");
	print("Error not document:" + countNotDocument + "</br>");
	print("Error more then one document found:" + moreThanOneDocumentFound + "</br>");
	print("Error not found:" + notFound + "</br>");
	print("Error name empty:" + notFound + "</br>");

}

//************************* AUTOINDEXING END**************************************/

//********************* AUTOFOLDER MEDICAL RECORD START **************************


print("Auto-folder starting..."+ "</br>");

String medicalrecordpath = "/okm:root/MR_temp/";
String propertyGroup =  "okg:medicalrecord";
String scanFolder = "/okm:root/scanner/MR/"; 
int    countMove = 0;
ArrayList doclist = OKMDocument.getInstance().getChildren(token,scanFolder);// ADDED TOKEN


for (i=0;i<doclist.size();i++){
	
	if (OKMPropertyGroup.getInstance().hasGroup(token,doclist.get(i).getUuid(),propertyGroup))  { // ADDED TOKEN
		
		OKMDocument.getInstance().move(token,doclist.get(i).getUuid(),medicalrecordpath); // ADDED TOKEN
		countMove++;
	}

}


print("Moved " + countMove + " files to " + medicalrecordpath + "</br>");
print("Auto-folder end.");
//********************* AUTOFOLDER MEDICAL RECORD END ******************************/

The text below is the metadata.csv file.
Code: Select all
Name,DP_uniqueFileName,DP_PatientType,DP_MRN,DP_AdmDate,DP_Name,DP_IC,DP_CaseNo,DP_Episode
"DayPatient 53","/okm:root/scanner/MR/Batch 1/00546598 - 20180628.pdf","dp","00546598","20180628120000","CHIN YOONG (FOLDER)","420401065018","201800007217",""
"InPatient 54","/okm:root/scanner/MR/Batch 1/00554001 - 20180627.pdf","ip","00554001","20180627120000","MOHANA SUN DRAM A/L PARAMUCHUWA","800107065847","201800007119",""

I don't get why system is unable to locate the files. Please help.
 #46305  by jllort
 
I do not like scripting an specially the big scripts because you can not debug on it. I suggest convert as a crontab https://docs.openkm.com/kcenter/view/ok ... b-job.html

In the output of the script are shown errors caused by document already exists, this should not affect.

You should verify the property group code:
Check if it works the code below:
Code: Select all
String docPath = "/okm:root/scanner/MR/Batch 1/00546598 - 20180628.pdf";
OKMRepository.getInstance().hasNode(null,docPath);
if it works then check:
Code: Select all
Map mtdata = new HashMap();
mtdata.put(grpLabel1,row[column1]);
mtdata.put(grpLabel2,row[column2]);
mtdata.put(grpLabel3,row[column3]);
mtdata.put(grpLabel4,row[column4]);
mtdata.put(grpLabel5,row[column5]);
mtdata.put(grpLabel6,row[column6]);
mtdata.put(grpLabel7,row[column7]);
					
// ADD GROUP
OKMPropertyGroup.getInstance().addGroup(token, docPath, grpName);
// Set property value
OKMPropertyGroup.getInstance().setPropertiesSimple(token, docPath, grpName, mtdata);
 #46307  by jorseng
 
Dear Jllort,

I have checked as you have suggested, it returned true.

I will try to convert it into a crontab job, meanwhile i have tested with a newly created csv file.
It seems to be working fine with the newly create csv file.
Funny thing is, the wording and separator is exactly the same. Thus I am quite certain the issue lies with the csv file.

Is there any behavior of csv files which i should be aware of?
 #46317  by jllort
 
Did you download the CSV file from our documentation or you have created ?
If you have downloaded, tell me from where and I will review it.

About Us

OpenKM is part of the management software. A management software is a program that facilitates the accomplishment of administrative tasks. OpenKM is a document management system that allows you to manage business content and workflow in a more efficient way. Document managers guarantee data protection by establishing information security for business content.