Open Source Document Management System | OpenKM

PostPosted:**Thu Feb 23, 2017 11:19 am**

Hi there,

I want to take my OpenKM repository and database backup to remote server. I did some R&D and found this is possible. below is the link
https://www.openkm.com/wiki/index.php/Backup_with_rsync

But this is in not for windows. I need this type of script to take incremental and full backup of OpenKM.

Please help me with this.

Thanks
Sunil

PostPosted:**Fri Feb 24, 2017 7:43 am**

Really what you need is incremental backup software for windows. Really windows backup does something like it, but I do not like much the output what it does. I suggest at google do some query like "windows incremental filesystem backup".

Also you can take a look at software like "bacula" see http://blog.bacula.org/ or alternatives to bacula http://alternativeto.net/software/bacula/

From my personal opinion is more easy get incremental backup from linux tools rather windows scenario. Anyway is your application is under virtual machine, you can think in periodical snapshots or also incremental backup tools for virtual machines. All it depends on what is your scenario.

PostPosted:**Thu Mar 02, 2017 1:57 pm**

I don't have a working script for a incremental backup working, but I am happy to share my differential backup batch file.
You need 7zip installed.

Code: Select all

:: @echo off
:: variables
set location="[backup location]"
set serverlocation=[root of server]
set openkmlocation=%serverlocation%\tomcat
set pathto7zip="c:\Program Files\7-Zip"
set currentlocation=%cd%
set folder="Full_Backup"

if exist "%location%\%folder%" goto differential

:full_backup
:: create backup location
mkdir "%location%\%folder%"
echo %time% %date% ### Making a Full Backup >> "%location%\%folder%\log.txt"
echo %time% %date% ### Going to stop_servers >> "%location%\%folder%\log.txt"
SET RETURN=Label_Full
goto stop_servers

:Label_Full
::OpenKM
cd %openkmlocation%
echo %time% %date% ### Backing up your OpenKM Property Groups... >> "%location%\%folder%\log.txt"
copy %openkmlocation%\PropertyGroups.xml "%location%\%folder%\PropertyGroups_backup.xml" >> "%location%\%folder%\log.txt"
echo %time% %date% ### Backing up your OpenKM SQL Data... >> "%location%\%folder%\log.txt"
"%serverlocation%\database\bin\mysqldump" -h localhost -u openkm -p[password] okmdb > %location%\"%folder%\openkm_db.sql"
echo %time% %date% ### Backing up your OpenKM Files... >> "%location%\%folder%\log.txt"
%pathto7zip%\7z.exe a -t7z -mx9 %location%\"%folder%\openkm_repository.7z" "%openkmlocation%\repository\" >> "%location%\%folder%\log.txt"

echo %time% %date% ### Going to restart_servers >> "%location%\%folder%\log.txt"
goto restart_servers

:differential
:: create differential backup location
set folder=%date:~0,2%_%date:~3,2%_%date:~6,4%
mkdir "%location%\%folder%"
echo %time% %date% ### Making a Differential Backup >> "%location%\%folder%\log.txt"
echo %time% %date% ### Going to stop_servers >> "%location%\%folder%\log.txt"
SET RETURN=Label_Diff
goto stop_servers

:Label_Diff
::OpenKM
cd %openkmlocation%
echo %time% %date% ### Backing up your OpenKM Property Groups... >> "%location%\%folder%\log.txt"
copy %openkmlocation%\PropertyGroups.xml "%location%\%folder%\PropertyGroups_backup.xml" >> "%location%\%folder%\log.txt"
echo %time% %date% ### Backing up your OpenKM SQL Data... >> "%location%\%folder%\log.txt"
"%serverlocation%\database\bin\mysqldump" -h localhost -u openkm -p[password] okmdb > %location%\"%folder%\openkm_db.sql"
echo %time% %date% ### Backing up your OpenKM Files... >> "%location%\%folder%\log.txt"
%pathto7zip%\7z.exe u -t7z -mx9 %location%\"Full_Backup\openkm_repository.7z" "%openkmlocation%\repository\" -u- -up0q3r2x2y2z0w2!%location%\"%folder%\openkm_repository.7z" >> "%location%\%folder%\log.txt"

echo %time% %date% ### Going to restart_servers >> "%location%\%folder%\log.txt"
goto restart_servers

:: Stop servers to prevent data from being changed during backup
:stop_servers
echo %time% %date% ### Stopping [Name of tomcat service] Server... >> "%location%\%folder%\log.txt"
net stop [Name of tomcat service]
if ERRORLEVEL 1 goto error
GOTO %RETURN%

:: Restart stopped servers
:restart_servers
echo %time% %date% ### Starting [Name of tomcat service] Server... >> "%location%\%folder%\log.txt"
net start [Name of tomcat service]
if ERRORLEVEL 1 goto error

::Tidy Up
echo %time% %date% ### Compressing SQL Databases... >> "%location%\%folder%\log.txt"
%pathto7zip%\7z.exe a -t7z -mx9 %location%\"%folder%\databases.7z" %location%\"%folder%\*.sql" >> "%location%\%folder%\log.txt"
if exist %location%\"%folder%\databases.7z" (
    	del %location%\"%folder%\*.sql" >> "%location%\%folder%\log.txt"
) else (
    echo %time% %date% ### Compressed Databases not found... >> "%location%\%folder%\log.txt"
	echo %time% %date% ### Not Deleting SQL files... >> "%location%\%folder%\log.txt"
		goto error
)
cd %currentlocation%
echo %time% %date% ### Backup Complete - Your Backup is located at %location%\"%folder%" >> "%location%\%folder%\log.txt"
echo %time% %date% ### Going to Finish >> "%location%\%folder%\log.txt"
goto finish

:error
echo %time% %date% ### There was some kind of problem. Check the log to find out where. >> "%location%\%folder%\log.txt"
echo %time% %date% ### Going to Finish >> "%location%\%folder%\log.txt"
goto finish

:finish
echo %time% %date% ### Finished.

The differential magic happens in the options "-up0q3r2x2y2z0w2!", see here for more info: http://www.wpctips.com/7zipdiff.html

Full disclosure, I have not fully tested the differential aspects of this, but I have restored from the full backup without any problems that I can see several times.

Use at your own risk!

PostPosted:**Fri Mar 03, 2017 7:36 pm**

Thanks for sharing this script, I think might be useful to other users.

Could you share with us the number of files, and repository size of your installation. Also will be interesting know how much time it needs the process to be completed.

Are you using default database ( embeded hypersonic ) ? because in the script I'm identied the location of the database dump.

PostPosted:**Mon Mar 06, 2017 9:41 am**

No problem.

The result from the last full backup was:

Code: Select all

 1:02:27.22 03/02/2017 ### Backing up your OpenKM Files... 

7-Zip [64] 15.14 : Copyright (c) 1999-2015 Igor Pavlov : 2015-12-31

Scanning the drive:
131694 folders, 20173 files, 14536712955 bytes (14 GiB)

Creating archive: [location]\Full_Backup\openkm_repository.7z

Items to compress: 151867


Files read from disk: 20139
Archive size: 10766276226 bytes (11 GiB)
Everything is Ok
 3:01:58.24 03/02/2017 ###

Performance is circa 5GiB/hour for the maximum (-mx9), or ultra level of 7zip encryption. I would guess that the majority of the 20k files we have are scanned PDFs, so they are fairly well compressed in advance. I think the compression routine would be much faster if I switched to a lower level of compression (-mx3 or -mx5), without gaining a whole lot of archive size.

The result of the last differential:

Code: Select all

 1:02:37.17 04/03/2017 ### Backing up your OpenKM Files... 

7-Zip [64] 15.14 : Copyright (c) 1999-2015 Igor Pavlov : 2015-12-31

Open archive: [...]\Full_Backup\openkm_repository.7z
--
Path = [...]\Full_Backup\openkm_repository.7z
Type = 7z
Physical Size = 10766276226
Headers Size = 1157713
Method = LZMA2:26
Solid = +
Blocks = 4

Scanning the drive:
138013 folders, 22810 files, 16117143810 bytes (16 GiB)

Creating archive: [...]\04_03_2017\openkm_repository.7z

Items to compress: 10605


Files read from disk: 2680
Archive size: 1037489462 bytes (990 MiB)
Everything is Ok
 1:14:00.73 04/03/2017 ###

You can see we have added or changed 2680 files in the last month. This added another couple of GiB to the total repo size. However, running a differential compression, even at max (-mx9) only takes 12 minutes. (This is still bang on 5GiB/hour performance).

We are using MySQL installed to [base install location}\database. The sql dump file is about 500Mb, but compresses down (with some much smaller and unconnected sql dumps) to less than 30Mb.

Hope this helps.

PostPosted:**Wed Mar 08, 2017 7:41 pm**

Really helpful information I will consider add into our documentation as a sample windows backup with your performance feedback.

PostPosted:**Mon Dec 06, 2021 11:33 am**

I thought it might be worth an update on the repository size, just for info. i am still using the same script commands to back it up nearly 5 years later.

Code: Select all

 1:25:24.87 04/12/2021 ### Backing up your OpenKM Files... 

7-Zip [64] 15.14 : Copyright (c) 1999-2015 Igor Pavlov : 2015-12-31

Scanning the drive:
746479 folders, 313471 files, 148368463287 bytes (139 GiB)

Creating archive:[...]\Full_Backup\openkm_repository.7z

Items to compress: 1059950


Files read from disk: 313390
Archive size: 112946963136 bytes (106 GiB)
Everything is Ok
10:06:18.92 04/12/2021 ###

We've gained data at around 20GiB/year. It now takes nearly 9 hours to back that all up.

Still not fully tested the differential restore, but I have restored the Full back up a couple of times, and it works fine.

Open Source Document Management System | OpenKM

Remote OpenKM repository incremental backup for windows

Remote OpenKM repository incremental backup for windows

Re: Remote OpenKM repository incremental backup for windows

Re: Remote OpenKM repository incremental backup for windows

Re: Remote OpenKM repository incremental backup for windows

Re: Remote OpenKM repository incremental backup for windows

Re: Remote OpenKM repository incremental backup for windows

Re: Remote OpenKM repository incremental backup for windows