Remote OpenKM repository incremental backup for windows

We tried to make OpenKM as intuitive as possible, but an advice is always welcome.
Forum rules
Please, before asking something see the documentation wiki or use the search feature of the forum. And remember we don't have a crystal ball or mental readers, so if you post about an issue tell us which OpenKM are you using and also the browser and operating system version. For more info read How to Report Bugs Effectively.
Post Reply
sunil
Junior Boarder
Junior Boarder
Posts: 20
Joined: Wed Sep 02, 2015 1:47 pm

Remote OpenKM repository incremental backup for windows

Post by sunil » Thu Feb 23, 2017 11:19 am

Hi there,

I want to take my OpenKM repository and database backup to remote server. I did some R&D and found this is possible. below is the link
https://www.openkm.com/wiki/index.php/Backup_with_rsync

But this is in not for windows. I need this type of script to take incremental and full backup of OpenKM.

Please help me with this.

Thanks
Sunil

jllort
Moderator
Moderator
Posts: 9594
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: Remote OpenKM repository incremental backup for windows

Post by jllort » Fri Feb 24, 2017 7:43 am

Really what you need is incremental backup software for windows. Really windows backup does something like it, but I do not like much the output what it does. I suggest at google do some query like "windows incremental filesystem backup".

Also you can take a look at software like "bacula" see http://blog.bacula.org/ or alternatives to bacula http://alternativeto.net/software/bacula/

From my personal opinion is more easy get incremental backup from linux tools rather windows scenario. Anyway is your application is under virtual machine, you can think in periodical snapshots or also incremental backup tools for virtual machines. All it depends on what is your scenario.

ofstab
Junior Boarder
Junior Boarder
Posts: 37
Joined: Tue Jan 05, 2016 9:21 am

Re: Remote OpenKM repository incremental backup for windows

Post by ofstab » Thu Mar 02, 2017 1:57 pm

I don't have a working script for a incremental backup working, but I am happy to share my differential backup batch file.
You need 7zip installed.

Code: Select all

:: @echo off
:: variables
set location="[backup location]"
set serverlocation=[root of server]
set openkmlocation=%serverlocation%\tomcat
set pathto7zip="c:\Program Files\7-Zip"
set currentlocation=%cd%
set folder="Full_Backup"

if exist "%location%\%folder%" goto differential

:full_backup
:: create backup location
mkdir "%location%\%folder%"
echo %time% %date% ### Making a Full Backup >> "%location%\%folder%\log.txt"
echo %time% %date% ### Going to stop_servers >> "%location%\%folder%\log.txt"
SET RETURN=Label_Full
goto stop_servers

:Label_Full
::OpenKM
cd %openkmlocation%
echo %time% %date% ### Backing up your OpenKM Property Groups... >> "%location%\%folder%\log.txt"
copy %openkmlocation%\PropertyGroups.xml "%location%\%folder%\PropertyGroups_backup.xml" >> "%location%\%folder%\log.txt"
echo %time% %date% ### Backing up your OpenKM SQL Data... >> "%location%\%folder%\log.txt"
"%serverlocation%\database\bin\mysqldump" -h localhost -u openkm -p[password] okmdb > %location%\"%folder%\openkm_db.sql"
echo %time% %date% ### Backing up your OpenKM Files... >> "%location%\%folder%\log.txt"
%pathto7zip%\7z.exe a -t7z -mx9 %location%\"%folder%\openkm_repository.7z" "%openkmlocation%\repository\" >> "%location%\%folder%\log.txt"

echo %time% %date% ### Going to restart_servers >> "%location%\%folder%\log.txt"
goto restart_servers

:differential
:: create differential backup location
set folder=%date:~0,2%_%date:~3,2%_%date:~6,4%
mkdir "%location%\%folder%"
echo %time% %date% ### Making a Differential Backup >> "%location%\%folder%\log.txt"
echo %time% %date% ### Going to stop_servers >> "%location%\%folder%\log.txt"
SET RETURN=Label_Diff
goto stop_servers

:Label_Diff
::OpenKM
cd %openkmlocation%
echo %time% %date% ### Backing up your OpenKM Property Groups... >> "%location%\%folder%\log.txt"
copy %openkmlocation%\PropertyGroups.xml "%location%\%folder%\PropertyGroups_backup.xml" >> "%location%\%folder%\log.txt"
echo %time% %date% ### Backing up your OpenKM SQL Data... >> "%location%\%folder%\log.txt"
"%serverlocation%\database\bin\mysqldump" -h localhost -u openkm -p[password] okmdb > %location%\"%folder%\openkm_db.sql"
echo %time% %date% ### Backing up your OpenKM Files... >> "%location%\%folder%\log.txt"
%pathto7zip%\7z.exe u -t7z -mx9 %location%\"Full_Backup\openkm_repository.7z" "%openkmlocation%\repository\" -u- -up0q3r2x2y2z0w2!%location%\"%folder%\openkm_repository.7z" >> "%location%\%folder%\log.txt"

echo %time% %date% ### Going to restart_servers >> "%location%\%folder%\log.txt"
goto restart_servers

:: Stop servers to prevent data from being changed during backup
:stop_servers
echo %time% %date% ### Stopping [Name of tomcat service] Server... >> "%location%\%folder%\log.txt"
net stop [Name of tomcat service]
if ERRORLEVEL 1 goto error
GOTO %RETURN%

:: Restart stopped servers
:restart_servers
echo %time% %date% ### Starting [Name of tomcat service] Server... >> "%location%\%folder%\log.txt"
net start [Name of tomcat service]
if ERRORLEVEL 1 goto error

::Tidy Up
echo %time% %date% ### Compressing SQL Databases... >> "%location%\%folder%\log.txt"
%pathto7zip%\7z.exe a -t7z -mx9 %location%\"%folder%\databases.7z" %location%\"%folder%\*.sql" >> "%location%\%folder%\log.txt"
if exist %location%\"%folder%\databases.7z" (
    	del %location%\"%folder%\*.sql" >> "%location%\%folder%\log.txt"
) else (
    echo %time% %date% ### Compressed Databases not found... >> "%location%\%folder%\log.txt"
	echo %time% %date% ### Not Deleting SQL files... >> "%location%\%folder%\log.txt"
		goto error
)
cd %currentlocation%
echo %time% %date% ### Backup Complete - Your Backup is located at %location%\"%folder%" >> "%location%\%folder%\log.txt"
echo %time% %date% ### Going to Finish >> "%location%\%folder%\log.txt"
goto finish

:error
echo %time% %date% ### There was some kind of problem. Check the log to find out where. >> "%location%\%folder%\log.txt"
echo %time% %date% ### Going to Finish >> "%location%\%folder%\log.txt"
goto finish

:finish
echo %time% %date% ### Finished.
The differential magic happens in the options "-up0q3r2x2y2z0w2!", see here for more info: http://www.wpctips.com/7zipdiff.html

Full disclosure, I have not fully tested the differential aspects of this, but I have restored from the full backup without any problems that I can see several times.

Use at your own risk!

jllort
Moderator
Moderator
Posts: 9594
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: Remote OpenKM repository incremental backup for windows

Post by jllort » Fri Mar 03, 2017 7:36 pm

Thanks for sharing this script, I think might be useful to other users.

Could you share with us the number of files, and repository size of your installation. Also will be interesting know how much time it needs the process to be completed.

Are you using default database ( embeded hypersonic ) ? because in the script I'm identied the location of the database dump.

ofstab
Junior Boarder
Junior Boarder
Posts: 37
Joined: Tue Jan 05, 2016 9:21 am

Re: Remote OpenKM repository incremental backup for windows

Post by ofstab » Mon Mar 06, 2017 9:41 am

No problem.

The result from the last full backup was:

Code: Select all

 1:02:27.22 03/02/2017 ### Backing up your OpenKM Files... 

7-Zip [64] 15.14 : Copyright (c) 1999-2015 Igor Pavlov : 2015-12-31

Scanning the drive:
131694 folders, 20173 files, 14536712955 bytes (14 GiB)

Creating archive: [location]\Full_Backup\openkm_repository.7z

Items to compress: 151867


Files read from disk: 20139
Archive size: 10766276226 bytes (11 GiB)
Everything is Ok
 3:01:58.24 03/02/2017 ### 
Performance is circa 5GiB/hour for the maximum (-mx9), or ultra level of 7zip encryption. I would guess that the majority of the 20k files we have are scanned PDFs, so they are fairly well compressed in advance. I think the compression routine would be much faster if I switched to a lower level of compression (-mx3 or -mx5), without gaining a whole lot of archive size.

The result of the last differential:

Code: Select all

 1:02:37.17 04/03/2017 ### Backing up your OpenKM Files... 

7-Zip [64] 15.14 : Copyright (c) 1999-2015 Igor Pavlov : 2015-12-31

Open archive: [...]\Full_Backup\openkm_repository.7z
--
Path = [...]\Full_Backup\openkm_repository.7z
Type = 7z
Physical Size = 10766276226
Headers Size = 1157713
Method = LZMA2:26
Solid = +
Blocks = 4

Scanning the drive:
138013 folders, 22810 files, 16117143810 bytes (16 GiB)

Creating archive: [...]\04_03_2017\openkm_repository.7z

Items to compress: 10605


Files read from disk: 2680
Archive size: 1037489462 bytes (990 MiB)
Everything is Ok
 1:14:00.73 04/03/2017 ###
You can see we have added or changed 2680 files in the last month. This added another couple of GiB to the total repo size. However, running a differential compression, even at max (-mx9) only takes 12 minutes. (This is still bang on 5GiB/hour performance).

We are using MySQL installed to [base install location}\database. The sql dump file is about 500Mb, but compresses down (with some much smaller and unconnected sql dumps) to less than 30Mb.

Hope this helps.

jllort
Moderator
Moderator
Posts: 9594
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: Remote OpenKM repository incremental backup for windows

Post by jllort » Wed Mar 08, 2017 7:41 pm

Really helpful information I will consider add into our documentation as a sample windows backup with your performance feedback.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest