I am having a reoccurring issue where OpenKM isn't stopping as it should during backups and this morning I found the site was down. A simple service tomcat start/stop would not reset the instance. I had to "kill -9 PID" to kill the java PIDs holding the ports used for OpenKM.
I am not sure where to look for clues, but here is what I am seeing in the syslog at the time of backup...
The start up if that in the syslog...
backup.sh called by cron...
I am not sure where to look for clues, but here is what I am seeing in the syslog at the time of backup...
Code: Select all
When I observed that the site was down this morning I tried to stop the service and this was in the syslog for that...Apr 11 00:00:01 RDDocMan CRON[1891]: (root) CMD (/root/backup.sh | tee /root/logs/backup.$(date +%Y.%m.%d_%H.%M.%S).log >/dev/null 2>&1)
Apr 11 00:00:02 RDDocMan systemd[1]: Started Session 4 of user openkm.
Code: Select all
I have learned to kill the PIDs holding this java ports used by OpenKM, so at this point I searched for the PIDs using a 'netstat -tupln' and killed the java PID holding two of the ports. Followed by a 'service tomcat start'Apr 11 08:28:00 RDDocMan systemd[1]: Stopping LSB: Start and stop Apache Tomcat...
Apr 11 08:28:00 RDDocMan systemd[1]: Started Session c6 of user openkm.
Apr 11 08:28:00 RDDocMan tomcat[15115]: Stopping TomcatPID file found but no matching process was found. Stop aborted.
Apr 11 08:28:05 RDDocMan AptDaemon.Worker: INFO: Finished transaction /org/debian/apt/transaction/98792cb976224b8babab9f99305a8cec
Apr 11 08:28:05 RDDocMan org.debian.apt[828]: 08:28:05 AptDaemon.Worker [INFO]: Finished transaction /org/debian/apt/transaction/98792cb976224b8babab9$
Apr 11 08:32:46 RDDocMan tomcat[15115]: ..........................................................
Apr 11 08:32:46 RDDocMan systemd[1]: Stopped LSB: Start and stop Apache Tomcat.
The start up if that in the syslog...
Code: Select all
For reference the backup log...Apr 11 08:33:05 RDDocMan systemd[1]: Starting LSB: Start and stop Apache Tomcat...
Apr 11 08:33:05 RDDocMan systemd[1]: Started Session c7 of user openkm.
Apr 11 08:33:05 RDDocMan tomcat[16129]: Starting Tomcat.
Apr 11 08:33:05 RDDocMan systemd[1]: Started LSB: Start and stop Apache Tomcat.
Apr 11 08:33:05 RDDocMan tomcat[16129]: Tomcat started.
Code: Select all
Here are my backup scripts...### BEGIN: 04/11/2018 12:00:01 AM ###
Stopping TomcatPID file found but no matching process was found. Stop aborted.
......................................................................................................................................................$
* Backuping MySQL data from okmdb...
-------------------------------------
Number of files: 69,246 (reg: 24,851, dir: 44,395)
Number of created files: 4,010 (reg: 3,623, dir: 387)
Number of deleted files: 231 (reg: 168, dir: 63)
Number of regular files transferred: 3,632
Total file size: 8.61G bytes
Total transferred file size: 130.55M bytes
Literal data: 130.55M bytes
Matched data: 0 bytes
File list size: 196.59K
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 47.52M
Total bytes received: 145.08K
sent 47.52M bytes received 145.08K bytes 274.73K bytes/sec
total size is 8.61G speedup is 180.67
Clean Tomcat temporal files.
Starting Tomcat.
### END: 04/11/2018 08:35:59 AM ###
Calling the tomcat_backup.sh script
Existing PID file found during start.
/home/openkm/ABTopenkm/
/home/openkm/ABTopenkm/db/
/home/openkm/ABTopenkm/db/mysql_okmdb.sql
Tomcat appears to still be running with PID 16145. Start aborted.
/home/openkm/ABTopenkm/home/
/home/openkm/ABTopenkm/home/openkm/
/home/openkm/ABTopenkm/home/openkm/tomcat-7.0.61/
/home/openkm/ABTopenkm/home/openkm/tomcat-7.0.61/OPENKM-README.txt
/home/openkm/ABTopenkm/home/openkm/tomcat-7.0.61/webapps/
/home/openkm/ABTopenkm/home/openkm/tomcat-7.0.61/webapps/ROOT/
backup.sh called by cron...
Code: Select all
tomcat_backup.sh...
#!/bin/bash
#
## BEGIN CONFIG ##
HOST=$(uname -n)
DATABASE_PASS="abt511"
OPENKM_DB="okmdb"
OPENKM_HOME="/home/openkm"
TOMCAT_HOME="$OPENKM_HOME/tomcat-7.0.61"
DATABASE_EXP="$OPENKM_HOME/ABTopenkm/db"
BACKUP_DIR="/home/openkm/ABTopenkm"
RSYNC_OPTS="-apzhR --stats --delete --exclude=*~ --delete-excluded"
## END CONFIG ##
# Check root user
if [ $(id -u) != 0 ]; then echo "You should run this script as root"; exit; fi
# Delete older local database backup
echo -e "### BEGIN: $(date +"%x %X") ###\n"
rm -rf $DATABASE_EXP
mkdir -p $DATABASE_EXP
# Mount disk
#if mount | grep "$BACKUP_DIR type" > /dev/null; then
# echo "$BACKUP_DIR already mounted";
#else
# mount "$BACKUP_DIR";
#
# if mount | grep "$BACKUP_DIR type" > /dev/null; then
# echo "$BACKUP_DIR mounted";
# else
# echo "$BACKUP_DIR error mounting";
# exit -1;
# fi
#fi
#
# Stop Tomcat
/etc/init.d/tomcat stop
#
# Backup database
if [ -n "$DATABASE_PASS" ]; then
echo "* Backuping MySQL data from $OPENKM_DB..."
mysqldump -h localhost -u root -p$DATABASE_PASS $OPENKM_DB > $DATABASE_EXP/mysql_$OPENKM_DB.sql
echo "-------------------------------------";
fi
#
# Create backup
rsync $RSYNC_OPTS $TOMCAT_HOME $BACKUP_DIR/
#
# Clean logs
echo "Clean Tomcat temporal files."
rm -rf $TOMCAT_HOME/logs/*
rm -rf $TOMCAT_HOME/temp/*
rm -rf $TOMCAT_HOME/work/Catalina/localhost
#
# Start Tomcat
/etc/init.d/tomcat start
echo -e "\n### END: $(date +"%x %X") ###"
#
# Umount disk
#sync
#umount "$BACKUP_DIR"
#
# run tomcat_backup.sh
echo "Calling the tomcat_backup.sh script"
sh /home/openkm/tomcat-7.0.61/tomcat_backup.sh
echo "tomcat_backup.sh complete"
#
Code: Select all
#!/bin/bash
#
#
# tar up tomcat application folder
#
cd /home/openkm/
#tar -czvf /home/openkm/DatabaseArchives/tomcat-$(date +%Y%m%d).tar.gz tomcat-7.0.61
tar -czvf /home/openkm/DatabaseArchives/ABTopenkmBackup-$(date +%Y%m%d).tar.gz /home/openkm/ABTopenkm
#
# mount USB drive
#
sudo mount /dev/sdb1 /mnt/backup
echo "backup mounted"
sudo cp /home/openkm/DatabaseArchives/ABTopenkmBackup-$(date +%Y%m%d).tar.gz /mnt/backup
echo "move sql dump and app folder tar"
find /mnt/backup -mtime +20 -mtime -100 -type f -exec rm -f {} \;
echo "purge old files on backup drive"
sudo sync
sudo umount /mnt/backup
echo "unmount backup"
#
# mount abt server
#
sudo mount -t cifs -o username=username,password=******** //192.168.255.3/"ABT Server"/"Project Management"/DatabaseBackups /mnt/DatabaseBackups
echo "mount ABT server"
sudo rm -rf /mnt/DatabaseBackups/*
echo "purge old file"
sudo cp /home/openkm/DatabaseArchives/ABTopenkmBackup-$(date +%Y%m%d).tar.gz /mnt/DatabaseBackups
echo "copy database tar"
sudo sync
sudo umount /mnt/DatabaseBackups
echo "unmount ABT server"
#
# delete backups older than 5 days
find /home/openkm/DatabaseArchives -mtime +5 -mtime -100 -type f -exec rm -f {} \;
echo "purge old files locally"
#
#