I've deployed Ubuntu 18.04.1 LTS in a virtual machine (VMWare Workstation Pro12, Windows 10 Pro).
The office manager has quit again this morning and I tried to restart and it hung with these results.
I aborted the restart and deleted the catalina PID and then restarted but found this in the startup log
Code: Select all2019-01-22 08:12:04,447 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Start OpenOffice manager ***
2019-01-22 08:12:04,454 [localhost-startStop-1] INFO com.openkm.util.DocConverter - *** Build Office Manager ***
2019-01-22 08:12:04,455 [localhost-startStop-1] INFO com.openkm.util.DocConverter - system.openoffice.path=/usr/lib/libreoffice
2019-01-22 08:12:04,455 [localhost-startStop-1] INFO com.openkm.util.DocConverter - system.openoffice.tasks=200
2019-01-22 08:12:04,455 [localhost-startStop-1] INFO com.openkm.util.DocConverter - system.openoffice.port=2002
22-Jan-2019 08:12:04.535 INFO [localhost-startStop-1] org.artofsolving.jodconverter.office.ProcessPoolOfficeManager.<init> ProcessManager implementation is SigarProcessManager
22-Jan-2019 08:12:04.583 INFO [OfficeProcessThread-0] org.artofsolving.jodconverter.office.OfficeProcess.start starting process with acceptString 'socket,host=127.0.0.1,port=2002,tcpNoDelay=1' and profileDir '/home/openkm/tomcat-8.5.24/temp/.jodconverter_socket_host-127.0.0.1_port-2002'
22-Jan-2019 08:12:04.615 INFO [OfficeProcessThread-0] org.artofsolving.jodconverter.office.OfficeProcess.start started process; pid = 9510
22-Jan-2019 08:12:07.701 WARNING [OfficeProcessThread-0] org.artofsolving.jodconverter.office.ManagedOfficeProcess$6.attempt office process died with exit code 81; restarting it
22-Jan-2019 08:12:07.719 INFO [OfficeProcessThread-0] org.artofsolving.jodconverter.office.OfficeProcess.start starting process with acceptString 'socket,host=127.0.0.1,port=2002,tcpNoDelay=1' and profileDir '/home/openkm/tomcat-8.5.24/temp/.jodconverter_socket_host-127.0.0.1_port-2002'
22-Jan-2019 08:12:07.735 INFO [OfficeProcessThread-0] org.artofsolving.jodconverter.office.OfficeProcess.start started process; pid = 9518
22-Jan-2019 08:12:08.359 INFO [OfficeProcessThread-0] org.artofsolving.jodconverter.office.OfficeConnection.connect connected: 'socket,host=127.0.0.1,port=2002,tcpNoDelay=1'
2019-01-22 08:12:08,363 [localhost-startStop-1] INFO c.o.extension.core.ExtensionManager - Initialize and load plugins...
22-Jan-2019 08:12:08.422 WARNING [localhost-startStop-1] net.xeoh.plugins.base.impl.classpath.loader.FileLoader.loadFrom Supplied path does not exist. Unable to add plugins from there.
2019-01-22 08:12:08,423 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Execute start script ***
2019-01-22 08:12:08,425 [localhost-startStop-1] WARN com.openkm.util.ExecutionUtils - Unable to read script: /home/openkm/tomcat-8.5.24/start.bsh
2019-01-22 08:12:08,425 [localhost-startStop-1] WARN com.openkm.util.ExecutionUtils - Unable to read jar: /home/openkm/tomcat-8.5.24/start.jar
2019-01-22 08:12:08,426 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Execute start SQL ***
2019-01-22 08:12:08,426 [localhost-startStop-1] WARN c.o.servlet.RepositoryStartupServlet - Unable to read sql: /home/openkm/tomcat-8.5.24/start.sql
2019-01-22 08:12:08,431 [localhost-startStop-1] WARN o.a.c.o.s.i.a.CmisAtomPubServlet - CMIS version is not defined! Setting it to CMIS 1.0.
22-Jan-2019 08:12:08.532 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/openkm/tomcat-8.5.24/webapps/OpenKM.war] has finished in [186,540] ms
22-Jan-2019 08:12:08.534 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/home/openkm/tomcat-8.5.24/webapps/ROOT]
22-Jan-2019 08:12:08.537 WARNING [localhost-startStop-1] org.apache.catalina.startup.SetContextPropertiesRule.begin [SetContextPropertiesRule]{Context} Setting property 'antiJARLocking' to 'true' did not find a matching property.
22-Jan-2019 08:12:09.083 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
22-Jan-2019 08:12:09.085 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/home/openkm/tomcat-8.5.24/webapps/ROOT] has finished in [551] ms
22-Jan-2019 08:12:09.116 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-127.0.0.1-8009"]
22-Jan-2019 08:12:09.159 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 187313 ms
22-Jan-2019 08:12:09.160 SEVERE [main] org.apache.catalina.core.StandardServer.await StandardServer.await: create[localhost:8005]:
java.net.BindException: Address already in use (Bind failed)
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:375)
at java.net.ServerSocket.<init>(ServerSocket.java:237)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:440)
at org.apache.catalina.startup.Catalina.await(Catalina.java:758)
at org.apache.catalina.startup.Catalina.start(Catalina.java:704)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:353)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:493)
22-Jan-2019 08:12:09.161 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-0.0.0.0-8080"]
22-Jan-2019 08:12:09.161 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-nio-127.0.0.1-8009"]
22-Jan-2019 08:12:09.214 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
2019-01-22 08:12:09,271 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Shutting down OpenOffice manager ***
22-Jan-2019 08:12:09.271 INFO [localhost-startStop-1] org.artofsolving.jodconverter.office.ProcessPoolOfficeManager.stop stopping
22-Jan-2019 08:12:09.377 INFO [MessageDispatcher] org.artofsolving.jodconverter.office.OfficeConnection$1.disposing disconnected: 'socket,host=127.0.0.1,port=2002,tcpNoDelay=1'
22-Jan-2019 08:12:09.570 INFO [OfficeProcessThread-0] org.artofsolving.jodconverter.office.ManagedOfficeProcess.doEnsureProcessExited process exited with code 0
22-Jan-2019 08:12:09.628 INFO [localhost-startStop-1] org.artofsolving.jodconverter.office.ProcessPoolOfficeManager.stop stopped
2019-01-22 08:12:09,628 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Shutting down UI Notification... ***
2019-01-22 08:12:09,629 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Shutting down cron... ***
2019-01-22 08:12:09,629 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Shutting down update info... ***
2019-01-22 08:12:09,630 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Shutting pending task executor... ***
2019-01-22 08:12:09,634 [localhost-startStop-1] INFO c.o.u.pendtask.PendingTaskExecutor - ### All threads shutdown requested ###
2019-01-22 08:12:09,635 [localhost-startStop-1] INFO c.o.u.pendtask.PendingTaskExecutor - ### All threads have finished ###
2019-01-22 08:12:09,636 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Shutting down repository... ***
2019-01-22 08:12:09,636 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Cache serialization ***
2019-01-22 08:12:09,667 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Repository shutted down ***
2019-01-22 08:12:09,667 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Execute stop script ***
2019-01-22 08:12:09,668 [localhost-startStop-1] WARN com.openkm.util.ExecutionUtils - Unable to read script: /home/openkm/tomcat-8.5.24/stop.bsh
2019-01-22 08:12:09,668 [localhost-startStop-1] WARN com.openkm.util.ExecutionUtils - Unable to read jar: /home/openkm/tomcat-8.5.24/stop.jar
2019-01-22 08:12:09,668 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Execute stop SQL ***
2019-01-22 08:12:09,668 [localhost-startStop-1] WARN c.o.servlet.RepositoryStartupServlet - Unable to read sql: /home/openkm/tomcat-8.5.24/stop.sql
2019-01-22 08:12:09,668 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Shutting down workflow engine... ***
22-Jan-2019 08:12:09.669 INFO [JbpmJobExecutor:127.0.1.1:1] org.jbpm.job.executor.JobExecutorThread.run inactive job executor thread 'JbpmJobExecutor:127.0.1.1:1' got interrupted
22-Jan-2019 08:12:09.670 INFO [JbpmJobExecutor:127.0.1.1:1] org.jbpm.job.executor.JobExecutorThread.run JbpmJobExecutor:127.0.1.1:1 leaves cyberspace
2019-01-22 08:12:09,716 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Shutting down cache manager... ***
2019-01-22 08:12:09,740 [localhost-startStop-1] INFO c.o.servlet.RepositoryStartupServlet - *** Hibernate shutdown ***
22-Jan-2019 08:12:09.742 INFO [localhost-startStop-1] org.springframework.context.support.AbstractApplicationContext.doClose Closing Root WebApplicationContext: startup date [Tue Jan 22 08:11:34 EST 2019]; root of context hierarchy
22-Jan-2019 08:12:09.745 INFO [localhost-startStop-1] org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@56465cf2: defining beans [dbAuthModule,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,org.springframework.context.annotation.internalPersistenceAnnotationProcessor,authService,bookmarkService,documentService,folderService,mailService,noteService,notificationService,propertyGroupService,propertyService,repositoryService,searchService,dashboardService,workflowService,testService,restAuth,restDocument,restFolder,restMail,restNote,restPropertyGroup,restSearch,restRepository,restProperty,restTest,cmisNavigationService,cmisPolicyService,cmisDiscoveryService,cmisMultiFilingService,cmisRepositoryService,cmisRelationshipService,cmisVersioningService,cmisObjectService,cmisAclService,CmisLifecycleBean,CmisServiceFactory,org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource#0,org.springframework.security.access.vote.AffirmativeBased#0,org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor#0,org.springframework.security.methodSecurityMetadataSourceAdvisor,org.springframework.aop.config.internalAutoProxyCreator,roleVoter,org.springframework.security.filterChains,org.springframework.security.filterChainProxy,org.springframework.security.web.PortMapperImpl#0,org.springframework.security.web.PortResolverImpl#0,org.springframework.security.config.authentication.AuthenticationManagerFactoryBean#0,org.springframework.security.authentication.ProviderManager#0,org.springframework.security.web.context.NullSecurityContextRepository#0,org.springframework.security.web.savedrequest.NullRequestCache#0,org.springframework.security.access.vote.AffirmativeBased#1,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#0,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#0,org.springframework.security.authentication.AnonymousAuthenticationProvider#0,org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint#0,org.springframework.security.userDetailsServiceFactory,org.springframework.security.web.DefaultSecurityFilterChain#0,org.springframework.security.web.PortMapperImpl#1,org.springframework.security.web.PortResolverImpl#1,org.springframework.security.config.authentication.AuthenticationManagerFactoryBean#1,org.springframework.security.authentication.ProviderManager#1,org.springframework.security.web.context.NullSecurityContextRepository#1,org.springframework.security.web.savedrequest.NullRequestCache#1,org.springframework.security.access.vote.AffirmativeBased#2,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#1,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#1,org.springframework.security.authentication.AnonymousAuthenticationProvider#1,org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint#1,org.springframework.security.web.DefaultSecurityFilterChain#1,org.springframework.security.web.PortMapperImpl#2,org.springframework.security.web.PortResolverImpl#2,org.springframework.security.config.authentication.AuthenticationManagerFactoryBean#2,org.springframework.security.authentication.ProviderManager#2,org.springframework.security.web.context.NullSecurityContextRepository#2,org.springframework.security.web.savedrequest.NullRequestCache#2,org.springframework.security.access.vote.AffirmativeBased#3,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#2,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#2,org.springframework.security.authentication.AnonymousAuthenticationProvider#2,org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint#2,org.springframework.security.web.DefaultSecurityFilterChain#2,org.springframework.security.web.PortMapperImpl#3,org.springframework.security.web.PortResolverImpl#3,org.springframework.security.config.authentication.AuthenticationManagerFactoryBean#3,org.springframework.security.authentication.ProviderManager#3,org.springframework.security.web.context.NullSecurityContextRepository#3,org.springframework.security.web.savedrequest.NullRequestCache#3,org.springframework.security.access.vote.AffirmativeBased#4,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#3,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#3,org.springframework.security.authentication.AnonymousAuthenticationProvider#3,org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint#3,org.springframework.security.web.DefaultSecurityFilterChain#3,org.springframework.security.web.PortMapperImpl#4,org.springframework.security.web.PortResolverImpl#4,org.springframework.security.config.authentication.AuthenticationManagerFactoryBean#4,org.springframework.security.authentication.ProviderManager#4,org.springframework.security.web.context.NullSecurityContextRepository#4,org.springframework.security.web.savedrequest.NullRequestCache#4,org.springframework.security.access.vote.AffirmativeBased#5,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#4,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#4,org.springframework.security.authentication.AnonymousAuthenticationProvider#4,org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint#4,org.springframework.security.web.DefaultSecurityFilterChain#4,org.springframework.security.web.PortMapperImpl#5,org.springframework.security.web.PortResolverImpl#5,org.springframework.security.config.authentication.AuthenticationManagerFactoryBean#5,org.springframework.security.authentication.ProviderManager#5,org.springframework.security.web.context.NullSecurityContextRepository#5,org.springframework.security.web.savedrequest.NullRequestCache#5,org.springframework.security.access.vote.AffirmativeBased#6,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#5,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#5,org.springframework.security.authentication.AnonymousAuthenticationProvider#5,org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint#5,org.springframework.security.web.DefaultSecurityFilterChain#5,org.springframework.security.web.PortMapperImpl#6,org.springframework.security.web.PortResolverImpl#6,org.springframework.security.config.authentication.AuthenticationManagerFactoryBean#6,org.springframework.security.authentication.ProviderManager#6,org.springframework.security.web.context.NullSecurityContextRepository#6,org.springframework.security.web.savedrequest.NullRequestCache#6,org.springframework.security.access.vote.AffirmativeBased#7,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#6,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#6,org.springframework.security.authentication.AnonymousAuthenticationProvider#6,org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint#6,org.springframework.security.web.DefaultSecurityFilterChain#6,org.springframework.security.web.PortMapperImpl#7,org.springframework.security.web.PortResolverImpl#7,org.springframework.security.config.authentication.AuthenticationManagerFactoryBean#7,org.springframework.security.authentication.ProviderManager#7,org.springframework.security.web.context.NullSecurityContextRepository#7,org.springframework.security.web.savedrequest.NullRequestCache#7,org.springframework.security.access.vote.AffirmativeBased#8,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#7,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#7,org.springframework.security.authentication.AnonymousAuthenticationProvider#7,org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint#7,org.springframework.security.web.DefaultSecurityFilterChain#7,org.springframework.security.web.PortMapperImpl#8,org.springframework.security.web.PortResolverImpl#8,org.springframework.security.config.authentication.AuthenticationManagerFactoryBean#8,org.springframework.security.authentication.ProviderManager#8,org.springframework.security.web.context.NullSecurityContextRepository#8,org.springframework.security.web.savedrequest.NullRequestCache#8,org.springframework.security.access.vote.AffirmativeBased#9,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#8,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#8,org.springframework.security.authentication.AnonymousAuthenticationProvider#8,org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint#8,org.springframework.security.web.DefaultSecurityFilterChain#8,org.springframework.security.web.PortMapperImpl#9,org.springframework.security.web.PortResolverImpl#9,org.springframework.security.config.authentication.AuthenticationManagerFactoryBean#9,org.springframework.security.authentication.ProviderManager#9,org.springframework.security.web.context.HttpSessionSecurityContextRepository#0,org.springframework.security.web.authentication.session.CompositeSessionAuthenticationStrategy#0,org.springframework.security.web.savedrequest.HttpSessionRequestCache#0,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#9,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#9,org.springframework.security.authentication.AnonymousAuthenticationProvider#9,org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter#0,org.springframework.security.web.DefaultSecurityFilterChain#9,accessDecisionManager,loggerListener,dataSource,org.springframework.security.provisioning.JdbcUserDetailsManager#0,org.springframework.security.authentication.dao.DaoAuthenticationProvider#0,org.springframework.security.authentication.DefaultAuthenticationEventPublisher#0,org.springframework.security.authenticationManager,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor]; root of factory hierarchy
22-Jan-2019 08:12:09.790 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [OpenKM] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
22-Jan-2019 08:12:09.791 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [OpenKM] appears to have started a thread named [OfficeProcessThread-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
22-Jan-2019 08:12:09.791 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [OpenKM] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@40c00383]) and a value of type [org.springframework.security.core.context.SecurityContextImpl] (value [org.springframework.security.core.context.SecurityContextImpl@ffffffff: Null authentication]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
22-Jan-2019 08:12:09.816 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-127.0.0.1-8009"]
22-Jan-2019 08:12:09.820 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-0.0.0.0-8080"]
22-Jan-2019 08:12:09.820 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-0.0.0.0-8080"]
22-Jan-2019 08:12:09.821 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-nio-127.0.0.1-8009"]
Again, aborted restart, then killed the process java process holding the ports open, thrashed the catalina PID file again and restarted tomcat. This time everything booted fine.
Unfortunately hard shut downs due to power outages are real scenario. How can I get this to be more robust in recovering form a power outage or other hard shut down? It appears that my installation just can't deal with destroying old processes.