Possible yes, but why ? I do not suggest it, if you want best perfomance install only under tomcat. In other hand if you got middle sized repository you need a lot of hardware resources, share server with other apps is not good scenario. Finally OpenKM is a web application under tomcat. There's some xml descriptors what should be changed to get it running in jboss. Here you got our developer guide to get openkm development environment. If you want to work on get it running in jboss I suggest start from here
http://wiki.openkm.com/index.php/Developer_Guide
Any support from your side about get openkm running in jboss is wellcome, but sincerally if we go out jboss from version 5.x was for performance reasons. From my point of view get it running under jboss can take some interest for you because jboss is mandatory in your environment, but from openkm view and performance not.