I can accept without problems what you say about tree perfomance, but now GWT does not provide us an easy solution for solving it. Here we have a technical problem and a restriction on the way the UI should be used.
You can be created a custom UI, it's quite easy and fast working on it with SDK and spring boot sample. From that scenario you will sucess in two points, exact fit of features in your needs and best performance only showing features you need in manner you wish. Really the users need to navigate across the folder structure or might be more easy a simply search with two or three input boxes and then retrieving data. In a huge scenario where OpenKM is used for a very specific data container as you are explaining now, I suggest small cutom UI in combination with automatic catalog feature. User should not know that behind the UI they have the DMS.
The last time I wach on documentum was focused on something more near a search engine view ( filtering and paginating ). But I do not remember a tree navigation in combination with table list ... if you make some screenshot of it, will be welcome, because always we are open to do improvements.
I have reviewed a lot of dms tools, from now I think are two main directions:
1- solutions based on search filtering ( work in cabinet concepts or document type concept ). You choose one cabinet or document type and from there you are able to apply filters what shown a paginated table with results ( here you do not have any kind of tree )
2- solutions based on tree ( taxonomy ), where you are able to navigate from left side, and from right panel you have a table with nodes paginated.
Both solutions have advantatges and disadvantatges ( I could tell you a list of application what going in one directions and other what going in another, really older dms started with option 1 and seems later option 2 has been applied by newer. It does means first is better than second, in my opinion tree option have some disadvantatges and in some scenarios might be not a good idea ). In deep, I have arrived the conclusion that is not possible building an UI what fit all user scenarios. Arrived this point you can go in three directions:
1- User must adapt to UI restrictions
2- UI can be extended and or modified ( us we have discarted this scenario, I will not explain all the reasons behind the decision, but a will share what for me is the most relevant, here we are in OSGI architecture what is quite complex and you need a very high qualified staff for working on it ... this scenarios have some disadvantatges )
3- Custom UI based on SDK + sample project ( for me this is the best aproach, what has less disadvantatges in comparison of what you get and the maintenance of it ).