fail to permission files after using bulk uploader OpenKM 6.3 CE

We tried to make OpenKM as intuitive as possible, but an advice is always welcome.
Forum rules
Please, before asking something see the documentation wiki or use the search feature of the forum. And remember we don't have a crystal ball or mental readers, so if you post about an issue tell us which OpenKM are you using and also the browser and operating system version. For more info read How to Report Bugs Effectively.
Post Reply
adrian.sima
Fresh Boarder
Fresh Boarder
Posts: 6
Joined: Tue Jun 08, 2021 11:44 am

fail to permission files after using bulk uploader OpenKM 6.3 CE

Post by adrian.sima »

Hello,

We are currently using OpenKM 6.3 CE and last week I managed to migrate over 50GB of documents into our OpenKM production repository using built-in OpenKM tool: Bulk Uploader/File Uploader. The files were uploaded using 'Administrator' user.

After setting up permissions for other users ( different Roles or separate user access) I noticed there's no visibility for them. Before I opened up this thread I tried several possibilities to grant at least read permissions for the newly migrated resources:

- created a new user and set up permissions read/write ( including recursive permissions) for our main directory
The result here was that the main directory was visible under taxonomy but the files/subfolders within the main directory were not.
- afterwards drilled into the subfolders and granted same permissions ( read/write, including recursive perm.) but the result wasn't access to files but only to the subfolder. If there were more subfolders these weren't available for the end user, just the one I granted permission earlier.

I also researched on various problems with other users using 'Bulk Uploader' but I didn't find anything similar.

The conclusion is that at the moment only the admin who uploaded the files into the application repository can view/edit/grant permissions and there's no workaround in order to grant permissions to the newly migrated files.

Would really be of help if we could determine if wether this is an issue with permissions at OS Level over the uploaded documents or if it's a bug in the application which isn't letting me to make visible the files after using 'Bulk Uploader' openkm tool.

If you require additional printscreens I can deliver them over the e-mail because we can't replicate this issue into the test environment since I have uploaded the files directly into the production environment.

Thanks in advance,

Kind Regards,
Adrian.
adrian.sima
Fresh Boarder
Fresh Boarder
Posts: 6
Joined: Tue Jun 08, 2021 11:44 am

Re: fail to permission files after using bulk uploader OpenKM 6.3 CE

Post by adrian.sima »

Hello,

I attached a test case from an OpenKM 6.3 CE test environment ilustrating the replication steps for the presented issue.

Kind Regards,
Adrian.
Attachments
2021-07-06 TestCase Bulk Uploader.docx
(153.7 KiB) Downloaded 77 times
jllort
Moderator
Moderator
Posts: 11570
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: fail to permission files after using bulk uploader OpenKM 6.3 CE

Post by jllort »

* You should create a ROLE -> assign to a users
* Go to some folder and then change security adding this ROLE -> ensure you check the checkbox named apply security recursive ( if the security is applied to a lot of nodes may take a lot of time until be finished )
Selección_033.png
Another way to propagate security at low level is this script ( after the execution is it required you rebuild lucene indexes, because changes done directly from database are not propagated to the search engine )
:https://docs.openkm.com/kcenter/view/ok ... level.html
adrian.sima
Fresh Boarder
Fresh Boarder
Posts: 6
Joined: Tue Jun 08, 2021 11:44 am

Re: fail to permission files after using bulk uploader OpenKM 6.3 CE

Post by adrian.sima »

Hello Jilort,

Please check step 8 of the previous attached documentation and we see there ROLE_USER already assigned to the Dummy user.

Also check currently attached document which I assigned another role, granted permissions in the directory where I bulk uploaded the files and then logged in with dummy user and there's the same behavior: no visibility over the files.

*I also included recursive permissions in every tested scenario. Even in production environment where I migrated over 50 GB of data I tried all mentioned possibilities and didn't get any positive result in seeing the files with any other user than 'Administrator' user ( the one which I used for Bulk Uploader activity).
2021-07-16 Updates Bulk Uploader.docx
(106.1 KiB) Downloaded 73 times
In reviewed the other documentation to propagate security at low level but at some point there's rebuilding lucene indexes which might affect performance of the application? We have 1.2 T of data and I guess you can't rebuild those indexes just for the migrated data, right?

Do you have any other suggestions?

Kind Regards,
Adrian.
jllort
Moderator
Moderator
Posts: 11570
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: fail to permission files after using bulk uploader OpenKM 6.3 CE

Post by jllort »

When you talk about bulk uploader -> about what tools are you talking about?
Seems you are doing the initial import, why are not using OpenKM repository import https://docs.openkm.com/kcenter/view/ok ... mport.html ? the files to be uploaded are not accessible directly from the OpenKM server?
adrian.sima
Fresh Boarder
Fresh Boarder
Posts: 6
Joined: Tue Jun 08, 2021 11:44 am

Re: fail to permission files after using bulk uploader OpenKM 6.3 CE

Post by adrian.sima »

The bulk uploader I was talking about is referenced in your OpenKM CE 6.3 documentation, accessible at URL: https://docs.openkm.com/kcenter/view/ok ... oader.html

The files I needed to upload in OpenKM aren't from the current OpenKM repository which in our case is underlying on the application server.

The files I uploaded reside in another server, were uploaded using RDP connection to that server, connected with admin user and used the procedure above.

Kind Regards,
Adrian.
jllort
Moderator
Moderator
Posts: 11570
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: fail to permission files after using bulk uploader OpenKM 6.3 CE

Post by jllort »

It seems the java tool for bulk upload I suggest do not use it, you can drag & drop directly from OpenKM UI https://www.youtube.com/watch?v=JbFvSNJ ... e=emb_logo

In case of a large upload with a lot of folders and files I suggest transfer files from your computer to the OpenKM server and from there use the OpenKM repository import ( Administration > tools > repository import ). It requires an extra step - copy to the server - but sure will work better and you'll get more control of the whole process.
adrian.sima
Fresh Boarder
Fresh Boarder
Posts: 6
Joined: Tue Jun 08, 2021 11:44 am

Re: fail to permission files after using bulk uploader OpenKM 6.3 CE

Post by adrian.sima »

Hello again,

Can we validate 100 % percent that we cannot do anything in regards of the migration process I already did on production environement?

We're talking about over 50GB of data, time consumed resources and a planification which went smoothly until I couldn't access the files from the system on end-users side.

Since we're also involved in a upgrade process for OpenKM Pro (with your partner in Romania, Softex) I'd like from your side some support or a valid workaround to gain time and use the files which were already uploaded.

There's lot of data to be stored locally, there's also sensitive data and might take time in order to get approvals on your suggested scenario.

Thanks again for support but I'd appreciate a little more research on this java tool on your side.

Kind Regards,
Adrian.
jllort
Moderator
Moderator
Posts: 11570
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: fail to permission files after using bulk uploader OpenKM 6.3 CE

Post by jllort »

Ir you are going to upgrade to OpenKM pro do not have the sense to use the CE forum. For this purpose, there's the OpenKM professional support website.

In case of huge import should analyze data to transfer and when analyzed decide what is the best manner for doing it. There's no magic process for doing it because although most scenarios are very similar not in all of them can use the same way to import data. I suggest talking with your sales manager in order to coordinate this matter.
adrian.sima
Fresh Boarder
Fresh Boarder
Posts: 6
Joined: Tue Jun 08, 2021 11:44 am

Re: fail to permission files after using bulk uploader OpenKM 6.3 CE

Post by adrian.sima »

Hello,

At the moment we're using OpenKM CE therefore I must use the forum, this migration process is at the moment at project phase and might take from 6 months to 1 year to complete.

Since this tool was available and since I've done so much effort in using this tool can't you propose a decent solution for our blocked scenario?

It's quite simple:

1. 50 GB of data uploaded using Bulk Uploader
2. Files available just for application admin user ( the same user I've uploaded the files with )
3. Tested all sorts of workarounds and can't seem to make data available for other end users.
4. That's why I created a CE Forum user to get professional instructions on this available function.

Scope of work at the moment is to benefit from some kind of support using CE, would be nice at least to find out if at server level can be some checks in order to make files available.

Kind Regards,
Adrian.
jllort
Moderator
Moderator
Posts: 11570
Joined: Fri Dec 21, 2007 11:23 am
Location: Sineu - ( Illes Balears ) - Spain
Contact:

Re: fail to permission files after using bulk uploader OpenKM 6.3 CE

Post by jllort »

Do not exist a quick answer for your question, because the problem is that your scenario should be analyzed to understanding the problem you are trying to solve and usually is not a task that can be completed in 5 minutes ( bulk upload is not the problem is the solution you tried to apply to the problem ).

Anyway I suggest do not use bulk upload for this huge import you should:
1- copy files in the server and use import tools ( it creates a log -> you need a log - may be this log is not exactly what you need - of what happens and usually have an acceptable speed )
2- OpenKM cron job to upload files from disk location, network location, or another kind of localtion like remote ftp
3- create a java client or c# client and use the application to upload ( can create your own log and take more control -> for example the number of parallel uploads allowed )
4- use hot folder windows client -> do not upload folders only files in the same folder -> that means maybe is not useful to solve the problem you are trying to solve

When having large uploading process you need always ( may be other too, but if you consider these requisits then can thinking about what option may fit better for these requirements ):
* control what happened -> log
* control errors -> log of errors
* When possible do in the background during nonworking hours
Post Reply