Trying to run the monojet workflow on reana, some of the issues I’ve run into is to do with sudo privileges and permissions.
Sudo privileges
When any command in the steps.yaml file tries to use sudo privileges, which works fine when running directly with yadage locally, it results in the following error message, as shown in the reana logs on https://reana-qa.cern.ch:
sudo: unknown uid 1000: who are you?
Example: uncertainty_tool_stage
reana output for this step:
job:
sudo: unknown uid 1000: who are you?
sudo: unknown uid 1000: who are you?
Permissions
When I go through and remove the use of sudo from the steps.yml file, the above step mentioned under ‘Sudo privileges’ succeeds, but others fail with error messages indicating insufficient permissions to create files/directories.
Example 1
Bash command in steps.yml: https://gitlab.cern.ch/atlas-phys/exot/jdm/ANA-EXOT-2018-06/workflow/blob/without_kerberos/steps.yml#L36
reana log output: mkdir: cannot create directory mc_dir’: Permission denied`
Example 2
Bash command in steps.yml: https://gitlab.cern.ch/atlas-phys/exot/jdm/ANA-EXOT-2018-06/workflow/blob/without_kerberos/steps.yml#L41
reana log output: mkdir: cannot create directory SignalMC’: Permission denied`
Is there a way around these permissions issues without the need for sudo privileges?
Hi @damacdon ,
I think I might be running into similar issue. In my case it is when python script wants to create file but I get “Permission denied” error. Everything seems to work fine when I am not running on REANA. I know this topic is really old but don’t you remember if you managed to resolve the issue? And if so then how?
Cheers,
Ondřej
Hi @otheiner Yes we solved that issue with @damacdon at the time. Sorry for not documenting the solution here.
Basically, there are two things to know when comparing standalone docker run ... processes and REANA jobs:
- By default, REANA runs your processes under special user identity (UID=1000, GID=0), i.e. not as “root”. This is due to security concerns.
- By default, REANA runs your workflows in a directory such as
/var/reana/users/11111/workflows/22222 that is mounted into the container, i.e. not in an absolute path created in entry points such as WORKDIR. This is due to storage space concerns, since the ephemeral storage is O(10GB) and specific to each cluster node, whilst the workflow’s workspace storage is O(200TB) and shared between nodes.
Therefore, if your job cannot write to some particular directory, please check its permissions, and see whether the given user has rights to write there.
For example, if you write files outside of the workflow’s default working directory quoted above, into an absolute path such as /mydir/ inside the container, then this is using volatile container storage, not the shared directory, and it can happen that user UID=1000 GID=0 won’t have rights to write there, unlike “root” user. You could solve the issue by modifying Dockerfile to allow GID=0 write rights into those places via chmod g+w /mydir and the like.
Note also that if you need to use another user identity than UID=1000, you can specify kubernetes_uid workflow hint in order to indicate the desired user ID for each workflow step. See for example the ATLAS RECAST demo example using UID=500: reana-demo-atlas-recast/steps.yml at master · reanahub/reana-demo-atlas-recast · GitHub
If you create your container images with the above in mind, then REANA should behave very much like the “standalone” Docker processes with docker run --user 1000:0 ... ran locally.
(And those sudo commands wouldn’t be necessary inside the workflow definition.)
1 Like