Accessing EOS from REANA

1

I’m currently trying to understand how to access EOS files in reana and running into issues.

I followed the instructions here http://docs.reana.io/advanced-usage/storage-backends/eos/ but don’t seem to be able to get it to work.

As an example I’ve set up the correct secrets and submitted a very simple job:

version: 0.6.0
workflow:
      type: serial
      specification:
        steps:
          - name: test
            environment: 'reanahub/krb5'
            kerberos: true
            commands:
            - klist
            - cat /eos/user/b/broberts/test/test.txt

The log for this is:

==> Job logs
==> Step: test
==> Workflow ID: a8e67d1f-874e-48c5-8f17-38f467a24d9d
==> Compute backend: Kubernetes
==> Job ID: 86a50509-90eb-4a60-b622-5106b55d5d8b
==> Docker image: reanahub/krb5
==> Command: klist
==> Status: finished
==> Logs:
krb5: 
  
job: 
 Ticket cache: FILE:/krb5_cache/krb5_1000
Default principal: broberts@CERN.CH

Valid starting     Expires            Service principal
05/07/20 14:00:01  05/08/20 15:00:01  krbtgt/CERN.CH@CERN.CH
	renew until 05/12/20 14:00:01
 

==> Step: test
==> Workflow ID: a8e67d1f-874e-48c5-8f17-38f467a24d9d
==> Compute backend: Kubernetes
==> Job ID: 54e84ca2-a127-4815-be52-df8ebe83b12b
==> Docker image: reanahub/krb5
==> Command: cat /eos/user/b/broberts/test/test.txt
==> Status: failed
==> Logs:
krb5: 
  
job: 
 cat: /eos/user/b/broberts/test/test.txt: No such file or directory

(I have triple checked and the file definitely exists in that location.)
If anyone could offer help or guidance on this issue that would be much appreciated.

2

Hi @broberts, I confirm there are indeed troubles accessing EOS volumes on REANA. We shall look at fixing the problem.

3

Hello @broberts, welcome to the REANA forum.

We have identified the root cause of this issue and it is now solved in the production instance https://reana.cern.ch. We are now investigating with the CERN cloud team how to prevent this from happening in the future.

Should you have any other problem related to this issue please do not hesitate to tell us here.

Thanks for contributing!

1 Like
4

Hi,

I also have an issue accessing eos files from REANA. It looks like my REANA job cannot access this file of mine:

/eos/project/r/recast/atlas/ANA-EXOT-2018-54/DAODs/DAOD_EXOT17.21861296._000001.pool.root.1

, even though I have

resources:
  - kerberos: true
  - kubernetes_uid: 1000

set up in the workflow.yml. The job fails with

[ERROR] Server responded with an error: [3010] Unable to open directory /eos/project/r/recast/atlas/ANA-EXOT-2018-54; Operation not permitted

I can however confirm that a standalone docker job on my own laptop can access this file:

$ docker run --rm -it gitlab-registry.cern.ch/atlas-phys/exot/ueh/multichargedparticles/2015-2018/analysiscode:latest
[in docker] $ kinit ysmirnov@CERN.CH
[in docker] $ source /home/atlas/release_setup.sh
[in docker] $ source x86_64-centos7-gcc8-opt/setup.sh
[in docker] $ cd ../
[in docker] $ root -b -q $ROOTCOREDIR/scripts/load_packages.C ATestRun.cxx\(\"submitDir0001\",\ 500,\ 7,\ \"\",\ true\)
[in docker] $ exit
$ exit

I see two possible differences:

  1. I do kinit manually in Docker, but the same should be achieved by the .keytab secrets in the REANA cloud, shouldn’t it?
  2. when I run in Docker installed on my own PC, the user is atlas, and when I run in the REANA cloud, the user is probably reana (?)

Anyway, what can cause this issue and how to solve it?

Thanks!

5

Hi,

this is understood and fixed now. There was something wrong with my keytab secrets. I regenerated the file, reuploaded it to the cloud and the issue was gone. It probably has to do with the fact that in the ATLAS tutorial we used to generate this file for a special user, who probably does not have access to my file in EOS (?). This is something I completely forgot about. So when I was regenerating the .keytab file, I did it for my own account now, the one I’m sure had access to my EOS file, and as I said the issue was gone.