Using the output of previous multistep-stage in next step

algomez · 6 April 2021 13:08

Hi all

I am having problems using the output of one step into the next step. I am using yadage and this is my reana.yml:

version: 0.7.2
inputs:
    directories:
        - AOD2NanoAOD/data
    files:
        - scripts/steps.yaml
        - scripts/workflow.yaml
        - AOD2NanoAOD/configs/general_cfg.py
        - Analyzer/rootToPandas.py
    parameters:
        aodnano: AOD2NanoAOD/configs/general_cfg.py
        rootpandas: Analyzer/rootToPandas.py
        mcfiles:
            - AOD2NanoAOD/data/CMS_MonteCarlo2012_Summer12_DR53X_DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball_AODSIM_PU_RD1_START53_V7N-v1_20001_01.txt
            - AOD2NanoAOD/data/CMS_MonteCarlo2012_Summer12_DR53X_DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball_AODSIM_PU_RD1_START53_V7N-v1_20001_02.txt
            - AOD2NanoAOD/data/CMS_MonteCarlo2012_Summer12_DR53X_DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball_AODSIM_PU_RD1_START53_V7N-v1_20001_03.txt
        datafiles:
            - AOD2NanoAOD/data/CMS_Run2012B_DoubleMuParked_AOD_22Jan2013-v1_20000_01.txt
            - AOD2NanoAOD/data/CMS_Run2012B_DoubleMuParked_AOD_22Jan2013-v1_20000_02.txt
            - AOD2NanoAOD/data/CMS_Run2012B_DoubleMuParked_AOD_22Jan2013-v1_20000_03.txt
        maxevents: 1000
        mcflag: 'False'
        dataflag: 'True'

workflow:
    type: yadage
    file: scripts/workflow.yaml

my step.yaml:

createnano:
    process:
        process_type: 'interpolated-script-cmd'
        interpreter: bash
        script: |
            pwd -LP
            source /opt/cms/cmsset_default.sh
            cd CMSSW_5_3_32/src/
            eval `scramv1 runtime -sh`
            cd {workingdir}
            pwd -LP
            echo "cmsRun {torun} maxEvents={maxevents} txtFile={txtfile} isData={isdata}"
            cmsRun {torun} maxEvents={maxevents} txtFile={txtfile} isData={isdata} outputFile={outputroot}
            rename "_numEvent1000" "" createnano*/*root
            cat {outputroot}
    environment: 
        environment_type: 'docker-encapsulated'
        image: gitlab-registry.cern.ch/algomez/cms-open-data
        imagetag: latest
    publisher:
        publisher_type: frompar-pub
        outputmap:
            outputfile: outputroot

skimpandas:
    process:
        process_type: 'interpolated-script-cmd'
        interpreter: bash
        script: |
            ls -lh
            cd {workingdir}
            echo {inputfile}
            python {torun} --inputFile={inputfile} --outputFile={outname}
            mv *png {outdir}
            mv *csv {outdir}
            mv *pkl {outdir}
            cat {outpng}
    environment: 
        environment_type: 'docker-encapsulated'
        image: gitlab-registry.cern.ch/algomez/cms-open-data/rootpython
        imagetag: latest
    publisher:
        publisher_type: frompar-pub
        outputmap:
            outputpng: outpng

and my workflow.yaml:

stages:
- name: createnanosimulation
  dependencies: [init]
  scheduler:
      scheduler_type: 'multistep-stage'
      parameters:
          torun: {step: init, output: aodnano}
          workingdir: '$REANA_WORKSPACE'
          maxevents: {step: init, output: maxevents}
          isdata: {step: init, output: mcflag}
          txtfile: {step: init, output: mcfiles, unwrap: true}
          outputroot: '{workdir}/output_simulation.root'
          #outputdir: '{workdir}'
      scatter:
          method: zip
          parameters: [txtfile]
      step: {$ref: 'scripts/steps.yaml#/createnano'}

- name: skimsim
  dependencies: [createnanosimulation]
  scheduler:
      #scheduler_type: 'singlestep-stage'
      scheduler_type: 'multistep-stage'
      parameters:
          inputfile: {step: createnanosimulation, output: outputfile}
          torun: {step: init, output: rootpandas}
          workingdir: '$REANA_WORKSPACE'
          outname: 'output_sim'
          outpng: '{workdir}/nMuonscheck_output_sim.png'
          outdir: '{workdir}'
      scatter:
          method: zip
          parameters: [inputfile]
      step: {$ref: 'scripts/steps.yaml#/skimpandas'}

and I am getting this error:

(reana) algomez@lxplus779:~/workingArea/OpenData/cms-open-data$ reana-client logs -w opendatatest 
==> Workflow engine logs
Workflow exited unexpectedly.
max() arg is an empty sequence
2021-04-06 13:06:49,596 | reana-workflow-engine-yadage | MainThread | INFO | getting socket..
2021-04-06 13:06:49,683 | yadage.creators | MainThread | INFO | initializing workflow with initdata: {'aodnano': 'AOD2NanoAOD/configs/general_cfg.py', 'datafiles': ['AOD2NanoAOD/data/CMS_Run2012B_DoubleMuParked_AOD_22Jan2013-v1_20000_01.txt', 'AOD2NanoAOD/data/CMS_Run2012B_DoubleMuParked_AOD_22Jan2013-v1_20000_02.txt', 'AOD2NanoAOD/data/CMS_Run2012B_DoubleMuParked_AOD_22Jan2013-v1_20000_03.txt'], 'dataflag': 'True', 'maxevents': 1000, 'mcfiles': ['AOD2NanoAOD/data/CMS_MonteCarlo2012_Summer12_DR53X_DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball_AODSIM_PU_RD1_START53_V7N-v1_20001_01.txt', 'AOD2NanoAOD/data/CMS_MonteCarlo2012_Summer12_DR53X_DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball_AODSIM_PU_RD1_START53_V7N-v1_20001_02.txt', 'AOD2NanoAOD/data/CMS_MonteCarlo2012_Summer12_DR53X_DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball_AODSIM_PU_RD1_START53_V7N-v1_20001_03.txt'], 'mcflag': 'False', 'rootpandas': 'Analyzer/rootToPandas.py'} discover: True relative: True
2021-04-06 13:06:49,684 | reana-workflow-engine-yadage | MainThread | INFO | running workflow on context: {'publisher': <reana_commons.publisher.WorkflowStatusPublisher object at 0x7f85af2fe910>, 'rjc_api_client': <reana_commons.api_client.JobControllerAPIClient object at 0x7f85af2feb20>, 'workflow_uuid': '04d87c3a-933d-44e4-a912-7c73a44a286d', 'workflow_workspace': '/var/reana/users/3938cb60-6819-4c9a-89e5-657f0beadae2/workflows/04d87c3a-933d-44e4-a912-7c73a44a286d', 'workflow_file': 'scripts/workflow.yaml', 'workflow_parameters': {'aodnano': 'AOD2NanoAOD/configs/general_cfg.py', 'datafiles': ['AOD2NanoAOD/data/CMS_Run2012B_DoubleMuParked_AOD_22Jan2013-v1_20000_01.txt', 'AOD2NanoAOD/data/CMS_Run2012B_DoubleMuParked_AOD_22Jan2013-v1_20000_02.txt', 'AOD2NanoAOD/data/CMS_Run2012B_DoubleMuParked_AOD_22Jan2013-v1_20000_03.txt'], 'dataflag': 'True', 'maxevents': 1000, 'mcfiles': ['AOD2NanoAOD/data/CMS_MonteCarlo2012_Summer12_DR53X_DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball_AODSIM_PU_RD1_START53_V7N-v1_20001_01.txt', 'AOD2NanoAOD/data/CMS_MonteCarlo2012_Summer12_DR53X_DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball_AODSIM_PU_RD1_START53_V7N-v1_20001_02.txt', 'AOD2NanoAOD/data/CMS_MonteCarlo2012_Summer12_DR53X_DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball_AODSIM_PU_RD1_START53_V7N-v1_20001_03.txt'], 'mcflag': 'False', 'rootpandas': 'Analyzer/rootToPandas.py'}, 'operational_options': {'accept_metadir': True, 'toplevel': '/var/reana/users/3938cb60-6819-4c9a-89e5-657f0beadae2/workflows/04d87c3a-933d-44e4-a912-7c73a44a286d/', 'initdir': '/var/reana/users/3938cb60-6819-4c9a-89e5-657f0beadae2/workflows/04d87c3a-933d-44e4-a912-7c73a44a286d/', 'initfiles': []}, 'kwargs': {'workflow_json': ''}, 'cap_backend': <yadage.backends.packtivitybackend.PacktivityBackend object at 0x7f85af27f8b0>, 'workflow_file_abs_path': '/var/reana/users/3938cb60-6819-4c9a-89e5-657f0beadae2/workflows/04d87c3a-933d-44e4-a912-7c73a44a286d/scripts/workflow.yaml', 'schema_name': 'yadage/workflow-schema', 'schemadir': None, 'specopts': {'toplevel': '/var/reana/users/3938cb60-6819-4c9a-89e5-657f0beadae2/workflows/04d87c3a-933d-44e4-a912-7c73a44a286d/', 'schema_name': 'yadage/workflow-schema', 'schemadir': None, 'load_as_ref': False}, 'validopts': {'schema_name': 'yadage/workflow-schema', 'schemadir': None}, 'workflow_json': {'stages': [{'name': 'createnanosimulation', 'dependencies': {'dependency_type': 'jsonpath_ready', 'expressions': ['init']}, 'scheduler': {'scheduler_type': 'multistep-stage', 'parameters': [{'key': 'torun', 'value': {'step': 'init', 'output': 'aodnano', 'expression_type': 'stage-output-selector'}}, {'key': 'workingdir', 'value': '$REANA_WORKSPACE'}, {'key': 'maxevents', 'value': {'step': 'init', 'output': 'maxevents', 'expression_type': 'stage-output-selector'}}, {'key': 'isdata', 'value': {'step': 'init', 'output': 'mcflag', 'expression_type': 'stage-output-selector'}}, {'key': 'txtfile', 'value': {'step': 'init', 'output': 'mcfiles', 'unwrap': True, 'expression_type': 'stage-output-selector'}}, {'key': 'outputroot', 'value': '{workdir}/output_simulation.root'}], 'scatter': {'method': 'zip', 'parameters': ['txtfile']}, 'step': {'process': {'process_type': 'interpolated-script-cmd', 'interpreter': 'bash', 'script': 'pwd -LP\nsource /opt/cms/cmsset_default.sh\ncd CMSSW_5_3_32/src/\neval `scramv1 runtime -sh`\ncd {workingdir}\npwd -LP\necho "cmsRun {torun} maxEvents={maxevents} txtFile={txtfile} isData={isdata}"\ncmsRun {torun} maxEvents={maxevents} txtFile={txtfile} isData={isdata} outputFile={outputroot}\nrename "_numEvent1000" "" createnano*/*root\ncat {outputroot}\n'}, 'environment': {'environment_type': 'docker-encapsulated', 'image': 'gitlab-registry.cern.ch/algomez/cms-open-data', 'imagetag': 'latest', 'resources': [], 'envscript': '', 'env': {}, 'workdir': None, 'par_mounts': []}, 'publisher': {'publisher_type': 'frompar-pub', 'outputmap': {'outputfile': 'outputroot'}}}}}, {'name': 'skimsim', 'dependencies': {'dependency_type': 'jsonpath_ready', 'expressions': ['createnanosimulation']}, 'scheduler': {'scheduler_type': 'multistep-stage', 'parameters': [{'key': 'inputfile', 'value': {'step': 'createnanosimulation', 'output': 'outputfile', 'expression_type': 'stage-output-selector'}}, {'key': 'torun', 'value': {'step': 'init', 'output': 'rootpandas', 'expression_type': 'stage-output-selector'}}, {'key': 'workingdir', 'value': '$REANA_WORKSPACE'}, {'key': 'outname', 'value': 'output_sim'}, {'key': 'outpng', 'value': '{workdir}/nMuonscheck_output_sim.png'}, {'key': 'outdir', 'value': '{workdir}'}], 'scatter': {'method': 'zip', 'parameters': ['inputfile']}, 'step': {'process': {'process_type': 'interpolated-script-cmd', 'interpreter': 'bash', 'script': 'ls -lh\ncd {workingdir}\necho {inputfile}\npython {torun} --inputFile={inputfile} --outputFile={outname}\nmv *png {outdir}\nmv *csv {outdir}\nmv *pkl {outdir}\ncat {outpng}\n'}, 'environment': {'environment_type': 'docker-encapsulated', 'image': 'gitlab-registry.cern.ch/algomez/cms-open-data/rootpython', 'imagetag': 'latest', 'resources': [], 'envscript': '', 'env': {}, 'workdir': None, 'par_mounts': []}, 'publisher': {'publisher_type': 'frompar-pub', 'outputmap': {'outputpng': 'outpng'}}}}}]}, 'workflow_kwargs': {'workflow_json': {'stages': [{'name': 'createnanosimulation', 'dependencies': {'dependency_type': 'jsonpath_ready', 'expressions': ['init']}, 'scheduler': {'scheduler_type': 'multistep-stage', 'parameters': [{'key': 'torun', 'value': {'step': 'init', 'output': 'aodnano', 'expression_type': 'stage-output-selector'}}, {'key': 'workingdir', 'value': '$REANA_WORKSPACE'}, {'key': 'maxevents', 'value': {'step': 'init', 'output': 'maxevents', 'expression_type': 'stage-output-selector'}}, {'key': 'isdata', 'value': {'step': 'init', 'output': 'mcflag', 'expression_type': 'stage-output-selector'}}, {'key': 'txtfile', 'value': {'step': 'init', 'output': 'mcfiles', 'unwrap': True, 'expression_type': 'stage-output-selector'}}, {'key': 'outputroot', 'value': '{workdir}/output_simulation.root'}], 'scatter': {'method': 'zip', 'parameters': ['txtfile']}, 'step': {'process': {'process_type': 'interpolated-script-cmd', 'interpreter': 'bash', 'script': 'pwd -LP\nsource /opt/cms/cmsset_default.sh\ncd CMSSW_5_3_32/src/\neval `scramv1 runtime -sh`\ncd {workingdir}\npwd -LP\necho "cmsRun {torun} maxEvents={maxevents} txtFile={txtfile} isData={isdata}"\ncmsRun {torun} maxEvents={maxevents} txtFile={txtfile} isData={isdata} outputFile={outputroot}\nrename "_numEvent1000" "" createnano*/*root\ncat {outputroot}\n'}, 'environment': {'environment_type': 'docker-encapsulated', 'image': 'gitlab-registry.cern.ch/algomez/cms-open-data', 'imagetag': 'latest', 'resources': [], 'envscript': '', 'env': {}, 'workdir': None, 'par_mounts': []}, 'publisher': {'publisher_type': 'frompar-pub', 'outputmap': {'outputfile': 'outputroot'}}}}}, {'name': 'skimsim', 'dependencies': {'dependency_type': 'jsonpath_ready', 'expressions': ['createnanosimulation']}, 'scheduler': {'scheduler_type': 'multistep-stage', 'parameters': [{'key': 'inputfile', 'value': {'step': 'createnanosimulation', 'output': 'outputfile', 'expression_type': 'stage-output-selector'}}, {'key': 'torun', 'value': {'step': 'init', 'output': 'rootpandas', 'expression_type': 'stage-output-selector'}}, {'key': 'workingdir', 'value': '$REANA_WORKSPACE'}, {'key': 'outname', 'value': 'output_sim'}, {'key': 'outpng', 'value': '{workdir}/nMuonscheck_output_sim.png'}, {'key': 'outdir', 'value': '{workdir}'}], 'scatter': {'method': 'zip', 'parameters': ['inputfile']}, 'step': {'process': {'process_type': 'interpolated-script-cmd', 'interpreter': 'bash', 'script': 'ls -lh\ncd {workingdir}\necho {inputfile}\npython {torun} --inputFile={inputfile} --outputFile={outname}\nmv *png {outdir}\nmv *csv {outdir}\nmv *pkl {outdir}\ncat {outpng}\n'}, 'environment': {'environment_type': 'docker-encapsulated', 'image': 'gitlab-registry.cern.ch/algomez/cms-open-data/rootpython', 'imagetag': 'latest', 'resources': [], 'envscript': '', 'env': {}, 'workdir': None, 'par_mounts': []}, 'publisher': {'publisher_type': 'frompar-pub', 'outputmap': {'outputpng': 'outpng'}}}}}]}}, 'dataopts': {'initdir': '/var/reana/users/3938cb60-6819-4c9a-89e5-657f0beadae2/workflows/04d87c3a-933d-44e4-a912-7c73a44a286d/'}, 'initdata': {'aodnano': 'AOD2NanoAOD/configs/general_cfg.py', 'datafiles': ['AOD2NanoAOD/data/CMS_Run2012B_DoubleMuParked_AOD_22Jan2013-v1_20000_01.txt', 'AOD2NanoAOD/data/CMS_Run2012B_DoubleMuParked_AOD_22Jan2013-v1_20000_02.txt', 'AOD2NanoAOD/data/CMS_Run2012B_DoubleMuParked_AOD_22Jan2013-v1_20000_03.txt'], 'dataflag': 'True', 'maxevents': 1000, 'mcfiles': ['AOD2NanoAOD/data/CMS_MonteCarlo2012_Summer12_DR53X_DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball_AODSIM_PU_RD1_START53_V7N-v1_20001_01.txt', 'AOD2NanoAOD/data/CMS_MonteCarlo2012_Summer12_DR53X_DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball_AODSIM_PU_RD1_START53_V7N-v1_20001_02.txt', 'AOD2NanoAOD/data/CMS_MonteCarlo2012_Summer12_DR53X_DYJetsToLL_M-50_TuneZ2Star_8TeV-madgraph-tarball_AODSIM_PU_RD1_START53_V7N-v1_20001_03.txt'], 'mcflag': 'False', 'rootpandas': 'Analyzer/rootToPandas.py'}, 'ys': <yadage.steering_object.YadageSteering object at 0x7f85af1c1820>}
2021-04-06 13:06:49,702 | reana-workflow-engine-yadage | MainThread | INFO | initializing REANA workflow tracker for id 04d87c3a-933d-44e4-a912-7c73a44a286d
2021-04-06 13:06:49,703 | adage.pollingexec | MainThread | INFO | preparing adage coroutine.
2021-04-06 13:06:49,703 | adage | MainThread | INFO | starting state loop.
2021-04-06 13:06:49,707 | reana-workflow-engine-yadage | MainThread | INFO | sending progress information
2021-04-06 13:06:49,713 | reana-workflow-engine-yadage | MainThread | INFO | sending to REANA
                    uuid: 04d87c3a-933d-44e4-a912-7c73a44a286d
                    json:
                    {
    "engine_specific": {
        "dag": {
            "edges": [],
            "nodes": []
        }
    },
    "planned": {
        "total": 0,
        "job_ids": []
    },
    "failed": {
        "total": 0,
        "job_ids": []
    },
    "total": {
        "total": 0,
        "job_ids": []
    },
    "running": {
        "total": 0,
        "job_ids": []
    },
    "finished": {
        "total": 0,
        "job_ids": []
    }
}
                    message:
                    this is a tracking log at 2021-04-06T13:06:49.712964
                    
2021-04-06 13:06:49,768 | yadage.wflowview | MainThread | INFO | added </init:0|defined|unknown>
2021-04-06 13:06:50,009 | yadage.handlers.scheduler_handlers | MainThread | INFO | scheduling multistep stage with spec:
{'scheduler_type': 'multistep-stage', 'parameters': [{'key': 'torun', 'value': {'step': 'init', 'output': 'aodnano', 'expression_type': 'stage-output-selector'}}, {'key': 'workingdir', 'value': '$REANA_WORKSPACE'}, {'key': 'maxevents', 'value': {'step': 'init', 'output': 'maxevents', 'expression_type': 'stage-output-selector'}}, {'key': 'isdata', 'value': {'step': 'init', 'output': 'mcflag', 'expression_type': 'stage-output-selector'}}, {'key': 'txtfile', 'value': {'step': 'init', 'output': 'mcfiles', 'unwrap': True, 'expression_type': 'stage-output-selector'}}, {'key': 'outputroot', 'value': '{workdir}/output_simulation.root'}], 'scatter': {'method': 'zip', 'parameters': ['txtfile']}, 'step': {'process': {'process_type': 'interpolated-script-cmd', 'interpreter': 'bash', 'script': 'pwd -LP\nsource /opt/cms/cmsset_default.sh\ncd CMSSW_5_3_32/src/\neval `scramv1 runtime -sh`\ncd {workingdir}\npwd -LP\necho "cmsRun {torun} maxEvents={maxevents} txtFile={txtfile} isData={isdata}"\ncmsRun {torun} maxEvents={maxevents} txtFile={txtfile} isData={isdata} outputFile={outputroot}\nrename "_numEvent1000" "" createnano*/*root\ncat {outputroot}\n'}, 'environment': {'environment_type': 'docker-encapsulated', 'image': 'gitlab-registry.cern.ch/algomez/cms-open-data', 'imagetag': 'latest', 'resources': [], 'envscript': '', 'env': {}, 'workdir': None, 'par_mounts': []}, 'publisher': {'publisher_type': 'frompar-pub', 'outputmap': {'outputfile': 'outputroot'}}}}
2021-04-06 13:06:50,142 | yadage.handlers.scheduler_handlers | MainThread | INFO | scattering
2021-04-06 13:06:50,142 | yadage.handlers.scheduler_handlers | MainThread | INFO | scattering
2021-04-06 13:06:50,154 | yadage.wflowview | MainThread | INFO | added </createnanosimulation:0|defined|unknown>
2021-04-06 13:06:50,168 | yadage.wflowview | MainThread | INFO | added </createnanosimulation:1|defined|unknown>
2021-04-06 13:06:50,186 | yadage.wflowview | MainThread | INFO | added </createnanosimulation:2|defined|unknown>
2021-04-06 13:06:50,628 | yadage.handlers.scheduler_handlers | MainThread | INFO | scheduling multistep stage with spec:
{'scheduler_type': 'multistep-stage', 'parameters': [{'key': 'inputfile', 'value': {'step': 'createnanosimulation', 'output': 'outputfile', 'expression_type': 'stage-output-selector'}}, {'key': 'torun', 'value': {'step': 'init', 'output': 'rootpandas', 'expression_type': 'stage-output-selector'}}, {'key': 'workingdir', 'value': '$REANA_WORKSPACE'}, {'key': 'outname', 'value': 'output_sim'}, {'key': 'outpng', 'value': '{workdir}/nMuonscheck_output_sim.png'}, {'key': 'outdir', 'value': '{workdir}'}], 'scatter': {'method': 'zip', 'parameters': ['inputfile']}, 'step': {'process': {'process_type': 'interpolated-script-cmd', 'interpreter': 'bash', 'script': 'ls -lh\ncd {workingdir}\necho {inputfile}\npython {torun} --inputFile={inputfile} --outputFile={outname}\nmv *png {outdir}\nmv *csv {outdir}\nmv *pkl {outdir}\ncat {outpng}\n'}, 'environment': {'environment_type': 'docker-encapsulated', 'image': 'gitlab-registry.cern.ch/algomez/cms-open-data/rootpython', 'imagetag': 'latest', 'resources': [], 'envscript': '', 'env': {}, 'workdir': None, 'par_mounts': []}, 'publisher': {'publisher_type': 'frompar-pub', 'outputmap': {'outputpng': 'outpng'}}}}
2021-04-06 13:06:50,649 | adage | MainThread | ERROR | some weird exception caught in adage process loop
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/adage/__init__.py", line 51, in run_polling_workflow
    for stepnum, controller in enumerate(coroutine):
  File "/usr/local/lib/python3.8/site-packages/adage/pollingexec.py", line 92, in adage_coroutine
    update_dag(controller, extend_decider,recursive_updates)
  File "/usr/local/lib/python3.8/site-packages/adage/pollingexec.py", line 48, in update_dag
    update_dag(controller, decider, recurse)
  File "/usr/local/lib/python3.8/site-packages/adage/pollingexec.py", line 48, in update_dag
    update_dag(controller, decider, recurse)
  File "/usr/local/lib/python3.8/site-packages/adage/pollingexec.py", line 43, in update_dag
    update_loop.send(command)
  File "/usr/local/lib/python3.8/site-packages/adage/pollingexec.py", line 21, in update_coroutine
    controller.apply_rules([rule])
  File "/usr/local/lib/python3.8/site-packages/adage/wflowcontroller.py", line 53, in apply_rules
    ctrlutils.apply_rules(self.adageobj, rules)
  File "/usr/local/lib/python3.8/site-packages/adage/controllerutils.py", line 127, in apply_rules
    rule.apply(adageobj)
  File "/usr/local/lib/python3.8/site-packages/yadage/stages.py", line 52, in apply
    self.rule.apply(WorkflowView(adageobj, self.offset))
  File "/usr/local/lib/python3.8/site-packages/yadage/stages.py", line 101, in apply
    self.schedule()
  File "/usr/local/lib/python3.8/site-packages/yadage/stages.py", line 144, in schedule
    scheduler(self, self.stagespec)
  File "/usr/local/lib/python3.8/site-packages/yadage/handlers/scheduler_handlers.py", line 302, in multistep_stage
    parameters = {
  File "/usr/local/lib/python3.8/site-packages/yadage/handlers/scheduler_handlers.py", line 303, in <dictcomp>
    k: select_parameter(stage.view, v)
  File "/usr/local/lib/python3.8/site-packages/yadage/handlers/scheduler_handlers.py", line 49, in select_parameter
    value = handler(wflowview, parameter)
  File "/usr/local/lib/python3.8/site-packages/yadage/handlers/expression_handlers.py", line 158, in stage_output_selector
    assert len(steps) == 1
AssertionError
2021-04-06 13:06:50,650 | adage | MainThread | INFO | unsubmittable: 0 | submitted: 0 | successful: 0 | failed: 0 | total: 4 | open rules: 1 | applied rules: 2
2021-04-06 13:06:50,653 | yadage.steering_api | MainThread | INFO | done. dumping workflow to disk.

If I remove the step skimsim, my test workflow runs fine. For this one, I have tried in multistep-stage and singlestep-stage.
Any suggestion?

thanks a lot for the help.

brottler · 15 December 2021 16:19

Not sure if you are still looking for an answer, but since this issue is unresolved and I did recently run into exactly the same problem I wanted to post the solution here in case somebody else has the same problem.

The issue is that when you set up parameters using the output from a multistep-stage you need to use {steps: mystagename, ...} or {stages: mystagename, ...} and not {step: mystagename, ...}, as step assumes exactly one step for this stage.

So for original issue in this thread one would need to change

inputfile: {step: createnanosimulation, output: outputfile}

to e.g.

inputfile: {steps: createnanosimulation, output: outputfile}

See the corresponding part of the code for more details: yadage/expression_handlers.py at master · yadage/yadage · GitHub

mvidalgarcia · 7 January 2022 17:36

Hi @brottler, this post might have slipped away, sorry for that. I’m glad to read you’ve found the solution!