Fix NodeJS retrieval when Docker is unavailable#1054
Open
GlassOfWhiskey wants to merge 1 commit into
Open
Conversation
GlassOfWhiskey
force-pushed
the
node-with-singularity
branch
from
640c201 to
bec7ab6
Compare
kinow
reviewed
May 5, 2026
Contributor
There was a problem hiding this comment.
I accidentally used a micromamba env with nodejs first, and thought it worked. I couldn't find any .sif files like I did with cwltool which was odd.
So I was cleaning up the files and reviewing what I did when I saw that and uninstalled NodeJS. I tried this again, and the .sif file for NodeJS was actually downloaded, but one test failed.
========================================================================================== FAILURES ==========================================================================================
_____________________________________________________________________________ cwl test: step_input_default_value _____________________________________________________________________________
[gw6] linux -- Python 3.14.4 /gpfs/home/bsc/<USER>/cwl/streamflow/cwl-conformance-venv/bin/python
CWL test execution failed.
Returned non-zero but it should be zero
Test: job:
file:///gpfs/home/bsc/<USER>/cwl/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/empty.json
output:
count_output: 16
tool:
file:///gpfs/home/bsc/<USER>/cwl/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/count-lines9-wf.cwl
label: step_input_default_value
id: 49
doc: Test default value on step input parameter
tags:
- inline_javascript
- workflow
line: '493'
------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------
2026-05-05 09:09:38.658 INFO Processing workflow 14c86613-091a-4050-94e9-8faaf8bea15a
2026-05-05 09:09:38.658 INFO Building workflow execution plan
2026-05-05 09:09:38.679 INFO COMPLETED building of workflow execution plan
2026-05-05 09:09:38.679 INFO EXECUTING workflow 14c86613-091a-4050-94e9-8faaf8bea15a
2026-05-05 09:09:38.737 INFO COPYING from /gpfs/home/bsc/<USER>/cwl/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt to /scratch/tmp/streamflow/477fa312-1387-4adf-918c-ac5741864669/657250c5-4351-4ad2-a901-70e6e9a65618/whale.txt on local file-system
2026-05-05 09:09:38.738 INFO COMPLETED copy from /gpfs/home/bsc/<USER>/cwl/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt to /scratch/tmp/streamflow/477fa312-1387-4adf-918c-ac5741864669/657250c5-4351-4ad2-a901-70e6e9a65618/whale.txt on local file-system
2026-05-05 09:09:38.740 INFO EXECUTING step /step1 (job /step1/0) locally into directory /scratch/tmp/streamflow/81c928e1-95a2-4000-b6ca-61f912dfa918:
wc -l
2026-05-05 09:09:38.760 INFO COMPLETED Step /step1
2026-05-05 09:09:38.763 INFO Evaluating expression for step /step2 (job /step2/0)
FATAL: Image file already exists: "node_alpine.sif" - will not overwrite
2026-05-05 09:09:38.793 ERROR NodeJSEngine requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, singularity run node:alpine
Traceback (most recent call last):
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/core/recovery.py", line 42, in wrapper
await func(*args, **kwargs)
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/workflow/step.py", line 736, in _execute_command
command_output := await command_task
^^^^^^^^^^^^^^^^^^
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/cwl/command.py", line 1304, in execute
result = utils.eval_expression(
expression=self.expression,
...<2 lines>...
expression_lib=self.expression_lib,
)
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/cwl/utils.py", line 742, in eval_expression
cwl_utils.expression.interpolate(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
expression,
^^^^^^^^^^^
...<9 lines>...
container_engine=_get_container_engine(),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/gpfs/home/bsc/<USER>/cwl/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/cwl_utils/expression.py", line 227, in interpolate
e = evaluator(
js_engine, scan[w[0] + 1 : w[1]], rootvars, jslib, fullJS, **kwargs
)
File "/gpfs/home/bsc/<USER>/cwl/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/cwl_utils/expression.py", line 173, in evaluator
return cast(CWLOutputType, js_engine.eval(ex, jslib, **kwargs))
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/home/bsc/<USER>/cwl/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/cwl_utils/sandboxjs.py", line 473, in eval
returncode, stdout, stderr = self.exec_js_process(
~~~~~~~~~~~~~~~~~~~~^
fn,
^^^
...<3 lines>...
container_engine=container_engine,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/gpfs/home/bsc/<USER>/cwl/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/cwl_utils/sandboxjs.py", line 201, in exec_js_process
new_proc = self.new_js_proc(
js_engine_code,
force_docker_pull=force_docker_pull,
container_engine=container_engine,
)
File "/gpfs/home/bsc/<USER>/cwl/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/cwl_utils/sandboxjs.py", line 442, in new_js_proc
raise JavascriptException(
...<5 lines>...
)
cwl_utils.errors.JavascriptException: NodeJSEngine requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, singularity run node:alpine
2026-05-05 09:09:38.811 WARNING Job /step2/0 failure can not be recovered. Failure manager is not enabled.
2026-05-05 09:09:38.812 ERROR NodeJSEngine requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, singularity run node:alpine
2026-05-05 09:09:38.814 INFO FAILED Step /step2
2026-05-05 09:09:38.816 ERROR FAILED Workflow execution
Traceback (most recent call last):
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/cwl/runner.py", line 112, in main
asyncio.run(_async_main(parsed_args))
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/home/bsc/<USER>/.local/share/uv/python/cpython-3.14.4-linux-x86_64-gnu/lib/python3.14/asyncio/runners.py", line 204, in run
return runner.run(main)
~~~~~~~~~~^^^^^^
File "/gpfs/home/bsc/<USER>/.local/share/uv/python/cpython-3.14.4-linux-x86_64-gnu/lib/python3.14/asyncio/runners.py", line 127, in run
return self._loop.run_until_complete(task)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/gpfs/home/bsc/<USER>/.local/share/uv/python/cpython-3.14.4-linux-x86_64-gnu/lib/python3.14/asyncio/base_events.py", line 719, in run_until_complete
return future.result()
~~~~~~~~~~~~~^^
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/cwl/runner.py", line 93, in _async_main
await streamflow.cwl.main.main(
workflow_config=workflow_config, context=context, args=args
)
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/cwl/main.py", line 93, in main
output_tokens = await executor.run()
^^^^^^^^^^^^^^^^^^^^
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/workflow/executor.py", line 175, in run
raise WorkflowExecutionException("FAILED Workflow execution")
streamflow.core.exception.WorkflowExecutionException: FAILED Workflow execution
Test failed unexpectedly: common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/count-lines9-wf.cwl common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/empty.json
Test default value on step input parameter
------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------
WARNING cwltest:plugin.py:82 Test failed unexpectedly: common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/count-lines9-wf.cwl common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/empty.json
WARNING cwltest:plugin.py:83 Test default value on step input parameter
----------------------------------------------------------- generated xml file: /gpfs/home/bsc/<USER>/cwl/streamflow/junit.xml ------------------------------------------------------------
================================================================================== short test summary info ===================================================================================
SKIPPED [1] common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/conformance_test_v1.0.cwltest.yaml: Test 'docker_entrypoint' is in the exclude list.
========================================================================= 1 failed, 195 passed, 1 skipped in 16.56s ==========================================================================Retrying that, it worked, and all tests passed, without the need to install or load NodeJS modules.
Thanks @GlassOfWhiskey
Member
Author
|
@kinow I found the time to investigate the problem. This is due to the fact that |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1054 +/- ##
=======================================
Coverage 64.16% 64.16%
=======================================
Files 93 93
Lines 17510 17518 +8
Branches 2149 2151 +2
=======================================
+ Hits 11235 11241 +6
- Misses 5997 5998 +1
- Partials 278 279 +1 ☔ View full report in Codecov by Sentry. |
LanderOtto
approved these changes
May 18, 2026
Fix #1053 by adding `_get_container_engine()` to detect the available container runtime by probing `docker`, `singularity`, `podman`, and `udocker` in order, caching the result with `@cached(FIFOCache(1))` from `cachebox`. The detected engine is passed as `container_engine` to `cwl_utils.expression.interpolate()` in `eval_expression()`, fixing CWL JavaScript expression evaluation on systems without Docker.
GlassOfWhiskey
force-pushed
the
node-with-singularity
branch
from
bec7ab6 to
30203cd
Compare
kinow
reviewed
May 30, 2026
Contributor
kinow
left a comment
kinow
left a comment
There was a problem hiding this comment.
Hmmm, I'm really not able to test this on MN5. It keeps insisting that NodeJS is not found, but it doesn't try to use Singularity. See my comment in the cwl-utils PR (which is related to this too): common-workflow-language/cwl-utils#424 (review)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #1053 by adding
_get_container_engine()to detect the available container runtime by probingdocker,singularity,podman, andudockerin order, caching the result with@cached(FIFOCache(1))fromcachebox. The detected engine is passed ascontainer_enginetocwl_utils.expression.interpolate()ineval_expression(), fixing CWL JavaScript expression evaluation on systems without Docker.