Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release: 0.10.0 #64

Merged
merged 123 commits into from
Jan 18, 2025
Merged
Changes from 1 commit
Commits
Show all changes
123 commits
Select commit Hold shift + click to select a range
2e7e6e7
wip(renderers): can pick renderers dynamically
philtweir Jul 20, 2024
aad87a5
fix(renderers): make the renderer semantics less janky
philtweir Jul 20, 2024
c58c17e
feat(renderers): add the ability to pass renderer arguments via CLI/YAML
philtweir Jul 20, 2024
d792f76
feat(renderers): add way to output subworkflows to files
philtweir Jul 20, 2024
b2c10e2
feat: support list/tuple return values
philtweir Jul 24, 2024
f9e6c9b
feat(annotations): add ability to annotate variables
philtweir Aug 3, 2024
d5b8c65
fix: correct tests
philtweir Aug 3, 2024
470de20
wip: add configuration
philtweir Aug 3, 2024
be9b862
fix: scope thread pools to a single render call and add context copying
philtweir Aug 3, 2024
7c34dbe
fix: make sure the globals are updated when a subworkflow is running
philtweir Aug 4, 2024
c6dc355
fix: tests
philtweir Aug 6, 2024
a7c1fe9
fix: integrate flatten_all_nested configuration
philtweir Aug 6, 2024
dee6815
fix: correctly handle flattened nested tasks
philtweir Aug 6, 2024
363be81
fix: confirm tuples can be returned
philtweir Aug 6, 2024
7aeccd9
fix: step reference works as tuple
philtweir Aug 6, 2024
03c8c8b
fix: tidy up type parsing
philtweir Aug 6, 2024
b50b150
fix: nested raw is parseable
philtweir Aug 6, 2024
109dbf9
fix(annotations): correct name to AtRender
philtweir Aug 7, 2024
6cb2ca1
feat(sympy): handle arithmetic/structure manipulation of references
philtweir Aug 10, 2024
0f846e7
feat(sympy): more flexible parameter support
philtweir Aug 10, 2024
ff81ad4
fix: fix render load order
philtweir Aug 10, 2024
e3b9b28
feat(iterated): allow spreading
philtweir Aug 11, 2024
38825fa
feat(fields): make it possible to use a plain dict if configured for …
philtweir Aug 11, 2024
0f047fb
fix: is_raw_type should handle union types
philtweir Aug 11, 2024
9468595
feat(positional): allow positional arguments if configuration overrid…
philtweir Aug 11, 2024
333967f
fix(sympy): support none in dicts
philtweir Aug 11, 2024
079c00b
fix(sympy): support none in dicts
philtweir Aug 11, 2024
213c514
fix: expressions should consider Raw as a raw type
philtweir Aug 11, 2024
08db69f
fix: sorted should not break for two identical steps
philtweir Aug 11, 2024
3dd35d1
fix: efficiency improvements
philtweir Aug 12, 2024
7e8ab1a
fix: efficiency improvements
philtweir Aug 12, 2024
2aba6e6
fix: efficiency improvements
philtweir Aug 12, 2024
36c74f1
feat: add construct args to CLI
philtweir Aug 13, 2024
0517594
fix: make field separator configurable
philtweir Aug 14, 2024
9b5d03a
fix: construct kwargs used to set_configuration
philtweir Aug 14, 2024
77c46bf
fix: construct kwargs used to set_configuration
philtweir Aug 14, 2024
19f1a38
fix: ignore annotations when looking at fieldability
philtweir Aug 14, 2024
ca47bb8
feat(imports): retrieve annotations from imported modules
philtweir Aug 17, 2024
4bdcd8b
feat(fixed): add loopable parameters
philtweir Aug 17, 2024
5773fae
feat(fixed): add loopable parameters
philtweir Aug 17, 2024
0c2cbdc
fix: better error message for iterating a non-iterable
philtweir Aug 17, 2024
00036d0
fix: add is_firm
philtweir Aug 17, 2024
5b83266
fix: move indexing to find_field
philtweir Aug 18, 2024
fc5309e
fix: resolve the parameter deduplication
philtweir Aug 18, 2024
36d15c6
fix: resolve the parameter deduplication
philtweir Aug 18, 2024
f6de37f
fix: fixups in iteration
philtweir Aug 18, 2024
3dbe777
fix(typing): make types consistent in core
philtweir Aug 19, 2024
bd7ee4a
fix(typing): move @subworkflow to @workflow
philtweir Aug 19, 2024
71a35c3
fix: allow workflows to come from packages
philtweir Aug 19, 2024
03e5209
fix: allow workflows to come from packages
philtweir Aug 19, 2024
7d880c0
fix: allow workflows to come from packages
philtweir Aug 19, 2024
aee1ce1
fix: allow workflows to come from packages
philtweir Aug 19, 2024
fc7495e
feat(iterable): provide len for Fixed
philtweir Aug 19, 2024
093b1d6
fix: allow fields of parameters
philtweir Aug 19, 2024
362af32
fix: reduce number of workflow merges
philtweir Aug 19, 2024
828d58c
fix: render configuration
philtweir Aug 20, 2024
8988037
fix: render configuration
philtweir Aug 20, 2024
f03b58c
fix: address field and hashing consistency
philtweir Aug 20, 2024
8c00247
fix: correct logic for remapping before a workflow has a name
philtweir Aug 20, 2024
0be1fea
fix: remove duplicate __hash__ method
philtweir Aug 20, 2024
0bae23c
fix: tidy up 0.9.2 merge, mostly label corrections
philtweir Aug 24, 2024
8e03137
fix: tests not working
philtweir Aug 24, 2024
173b2c7
fix: make sure we can identify an iterated ref
philtweir Aug 24, 2024
0d8b3d1
docs: tidyup docstrings and structures
philtweir Aug 24, 2024
f1dd479
fix: throw import exceptions
philtweir Aug 24, 2024
5c2d7a2
fix: add configuration for simplify_ids
philtweir Aug 24, 2024
4aef9c4
docs: tidyup docstrings and structures
philtweir Aug 24, 2024
8c84ab2
chore: fix up typehints and disable mypy sympy checking, as sympy doe…
philtweir Aug 24, 2024
b4a4cbc
fix: unbreak the doublequotes
philtweir Aug 24, 2024
394e5a6
fix: unbreak the doublequotes
philtweir Aug 24, 2024
88472e4
fix: unbreak the doublequotes
philtweir Aug 24, 2024
861420b
fix: unbreak the doublequotes
philtweir Aug 24, 2024
99f26fd
wip: try 3.12 in test
philtweir Aug 24, 2024
3274ea9
fix: sort examples - TODO: fix explanatory text
philtweir Aug 24, 2024
1dc5692
chore: fix test linting - TODO: add docstrings
philtweir Aug 25, 2024
0a5944f
version: bump to 0.10.0 (starting branch)
philtweir Aug 25, 2024
3871ef4
fix: make result ordering consistent
philtweir Aug 25, 2024
fc88fb4
fix: ensure that fieldability doesn't mean dask matches references as…
philtweir Aug 25, 2024
780a7d6
fix: restore 3.11 in test
philtweir Aug 25, 2024
3e589ee
ci: add 3.12 to the build matrix
philtweir Aug 25, 2024
0c45e51
fix: restore commented test
philtweir Aug 25, 2024
269e591
fix: mypy issue in tests on 3.11
philtweir Aug 25, 2024
4dfa8bd
chore: fix tests and types for mypy and ruff on 3.11 and 3.12
philtweir Aug 25, 2024
33874a3
ci: allow subclassing any for mypy to enable subclassing Symbol
philtweir Aug 25, 2024
3ec7e39
fix: make __make_reference__ pass workflow if none
philtweir Aug 25, 2024
d0e0d37
fix: remove variable re-use
philtweir Aug 25, 2024
e27db97
feat: can go upwards in fields
philtweir Aug 25, 2024
d890bd3
fix: do not lose the field structure when iterating
philtweir Aug 25, 2024
351d344
fix: add a catch for a bug in the first import
philtweir Aug 29, 2024
3bd55e4
fix: remove unnecessary abstractmethod annotation
philtweir Aug 30, 2024
a7c0d8d
fix: remove unnecessary abstractmethod annotation
philtweir Aug 30, 2024
c32469e
chore: Replace all Unions with | for consistency
KamenDimitrov97 Sep 2, 2024
f6f478b
chore: Refactored render.py:get_render_method for better readability,…
KamenDimitrov97 Sep 2, 2024
e76a0a9
chore: Formated methods
KamenDimitrov97 Sep 2, 2024
a0655c0
docs: Added docstrings to annotation tests
KamenDimitrov97 Sep 2, 2024
aa7d6e4
docs: Added docstrings for tests
KamenDimitrov97 Sep 2, 2024
3bff5e5
fix: Formating
KamenDimitrov97 Sep 2, 2024
7ae5a7a
fix: double-check importing correctly errors if render neither behave…
philtweir Sep 9, 2024
2a2d790
fix: check that importing a non-compliant render module throws an error
philtweir Sep 9, 2024
241f338
fix: double-check importing correctly errors if render neither behave…
philtweir Sep 9, 2024
fe5253c
fix: double-check importing correctly errors if render neither behave…
philtweir Sep 9, 2024
c28a31e
fix: ensure pytest ignores tests._lib
philtweir Sep 9, 2024
713882a
fix: correct name of default_config
philtweir Sep 10, 2024
6714786
fix: remove Graph import from dask.typing in backend_dask as it seems…
philtweir Sep 10, 2024
dd27801
ops: enable example tests in CI
KamenDimitrov97 Sep 17, 2024
b6677c0
feat: Added a test for default renderer
KamenDimitrov97 Sep 17, 2024
3b9ccfe
feat: Added a test for default renderer
KamenDimitrov97 Sep 17, 2024
f519c47
feat: Abstracted write_rendered_output from the main dewret CLI
KamenDimitrov97 Sep 30, 2024
a25d3c2
docs: Updated renderer tutorial with the protocols implementation nee…
KamenDimitrov97 Sep 30, 2024
229102b
Merge pull request #55 from flaxandteal/feature/parameter-type-improv…
KamenDimitrov97 Sep 30, 2024
0051264
fix: Fixed mypy pre-commit hook
KamenDimitrov97 Oct 11, 2024
ff529fe
docs(lint): explain type: ignore for ExprType
philtweir Jan 8, 2025
84bbe84
docs(lint): dataclass field types now cause mypy errors
philtweir Jan 8, 2025
b0635bf
docs(lint): ensure callback names unused parameter consistently for mypy
philtweir Jan 8, 2025
a56d3d9
Merge pull request #58 from flaxandteal/fix/mypy-hook-errors-for-new-…
philtweir Jan 13, 2025
8098da5
fix(ci): typo in workflow
philtweir Jan 16, 2025
bd2f625
fix(ci): typo in workflow
philtweir Jan 16, 2025
6abb6d5
fix(cwl): deal with fact we cannot comprehensively type-check to_outp…
philtweir Jan 18, 2025
71f06f6
wip(ci): confirm test failure on self-hosted
philtweir Jan 18, 2025
a586ebe
fix(annotations): better inspection, working-around cpython bugs
philtweir Jan 18, 2025
71aa956
fix(annotations): minor refactor
philtweir Jan 18, 2025
6d91f08
fix(inspection): function inspection does not work right with python>…
philtweir Jan 18, 2025
dbd29ee
fix(ci): restore runner to ubuntu-latest
philtweir Jan 18, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: tests
philtweir committed Aug 22, 2024
commit c6dc355b564a7e4a2ad06899148997ce618200cf
22 changes: 14 additions & 8 deletions src/dewret/renderers/cwl.py
Original file line number Diff line number Diff line change
@@ -28,6 +28,7 @@
from dewret.workflow import (
FactoryCall,
Reference,
FieldableMixin,
Raw,
Workflow,
BaseStep,
@@ -91,7 +92,7 @@ def from_reference(cls, ref: Reference) -> "ReferenceDefinition":
Args:
ref: reference to convert.
"""
return cls(source=ref.name)
return cls(source=to_name(ref))

def render(self) -> dict[str, RawType]:
"""Render to a dict-like structure.
@@ -392,11 +393,11 @@ def from_parameters(
"""
return cls(
inputs={
input.name: cls.CommandInputParameter(
label=input.name,
default=input.default,
input.__name__: cls.CommandInputParameter(
label=input.__name__,
default=input.__default__,
type=raw_to_command_input_schema(
label=input.name, value=input.default
label=input.__name__, value=input.__default__
),
)
for input in parameters
@@ -424,6 +425,11 @@ def render(self) -> dict[str, RawType]:
return result


def to_name(result: FieldableMixin):
if not result.__field__ and isinstance(result, StepReference):
return f"{result.__name__}/out"
return result.__name__

@define
class OutputsDefinition:
"""CWL-renderable set of workflow outputs.
@@ -450,12 +456,12 @@ def from_results(
return cls(
outputs=[
to_output_schema(
result.field, result.return_type, output_source=result.name
"/".join(result.__field__) or "out", result.__type__, output_source=to_name(result)
) for result in results
]
if isinstance(results, list | tuple) else {
key: to_output_schema(
result.field, result.return_type, output_source=result.name
"/".join(result.__field__) or "out", result.__type__, output_source=to_name(result)
)
for key, result in results.items()
}
@@ -524,7 +530,7 @@ def from_workflow(
outputs=OutputsDefinition.from_results(
workflow.result
if isinstance(workflow.result, list | tuple) else
{workflow.result.field: workflow.result}
{"/".join(workflow.result.__field__) or "out": workflow.result}
if workflow.has_result else
{}
),
20 changes: 9 additions & 11 deletions src/dewret/tasks.py
Original file line number Diff line number Diff line change
@@ -496,8 +496,8 @@ def add_numbers(left: int, right: int):
# We leave this reference dangling for a consumer to pick up ("tethered"), unless
# we are in a nested task, that does not have any existence of its own.
kwargs[var] = ParameterReference(
workflow,
param(
workflow=workflow,
parameter=param(
var,
value,
tethered=(
@@ -507,7 +507,7 @@ def add_numbers(left: int, right: int):
),
)
elif isinstance(value, Parameter):
kwargs[var] = ParameterReference(workflow, value)
kwargs[var] = ParameterReference(workflow=workflow, parameter=value)
original_kwargs = dict(kwargs)
fn_globals = analyser.globals

@@ -518,13 +518,13 @@ def add_numbers(left: int, right: int):
# raise TypeError(
# "Captured parameter {var} (global variable in task) shadows an argument"
# )
if analyser.is_at_construct_arg(var):
if analyser.is_at_construct_arg(var) or isinstance(value, Reference):
kwargs[var] = value
elif isinstance(value, Parameter):
kwargs[var] = ParameterReference(workflow, value)
elif is_raw(value):
kwargs[var] = ParameterReference(workflow=workflow, parameter=value)
elif is_raw(value) or ((attrs_has(value) or is_dataclass(value)) and not inspect.isclass(value)):
kwargs[var] = ParameterReference(
workflow, param(var, value, tethered=False)
workflow=workflow, parameter=param(var, value, tethered=False)
)
elif is_task(value) or ensure_lazy(value) is not None:
if not nested and _workaround_check_value_is_task(
@@ -546,8 +546,6 @@ def {fn.__name__}(...) -> ...:
...
"""
)
elif attrs_has(value) or is_dataclass(value):
...
elif nested:
raise NotImplementedError(
f"Nested tasks must now only refer to global parameters, raw or tasks, not objects: {var}"
@@ -565,8 +563,8 @@ def {fn.__name__}(...) -> ...:
nested_workflow = Workflow(name=fn.__name__)
nested_globals: Param.kwargs = {
var: ParameterReference(
nested_workflow,
param(
workflow=nested_workflow,
parameter=param(
var,
typ=(
value.__type__
354 changes: 211 additions & 143 deletions src/dewret/workflow.py

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions tests/test_cwl.py
Original file line number Diff line number Diff line change
@@ -153,7 +153,7 @@ def test_cwl_with_parameter() -> None:
workflow = construct(result)
rendered = render(workflow)["__root__"]
num_param = list(workflow.find_parameters())[0]
hsh = hasher(("increment", ("num", f"int|:param:{num_param.unique_name}")))
hsh = hasher(("increment", ("num", f"int|:param:{num_param._.unique_name}")))

assert rendered == yaml.safe_load(f"""
cwlVersion: 1.2
@@ -321,9 +321,9 @@ def test_cwl_references() -> None:
rendered = render(workflow)["__root__"]
num_param = list(workflow.find_parameters())[0]
hsh_increment = hasher(
("increment", ("num", f"int|:param:{num_param.unique_name}"))
("increment", ("num", f"int|:param:{num_param._.unique_name}"))
)
hsh_double = hasher(("double", ("num", f"increment-{hsh_increment}/out")))
hsh_double = hasher(("double", ("num", f"increment-{hsh_increment}")))

assert rendered == yaml.safe_load(f"""
cwlVersion: 1.2
64 changes: 64 additions & 0 deletions tests/test_fieldable.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
import yaml
from dataclasses import dataclass
from dewret.tasks import task, construct, subworkflow
from dewret.workflow import param
from dewret.renderers.cwl import render

from ._lib.extra import double, mod10, sum

@dataclass
class Sides:
left: int
right: int

SIDES: Sides = Sides(3, 6)

@subworkflow()
def sum_sides():
return sum(left=SIDES.left, right=SIDES.right)

def test_fields_of_parameters_usable() -> None:
result = sum_sides()
workflow = construct(result, simplify_ids=True)
rendered = render(workflow, allow_complex_types=True)["sum_sides-1"]

assert rendered == yaml.safe_load("""
class: Workflow
cwlVersion: 1.2
inputs:
SIDES/left:
label: SIDES/left
type: Sides
SIDES/right:
label: SIDES/right
type: Sides
outputs:
out:
label: out
outputSource: sum-1-1/out
type:
- int
- double
steps:
sum-1-1:
in:
left:
source: SIDES/left
right:
source: SIDES/right
out:
- out
run: sum
""")

def test_can_get_field_reference_iff_parent_type_has_field():
@dataclass
class MyDataclass:
left: int
my_param = param("my_param", typ=MyDataclass)
result = sum(left=my_param, right=my_param)
workflow = construct(result, simplify_ids=True)
param_reference = list(workflow.find_parameters())[0]

assert str(param_reference.left) == "my_param/left"
assert param_reference.left.__type__ == int
9 changes: 3 additions & 6 deletions tests/test_subworkflows.py
Original file line number Diff line number Diff line change
@@ -59,13 +59,11 @@ def get_global_queues(num: int | float) -> list["Queue[int] | int"]:
@subworkflow()
def add_constant(num: int | float) -> int:
"""Add a global constant to a number."""
print(CONSTANT, type(CONSTANT))
return to_int(num=sum(left=num, right=CONSTANT))

@subworkflow()
def add_constants(num: int | float) -> int:
"""Add a global constant to a number."""
print(CONSTANT, type(CONSTANT))
return to_int(num=sum(left=sum(left=num, right=CONSTANT), right=CONSTANT))


@@ -257,9 +255,8 @@ def test_subworkflows_can_return_lists() -> None:
num:
label: num
type: int
sum-1-1-1-right:
default: 3
label: sum-1-1-1-right
CONSTANT:
label: CONSTANT
type: int
outputs:
out:
@@ -272,7 +269,7 @@ def test_subworkflows_can_return_lists() -> None:
left:
source: num
right:
source: sum-1-1-1-right
source: CONSTANT
out:
- out
run: sum