Repository navigation
Conversation
A complex variable substituted into for_each_task.inputs was dropped by normalization with 'expected string, found sequence', leaving the required field unset. Serialize pure variable references to complex values into the JSON string the Jobs API expects, before variable resolution runs. Fixes databricks#6901.
|
Thanks for the contribution! This pull request was opened from a fork. If you're a member of the Databricks organization, please request access to the If you're an external contributor, no action is needed; a maintainer will review your pull request. |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved serialization and reference-resolution interactions can still drop inputs or alter literal data.
3 open findings
What changed in this PR
Adds JSON serialization for complex variables referenced by for_each_task.inputs, addressing the local type mismatch reported in #6901.
Changes:
- Adds a field-specific serializer before resource variable resolution.
- Adds unit tests and an acceptance reproduction.
- Documents the behavior in a changelog entry.
| File | Description |
|---|---|
| bundle/config/mutator/resourcemutator/serialize_for_each_task_inputs.go | Implements serialization. |
| bundle/config/mutator/resourcemutator/serialize_for_each_task_inputs_test.go | Tests serialization and passthrough behavior. |
| bundle/config/mutator/resourcemutator/resource_mutator.go | Integrates serialization into normalization. |
| bundle/config/mutator/resourcemutator/process_static_resources.go | Integrates serialization into static processing. |
| acceptance/bundle/variables/complex-foreach-inputs/test.toml | Configures ignored test artifacts. |
| acceptance/bundle/variables/complex-foreach-inputs/script | Validates serialized inputs. |
| acceptance/bundle/variables/complex-foreach-inputs/output.txt | Records expected command output. |
| acceptance/bundle/variables/complex-foreach-inputs/out.validate.json | Records expected configuration. |
| acceptance/bundle/variables/complex-foreach-inputs/out.test.toml | Records test execution settings. |
| acceptance/bundle/variables/complex-foreach-inputs/hello.py | Supplies the task fixture. |
| acceptance/bundle/variables/complex-foreach-inputs/databricks.yml | Defines the reproduction bundle. |
| .nextchanges/bundles/6901.md | Announces the fix. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Serializing before resolution missed inputs nested inside variable-defined tasks, and the reference guard did not cover map keys. Serialize after substitution in resolveOnce instead, and scan keys too.
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
Complex variables substituted into
for_each_task.inputswere dropped by normalization withexpected string, found sequence, leaving the required field unset. The Jobs API never saw a value.This serializes complex values at
for_each_task.inputsinside variable resolution, right after substitution and before normalization, into the JSON string the Jobs API expects. That covers both direct references (inputs: ${var.table_list}) and inputs nested inside variable-defined tasks (tasks: ${var.job_tasks}). Values that still contain references are skipped and keep the existing diagnostics, since baking them into JSON would change substitution into text interpolation. Validation in unrelated fields is unchanged.Tests: unit coverage for the serializer and the resolution pipeline (serialization, empty arrays, escaping, passthrough cases, preserved validation errors, variable-defined tasks), plus an acceptance test running both reference shapes through
bundle validate.Fixes #6901.