Hamilton Tightens Driver Lookup, Column Validation, and Graph Display

Hamilton is a Python framework for declaring dataflows as functions, and this recent activity is small but useful for teams that inspect DAGs, wrap drivers, and keep column transforms predictable. Across three commits, the project added public graph lookup helpers, fixed a validation gap in column injection, and made Graphviz output clearer for dynamic parallel execution.

The driver variable lookup helpers add two small methods in hamilton/driver.py. Driver.get_variable(name) returns the public Variable for one graph node and raises KeyError when the name is missing. Driver.get_graph() returns a HamiltonGraph view over the driver graph.

This is not a large API surface. That is the point. Operators and tool authors often need to answer basic questions before execution: is this input external, what node owns this variable, what does the graph look like after config has been applied. Before this change, callers could use list_available_variables, but single variable access meant filtering a list or reaching into internal graph state.

The test addition in tests/test_hamilton_driver.py is also useful. It asserts that input a is external, node b is present, and a missing name still fails with KeyError. That gives wrapper code a crisp contract. If an orchestration layer asks for a variable that is not in the DAG, it gets a normal Python lookup failure rather than a quiet empty result.

The exactly one validation change fixes with_columns_base argument handling in hamilton/function_modifiers/recursive.py. The constructor now counts how many of pass_dataframe_as, columns_to_pass, and on_input are set. Anything other than one raises ValueError.

The previous condition only caught a narrower shape. That matters because these arguments decide how a dataframe enters a subdag and which columns get passed into generated nodes. If two selectors are set at once, the user intent is unclear. If none are set, the subdag has no explicit route for the dataframe. Both cases are better rejected during construction than debugged after a pipeline run builds the wrong graph.

The new tests in tests/function_modifiers/test_recursive.py cover no argument, two arguments, all three arguments, and each valid single argument case. This is the right test shape for this kind of change. The behavior is a small guardrail, but it protects a part of Hamilton that can create many generated nodes from one decorator.

The newest commit, a collect edge styling fix, changes how hamilton/graph.py styles Graphviz edges for Collect[...] nodes. The code now checks whether a dependency is the actual collect_dependency before applying the collect edge style. Other inputs into the same collect node keep normal edges.

That distinction is small on paper and very visible in a graph. Dynamic parallelism can put several inputs near a collect node. If every incoming edge gets the collect marker, the rendered DAG suggests that every input participates in the expansion and collection pattern. The fix limits the crow tail style to the edge from double into summed in the test module, while keeping supporting inputs plain.

tests/test_graph.py captures the exact Graphviz lines. The test checks double -> summed for the collect style, number_to_repeat -> double for the expand style, and two other inputs into summed as plain edges. That is more valuable than a vague snapshot test here. It pins the semantic meaning of each edge.

This is only 6 files changed, with 121 insertions and 6 deletions. It still touches three places that matter in production data work: public driver inspection, dataframe column injection, and DAG visualization.

For data engineering teams, the shared theme is earlier feedback. get_variable lets integration code check one symbol directly. The with_columns_base validation moves invalid decorator use to object creation. The graph display change makes visual review of dynamic parallel DAGs less misleading. None of these changes alters the core execution model, which is a good property for a maintenance week.

The repository link is Hamilton, and this activity reads like normal framework hardening rather than a release push. That is useful context. Small API and validation changes tend to be where data workflow libraries become easier to operate over time.

First, any internal wrapper that has been reading driver.graph.nodes directly can now consider get_variable and get_graph. The old access path may still work, but the new methods are a cleaner public boundary.

Second, existing uses of with_columns should be checked if they relied on loose combinations of pass_dataframe_as, columns_to_pass, and on_input. The new rule is exact. One of the three should be set.

Third, teams that use Hamilton graph images for review should refresh diagrams for dynamic parallel jobs. The rendered graph may now show fewer collect styled edges, which is the intended correction.