dbt-core This Week: Compute Routing, Snowflake WIF, Parser Gaps

dbt-core is the compiler and runner for dbt projects. The tree on main is the Rust Fusion rewrite, and this week it took 97 commits, including a bump to 2.0.0-beta.2. The useful slice is warehouse connections, auth, and YAML shapes that used to fail parse on Fusion while Python Core loaded them.

Databricks lets a model pick a SQL warehouse through databricks_compute. Fusion was running that SQL on the default connection anyway. A Databricks connection routing fix now reads the node config, fingerprints the connection that config would open, and refuses to reuse a pooled handle unless the fingerprint matches.

The pool used to pop any recycled connection and compare fingerprints after the fact. recycle_connection now takes the fingerprint as an argument and scans for a match. A profile with no per node override still uses the cached engine fingerprint from the adapter engine, so the extra Auth::configure() pass is skipped when nothing changed.

pub fn recycle_connection(
    node_id: Option<&String>,
    fingerprint: u64,
) -> Option<Box<dyn Connection>>

DuckDB is excluded from that pool. Parking a DuckDB handle for the life of the process left a file lock that collided with the next open of the same database file. The factory now drops DuckDB connections as soon as the caller is done. That is the right split: reuse for warehouses that can share sessions, hard close for an embedded file database.

Seeds still do not carry databricks_attr. Per model compute on seeds will keep falling back to the default warehouse until that schema gap is filled. Probe errors are logged instead of swallowed, which is better than a silent fallback, but you will still miss the warehouse you asked for if the node shape cannot be serialized.

Snowflake profiles can now set authenticator: workload_identity. The Snowflake WIF support commit accepts OIDC, AZURE, GCP, and AWS as workload_identity_provider. AWS and GCP fetch attestation from the local cloud metadata service. Azure is the only provider that may set workload_identity_entra_resource. The Snowflake Arrow ADBC driver was bumped in the same change, recorded in the ADBC drivers list.

authenticator: workload_identity
workload_identity_provider: AZURE
workload_identity_entra_resource: api://your-resource

On BigQuery, adapter.get_relation() could not find UDFs or routines even right after a successful create. Both the schema wide list and the single relation lookup only queried INFORMATION_SCHEMA.TABLES, which BigQuery never fills with routines. A BigQuery UDF lookup fix unions in INFORMATION_SCHEMA.ROUTINES, skips stored procedures, and maps BigQuery FUNCTION to RelationType::Function. The routine name is escaped as a SQL literal so a quote in the identifier cannot break out of the string.

If a model creates a UDF and a later node resolves it in the same run, that path should now match what Python Core already did. Replay tests and lookups that bypass the schema cache use the same ROUTINES fallback when ADBC GetTableSchema reports not found.

Fusion was rejecting project YAML that Python Core loaded without complaint. A YAML acceptance patch loosens the model config schema, plus the snapshot and seed siblings. column_types can be a sequence. event_time can be a mapping. policy_tags can be a scalar. Legacy data paths still parse.

That is compatibility, not new semantics. A mapping event_time is stored as a JSON string placeholder, matching Core’s event_time: Any. A sequence column_types entry is joined into one string. Parse no longer dies. The adapter still has to make sense of the value.

column_types:
  id:
    - integer
event_time:
  column: event_at

Two nearby parser fixes matter if you consume packages. A root project enabled overlay now honors +enabled: true in the root project over a package’s inline config(enabled=false). Versioned model column inheritance in the model resolver now defaults to include all model level columns when a version lists columns but no include or exclude rules. Version local {{ doc() }} descriptions are actually rendered.

If a project parses on Python Core and failed on Fusion last month, these three diffs are the first place to look. Fusion is still chasing Core’s YAML leniency. Treat the new deserializers as “parse no longer crashes”, not as a typed contract you should start emitting from macros.

Packages such as Elementary stash cache on the Jinja graph object. Core keeps graph as one dict on the Manifest for the whole invocation. Fusion’s parse context had graph as undefined, so graph.setdefault("elementary", {}) did nothing while hooks and model bodies were rendered to collect ref() and source() edges. A graph mutation persist fix hands parse the same mutable mapping used at compile and run, then resets it at the resolver entry so a long lived LSP process cannot leak one project’s scratch keys into the next.

State handling moved in the same window. Auto deferral can resolve unselected relations through the server, wired through the compilation entry. Reused State tests now show up in run_results.json instead of looking like a fresh pass. --empty now emits where false in the SQL, matching Core.

A large share of the 97 commits is docs v2 UI, swapping sourdough components for radix and adding Storybook. That work does not change compile or run. Skip it unless you ship the embedded docs server.

Databricks databricks_compute now implies a distinct connection. Watch warehouse counts and idle timeouts if many models each pick a different warehouse. The pool will not hand you a handle from warehouse A when the next node asked for warehouse B.

Fusion YAML acceptance is still a treadmill. Sequence column_types and mapping event_time parse, but they are stored as flattened strings. Do not start generating those shapes on purpose.

Snowflake WIF needs a current ADBC driver and a valid workload_identity_provider. Azure is the only provider that may set workload_identity_entra_resource. Everything else is a config error, not a silent ignore.