Skip to content

feat(duckdb): Add transpilation support for ARRAY_UNION_AGG function#7495

Open
fivetran-amrutabhimsenayachit wants to merge 1 commit intomainfrom
RD-1069214-array-union-agg
Open

feat(duckdb): Add transpilation support for ARRAY_UNION_AGG function#7495
fivetran-amrutabhimsenayachit wants to merge 1 commit intomainfrom
RD-1069214-array-union-agg

Conversation

@fivetran-amrutabhimsenayachit
Copy link
Copy Markdown
Collaborator

Snowflake's ARRAY_UNION_AGG computes the set-union of multiple input arrays across rows — returning one flat array of distinct elements. DuckDB has no equivalent function. SQLGlot was emitting it verbatim, causing DuckDB to raise a Catalog Error at runtime.

The window form added a second wrinkle: even if we replaced the inner function correctly, DuckDB requires the OVER (...) clause to wrap the innermost aggregate (LIST), not the outer transformation functions. Simply swapping the function name and appending OVER would produce structurally invalid SQL.

Fix

Two changes to the DuckDB generator:

  1. Regular aggregate form — Added arrayunionagg_sql, which rewrites ARRAY_UNION_AGG(col) as LIST_DISTINCT(FLATTEN(LIST(col))). LIST collects all input arrays into a list-of-lists, FLATTEN unwraps them into one flat list, and LIST_DISTINCT removes duplicates — together approximating the set-union semantics.

  2. Window form — Extended the existing window_sql override to intercept ARRAY_UNION_AGG(...) OVER (...). It deep-copies the Window node, swaps the inner function from ArrayUnionAgg to LIST, then wraps the restructured window in FLATTEN and LIST_DISTINCT — producing LIST_DISTINCT(FLATTEN(LIST(col) OVER (PARTITION BY grp))) where the OVER correctly wraps only the aggregate.

Known limitation: DuckDB LIST_DISTINCT uses strict set semantics. Snowflake's multiset semantics (where duplicates from a single input array are preserved) cannot be reproduced in DuckDB with built-in functions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant