DuckDBArray 0.99.7
This vignette is for developers: it documents how the DuckDB backend satisfies the DelayedArray seed contract and how array operations become SQL, so you can maintain the package, extend it, or implement a similar backend. For day-to-day use see Introduction to DuckDBArray; for performance see Benchmarking DuckDBArray.
The design follows the pattern DelayedArray prescribes for backends (see its
Implementing a DelayedArray backend vignette): a lightweight seed class that
knows how to read blocks of data, wrapped by the standard DelayedArray /
DelayedMatrix classes that supply the lazy-evaluation machinery.
library(DuckDBArray)
library(DelayedArray)DuckDBArray layers three classes on the DuckDBDataFrame foundation:
| Class | Extends | Role |
|---|---|---|
DuckDBArraySeed |
Array |
the backend seed: wraps a DuckDBTable, reads blocks |
DuckDBArray |
DelayedArray |
user-facing N-dimensional array |
DuckDBMatrix |
DelayedMatrix |
2-D specialization |
The seed holds no data; it holds a DuckDBTable (from
DuckDBDataFrame) describing a Parquet file in coordinate (COO)
form, one or more key columns giving each entry’s position and a single
data column giving its value. DelayedArray calls the seed’s extract_array() /
extract_sparse_array() methods to materialize only the blocks an operation
needs; everything else (subsetting, arithmetic, log1p(), …) is recorded lazily
by DelayedArray and resolved against the seed on demand.
A DelayedArray seed must answer three questions: its shape, and how to read a
dense or a sparse block. DuckDBArraySeed implements all three.
dim() and dimnames()The dimensions come from the key columns: each key column is one dimension, its levels are that dimension’s extent and names.
array_df <- expand.grid(i = 1:4, j = 1:3, k = 1:2)
array_df$value <- rpois(nrow(array_df), lambda = 2)
array_path <- tempfile(fileext = ".parquet")
arrow::write_parquet(array_df, array_path)
seed <- DuckDBArraySeed(array_path, datacol = "value",
keycols = list(i = 1:4, j = 1:3, k = 1:2))
dim(seed)
#> [1] 4 3 2extract_array()extract_array(seed, index) returns an ordinary array for the requested
index (a list of integer subscripts, one per dimension). Internally it subsets
the underlying DuckDBTable to those positions, materializes the result with
as.data.frame(), scatters the values into an array of the block’s shape, and
fills absent coordinates with the seed’s fill value.
extract_array(seed, list(1:2, 1:2, 1))
#> , , k = 1
#>
#> j
#> i 1 2
#> 1 1 4
#> 2 1 0extract_sparse_array()Because the data is stored as COO, sparse extraction is natural: the seed returns
a COO_SparseArray (from SparseArray) built directly from the rows
DuckDB returns, with no densification.
sparse_df <- data.frame(i = c(1, 1, 2, 4), j = c(1, 3, 2, 3),
k = c(1, 1, 2, 1), value = c(5, 3, 8, 7))
sparse_path <- tempfile(fileext = ".parquet")
arrow::write_parquet(sparse_df, sparse_path)
sparse_seed <- DuckDBArraySeed(sparse_path, datacol = "value",
keycols = list(i = 1:4, j = 1:3, k = 1:2))
extract_sparse_array(sparse_seed, list(1:4, 1:3, 1))
#> <4 x 3 x 1 SparseArray> of type "double" [nzcount=3 (25%)]:
#> ,,1
#> j
#> i 1 2 3
#> 1 5 0 3
#> 2 0 0 0
#> 3 0 0 0
#> 4 0 0 7is_sparse() on the seed reports whether sparse extraction is available, which is
how DelayedArray decides to call extract_sparse_array() in preference to
extract_array().
DuckDBArray() / DuckDBMatrix() wrap a seed (or construct one from a path) in
the standard DelayedArray classes:
arr <- DuckDBArray(seed)
class(arr)
#> [1] "DuckDBArray"
#> attr(,"package")
#> [1] "DuckDBArray"
seedClass <- class(seed(arr))
seedClass
#> [1] "DuckDBArraySeed"
#> attr(,"package")
#> [1] "DuckDBArray"From here the full DelayedArray API applies unchanged, lazy
arithmetic, subsetting, blockApply(), realize(), because DuckDBArray
is a DelayedArray.
The performance story rests on one idea: rather than let DelayedArray’s generic
block machinery loop over blocks, DuckDBArray overrides the
reductions so they become single SQL aggregations on the DuckDBTable. A
rowSums() on a DuckDBMatrix is
SELECT <row-key>, SUM(value) FROM table GROUP BY <row-key>and rowVars() the same with VAR_SAMP(value). The matrixStats /
MatrixGenerics methods (rowSums/colSums, rowMeans, rowVars,
rowSds, rowMins/rowMaxs, rowNnzs/colNnzs, rowDeviances) are all
implemented this way, at the DuckDBTable level, so DuckDB does the scan and the
arithmetic and only the (small) per-margin result crosses back into R.
mat <- as(arr[, , 1], "DuckDBMatrix")
rowSums(mat)
#> i
#> 1 2 3 4
#> 6 5 4 10Operations without an aggregation form (element-wise math, subsetting) fall back
to DelayedArray’s lazy machinery over extract_array(), correct for any
backend, and cheap because the seed reads only the needed blocks.
For large arrays, createDimTables() builds per-dimension lookup tables that map
indices to grid partitions, letting DuckDB prune to just the Parquet partitions a
query touches (and enabling independent, parallel processing of partitions).
library(S4Arrays)
grid <- RegularArrayGrid(dim(state.x77), c(10, 4))
dim_tables <- createDimTables(state.x77, grid = grid)
dim_tables
#> $index1
#> index1_group
#> 1 1
#> 2 1
#> 3 1
#> 4 1
#> 5 1
#> 6 1
#> 7 1
#> 8 1
#> 9 1
#> 10 1
#> 11 2
#> 12 2
#> 13 2
#> 14 2
#> 15 2
#> 16 2
#> 17 2
#> 18 2
#> 19 2
#> 20 2
#> 21 3
#> 22 3
#> 23 3
#> 24 3
#> 25 3
#> 26 3
#> 27 3
#> 28 3
#> 29 3
#> 30 3
#> 31 4
#> 32 4
#> 33 4
#> 34 4
#> 35 4
#> 36 4
#> 37 4
#> 38 4
#> 39 4
#> 40 4
#> 41 5
#> 42 5
#> 43 5
#> 44 5
#> 45 5
#> 46 5
#> 47 5
#> 48 5
#> 49 5
#> 50 5
#>
#> $index2
#> index2_group
#> 1 1
#> 2 1
#> 3 1
#> 4 1
#> 5 2
#> 6 2
#> 7 2
#> 8 2Dimension tables are attached to a seed via the dimtbls= argument (this is what
writeParquet() in BiocDuckDB does when it writes an experiment
object) and read back with dimtbls().
A DuckDBArray interoperates with the rest of the DelayedArray ecosystem: it can
be realized to another backend or to an ordinary array, and combined in delayed
expressions with other seeds.
as.array(arr[1:2, 1:2, 1])
#> j
#> i 1 2
#> 1 1 4
#> 2 1 0Because the on-disk representation is plain Parquet, the same files are readable outside R (Python, Julia, DuckDB itself, cloud query engines), a property the BiocDuckDB package relies on for its language-neutral on-disk experiment format.
sessionInfo()
#> R version 4.6.1 (2026-06-24)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 24.04.4 LTS
#>
#> Matrix products: default
#> BLAS: /home/biocbuild/bbs-3.24-bioc/R/lib/libRblas.so
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.12.0 LAPACK version 3.12.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_GB LC_COLLATE=C
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: America/New_York
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats4 stats graphics grDevices utils datasets methods
#> [8] base
#>
#> other attached packages:
#> [1] DuckDBArray_0.99.7 DelayedArray_0.39.5 SparseArray_1.13.2
#> [4] S4Arrays_1.13.0 abind_1.4-8 MatrixGenerics_1.25.0
#> [7] matrixStats_1.5.0 Matrix_1.7-6 DuckDBDataFrame_0.99.20
#> [10] IRanges_2.47.2 S4Vectors_0.51.6 BiocGenerics_0.59.12
#> [13] generics_0.1.4 bit64_4.8.2 BiocStyle_2.41.0
#>
#> loaded via a namespace (and not attached):
#> [1] sass_0.4.10 lattice_0.23-1 digest_0.6.39
#> [4] magrittr_2.0.5 evaluate_1.0.5 grid_4.6.1
#> [7] bookdown_0.47 blob_1.3.0 fastmap_1.2.0
#> [10] jsonlite_2.0.0 DBI_1.3.0 BiocManager_1.30.27
#> [13] purrr_1.2.2 jquerylib_0.1.4 duckdb_1.5.5
#> [16] cli_3.6.6 rlang_1.3.0 dbplyr_2.6.0
#> [19] XVector_0.53.0 withr_3.0.3 cachem_1.1.0
#> [22] yaml_2.3.12 otel_0.2.0 tools_4.6.1
#> [25] dplyr_1.2.1 assertthat_0.2.1 vctrs_0.7.3
#> [28] R6_2.6.1 lifecycle_1.0.5 bit_4.6.0
#> [31] arrow_25.0.0 pkgconfig_2.0.3 bslib_0.12.0
#> [34] pillar_1.11.1 glue_1.8.1 xfun_0.60
#> [37] tibble_3.3.1 tidyselect_1.2.1 knitr_1.51
#> [40] htmltools_0.5.9 rmarkdown_2.31 compiler_4.6.1