---
title: "dnaEPICO Overview"
author:
    -
        name: Paul Ruiz
        affiliation:
            - Queensland University of Technology
        email: ruizpint@qut.edu.au
output:
    BiocStyle::html_document:
        self_contained: yes
        toc: true
        toc_float: true
        toc_depth: 2
date: "`r Sys.Date()`"
package: dnaEPICO
resource_files:
    - preprocessingMinfiEwasWater_pipeline_overview.svg
    - svaEnmix_pipeline_overview.svg
    - preprocessingPheno_pipeline_overview.svg
    - methylationGLM_pipeline_overview.svg
    - methylationLME_pipeline_overview.svg
    - dnamReport_pipeline_overview.svg
vignette: >
    %\VignetteIndexEntry{1. dnaEPICO Overview}
    %\VignetteEncoding{UTF-8}
    %\VignetteEngine{knitr::rmarkdown}
---

```{r setup, include = FALSE}
startTime <- Sys.time()
knitr::opts_chunk$set(
    collapse = TRUE,
    comment = "#>",
    crop = NULL
)

library(dnaEPICO)
```

# Introduction

This vignette shows the roles, inputs, processing stages, and outputs of the
main `r Biocpkg("dnaEPICO")` functions.

Use the diagrams to choose a function and follow outputs between stages. See
the local-use and pipeline-use vignettes for arguments and executable examples.

Read each overview as follows:

- The arrows show the expected flow from inputs to derived objects and files.
- The grouped regions show the main analysis tasks performed inside a function.
- The final nodes show the outputs that are returned to R or written to disk.
- The same function can be used interactively or as part of a file-based
    pipeline, depending on the value of `saveOutputs`.

# Required knowledge

`r Biocpkg("dnaEPICO")` is built on core Bioconductor infrastructure for
high-dimensional genomic data, with a focus on Illumina DNA methylation arrays.
This vignette assumes familiarity with a general DNA methylation workflow. For
an introduction, see this tutorial:
<https://paulyrp.github.io/2025-cpgpneurogenomics-workshop/tutorial.html>.
It covers the main concepts and analysis steps.

Preprocessing and quality control are performed using established
Bioconductor tools, including `r Biocpkg("minfi")`, `r Biocpkg("ENmix")`, and
`r Biocpkg("wateRmelon")`. Downstream statistical modelling relies on base R
and CRAN frameworks, including generalised linear models and linear
mixed-effects models. Users are expected to have basic familiarity with R,
Bioconductor pipelines, command-line execution, and Illumina IDAT file
structures.

For an introduction to Bioconductor, see the
[installation guide](https://www.bioconductor.org/install/).

## Probe-exclusion reference files

The lists of excluded probes depend on the Illumina methylation-array platform.

- For **Illumina HumanMethylationEPIC v2.0**, use the cross-reactive
    probe-exclusion file from
    [Peters et al. (2024)](https://doi.org/10.1186/s12864-024-10027-5).

- For **Illumina MethylationEPIC**, also known as the **850k array**, use
    the probe-exclusion resources from
    [Pidsley et al. (2016)](https://doi.org/10.1186/s13059-016-1066-1).
    The supporting files commonly used together are
    `13059_2016_1066_MOESM1_ESM.csv`,
    `13059_2016_1066_MOESM4_ESM.csv`,
    `13059_2016_1066_MOESM5_ESM.csv`, and
    `13059_2016_1066_MOESM6_ESM.csv`.

- For **Illumina HumanMethylation450k**, use the cross-reactive and
    polymorphic probe resources from
    [Chen et al. (2013)](https://doi.org/10.4161/epi.23470).

- Multiple probe-exclusion files can be supplied as a semicolon-separated
    value in `probeExclusionPath`. `dnaEPICO` reads probe IDs from each file,
    uses `probeExclusionIdColumn` when supplied, or otherwise auto-detects
    common probe-ID columns such as `ProbeID`, `TargetID`, `IlmnID`, and
    `Name`. The unique union of all probe IDs is then used to filter the
    normalised object.

- For **EPICv2**, setting `useEpicV2Manifest = TRUE` also retrieves the
    expanded Peters et al. manifest from AnnotationHub resource
    [AH116484](https://bioconductor.org/packages/EPICv2manifest). Probes
    flagged in selected manifest columns are added to the same exclusion set.
    By default, probes flagged by `CH_WGBS_evidence`, `CH_BLAT`, or
    `MissingPos` are removed, while `MismatchPos` is retained unless explicitly
    enabled.

```{r overview-helpers, echo = FALSE, results = "asis"}
cat("
<style>
.overview-figure {
    margin: 1.5rem 0 2rem;
}

.overview-controls {
    display: flex;
    flex-wrap: wrap;
    gap: 8px;
    margin: 0 0 0.5rem;
}

.overview-zoom-button {
    background: #ffffff;
    border: 1px solid #2c7fb8;
    border-radius: 6px;
    color: #2c7fb8;
    cursor: pointer;
    font-size: 0.875rem;
    line-height: 1.2;
    padding: 5px 9px;
}

.overview-zoom-button:hover,
.overview-zoom-button:focus {
    background: #edf6fb;
}

.overview-zoom-value {
    align-items: center;
    color: #4b5563;
    display: inline-flex;
    font-size: 0.875rem;
    min-width: 44px;
}

.overview-svg-frame {
    background: #ffffff;
    border: 1px solid #d9e2ec;
    border-radius: 6px;
    max-height: 680px;
    overflow: auto;
    padding: 10px;
}

.overview-svg-frame img {
    display: block;
    height: auto;
    max-width: none;
    width: calc(1600px * var(--overview-scale, 1));
}

.overview-figure:fullscreen {
    background: #ffffff;
    margin: 0;
    overflow: hidden;
    padding: 16px;
}

.overview-figure:fullscreen .overview-svg-frame {
    height: calc(100vh - 88px);
    max-height: none;
}

@media (max-width: 768px) {
    .overview-svg-frame {
    max-height: 520px;
    }

    .overview-svg-frame img {
    width: calc(1200px * var(--overview-scale, 1));
    }
}
</style>
<script>
function setOverviewZoom(figureId, scale) {
    var figure = document.getElementById(figureId);
    if (!figure) {
    return;
    }

    var next = Math.max(0.5, Math.min(3, scale));
    figure.dataset.scale = next.toFixed(2);
    figure.style.setProperty('--overview-scale', next.toString());

    var label = figure.querySelector('.overview-zoom-value');
    if (label) {
    label.textContent = Math.round(next * 100) + '%';
    }
}

function changeOverviewZoom(figureId, delta) {
    var figure = document.getElementById(figureId);
    var current = figure ? parseFloat(figure.dataset.scale || '1') : 1;
    setOverviewZoom(figureId, current + delta);
}

function openOverviewFullscreen(figureId) {
    var figure = document.getElementById(figureId);
    if (!figure || !figure.requestFullscreen) {
    return;
    }

    figure.requestFullscreen();
}
</script>
")

overview_svgs <- c(
    preprocessingMinfiEwasWater =
        "preprocessingMinfiEwasWater_pipeline_overview.svg",
    svaEnmix =
        "svaEnmix_pipeline_overview.svg",
    preprocessingPheno =
        "preprocessingPheno_pipeline_overview.svg",
    methylationGLM =
        "methylationGLM_pipeline_overview.svg",
    methylationLME =
        "methylationLME_pipeline_overview.svg",
    dnamReport =
        "dnamReport_pipeline_overview.svg"
)

html_escape <- function(x) {
    x <- gsub("&", "&amp;", x, fixed = TRUE)
    x <- gsub("<", "&lt;", x, fixed = TRUE)
    x <- gsub(">", "&gt;", x, fixed = TRUE)
    x <- gsub('"', "&quot;", x, fixed = TRUE)
    x
}

svg_to_html <- function(svg_path, title = NULL) {
    if (!file.exists(svg_path)) {
        stop("Missing overview SVG: ", svg_path)
    }

    if (is.null(title)) {
        title <- sub(
            "_pipeline_overview$",
            "() overview",
            tools::file_path_sans_ext(basename(svg_path))
        )
    }

    figure_id <- gsub("[^A-Za-z0-9_-]", "-", tools::file_path_sans_ext(
        basename(svg_path)
    ))

    cat(sprintf(
        '<div class="overview-figure" id="%s" data-scale="1">\n',
        figure_id
    ))
    cat('<div class="overview-controls">\n')
    cat(sprintf(
        paste0(
            '<button class="overview-zoom-button" type="button" ',
            'onclick="changeOverviewZoom(\'%s\', 0.25)">',
            'Zoom +</button>\n'
        ),
        figure_id
    ))
    cat(sprintf(
        paste0(
            '<button class="overview-zoom-button" type="button" ',
            'onclick="changeOverviewZoom(\'%s\', -0.25)">',
            'Zoom -</button>\n'
        ),
        figure_id
    ))
    cat(sprintf(
        paste0(
            '<button class="overview-zoom-button" type="button" ',
            'onclick="setOverviewZoom(\'%s\', 1)">',
            'Reset</button>\n'
        ),
        figure_id
    ))
    cat(sprintf(
        paste0(
            '<button class="overview-zoom-button" type="button" ',
            'onclick="openOverviewFullscreen(\'%s\')">',
            'Full screen</button>\n'
        ),
        figure_id
    ))
    cat('<span class="overview-zoom-value">100%</span>\n')
    cat('</div>\n')
    cat('<div class="overview-svg-frame">\n')
    cat(sprintf(
        '<img src="%s" alt="%s">\n',
        html_escape(svg_path),
        html_escape(title)
    ))
    cat('\n</div>\n</div>\n')
}
```

# Functions

The main functions follow the analysis path from preprocessing and
surrogate-variable estimation through phenotype preparation, modelling, and
report generation.

## preprocessingMinfiEwasWater()

`preprocessingMinfiEwasWater()` reads the phenotype table and IDAT files,
builds the methylation
objects, performs quality control and normalisation, filters probes, and
estimates cell composition.

Its role in the package is to create analysis-ready methylation data:

- Inputs: phenotype metadata, IDAT files, array annotation, and filtering
    settings.
- Processing tasks: import, quality control, normalisation, probe filtering,
    metric extraction, and cell-composition estimation. When
    `removeSexMismatch = TRUE`, samples are removed only when both reported and
    predicted sex are available and disagree; samples with missing or unknown
    sex information remain because their mismatch status cannot be determined.
- Outputs: filtered phenotype data, `RGSet`, beta values, M-values,
    copy-number values, quality-control figures, and `phenoLC`. The log reports
    the observed finite minimum and maximum for each methylation matrix without
    modifying its values.

```{r preprocessingMinfiEwasWater-overview, echo = FALSE, results = "asis"}
svg_to_html(
    overview_svgs[["preprocessingMinfiEwasWater"]]
)
```

## svaEnmix()

`svaEnmix()` estimates surrogate variables from control-probe information and
adds them to the phenotype table. This step helps represent technical
variation that may otherwise influence downstream association models.

Its role is to prepare covariates for batch and technical adjustment:

- Inputs: the phenotype table from preprocessing and the saved `RGSet`.
- Processing tasks: control-probe extraction, surrogate-variable
    estimation, association checks with array-position metadata, and phenotype
    merging.
- Outputs: surrogate-variable matrices, an updated phenotype table, and
    diagnostic summaries.

```{r svaEnmix-overview, echo = FALSE, results = "asis"}
svg_to_html(
    overview_svgs[["svaEnmix"]])
```

## preprocessingPheno()

`preprocessingPheno()` aligns phenotype information with methylation metrics.
It prepares timepoint-specific data, combines longitudinal records, and
creates export-ready files for external methylation-age tools.

Its role is to organise samples and methylation matrices for modelling:

- Inputs: phenotype data, beta values, M-values, copy-number values, and
    timepoint settings.
- Processing tasks: sample alignment, timepoint splitting, longitudinal
    merging, and Clock Foundation export preparation.
- Outputs: timepoint-specific phenotype-methylation tables, combined
    longitudinal tables, and Clock Foundation input files.

```{r preprocessingPheno-overview, echo = FALSE, results = "asis"}
svg_to_html(
    overview_svgs[["preprocessingPheno"]])
```

## methylationGLM()

`methylationGLM()` fits cross-sectional methylation association models. It
is designed for analyses where one phenotype is tested against CpG-level
methylation while adjusting for selected covariates.

Its role is to run single-timepoint association testing:

- Inputs: a phenotype-methylation table, phenotype variables,
    covariates, categorical and scaled-variable selections, optional PRS
    mappings, and annotation settings.
- Processing tasks: model preparation, CpG-GLM fitting, coefficient and
    optional omnibus testing, diagnostic plotting, multiple-testing adjustment,
    summarisation, and annotation.
- Outputs: compact phenotype summaries, summary tables, diagnostic figures,
    significant-CpG exports, and annotated workbooks. Native glm2 conditions
    are stored in `<Phenotype>_Model.Message`; CpGs without a returned p-value
    remain in the compact summary and are counted in workbook metadata.
    GRCh38 analyses can optionally append release-aware GENCODE gene-body and
    nearest-TSS annotations through AnnotationHub.

```{r methylationGLM-overview, echo = FALSE, results = "asis"}
svg_to_html(
    overview_svgs[["methylationGLM"]])
```

## methylationLME()

`methylationLME()` fits longitudinal mixed-effects models. It supports
repeated-measures designs with a participant-level random intercept and
timepoint-related fixed effects. For lmerTest/lme4 models, an optional omnibus
F test jointly evaluates all estimable coefficients for a phenotype main
effect or phenotype-by-interaction term.
If the participant column is absent, the configured `SampleID` column can be
used to derive it from identifiers ending in the supported `A`/`B` visit code.

Its role is to model methylation change across repeated observations:

- Inputs: combined longitudinal phenotype-methylation data, participant
    identifiers, timepoint variables, phenotypes, covariates, categorical and
    scaled-variable selections, and annotation settings.
- Processing tasks: longitudinal model preparation, CpG-LME model
    fitting, coefficient and optional omnibus testing, diagnostic plotting,
    summarisation, multiple-testing adjustment, and annotation.
- Outputs: compact phenotype summaries, interaction summaries, diagnostic
    figures, significant-interaction exports, and annotated longitudinal
    workbooks.
    Native lmerTest/lme4 or nlme conditions are stored in
    `<Phenotype>_Model.Message`; CpGs without a returned coefficient or omnibus
    p-value remain in the compact summary and are counted in workbook
    metadata. The same optional AnnotationHub annotation is supported by both
    the lmerTest/lme4 and nlme engines for GRCh38 results.

```{r methylationLME-overview, echo = FALSE, results = "asis"}
svg_to_html(
    overview_svgs[["methylationLME"]])
```

## dnamReport()

`dnamReport()` assembles the main tables, figures, logs, and model summaries
into a report website. It can be run after preprocessing and modelling
outputs have been written to disk.

Its role is to make the package outputs easier to inspect and share:

- Inputs: phenotype tables, QC figures, ENmix figures, SVA figures,
    metric figures, model annotation tables, detection P-value data, and logs.
- Processing tasks: input validation, report file preparation, Quarto
    rendering, and post-processing.
- Outputs: a responsive dark report site with persistent navigation, a wide
    central table or figure panel, contextual notes and controls, and organised
    tabs for data, preprocessing, modelling, logs, and supporting diagnostics.

```{r dnamReport-overview, echo = FALSE, results = "asis"}
svg_to_html(
    overview_svgs[["dnamReport"]])
```

# Summary

The preprocessing functions create quality-controlled methylation data and
analysis-ready phenotype tables. The modelling functions fit cross-sectional and
longitudinal association models. The report function gathers the resulting
tables, figures, and logs into a browsable output.

The main workflow is:

- run `preprocessingMinfiEwasWater()` to prepare methylation objects and QC
    outputs,
- run `svaEnmix()` when control-probe surrogate variables are needed,
- run `preprocessingPheno()` to prepare modelling tables,
- run `methylationGLM()` or `methylationLME()` for association
    testing, and
- run `dnamReport()` to review the completed outputs.

# Basics

Date the vignette was generated.

```{r vignette-date, echo = FALSE}
Sys.time()
```

Wallclock time spent generating the vignette.

```{r vignette-wallclock, echo = FALSE}
totalTime <- diff(c(startTime, Sys.time()))
round(totalTime, digits = 3)
```

`R` session information.

```{r vignette-session-info, echo = FALSE}
sessionInfo()
```

## Asking for help

Use the [Bioconductor support site](https://support.bioconductor.org/) for
package questions. Add the `dnaEPICO` tag, review
[previous posts](https://support.bioconductor.org/tag/dnaEPICO/), and include a
small reproducible example with session information.
