KADAIF: an anomaly detection method for complex microbiome data

Motivation The gut microbiome plays an important role in human health and disease, prompting large-scale studies that generate extensive datasets. A critical preprocessing step in analysing such datasets is anomaly detection, which aims to identify erroneous samples and prevent misleading statistical outcomes. Microbiome data, however, pose unique challenges such as compositionality, sparsity, interdependencies, and high dimensionality, limiting the effectiveness of conventional methods and highlighting the need for specifically-tailored approaches for anomaly detection in microbiome data.Implementation To address this challenge, we introduce KADAIF, a microbiome-specific anomaly detection method that generalizes the common Isolation Forest (IF) approach. As in IF, KADAIF builds an ensemble of trees, each recursively partitioning the data along randomly selected features, and measures the average depth at which samples are isolated, assuming that anomalous samples will be isolated closer to the root. Unlike IF, however, KADAIF partitions samples based on subsets of features (coupled with dimensionality reduction), addressing microbiome-specific properties such as sparsity and species interactions.Results We evaluate KADAIF by simulating common scenarios that introduce anomalous behavior, demonstrating that KADAIF outperforms alternative methods across various settings and datasets. Furthermore, we show that KADAIF outperforms IF in detecting anomalies also in other types of high-dimensional sparse biological data. Finally, we show KADAIF's application for identifying disease onset in longitudinal microbiome data and for partitioning cases versus controls based on the Anna Karenina principle. Combined, our work highlights KADAIF's potential to enhance microbiome data processing and downstream analyses, with beneficial implications for precision medicine studies.Availability and implementation An implementation of KADAIF, as well as all the code used for the analysis, is available on GitHub (https://github.com/borenstein-lab/KADAIF).