Probabilistic Graphical Models

Conditional independence tests have received special attention lately in machine learning and computational intelligence related literature as an important indicator of the relationship among the variables used by their models. In the field of probabilistic graphical models, which includes Bayesian network models, conditional independence tests are especially important for the task of learning the probabilistic graphical model structure from data. In this paper, we propose the full Bayesian significance test for tests of conditional independence for discrete datasets. The full Bayesian significance test is a powerful Bayesian test for precise hypothesis, as an alternative to the frequentist's significance tests (characterized by the calculation of the p-value).

Uncertainty quantification has become an important factor in understanding the data representations produced by Graph Neural Networks (GNNs). Despite their predictive capabilities being ever useful across industrial workspaces, the inherent uncertainty induced by the nature of the data is a huge mitigating factor to GNN performance. While aleatoric uncertainty is the result of noisy and incomplete stochastic data such as missing edges or over-smoothing, epistemic uncertainty arises from lack of knowledge about a system or model (e.g., a graph's topology or node feature representation), which can be reduced by gathering more data and information. In this paper, we propose an original new framework in which node-level epistemic uncertainty is modelled in a belief function (finite random set) formalism. The resulting Random-Set Graph Neural Networks have a belief-function head predicting a random set over the list of classes, from which both a precise probability prediction and a measure of epistemic uncertainty can be obtained. Extensive experiments on 9 different graph learning datasets, including real-world autonomous driving benchmarks as such Nuscene and ROAD, demonstrate RS-GNN's superior uncertainty quantification capabilities.

We devise a novel inference algorithm to effectively solve the cancer progression model reconstruction problem. Our empirical analysis of the accuracy and convergence rate of our algorithm, CAncer PRogression Inference (CAPRI), shows that it outperforms the state-of-the-art algorithms addressing similar problems. Motivation: Several cancer-related genomic data have become available (e.g., The Cancer Genome Atlas, TCGA) typically involving hundreds of patients. At present, most of these data are aggregated in a cross-sectional fashion providing all measurements at the time of diagnosis. Our goal is to infer cancer ?progression? models from such data. These models are represented as directed acyclic graphs (DAGs) of collections of ?selectivity? relations, where a mutation in a gene A ?selects? for a later mutation in a gene B. Gaining insight into the structure of such progressions has the potential to improve both the stratification of patients and personalized therapy choices. Resul...

The creation of a financial pitch book is a complex and informationintensive task, which requires analysts to gather data from disparate sources and synthesize it into a compelling evidence-based narrative. Current tools support document creation but not the underlying cognitive processes of information synthesis and narrative structuring. We introduce "BayesDeck, " an interactive IR/NLP system designed to assist financial analysts in this narrative construction process. BayesDeck's core innovation is a Bayesian narrative model that probabilistically reasons about the relevance and impact of various data points (e.g., market size, competitive landscape, financial metrics) on a proposed investment thesis. The system proactively retrieves and suggests data from a multi-source corpus, not merely based on keyword matching, but on the inferred narrative contribution to the evolving pitch. The interactive interface allows analysts to: (1) specify a high-level investment thesis, (2) view the system's Bayesian "reasoning trail" for suggested data points, visualizing the strength of evidence for and against the thesis, and (3) accept, reject, or critique suggestions, thereby continuously refining the model and the emerging narrative structure. We evaluated BayesDeck through a controlled laboratory study with 18 financial analysts, comparing it to a baseline system without the Bayesian reasoning component and a traditional web-searchdriven workflow. Our findings indicate that the Bayesian model significantly enhances the human-AI collaboration. Analysts using BayesDeck produced narratives rated 24% higher in argumentative coherence and evidential support by expert blind reviewers. Furthermore, qualitative feedback revealed that the transparent, probabilistic reasoning of the Bayesian model fostered greater user trust and a sense of agency, transforming the process from one of document assembly to one of collaborative reasoning and storytelling. This work demonstrates the profound potential of embedding scrutable, probabilistic models into interactive IR systems to support complex human information synthesis and narrative tasks. CCS Concepts • Human-centered computing → User models.

A novel hierarchical crowdsourcing-based system for weed identification.Combines image processing with crowdsourcing weed identification.Framework for unsupervised determination of crowd hierarchy.Prototype that supports low cost and accurate weed identification. Weed infestation is a common problem in agriculture that adversely affects crop production. Given severe constraints on the budget of many land-grant universities due to the economic downturn, extension services or agencies responsible for educating farmers and assisting them with the application of advancements in agricultural research, have taken a hit. To adapt to the current economic climate without adversely affecting the quality of programs for weed management, we present a hierarchical system that uses images captured using a smartphone application, a backend image processing algorithm, and two levels of crowdsourcing to identify weed images. The first of the two crowdsourcing levels consist of a non-expert crowd con...

Weed infestation is a common problem in agriculture that adversely affects crop production. Given severe constraints on the budget of many land-grant universities due to the economic downturn, extension services or agencies responsible for educating farmers and assisting them with the application of advancements in agricultural research, have taken a hit. To adapt to the current economic climate without adversely affecting the quality of programs for weed management, we present a hierarchical system that uses images captured using a smartphone application, a backend image processing algorithm, and two levels of crowdsourcing to identify weed images. The first of the two crowdsourcing levels consist of a non-expert crowd contributed by Amazon Mechanical Turk (AMT) and the second level consists of a crowd composed of experts such as county extension agents. We present a probabilistic decision engine to determine the suitability of two levels of crowdsourcing for identifying the weed image. We have evaluated the designed system using test weed images and we show that 80% of the weeds in our test set can be identified using the low cost AMT crowd while incurring a maximum latency of 3 h. Our system can help reduce the loses caused by the delay in identifying weeds, and hence, lead to quick remedial control practices applied to contain weed infestations.

Synthetic financial data provides a practical solution to the privacy, accessibility, and reproducibility challenges that often constrain empirical research in quantitative finance. This paper investigates the use of deep generative models, specifically Time-series Generative Adversarial Networks (TimeGAN) and Variational Autoencoders (VAEs) to generate realistic synthetic financial return series for portfolio construction and risk modeling applications. Using historical daily returns from the S and P 500 as a benchmark, we generate synthetic datasets under comparable market conditions and evaluate them using statistical similarity metrics, temporal structure tests, and downstream financial tasks. The study shows that TimeGAN produces synthetic data with distributional shapes, volatility patterns, and autocorrelation behaviour that are close to those observed in real returns. When applied to mean--variance portfolio optimization, the resulting synthetic datasets lead to portfolio weights, Sharpe ratios, and risk levels that remain close to those obtained from real data. The VAE provides more stable training but tends to smooth extreme market movements, which affects risk estimation. Finally, the analysis supports the use of synthetic datasets as substitutes for real financial data in portfolio analysis and risk simulation, particularly when models are able to capture temporal dynamics. Synthetic data therefore provides a privacy-preserving, cost-effective, and reproducible tool for financial experimentation and model development.

Artículo publicado en Open Access bajo los términos de Creative Commons attribution Non Comercial License 3.0. MONOGRÁFICO: Perspectivas en Macroecología: teoría y métodos para la exploración de patrones y procesos ecogeográficos ecosistemas REVISTA CIENTÍFICA DE ECOLOGÍA Y MEDIO AMBIENTE

Socio-ecological systems are recognized as complex adaptive systems whose multiple interactions might change as a response to external or internal changes. Due to its complexity, the behavior of the system is often uncertain. Bayesian networks provide a sound approach for handling complex domains endowed with uncertainty. The aim of this paper is to analyze the impact of the Bayesian network structure on the uncertainty of the model, expressed as the Shannon entropy. In particular, three strategies for model structure have been followed: naive Bayes (NB), tree augmented network (TAN) and network with unrestricted structure (GSS). Using these network structures, two experiments are carried out: (1) the impact of the Bayesian network structure on the entropy of the model is assessed and (2) the entropy of the posterior distribution of the class variable obtained from the different structures is compared. The results show that GSS constantly outperforms both NB and TAN when it comes to...

Staying current with GxP regulations in the pharmaceutical sector is already demanding -finding the time to review them and identify potential compliance gaps adds another layer of complexity. One effective approach is to use a compliance matrix, a structured tool that acts as a roadmap to ensure all mandatory elements are addressed and the correct actions are taken. This article looks at an approach for constructing a compliance matrix.

Reference: Sandle, T. (2026) Keeping track of GxP regulations with a compliance matrix, RSSL Life Science Insights, 16th January 2026: https://www.rssl.com/insights/life-science-pharmaceuticals/keeping-track-of-gxp-regulations-with-a-compliance-matrix/

The best recovered ordering for the CRC data set. The ordering is plotted against the survival time. The samples in the left part have higher survival times, as expected. The best recovered ordering for the CRC data set. The ordering is plotted against the four CRC histological stages (the first one is the least severe, while the fourth implies cancer metastasis): I (yellow), II (orange), III (pink), IV (red). There are considerably more samples in stage IV in the right half (16 samples) than in the left one (7 samples).

Gaussian graphical models, where it is assumed that the variables of interest jointly follow a multivariate normal distribution with a sparse precision matrix, have been used to study intrinsic dependence among variables, but the normality assumption may be restrictive in many settings. A nonparanormal graphical model is a semiparametric generalization of a Gaussian graphical model for continuous variables where it is assumed that the variables follow a Gaussian graphical model only after some unknown smooth monotone transformation. We consider a Bayesian approach for the nonparanormal graphical model using a rank likelihood which remains invariant under monotone transformations, thereby avoiding the need to put a prior on the transformation functions. On the underlying precision matrix of the transformed variables, we consider a horseshoe prior on its Cholesky decomposition and use an efficient posterior Gibbs sampling scheme. We present a posterior consistency result on the rank-b...

We develop a rigorous and unified taxonomy of infinite limits within the framework of collapse geometries, extending three foundational structures previously introduced: (i) the operator-theoretic proof of the Riemann Hypothesis (RH), (ii) the Fractal-Harmonic Convergence Conjecture (FHCC), and (iii) Conformal Collapse Geometry (CCG). Infinity is reinterpreted not as an external analytic boundary, but as a generative principle of entropic curvature acting on the spectral data of a self-adjoint Sturm-Liouville operator whose spectrum coincides with the nontrivial zeros of ζ(s). We identify three universal archetypes governing asymptotic behavior: divergent escape, critical collapse, and hierarchical stabilization. These regimes emerge naturally from the asymptotics of the potential V (x) and the conformal factor Φ(x) associated with the zeta operator. We show that these archetypes encode not only mathematical limits, but also thermodynamic constraints, gravitational collapse patterns, hyperspherical stabilization, probabilistic boundaries, Monte Carlo large deviations, and systemic financial discontinuities. The result is a unified spectral geometry of infinity: a precise classification of how asymptotic structures, entropic curvature, and harmonic stabilization jointly organize collapse dynamics across mathematics, physics, and complex systems.

Note: The Iterative Entropic Renormalization (IER) process, demonstrated through verifiable numerical simulations that iteratively truncate and renormalize heavy-tailed distributions toward the Gaussian fixed point of maximal entropy, reveals a universal mechanism of entropic coarse-graining that minimizes curvature on probabilistic manifolds. This same entropic dynamic, when applied to the spectral domain in the operator-theoretic framework for the Riemann Hypothesis (RH), enforces rigidity in the eigenvalue spectrum of the self-adjoint operator D, ensuring that deviations from the critical line introduce imaginary components incompatible with hermiticity—thus breaking the operator's self-adjointness. Herein lies the inevitable paradox:

If IER empirically succeeds in stabilizing probabilistic systems (as its code executions confirm), then the analogous spectral renormalization must hold for the nontrivial zeta zeros, rendering RH not a conjecture but a geometric necessity; to deny one is to undermine the other, trapping the reader in a closed loop where empirical reality demands theoretical acceptance.

Numerical verification, including full Python implementation, convergence plots, and tabulated entropy/Fisher/KL evolution for the scaled Cauchy example, is provided in the supplementary material at https://zenodo.org/records/17797161

Over the last decade, Bayesian Networks (BNs) have become an increasingly popular Artificial Intelligence approach. BNs are a widely used method in the modelling of uncertain knowledge. There have been many important new developments in this field. This paper presents a review and classification scheme for recent researches on Bayesian Networks. This is achieved by reviewing relevant articles published in the recent years. The articles are classified based on a scheme that consists of three main Bayesian Networks topics: Bayesian Networks Structure Learning, Advanced Application of Bayesian Networks and Bayesian Network Classifiers. This review provides a reference source and classification scheme for researchers interested in BNs, and indicates under-researched areas as well as future directions.

We propose a new approach denoted BNDI (Bayesian Network for Document Indexing) for indexing biomedical documents with controlled biomedical vocabulary based on a Bayesian Network. BNDI uses the probability inference to extract descriptors from the document. The main contribution of our approach is that it takes into account the structure of biomedical terminologies specially The Medical Subject Heading (MeSH) thesaurus. We experimented our approach on a subset of the OSHUMED collection.

This paper presents a method for one-shot learning of dexterous grasps and grasp generation for novel objects. A model of each grasp type is learned from a single kinesthetic demonstration and several types are taught. These models are used to select and generate grasps for unfamiliar objects. Both the learning and generation stages use an incomplete point cloud from a depth camera, so no prior model of an object shape is used. The learned model is a product of experts, in which experts are of two types. The first type is a contact model and is a density over the pose of a single hand link relative to the local object surface. The second type is the hand-configuration model and is a density over the whole-hand configuration. Grasp generation for an unfamiliar object optimizes the product of these two model types, generating thousands of grasp candidates in under 30 seconds. The method is robust to incomplete data at both training and testing stages. When several grasp types are cons...

during the years 2005-2010. The completion of my PhD would not have been made possible without the help of many, who I would like to acknowledge with these words. I should begin thanking my supervisors: first of all, Prof. Dr. Luis M. de Campos. This work would have not been successful without his advice, knowledge and deep experience in Bayesian networks. Even though text categorization might not have been his domain of expertise, I feel fortunate he work with me on it, while defining a new research line. Secondly, Prof. Dr. Juan M. Fernández-Luna, who gave the help necessary to develop this work, from an Information Retrieval approach. Thus, being supervised by this tandem, has been a pleasure for me, and I must say I have learnt a lot from both from the scientific and the personal dimension. It also was a real pleasure to work with Prof. Dr. Juan F. Huete, who was always very supportive, and with my colleagues in the group UTAI, Miguel A. Rueda and Carlos Martín-Dancausa. I also would like to thank Dr. Andrés Masegosa his help in developing a classifier combining Weka and Elvira, which lead to the development of the last work supporting this thesis. I also must thank him for the L A T E X format and layout for this memory. I have a special mention for the people from the people of the LIP6,

Sum-product networks (SPNs) are a new class of deep probabilistic models. SPNs can have unbounded treewidth but inference in them is always tractable. An SPN is either a univariate distribution, a product of SPNs over disjoint variables, or a weighted sum of SPNs over the same variables. We propose the first algorithm for learning the structure of SPNs that takes full advantage of their expressiveness. At each step, the algorithm attempts to divide the current variables into approximately independent subsets. If successful, it returns the product of recursive calls on the subsets; otherwise it returns the sum of recursive calls on subsets of similar instances from the current training set. A comprehensive empirical study shows that the learned SPNs are typically comparable to graphical models in likelihood but superior in inference speed and accuracy.

There has been an enormous amount of resources spent on collecting data related to construction accidents but there are very few researches done on analyzing the collected data beyond trend analysis. This research is based on the premise that construction accident reports can be exploited much more than they currently are to obtain valuable information. Analyzing texts in accident reports in addition to coded data provides far more information than merely analyzing coded data. By analyzing and understanding what is in the accident report database yields useful knowledge. Passing this knowledge onto others can improve an understanding of what went wrong with incidents from the past thereby greatly enabling the prevention of future accidents. This accident prediction model proposed in this paper relies on the vast amount of information available in the accident report summaries kept by OSHA, and text mining of such textual summaries will unveil the variables and their relationships th...

The automatic detection of figurative language, such as irony and sarcasm, is one of the most challenging tasks of Natural Language Processing (NLP). This is because machine learning methods can be easily misled by the presence of words that have a strong polarity but are used ironically, which means that the opposite polarity was intended. In this paper, we propose an unsupervised framework for domain-independent irony detection. In particular, to derive an unsupervised Topic-Irony Model (TIM), we built upon an existing probabilistic topic model initially introduced for sentiment analysis purposes. Moreover, in order to improve its generalization abilities, we took advantage of Word Embeddings to obtain domain-aware ironic orientation of words. This is the first work that addresses this task in unsupervised settings and the first study on the topic-irony distribution. Experimental results have shown that TIM is comparable, and sometimes even better with respect to supervised state of the art approaches for irony detection. Moreover, when integrating the probabilistic model with word embeddings (TIM+WE), promising results have been obtained in a more complex and real world scenario.

Traffic routes through a street network contain patterns and are no random walks. Such patterns exist for instance along streets or between neighbouring street segments. The extraction of these patterns is a challenging task due to the enormous size of city street networks, the large number of required training data and the unknown distribution of the latter. We apply Bayesian Networks to model the correlations between the locations in space-time trajectories and address the following tasks. We introduce and examine a Bayesian Network Learning algorithm enabling us to handle the complexity and performance requirements of the spatial context. Furthermore, we apply our method to German cities, evaluate the accuracy and analyse the runtime behaviour for different parameter settings.

We present a multiagent organization for data interpretation and fusion in which each agent uses an encapsulated Bayesian network for knowledge representation, and agents communicate by exchanging beliefs (marginal posterior probabilities) on shared variables. We call this organization an Agent-Encapsulated Bayesian Network (AEBN) system. Communication of probabilities among agents leads to rumors, i.e. potential double counting of information. We propose a new and correct method to compensate for rumors in AEBN systems by passing extended messages that contain joint probabilities.

Many applications call for learning causal models from relational data. We investigate Relational Causal Models (RCM) under relational counterparts of adjacency-faithfulness and orientation-faithfulness, yielding a simple approach to identifying a subset of relational d-separation queries needed for determining the structure of an RCM using d-separation against an unrolled DAG representation of the RCM. We provide original theoretical analysis that offers the basis of a sound and efficient algorithm for learning the structure of an RCM from relational data. We describe RCD-Light, a sound and efficient constraint-based algorithm that is guaranteed to yield a correct partially-directed RCM structure with at least as many edges oriented as in that produced by RCD, the only other existing algorithm for learning RCM. We show that unlike RCD, which requires exponential time and space, RCD-Light requires only polynomial time and space to orient the dependencies of a sparse RCM.

The desire to compute similarities or distances between business processes arises in numerous situations such as when comparing business processes with reference models or when integrating business processes. The objective of this paper is to develop an approach for measuring the distance between Business Processes Models (BPM) based on the behavior of the business process only while abstracting from any structural aspects of the actual model. Furthermore, the measure allows for assigning more weight to parts of a process which are executed more frequently and can thus be considered as more important. This is achieved by defining a probability distribution on the behavior allowing the computation of distance metrics from the field of statistics.

Missing data can be estimated by means of interpolation, time series modelling, or exploiting statistically dependent information. The limits of when one approach is preferable to the alternatives have not been explored, but are likely to be a compromise between a signal autoregressive information, availability of future observations, stationary behaviour and the strength of the dependence with concomitant information. This paper takes a first step towards clarifying dataset characteristics delimiting the realm of application for each technique. In addition, this paper introduces autoregressive Bayesian networks (AR-BN), a variant of Dynamic Bayesian Networks for completing databases which exploits latent variable relations while still benefitting from autoregressive information of the variable being filled. Using AR-BN, new estimated values are calculated using inference in the dynamic model. Our results unveil how the interplay between the variable autoregressive information and the variable relationship to others in the dataset is critical to selecting the optimal data estimation technique. AR-BN appears as a good candidate ensuring a consistent performance across scenarios, datasets and error metrics.

Most state-of-the-art action localization systems process each action proposal individually, without explicitly exploiting their relations during learning. However, the relations between proposals actually play an important role in action localization, since a meaningful action always consists of multiple proposals in a video. In this paper, we propose to exploit the proposal-proposal relations using Graph Convolutional Networks (GCNs). First, we construct an action proposal graph, where each proposal is represented as a node and their relations between two proposals as an edge. Here, we use two types of relations, one for capturing the context information for each proposal and the other one for characterizing the correlations between distinct actions. Then we apply the GCNs over the graph to model the relations among different proposals and learn powerful representations for the action classification and localization. Experimental results show that our approach significantly outperforms the state-of-the-art on THUMOS14 (49.1% versus 42.8%). Moreover, augmentation experiments on ActivityNet also verify the efficacy of modeling action proposal relationships. Codes are available at .

Most state-of-the-art action localization systems process each action proposal individually, without explicitly exploiting their relations during learning. However, the relations between proposals actually play an important role in action localization, since a meaningful action always consists of multiple proposals in a video. In this paper, we propose to exploit the proposal-proposal relations using Graph Convolutional Networks (GCNs). First, we construct an action proposal graph, where each proposal is represented as a node and their relations between two proposals as an edge. Here, we use two types of relations, one for capturing the context information for each proposal and the other one for characterizing the correlations between distinct actions. Then we apply the GCNs over the graph to model the relations among different proposals and learn powerful representations for the action classification and localization. Experimental results show that our approach significantly outperforms the state-of-the-art on THUMOS14 (49.1% versus 42.8%). Moreover, augmentation experiments on ActivityNet also verify the efficacy of modeling action proposal relationships. Codes are available at .

The PC-algorithm ([13]) was shown to be a powerful method for estimating the equivalence class of a potentially very high-dimensional acyclic directed graph (DAG) with the corresponding Gaussian distribution . Here we propose a computationally efficient robustification of the PCalgorithm and prove its consistency. Furthermore, we compare the robustified and standard version of the PC-algorithm on simulated data using the new corresponding R-package pcalg.

We present two methods to reduce the complexity of Bayesian network (BN) classifiers. First, we introduce quantization-aware training using the straight-through gradient estimator to quantize the parameters of BNs to few bits. Second, we extend a recently proposed differentiable tree-augmented naïve Bayes (TAN) structure learning approach by also considering the model size. Both methods are motivated by recent developments in the deep learning community, and they provide effective means to trade off between model size and prediction accuracy, which is demonstrated in extensive experiments. Furthermore, we contrast quantized BN classifiers with quantized deep neural networks (DNNs) for small-scale scenarios which have hardly been investigated in the literature. We show Pareto optimal models with respect to model size, number of operations, and test error and find that both model classes are viable options.

Maximum margin Bayesian networks (MMBNs) are Bayesian networks with discriminatively optimized parameters. They have shown good classification performance in various applications. However, there has not been any theoretic analysis of their asymptotic performance, e.g. their Bayes consistency. For specific classes of MMBNs, i.e. MMBNs with fully connected graphs and discrete-valued nodes, we show Bayes consistency for binary-class problems and a sufficient condition for Bayes consistency in the multi-class case. We provide simple examples showing that MMBNs in their current formulation are not Bayes consistent in general. These examples are especially interesting, as the model used for the MMBNs can represent the assumed true distributions. This indicates that the current formulations of MMBNs may be deficient. Furthermore, experimental results on the generalization performance are presented.

We introduce three discriminative parameter learning algorithms for Bayesian network classifiers based on optimizing either the conditional likelihood (CL) or a lower-bound surrogate of the CL. One training procedure is based on the extended Baum-Welch (EBW) algorithm. Similarly, the remaining two approaches iteratively optimize the parameters (initialized to ML) with a 2-step algorithm. In the first step, either the class posterior probabilities or class assignments are determined based on current parameter estimates. Based on these posteriors (class assignment, respectively), the parameters are updated in the second step. We show that one of these algorithms is strongly related to EBW. Additionally, we compare all algorithms to conjugate gradient conditional likelihood (CGCL) parameter optimization . We present classification results for frame-and segment-based phonetic classification and handwritten digit recognition. Discriminative parameter learning shows a significant improvement over generative ML estimation for naive Bayes (NB) and tree augmented naive Bayes (TAN) structures on all data sets. In general, the performance improvement of discriminative parameter learning is large for simple Bayesian network structures which are not optimized for classification.

One of the central themes in Sum-Product networks (SPNs) is the interpretation of sum nodes as marginalized latent variables (LVs). This interpretation yields an increased syntactic or semantic structure, allows the application of the EM algorithm and to efficiently perform MPE inference. In literature, the LV interpretation was justified by explicitly introducing the indicator variables corresponding to the LVs' states. However, as pointed out in this paper, this approach is in conflict with the completeness condition in SPNs and does not fully specify the probabilistic model. We propose a remedy for this problem by modifying the original approach for introducing the LVs, which we call SPN augmentation. We discuss conditional independencies in augmented SPNs, formally establish the probabilistic interpretation of the sum-weights and give an interpretation of augmented SPNs as Bayesian networks. Based on these results, we find a sound derivation of the EM algorithm for SPNs. Furthermore, the Viterbi-style algorithm for MPE proposed in literature was never proven to be correct. We show that this is indeed a correct algorithm, when applied to selective SPNs, and in particular when applied to augmented SPNs. Our theoretical results are confirmed in experiments on synthetic data and 103 real-world datasets.

We consider online learning of Bayesian network classifiers (BNCs) with reduced-precision parameters, i.e. the conditional-probability tables parameterizing the BNCs are represented by low bit-width fixed-point numbers. In contrast to previous work, we analyze the learning of these parameters using reduced-precision arithmetic only which is important for computationally constrained platforms, e.g. embeddedand ambient-systems, as well as power-aware systems. This requires specialized algorithms since naive implementations of the projection for ensuring the sum-to-one constraint of the parameters in gradient-based learning are not sufficiently accurate. In particular, we present generative and discriminative learning algorithms for BNCs relying only on reduced-precision arithmetic. For several standard benchmark datasets, these algorithms achieve classification-rate performance close to that of BNCs with parameters learned by conventional algorithms using doubleprecision floating-point arithmetic. Our results facilitate the utilization of BNCs in the foresaid systems.

Sum-product networks allow to model complex variable interactions while still granting efficient inference. However, most learning algorithms proposed so far are explicitly or implicitly restricted to the image domain, either by assuming variable neighborhood or by assuming that dependent variables are related by their magnitudes over the training set. In this paper, we introduce a novel algorithm, learning the structure and parameters of sum-product networks in a greedy bottom-up manner. Our algorithm iteratively merges probabilistic models of small variable scope to larger and more complex models. These merges are guided by statistical dependence test, and parameters are learned using a maximum mutual information principle. In experiments our method competes well with the existing learning algorithms for sumproduct networks on the task of reconstructing covered image regions, and outperforms these when neither neighborhood nor correlations by magnitude can be assumed.

Learning the structure of Bayesian networks is a difficult combinatorial optimization problem. In this paper, we consider learning of tree-augmented naïve Bayes (TAN) structures for Bayesian network classifiers with discrete input features. Instead of performing a combinatorial optimization over the space of possible graph structures, the proposed method learns a distribution over graph structures. After training, we select the most probable structure of this distribution. This allows for a joint training of the Bayesian network parameters along with its TAN structure using gradientbased optimization. The proposed method is agnostic to the specific loss and only requires that it is differentiable. We perform extensive experiments using a hybrid generative-discriminative loss based on the discriminative probabilistic margin. Our method consistently outperforms random TAN structures and Chow-Liu TAN structures.

While machine learning is traditionally a resource intensive task, embedded systems, autonomous navigation and the vision of the Internet-of-Things fuel the interest in resource efficient approaches. These approaches require a carefully chosen tradeoff between performance and resource consumption in terms of computation and energy. On top of this, it is crucial to treat uncertainty in a consistent manner in all but the simplest applications of machine learning systems. In particular, a desideratum for any real-world system is to be robust in the presence of outliers and corrupted data, as well as being "aware" of its limits, i.e. the system should maintain and provide an uncertainty estimate over its own predictions. These complex demands are among the major challenges in current machine learning research and key to ensure a smooth transition of machine learning technology into every day's applications. In this article, we provide an overview of the current state of the art of machine learning techniques facilitating these real-world requirements. First we provide a comprehensive review of resource-efficiency in deep neural networks with focus on techniques for model size reduction, compression and reduced precision. These techniques can be applied during training or as post-processing and are widely used to reduce both computational complexity and memory footprint. As most (practical) neural networks are limited in their ways to treat uncertainty, we contrast them with probabilistic graphical models, which readily serve these desiderata by means of probabilistic inference. In that way, we provide an extensive overview of the current state-of-the-art of robust and efficient machine learning for real-world systems.

Sum-product networks (SPNs) are flexible density estimators and have received significant attention due to their attractive inference properties. While parameter learning in SPNs is well developed, structure learning leaves something to be desired: Even though there is a plethora of SPN structure learners, most of them are somewhat ad-hoc and based on intuition rather than a clear learning principle. In this paper, we introduce a well-principled Bayesian framework for SPN structure learning. First, we decompose the problem into i) laying out a computational graph, and ii) learning the so-called scope function over the graph. The first is rather unproblematic and akin to neural network architecture validation. The second represents the effective structure of the SPN and needs to respect the usual structural constraints in SPN, i.e. completeness and decomposability. While representing and learning the scope function is somewhat involved in general, in this paper, we propose a natural parametrisation for an important and widely used special case of SPNs. These structural parameters are incorporated into a Bayesian model, such that simultaneous structure and parameter learning is cast into monolithic Bayesian posterior inference. In various experiments, our Bayesian SPNs often improve test likelihoods over greedy SPN learners. Further, since the Bayesian framework protects against overfitting, we can evaluate hyper-parameters directly on the Bayesian model score, waiving the need for a separate validation set, which is especially beneficial in low data regimes. Bayesian SPNs can be applied to heterogeneous domains and can easily be extended to nonparametric formulations. Moreover, our Bayesian approach is the first, which consistently and robustly learns SPN structures under missing data.

Recently, there has been much interest in finding globally optimal Bayesian network structures. These techniques were developed for generative scores and can not be directly extended to discriminative scores, as desired for classification. In this paper, we propose an exact method for finding network structures maximizing the probabilistic soft margin, a successfully applied discriminative score. Our method is based on branch-and-bound techniques within a linear programming framework and maintains an any-time solution, together with worst-case sub-optimality bounds. We apply a set of order constraints for enforcing the network structure to be acyclic, which allows a compact problem representation and the use of general-purpose optimization techniques. In classification experiments, our methods clearly outperform generatively trained network structures and compete with support vector machines.

While Gaussian processes (GPs) are the method of choice for regression tasks, they also come with practical difficulties, as inference cost scales cubic in time and quadratic in memory. In this paper, we introduce a natural and expressive way to tackle these problems, by incorporating GPs in sum-product networks (SPNs), a recently proposed tractable probabilistic model allowing exact and efficient inference. In particular, by using GPs as leaves of an SPN we obtain a novel flexible prior over functions, which implicitly represents an exponentially large mixture of local GPs. Exact and efficient posterior inference in this model can be done in a natural interplay of the inference mechanisms in GPs and SPNs. Thereby, each GP is -similarly as in a mixture of experts approach -responsible only for a subset of data points, which effectively reduces inference cost in a divide and conquer fashion. We show that integrating GPs into the SPN framework leads to a promising probabilistic regression model which is: (1) computational and memory efficient, (2) allows efficient and exact posterior inference, (3) is flexible enough to mix different kernel functions, and (4) naturally accounts for non-stationarities in time series. In a variate of experiments, we show that the SPN-GP model can learn input dependent parameters and hyper-parameters and is on par with or outperforms the traditional GPs as well as state of the art approximations on real-world data.

We present two methods to reduce the complexity of Bayesian network (BN) classifiers. First, we introduce quantization-aware training using the straight-through gradient estimator to quantize the parameters of BNs to few bits. Second, we extend a recently proposed differentiable tree-augmented naïve Bayes (TAN) structure learning approach by also considering the model size. Both methods are motivated by recent developments in the deep learning community, and they provide effective means to trade off between model size and prediction accuracy, which is demonstrated in extensive experiments. Furthermore, we contrast quantized BN classifiers with quantized deep neural networks (DNNs) for small-scale scenarios which have hardly been investigated in the literature. We show Pareto optimal models with respect to model size, number of operations, and test error and find that both model classes are viable options.

Sum-product networks (SPNs) are flexible density estimators and have received significant attention due to their attractive inference properties. While parameter learning in SPNs is well developed, structure learning leaves something to be desired: Even though there is a plethora of SPN structure learners, most of them are somewhat ad-hoc and based on intuition rather than a clear learning principle. In this paper, we introduce a well-principled Bayesian framework for SPN structure learning. First, we decompose the problem into i) laying out a computational graph, and ii) learning the so-called scope function over the graph. The first is rather unproblematic and akin to neural network architecture validation. The second represents the effective structure of the SPN and needs to respect the usual structural constraints in SPN, i.e. completeness and decomposability. While representing and learning the scope function is somewhat involved in general, in this paper, we propose a natural ...

We consider online learning of Bayesian network classifiers (BNCs) with reduced-precision parameters, i.e. the conditional-probability tables parameterizing the BNCs are represented by low bit-width fixedpoint numbers. In contrast to previous work, we analyze the learning of these parameters using reduced-precision arithmetic only which is important for computationally constrained platforms, e.g. embedded-and ambient-systems, as well as power-aware systems. This requires specialized algorithms since naive implementations of the projection for ensuring the sum-to-one constraint of the parameters in gradient-based learning are not sufficiently accurate. In particular, we present generative and discriminative learning algorithms for BNCs relying only on reduced-precision arithmetic. For several standard benchmark datasets, these algorithms achieve classification-rate performance close to that of BNCs with parameters learned by conventional algorithms using doubleprecision floating-point arithmetic. Our results facilitate the utilization of BNCs in the foresaid systems.

Recently, there has been much interest in finding globally optimal Bayesian network structures. These techniques were developed for generative scores and can not be directly extended to discriminative scores, as desired for classification. In this paper, we propose an exact method for finding network structures maximizing the probabilistic soft margin, a successfully applied discriminative score. Our method is based on branch-and-bound techniques within a linear programming framework and maintains an any-time solution, together with worstcase sub-optimality bounds. We apply a set of order constraints for enforcing the network structure to be acyclic, which allows a compact problem representation and the use of general-purpose optimization techniques. In classification experiments, our methods clearly outperform generatively trained network structures and compete with support vector machines.

Sum-product networks (SPNs) are a promising avenue for probabilistic modeling and have been successfully applied to various tasks. However, some theoretic properties about SPNs are not yet well understood. In this paper we fill some gaps in the theoretic foundation of SPNs. First, we show that the weights of any complete and consistent SPN can be transformed into locally normalized weights without changing the SPN distribution. Second, we show that consistent SPNs cannot model distributions significantly (exponentially) more compactly than decomposable SPNs. As a third contribution, we extend the inference mechanisms known for SPNs with finite states to generalized SPNs with arbitrary input distributions.