Understanding rules

A rule computes one message of one factor node: the message towards one of the node's interfaces, from what the node receives on the others. RxInfer finds a rule for every message a model needs. This page explains how it finds one, and why some rules take messages while others take marginals.

Messages by hand and Variational message passing by hand call the rules of small models one by one, and are the practical companion to this page.

using RxInfer

What a rule computes

Consider a node $f(x, y, z)$ and the message it sends towards $x$. Which formula computes it depends on the approximation.

Belief propagation integrates the node function against the messages on the other interfaces:

\[\mu_{f \to x}(x) = \int f(x, y, z)\, \mu_{y \to f}(y)\, \mu_{z \to f}(z) \, \mathrm{d}y\, \mathrm{d}z .\]

Its inputs are the messages $\mu_{y \to f}$ and $\mu_{z \to f}$, written m[:y] and m[:z].

Variational message passing takes the expected log-density under the marginals $q(y)$ and $q(z)$, the current posterior beliefs about the neighbors:

\[\mu_{f \to x}(x) \propto \exp \mathbb{E}_{q(y)\, q(z)}\big[\log f(x, y, z)\big] .\]

Its inputs are the marginals, written q[:y] and q[:z]. An expectation needs the whole posterior of a neighbor, not only the message the neighbor sends.

Structured variational message passing mixes the two. When $x$ and $y$ stay joint and $z$ is separate, the rule integrates over $y$ with its message and takes the expectation over $z$ with its marginal:

\[\mu_{f \to x}(x) \propto \int \mu_{y \to f}(y) \exp \mathbb{E}_{q(z)}\big[\log f(x, y, z)\big] \, \mathrm{d}y .\]

Its inputs are m[:y] and q[:z].

How RxInfer finds a rule

RxInfer finds a rule from four things:

• the node, such as NormalMeanPrecision or the function +;
• the target, the interface the message is heading to;
• the node's algorithm, which is DefaultAlgorithm() for most nodes;
• the inputs: which messages and marginals the rule receives, and their types.

Among the rules with the same inputs, the types select one, as Julia's multiple dispatch selects a method. which_message_update_rule finds the rule for a call without running it:

which_message_update_rule(
    NormalMeanPrecision, :μ;
    m = (out = NormalMeanVariance(1.0, 2.0), τ = PointMass(4.0)),
)

args->begin
        (out_mean, out_var) = mean_var(args.m[:out])
        NormalMeanVariance(out_mean, out_var + inv(mean(args.m[:τ])))
    end

The card shows the inputs the rule takes, where it is defined, and its body. With a normal message on out and a known precision, the message towards the mean is a normal with the two variances added.

rule_coverage tabulates every rule of a node:

MessagePassingRulesBase.rule_coverage(NormalMeanPrecision)

A row is a target, the average energy, or the joint marginal of a cluster. A column is an algorithm, and each cell counts the rules. The six rules towards μ differ in their inputs: messages, marginals or a mix, from a known value or from a distribution.

Rule packages: what RxInfer loads

The nodes and their rules live in packages, listed on ReactiveMP's ecosystem page. using RxInfer loads the packages of the nodes most models use, and a rule is found once its package is loaded:

The other node packages are not loaded, so that RxInfer depends on as little as possible. A model that uses one of their nodes loads the package next to RxInfer:

A name from a package that is not loaded is an UndefVarError, whose hint names the package. The same holds for one optimisation package: the default optimiser of a PointMassFormConstraint on a continuous variable runs Optim.jl, which a model loads with using Optim.

Messages or marginals: the factorization decides

A rule does not choose whether it takes messages or marginals. The factorization of the posterior does, which you state with @constraints. It splits each node's interfaces into clusters. The rule towards a target then takes:

• the messages on the other interfaces of the target's own cluster;
• the marginals of the other clusters, a joint marginal for a cluster of several interfaces.

For a node x ~ NormalMeanPrecision(μ, τ), with the interfaces out, μ and τ:

Under the mean-field factorization, the rule towards out reads the marginals. The card draws marginal inputs as dashed arrows:

@call_message_update_rule(
    node = NormalMeanPrecision, target = :out,
    q = (μ = NormalMeanVariance(1.0, 2.0), τ = GammaShapeRate(2.0, 1.0)),
)

args->NormalMeanPrecision(mean(args.q[:μ]), mean(args.q[:τ]))

Under the structured factorization, the rule towards τ reads the joint marginal of the cluster (out, μ), written q[:out, :μ]. A call passes a joint marginal with the clusters keyword:

which_message_update_rule(
    NormalMeanPrecision, :τ;
    clusters = ((:out, :μ) => MvNormalMeanCovariance([1.0, 0.0], [1.0 0.5; 0.5 2.0]),),
)

args->begin
        (m, V) = mean_cov(args.q[:out, :μ])
        θ = 2 / (((V[1, 1] - V[1, 2]) - V[2, 1]) + V[2, 2] + abs2(m[1] - m[2]))
        Gamma(convert(typeof(θ), 1.5), θ)
    end

Observed data and constants are known values, and RxInfer keeps each of them in a cluster of its own. A stochastic node's rules therefore receive them as point-mass marginals, such as q[:out]::PointMass for an observation. A point mass is the same distribution as a message or as a marginal, and the rules return the same result for either.

A deterministic node, such as + or *, relates its output to its inputs by a function. Its clusters are always its output and the joint over its inputs, whatever the factorization, so its rules take messages.

A node's algorithm

An algorithm selects which rules a node runs and carries their parameters. Most nodes run under DefaultAlgorithm(), where the factorization alone decides the inputs. The node * runs under its own default, MultiplicationSampling. Its parameter is the number of samples that its rules for two uncertain factors draw:

MessagePassingRulesBase.rule_coverage(*)

A nonlinear function in a model is a Delta node, whose algorithm names the approximation, such as Linearization() or Unscented(). Algorithm specification shows how a model gives a node its algorithm, with where { algorithm = … } or @algorithm. Algorithms and dependencies describes the scheme in full.

When no rule fits

When no rule takes the inputs a message needs, inference stops with a RuleNotFoundError. The error names the node, the target and the inputs, and explains, for every rule of that node and target, why the rule does not fit. Variational message passing by hand shows one, and Rule Not Found Error lists the ways out: another factorization, a functional form constraint, or a rule of your own.

Implementing a custom node

A model may need a node or a rule that no package defines. Creating your own custom nodes declares a node and writes its rules for RxInfer. MessagePassingRulesBase's tutorial, Your first node, writes the belief propagation and variational rules of one node and checks them.