Combine VAE

Here we describe the generative process of VDGEC. With $K$ clusters, an observed sample $\mathbf{x} \in \mathbb{R}^D$ is generated by the following process:

1. choose a cluster $c \sim Cat(\pi)$, i.e.

2. choose a latent vector $\mathbf{z} \sim \mathcal{N}(\mathbf{\mu_c}, \mathbf{\sigma_c}^2\mathbf{I})$, i.e.

3. compute $\mathbf{\mu_x}, \mathbf{\sigma_x^2}$:

   $$[\mathbf{\mu_x};\log \mathbf{\sigma_x}^2] = f(\mathbf{z}:\mathbf{\theta})$$

4. choose a sample $\mathbf{x }\sim \mathcal{N}(\mathbf{\mu_x}, \mathbf{\sigma_x}^2\mathbf{I})$, i.e.

where $K$ is a predefined parameter, $\pi_k$ is the prior probability for cluster $k$, $\mathbf{\pi} \in \mathbb{R}^{K}_{+}, \sum_{k=1}^K\pi_k = 1$. $Cat(\mathbf{\pi})$ is the categorical distribution parameterized by $\mathbf{\pi}$. $\mathbf{\mu_c}$ and $\mathbf{\sigma_c^2}$ are the mean and the variance of the Gaussian distribution corresponding to cluster $c$. $\mathbf{I}$ is an identity matrix, $f(\mathbf{z};\mathbf{\theta})$ is a neural network whose input is $\mathbf{z}$ and is parameterized by $\mathbf{\theta}$. $\mathcal{N}(\mathbf{\mu_x, \sigma_x^2})$ is multivariate Gaussian distribution parameterized by $\mathbf{\mu_x, \sigma_x}$.

According to the generative process above, the joint probability $p(\mathbf{x,z},c)$ can be factorized as:

as $\mathbf{x}$ and c are independent conditioned on $\mathbf{z}$.

To maximize the likelihood of the given data points base on the generative process, by using Jensen's inequality, the log-likelihood can be written as:

where $\mathcal{L}(\mathbf{x})$ is the evidence lower bound, which can be written as:

or as:

where $q(\mathbf{z},c|\mathbf{x})$ is the variational posterior to approximate the true posterior $p(\mathbf{z},c|\mathbf{x})$.

We assume $q(\mathbf{z},c|\mathbf{x})$ to be a meanfield distribution which can be factorized as:

In Equation 7, the first term has no relationship with $c$ and the second term is non-negative.

Hence, in order to maximize $\mathcal{L}(\mathbf{x})$, $\mathit{D_{KL}(q(c|\mathbf{x}) || p(c|\mathbf{z})) \equiv 0}$ should be satisfied, which means,

In Equation 8, the first term is the reconstruction term, which encourages VDGEC to explain the dataset well. And the second term is the Kullback-Leibler divergence from the gaussian mixture model prior $p(\mathbf{z}, c)$ to the variational posterior $q(\mathbf{z}, c | \mathbf{x})$, which regularizes the latent embedding $z$ to lie on a mixture of gaussion manifold.

According to Equation 5 and 9, the $\mathcal{L}(\mathbf{x})$ can be rewritten as:

We compute $\mathit{E}_{q(\mathbf{z},c | \mathbf{x})}[\log p(\mathbf{x|z})]$ as:

Where $[\mathbf{\mu_x};\log \mathbf{\sigma_x}^2] = f(\mathbf{z}:\mathbf{\theta})$

We compute $\mathit{E}_{q(\mathbf{z},c | \mathbf{x})}[\log p(\mathbf{z}|c)]$ as:

We compute $\mathit{E}_{q(\mathbf{z},c | \mathbf{x})}[\log p(c)]$ as:

Following VAE, we use a neural network $g$ to model $q(\mathbf{z|x})$:

Then, we compute $\mathit{E}_{q(\mathbf{z},c | \mathbf{x})}[\log p(\mathbf{z}|\mathbf{x})]$ as:

We compute $\mathit{E}_{q(\mathbf{z},c | \mathbf{x})}[\log p(c|\mathbf{x})]$ as:

As for $q(c|\mathbf{x})$, according to Equation 10, we compute as follow:

Using the SGVB estimator and the reparameterization trick, the $\mathcal{L}_{ELBO}(\mathbf{x})$ can be rewritten as:

where $L$ is batch size, $D$ is the dimensionality of $\mathbf{x}$ and $\mu_x^{(l)}$, $J$ is the dimensionality of $\mu_c,\sigma_c^2, \tilde{\mu}$ and $\tilde{\sigma}^2$, and $*|_j$ donotes the $j^{th}$ element of $*$, $K$ is the number of clusters, $\pi_c$ is the prior probability of cluster $c$, and $\gamma_c$ denotes $q(c|\mathbf{x})$. where $\mathbf{z}^{(l)}$ is the $l^{th}$ sample.