Locally maximizing the Rényi entropies

As I was rewriting my website, I found some visualizations I had stored on my old website to show a collaborator, and I figured it was worth writing a little to have a more proper place to put them; hence this post 😊.

Probability distributions on three letters consist just of three non-negative numbers which add up to 1, which we can see as a vector in $\mathbb{R}^3$. The set of all such distributions form a simplex, which looks like a 2D triangle laying in $\mathbb{R}^3$:

We can parametrize such distributions just by their $x$- and $y$-coordinates, since their $z$-coordinate is given by $z=1-x-y$. This allows us to plot functions $f$ that vary over the set of probability distributions on three letters in $\mathbb{R}^3$: for each valid choice of $(x,y)$ coordinates, we plot the number $f(x,y,1-x-y)$ at the point $(x,y)$. One particular function of probability distributions that I’m interested in is the $\alpha$-Rényi entropy. When considering probability distributions on three letters, it’s given by $H_\alpha( \vec p) = \frac{1}{1-\alpha}\log(x^\alpha + y^\alpha + z^\alpha)$ where $\vec p = (x,y,z)$, and $\alpha \in (0,1)\cup(1,\infty)$ is a parameter. From this function, we can define another function of probability measures, $\Delta_\varepsilon(\vec p) = \max_{ \vec q \in B_\varepsilon( \vec p) } H_\alpha (\vec q) - H_\alpha(\vec p),$ where $B_\varepsilon(\vec p)$ is called the $\varepsilon$-ball around $\vec p$, and consists of all probability measures which are $\varepsilon$-close to $\vec p$ in total variation distance. For example if $\vec r = (0.21, 0.24, 0.55)$, then $B_\varepsilon(\vec r)$ is given by the filled purple hexagon in Figure 2:

$B_\varepsilon(\vec r)$ is the purple hexagon.

It turns out that this maximum is achieved at one unique pointSee arxiv/1706.02212

; for the case before, it’s shown here:

The maximizer of $H_\alpha$ over the ball $B_\varepsilon(\vec r)$ is the unlabelled black point at the bottom of the hexagon.

and we can write down a form for the maximizer. This allows us to plot the value of $\Delta_\varepsilon$ as it varies over the set of probability distributions, for a given $\varepsilon$ and $\alpha$. The quantity $\Delta_\varepsilon$ is useful for proving continuity boundsSee arxiv/1707.04249

. I’ve included some of these plots of it below.