Guestimations and Heuristics

For a first approximation, we can look at the mean and variance of the number of distinct numbers. Letting \(X_i\) be the event that we have at least one \(i\), \(X\) the number of distinct, then by linearity of expectation,

which is very close to \(4\). Hmm… this seems intentionally designed. Let’s now calculate the variance: for \(i \ne j\), then

so that

A simple Chebyschev inqualities gives us that

and a normal approximation gives us

As an aside, this ratio \((1 - (1 - 1/n)^n ) \to 1 - \frac{1}{e} \approx 0.63\) is related to how hash maps set their maximum load. If we hash \(n\) elements into \(n\) buckets uniformly, this is our expected occupancy!

Solutions

Some code

Not the most efficient, but some pretty clean usage brute-force calculation via the Python standard library.

> Counter(len(set(x)) for x in it.product(range(6), repeat=6))
Counter({4: 23400, 3: 10800, 5: 10800, 2: 930, 6: 720, 1: 6})

Combinatorics

There are \(6^6 = 46'656\) possible strings of 6 dice. We want to derive an expression for \(c_k\), the number of ways for there to be exactly \(k\) distinct numbers in that string.

We first pick the \(k\) distinct numbers, with there being \(\binom{6}{k}\) different ways of doing so. Then, we need to pick the histogram of counts for each of the \(k\) numbers. For \(c_4\), then this is

And the answer is \(23'400 / 46'656 \approx 50.2\%\). For some reason, \(c_3 = c_5\); seems like a coincidence.

There are many other ways to do this combinatorially, e.g. with generating functions or inclusion-exclusion. These are besides the point here, but there are some more interesting approaches (many of which were in the pursuit of trying to find an intuitive reason for such a large probability mass)!

Dynamic Programming

Another approach that is maybe less error prone is to do some dynamic programming. Let \(f(i, j)\) be the probability that we have \(i\) distinct numbers after \(j\) rolls. Then, we get the recurrence

i.e. we sum up the cases where after a new roll, we get a distinct or not.

Summing up each column gives a probability of exactly \(1\). This also gives us a simpler closed form, namely, we just sum up all the paths! Indeed,

Calculating \(f(4, 6)\), the probability of getting \(4\) distinct numbers after \(6\) rolls, we have

Hitting Times

Let us consider aa related problem: how many trials does it take until we get \(k\) distinct rolls. The number of rolls until we get our first distinct follows \(\text{Geometric}(p = 1)\). Our second distinct element follows \(\text{Geometric}(p = 1 - 1/6)\). In general, to get the \(k\) -th distinct, it follows \(G_k = \text{Geometric}(p_k = \frac{7 - k}{6})\). The expected number of rolls to get \(k = 1, \ldots, 6\) distinct elements are thus:

But, if we recall from our introductory statistics classes, this feels a lot like the Poisson and Gamma duality. The probability of counting \(\ge 4\) successes in \(6\) rolls is the same as it taking \(\le 6\) rolls to get the \(4\) -th success.

We can proceed in a similar path, deriving an analogous for the Gamma distribution. In particular, let \(T_k = G_1 + \ldots + G_k\) be the rolls until we get the \(k\) -th distinct element. Then, we want to calculate

The full calculation is a bit annoying, and I have instead attached it. In short, probability generating functions are useful.

Embeddings

An extremely powerful tool in stochastic processes, which I hope to learn more about, is to embed this discrete chain into some more general process. Finally, we restrict our view of that more general process to the events that we care about.

This in general is known as Poissonification, and I first saw the idea with context of hashing in a UIUC theory seminar. We suppose we throw \(m = \textbf{Poisson}(\lambda)\) die, and then condition on the event that \(m = 6\). The really elegant result from this is that we avoid the negative correlation of the occupancy indicators \(X_i\). In particular, if we look instead at the Multinomial distribution of the counts of each dice, then

where we recognize each individual term in the product as the mass function for \(\textbf{Poisson}( \lambda p_i )\). We not only avoid the negative correlation, but we get pure independence! At this point, by conditioning on \(m\), we can do the calculation and get an answer.

In fact, if one remembers the construction of a Poisson process, then it should be clear why Possionification leads to this nice independence property. Suppose we set a timer \(T \sim \textbf{Exponential}(\lambda / 6)\) independently for each of the \(6\) bins. Then, a dice is rolled whenever a timer is fired, and the timer gets reset.

Here, the waiting time between rolls of the dice follow the minimum of \(6\) independent \(\textbf{Exponential}(\lambda / 6)\) timers, i.e. the gap between each of the each of the dice rolls follows \(\textbf{Exponential}(\lambda)\). Thus, by the construction of Poisson process, then the number of dice rolls at a given time \(t\) follows \(\textbf{Poisson}( \lambda t )\). Moreover, because exponential random variables are memoryless, then the increment between a dice rolling in each bin follows \(\textbf{Exponential}(\lambda / 6)\), and again by construction of Poisson, then the number of balls in a given bin follows \(\textbf{Poisson}(\lambda t / 6)\).

A Long Thread

It feels like every stochastic process is just some variation on balls and bins. Immediately, it is connected to hashing. More generally, we can look at the waiting time, and other ways of embedding this into general chains, such as the Poisson process. The DP is the simplest and hardest to make a mistake on (something about computing permutations just never goes right for me). Then there are the more powerful generalizations, where looking at a continuous embedding (in the final clause) made the calculation much smoother than its discrete time analogs. Generating functions, which I’ve hardly used before, were recurring, as the bridge to combine independent events. While we have negatively correlated dependency structure, the right lens cast that away.

The goal of this multitude of approaches is to derive intuition for why we find such a large percentage of having 4 unique dice values. My first instinct is to wave my hands with an exclaimation about the law of small numbers. But this is a really good reminder about how common concentration really is – while these situations are rarely presented as a sum of independent events, the right view exposes it. Whenever the probability generating function has all real roots (which one can indeed verify), then we can decompose it into a sum of independent Bernoullis, and so typically concentration holds. More powerful is that the number of balls in a given bin is negatively correlated with another bin – if bin \(2\) has many balls, then we naturally expect bin \(1\) to have few balls.

Next semester, when I take a course in stochastic physics, I expect many more such problems to show up. I should remember myself of the suggestion of Sariel Har-Peled: your life gets easier when you pretend random variables are point-masses around their mean.

Lattice Bacteria Puzzle

Puzzle Statement

Consider an integer 2-D grid. A bacterium begins at the origin. Each bacteria may divide; that is, a bacterium at the point \((x, y)\) can be replaced by one at \((x+1, y)\) and \((x, y+1)\), supposing those two squares were previously bacteria-free. Roughly, how many divisions must take place before the closed box with corners at \((0,0), (0,3), (3,3)\), and \((3,0)\) is clear of bacteria?

An attempt via dynamic programming

First was via dynamic programming. Let \(f(w, h)\) be the amount of divisions for a bacteria to clear a \(w \times h\) grid. The problem asks to find \(f(4,4)\).

If \(w = h = 1\), then it takes only one division. Otherwise, after one division, the top bacteria must clear a \(w \times (h-1)\) grid, and the right bacteria must clear a \((w-1) \times h\) grid. Thus, the recurrence I formulated is:

Note that the general solution is that

This does not work. After a bacteria clears the grid, it does not magically disappear. In other words, I cannot isolate the two bacteria separately.

Simulations

Ok, at this point I gave up and wrote a simple simulation to see how long it would take. In a loop, I would:

• check if there are any bacteria in the \(w \times h\) grid
• try to get that bacteria to divide
• if there was a bacteria in the way, then recurse to split that other bacteria

So I wrote this up and ran it. Then waited. Then waited. Then waited…

At this point, I got suspicious. Looking through the states, it seemed that I entered an infinite recursive descent.

The Solution

Finally, it got through in my head that it may not be possible to clear the grid. How would I prove this?

I recalled a similar problem my friend showed me. Similar, there was an integer grid of bacteria, but now there were infinite bacteria at every coordinate \((x, y)\) for $x \le 0$$. They divide in a peculiar way, with two (orthogonally) adjacent bacteria mating to spawn a new bacteria. That is, there are two ways they can divide:

• a bacteria at \((x, y)\) and \((x+1, y)\) being replaced with \((x+2, y)\)
• a bacteria at \((x, y)\) and \((x, y+1)\) being replaced with \((x, y+2)\) The question is can a bacteria reach \(x=5\)?

A general approach towards solving such problems is to find invariants. Suppose each configuration of bacteria has some potential that is maintained by these divisions. I’ll leave the solution to this related problem hidden, but as a hint, the golden ratio shows up!

Suppose we assign a value to each grid \((x, y)\), so that \(\phi(x,y) = 2^{-(x+y)}\). The potential is the sum of \(\phi\) over occupied grid positions. With this, upon a split,

so that the potential is constant! The starting potential is \(1\). What must be the potential be if the \(4 \times 4\) grid was empty? Well, there can be at most one bacteria on each grid square, so

So the potential must decrease! However, divisions do not change the potential, so it is impossible to clear the grid.

Note that even clearing a \(3 \times 3\) grid is impossible, as the remaining potential adds up to \(\frac{30}{32} < 1\).

Equal-Sum Subsets Puzzle

Your friend chooses 10 distinct integers from \(1\) to \(100\). Is it always possible to find two disjoint, non-empty subsets of their 10 numbers that have the same sum?

Pigeonhole and Solution

Unfortunately, I have seen this puzzle before as an application of the pigeonhole principle.

The first observation to make is that the disjointness only means that you need to choose two distinct subsets. That is, given two overlapping but distinct subsets that add to the same sum, then removing the overlapping numbers would find a winning pair of disjoint subsets.

Given 10 numbers \(x_1, \ldots, x_10\), there are \(2^{10} - 2 = 1022\) possible subsets (ignoring trivial ones). However, any subset must sum to at least \(1+2+\ldots+10 = 55\), and at most \(100+99+\ldots+91 = 955\), so there are \(901\) possible sums. As \(1022\) subsets must have a range of \(901\) values, two of them must have the same sum, by pigeonhole!

Pick those two subsets, remove the overlapping values, and thus you can always find two disjoint non-empty subsets of the same value!

For example, the sequence \(HHTHHH\) gets \(2\) points. With \(6\) flips, the average score is \(1/3\).

From a quick simulation with \(p=1/2\), (flips on the \(x\)-axis, on a square-root scale), it seems like the ratio converges to \(1/6\). Let us show that analytically.

Solution via Mutual Recursion

Consider a (sufficiently large) index \(n\) in our sequence. The quantity we want to find is the probability that the \(n\)-th coin flip gains one additional point.

There are three possible cases:

• the \((n-1)\)-th flip was tails (state T)
• the \((n-1)\)-th flip was heads, and did not gain a point (state H)
• the \((n-1)\)-th flip was heads, and gained a point (state HH)

For \(n\) large, the probabilities are stationary, meaning that flipping one more coin does not change their probabilities. Of course, \(\pi_T\), probability of being in state \(T\), is \((1-p)\), the probability of flipping tails. Under the stationary distribution, we also have the relations

The first relation says that the \(H\) state can come from flipping heads from either the \(T\) or \(HH\) states. The second relation says that to get to the \(HH\) state, we must flip heads from the \(H\) state.

To solve this recurrence, we can unroll:

and therefore, \(\pi_{HH} = \frac{p^2}{1+p}\), the solution to our puzzle. For a fair coin (\(p = 0.5\)), then \(\pi_{HH} = \frac{1}{6}\).

Markov Chains

What we have just done is find the stationary distribution of the following Markov Chain:

The arcs represent the state changes after a coin flip. We want to find to find the long-term probability of being in the \(HH\) state.

Let \(P(x, y)\) be the probability of transitioning between states \(x\) and \(y\) (zero if there is no arc). Then, a stationary distribution satisfies the recurrence

Towards a Generalization

In this problem, we gain points whenever we visit a state. It is straightforward to generalize to gaining points whenever we visit one of a set of states, by just adding the probabilities!

However, what if we want to gain points instead on an arc? For example, we can represent this three state markov chain instead using two states, where we get points on the top edge.

As a computer scientist, I care about finite state Markov chains. For irreducible chains (i.e. corresponding to strongly connected digraphs), there is an alternative way of finding these probabilities.

Notation

Let us introduce some notation.

• \(X_m\), the state of the markov chain after \(m\) steps
• \(P^m(x,y)\) be the probability of moving from state \(x\) to \(y\) in exactly \(m\) steps.
• \(N_n(x,y) = \sum_{m=1}^n 1_{y}(X_m)\), the number of visits to state \(y\) in \(n\) steps, starting at state \(x\).
• \(G_n(x,y) = E(N_n(x,y)) = \sum_{m=1}^n P^m(x,y)\), the expected number of visits to \(y\) from \(x\) in \(n\) steps.
• \(T_{x,y}^r = \min \{ n \ge 1 \mid N_n(x,y) \ge r \}\), the earliest time to hit \(y\) exactly \(r\) times
• \(m_{x,y} = E(T_{x,y}^1)\), the expected time to hit \(y\) starting at state \(x\), i.e. the hitting time

For brevity, \(m_y = m_{y,y}, T_y = T_{y,y},\)…

Hitting Times and Stationary Distribution

Recall we want to find \(\pi_y\), the stationary distribution of the markov chain.

Theorem: The stationary distribution is the reciprocal of the hitting time.

proof: First, we show the second equality, that the ratio \(N_n(y, y) / n\) approaches \(1/m_{y}\). For \(i \ge 1\), we consider \(\omega^i = T_{y}^i - T_{y}^{i-1}\), the marginal time to cycle back to \(y\) for the \(i\)-th time. Since our transitions are memoryless, then \(\omega^i\) are iid. Now, by the law of large numbers,

Finally,

where the left and right both converge to \(m_y\). Thus, \(\frac{n}{N_n(y)}\), and moreover \(\frac{n}{G_n(y)}\), both converge to \(m_y\) as well.

Now, for the first equality, note that for any \(m\), \(\pi_y = \sum_{x \in S} \pi(x) P^{m}(x,y)\). Therefore, when we sum over \(m\), we have

Moreover, this holds as \(n \to \infty\), so recalling that \(\sum_x \pi(x) = 1\), then we find that

Thus concludes the proof.

Computing Hitting Times using Cycles

The hitting time \(m_y\) is the expected number of steps starting at \(y\) before reaching \(y\) again. We can compute this by enumerating all (possibly countable infinite) cycles, computing the probability of taking it, and find the expected length.

I illustrate this by example. Consider the following two state Markov chain with states \(T\) and \(H\). Let \(P(T, H) = p\), \(P(H, T) = q\), and self loops \(1-p\) and \(1-q\) respectively.

To find the hitting time (and therefore, the stationary distribution) \(m_T\), then we have the following cycles: \(e_4\), \(e_1 e_3\), \(e_1 e_2 e_3\), \(e_1 e_2 e_2 e_3\), …

Therefore, we can find the average cycle length, i.e. the hitting time, as

Thus, \(\pi_T = \frac{p}{p+q}\).

We can now find \(\pi_H\) in one of several ways. First, \(\pi_H + \pi_T = 1\), so that \(\pi_H = \frac{p}{p+q}\).

Alternatively, we can note that \(\pi_H / \pi_T\) is the averaged number of times we visit \(H\) in the above cycles. That is,

so that indeed, \(\pi_H = \frac{p}{q} \frac{q}{p+q} = \frac{p}{p+q}\).

We can summarize this procedure into the following theorem:

Theorem: Let \(x\) be any state in a finite irreducible Markov chain, and \(C_x\) be all cycles from \(x\). Then, for any state \(y\),

Note that here, we say cycles to allow repeated links, but passing through the start state exactly once.

proof: We partition the walk into stages, where each stage is delimited by a cycle in \(C_x\). Then, \(N_n(y)\) is sum of visits to \(y\) in the \(k\)-th stage.

Now, \(\frac{N_n(x)}{G_n(x)} \to 1\), and \(\frac{1}{N_n(x)} \sum_{k=1}^{N_n(x)} (..)\) approaches the average visits to \(y\) in \(C_x\). Moreover, the LHS approaches \(\pi_y\), and \(m_x\) is the average length of \(C_x\).

Thus, taking limits, indeed

Points on Links

Let us return to our two-state Markov chain, where we get a point whenever taking the \(H \overset{H}{\to} S\) link.

In general, as we can bisect links into states, then for any start state \(x\),

Now, we have the cycles \(T\), \(HT\), and \(HH\) with probabilities \(1-p\), \(p(1-p)\), and \(p^2\) respectively. The average cycle length is \(1 \cdot (1-p) + 2 \cdot p(1-p) + 2 \cdot p^2 = 1 + p\). The average points per cycle is \(0 \cdot (1-p) + 0 \cdot p(1-p) + 1 \cdot p^2 = p^2\). Finally,

Indeed this is the same solution as via mutual recursion!

Concluding Remarks

Here, we discussed a probability question that could be represented by finding the stationary probability of an irreducible Markov chain. Then, we described an alternative approach that often leads to smaller Markov chains!

When the underlying digraph is not strongly connected, then this analysis does NOT work. In particular, the start state now matters.

Addressing this, we need to consider the probability of ending up at each of the strongly connected components. The general way of handling this is to represent each connected component as an absorbing state, with \(Q\) the new transition probability. Then, the probability of ending at state \(t\) is

where \(\sum_{m=0}^\infty A^m = (1-A)^{-1}\) by Geometric series formula. \((1-Q)^{-1}\) is called the fundamental matrix.

Apparently, this is a special case of a renewal-reward process.

Acknowledgements

The cycle analysis technique was introduced to me by my friend taking CS 438 at UIUC. Many probablistic networking protocols exhibit simple irreducible behavior which lend themselves nicely to such analysis.

It will be useful to establish consistent notation. Let \(p_y\) be the probability that \(y\) is the last new number the ladybug visits. Let \(X_t\) be the position of the ladybug. Let \(T_y\) be the first time where \(X_t = y\).

I will also use \(P_x( ... )\) to denote the probability of an event given the starting point of a random walk of \(x\).

Gamblers Ruin

The first solution I came up with relies on the setup called “Gambler’s Ruin”. Consider a 1-D symmetric random walk starting at \(0 < x < d\). I claim that \(P_x(T_d < T_0) = \frac{x}{d}\).

First, observe that \(E(X_t) = x\) (this is called a Martingale process). This is because at each time step, we make no expected progress towards either endpoint. Therefore, at time \(T = \min( T_d, T_0)\), then we obtain the equation

which proves our claim.

Solution One

Now, to solve the puzzle, it remains to enumerate some simple cases. Consider the sequence in which the ladybug sees new numbers. Then, in order for 6 to be last, then it must either see 5 before 7 before 6, or 7 before 5 before 6. That is,

We can decompose

and

so that plugging in our expression

Solution Two

Following this, I immediately noticed that \(p_1 = p_2 = \ldots = p_{11} = 1/11\). All the probabilities are uniform! This hints that there is a simpler explanation.

Indeed, in order for 6 to be the last new number visited, there must be an arc between 5 to 7, or from 7 to 5. All the steps before reaching either 5 or 7 are superfluous. Therefore, $p_x = P_5(T_7 < T_6)$.

In words, the probability of seeing 6 last is the probability of starting to one side of 6 and reaching the other side of 6 without hitting 6. This holds for any starting position.

Now, as there are 11 probabilities, the final step is to show that they sum to 1. One way of doing this (although there are likely many) is to shift a perspective: instead of varying the starting point, vary the end point. As there must be some number that is the last one visited, those probabilities must add to 1.