Machine Learning, like several other disciplines, models many phenomena as random variables (RVs): variables that can take on any value in some set, each with some probability.1 These RVs can be pixel intensities in an image, words in a sentence, sensor readings from a robot’s surroundings, or even the exchange of observations and actions as an agent interacts with its environment in time. Information theory is one way to study these RVs. One of the many interesting results in information theory is the conservation of directed information: a law that tells us that as two entities exchange RVs back and forth, the total information they share is neatly partitioned into what flows from one to the other, and what flows back. This post’s focus will be the proof of that conservation law. ...

Should we really believe that a cat can be both dead and alive because ‘science’ told us so? Are you telling me that everything we ever have and ever will see in this gigantic Universe was once squashed into a point smaller than an atom? When we try to look into the scientific reasoning and explanations for why these absurd conclusions could be true, they’re often shrouded in a black box of equations and jargon that really doesn’t feel like it’s worth the time to understand completely. At least that’s how I feel about my reading assignments. ...