On the Conservation of Mutual Information
Machine Learning, like many disciplines, uses random vairables (RVs) to model the outcomes of events. Loosely speaking, these are variables that can take on any value in some set, each with some probability.1 As you could imagine, such a general definition allows RVs to describe the outcomes of a wide range of events. These could be the pixel values from taking a photograph, the words that follow a paragraph, the temperature readings from a doctor’s thermometer, or even the observations and actions from an agent interacting with its environment. ...