On the Conservation of Mutual Information
Machine Learning, like several other disciplines, models many phenomena as random variables (RVs): variables that can take on any value in some set, each with some probability \footnote{probability density, in the case of continuous sets}. These RVs can be pixel intensities in an image, words in a sentence, sensor readings from a robot’s surroundings, or even the exchange of observations and actions as an agent interacts with its environment in time....