Joint and Conditional Entropies
Joint and conditional entropies, as defined in this chapter and as used in information theory, apply to two systems or dimensions; for more than two dimensions (discussed later in the chapter), terminology is still developing. For simplicity, we'll usually deal here with two systems. The first, system X, has discrete observations xi; the second, system Y, has values yj.
The probability of seeing particular values of two or more variables in some sort of combination. In fact, we rolled a pair of dice (one white and one green, representing separate systems) and built a joint probability distribution. The joint probability distribution gave the probability of the joint occurrence of each possible combination of values of the two systems. Since entropy is based solely on probability, the idea of joint probability easily applies to a so-called joint entropy—the average amount of information obtained (or uncertainty reduced) by individual measurements of two or more systems.
Having built a word concept of joint entropy, let's follow our standard procedure and express it in symbols. There are two cases. In the first, knowing value x from system X has no effect on the entropy of system Y. (In other words, individual observations x and y aren't mutually related.) In the second, knowing x affects our estimate of the entropy of system Y (and so x and y have some kind of association). For each case, the symbols (and hence the form of the definition) can be either those of probabilities or of entropies.
Mutually unrelated systems
Let's begin by talking about the first case, in which x and y aren't mutually related. We can express joint entropy in terms of probabilities or in terms of entropies. First we'll do the probability form, then the entropy form.
