Consider a sequence of 8 binary 0 or 1 bits containing only

Consider a sequence of 8 binary (0 or 1) bits containing only 2 bits that are equal to 1. Assuming each bit is independent of any other and equals 1 with probability s,

a) Find the entropy of the sequence

b) Is there a single value of s that maximizes the entropy and, if so, what is it?

c) What is the maximum amount of entropy reduction that can occur when you make a guess that one of the bits is equal to 1? Note, your guess can be correct or incorrect.

Solution

a) entropy = 2/8 = 1/4

b) value of s maximise the entropy = probabilty of happening bit 1 in sequence

if all the bit become 1, i.e 11111111

s=1

c) maximum reduction in entropy = 1-1/4 = 3/4