2011 AMC 12A Problem 23

Direct composition gives $$g(z)=\dfrac{Az+B}{Cz+D},$$ where $A=1+a,$ $B=a(1+b),$ $C=1+b,$ and $D=a+b^2.$ The matrix $\begin{pmatrix}A&B\\C&D\end{pmatrix}$ represents $g.$ For $g\circ g$ to be the identity, its square must be scalar. Comparing the off-diagonal entries and the two diagonal entries gives two possibilities: either $B=C=0$ and $A=D,$ or $A+D=0.$ The first gives $b=-1$ (with $a\ne-1$); the second gives $$b^2=-(1+2a).$$ In the second case, $|b|^2=|1+2a|.$ As $a$ runs around the unit circle, $|1+2a|$ ranges from $1$ to $3,$ so $1\le|b|\le\sqrt3.$ Both endpoints occur: take $(a,b)=(-1,1)$ and $(a,b)=(1,i\sqrt3).$ The separate case $b=-1$ also has $|b|=1.$ Therefore the requested difference is $\sqrt3-1.$ Thus, the correct answer is C.