AMC 12 Mock Exam 3 Problem 18

Since $ABCD$ is cyclic, $\angle B+\angle D=180^\circ$, so $\cos D=-\cos B$. Apply the law of cosines to diagonal $\overline{AC}$ in triangles $ABC$ and $ADC$: $$AC^2=2^2+3^2-12\cos B=5^2+6^2+60\cos B.$$ Thus $13-12\cos B=61+60\cos B$, so $\cos B=-\frac23$ and $AC^2=13+8=21$. Then $\sin B=\sqrt{1-\frac49}=\frac{\sqrt5}{3}$.

The circle is the circumcircle of $\triangle ABC$, so by the extended law of sines $$R=\frac{AC}{2\sin B}=\frac{\sqrt{21}}{2\cdot\frac{\sqrt5}{3}}=\frac{3\sqrt{21}}{2\sqrt5}=\boxed{\tfrac{3\sqrt{105}}{10}}\approx3.07.$$ (Choice (E) is $AC$ itself; choice (A) would be the radius if $AC$ were a diameter, which it is not since $\angle B\ne90^\circ$.)