AMC 12 Mock Exam 2 Problem 25

Let $p=xyz$. From $x+y+z=0$ and $x^2+y^2+z^2=6$ we get $xy+yz+zx=-3$, so $x,y,z$ are the roots of $t^3-3t-p$.

Let $S=x^2y+y^2z+z^2x$ and $T=xy^2+yz^2+zx^2$. Then $S+T=(x+y+z)(xy+yz+zx)-3xyz=-3p$, and expanding shows $S-T=-(x-y)(y-z)(z-x)$. The square of $(x-y)(y-z)(z-x)$ is the discriminant of $t^3-3t-p$, namely $-4(-3)^3-27p^2=108-27p^2$; in particular $|p|\le2$. Hence $$S=\frac{-3p-(x-y)(y-z)(z-x)}{2}\le\frac{-3p+\sqrt{108-27p^2}}{2}=\frac32\left(-p+\sqrt3\sqrt{4-p^2}\right).$$ By the Cauchy–Schwarz inequality, $-p\cdot1+\sqrt{4-p^2}\cdot\sqrt3\le\sqrt{p^2+(4-p^2)}\cdot\sqrt{1+3}=4$, so $S\le6$.

Equality requires $(-p,\sqrt{4-p^2})$ to be proportional to $(1,\sqrt3)$, i.e., $p=-1$. Then $x,y,z$ are the three real roots of $t^3-3t+1$ (its discriminant is $81>0$), ordered so that $(x-y)(y-z)(z-x)=-9$. This gives $S=\frac{3+9}{2}=6$. The maximum is $\boxed{6}$.

(Trying integer points such as $(1,1,-2)$ gives only $3$; the maximum occurs at irrational values, namely $2\cos\frac{2\pi}9$, $2\cos\frac{4\pi}9$, $2\cos\frac{8\pi}9$ in a suitable order.)