Anomalous and Anisotropic Nonlinear Susceptibility in the Proximate Kitaev Magnet $α$-RuCl$_3$

The leading order nonlinear (NL) susceptibility, $χ_3$, in a paramagnet is negative and diverges as $T \rightarrow 0$. This divergence is destroyed when spins correlate and the NL response provides unique insights into magnetic order. Dimensionality, exchange interaction, and preponderance of quantum effects all imprint their signatures in the NL magnetic response. Here, we study the NL susceptibilities in the proximate Kitaev magnet $α$-RuCl$_3$ which differs from the expected antiferromagnetic behavior. For $T< T_c$ = 7.5 K and field $B$ in the ab-plane, we obtain contrasting NL responses in low ($<$ 2 ${T}$) and high field regions. For low fields the NL behavior is dominated by a quadratic response (positive $χ_2$), which shows a rapid rise below $T_c$. This large $χ_2 >0$ implies a broken sublattice symmetry of magnetic order at low temperatures. Classical Monte Carlo (CMC) simulations in the standard ${K-H-Γ}$ model secure such a quadratic ${B}$ dependence of ${M}$, only for ${T}$ $\approx$ ${T}_c$ with $χ_2$ being zero as ${T}$ $\rightarrow$ 0. It is also zero for all temperatures in exact diagonalization calculations. On the other hand, we find an exclusive cubic term ($χ_3$) describes the high field NL behavior well. $χ_3$ is large and positive both below and above ${T}_c$ crossing zero only for ${T}$ $>$ 50 K. In contrast, for $B$~$\parallel$~c-axis, no separate low/high field behaviors is measured and only a much smaller $χ_3$ is apparent.