Hierarchical modeling of gravitational-wave populations for disentangling environmental and modified-gravity effects

The upcoming Laser Interferometer Space Antenna (LISA) will detect up to thousands of extreme-mass-ratio inspirals (EMRIs). These sources will spend $\sim 10^5$ cycles in band, and are therefore sensitive to tiny changes in the general-relativistic dynamics, potentially induced by astrophysical environments or modifications of general relativity (GR). Previous studies have shown that these effects can be highly degenerate for a single source. However, it may be possible to distinguish between them at the population level, because environmental effects should impact only a fraction of the sources, while modifications of GR would affect all. We therefore introduce a population-based hierarchical framework to disentangle the two hypotheses. Using simulated EMRI populations, we perform tests of the null vacuum-GR hypothesis and two alternative beyond-vacuum-GR hypotheses, namely migration torques (environmental effects) and time-varying $G$ (modified gravity). We find that with as few as $\approx 20$ detected sources, our framework can statistically distinguish between these three hypotheses, and even indicate if both environmental and modified gravity effects are simultaneously present in the population. Our framework can be applied to other models of beyond-vacuum-GR effects available in the literature.