Is it this simple?
“Not every civilization will bother to invest scarce resources in projects that by the nature of light-speed lag and distance can do them no direct good. Furthermore, light-speed lag and distance mean that child civilizations must be functionally independent of parent cultures. That could be a full stop to further expansions, as there’s no guarantee that colony worlds will want to invest in spawning their own colonies.”
https://reactormag.com/the-fermi-paradox-percolation-and-inbreeding/