@testbomber not really, I have almost the same setup, except the 4GB RAM version of the CM4, but 2 GB should be enough. If I understood correctly, you don't even get to the bootanimation?
Do you have a UART Bridge to get some logs? You might try a different SD card, but that's a wild guess.
Do you have any other periphery connected?