It’s programmed so that you won’t take fall damage if you jump/fall from a height around 20+ blocks high (this has been here for like 2-3 years, dw about it)
I never tested the actual height where you begin taking fall damage, but an estimate is that if you’re on top of the ladder in the towers...