I was having the same issue after building a tower on PS4. However, I discovered that if I face the ladder, standing up, from the top, and get as close as I can just as the dot cursor passes the first rung and HOLD X, it allows me to climb down the ladder.
The scripted prompt that says “climb” does not appear but I have now been successful several times in climbing down the ladder with this technique.