go down in the world
US /ɡoʊ daʊn ɪn ðə wɜːrld/
UK /ɡoʊ daʊn ɪn ðə wɜːrld/

1.
to become less successful or to lose social status or money
:
•
After losing his job, he really started to go down in the world.
•
It's sad to see how some once-wealthy families can go down in the world so quickly.