a millstone around your neck
US /ə ˈmɪlˌstoʊn əˈraʊnd jʊər nɛk/
UK /ə ˈmɪlˌstoʊn əˈraʊnd jʊər nɛk/

1.
a heavy burden of responsibility or worry
:
•
His gambling debts became a millstone around his neck.
•
The old factory was a millstone around the company's neck, costing them a fortune in repairs.