The Supreme Court said Monday it won't hear an appeal from the family on TV's "Sister Wives" challenging Utah's law banning polygamy.

The justices left in place a lower court ruling that said Kody Brown and his four wives can't sue over the law because they weren't charged under it.

A federal judge sided with the Browns and overturned key parts of the state's bigamy law in 2013, but an appeals court overturned that decision last year.

The Browns claim the law infringes on their right to freedom of speech and religion. The family said they should be able to challenge the law because the threat of prosecution forced them to flee to Nevada and still looms over them when they return to Utah.

Police investigated the family after their show premiered in 2010, but closed the case without filing any charges. The family argued in legal briefs that the state should not be able to thwart a constitutional challenge to the law "by changing its enforcement policy during the pendency of litigation."

Utah's law forbids married people from living with a second purported "spouse," making it stricter than anti-bigamy laws in other states and creating a threat of arrest for plural families. But state officials have followed a long tradition of not prosecuting polygamists unless they commit some other crime, such as child or spousal abuse, domestic violence or fraud.