Typically what happens is you get sued in civil court by the government. If you lose you will be required to tear down the structure. If you don't comply it will then become a case where the government tears it down and files a lein on the property to recoup the costs. Failure to pay this lein will result in foreclosure and seizure by the government.

Obviously defying a court order can wind up with contempt of court where you then can serve jail time, but the general process is civil.