Get an attorney
Your contractor payment process has been set up wrong. Go to an attorney immediately. Assuming that you acquired the construction loan, you have control over all payments. You should set it up with your title company, or company servicing the payment of all expenses, to pay the sub contractors directly. Never have the builder pay the subs and get a Lien Release with each payment. You could end up paying for your home twice. Remember, the one who holds and controls the money is the one who controls the entire project.
Also, since the builder is not paying the subs he is probably cutting corners in the quality of construction. I would definetly hire a home inspection company to make phase inspections during the building process. Subject to the language of your builders' contract, you can make your builder go back and correct all mistakes at his expense. If he does not correct the mistakes, don't pay him.
Building a home is an eye opening experience. The builder is not your friend, he is your adversary. His job is to make as much money as possible. I had a home built a few years ago and that process was a nightmare. The final result was that I stood my ground, ultimately refusing to pay the builder his earnings until final closing (I paid all the subs). The result was that the $32,000. he was suppose to make (profit) was reduced to $11,000. after he paid for all mistakes. And "No" the builder and I are definetly not friends, but my home was built correctly. GOOD LUCK AND GET AN ATTORNEY.