It really depends on the reason why the contractor isn't giving the source files.
In any case, please try to put yourself on both sides of the equation first. I have a similar problem right now, but I'm on the other side. I'm the contractor, I've provided all of the source files, submitted my invoice, and almost 3 weeks now, but no payment has been received from the MassChallenge startup. The founder is refusing to pay me. Had I known, I wouldn't have provided the source files too.
I'm in no way saying that you're in the wrong or anything like that. As a contractor myself, I can tell you that contractors generally just like people who are reasonable and understanding, and people who actually understand how contracting works. So if I were you, I'd first try to figure out the reason behind all of this, examine it carefully, equally considering both sides, be reasonable with each other and work out a solution. There are two things that, as an early-stage startup, you really don't want to come out of this: (1) having to take the matter to court and (2) getting your reputation destroyed by one angry or unsatisfied contractor.
That's just my 2¢. I hope you're able to work this out and find a good solution.