The challenge is that government agencies are generally risk averse to disruption. Startups are usually about disruption. Governments will change when things reach a crisis point, but it takes a long time to reach a crisis that warrants a dramatic disruption in the current business processes.
Instead, governments work best with startups that try to incrementally improve business processes.
With all that said, I'd say the biggest problem for startups to work with governments is that onerous and time consuming procurement policies make it almost impossible for most startups to hang in long enough to make lots of sales. This is particularly true if you're selling anything that requires a competitive bid. Competitive bids take 6 months. Startups aren't well equipped to slog out a six month RFP process with an uncertain outcome (maybe you get the business, maybe you don't). Small purchases are easy enough, but there's not a ton of money to be made if your max sale is $10K per year per customer. There are only 1,600 government agencies in the U.S. of any demonstrable size and so you have to get significant market penetration to hit significant revenue. I think that's another limiting factor for startups to focus on the government sector.