Given that the failure rate of most startups is very high - in the 90% range - choosing a CEO likely to make the startup successful is pretty important.
In my experience, a good CEO is multi-dimensional. He is a good strategist, but willing to pivot if the market feedback early on sends him in a particular direction. He should be charismatic - able to pain a picture of the future of the company - but not narcissistic. He should have a sold background in putting together a team, but be able to pinch hit in multiple roles. Different companies in different industries may have different CEO needs, but in technology companies, an understanding of technology, the ability to talk about the technology in terms of implications and benefits to end users, along with the ability to segment the early market and target those most likely to buy.