As a developer I participated in developing a bunch of startup, I realised that without a co-founder who has the domain knowledge is indispensable. But really, is this the only way to find these problems? Or how your startup story is coming?
Find a problem you would be passionate to solve, then find a person who has a network in that industry (if you don't), this would make you a killer team. You need to fell in love with the problem, it doesn't matter which industry it is, you have problem-solving skill as a developer :) Hackathon ideas rarely grow up into a real business, and often it is the problem and willingness to solve it is what united people in teams, "bunch of startups" just doesn't sound like there was any passion in it.
This is a great question and one that I have thought about numerous times so I am glad you asked. I see a number of good ideas here already. I have even pondered doing a kickstarter simply to create a forum where this is the main idea: to voice then groom and perfect market needs with the crowd and then perfect a scope from the ones that bubble to the top. Sprinkle in git and slack and you have a team and a distributed development project. Right now, I know of no such thing that coalesces all of this into one seamless experience. However, in the age of crowd sourcing it pays to simply "ask". http://www.Quora.com is the best site I've found for this. Some people also swear by Reddit. On Quora you can get a lot of experts chime in.
Another source I look at are package managers. There are thousands of problems solved already (just think npm or yarn packages). Many of these are juicy enough that they could be easily spun up (if they are open source) into larger offerings and then sold for profit. You may even be able to get the original developer interested.
You won't have to have a co-founder in the industry you want to serve. You just need to talk to people in that industry. If you have an existing relationship with someone, go to lunch or get coffee and ask them the hardest problem they're dealing with at work right now.
If you don't already know someone, you can try to find forums where your industry hangs out online. You can then try to develop relationships with members or you can mine their problems from forum posts.
First decide what category of problem you want to solve - social/societal, consumer/ business problem; I'm not being flippant but solving part of the world hunger problem, or something that moves the ball forward on curing cancer, is not the same as solving a problem for mass market retailers' (e.g.consumers). Once you decide what kind of problems intrigue or excite you, the direction to find specific problems becomes much simpler.
If you are looking to find pre-validated problems and a pre-qualified market sample, try industry associations as a resource. . If you're interested in manufacturing and Industry 4.0 related matters, feel free to ping me.
Adam, there are a couple of ways that I would use to find a problem in industries which are ready or waiting to be disrupted,
1) Read, knowledge about an industry compounds by reading. You'll be amazed by the number of ideas that you'll be able to generate and connect once you have started to read about an specific industry.
2) Ideas may be born in isolation but they need to be nurtured over a period of time by discussing with folks from the industry, make a list of the top 10 things that they complain about and one of the line items will have you startup idea.
3) For example, identify markets where the end products are over priced and where there is information asymmetry and sellers are taking advantage of this and trying to push their product or service aggressively. If your startups solves the information or the price problem, you have a winning startup idea.
Hmmm... I may not understand the question so I apologize if this makes no sense. Are you only wanting to solve a tech or software problem? Or what about solving a problem from another industry with tech? I am among many entrepreneurs on here who need engineers to help make our startup dreams come true (and then help many others realize their dreams as well...). If you are open to working in other industries, such as Travel, for instance, contact me:).
Adam - You have asked a number of questions - I am assuming that you want some thoughts on how you might go about starting a company. 1. I agree with you and others that having a co-founder is key. The domain expertise is not as important as someone who can hold you accountable. As they say in Americs, call you on your bulls*!t. 2) The key to having a chance at a successful startup lies in the convergence of the aforementioned relationship, something you care deeply about and are determined to address and solving a problem that is worth solving with something that a large enough market will pay you for. 3) Having said all that, the advice I give early entrepreneurs is to make sure this is really what you want. I have compiled some info for you t consider on these topics in the following blog posts.
Steve Blank, Eric Ries, Cindy Alvarez, Ash Maurya, Justin Wilcox, Bob Moesta, Alan Klement, and LIFFFT are some other folks to look at to learn more about the early phase of startups.