I think the first the thing you have to decide is how much of the accounting you want to do yourself. I've used Quickbooks since it came out in the early 90's and it's what I usually recommend but there are other programs out there that could work for your situation. You don't say what type of business you have, how large it is, or what kind of record keeping you do/need done; these can be important factors in deciding on a bookkeeper. In order to keep tabs on things, whatever accounting system you use HAS to be something that you understand and can pull any information you need from it without having to ask your bookkeeper.
Having your CPA do your books AND do your taxes, while it may seem logical, is always more costly and in my opinion doesn't allow for proper separation. I handle the accounting for a small business; I am also an EA, a Tax Professional. I told my client straight up that while I was more than capable of doing his taxes that I didn't feel it advisable to do both his bookkeeping AND his taxes. So at year end, I balance all of his accounts, make copies of statements, reconciliations and other pertinent data and give him a package to take to his CPA. His wife handles the bill paying and writing the checks, I just enter that data and take care of other things like posting the payroll that is processed through a separate agency.
When it comes to deciding whether or not someone is trustworthy, in part it depends if you're looking to hire an in house employee or using someone who is a professional bookkeeper. If you're looking for a part time in house employee, then your normal hiring protocols would be fine. If the bookkeeper can't write checks or take money, they can't, practically speaking, embezzle from you which I believe is your primary concern. Both an in house employee and a professional bookkeeper should be able to provide their experience / job references. A good way to check their discretion is to look at their facebook public view and then as a friend (if they will accept a friend request from you or long in to let you review the past couple of weeks). Someone who talks smack about clients or their employers is someone to avoid.
Lastly, you need to look for someone who will work with you and be willing to work with your CPA. CPAs often have a static account listing that they prefer their clients use when giving them info for taxes. They could potentially charge more if they have to figure out what your odd accounts names mean. Look for their experience, credibility and discretion and not so much about them having a college degree. Accounting software makes things incredibly easy these days and there may be a very good bookkeeper out there that doesn't have a piece of paper. (FYI yes, I do have a degree, one in Accounting and Finance, so this isn't a tainted opinion).