At my first startup (I was CTO - a day 1 employee) I delayed taking a honeymoon for three years, thinking that if I was away for a minute everything would fall apart.
At some point, the 2nd in charge of technology went away on vacation: and I realized that everything went smoothly, and that we didn't even know he was gone (and he was an incredibly valuable employee).
The lesson, I learned, is that we overthink our importance, even if we are the most important person at the organization. This doesn't mean there aren't crucial times where you can't be away - it just means that every moment isn't such a time.
Schedule a day off, but schedule it filled with stuff (meet someone for breakfast, then go to a museum, then meet someone for lunch, etc.) so that you don't have any free time to think about how you should be at work (or worse, so you don't grab your laptop and start coding).