Have you considered that he may have a food allergy? This could be one reason to refuse food/drink or not to want to be around others when eating. Some people are so allergic that smelling the food or being touched by someone with that food on their fingers can cause a reaction.
The other possibility is that he is strongly introverted. People who are introverted become mentally and physically exhausted being around others and need a break after some period. So if you have all been working intensely together for awhile, he may need some time alone to recuperate. It can be hard for extroverts to understand this because they get energized by being around others in contrast to introverts losing energy in the same situation.
Also, please keep in mind that being shy and being introverted are different things. People who are introverted are not afraid or scared of others, they simply get worn out being around others. Shy people are nervous or have anxiety being around others.
If this person is an introvert, the best thing you can do is give him space to recharge himself after any intense team-work.
Some people may also find it hard to engage in conversations that aren't in their areas of interest. So that could play a role as well. You will notice if his attention drifts on any subjects outside his expertise. That is tough to improve on, but I recommend keeping conversations short and to the point when outside his interest areas if that seems to be the problem.