Bridging the gap between symbolic and geometric planning has received much attention in recent years. An important issue in some of the works that combine the two approaches is finding the right balance between backtracking at the symbolic level versus at the geometric planning level. We present in this work a new approach to interleaved backtracking, where the symbolic planner backtracks to try alternative action branches that naturally map to different geometric solutions. This eliminates the need to “protect” certain symbolic conditions when backtracking at the geometric level, and addresses a completeness issue in our previous approach to interleaved backtracking. We discuss a concrete, non-trivial symbolic-geometric planning example in the context of Human-Robot Interaction, a full implementation of the combined planning technique, and an evaluation of performance as well as the effect of increasing the symbolic-action branching factor.