This paper presents a reactive trajectory controller for manipulating objects in Human Robot Interaction (HRI) context. The trajectories to be followed by the robot are provided by a human aware motion planner. The controller is based on an online trajectory generator, which is capable of calculating a trajectory from an arbitrary initial condition to a target within one control cycle. The controller is capable of switching to a new trajectory each time the motion planner provides a new trajectory, changing the frame in which the input trajectory is controlled and tracking a target or a trajectory in a frame, which moves with respect to the robot frame. The controller chooses different control modes for different situations. Visual servoing by trajectory generation is considered as one case of the control situations. Some results obtained with this controller are presented to illustrate the potential of the approach.