Even 25–50ms is noticeable in certain situations. See http://www.youtube.com/watch?v=vOvQCPLkPt4 for an illustration of how things get noticeably better, even as you go below 10ms.
In situations like this it's definitely important to get that response lag down as much as possible — you're looking at direct 1:1 control with that though which is slightly different than a tap. Like you said, it all depends on the situation.
