I am in mouse down camp. When it comes to dragging off a button to avoid triggering, I've done it many times, but it always feels wrong. It's like I'm taking advantage of faulty behaviour. Because of that instinct I'm inclined to believe that there's a better alternative available, like being able to undo easily.
On the note of draggable elements: don't overload affordances on UI elements. If it's draggable, don't make it clickable too. It creates a conceptually simpler experience. As for drag scrolling on mobile, that requires intervention to avoid erroneous triggering. Though when we're talking about "clicking" we're talking about Desktop + Mouse.
But that simply just doesn't work. For example, hyperlinks are both clickable and draggable. It doesn't make sense to remove one or the other because of some strange reason. There's a reason why actions are taken upon release and not upon down. There's literally no reason to take action on down.
On the note of draggable elements: don't overload affordances on UI elements. If it's draggable, don't make it clickable too. It creates a conceptually simpler experience. As for drag scrolling on mobile, that requires intervention to avoid erroneous triggering. Though when we're talking about "clicking" we're talking about Desktop + Mouse.