Yes. Even and especially surveys where the participants are not only compensated for their time, but are also selected from a pool of people who are specifically seeking out such piece-male work (a la Amazon Turk).
There's a big difference b/w posting flyers/ads in target-rich environments and providing a low friction way for people to get involved in the study. But if you do something on Amazon Turk you're basically guaranteeing a very particular type of person, because random guy with a high paying job probably isn't going to have a turk account and won't have the patience to create one just for your survey.
In some cases you can make a good case that this doesn't totally invalidate any sort of generality in the results, but for the most part I throw out papers -- on face -- that proceed in this ways without extended, evidence-based discussion of why the hell I can expect any sort of conclusion except "our result generalizes to a population of people who really need an extra dollar or two (literally) and are looking for pretty much any way to get it."
There's a big difference b/w posting flyers/ads in target-rich environments and providing a low friction way for people to get involved in the study. But if you do something on Amazon Turk you're basically guaranteeing a very particular type of person, because random guy with a high paying job probably isn't going to have a turk account and won't have the patience to create one just for your survey.
In some cases you can make a good case that this doesn't totally invalidate any sort of generality in the results, but for the most part I throw out papers -- on face -- that proceed in this ways without extended, evidence-based discussion of why the hell I can expect any sort of conclusion except "our result generalizes to a population of people who really need an extra dollar or two (literally) and are looking for pretty much any way to get it."