In this article, Zed demonstrates one of the tremendous screwups with the academic human trials system: actions that are perfectly acceptable normally suddenly become a problem if done for science.
I.e., if I personally want to pay people $5 to play a video game, and then pay them $10 if they win, there is absolutely nothing preventing me from doing this. I can also gather statistics on the result. But if I were a scientist at a university doing this, I'd suddenly need to get permission from an IRB.
This is a flaw with academia, not Riot Games.
The comparison to the Stanford Prison Experiment is nonsense. That experiment literally locked people in jail, an action that is generally illegal in general. This experiment literally involves nothing more than a group of people statistically accessing records they have legal permission to access and then doing some math.
Note that I say this with no positive feelings whatsoever about Riot Game's PC-police behavior. I don't generally favor "no-asshole rules", or any of the other feelings-over-facts policies currently favored by the PC-left.
[edit: since folks reading this clearly think I'm criticizing Zed's entire post, Zed has 8 arguments against Riot. I'm disputing (3) and (4) only.]
I like Zed Shaw, his writing and learncodethehardway. I am not sure I agree with someof his conclusions here but I am not sure yours are correct.
One of the key issues for me, Shaw and many others based on comments, is lack of explicit permission. Most academic studies get explicit permission but they also anonymize the data. They also are objective researchers who do not take negative action against their subjects based on the results of their paper/findings.
For me, this hinges on how much was communicated to the employees and how it was handled internally. I do not have information about that which is broad and reliable so I won't speculate on the actual decision.
However, Zed is a hilarious and prolific writer and always has an agenda. If his rhetoric was an overreach or not entirely sound, it doesnt discount that the area between playing a competitive game with assumed privacy and professiomal overlap as an employee at the company is pretty gray.
So if you run some statistics and then use the result, that's somehow unethical?
Riot's decisionmaking process may be flawed - I have no evidence of this, but my prior is that it's likely (same as Zed). If this is true, they are firing good employees and costing themselves money.
But the fact that they didn't obey ridiculous university IRB requirements is NOT really a good argument against them. And that's all I'm criticizing.
I took his article and the analogies to point out that these studies must get approval before engaging subjects. He also states that the law requires explicit permission unless it is on or using company property.
Idk the legal requirements, but I thin, to underscore my previous point (and how I interpreted zsfa) that the issue is in how counterparties agree and the obligations they owe to one another.
I also find myself agreeing that it would be difficult to charchterize snarky. Although, I suspect that they say snarky but mean using racial slurs and other threatening language. Clearly, saying horrible things to people is not positive but free speech is more important. I find in super reasonable that someone with a tagged Riot employee un shouting racial epithets gets fired. However, a private personal account without explicit consent should probably enjoy the expectation of privacy.
super complicated issue, however, I believe many people would look at this differently if it came out that every employee was told, signed paperwork, and was aware that it would be company policy to monitor and enforce known etiquette with known repetcussions
edit: also there have been studies that show people behave differently when watched. It seems reasonable simply telling everyone about this policy would bring about the desired results.
Riot games moves people to Santa Monica where rent on a two bedroom is $4K/month, pays them the same amount they will make elsewhere, and then fires them for weird shit after they already have a lease on an apartment, and you think that's unlike the Stanford prison experiment? Those people will likely take a year or more to recover from the financial disaster of working for that company.
The premise "anything that's not illegal should be acceptable in scientific research" is flawed. Research has to be verifiable to be valid, and therefore must be held to a higher standard (both ethically and technically) than casual study.
But more importantly: There was no IRB here. Zed is arguing that this action is illegal and unethical in an employment context, not a research context. There's a world of difference between "take this $5 and play this game for me" and "take this $5 and play this game for me and while you do I'll rifle through your phone without telling you. It's okay, because I'm paying you." That latter example is closer to what's happening here. The fact that Riot is the holder of the data doesn't make their use of the data ethical or legal.
California is an at-will employement state.[1] Riot can terminate those employees for any reason not protected under federal or state law, such as being a member of a protected class. Being a gigantic dick does not grant protected status. The only liability Riot has here is for their unemployment, unless they can show cause for termination, which is probably more of a gray area.
If the IQ test anecdote about the horse is real I'd like to see a source. It's extremely hard to believe that even a malicious scientist could convince himself that a horse cannot be ridden into town.
> One question in the early test asks, “A man comes into town but his feet do not touch the ground. What is he riding?” The story goes that a young Native American boy responded, “A horse” to which the tester yelled, “Wrong! It’s a bicycle!”
Yes, indeed. The stench of bullshit is quite strong there, as is with the claim that the tests were adopted with the purpose of "resegregating the schools".
The closest I've been able to find is a claim (without references of any kind) in Gould's The Mismeasure of Man, where the actual question involved a man "walking without his feet touching the ground". Clearly "bicycle" is a better answer for that question than "horse" (because your feet move in a quasi-walking motion when you're riding a bike, but not when you're riding a horse).
The "yelling" part appears to have been invented out of whole cloth. Also, I'm not sure where the notion that Native Americans were unfamiliar with wheeled conveyances in the 1920s came from.
I would not comment on that particular anecdote, but the entire structure of literacy tests in the South during segregation was that an ambiguous or downright impossible test was written, and the testers applied it inconsistently. http://www.slate.com/blogs/the_vault/2013/06/28/voting_right...
I thought it was clear enough that I was making a general point about how tests can be applied, and illustrating it with an extreme example from the South.
In general, I think asking for clarification is better than being condescending.
But to make clear what I was saying, I really should have responded to the parent, who says: "It's extremely hard to believe that even a malicious scientist could convince himself that a horse cannot be ridden into town."
My point being that a malicious test-giver could certainly deny that as a valid answer. I don't know that I believe that anecdote happened, but it is assuming good faith to think that it couldn't happen because no one would ever reject "horse" as an answer.
These questions are indeed atrocious. The author's point would have been better served with these real examples. Over-exaggerating is like jumping the shark, and makes it difficult to entertain the premise.
Another point of difference in the Gould version of the...I'm just going to go ahead and call it an "urban legend"...and the Shaw version is that in the Gould version the Native American character is in the text of the question "An Indian who had come to town for the first time in his life...", while in the Shaw version the Native American character is the person being given the test, "...a young Native American boy..." Also Gould's version does not mention age or sex. Shaw's does. Neither gives a source. Both assume an incredible amount of naïveté on the part of the Native Americans by stating that they were unfamiliar with bicycles.
Until someone provides an actual citation to a first-hand witness, I'm going to go ahead and call this myth busted.
It shows :( I have a number of issues with this article.
First, the "I don't know what methodology they used but I bet it's bad" is not something you can credibly say about Riot. They famously hired a big team of academics and they've been studying the issue for several years now. They have more data to analyze than any academic could dream of, they've had time to iterate (institute a new policy or algorithm, punish players based on it, and see if it reduces bad behavior by some metric) multiple times, and at this point it's likely that they know more about what constitutes bad behavior in a video game than any other organization on Earth.
Second, if Riot's analysis of player behavior constitutes an illegal trial on human subjects (which sounds dubious to me), their EULA certainly inoculates them for it. Riot has been very publicly doing this kind of analysis on players and using it to punish them (up to and including deleting the stuff they've purchased in game) and if that was actionable it seems like it would've been tried by now. Maybe the EULA isn't good enough or would fail at trial, but it's a little presumptuous to just casually assume that such a novel claim would be a slam dunk.
Third, I'm no lawyer but it seems to me that CA 980 is irrelevant. It pertains to employers demanding access to social media they wouldn't otherwise have access to, not just social media in general. Under 980 it's illegal to demand an employee's Facebook password, but AFAICT perfectly legal to fire them for something they post on Facebook publicly. In this case, not only was the speech in question made publicly, it was made in a way that Riot indisputably has a legal right to access.
Finally, the author repeatedly refers to the behavior that got these people fired as "being snarky," when a much more reasonable assumption is that the behavior was significantly worse, e.g. racial slurs and death threats (both of which are very common in LOL chat). The source article tells us that, once Riot had identified the group of problem players, a committee examined the cases individually and decided which ones to reprimand and which ones to fire. Do you really think that "would we lose if this person sued us" wasn't one of the criteria? Is it not reasonable to assume that each of the people who got fired did or said something so bad that Riot felt legally justified in firing them?
tl;dr: "I don't know by what methodology these people were measured or by whom, or what they signed, or what they did to get fired, but I'm dead certain they have a case" does not sound persuasive to me.
> In addition to this, the employees are most likely expected to play the game or have to play it to do their jobs.
It is an expectation, and the vibe I got from how they described it (albeit as a visiting contractor) it seemed like anyone who says "you know what chuck, I don't think I want to play the game" would basically "not fit in".
Honestly it seemed like a kinda weird place to work, but maybe that was culture too (I'm not American and was just there for a few weeks)
The article talked about mining employee data before they became employees and flagging potential new hires based on automated sentiment analysis. I don't have a problem with a company trying to find examples of employees representing the company poorly. I do have a problem with using automated system to classify people into fire and don't hire categories based on something as inexact as sentiment analysis.
Any company should be terrified of using any kind of automated system like this to reject employees. If for whatever reason it starts kicking out a higher percentage of employees from a protected class, you open yourself up to huge lawsuits.
Sentiment analysis is an art, and I'd be surprised if their algorithm didn't have different accuracy rates for different dialects.
People that work for Riot do not usually use their riot accounts when just casually playing.
To put into a perspective this isn't about someone using a "GM" or some community manager account, it's about people using their own private account.
This is a bad precedent, it's not even about people trashing the company, or even doing anything "wrong" it's about people participating in a behaviour that the company doesn't like.
Let me put it into another frame of reference for you would you support an employer that fired or did not hire someone because they liked, or even wrote a political post on facebook?
Keep in mind that most of the political speech on facebook doesn't exactly look like an Aaron Sorkin script draft.
We already had a CEO of a company having to step down for donating to a political cause, and no matter how vile that cause might be, it's still a horribly regressive thing, not that they've donated to an anti-gay cause but that they had to step down over it.
Now CEO's and other fiduciary roles might be the "exception" rather than the rule, but let's face it employers datamine the hell out of your "online footprint" and it's a bad thing, I know people that weren't "hired" not because they had a bad social presence but because they didn't had one and the hiring manager thought that they were untrustworthy because of it.
What you say in public and in private should not have any bearing on your employment unless you wave the corporate flag at the same time, what Riot is doing is Philip K. Dick level of shady, and there are sadly way too many employers especially in the tech business that do the same thing and even worse.
We've already came to a point where your right to employment conflicts with your more important rights of privacy and free speech and we should strive to revert these changes not hail them as something good and somehow a victory for liberalism and social justice.
> Let me put it into another frame of reference for you would you support an employer that fired or did not hire someone because they liked, or even wrote a political post on facebook?
It doesn't matter whether I support this or not; its legal.
>We already had a CEO of a company having to step down for donating to a political cause, and no matter how vile that cause might be, it's still a horribly regressive thing, not that they've donated to an anti-gay cause but that they had to step down over it.
I'm assuming you're talking about Brenden Eich. He stepped down as a result of the PR optics of his politics, which made him untenable as the de facto head of the Mozilla community. He was not fired. I do not agree that it is repressive. If your views are so vile as to convince the majority of your users/customers that they no longer want to do business with you, then maybe the problem is your views.
> We've already came to a point where your right to employment conflicts with your more important rights of privacy and free speech and we should strive to revert these changes not hail them as something good and somehow a victory for liberalism and social justice.
One does not have a right to employment. You have a right to not be discriminated against for status as a member of a protected class. These are not the same thing.
That's not true. Riot expects you to have a level 12 account before hiring you, and then they convert that account to a riot account after hiring you. So the time you spent playing on your private account before joining the company counts toward your total player behavior. Additionally, I know several riot employees and they all play on their main riot account in their free time.
I know of other companies that have similar policies - e.g eBay - it's part of the application process to provide your username and background checks are done on it.
If you think you can be a jerk on the Internet and have it not potentially affect employment then you're being naive. Most interviewers will do an Internet search on your name and dig up as much as they can about your behaviour on platforms and social networks.
If true, I'd argue this is evidence the problem is more widespread than realized (rather than "normal." Just because something is common doesn't mean we should accept it).
I think this issue is particularly daunting because free-information seems good, but having been to ridiculous interviews before we are all aware how biasing a small amount of irrelevant data can be.
I'm not sure I see anything illegal in there (from the perspective of PA employment law anyway). His biggest beef seems to be that they mined data without their employee's consent, but most of the places I work explicitly state that emails, chats, electronic and physical documentation (work products) all belong to the organization.
On one hand, this feels a bit scummy - on the other hand, I'd rather work somewhere that's not toxic. I think the biggest issue is that I doubt their technique is good enough to a) find all the toxic people (one of the most toxic people I ever met was outstanding at never putting it in writing) and b) will exclude people that could have improved the company.
I respect Zed for his many contributions, but this is among the most ridiculous diatribe I've read all year. We'll ignore the whole at-will and protected class thing which is all that matters (legally).
> Their use of statistics to bucket employees into “reprimand” or “fire” buckets is incredibly dubious
Do you really think HR looked at some bin graphs and said "Ok lets fire these aholes!".
Software helped surface specific data from a larger data set based on specific rules. This was then reviewed by humans - remember them? - that decided some examples of behavior were so horrible that they felt compelled to confront specific individuals. I'm going to give them the benefit of the doubt [1].
> ... using data based on their behavior without their knowledge
Anyone have a Riot employment contract handy? Even if it doesn't explicitly mention data collection, I'm willing to wager the LoL terms of service has it covered. Every employment contract I've at looked at in over 15 years in tech has had some sort of clause around data collection/monitoring being an open option. Does it give me the Orwellian chills? Sure. But it's not illegal or unethical. Some industries like finance are legally required to in many situations. Boo hoo.
> I’ll bet you $100 the second this study was discussed all the managers went and deleted their history.
Riot centrally logs chat deemed (subjectively) toxic. Middle managers at a company this size do not just walk up to someone with production access and ask them to delete data in a complex system, especially not en masse.
From the original re:Work article:
> Riot identified the 30 most toxic employees (all of whom were more junior Rioters, new to the working world)
> While these logs did factor into the exits of a couple employees who had already had serious problems
Let's set some context. Smart folks at Riot had a hypothesis that ended up identifying 30 out of some ~2,500 [2] employees (~1.2%) by legally analyzing chat logs. Riot discovered, unsurprisingly, that all of the employees identified happened to be new to the workforce and the company and used this opportunity to meet with them individually to work out a plan to improve their behavior.
Unfortunately a "couple" with past incidents were let go. The question is whether or not these specific employees would have been terminated if not for this hypothesis. Bigger question is what Riot will do so that the problem doesn't escalate to this level again (new hire orientation sounds like a good start).
Great job Riot for using human process and technology to try and make a workplace better. Poor job re:Work for over-romancing the software involvement in the process and turning an awesome example of work place improvement into a bad PR puff piece on "people operations", and worse job Zed for turning this into an emotional appeal of fear of an authority state. "But what if they use your chat logs before you worked there?! Or recorded headset audio?" It holds as much intellectual merit as the mob responses we see when Facebook or Instagram make changes to how stories are sorted. Sorry pal, I've lived in California too long waiting for "the big one" to buy into world-ending FUD.
"To understand how this happens you have to understand that the Stanford-Binet IQ test was explicitly created to re-segregate the California school system and prove that whites were better. "
This is nonsense... the IQ testing came out around the first world war. It was first used to test disabled children and then used to test military recruits.
----
Two people without the courage to actually respond voted me down... sorry, but Zed Shaw is talking out of his arse... the idea that the "Stanford-Binet IQ test was explicitly created to re-segregate the California school system and prove that whites were better" is just incredibly stupid.
I.e., if I personally want to pay people $5 to play a video game, and then pay them $10 if they win, there is absolutely nothing preventing me from doing this. I can also gather statistics on the result. But if I were a scientist at a university doing this, I'd suddenly need to get permission from an IRB.
This is a flaw with academia, not Riot Games.
The comparison to the Stanford Prison Experiment is nonsense. That experiment literally locked people in jail, an action that is generally illegal in general. This experiment literally involves nothing more than a group of people statistically accessing records they have legal permission to access and then doing some math.
Note that I say this with no positive feelings whatsoever about Riot Game's PC-police behavior. I don't generally favor "no-asshole rules", or any of the other feelings-over-facts policies currently favored by the PC-left.
[edit: since folks reading this clearly think I'm criticizing Zed's entire post, Zed has 8 arguments against Riot. I'm disputing (3) and (4) only.]