Reflections and Expectations of Smogon's OU

[Freze]

If the theme question was to decide on an issue of whether themes are good then I must say the answers were poorly worded. One was if themes are good and the other was if broken themes are bad... I mean I can't get the quote, but really that is a yes yes question and if its task was to find if themes are good or bad then I find it a bad set of answers. Not bashing as I understand you are also human therefore not perfect and the survey as a whole was good.

 

[Cafemocha]

Interesting quiz. Even though i'm more of a lurker and never got really into OU, I think the quiz really made me realize why I can't keep up with it at all. I didn't really have any problems answering the questions though, some of the questions were worded in a way where I wanted to pick both but, that's forcing an answer for you. It'll be really interesting to see the results. [Wow I feel so out of place with my low post count and off putting join date.]

 

[Chou Toshio]

If the theme question was to decide on an issue of whether themes are good then I must say the answers were poorly worded. One was if themes are good and the other was if broken themes are bad... I mean I can't get the quote, but really that is a yes yes question and if its task was to find if themes are good or bad then I find it a bad set of answers. Not bashing as I understand you are also human therefore not perfect and the survey as a whole was good.

There's no good or bad-- that's not the point either. The point is to find out which of these 2 reasonable sentiments do you sympathize with more.

 

[-Sparkbeat-]

The thing about the theme question I had a problem with was grouping perma weather with turn limited things like Trick Room, since having something permanent unless it is replaced by something else is very different (And obviously much more viable and powerful, and game breaking) than something limited by the amount of time it lasts.

 

[Chou Toshio]

Okay, I just found out that my free account only lets me collect up to 250 responses, and I already have 240. So, I will post initial results soon most likely.

Edit: Thank you for participating. I passed 250 responses, so I closed the thread. I will post results soon, along with some of my own analysis. There's a bunch of things I want to check, but cross-tab or stat analysis may take more time than just posing the results (which I will do sooner).

I'm still just getting my feet wet in market research, so I'm sure I made a lot of silly amateur mistakes in putting the survey together, but thanks for participating. :)

 

[Chou Toshio]

Alright guys, so click the link here to take a look at the Survey Results:

Survey Results

First off I'd like to thanks everyone for participating. We got over 300 respondents which is cool.

I haven't done a whole lot of analysis yet, but here are some interesting facts I've looked at/know:

A) Do most non-OU players play other tiers/metagames because of a grievance with OU?

9.Did you choose your primary metagame in-part because of a grievance with the current OU?
The results are not statistically significant. I ran a chi squared analysis (null hypothesis Yes = No in population), and I only came out with a p-value of .3 (.2 earlier in the survey, got less significant later).
So while more people cited that they had a grievance with OU as a reason to play other tiers, there's no way to prove these answers are not a result of normal variation. You can't say statistical certainty that most non-OU players play their tiers because of a grievance with OU
This is further supported in that question 10:

A-b) Does the popularity of other tiers reflect poorly on OU?
10.Please tell me if you agree with the following statement:"OU should be made to be the ideal metagame. If other metagames are greatly popular, it reflects poorly on the design of OU."
There is no statistically significant consensus about whether other metagames being popular reflects poorly on OU. Generally though, it looks like people disagree with the idea that other tiers being popular means OU is a bad tier.
I did a cross tab with question 4, and the Agree v. Disagree is ratio is close to uniform across every tier.
VGC is the only metagame who has more Agree than disagree. It's hard to conclude ANYTHING about VGC though, considering barely any VGC primary players took the survey. lol




B) "ADV is the best gen ever~" Do most players who have ADV experience prefer ADV over other tiers?
Turns out no.
I did a cross tab between questions 4 and 5, and while 77 players indicated that they have ADV experience, less than half of them indicated that ADV was their favorite gen.
Out of 77 respondents with ADV experience, their favorite OU were:
RBY 1
GSC 5
ADV 32
DPPt 36
BW 3
So not only was ADV not picked by a majority of ADV players, but ADV was not even the most popular gen, with DPPt having the most votes.
This is especially noteworthy alongside

C) DPPt, BEST GEN EVER???
Out of 178 respondents with DPPt experience who answered question 4, 116 (65%) indicated DPPt as their favorite gen OU. Furthermore, DPPt got the most votes in question 5

5.Which generation did you consider to have the best OU?
(Note, question 5 was only asked to players who have experience in more than 1 gen's OU)
With 119 votes (3 players with no DPPt experience indicated DPPt as the best OU gen, lol), DPPt got 62% of the votes in question 5 (2nd place goes to ADV at 22%).
Yes, a lot more players have DPPt experience than ADV in this survey, but I think the fact that DPPt is more popular than ADV even amongst ADV players and the fact that DPPt got almost 3 times the votes of DPPt in the question 5 vote (far surpassing the percent difference in players with ADV v. DPPt experience) speaks a lot of DPPt's popularity.
I find this particularly interesting because during DPPt, people constantly bitched about DPPt being too "out of control" offensive. People constantly indicated that DPPt was too fast paced, and ADV had a better balance. Maybe BW has changed people's perceptions of what "out of control" offensive really is. lol

Edit:

D) People want a more offensive metagame

15.On a scale of 1 to 10, where 10 is "Extremely Offensive" (where Heavy Offense is the primary style) and 1 is "Extremely Stall-Based" (where Full/Traditional Stall is the primary style), how offensive would your ideal metagame be?

So I ran some hypothesis testing and got the following:

5.000 hypothesized value
6.354 mean Data
1.423 std. dev.
0.091 std. error
246 n
245 df

14.918 t
1.53E-36 p-value (one-tailed, upper)

Null Hypothesis: M = 5

Basically the odds of getting a mean of 6.35 from a population with a mean of 5 or less is 1.53E-36

Put plainly, The odds of getting a mean of 6.35 (slightly offensive metagame) from a population that actually wants 5 (neutral) or stally (less than 5) metagame is extremely low. It's far less than 5% (2 standard deviations).


There is a bunch of other stuff I wanted to look into but haven't gotten into yet. In the mean time feel free to scrounge through the data yourself guys. Have fun. :)

 

[Notty]

The responses to 11 and 12 are seemingly contradictory.

A strong majority (77%) agree that an ideal metagame has great diversity, but then a plurality (53%) turns around and says that there should be a limited number of threats so as not to overwhelm team building.

If you assume that the 11% that disagreed with the notion of promoting diversity remained consistent, that still leaves 42% of respondents who think that we ought to have as many threats as possible, yet should still be limited enough to be able to account for the majority of them in team-building. Which is just huh?

Anyway, fantastic work on this, Chou. Looking forward to seeing more in-depth analysis.

 

[MMII]

I think it mostly says that players want diversity within reason. They want as many options as they can without having so many that they can't account for every single one.

 

[Champion Steve]

Well, something can be viable without being a threat that you HAVE to deal with (For example, when was the last time you complained about a team being Espeon weak). That the questions were phrased differently might have led to that. Perhaps some people want diversity like that? This is merely a hypothesis, of course.

 

[Chou Toshio]

I think it mostly says that players want diversity within reason. They want as many options as they can without having so many that they can't account for everything single one.

This is exactly my thoughts on the subject. I built these 2 questions knowing they were contradictory, but I pretty much expected results like this. It's not a matter of hypocrisy, but rather that people are looking for moderation between "diversity" and "reasonable limit" in the metagame-- there's a sweet spot of not too little, and not too much.

Looking back on it, I really should have randomized the order of those 2 questions in order to avoid bias... ah... live and learn... :/




My thoughts (extrapolation, unfounded in the data)--

In DPPt, you'd have probably seen a LOT more "Agree" in question 11-- because "lack of diversity" / "Over centralization" was the hot subject, the king pin, the thing everyone always bitched about. And that's because almost everything outside the top 30 or so mons was extremely lacking in viability.

That culture has weakened in the face of BW, where even countless UU and lower mons like Victini, Amoongus, Ditto and Zapdos can rip your OU team a new one if you underestimate them-- but the roots of it are still there, the sentiment still very strong. The whole system/culture of tiering that was built in DPPt and carried over to BW was focused on this concern about over-centralization; so we haven't been able to shake it completely even though BW definitely has the most diverse OU with the greatest number of viable threats ever.

It's pretty ironic to remember the attitudes of DPPt-- how EVERYONE bitched about the game being overly centralized and lacking diversity-- and now that we HAVE a metagame with TREMENDOUS diversity, NO ONE IS HAPPY. It is far less satisfactory than DPPt (looking at the numbers, I don't think we can argue that DPPt OU has a lot stronger positive support from the respondents).

On the other hand though, the reverse attitude is starting to become more popularized. Centralizing mostly around tournament players, attitudes such as "too much diversity is bad"; "games are coming down to just team match ups"; "You can't prepare for everything, so the game comes down to luck over skill"; etc. is emerging-- based on question 12, it's getting some sway.

I think this small but growing sentiment is also behind the results of question 14:

14.Which is the more important skill to be valued and encouraged in competitive Pokemon-- "Team Building," or "In-Battle Decision Making"?

While the greater majority of players indicated that the two skill sets are to be equally fostered, the number of respondents to answer "In-Battle Decision Making" was more than double the number who answered "Team Building."

 

[jpw234]

While the greater majority of players indicated that the two skill sets are to be equally fostered, the number of respondents to answer "In-Battle Decision Making" was more than double the number who answered "Team Building."

What? Why? I actually don't really get this, will somebody who holds this opinion (in-battle decision making > team building) explain this to me?

 

[Houndoomsday]

What? Why? I actually don't really get this, will somebody who holds this opinion (in-battle decision making > team building) explain this to me?

i might elaborate on this later but i believe that a pokemon game should be won in the game itself and not pre-battle.

 

[ShootingStarmie]

What? Why? I actually don't really get this, will somebody who holds this opinion (in-battle decision making > team building) explain this to me?

While I'm not saying I agree or disagree with this post (I voted they were equal of importance), I believe people think that someone who has an awesome team (maybe took it from the RMT section), but don't know how to use it will not do well in battling, hence why in battle skills are more important. However, if two equally skilled players face each other, but one has a slightly less well built team, chances are the player with the worse team is going to lose. I'd say both are equally as important, as creating a bad team gives you less opportunity to use your in battle skills, while having a good team and not knowing how to use it isn't the way forward either.

 

[Chou Toshio]

^Actually, the question is not "which is more important", the question is "which should be more valued (by the community)".

ie. As a community, which should we try to emphasize/foster?

If we are going to build a metagame (with bans), is it better to build a metagame that:
a) Supports team building-- by having a great diversity of threats, you give people more team building options AND FORCE THEM to deal with a greater number of threats in team building. Great meta diversity puts emphasis on team building skills.
b) Supports in-game decision making-- by having a lower diversity of threats (fewer threats), you allow people to be more educated about what their opponent can use. You also take the pressure off their team building skills (they don't have to be as prepared for as many threats), and lower the chance of losing "because of bad team match up."

Example: Losing because of bad team match ups is a frequent complaint in BW (where we have the most diverse OU metagame). Losing because of team match up is basically non-existant in RBY (where there are like, 10 OU Pokemon, lol)

 

[asbdsp]

D) People want a more offensive metagame

15.On a scale of 1 to 10, where 10 is "Extremely Offensive" (where Heavy Offense is the primary style) and 1 is "Extremely Stall-Based" (where Full/Traditional Stall is the primary style), how offensive would your ideal metagame be?

So I ran some hypothesis testing and got the following:

5.000 hypothesized value
6.354 mean Data
1.423 std. dev.
0.091 std. error
246 n
245 df

14.918 t
1.53E-36 p-value (one-tailed, upper)

Null Hypothesis: M = 5

Basically the odds of getting a mean of 6.35 from a population with a mean of 5 or less is 1.53E-36

Put plainly, The odds of getting a mean of 6.35 (slightly offensive metagame) from a population that actually wants 5 (neutral) or stally (less than 5) metagame is extremely low. It's far less than 5% (2 standard deviations).
There is a bunch of other stuff I wanted to look into but haven't gotten into yet. In the mean time feel free to scrounge through the data yourself guys. Have fun. :)

I remember this question quite well and it ticking me off a bit. The problem is that the middle of 1-10 is 5.5, not 5.

 

[Houndoomsday]

while i definitely liked the survey and feel it provided a lot of talking points about our current meta and what we want the perfect meta to be, some of the aforementioned errors in surveying put the validity of the results in question

 

[Princess Bubblegum]

Losing because of bad team match ups is a frequent complaint in BW (where we have the most diverse OU metagame). Losing because of team match up is basically non-existant in RBY (where there are like, 10 OU Pokemon, lol)

Eh, I wouldn't go this far, if your teams lacks some key Pokemon one can have issues with some BL/UU Pokemon. Also in the standard wrap-meta, one is arguably at a disadvantage if they face Dragonite without a Gengar on hand. There can also be situation problems, like if someone gets rid of Slowbro via Exploding, but they have a Gengar, you are certainly having issues.

 

[Chou Toshio]

while i definitely liked the survey and feel it provided a lot of talking points about our current meta and what we want the perfect meta to be, some of the aforementioned errors in surveying put the validity of the results in question

I don't think 'validity in question' is true, nor would I say I committed great errors in survey construction, however from this first one I learned a lot about how to make a better one! Hopefully I can get permission to do so, and hopefully it will still be interesting enough that people want another one (probably on similar subjects though).

As for the questions included here though, there's no reason not to believe or use the data here. The improvements I would make would get be more data, and more useful data-- not more accurate data most likely.

 

[Houndoomsday]

i dont want to get into an argument about this, and i really like it in general. what i was trying to say was this had a few things that could have affected the user's response, 1-10 scale with 5 as the median being an example, but it's just small stuff that you dont pay attention to can have a huge effect on survey results. small changes in the wordings of questions can make a huge difference.

i wasnt trying to attack your survey or anything, and really do like the concept, but i was just trying to point out running statistically analyses and expecting them to completely hold up an argument isnt going to work with this survey. basically if someone uses the survey results, use them to bolster a claim, not make a claim

 

[Tassa]

What? Why? I actually don't really get this, will somebody who holds this opinion (in-battle decision making > team building) explain this to me?

You can copy a team. You cannot as easily copy how to use it, what you should do against meta threats etc.