Most people don't like randomly being challenged. The quickest way to find a match is click 'Find' and load a team, then 'Find Match'. You can pick tier. Also, they can't see your team, and random challengers are notorious for carrying random Mewtwo etc. You can challenge them OU from the Unrated side of the Find tab, to make sure neither have Ubers, but it'll still be hard to get consistent matches. Most 'serious' battlers use the ladder, so it's a fast way to get good opponents.
To anyone reading this thread: if you hate being challenged, you can Status > Away to avoid them, just in case you didn't know...