Ban Confusion?
Yes, while we're at it, let's get rid of secondary effects, ban Cute Charm, kill crits, and make the RNG consistently roll 0 to ensure moves never miss. /sarcasm
The Pokémon that use it in question are annoying; that's what they do.
But one thing in common with all of them (Bar the Genies) is that they all pretty much suck stat-wise, so, taking care of it should not be a massive issue.
It doesn't even have to be an offensive solution, you have Magic Coat/Magic Bounce, Own Tempo to get them out, Taunt on a faster Prankster user, Torment to stop them going Foul Play mode, Lum Berry, and more. I understand that you wouldn't want to change your team around, but isn't the whole point of teambuilding to adapt to an occasionally changing metagame? If the popularity of this tactic annoys you, then respond to it!
On top of that, it wears off after a few turns. If you're still in, you can dent them quite a bit on the turn that you lose the confusion ailment.
Don't get me wrong, I do agree with it being noncompetitive and an underhanded tactic, but it's no different to those who constantly use Roar/Whirlwind on hazards, or ParaFlinch abusers. Regardless of how you complain, it will always be a viable tactic, as it does its job.
However, getting rid of it just because you can't deal with it is even more noncompetitive. That being said, those who use 6 mons with identical movesets + Prankster do need to be nipped in the bud.
tl;dr
My opinion? Don't ban Prankster + Swagger at its core, instead, a more complex ban that prohibits 2 or more mons from having Prankster + Swagger/etc on the same team would be more suitable. That way, those that do use the tactic to aid in teambuilding won't be affected, and this only hits those that aim to troll.
Who cares if it's luck based? For as long as the RNG exists, the game will -always- be skewed by luck.