The first thing we need to address is what a role playing game is. I don't mean the, "you create a character then pretend to be that character," aspect but what it means to actually play the game. Generally, I break the game down into four areas:
Mechanics refers to the actual rules of how one plays the game. A player who really enjoys deciphering the grappling rules in [any system ever] is taking pleasure in the Mechanics of the game. Mechanics might be considered the opposite of Dramatization, which is the assumption of role, acting, etc. Simulation is the intersection of Mechanics and Dramatization, where the rules are invoked specifically to represent some aspect of the game being described; usually this is combat but anything where the GM says, "Roll to see if X" is a Simulation moment. Finally, Socialization is everything that's not actually part of the game but that takes place when we play games, such as ordering pizza or chatting about Star Trek vs. Star Wars.
For the record, it is possible to create entirely different breakdowns, but this what works for me.
One of the biggest problems when trying to decide how to design a game is estimating how each group likes their balance of these four aspects. Some games lend themselves very well toward one aspect or another. Nearly every edition of D&D, for example, is very interesting from a Mechanical perspective to the point that many people "play" the game entirely by designing characters, traps, etc. because their fascinated by how the game rules interact with each other. Larps tend to be very rules light because they're largely focused on Dramatization so the rules that make Mechanics and Simulation function are less relevant. Other examples quickly become apparent.
While game design encompasses many aspects in each area, I want to talk about rules in specific because of their tendency to influence and be influenced by each area. In particular, I want to address the issue of rule complexity.
Rule complexity is an aspect of Simulation that directly affects both Mechanics and Dramatization. The more complex rules are, the better Simulation becomes possible (notice that I say "possible." Also note that I don't say "realistic") because the rules more accurately represent what the game designer is attempting to express. In Shadow Run there are very detailed rules about the effects of cyberware because one of the themes of the game is the definition of humanity. In 7th Sea there are complex rules regarding each magic system because the differences between each style of sorcery is an important part of the metaplot. Again, examples of games creating complex rules to create a more intricate Simulation become obvious.
Complex rules don't just draw attention to their area of Simulation, however, they also draw attention to the rules themselves. Thus the more complex rules the more emphasis on Mechanics and the less emphasis available for Dramatization. This is not to say that Mechanics and Dramatization are opposed, but it is a nearly-inevitable effect of rules to shift emphasis from one to the other because the more complex the rules are, the more the players become aware of those rules which breaks into the player's awareness, "Hey, I'm playing a game." At that point the player is less likely to think about what's happening in the game but rather how it is happening.*
Now there are many roleplaying purists, gamers whose favorite part of the game is Dramatization, who lament complex rules for this reason. I know a great many players who are incredibly intelligent and capable of understanding complex rules but who favor simple systems because it avoids the distraction of Mechanics. That's fine. Their games are largely about Dramatization and rules are only invoked when absolutely necessary for Simulation purposes.
Dana's 7th Sea larps at Gen Con are a great example of this trend. Dana has created his own system in which each character "sheet" is reduced to three or four numbers that can fit on half a playing card. For something like a duel, players each draw a card, add the number on their card for Combat, and whoever scores higher wins. That's the whole system and it works damn well. Does it leave out the intricacies of all the Swordsman styles that are part of 7th Sea? Absolutely, but his larp isn't about those styles, it's about the conversations between the people who pratice those styles.
The counterexample is 4th edition D&D, an incredibly complex (though not difficult to understand) game in which the mechanics are so pervasive that entire aspects of the setting have been altered to reflect them. While the formalization of roles within the party of leader, defender, striker, and controller has done wonders for balancing game play it specifically affects how the characters think of themselves within the game. This isn't a bad thing, however, as one of the main joys of D&D is small-group tactical combat. That's why people like to play D&D and it works for them. If you enjoy taking apart mechanics and looking for the most effective or the silliest or the most intriguing combination, D&D 4e is not a bad way to go.
When one is designing a game, one most consider what type of game one is creating and who is expected to be playing it. One must consider what sort of experience they're playing to have and what sort of experience your rules are going to give them. The ending of The Gamers II when Joanna has Daphne use her wish to advance Dramatization and Cass is incensed at her for not using it to advance Mechanics illustrates the conflict that can occur when the designer guesses wrong.
There is another issue of rules complexity that needs to be addressed, one that was actually my original idea behind this post and I've been trying to work my way around to the entire time: that of interaction. A min-maxer is a player who tries to find an optimal combination of rules and character options to achieve a specific effect, stereotypically combat effectiveness but I've seen other min-max builds such as social dominance, or even a single ability normally out of reach of most PCs. The more complex rules become, the more min-maxing is possible.
Let's consider a simple game. In our game, every character has one trait. Let's call if, "Effectiveness." A character's Effectiveness is compared to the Difficulty Rating of a task, such as climbing a wall, out-witting a sphinx, or beating an opponent in combat. Both Effectiveness and Difficulty Rating are numbers on a scale from 1-10. If Effectiveness is higher than the Difficulty Rating, the character succeeds. If not, he fails.
This seemingly simple system - there aren't even any dice! - actually affords quite a bit of complexity. We could introduce abilities that allow the character to boost his Effectiveness in certain situations, to lower Difficulty Ratings, to give a bonus to the next check if the character beats the Difficulty Rating by different amounts, to allow characters to add their Effectiveness, and so on. Each of these areas is something that can be manipulated. What it doesn't allow is min-maxing because most of it is out of the players' hand. This system may do very well on Simulation (in fact, I'd say such a system is perfect to simulate Jedi), but very poorly on Mechanics because it's not very Mechanically interesting.
We could make it more Mechanically interesting by dividing Effectiveness into three traits: Physical, Mental, and Spiritual. Now the GM needs to decide which to use for each challenge. The game gets more complex with abilities that swap or temporarily boost each score. Min-maxing becomes possible as players build up one score at the expense of others.
We can make it even more complex by splitting the traits down further or adding dice rolls to each trait when comparing against Difficulty Rating (the system used by most tabletop RPGs). Dice rolls allow for entire new set of complex mechanics based on interacting with the dice rules - think about all the rules to roll and keep dice or affect how they explode in l5r.
Each step to make the game better at a Mechanics level opens it up to abuse. An ability that says "You may add your Spirit to your Physical scores whenever you roll a die" may be powerful, and an ability that says, "Three times per session you may add your Physical score to your Spirit" may be fine, but taken together that means once per session that character is adding 3x his Physical and Spirit to a roll. This game doesn't even exist yet and we've already broken it!
This is why RPGs need to hit a reset button every few years and release a new edition. Seemingly innocuous rules become abusive once it's seen how they combine with existing rules. Core rules that were tested over and over in playtesting and found to be perfectly balanced and functional as core rules become unbalanced or overshadowed when less-tested rules are introduced in supplements (I'm looking at you Hojatsu's Legacy Duelist!).
The end result is that while a game may be established with an optimal level of complexity and balance between the four areas - or at least an acceptable amount of customization available for each group of players. With each new supplement, however, the complexity of the rules goes up by definition as new rules and character options are introduced. There is thus an inevitable creep toward a Mechanics-intensive game over time.
My response to this as a game designer, at least for my home games, is to simply not care about Mechanics. I expect players to have broken characters and I don't concern myself with that. If that's a part of the game they enjoy, if that's what makes the game fun for them, why would I want to fight that?
Rather than try and "fix" their character, I strive to recruit like-minded players who will enjoy that area of the game as well. This isn't to say that I will never jump in and block something with GM fiat (I'm looking at you every feat from Book of Exalted Deeds but as a GM who's their to have fun as much as all the other players, I'd rather say, "yes," then move on to the parts of the game that interest me.
This becomes much more difficult when designing games blind. When I write a module for Heroes of Rokugan, I don't know who's going to play it. Now each of the modules that I've contributed so far has, admittedly, been written with different players in mind. In fact, if you look over "Touch of Death" (the rejected module) you can figure out exactly who each scene was written for. That said, it still needs to be playable and enjoyable by people who've never met me.
The key to blind design is to put the focus on Simulation. Rather than writing mechanics, write for theme and story and tell the GM what they are so that she can adapt on the fly based on the group she has. As a game designer I don't want to have to list Target Numbers for every delegate in "An Arranged Marriage" in response to persuasion by coercion, bullying, bribery, seduction, deception, philosophical argument, enlightened self interest, etc., but GMs may need that information, and in fact they probably will because if I list just one method it's pretty unlikely to be the one the PCs try. On the other hand, I can say, "This character response very well to arguments based on moral virtue. A PC may roll Awareness/Lore: Bushido at TN 40," create appropriate listings about the best argument for each NPC, then a note at the end that says, "The GM should adjudicate other methods of persuasion as appropriate, but the TNs will be at least +10 higher." Thus players can use whatever methods they like rather than my reminding them they're playing a game by telling them what to roll every time.
There is one last area that affects all of this, and that is house rules. House rules are changes to the game specific to a given group of players, a campaign, etc. One of the things I'm loving that's been promised about 4e l5r is a section on house rules in the core book along with a description of what effect each will have.
As a GM, though, I try to keep my house rules rather flexible. At D&D on Sunday, for example, I adapted the rules of called shots (striking a particular location on a target) from Mongoose Publishing's Quintessential Fighter.
Most games handle called shots by saying something like, "You may voluntarily increase the difficulty of your attack by 4 before you roll. If you succeed, you hit the opponent's arm." There are two things I don't like about this approach. One, it makes no sense to me that if you're aiming for a particular point on a subject that you either hit them there or miss completely - if I miss my opponent's eye because I roll a 22 instead of a 23, why don't I hit the rest of his head? Second, it means that for upper-level characters there's no reason not to make a called shot every round, at which point they stop being special and start being paperwork.
Mongoose had a particularly creative solution to this when QF was published back in 2001. They said if you made a critical hit, you could choose to waive the extra damage and instead make it a called shot instead. What type of called shot you could do was limited by your Base Attack Bonus, but there were great rules for the effects.
I loved this approach because this is exactly what critical hits are supposed to represent: a strike to a vulnerable point. A standard critical hit that did extra damage now represented a called shot to the torso or head or other point vulnerable to wounding, but a player might forgo that extra damage to instead turn it into a called shot to the leg to slow their opponent down, to the hand to cause a penalty to attack, etc. Furthermore, because it was still dependent on those rare critical hits, high-level players couldn't abuse their high attack bonuses to make critical hits every time they attacked, thus keeping them special.
For my game I partially adopted the rule. On a critical threat the player can choose to make it a called shot (if they're going for the eye, groin, or other small, target area I may increase the opponent's AC; if the confirmation misses they still get a regular hit for the surrounding area, but an eye should be harder to hit than a leg). The effects of the called shot, however, are completely at my discretion as GM. This makes them more risky for players, who still have the option of just doing normal critical hits for extra damage, because I may have it do a weaker effect than they'd like, but it means I can adjust on the fly to keep the game balanced, while still allowing dramatic combat. Ergo, Mechanics, Dramatization, and Simulation.
Then if someone gets me hot wings we get all four areas.
* Note that I use a great many qualifiers in this paragraph, such as "nearly-inevitable" and "less likely." I am describing trends, not immutable laws. Each group - indeed, each player - may buck these trends, but the overall tendency will remain outside these counter-examples.