From a post I just made on the Paizo forums, which was long, complex, and potentially useful enough that I wanted to reproduce it here.
Rules cannot be copyrighted.
An individual expression of the rules can (e.g. the text describing it), but the underlying idea cannot. If I put a rule in a product stating that to move around the board you roll two dice and move that number of squares, copyright only applies to the specific text I write in that rulebook. Anyone else can use the same rule by rewording it.
It’s important to remember that “rules” here means the idea of how the rule works, not the descriptive text itself, and especially not “rulebooks” – the latter two are very, very copyrightable.
Terminology is copyrightable
At least, it is if it consists of a long enough string of words for copyright to apply. If I refer my rule of rolling two dice and moving that number of squares around the board as “The Cool Movement Track of Doom”, then chances are my claim to that specific phrase can hold up in court. There’s no real hard and fast rule here, unfortunately, as it’s still one of those murky areas where it could go one way or the other depending on exactly how unique a term or phrase I happen to use, and how blatantly someone else tries to copy it. The opinion of the judge on the day matters, as does how well my lawyer puts my case across, and how well your lawyer is able to weedle you out of it.
Spell names, for instance, are pretty easily covered here.
You can make your game with exactly the same rules as mine, written in your own words, and call the above rule “The Amazing Propulsion Range of Death”.
Rules can be Patented
…. That is, unless I patented that rule. If I did that, there’s no way around it, you just have to write a brand new rule of your own that works differently to mine.
Thankfully, I’m unaware of anyone having patented any RPG rules. There may well be some I’m unaware of, of course.
Terminology can be Trademarked
“Dungeon Master” for example, is a registered trademark of Wizards of the Coast, used here by me right now without permission for purely illustrative purposes (and for the truly curious, I’m able to do that through the nominative usage clause of Fair Use, as I’m only using it to refer to the term and its source and not trying it use it on my own product here.) We can make our own variants that mean the same thing, but we can’t use that term. Always check for other trademarked terms in the products you’re utilizing for inspiration.
The OGL gives you restrictions as well as rights
As has been mentioned, under the OGL you agree not to use other companies Product Identity, which means checking their product for what exactly they’ve claimed as Product Identity. Even if copyright law would let you get away with it, by using the OGL to licence parts of a work you’ve also agreed to some additional restrictions as well. The OGL works two ways – both to protect and sublicence different parts of the licensed work.
Mostly, we tend to use the OGL in order to make use of terminology that would previously make us the subject of a copyright claim. For example, setting up a game in which the attributes are Strength, Intelligence, Dexterity, Wisdom, Constitution, and Charisma, and in which your level of protection is known as an “Armor Class” could quite possibly be treading on somewhat murky legal ground. By using the OGL, we (and the OSR guys especially) are able to use these terms in our works with a degree of protection (and yes, you’ll need a lawyer to explain exactly how much of a degree) that we’ve been licensed the right to do so.
In addition, the OGL also gives us not only terminology but access to 95% of the base rules of the 3/3.5e, as those rules have been declared open content in the respective SRDs. Pathfinder goes a little further and makes it 100% of its base rules (sidenote: if you also use the Pathfinder Compatibility License, you may not produce a complete working game using the Pathfinder rules. If you only use the OGL, however, you can.)
What the OGL doesn’t give you
This one’s a bit of a pitfall for first-time authors, so I’m adding it in here because I’ve seen too many people get hurt by it. The OGL doesn’t give you the right to reference other products, outside of section 15 obligations. You can’t refer your readers to someone else’s product without that publisher’s permission, even if it would seem like a nice thing to do at the time (“hey, this is a great place to use The Leaping Idiot character class by Hairy Bert Publishing” – sorry, you can’t do that, the best you usually can do is reprint the rules to that class they’re available to you through the OGL. You can, of course, contact Hairy Bert and request permission, and many 3PPs are more than happy to agree if it results in promotion of their work from within yours.)
What the OSR people did
Here’s the nitty-gritty part that answers the OP’s original question:
To the best of my understanding.
The OSR people used the terminology available via the 3/3.5 SRDs through the OGL, as it matched the terminology from previous versions of the game (thus avoiding potential trademark/copyright problems with those terms). They then used the rules from those previous versions, rewritten to avoid copyright problems (in some cases they may also have been able to use the OGL’ed 3.5 rule if it matched the original closely enough). Finally, they had to ensure they’d filed off the serial numbers in the case of any terminology not provided through the SRD/OGL.
So, to put it very basically – they used the OGL in order to be able to reference the names of ability scores, hit points, armor class, and saving throws. They then used the fact that “rules cannot be copyrighted” to rewrite the rules from their desired edition of the game in their own words. Then they found certain terms were not granted via the OGL, and had to substitute similar but different terms in a few cases.
And the most important rule:
Don’t be a jerk
Bear in mind this is people’s livelihoods, and don’t try to come out with a product that directly lifts all the content from and then replaces someone else’s, unless you’re doing so purely through the OGL. Filing the serial numbers off a non-OGL product currently still in print (and I count PDF availability here too) isn’t going to make you any friends 🙂
While doing the same thing with an OGL product does just as much damage to the original publisher, it’s seen as somewhat more acceptable due to the fact said publisher knew (or should have known, at least) what they were getting into when they started doing OGL products, and should be prepared for OGL clones of their OGL product to spring up here and there.
Disclaimer and References
I’m not a lawyer, I don’t claim to be. I have, however, worked in this industry for enough years (and been poking at the OGL in particular ever since it was publicly available) long enough to be familiar with what I as a writer can and cannot do within the confines of the OGL without getting my publisher into hot water. I still wouldn’t want to push those confines myself without getting proper legal advice, but I believe I know enough to know what’s absolutely okay within the OGL, what absolutely isn’t okay, and what lies in the big grey area of “well, I need to get some proper legal advice on this”.