I tried to capture a DM-posessed NPC and experienced epic failure. I could swear I remember reading about a way to subdue NPCs, but I couldn't find any reference to it in the journals.

Am I missing something, or am I simply imagining this? If there is no scripted way to subdue NPCs, what is the proper way to capture an NPC?

Sadly there's no way to subdue NPCs without a DM watching. If a DM is watching send them a quick DM message stating who you're wishing to subduel and why. If you just did not want to kill random commoner xx that hostiled you and got in the fight, consider their death to be a knock out, but if you want to question or kidnap someone in particular then a DM has tools to help you.