A lot of exploit styles (e.g. "return-oriented programming") rely on jumping or calling into code that was never meant to be executed "stand-alone" --- i.e., jumping to an instruction that was only ever supposed to be executed by falling through from the previous instruction or via a branch within its function. On x86 this includes destination "instructions" that are actually part of another multi-byte instruction. It seems to me these exploits could be made much harder by somehow marking the start of "valid" branch/call/return targets and faulting when control is transferred to other instructions inappropriately.
Here's a more specific proposal for x86:
- Add a new flag bit for code pages to enable "destination checking" on a per-page basis.
- An indirect control transfer is any return instruction that pops EIP off the stack, or any jump or call instruction whose operand is not a constant. When an indirect control transfer jumps to a page marked for destination checking, and the instruction at EIP is not 0x90 (NOP), fault.
- A toolchain would take advantage of this feature by marking code pages for destination checking, avoiding use of 0x90 for regular NOPs, and placing a 0x90 at every function entry point and after every call instruction. For bonus points, avoid generating 0x90 bytes inside other instructions.
Obviously there are a lot of ways to tweak this. For example "PUSH EBP" is very often the first instruction of a function, so you could whitelist that instruction as a valid function entry point. You could avoid having to place a NOP after a direct CALL instruction by checking if the instruction at EIP-5 is 0x9A (direct CALL). You could make false returns even harder by extending that special case to require the address we're returning from to be "close to" the destination of the direct CALL instruction.
This is obvious enough that I presume someone's worked on it already.