This is, more or less, how I taught myself x86 assembly. By reverse-engineering Skype.
It wasn’t quite that difficult back in the 90s. And these guys got way further than I ever did. But a lot of the binary protection (polymorphic checksumming, dynamic calling, a really clever packer, RSA verification, etc) is close or identical to what I was playing with.
Play with that stuff long enough and, well, you’ll get really good at asm. A lot of really clever ideas of mine came out of things I saw in their binary.
I tracked down one of the exe packers I helped write way back when, and turns out that malware protected with it has been detected by McAfee over half a million times now. So that’s kinda cool… even if it’s a little scary.
But sort of the holy grail of reversing right now is Kaspersky. Those guys are hardcore. Their engine can pick apart a lot of packers these days, even stuff it’s never seen before. I’ve heard stories of people who have written decent custom packers that get detected automatically. They’ve got a static analysis tool that can follow not just off-by-one byte alignment tricks or decompress LZW blocks automatically, but it can simulate execution in a sandboxed environment and use the binary’s own decryption routines to decrypt it, even on binaries that only decrypt a few pieces at a time. It’s incredibly hardcore.