Wardle found a way around Gatekeeper earlier this year, and Apple patched the issue. But after studying it for a few days, he bypassed Gatekeeper again.
"Instead of them implementing a more generic patch or generic fix, they kind of did just what was the bare minimum," Wardle said. "That's a little worrisome."
His latest findings were presented two weeks ago at the Virus Bulletin security conference in Prague, and Apple has been notified.
Patrick Wardle of Synack has found two issues in Apple's Gatekeeper, which is designed to stop certain applications from installing. Credit: Patrick Wardle/Synack
Wardle talks to Apple's security team regularly about bugs he finds in OS X. They are "sharp guys," he says, but they may be fighting a company culture where usability in many cases trumps security.
Locking down OS X's core
Apple introduced a new defense in OS X El Capitan called System Integrity Protection (SIP), which is makes it a lot harder for malware writers to touch critical OS files.
SIP blocks the most powerful kind of access to the operating system, known as "root." That access is usually only prevented by a single password set by the Mac's user, who has administrative privileges.
If that password is compromised, an attacker with root access can disable other security protections, posing a great risk.
SIP greatly reduces the opportunity for malware writers to put something deeply rooted in OS X, said Rich Trouton, a Mac systems administrator in Middletown, Maryland, who writes the Der Flounder blog.
"People look at it and go 'Apple's not finding a bunch of malware,'" Trouton said. "And that's true at the moment, but the reason for that is that Apple's put a lot work into making OS X a less appealing target."
However, "who knows what the future may bring," he said.
More eyes on the code
Another interesting fact about OS X this year: about four times more software vulnerabilities have been disclosed than in past years.
A list shows 276 flaws have been found this year, which is about four times higher than the average number found annually over the last 15 years, said Claud Xiao, a security researcher with Palo Alto Networks.
"It's a huge increase," Xiao said. "This year, more and more researchers are focused on how to bypass security mechanisms or how to get code to execute remotely."
Malware writers also seemed more tuned into using vulnerabilities to infect OS X. Xiao said that in three cases this year, malware or adware used disclosed vulnerabilities to get onto systems.
Mac malware and adware programs have typically relied on tricking users into installing them rather than exploiting vulnerabilities. Apple's built-in AV product, called XProtect, is regularly updated to block nuisances like adware.
Sign up for CIO Asia eNewsletters.