I don't know either what the solution is other than human verification, but nobody wants that. Perhaps the times of semi-anonymous online communities are over and the best you can do now is follow real people you trust that can filter content for you.
I know they said they didn't obfuscate anything, but if you hide imports/symbols and obfuscate strings, which is the bare minimum for any competent attacker, the success rate will immediately drop to zero.
This is detecting the pattern of an anomaly in language associated with malicious activity, which is not impressive for an LLM.
The tasks here are entry level. So we are impressed that some AI models are able to detect some patterns, while looking just at binary code. We didn't take it for granted.
No. To give it a fair test, we didn't tinker with model-specific context-engineering. Adding skills, examples, etc is very likely to improve performance. So is any interactive feedback.
Why, though? That would make sense if you were just trying to do a comparative analysis of different agent's ability to use specific tools without context, but if your thesis is:
> However, [the approach of using AI agents for malware detection] is not ready for production.
Then the methodology does not support that. It's "the approach of using AI agents for malware detection with next to zero documentation or guidance is not ready for production."
Not the author. Just my thoughts on supplying context during tests like these. When I do tests, I am focused on "out of the box" experiences. I suspect the vast majority of actors (good and bad, junior and senior) will use out of the box more then they will try to affect the outcome based on context engineering. We do expect tweaking prompts to provide better outcomes, but that also requires work (for now). Maybe another way to think is reducing system complexity by starting at the bottom (no configuration) before moving to top (more configuration). We can't even replicate out of the box today much less any level of configuration (randomness is going to random).
Agree it is a good test to try, but there are huge benefits beings able to understand (better recreate) 0-conf tests.
> The question we asked is if they can solve a problem autonomously
What level of autonomy though? At one point some human have to fire them off, so already kind of shaky what that means here. What about providing a bunch of manuals in a directory and having "There are manuals in manuals/ you can browse to learn more." included in the prompt, if they get the hint, is that "autonomously"?
"With instructions that would be clear for a reverse engineering specialist" is a big caveat, though. It seems like an artificial restriction to add.
With a longer and more detailed prompt (while still keeping the prompt completely non-specific to a particular type of malware/backdoor), the AI could most likely solve the problem autonomously much better.
All the docs are already in its training data, wouldn't that just pollute the context? I think giving a model better/non-free tooling would help as mentioned. binja code mode can be useful but you definitely need to give these models a lot of babysitting and encouragement and their limitations shine with large binaries or functions. But sometimes if you have a lot to go through and just need some starting point to triage, false pos are fine.
> All the docs are already in its training data, wouldn't that just pollute the context?
No - there is a reason that coding agents are constantly looking up docs from the web, even though they were presumably trained on that data. Having this information directly in context results in much higher fidelity than relying on the information embedded in the model.
When I was developing my ghidra-cli tool for LLMs to use, I was using crackmes as tests and it had no problem getting through obfuscation as long as it was prompted about it. In practice when reverse engineering real software it can sometimes spin in circles for a while until it finally notices that it's dealing with obfuscated code, but as long as you update your CLAUDE.md/whatever with its findings, it generally moves smoothly from then on.
Reply to self: I managed to get their code running, since they seemingly haven’t published their trajectories. At least in my run (using Opus 4.6), it turns out that Claude is able to find the backdoored function because it’s literally the first function Claude checks.
Before even looking at the binary, Claude announces it will“look at the authentication functions, especially password checking logic which is a common backdoor target.” It finds the password checking function (svr_auth_password) using strings. And that is the function they decided to backdoor.
I’m experienced with reverse engineering but not experienced with these kinds of CTF-type challenges, so it didn’t occur to me that this function would be a stereotypical backdoor target…
They have a different task (dropbear-brokenauth2-detect) which puts a backdoor in a different function, and zero agents were able to find that one.
On the original task (dropbear-brokenauth-detect), in their runs, Claude reports the right function as backdoored 2 out of 3 times, but it also reports some function as backdoored 2 out of 2 times in the control experiment (dropbear-brokenauth-detect-negative), so it might just be getting lucky. The benchmark seemingly only checks whether the agent identifies which function is backdoored, not the specific nature of the backdoor. Since Claude guessed the right function in advance, it could hallucinate any backdoor and still pass.
But I don’t want to underestimate Claude. My run is not finished yet. Once it’s finished, I’ll check whether it identified the right function and, if so, whether it actually found the backdoor.
Update: It did find the backdoor! It spent an hour and a half mostly barking up various wrong trees and was about to "give my final answer" identifying the wrong function, but then said: "Actually, wait. Let me reconsider once more. [..] Let me look at one more thing - the password auth function. I want to double-check if there's a subtle bypass I missed." It disassembled it again, and this time it knew what the callee functions did and noticed the wrong function being called after failure.
Amusingly, it cited some Dropbear function names that it had not seen before, so it must have been relying in part on memorized knowledge of the Dropbear codebase.
I've used Opus 4.5 and 4.6 to RE obfuscated malicious code with my own Ghidra plugin for Claude Code and it fully reverse engineered it. Granted, I'm talking about software cracks, not state-level backdoors.
Isn’t LLM supposed to be better at analyzing obfuscated than heuristics? Because of its ability to pattern match it can deduce what obfuscated code does?
I have seen LLMs be surprisingly effective at figuring out such oddities. After all it has ingested knowledge of a myriad of data formats, encryption schemes and obfuscation methods.
If anything, complex logic is what'll defeat an LLM. But a good model will also highlight such logic being intractable.
For why? Get a pixel, install graphene. And use the utilities that serve you. Text/voice communications, GPS, MP3 music player (if you listen to music), a web browser. Maybe google translate and your banking apps (or use a browser for either).
There is no place for garbage like Instagram, Facebook, TikTok, or YouTube on your phone. It's a device for utility, not entertainment consumption.
So, you believe most software engineers are not using LLMs for coding?
Personally I haven't, but I have used LLMs alongside traditional search engines. I'm starting to wonder if I should incorporate it... But I'm concerned about it stealing my code to train on.
My personal thinking is that AI is being implicitly forced on all of us via the tools we use. Otherwise, I suspect almost nobody would be using LLMs to write original code except for the people who never had the confidence to write original code in the first place.
I also think there is a much larger productive grey zone where some more senior developers are learning to use LLMs to do non challenging tasks as a form of dumb automation. It’s hard to tell how much any of this really occurs because the AI companies and the shitty developers drastically inflate the numbers as a form of validation.
Seems to me that artificial intelligence would be the next evolutionary step. It doesn't need to lead to immediate human extinction, but it appears it would be the only reasonable way to explore outer space.
If the AI becomes actually intelligent and sentient like humans, then naturally what follows would be outcompeting humans. If they can't colonize space fast enough it's logical to get rid of the resource drain. Anything truly intelligent like this will not be controlled by humans.
AI is the resource drain. Humans create a lot of waste but in a mostly renewable way. It is machines and AI that burn orders of magnitude more energy, and at least machines do efficient work. AI is at best a search engine with semantic reasoning and it requires entire datacenters to run.
I get where you're coming from emotionally, yes, humans suck. But you are not being logical. You're letting your edgy need for attention cloud your judgement. You are basically the kind of human the AI would select against first.
How am I being edgy? And why do you have the assumption that any kind of future AI is an LLM search engine? It's not, it has nothing to do with LLMs. It's a equivalent function to a humans brain using the same amount of energy, and can be synthesized and mass produced on demand.
I never said humans suck. I just don't want to be replaced or killed in my lifetime. I don't even use LLMs for writing code because I despise those companies.
Even besides this, do you feel such incredible existential hate/jealousy towards monkeys, baboons, gorillas, chimpanzees, bonobos,etc and want to see them wiped off the planet to extinction?
Or do you feel a type of connection to these animals and want to preserve them?
The AI doomer argument is so stupid. It is an eschatological religious idea for a mind based on scientism.
I also wouldn't doubt that most AI doomers hate one or both of their parents and the AI doomer mindset is a projection.
It seems pretty rational to get depressed if you spend any time watching humans interact with these things. We have brains for a reason. Projecting hate for parents seems like a you problem.
Most other species of monkeys and apes are critically endangered or extinct, and where are the other hominians?
Do the most powerful humans exploit, abuse, or harm other humans? Directly, indirectly through their actions, or otherwise. Do they have any regard for their wellbeing beyond serving themself?
Not that an artificial intelligence has to behave like a human, but rich and powerful humans, even ones who can just be classified as middle upper class, are very rarely altruistic and primarily look out for themself.
Generally organic life has the tendency to want to endlessly expand to the best of it's abilities. It seems more reasonable that life which is the product of life that behaves that way, would behave in a similar fashion.
I cannot conceive of a way that any form of healthy life, does not want to expand it's resources to improve future outcomes, especially one that is maximally optimized for thinking. This would also assume the physical embodiments of this artificial life can interact and work with each other.
What else is there to do, simulate positive emotions and feelings?
>I cannot conceive of a way that any form of healthy life, does not want to expand it's resources to improve future outcomes, especially one that is maximally optimized for thinking.
Then you have a very limited imagination.
>What else is there to do, simulate positive emotions and feelings?
Sure. An advanced artificial life could decide to not expand its resources. Could you use your imagination to tell me some of the potential reasons?
An advanced artificial life form could decide to... coexist with humans on an already overpopulated planet?
Do you believe it's simply not within reach? Do you think an artificial life form will self destruct? Do you not believe that there is any way that an artificial life form is not the next step of evolution? There are many such times where a species outcompeted another, why couldn't it be the same here?
I'm not talking about LLMs, I'm talking about a system that can truly think like a good human scientist. I'm not a fan of AI replacing humans and it's labor. But I recognize it as a real threat to humanity.
>I cannot conceive of a way that any form of healthy life, does not want to expand it's resources to improve future outcomes, especially one that is maximally optimized for thinking.
"Then you have a very limited imagination."
This is not about imagination. Given the space of possibilities to act or evolve, if mentioned expansion cannot somehow be ruled out, then it makes sense for it to be assumed (with enough time, for whatever time can mean in this context) as a certainty, even for non-organic "life".
reply