The Cisco Dwell Community Operations Heart (NOC) deployed Cisco Umbrella for Area Identify Service (DNS) queries and safety. The Safety Operations Heart (SOC) workforce built-in the DNS logs into Splunk Enterprise Safety and Cisco XDR.
To guard the Cisco Dwell attendees on the community, the default Safety profile was enabled, to dam queries to identified malware, command and management, phishing, DNS tunneling and cryptomining domains. There are events when an individual must go to a blocked area, such a reside demonstration or coaching session.
In the course of the Cisco Dwell San Diego 2025 convention, and different conferences we’ve got labored up to now, we’ve got noticed domains which might be two to 3 phrases in a random order like “alphabladeconnect(.)com” for instance. These domains are linked to a phishing marketing campaign and are typically not but recognized as malicious.
Ivan Berlinson, our lead integration engineer, created XDR automation workflows with Splunk to establish Prime Domains seen within the final six and 24 hours from the Umbrella DNS logs, as this can be utilized to alert to an an infection or marketing campaign. We seen that domains that adopted the three random names sample began to exhibiting up, like 23 queries to shotgunchancecruel(.)com in 24 hours.
This bought me considering, “May we catch these domains utilizing code and with our push to make use of AI, might we leverage AI to seek out them for us?”
The reply is, “Sure”, however with caveats and a few tuning. To make this attainable, I first wanted to determine the classes of knowledge I needed. Earlier than the domains get marked as malicious, they’re normally categorized as procuring, ads, commerceor uncategorized.
I began off working a small LLM on my Mac and chatting with it to find out if the performance I would like is there. I informed it the necessities of needing to be two-three random phrases, and to inform me if it thinks it’s a phishing area. I gave it just a few domains that we already knew have been malicious, and it was capable of inform that they have been phishing in accordance with my standards. That was all I wanted to begin coding.
I made a script to drag down the allowed domains from Umbrella, create a de-duped set of the domains after which ship it to the LLM to course of them with an preliminary immediate being what I informed it earlier. This didn’t work out too effectively for me, because it was a smaller mannequin. I overwhelmed it with the quantity of knowledge and rapidly broke it. It began returning solutions that didn’t make sense and completely different languages.
I rapidly modified the habits of how I despatched the domains over. I began off sending domains in chunks of 10 at a time, then bought as much as 50 at a time since that gave the impression to be the max earlier than I assumed it might develop into unreliable in its habits.
Throughout this course of I seen variations in its responses to the information. It is because I used to be giving it the preliminary immediate I created each time I despatched a brand new chunk of domains, and it might interpret that immediate in a different way every time. This led me to switch the mannequin’s modelfile. This file is used as the basis of how the mannequin will behave. It may be modified to vary how a mannequin will reply, analyze knowledge, and be constructed. I began modifying this file from being a common objective, useful assistant, to being a SOC assistant, with consideration to element and responding solely in JSON.
This was nice, as a result of now it was persistently responding to how I needed it to, however there have been many false positives. I used to be getting a few 15–20% false constructive (FP) price. This was not acceptable to me, as I wish to have excessive constancy alerts and fewer analysis when an alert is available in.
Right here is an instance of the FP price for 50 at this level and it was oftentimes a lot greater:
I began tuning the modelfile to inform the mannequin to present me a confidence rating as effectively. Now I used to be capable of see how assured it was in its dedication. I used to be getting a ton of 100% on domains for AWS, CDNs, and the like. Tuning the modelfile ought to repair that although. I up to date the modelfile to be extra particular in its evaluation. I added that there shouldn’t be any delimiters, like a dot or sprint between the phrases. And I gave it detrimental and constructive samples it might use as examples when analyzing the domains fed to it.
This labored wonders. We went from a 15–20% FP price to about 10%. 10% is a lot better than earlier than, however that’s nonetheless 100 domains out of 1000 that may have to verify. I attempted modifying the modelfile extra to see if I might get the FP price down, however with no success. I swapped to a more moderen mannequin and was capable of drop the FP price to 7%. This reveals that the mannequin you begin with is not going to all the time be the mannequin you find yourself with or will fit your wants probably the most.
At this level, I used to be pretty pleased with it however ideally want to get the FP price down even additional. However with the mannequin’s present capabilities, it was capable of efficiently establish phishing domains that weren’t marked as malicious, and we added them to our block listing. Later, they have been up to date in Umbrella to be malicious.
This was an excellent feat for me, however I wanted to go additional. I labored with Christian Clasen, our resident Umbrella/Safe Entry knowledgeable and was capable of get a slew of domains related to the phishing marketing campaign and I curated a coaching set to nice tune a mannequin.
This job proved to be more difficult than I assumed, and I used to be not capable of nice tune a mannequin earlier than the occasion ended. However that analysis continues to be ongoing in preparation for Black Hat USA 2025.
We’d love to listen to what you assume! Ask a query and keep linked with Cisco Safety on social media.
Cisco Safety Social Media
LinkedIn
Fb
Instagram
X
Share:
