Prompt Power: Investigation Summarization

Imagine you could tell a powerful AI system what you want it to do using your own words. Sounds amazing, right? Well, that’s exactly what generative AI can do. It can understand natural language and follow your instructions, as long as you know how to phrase them. In this post, I will show you how to use the magic of prompting to create concise and accurate summaries for a security investigation.

Background

• I am using Security Copilot to perform my work and executed a promptbook to analyze a suspicious powershell script. This script silently downloads a file from a remote IP address and then executes it on the host system.
• Promptbooks solicit inputs from the user and then run a series of curated prompts that build off previous context to complete a workflow. They take a few minutes to run, but are incredibly powerful. If this concept is new here to learn more, https://learn.microsoft.com/en-us/security-copilot/using-promptbooks#what-are-promptbooks.
• Interactions such as individual prompts or promptbooks within Security Copilot create sessions. Sessions are saved in your workspace and can be shared with others. Our summary is derived from our session.
• Our script analysis promptbook explains what the script is doing, notes any malicious behavior, extracts IOCs, and correlates with Microsoft threat intelligence where it will pull in details like indicator reputation, known threat associations and threat articles.

Creating the Summary

Within my session, I have 5 prompts and responses that were associated with the promptbook. I want to generate a summary of the investigation so that I can preserve this in Security Copilot and share the results with my colleagues. While the contents of the prompts and session are important, they are not needed to demonstrate how different prompts can augment the response. Below, I am going to break down the differing ways I can achieve getting a summary and how crafting a detailed prompt is able to yield far superior results.

Summarize and nothing more

In the above example, I use a simple prompt to get a summary. The results are a bulleted list but the format isn't exactly what I was expecting. To me, a summary would be a paragraph, not necessarily bullet points. While this captures the spirit of the session, it is not leveraging Security Copilot to its fullest potential.

Summarize and guide me

In this prompt, we retain our simple summary request, but we have now extended the instructions to Security Copilot to form an executive summary and build a list of recommendations. Our output is a small paragraph capturing the session details and the list below becomes actionable next steps. These small changes make a big difference, especially if the user working the investigation has less experience. Also notable, the recommendations go beyond our session context and leverage the security knowledge of the foundation models to give more insight into what to do next. While this response is more actionable, we can do better.

Summarize, guide me and share your view

Again, we retain our simple request to summarize the session, but in this example, we ask Security Copilot to offer its view on the incident and to include evidence for the choice alongside our summary and recommendations. The phrasing of the request is designed to force a decision of true or false while confidence and reasons are included to ensure we understand why the model made the specific choice. It's worth noting that while we have a paragraph summary similar to our previous example, the contents are worded differently. GAI will have non-deterministic outputs depending on the model temperature and this must be accounted for when prompting. If there's specific details you want retained, specify those in your prompt.

Responsible AI Note: This sort of prompt highlights the power of GAI within security, though it also exposes risks. Even when a session is grounded with information, fabrications are still possible. While the above prompt is able to produce an accurate answer for the incident classification, analysts should still be involved in the triage process and verify the responses from any GAI solution.

Coming back to the actual response details, Security Copilot rates this as being a likely true positive with high confidence and proceeds to list the reasons for making this assessment. Dusting off my analyst hat, I find myself agreeing with Security Copilot.

Summarize for management

Any confirmed and active incident is going to require management notification. Business leaders outside of security may not possess the detailed knowledge of those doing the incident response. In this example, Security Copilot is instructed to call out the most noteworthy facts and write for leadership who only need to worry if there's a threat. The response is short and to the point, suitable for an initial check-in to management while the response efforts take place. The purpose of this example is to demonstrate the ability to adjust the same summary for different audiences without having to make any other material adjustments. I could have easily selected examples including writing for legal, a more technical audience or someone outside my organization who may not be privy to the sensitive details in the session.

Conclusion

In this post, I demonstrated the power of prompting, specifically when it came to summarizing an incident. Each example builds on the previous to show how a more specific prompt can lead to a much more refined response. Not only that, but how the underlying GAI models can be used to assist the analyst in their workflow to include the classification of the incident or a summary for their management. While these examples were interesting, the possibilities for prompt variations are endless.

If you had the ability to summarize your investigations in anyway possible, what kind of outputs would you want to see? If you have prompt ideas, I am happy to run them and share results.