Flip the Script: Learn from the Machines

Back in August of last year, I came across an article outlining an advanced prompting pattern called "flip the script". Instead of asking the language model questions, you would prompt the model to ask you questions. What I love about this pattern is that it lends itself well to learning a new topic or helping complete a task for which you may need support from someone with more experience. I've shared my views on how generative AI can help transform security and within that post, I indirectly reference the flip the script approach, specifically applied to assessing practitioners experience and continually qualifying them for their role.

In this post, I want to demonstrate applying this advanced pattern within ChatGPT and how anyone could use this approach to learn more about cybersecurity, get help on how to respond to an incident or even generate prompt ideas for a GAI security tool like Security Copilot. While I am using ChatGPT to frame these examples, it's important to note that each idea could stand alone within a product offering.

Evaluate My Knowledge

In this example, I am giving ChatGPT guidance for how to interact with me within this session. I ask for the model to assume the role of an instructor and provide some basic framing around what I feel would make a good teacher. There's a variety of "super/system prompts" cropping up across the Internet that may offer better outcomes beyond my initial version. I find these can be refined much like any other prompt and may take time to really hone in.

For the second portion of my prompt, I've requested the model to evaluate my current level using a series of 5 questions that well-represent the job position of a security operations analysts. Based on the answers to those questions, I have the model decide whether or not I am a junior, senior or an expert and use that to guide the future questions beyond. This small bit of guidance plays a larger role in customizing the learning more specifically to my knowledge and skill set.

Finally, I ask the model to score each of my responses and offer ways to make my answer better. Prior to implementing this in my sessions, I found myself feeling like my answers were good, but with no sense of where I could improve. Assessing each answer and providing feedback as I answer was effective in me crafting better answers in subsequent questions.

Practical Use

This application of "flip the script" was one of the most useful and fun for me. I found myself spending about 45 minutes answering questions. For many, I tried to provide quality answers, but I also purposefully gave incomplete or inaccurate answers. I ended up getting categorized as a "senior" analyst and found that future questions honed in on areas where I had previously low scoring responses. I was also able to interrupt the model and ask for focus areas, examples and other details to get a more diverse set of interactions.

What I would like to see in a system

• Blend of traditional learning with GAI features (explanations, examples, interactions, evaluations)
• Ability to "test in" to the system and evaluate my baseline
• Time-based learning sessions that meet my schedule and focus areas
• Tuned content based on my day job and most likely to encounter experiences
• Micro-evaluations that constantly check my growth/loss
• Ability to "certify" myself in specific learning paths for a period of time

Provide Operational Guidance

In this example, I take a more casual prompt-setting approach to have the model assist me with triaging an incident using questions. If I were to supply the incident details without the question guidance, the model would give me a list of steps and things to consider. Obviously, the question approach adheres to flipping the script, but I also found in practice that it feels better to interact with the model in this way for an operational task. While I know there's not a human on the other end, I feel like I have a collaborator working alongside me.

The above interaction captures the 5 questions the model asked me. I believe the number of questions was dictated based on coming up with a "verdict" or natural conclusion to the triage. I felt the model does a good job of framing the questions and using those previous responses to inform a proper next step or secondary level of detail.

Practical Use

While I was able to easily mock a fake incident and series of answers, I think this approach would struggle to be operationalized within a system like ChatGPT. The foundational models lack the necessary context to provide product-specific knowledge and understanding of my environment. This gap can be filled using plugins, but those face limitations that could be challenging to work through. I could see this approach being a good generic steering mechanism for newer analysts who may not know exactly what to do in an incident or other security task, though it would hit limitations quickly.

What I would like to see in a system

• Security specific implementation that helps users do security tasks
• Plugin library of popular solutions to pull in relevant context
• Suggestive next-steps to take when doing particular jobs
• Evaluation of my work based on signals collected
• Ability to collaborate with my team or others in my organization
• Tie-in to learning "sandbox" system to improve specific skills while performing my normal operating duties

Build My Prompts

As a final example, I go meta and ask the model to help me form a detailed prompt for a generative AI system. Specifically, I'd like a prompt that helps to craft an incident report assuming all the incident details were located in a single location. ChatGPT asked me 5 questions about the audience, format, and specifics of certain sections to come up with a prompt to meet my needs. Similar to the other examples, if I had not leveraged the flip the script approach, I would have gotten an answer, though I would have needed to get my initial prompt detailed enough at the start, or refine it numerous times to get an output like this. The flip the script approach let me iteratively get to where I needed without a lot of deep understanding of the GAI system.

Practical Use

GAI is full of meta examples and this specific "create my prompts" is one I use often, especially when generating images. In fact, the image on all my posts are achieved by feeding my blog content into ChatGPT and asking the model to create a detailed prompt for Midjourney. While I don't always use the exact outputs the model suggests, I find the starting points useful, especially when I am not fully decided on the format or style I want within an output. For those beginning in GAI, this is a superpower that must be applied for the prompts you use to engage the system often dictate the quality of the outputs. Weak prompts in lead to weak answers out.

What I would like to see in a system

• Ask me for my role and the task I need to complete
• Interview me on the subject of the task to collect context
• If context isn't clear, ask for examples or support
• Collect the audience for the outputs along with any formatting

Conclusion

Flip the Script is a powerful pattern that can be used within any subject matter to achieve better results using GAI. The approach removes many of the complexities of interacting with the model and mimics more of a collaborative relationship you'd find in a colleague or assistant. The three examples I selected for this blog are ones I personally apply to help increase my knowledge and improve my daily workflows. In a future post, I will select one of these examples to form a GPT and show a more repeatable experience could be created. Because I like to have direct control and less abstractions, I continue to employ my scaffolding technique when applying patterns like this.