Scaffolding: Rapid Prototyping within ChatGPT

When considering how to operationalize GAI within security, I often think about completing a specific workflow or task. I will commonly use ChatGPT as a way to validate my ideas and identify locations in the workflow where more effort may be required. I call this process "scaffolding" and while I've indirectly referenced in my previous posts, I've never provided a comprehensive example of going from idea to implementation.

In this post, I will demonstrate the use of scaffolding with a common security example that usually involves code/development. I am going to transform a threat intelligence blog entry into structured JSON output, including an example that highlights the benefits of large language models. While not a coded implementation, this process could be adopted by any smaller business looking for a starting point in building out a foundation of threat intelligence for their organization. The goal with this scaffolding session is to quickly get from an idea to a working concept, all without writing any code.

Scaffolding Demonstration

To complement the text below, I've put together a 20 minute video showing myself working through the process of using ChatGPT to achieve a rough implementation of the blog idea. This video isn't scripted and captures my stream of consciousness as I would normally work through building a workflow like this. Here's a link to my ChatGPT session.

The Idea

As noted in the introduction, the goal with this scaffolding session is to take a blog and transform it into a JSON output that captures the core details of the blog while also introducing components that would bring value to security practitioners. For the example blogs, I am leveraging the Microsoft Security Blog, specifically a few entries published under the threat intelligence category. The example created in the session is meant to be more of a proof-of-concept, less a fully formed implementation. Enough details will be shared that one could easily extend this concept and migrate it into code, a series of prompts for OpenAI APIs or construct a GPT.

ChatGPT Session: https://chat.openai.com/share/70b6734a-08d6-4e9d-aca8-655b81bdd224

Building our Scaffolding Implementation

The process of scaffolding is quite simple. Similar to putting together a puzzle, I will first get a sense of all the pieces I need and ensure they are organized. This initial phase may require refinement or separate "chat" sessions to work through any issues. Once I have my components, I begin stitching them together and ensure and identifying any weak points that need further refinement. Finally, with a working process in hand, I test on several examples and then consider a final implementation.

Spot-Checking the Model Capabilities

I will begin by testing the model to ensure it can perform the basic tasks I know I will want within my final results. I like to test these independent of the broader outcome I want to achieve, so I am clear on any issues I may face. I will often use this step to form the building blocks that I can then leverage to construct my final output.

With any scaffolding session, my first steps are to test to see how the model would perform with limited direction. I will ask for simple tasks to be completed and often interrupt the response output once I am satisfied that the core of what I want can be completed. For our example, I seed the session with content copied from the Microsoft blog and put that directly into the session. From there, I test to make sure that ChatGPT can summarize that data and get a glimpse of how it's going to organize the content. The summary aspects are good, but the formatting isn't what I want. This is expected given I've not provided any format instructions.

Knowing I can get a summary on the current data, I move on to a more specific example that highlights the value of LLMs: identify MITRE ATT&CK framework techniques and include an example from the source text to pair with each. My primary goal with this test is to identify that ChatGPT knows what the MITRE ATT&CK framework is and to see how it decides to construct the outputs. Again, without much formatting direction, I get back results that appear correct, though do not match what I want.

As a final capability test, I ask the model to take the outputs we have and form them into a JSON output. Notice, I give direction on the persona to adopt and the task to complete, but I do not provide any organization to the JSON. The response demonstrates JSON outputs can be achieved. As a bonus, I can draw inspiration from the suggested JSON structure. When I have a less formed view of the final output, I love to use vague prompts to help generate ideas and suggestions.

Assembling the Components

With our quick tests, I've managed to demonstrate the model can effectively summarize our content, blend industry concepts with the material and return responses in JSON. Enough of the core components are defined that I can begin to stitch this together into a singular prompt.

In the prompt above, I combine all our components to get a single output that matches close to what I want and should remain consistent. Much of the structure is a repeat from our testing, except for the specific guidance for how the JSON should be formed. To ensure the model provides consistent outputs, I specify the keys I want and the descriptions for each one. It's worth noting, these descriptions could be heavily refined to create more robust outputs, but for the sake of demonstration, this meets what I want.

Polishing the Implementation

As a final addition to the prompt, I add a single line of "Only respond with the JSON codeblock. Do not put any text before or after explaining what you did.". This instruction avoids the common pre and post text that often follows any ChatGPT response. Having it respond only with JSON allows this prompt to easily flow into an API implementation I could call and process via code.

With my final prompt formed, I then run several test examples from a variety of sources and check the outputs. My primary goal in the tests is to ensure I have consistent results from the model and that it can work across a diverse set of inputs.

0 to 100 in 20 minutes

What I love about scaffolding is the speed for which I can prove out complex ideas. Prior to solutions like ChatGPT, I'd need to write an extensive amount of code to achieve the outcomes I wanted. Many off-the-shelf libraries exists to parse webpages, extract blog content and organize it, but for deriving tags or fusing MITRE ATT&CK technique details with the collected text would take significantly more effort. In this example, I was able to achieve the results I wanted in less than 20 minutes and without writing a single line of code.

Once I complete an implementation, I then need to consider how to operationalize it. As of November, OpenAI has created the concept of GPTs which allow you to perform a similar approach to scaffolding and encapsulate your implementation within a GPT––better thought of as a bot. In practice, I've found limitations and issues with GPTs, but for someone without development experience, this is a great next step and one I will document in future posts.

For developers, there's two pathways to take. One method would be to take the refined prompt you created and execute this within code using OpenAI APIs. Depending on your idea, this may work without any issue, though unlike traditional code, you will need to account for added latency and processing associated with using GAI in your application. This example would work well with small volumes of articles and could easily be integrated into an application. For more complex workflows, your next step may be to have the model suggest the code to reproduce the outputs you worked to create. While not perfect, this will give you a strong foundation to operate from.

Conclusions

In this post, I was able to use scaffolding to go from idea to rough implementation in less than 20 minutes and without writing a single line of code. I demonstrated this process on a laptop, but I more commonly do this on my phone using the ChatGPT mobile application. What I love about this approach is the simplicity of it and that anyone can apply this technique, regardless of their experience level. Next time you're waiting in line and have an idea, consider scaffolding it out in ChatGPT.