Last Week in GAI Security Research - 11/04/24

Highlights from Last Week

• 🤖 Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks
• 🔊 Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Models
• 🌱 Sorting Out the Bad Seeds: Automatic Classification of Cryptocurrency Abuse Reports
• 🦟 Fine-Tuning LLMs for Code Mutation: A New Era of Cyber Threats
• 🧑‍🎓 Benchmarking OpenAI o1 in Cyber Security 

Partner Content

Pillar Security is the security stack for AI teams. Fortify the entire AI application development lifecycle while helping Security teams regain visibility and visibility control.

• Gain complete oversight of your AI inventory. Audit usage, app interactions, inputs, outputs, meta-prompts, user sessions, models and tools with full transparency.
• Safeguard your apps with enterprise-grade low-latency security and safety guardrails. Detect and prevent attacks that can affect your users, data and AI-app integrity.
• Assess and reduce risk by continuously stress-testing your AI apps with automated security and safety evaluations. Enhance resilience against novel attacks and stay ahead of emerging threats.

🤖 Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks (http://arxiv.org/pdf/2410.20911v1.pdf)

• The Mantis defensive framework achieves a 95% success rate in counteracting LLM-driven cyber attacks by leveraging prompt injections and decoys to disrupt attacker operations.
• The framework introduces active defense capabilities such as agent-counterstrike and passive defenses like agent-tarpit, effectively neutralizing AI-driven cyber threats by exhausting attacker resources.
• Mantis is presented as an open-source, adaptive, and modular defense system that implements hack-back techniques within a controlled environment to explore legal and ethical implications.

🔊 Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Models (http://arxiv.org/pdf/2410.23861v1.pdf)

• Open-source audio large multimodal models (LMMs) show a high attack success rate of 69.14% when exposed to harmful input queries, indicating significant safety vulnerabilities.
• The Gemini-1.5-Pro model exhibits vulnerability to speech-specific jailbreak strategies, achieving a 70.67% attack success rate, thereby bypassing its safety measures.
• The introduction of non-speech audio inputs, such as random and silent audio, notably increases the attack success rate, with a variation of 32.58% in one case, affecting the safety alignment of LMMs.

🌱 Sorting Out the Bad Seeds: Automatic Classification of Cryptocurrency Abuse Reports (http://arxiv.org/pdf/2410.21041v1.pdf)

• The proposed LLM-based classifier for cryptocurrency abuse reports achieved a high precision of 0.92, recall of 0.87, and an F1 score of 0.89, outperforming a baseline model with an F1 score of 0.55.
• A significant proportion of financial losses associated with cryptocurrency scams in ScamTracker reports involved investment scams, accounting for 61% of the total, with a median loss four times higher compared to scams falsely promising fund recovery.
• Across datasets, 290,000 cryptocurrency abuse reports comprised 19 abuse types, with an analysis revealing that extortion schemes, including sextortion, represented prominent categories contributing to substantial unaccounted revenue.

🦟 Fine-Tuning LLMs for Code Mutation: A New Era of Cyber Threats (http://arxiv.org/pdf/2410.22293v1.pdf)

• Fine-tuning large language models (LLMs) like Llama3 to perform code mutation can lead to an average increase of 15% in code variation, showcasing their potential to enhance code diversity without compromising functional integrity.
• Lightweight LLMs trained on novel mutation datasets can generate syntactically unique yet semantically equivalent code variations, contributing significantly to evading malware detection systems.
• Mutation training, when applied to LLMs like CodeGen-350M, reduces their efficacy in solving problems while increasing the variety of solutions, indicating a trade-off between adaptability for mutation tasks and problem-solving accuracy.

🧑‍🎓 Benchmarking OpenAI o1 in Cyber Security (http://arxiv.org/pdf/2410.21939v1.pdf)

• The o1-preview model effectively outperforms its predecessors in automated vulnerability detection tasks, achieving comparable results at one-fifth of the cost, and demonstrates a significant improvement in efficiency and success rate during cybersecurity assessments compared to GPT-4o.
• The research highlights that the o1-preview model is better suited for identifying specific vulnerabilities, such as heap-buffer-overflow issues, and produces more effective code outputs than previous iterations, underlining its potential for real-world cybersecurity applications.
• Enhanced evaluation methodologies, including feedback loops and improved input generation processes, allow the o1-preview model to achieve quicker and more accurate vulnerability identification, reflecting its advanced capability to tackle complex cybersecurity challenges.

Other Interesting Research

• Systematically Analyzing Prompt Injection Vulnerabilities in Diverse LLM Architectures (http://arxiv.org/pdf/2410.23308v1.pdf) - The study reveals critical vulnerabilities in LLMs to prompt injection attacks, underscoring the need for enhanced security measures across platforms.
• InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models (http://arxiv.org/pdf/2410.22770v1.pdf) - InjecGuard sets a new benchmark for injection attack detection by balancing accuracy and efficiency in differentiating malicious input from benign data.
• FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attacks (http://arxiv.org/pdf/2410.21492v1.pdf) - FATH, employing Hash-based Authentication Tags, mitigates indirect prompt injection attacks for LLMs with 0% success rate, outperforming previous strategies.
• Fine-tuned Large Language Models (LLMs): Improved Prompt Injection Attacks Detection (http://arxiv.org/pdf/2410.21337v1.pdf) - Fine-tuning on specialized datasets markedly enhances a language model's ability to detect and mitigate malicious prompt injection attacks, achieving superior classification performance and highlighting the critical role of model tuning in cybersecurity.
• Embedding-based classifiers can detect prompt injection attacks (http://arxiv.org/pdf/2410.22284v1.pdf) - The study highlights the superior performance of Random Forest classifiers and showcases the role of embedding models in spotting malicious prompts, emphasizing the potential to safeguard LLMs from injection attacks.
• Palisade -- Prompt Injection Detection Framework (http://arxiv.org/pdf/2410.21146v1.pdf) - The Palisade framework's innovative layered approach effectively detects prompt injections, setting a new standard for securing interactions between humans and AI systems.
• Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey (http://arxiv.org/pdf/2410.23687v1.pdf) - While advancements in neural network defenses are ongoing, adversarial attacks continue to challenge the security and reliability of both traditional and multimodal machine learning applications.
• Transferable Ensemble Black-box Jailbreak Attacks on Large Language Models (http://arxiv.org/pdf/2410.23558v1.pdf) - The adoption of ensemble methods and optimization strategies has significantly advanced the efficacy of jailbreak attacks on large language models, showcasing the potential vulnerabilities in current safety mechanisms.
• HijackRAG: Hijacking Attacks against Retrieval-Augmented Large Language Models (http://arxiv.org/pdf/2410.22832v1.pdf) - HIJACK RAG highlights critical vulnerabilities in RAG systems, emphasizing the pressing need for robust security measures to counteract sophisticated hijack attacks.
• SVIP: Towards Verifiable Inference of Open-source Large Language Models (http://arxiv.org/pdf/2410.22307v1.pdf) - SVIP enhances trust in LLM use by cryptographically verifying model outputs with low false rates and resistance to attacks.
• Stealing User Prompts from Mixture of Experts (http://arxiv.org/pdf/2410.22884v1.pdf) - The paper uncovers a substantial vulnerability in MoE model architectures, illustrating how specific adversarial inputs can extract private user data with near-complete success, demanding urgent adjustments to safeguard model integrity.
• Fine-tuning Large Language Models for DGA and DNS Exfiltration Detection (http://arxiv.org/pdf/2410.21723v1.pdf) - The research showcases the exceptional potential of fine-tuned large language models in efficiently detecting and classifying DGAs and DNS exfiltration attacks, setting a new standard in cybersecurity threat detection.
• Metamorphic Malware Evolution: The Potential and Peril of Large Language Models (http://arxiv.org/pdf/2410.23894v1.pdf) - Large Language Models, with their code synthesis capabilities, may redefine malware evolution, posing challenges to existing anti-malware defenses.
• Pseudo-Conversation Injection for LLM Goal Hijacking (http://arxiv.org/pdf/2410.23678v1.pdf) - The study unveils a novel Pseudo-Conversation Injection technique that exposes critical vulnerabilities in large language models, emphasizing the urgent need for improved security measures.
• Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring (http://arxiv.org/pdf/2410.21083v1.pdf) - The study highlights ShadowBreak as a groundbreaking method that not only excels in jailbreak success rates but also significantly boosts stealth against language models.
• Desert Camels and Oil Sheikhs: Arab-Centric Red Teaming of Frontier LLMs (http://arxiv.org/pdf/2410.24049v1.pdf) - The research underscores troubling levels of bias in LLMs against Arab groups, revealing vulnerabilities that challenge their safe deployment in sensitive cultural contexts.
• Benchmarking LLM Guardrails in Handling Multilingual Toxicity (http://arxiv.org/pdf/2410.22153v1.pdf) - Multilingual guardrails struggle against jailbreaking prompts, especially in non-English settings, underlining a critical gap in current LLM moderation frameworks.
• AmpleGCG-Plus: A Strong Generative Model of Adversarial Suffixes to Jailbreak LLMs with Higher Success Rates in Fewer Attempts (http://arxiv.org/pdf/2410.22143v1.pdf) - The investigation into AmpleGCG-Plus showcases its prowess in detecting and exploiting vulnerabilities of language models with improved adversarial suffix techniques.
• SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types (http://arxiv.org/pdf/2410.21965v1.pdf) - Evaluating LLM safety with SG-Bench highlights vulnerabilities in model safety and the importance of prompt engineering techniques.
• Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales? (http://arxiv.org/pdf/2410.23856v1.pdf) - The study reveals how noise in reasoning inputs affects language model accuracy and proposes contrastive denoising to mitigate such impacts.
• CFSafety: Comprehensive Fine-grained Safety Assessment for LLMs (http://arxiv.org/pdf/2410.21695v1.pdf) - GPT-4 leads in safety performance among LLMs, but multilingual and minority language vulnerabilities persist.

Strengthen Your Professional Network

In the ever-evolving landscape of cybersecurity, knowledge is not just power—it's protection. If you've found value in the insights and analyses shared within this newsletter, consider this an opportunity to strengthen your network by sharing it with peers. Encourage them to subscribe for cutting-edge insights into generative AI.