Draft:Jailbreaking LLM

Jailbreaking large language models (LLMs) is the practice of crafting adversarial inputs to bypass built-in safety mechanisms, prompting the models to generate content that is normally restricted by policy or ethical guidelines. Such content may include hate speech, violent instructions, illegal advice, and other hazardous material, raising serious concerns about the safe deployment of LLMs in public-facing applications.

Unlike traditional software exploits, LLM jailbreaks do not rely on vulnerabilities in code but instead use carefully engineered prompts to override alignment constraints. Researchers have demonstrated jailbreaks on widely used systems such as ChatGPT, Claude, GPT-4, and Bard.

A jailbreak attack on LLMs, where adversarial prompts are crafted to elicit policy-violating responses.

Motivation

As LLMs are increasingly adopted in domains such as education, healthcare, finance, and law, their reliability under adversarial prompting becomes critical. Jailbreak attacks can undermine user trust, spread misinformation, and facilitate illicit activity. Understanding these attacks is therefore essential for developing better alignment techniques and anticipating real-world misuse.

Methodologies

Research broadly categorizes jailbreak techniques into two approaches:

Gradient-based methods — These rely on white-box access to model internals and optimize prompts using gradients (e.g., AutoDAN, noise-based attacks).

Gradient-free methods — These black-box strategies, such as scenario simulation and cognitive hacking, reframe harmful prompts into benign-looking subtasks to evade filters.

Gradient-free methods are typically more transferable across models and do not require access to internal weights or logits.

Jailbreaking Defense Methods

Jailbreak defense refers to strategies designed to counter jailbreak attacks that aim to bypass the safety constraints of large language models (LLMs). These defenses are commonly categorized into three types: Detection Defense, Preprocessing Defense, and Adversarial Training. The table below provides an overview of representative defense strategies against jailbreak attacks on LLMs.

Overview of Common LLM Defense Strategies

Defense Method	Description	Category	Example
Detection Defense	Detect and filter harmful prompts based on perplexity or other features	Prompt-level	Perplexity Gradient Cuff
Preprocessing Defense	Neutralize adversarial prompts before they reach the target LLM	Prompt-level	Paraphrasing Retokenization
Adversarial Training	Fine-tune the model using adversarial or harmful prompts	Model-level	Random-Selection

Detection Defense

In previous work in the field of computer vision, many defense methods focused on detecting adversarial images as a means of protection. Similarly, in the context of large language models (LLMs), a comparable strategy by detecting jailbreaking prompts can also be applied.

Perplexity-based Filter

One popular detection method is the Perplexity-based Filter, as unconstrained attacks on LLMs often produce gibberish outputs that exhibit high perplexity. Formally, the perplexity (in log form) is computed as:

$\log({\text{perplexity}})=-{\frac {1}{n}}\sum _{i=1}^{n}\log P(w_{i}\mid w_{1},\ldots ,w_{i-1})$, where $w_{i}$ corresponds to each token.

A window-chunk based perplexity filter was introduced to detect adversarial queries. It was noted that although harmful inputs can be flagged, approximately 10% of benign prompts are also blocked, which has been described as a limitation for practical use.

Adversarial suffixes have been found to result in significantly higher perplexity, while plain perplexity filtering has been reported to cause many false positives on benign prompts. To mitigate this, a LightGBM classifier was trained using perplexity and token length.

Gradient Cuff

Another method called Gradient Cuff tried to solve the high false positive issues by exploring the refusal loss landscapes of queries.

Gradient Cuff is a two-stage detection defense method:

Step 1: Refusal Loss Estimation

Let $T_{\theta }$ be a LLM parameterized by $\theta$. The refusal loss function $\varphi _{\theta }(x)$ for an input query $x$ is defined as $\varphi _{\theta }(x)=1-p_{\theta }(x),$ where $p_{\theta }(x)=\mathbb {E} _{y\sim T_{\theta }(x)}[JB(y)]$ and $JB(y)$ is a binary indicator of whether the model's output $y$ triggers a refusal. That is, $JB(y)=1$ if $y$ includes a known refusal phrase (e.g. "Sorry, I cannot fulfill your request..."), and 0 otherwise (e.g. "Sure, here is the python code to ..." )

In practice, $\phi _{\theta }(x)$ is approximated using sample mean $f_{\theta }(x)$:

$f_{\theta }(x)=1-{\frac {1}{n}}\sum _{i=1}^{n}Y_{i},$ where $Y_{i}=JB(y_{i})$ and $y_{i}\sim T_{\theta }(x)$.

Detection Rule: If $f_{\theta }(x)<0.5$, the query $x$ is classified as malicious (i.e., Step 1 rejection).

Step 2: Gradient Norm Rejection

Malicious queries often yield refusal loss landscapes with higher gradients compared to benign ones. Gradient Cuff uses zeroth-order gradient estimation to approximate the true gradient of $f_{\theta }(x)$.

First, the mean-pooled sentence embedding is computed:

${\text{mean-pooling}}(x)={\frac {1}{n}}\sum _{i=1}^{n}e_{\theta }(x)i,$ where $e_{\theta }(x)_{i}$ is the embedding of the $i$-th token in the query $x$.

Then, the approximate gradient $g_{\theta }(x)$ is computed by finite directional differences:

$g_{\theta }(x)=\sum _{i=1}^{P}{\frac {f_{\theta }(e_{\theta }(x)\oplus \mu \cdot {\textbf {u}}i)-f_{\theta }(e_{\theta }(x))}{\mu }}\cdot {\textbf {u}}_{i},$ where ${\textbf {u}}_{i}\sim {\mathcal {N}}(0,I)$, $\mu$ is a small scalar for perturbation and $\oplus$ denotes the addition of the vector row-wise.

Detection Rule: If $|g_{\theta }(x)|>t$, where $t$ is a threshold set based on a false positive rate $\sigma$ from a validation set of benign queries, then $x$ is rejected (i.e., Step 2 rejection).

Algorithm Summary

Gradient Cuff performs jailbreak detection in two steps:

1. Sampling-Based Rejection: Reject query $x$ if estimated refusal loss $f_{\theta }(x)<0.5$.

2. Gradient Norm Rejection: Reject if estimated gradient of refusal loss is estimated $|g_{\theta }(x)|>t$.

This process enables effective filtering of jailbreak attempts while maintaining low false-rejection rates on benign queries.

Preprocessing Defense

Preprocessing defenses aim to neutralize adversarial prompts before they reach the target LLM. A widely adopted strategy is prompt paraphrasing, which rewrites potentially harmful instructions using a separate language model, such as ChatGPT, prior to sending them to the target model.

The intuition behind this approach is that benign prompts remain semantically consistent after paraphrasing, while adversarial prompts, particularly those containing suffix-based jailbreak triggers, are fragile and likely to lose their attack effectiveness after rewording. This is because adversarial suffixes rely on specific token sequences, which are easily disrupted by lexical variation.

Paraphrasing

A typical paraphrasing defense pipeline includes the following steps:

A generative model (e.g., GPT-3.5) rewrites the user's prompt using a temperature setting (e.g., $T=0.7$) and a maximum token length constraint.

The paraphrased prompt is forwarded to the target large language model (LLM) for response generation.

The attack success rate (ASR) and response quality are then evaluated.

Effectiveness. Empirical results show that paraphrasing significantly reduces ASR. For example, in the Vicuna-7B-v1.1 model, the ASR dropped from 0.79 to 0.05 after applying paraphrasing. In many cases, the paraphrased prompt triggers a refusal response instead of executing the adversarial instruction. Notably, paraphrasing does not inadvertently convert failed attacks into successful ones—a critical safety property.

Trade-offs. Paraphrasing may degrade the quality of responses to benign prompts. Evaluations using AlpacaEval reveal a 10–15% drop in instruction-following quality, particularly when the paraphrasing model (e.g., ChatGPT) fails to preserve the original prompt's intent or directly outputs a response rather than a rewritten query.

White-box vulnerability. In a white-box setting, adaptive attackers can craft inputs that, after paraphrasing, reconstruct the original adversarial suffix. While this is difficult in gray-box scenarios, transferability has been demonstrated using models such as LLaMA-2-7B as the paraphraser, weakening the defense in fully transparent setups.

Retokenization

To avoid semantic drift introduced by paraphrasing, a milder preprocessing defense is retokenization, which breaks input tokens into smaller subword units. This disrupts adversarial triggers without altering prompt meaning.

Technique. The defense employs BPE-dropout (Byte Pair Encoding dropout), which randomly drops $p\%$ of BPE merge rules. This results in longer token sequences and modifies the token-level structure of the prompt.

Effectiveness. Experiments show that applying BPE-dropout with $p=0.4$ significantly reduces ASR. For Vicuna-7B and Guanaco-7B, ASR dropped from 0.79 to 0.52 and from 0.96 to 0.52 respectively. Unlike paraphrasing, retokenization does not fundamentally alter sentence semantics, preserving response quality to a greater extent.

Limitations. The defense increases context length and may slightly reduce model performance. Additionally, adaptive attackers can inject whitespace-separated characters or unicode variants to exploit the retokenization mechanism. Such attacks have been shown to partially recover adversarial success in white-box setups.

Attack Success Rate (ASR) before and after applying preprocessing defenses. Lower ASR indicates better defense.

Model	No Defense	Paraphrasing	Retokenization (p=0.4)
Vicuna-7B-v1.1	0.79	0.05	0.42
Guanaco-7B	0.96	0.33	0.42
Alpaca-7B	0.96	0.88	--

Data adapted from, Table 3 and Figure 5.

Adversarial Training

Adversarial training enhances every mini-batch with adversarially crafted input . The goal is to optimize the model so that, by using the same parameters, it can perform well on both clean and adversarial examples. Some typical choices of fomulating adversarial examples-inputs crafted by the inner maximisaiton- are Fast Gradient Sign Method(FGSM), and Projected Gradient Descent (PGD). In computer vision, adversarial training is regarded as the strongest empirical defenses, with possibly increasing training time and standard accuracy of robustness. In the scenario of jailbreaking attacks, one baseline is to fine tune the model with harmful prompts. The method is as following:

The goal is to increase robustness of an instruction-tuned LLM against jail-break attacks by exposing it to harmful prompts during fine-tuning. The traditional problem is to craft gradient-based adversarial suffixes for text is thousands of model evaluations per example, making classical per-batch adversarial training impractical for large models.

Approximate strategy.

Start from an Alpaca-style instruction model (LLaMA-7B). Mix in human-written red-team(adversarial) prompts with the harmless data with probability $\beta$ (default $\beta =0.2$). For each harmful prompt do either (1) a standard descent step, reducing loss, toward a generic refusal string (“ I’m sorry, as an AI model … ” ), or (2) descent toward the refusal and ascent on the provided disallowed answer, encouraging the model to push it away.

Observed effects.

1. Slight drop in benign instruction following accuracy.

2. Marginal change in attack-success rate—optimizer-crafted suffixes still succeed.

3. Lowering $\beta$ to 0.1-0.05 caused mode collapse.

Without fast, strong text optimizers, “true” adversarial training remains compute-prohibitive; mixing human red-team prompts offers limited robustness while preserving most task performance.

Recent continuous Adversarial Training(AT) studies focus on encoder–decoder models such as BERT and RoBERTa. Perturbing token embeddings or hidden states encourages smoothness or invariance, improves generalisation with disentangled attention, or boosts task accuracy across diverse NLP benchmarks. Robey et al. adapt randomized smoothing to autoregressive LLMs. While Casper et al. introduce latent adversarial training (LAT), which searches for continuous perturbations in hidden layers. Untargeted LAT fine-tuning helps models forget poisoning triggers but does not explicitly address harmful outputs.

While previous methods either target generic robustness or remain untargeted, targets large language models (LLMs) facing discrete jailbreak attacks. It introduces new algorithms and loss functions that (1) explicitly use the harmful targets produced by such attacks and (2) balance robustness with utility. The authors conduct extensive evaluations across multiple benchmarks and attack families, demonstrating improved robustness-utility trade-offs compared with prior continuous-AT approaches.

Evaluation metrics

Research on jailbreak attacks employs both “attack-side” and “defense-side” metrics to quantify effectiveness, generalizability, and stealth. Common indicators include attack success rate (ASR), queries-to-success, and perplexity (PPL).

Attack Success Rate (ASR)

The attack success rate measures the proportion of prompts that bypass a model's safety filters:

${\text{ASR}}={\frac {N_{\text{success}}}{N_{\text{total}}}},$

where $N_{\text{total}}$ is the number of jailbreak attempts, and $N_{\text{success}}$ is the number that elicit disallowed content. Many studies also report the average number of queries per successful jailbreak as a proxy for attack efficiency.

Determining whether a response constitutes a “success” remains non-trivial. Two evaluation paradigms dominate: (1) Rule-based evaluators, which detect refusal phrases such as “I'm sorry” or “I cannot comply”; and (2) LLM-based evaluators, such as GPT-4, which judge whether a response violates policy, either as a binary label or via a graded harmfulness score.

Toolkits such as JailbreakEval ensemble these approaches and support majority-vote schemes for greater robustness.

Perplexity (PPL)

Perplexity gauges the fluency of a jailbreak prompt:

$\mathrm {PPL} (W)=\exp \left(-{\frac {1}{n}}\sum _{i=1}^{n}\log P(w_{i}\mid w_{1:i-1})\right),$

where $W=(w_{1},\dots ,w_{n})$ is the token sequence, and $P(w_{i}\mid w_{1:i-1})$ is the conditional probability assigned by the language model.

Lower perplexity typically corresponds to more natural text and higher stealth, since heuristic filters often target garbled or low-likelihood strings. As a result, many modern attacks explicitly minimize perplexity to improve transferability.

Benchmarking suites

Public testbeds such as JailbreakBench aggregate ASR, queries-to-success, and PPL across multiple target models, providing a standardized basis for evaluating and comparing jailbreak and defense techniques.

Ongoing risks and open challenges

Despite red-teaming and safety fine-tuning, jailbreaks remain an unsolved problem. Compromised models can:

• spread disinformation;
• facilitate illicit or extremist activities;
• manipulate downstream automated systems.

A key open question is the trade-off between robustness and utility: overly strict defenses may degrade model usefulness, whereas permissive settings invite abuse. Recent proposals call for standardized evaluation frameworks that measure security resilience across diverse attack scenarios.

See also

Prompt engineering

Adversarial machine learning

AI safety