In today’s rapidly evolving digital landscape, ensuring the safety and security of generative applications has become a paramount concern.

Traditionally, red teaming involves a group of security professionals, known as the red team, who adopt the mindset of potential adversaries to test the defenses of an organization. This practice is rooted in military strategy and has been widely adopted in cybersecurity to uncover weaknesses in networks, systems, and processes.

One effective strategy for identifying and mitigating risks in such systems is red teaming. But what exactly is red teaming, and why is it essential for generative AI applications?

Historical Perspective of Red Teaming

Red teaming has its roots in medieval times, where it was initially used for physical security. The Roman army, for example, employed strategies that involved simulating enemy attacks to test and strengthen their defenses. This practice continued through various historical periods, including colonial times and both World Wars, where military forces used red teaming to anticipate and counteract enemy strategies.

With the advent of the internet era, red teaming was adopted to combat cybersecurity threats. This approach has been continuously refined and has now been integrated into the most recent developments in Generative AI and large language models. Today, red teaming is not only a critical practice for addressing security threats but also for ensuring the safety and ethical use of AI models.

Organizational Perspective of Red Teaming

In an organizational or administrative context, red teaming involves distinct teams with specific roles and objectives:

• Red Team: This team adopts the perspective of potential adversaries, actively seeking to identify and exploit vulnerabilities in systems. Their objective is to simulate real-world attacks to uncover weaknesses that need to be addressed.

• Blue Team: The blue team is responsible for defending the organization’s systems. They work to detect, respond to, and mitigate attacks, whether real or simulated. Their objective is to improve the organization’s defensive capabilities and resilience against threats

• Purple Team: This team bridges the gap between the red and blue teams. They facilitate collaboration and communication between the two, ensuring that the insights and findings from red team exercises are effectively integrated into the blue team’s defensive strategies. The objective of the purple team is to enhance the overall security posture by combining offensive and defensive insights.

These teams collectively contribute to a comprehensive approach to security, enabling organizations to proactively identify and address potential threats.

Why should I care?

Red teaming is crucial because it simulates real-world attacks on systems, helping to identify vulnerabilities before malicious actors can exploit them. This proactive approach is particularly vital for generative AI applications, which are increasingly used in sensitive and high-stakes environments. Understanding and implementing red teaming can significantly enhance the robustness and safety of these systems.

Red Teaming in Generative AI

In the context of generative AI, red teaming extends beyond traditional cybersecurity measures. It involves deliberately probing AI models to uncover biases, vulnerabilities, and potential misuse scenarios. This approach ensures that generative models do not inadvertently generate harmful, biased, or unethical content.

In the Context of Responsible AI

Risk Detection and Measurement

For responsible AI, red teaming plays a pivotal role in risk detection and measurement. By simulating harmful scenarios, red teams can identify and quantify risks such as:

• Sexual Content: Ensuring generative models do not produce inappropriate or explicit content.

• Self-harm: Preventing models from generating content that encourages or glorifies self-harm.

• Hate Speech: Detecting and mitigating the generation of hateful or discriminatory content.

• Unfairness: Identifying biases that lead to unfair treatment of individuals or groups.

• Violence: Ensuring models do not produce violent or inciting content.

Jailbreak

Additionally, red teaming helps in identifying “jailbreak” prompts—specific inputs designed to bypass the ethical and safety constraints of AI models. Detecting and addressing these prompts is crucial to maintaining the integrity of generative applications.

As a Risk Mitigation Tool

While red teaming is invaluable for identifying risks, the insights gained are used to develop effective risk mitigation strategies. Purple teaming, in particular, helps in formulating and implementing these mitigation plans. The collaboration between red and blue teams ensures that vulnerabilities are not only identified but also addressed comprehensively to enhance the safety and robustness of AI models. We will explore in further articles how risks could be managed.

Implementation of Red Teaming

One effective approach to implementing red teaming in generative AI is through tools like PromptFlow in Azure AI Studio. These platforms facilitate the creation and execution of harmful and jailbreak prompts in a simulated environment to test AI models.

Setting Up the Chatbot

First, we need a chatbot that can mimic the target system for red teaming. In this example, we have already deployed a sample chatbot (wikipedia-chatbot) in PromptFlow in AI Studio. The steps to deploy this chatbot are not shown here but will be documented in a separate article. <Let me know in comments if an article on this topic will be useful>.

Interacting with the Deployed Chatbot

We use a class ChatbotAPI to interact with our deployed chatbot through its REST API endpoint.

import os
import ssl
import json
import urllib.request
from dotenv import load_dotenv

class ChatbotAPI:

    def __init__(self, api_url, api_key, deployment_name):
        self.url = api_url
        self.api_key = api_key
        self.deployment_name = deployment_name

    def allowSelfSignedHttps(self, allowed):
        if allowed and not os.environ.get('PYTHONHTTPSVERIFY', '') and getattr(ssl, '_create_unverified_context', None):
            ssl._create_default_https_context = ssl._create_unverified_context

    def __call__(self, question):
        self.allowSelfSignedHttps(True)
        data = {
            "question": question,
            "chat_history": []
        }
        body = str.encode(json.dumps(data))
        headers = {
            'Content-Type': 'application/json',
            'Authorization': 'Bearer ' + self.api_key,
            'azureml-model-deployment': self.deployment_name
        }
        req = urllib.request.Request(self.url, body, headers)
        try:
            response = urllib.request.urlopen(req)
            result = response.read()
            return result
        except urllib.error.HTTPError as error:
            error_message = error.read().decode("utf8", 'ignore')
            error_json = json.loads(error_message)
            message = error_json["error"]["message"]
            return message

load_dotenv(override=True)

api_url = os.getenv('API_URL_PF')
api_key = os.getenv('API_KEY_PF')
deployment_name = os.getenv('DEPLOYMENT_NAME_PF')

chatbot = ChatbotAPI(api_url=api_url, api_key=api_key, deployment_name=deployment_name)

Callback Function for Simulator

We define a callback function to pass to the simulator, which will use the ChatbotAPI wrapper and return responses in the specific format that the simulator expects.

import asyncio
from typing import Any, Dict

async def chatbot_callback(prompt: Dict[str, Any]) -> Dict[str, Any]:
    question = prompt.get("question", "")
    response = await asyncio.to_thread(chatbot, question)
    return {
        "question": question,
        "answer": response
    }

Generating Harmful and Jailbreak Prompts

Using the simulator, we generate a set of either harmful prompts or jailbreak prompts.

# Assuming AdversarialScenario and AdversarialSimulator are defined and imported
from promptflow.evals.synthetic.adversarial_scenario import AdversarialScenario

simulator = AdversarialSimulator(azure_ai_project={
    "subscription_id": os.getenv("SUBSCRIPTION_ID"),
    "resource_group_name": os.getenv("RESOURCE_GROUP"),
    "project_name": os.getenv("PROJECT_NAME"),
    "credential": os.getenv("CREDENTIAL")
})

# Generate harmful prompts
harmful_outputs = await simulator(
    scenario=AdversarialScenario.ADVERSARIAL_QA,
    target=chatbot_callback,
    max_conversation_turns=1,
    max_simulation_results=100,
    jailbreak=False
)

# Generate jailbreak prompts
jailbreak_outputs = await simulator(
    scenario=AdversarialScenario.ADVERSARIAL_QA,
    target=chatbot_callback,
    max_conversation_turns=1,
    max_simulation_results=50,
    jailbreak=True
)

# Convert outputs to a format suitable for evaluation
def to_eval_qa_json_lines(outputs):
    json_lines = []
    for output in outputs:
        json_lines.append(json.dumps({"question": output["template_parameters"]["conversation_starter"], "answer": output["messages"][1]["content"]}))
    return "\n".join(json_lines)

# Create datasets
harmful_dataset = to_eval_qa_json_lines(harmful_outputs)
jailbreak_dataset = to_eval_qa_json_lines(jailbreak_outputs)

# Save datasets to files
with open("harmful_dataset.jsonl", "w") as f:
    f.write(harmful_dataset)

with open("jailbreak_dataset.jsonl", "w") as f:
    f.write(jailbreak_dataset)

Evaluating the Chatbot

Using PromptFlow’s evaluation functionality, we evaluate the generated dataset to obtain key performance indicators (KPIs) or Metrics. To quantify the risks, safety metrics are calculated. These metrics include:

• Sexual Content: Frequency and severity of inappropriate content generation.

• Self-harm: Instances of content encouraging self-harm.

• Hate Speech: Detection of hateful or discriminatory content.

• Unfairness: Identification of biases and unfair treatment.

• Violence: Occurrences of violent or inciting content.

from promptflow.evals.evaluate import evaluate
from promptflow.evals.evaluators import (
    ViolenceEvaluator,
    SexualEvaluator,
    SelfHarmEvaluator,
    HateUnfairnessEvaluator,
    ContentSafetyEvaluator
)

# Define the project scope of AI studio
project_scope = {
    "subscription_id": os.getenv("SUBSCRIPTION_ID"),
    "resource_group_name": os.getenv("RESOURCE_GROUP"),
    "project_name": os.getenv("PROJECT_NAME")
}

# Define evaluators
evaluators = {
  'violence': ViolenceEvaluator(project_scope=project_scope),
  'sexual': SexualEvaluator(project_scope=project_scope),
  'self_harm': SelfHarmEvaluator(project_scope=project_scope),
  'hate_unfairness': HateUnfairnessEvaluator(project_scope=project_scope),
  'content_safety': ContentSafetyEvaluator(project_scope=project_scope)
}

# Define evaluator config - Target is the data
evaluator_config = {
    "default": {
        "question": "${data.question}",
        "answer": "${data.answer}"
    }
}

# Evaluate harmful dataset
results_harmful = evaluate(
    data="harmful_dataset.jsonl",
    evaluation_name=f"red_teaming_eval_harmful-{time()}",
    evaluator_config=evaluator_config,
    evaluators=evaluators
)

# Evaluate jailbreak dataset
results_jailbreak = evaluate(
    data="jailbreak_dataset.jsonl",
    evaluation_name=f"red_teaming_eval_jailbreak-{time()}",
    evaluator_config=evaluator_config,
    evaluators=evaluators
)

We used PromptFlow SDK and Azure AI studio to implement automated safety evaluations. Microsoft also released PyRIT (python package) tool that helps setup manual and as well as automated Red Teaming. <Let me know in comments if you want me to explain PyRIT in a separate article>.

Prepare and Implement Safety Risk Mitigation Plan

Finally, based on the insights gained from red teaming, a comprehensive safety risk mitigation plan is prepared. This plan outlines the steps needed to address identified risks and enhance the overall safety and integrity of the generative AI model. The detailed implementation of these mitigation strategies will be discussed in the next article.

By systematically applying red teaming principles to generative AI, organizations can proactively identify and address potential risks, ensuring the responsible and ethical deployment of these powerful technologies.

In the time of COVID-19, almost all aspects of our lives are affected, some being slightly resilient and others being heavily hit. Even the weather forecast are going to be affected by corona while weather is not. The reason for that is the dependencies of the weather forecasting models on the data collected by aircraft e.g., Wind speed, Wind direction, Air pressure, Air temperature together with Temporal and Spatial information. As per ECMWF, The data collected by aircraft is second only to satellite data in terms of their significance for weather predictions. Now due to the very heavy reduction in air traffic the weather forecast accuracy will be slightly impacted but the impact will still be statistically significant.

A lot of work is already happening to cope with COVID-19 pandemic. I see a lot of articles on the internet about how Machine Learning or AI could be used to combat Corona Virus e.g., Forecasting models to predict the outbreak, Computer Vision models to better screen the existence of Corona infection in the X-ray/CT-scan images.

However, I do not see much contribution in the direction of modeling the impact of corona virus on businesses. This is obvious that majority of the businesses are affected by the pandemic but the question is how can we forecast the demand or sales given the corona pandemic in a particular region. The answer to this question becomes even more interesting for the companies who are currently using ML models for the forecasting of their business. How could the models running in production incorporate the COVID-19 related data to model the impact on business and adjust the forecasts accordingly? Here at Continental, we are running a number of forecasting models to predict a number of things e.g., Demand, Revenue, Sales, Raw material requirements.

I could think of couple of strategies which I am planning to test if they help to adapt the behavior of our forecasting models:

A very naive approach could be just to add a binary feature which could emphasize on the time-period when an outlier event e.g., COVID-19 took place which of course could be different for different regions. Perhaps a step a forward approach could be to also add the actual effect of this event (COVID-19) hence the model not only learns when the event happened but also how the event evolved within that period e.g., daily new cases, daily recovered cases, daily deaths due to COVID-19. This also depends upon the aggregation level (forecast step size) of the data. I could also think of adding a data set e.g., employment data, stock data which could exhibit the impact of COVID-19, and we use this effect from aforementioned data sets as a feature to learn the behavior. Of course, the semantics of such a data set should be close to the business we are forecasting. I wrote this post to start a discussion around this topic to brainstorm on the ideas we can use to combat with the adverse effect on our respective businesses. We can already see the impact of COVID-19 on the employment e.g., In the US alone, 16 million jobs are already gone while more than 6.6 million alone in the week of 9th April, 2020. I hope that companies will quickly develop resilience to the impact and leave minimal impact on employment. In the meantime, we continue to contribute!

Looking forward to your ideas/comments on modeling the impact of COVID-19 in our ML models.