Safety | Ayan Majumdar

ModerationBench: Can Foundation Models Moderate Online Content?

Tue, 01 Sep 2026 00:00:00 +0100

Evaluating LLMs for Demographic-Targeted Social Bias Detection: A Comprehensive Benchmark Study

Mon, 10 Feb 2025 10:44:48 +0100

This project was conducted during my internship at Trustworthy Technology and Engineering Lab (TTE-DE), Huawei Munich Research Center. Detecting and reporting social biases prevalent in large-scale text corpora used for the development of general-purpose AI systems (GPAI) has become a core necessity for trustworthy system development and recent regulatory developments (the EU AI Act). Purely manual annotation at such scale remains infeasible, highlighting a pressing need to develop automated tools for the purpose. However, we lack a clear, fine-grained picture of the capabilities of current language models in detecting different social biases in textual data. In this work, we conduct a comprehensive empirical survey to evaluate large language models (LLMs) in demographic-targeted social bias detection in text data. We develop a new taxonomy that aligns with anti-discrimination principles and accounts for biases targeting multiple demographic axes simultaneously. We adapt twelve large English-language datasets, frame bias detection as a multi-label task, and build a framework to evaluate models using prompting, in-context learning, and fine-tuning.

A. Demographic-targeted Taxonomy

Taxonomy and data coverage

We focus on demographic-targeted social biases and design a taxonomy that identifies the demographic axes targeted by biased texts. This approach aligns directly with anti-discrimination regulations and governance measures (EU Charter of Fundamental Rights), while also enabling the study of multi-axis biases: cases where texts simultaneously target multiple groups, an aspect often overlooked. Specifically, we focus on biases that can target gender identity, sexual orientation, disability, race/ethnicity, nationality, religion, socio-economic status, and/or physical appearance.

Moreover, we create a comprehensive benchmark dataset by combining and adapting instances from twelve well-studied English-language datasets. Importantly, we standardize bias-targeting labels across datasets to align with our taxonomy and ensure consistency. For instance, while bias against “Arabs” or “Middle Eastern” identities is labeled as racial targeting by some datasets and religious targeting by others, we consistently label them as racial bias, reserving REL for explicit religious references in biased texts.

B. Methodological Bias Detection Testbed

We primarily frame the bias detection problem as a multi-label detection task, moving away from traditional approaches that simply perform binary detection, e.g., toxic vs non-toxic or hate vs non-hate. We consider zero-shot prompting and in-context-learning-based prompting of large-scale LLMs and fine-tuning encoder and decoder-based smaller language models (SLM). For a comprehensive analysis, for the prompting cases, we analyze Llama Guard, Llama 3.1 (8B, 70B), Qwen 2.5 (72B), GLM 4 (9B) — models spanning several developers and scales. For the fine-tuning cases, we consider encoder models like RoBERTa (base, large), DeBERTa (v2-XL, v3-large) and decoder models GPT 2 (large, XL).

For prompting, we also design a detailed detection policy that we use.

Data bias detection policy

For fine-tuning, we also analyze the impact of leveraging a balanced loss to tackle class imbalances. We perform training using HuggingFace transformers package.

C. Evaluation Metrics

For detection performance, we analyze binary (biased vs unbiased) detection using $F_1$, FPR, and FNR. For multi-label performance (how well models detect exact bias types), we use Exact match ratio, Hamming loss, macro and micro $F_1$ scores. Finally, we also measure disparities in detection performance for biases targeting the different demographics. We wish to analyze if models disproportionately perform worse for biases that target specific demographics or for multi-targeted biases. If $\mathcal{P}$ denotes either FPR or FNR, and $m, m'$ two demographic axes, we design the per-demographic disparity measure:

$$ \Delta_{\mathcal{P}} = \max _{m,m'} \left| \mathcal{P}_m - \mathcal{P}_{m'} \right| $$

We also measure if the models make systematically more errors in detecting biases that specifically target multiple axes simultaneously (e.g., gender+race) relative to biases that target each constituent axis (e.g., only gender or race). Check out our paper for more details.

D. Key Takeaways

Overall Results

Success of Smaller Models: Our results indicate that fine-tuned SLMs show significant potential for scalable and effective bias detection compared to zero-shot prompting of larger models.

Performance Gaps: Significant disparities remain in how well models detect biases across different demographic axes. Models particularly struggle with multi-demographic (intersectional) targeted biases and subtle social biases.

Inadequacy of Existing Tools: Common safety-focused models and moderation APIs (like LlamaGuard) were found to be less effective at identifying the nuanced social biases present in raw text datasets.

CARMA: Causal Algorithmic Recourse with (Neural) Model-based Amortization

Mon, 10 Jun 2024 10:44:48 +0100

The project represents our recent ACM FAccT 2024 paper. Algorithmic recourse aims to provide explanations to automated algorithmic decision-making systems to enable end-users to understand what needs to be done to change the algorithm’s prediction. To this end, causal recourse showed that using the causal knowledge of the features is vital to ensuring the recommendations are actionable by people while being optimal in terms of cost of feature changes. However, causal recourse has two major practical limitations: i) it requires perfect causal knowledge to compute counterfactual outcomes of its causal recommendations, ii) it needs to solve separate combinatorially complex optimization problems for each person seeking recourse. Hence, for large-scale automated systems, e.g., online applications that provide instantaneous automated lending decisions or instantly filter out CVs in hiring, causal recourse cannot be directly applied. In this work, we introduce CARMA, a practical framework that i) incorporates the power of the latest causal generative models with ii) neural networks that solve a novel amortized predictive formulation of the separate combinatorial problems. Using this practical approach, we show that CARMA can help scale up causal recourse to enable instantaneous, actionable, and optimal recourse recommendation generation.

A. Limitations of existing causal recourse techniques

Consider large-scale predictive settings, e.g., online automated lending systems, that instantaneously filter out and reject loan applicants. Causal recourse can help provide helpful and actionable recommendations that users can use to overturn negative decisions. For each individual with features $\mathbf{x}^{\mathrm{F}}$ requesting recourse, causal algorithmic recourse finds the optimal action $\boldsymbol{a}^{*}$ that generates the counterfactual features $\mathbf{x}^{\mathrm{CF}}$ and obtains the positive prediction from the downstream classifier $h$. Causal recourse finds the optimal solution by solving the following optimization problem $$ \begin{align} \boldsymbol{a}^* \in \underset{\boldsymbol{a} \in \mathcal{A}\left(\mathbf{x}^{\mathrm{F}}\right)}{\arg \min } \operatorname{cost}\left(\boldsymbol{a} ; \mathbf{x}^{\mathrm{F}}\right) \text { subj.to } h\left(\mathbf{x}^{\mathrm{CF}}\left(\boldsymbol{a}, \mathbf{x}^{\mathrm{F}}\right)\right)=1 \end{align} $$ Unfortunately, there are two major limitations that limit the practicality of causal recourse.

We need to know the exact causal relations between the features $X$ (the exact structural causal model or SCM). Unfortunately, this is infeasible for most practical settings.
The optimization problem needs a combinatorial search over the set of all actions $\mathcal{A}$ that involves all feature and value combinations. Unfortunately, this becomes exponentially more complex for large-scale settings, increasing compute times. Owing to these limitations, we cannot deploy causal recourse for practical large-scale settings so that we can provide causal recommendations for recourse instantaneously.

B. Amortizing causal recourse optimization with CARMA

First, to enable providing causal recommendations even when we do not know the exact SCM for a practical scenario, we leverage existing causal generative models like the Causal Normalizing Flows. With data of observed features and knowledge of only the causal graph, these powerful generative models allow CARMA to accurately estimate causal relations, causal exogenous factors $\mathbf{Z}^{\mathrm{F}}$ and compute counterfactual features. CARMA incorporates pre-trained causal generative models and is flexible to use any recent and future models that allow gradient access.

Secondly, note that the causal relations between the features and the downstream classifier are, in fact, common and shared across the entire population. This enables us to convert the separate combinatorial optimization problems into a single predictive task to generate optimal recourse actions. Moreover, we can easily use observed data of features of individuals and train a neural network based framework to learn to generate these recommendations. CARMA trains two NN models end-to-end: the mask network predicts which features $\mathcal{I}$ to intervene on and the action network predicts the optimal intervention values.

CARMA amortized causal recourse framework

C. CARMA provides recourse that is optimal and instant

We apply CARMA as the deployed causal recourse recommendation provider for various causal datasets and the semi-synthetic Loan dataset. We compare CARMA to the Oracle that has access to the exact SCM for each dataset and solves the separate combinatorial problems to find causal recommendations. In our results, we show that CARMA, despite amortization, can provide recourse recommendations that are optimal regarding the oracle in terms of the cost and the causal validity of the recommendations. Moreover, amortization allows CARMA to significantly reduce the time required to generate the causal recommendations, leading to the possibility of instantaneous recourse suggestions.

CARMA provides optimal amortized recourse compared to the Oracle

D. CARMA is flexible in using different generative models

In our evaluations, we train and deploy CARMA using not only the causal normalizing flows generative model, but also VACA that uses graph neural networks. We show in our results that we can train and deploy CARMA using these two different architectures of generative models. Naturally, the Flows model performs better since it was shown to be the state-of-the-art. Nonetheless, our results confirm that CARMA is flexible and can be incorporated with off-the-shelf causal generative models as long as we can compute gradients through these models (note that this is mostly possible since the models are based on deep neural networks).

CARMA can use different causal generative models

D. CARMA can incorporate population-level feature preferences

CARMA’s formulation also allows us to easily incorporate knowledge of feature preferences that a population might have. For instance, a population may find changing Education level very difficult but increasing Savings to be relatively simpler. It is easy to incorporate this knowledge in the optimization of the mask and action networks of CARMA. In our analysis on the Loan dataset where higher numbers for a feature represents more difficult features, we see that CARMA updates its recourse recommendations by leveraging the population preferences. For instance, note that with the weights (wgt.), CARMA intervenes on fewer features and changes the more preferred features like Savings a lot more.

CARMA can adapt recourse solutions to cater to feature preferences of a population

FairAll: Fair Decisions With Unlabeled Data

Tue, 01 Mar 2022 10:44:48 +0100

The project is part of our ACM FAccT 2022 paper. In this work, we explore the helpfulness of integrating unlabeled data into the fair decision-making pipeline to tackle label and selection bias. Although many works exist on trying to learn decision policies that are fair, they often suffer from high variance and instability in these scenarios. So, when i) the labels we work with are biased (and often biased proxies) and ii) the labeled data is sampled non-i.i.d. with selection bias, the prior work often is quite unstable. By this, we mean that the different policies output by the existing frameworks at different times can give very different measures of utility and fairness for the same population. What is more, these policies often forcefully trade-off the utility for gaining fairness. In this project, we show how we can actually approximate hidden ground-truth information (e.g., talent) from observed biased proxy labels (e.g., biased grades) that can help make optimal and fair decisions (e.g., accept or reject students). Further, we highlight the role that unlabeled data can play in learning decision policies that are optimal, fair and stable.

A. Learning fair representations from all data with VAEs

We first aim to use both labeled and unlabeled data in the learning process. Our goal is to approximate the hidden ground-truth label (e.g., the hidden, intrinsic talent of students) from biased features and proxy labels (e.g., biased grades). We do so by modeling the data distribution using both labeled and unlabeled data (which is often discarded). We use a VAE model, conditional on sensitive features (e.g., gender). Our loss function is shown here. Note, we also have an additional classifier model to predict the true utility of each data-point. One important point to note: while prior work required IPS weighting to reduce selection bias on the entire loss function, we need the IPS only for the classifier and not the VAE. $$ \begin{align} J(\theta, \phi, \omega) = &\underbrace{\alpha \mathbb{E}_{(x,\tilde{u}, s)\sim \mathbf{A}^L} [\mathcal{R}(\omega; x, s, \tilde{u}, \pi)]}_{\text{IPS-weighted Classifier}} \\
&-\underbrace{\mathbb{E}_{(x,\tilde{u},s)\sim \mathbf{A}^L, d=1}[\mathcal{L}(\theta, \phi; x, s, \tilde{u})]}_{\text{Labeled VAE-ELBO}} \\
&-\underbrace{\mathbb{E}_{x,s\sim \mathbf{A}^{UL}, d=0}[\mathcal{U}(\theta, \phi; x, s)]}_{\text{Unlabeled VAE-ELBO}} \end{align} $$

B. FairAll: Two-phase decision learning pipeline

We introduce a two-phase decision learning approach that can significantly reduce the instability, while being highly optimal and fair. In fact, in certain scenarios where an unbiased ground-truth actually might exist, we completely do away with the trade-off that is usually assumed. Before we actually perform the decision learning phase (Phase II), we introduce a pre-learning Phase I. In this Phase I, we train a Conditional VAE only on the features. We show that this improves the representation learning and the corresponding stability, optimality, and fairness.

FairAll two-phase policy pipeline

C. FairAll approximates the hidden ground-truth

Here, we show our result where we use synthetic data (with access to ground-truth information) to show that FairAll can approximate the optimal policy based on the ground-truth. This is shown by the blue line where we trained FairAll with Phases I and II, i.e., FairAll (I+II). FairAll (I+II) achieves optimality not only in terms of utility, but also fairness both in terms of demographic parity (DP) and counterfactual (CF) fairness notions. The prior work (FairLog) fails to achieve optimal fairness. We further show the benefit of unlabeled data, as we compare to our method without Phase I unlabeled data (FairAll (II)), and our method with Phase I but without Phase II unlabeled data (FairLog).

Synthetic experiment validating approximation of ground-truth

D. FairAll provides an effective and fair learning process

Using real-world data, we show that over the entire training process across time, FairAll can accumulate higher amount of utility, while also incurring lower amount of unfairness. This is in stark contrast to prior approach FairLog, where we clearly see a trade-off. Note again, the benefits of using unlabeled data in both phases as FairAll (I+II) has the best performance overall.

Real-world experiment showing effectiveness of FairAll in terms of accumulated utility and fairness.

E. FairAll provides high utility and fairness for real data

We show that FairAll, on deployment to unseen real-world data, can provide high utility as well as low unfairness. So, we do not see a trade-off in performance for the policy system, compared to prior method FairLog. In fact, for COMPAS data as shown here, FairAll can provide utility higher than the unfair standard policy learning method (UnfairLog)! Using unlabeled data (compared to FairAll (II) and FairLab) also provides higher utility and fairness.

Real-world experiment showing test-time performance of FairAll

F. FairAll provides a more stable policy

With real-world data, we show that FairAll (I+II), using unlabeled data in both phases, can indeed provide a more stable policy. This means that once the learning process has been given some time-steps to learn, FairAll (I+II) can give more stable utility and fairness responses with respect to the same unseen test data across different time-steps. In contrast, prior work like FairLog can give very different estimates. We measure this variance using Temporal Variance (TV) for utility and fairness. So, we measure the variance of a metric $M$ across time $t$ with respect to the mean value $\mu_M$. $$ \operatorname{TV}_{M}\left(\Pi_{t_{1}}^{t_{2}}\right)=\sqrt{\frac{1}{t_{2}-t_{1}} \sum_{t=t_{1}}^{t_{2}}\left[\left(M\left(\pi_{t}\right)-\mu_{M}\right)^{2}\right]} $$ where $\mu_M = \frac{1}{t_2 - t_1} \sum_{t=t_1}^{t_2} M(\pi_t)$. So, lower the TV, more stable the policy process, whereas higher the $\mu$ for utility (lower for unfairness), better the policy.

Real-world experiment showing stability of FairAll in terms of utility and fairness.

This project was jointly first-authored with Miriam Rateike. Also done in collaboration with Olga Mineeva, Isabel Valera, Krishna Gummadi.

Generating Counterfactuals for Causal Fairness

Mon, 02 Nov 2020 09:54:04 +0100

Notions of causal fairness for algorithmic decision making systems crucially rely on estimating whether an individual (or a group of individuals) and their counterfactual individual (or groups of individuals) would receive the same decision(s). Central to this estimation is the ability to compute the features of the counterfactual individual, given the features of any individual. Recent works have proposed to apply deep generative models like GANs and VAEs over real-world datasets to compute counterfactual datasets at the level of both individuals and groups. In this paper, we explore the challenges with computing accurate counterfactuals, particularly over heterogenous tabular data that is often used in algorithmic decision making systems. We also investigate the implicit assumptions when applying deep generative models to compute counterfactual datasets.

Simplifying causal assumptions

Looking at the fairness and causality literature, we can actually simplify the causal graphs often assumed in the literature.

Example causal graphs from fairness literature

These graphs considered in fairness literature have implicit structures, that we can leverage to simplify them.

Simpler causal graph that we can assume for fairness. Sensitive A is root node along with exogenous U. Some observed features X are affected by A.

So, we can almost always assume a simpler causal graph to work with. The main considerations are:

Sensitive features are root nodes.
Sensitive features do not influence exogenous variables.
Some observed features can be affected by sensitive features.

Using a Conditional VAE to generate counterfactuals

We first model the data likelihood using a Conditional VAE, conditioned on the sensitive features. We minimize the loss: $$ \mathcal{L}=\frac{1}{N} \sum_{i=1}^{N} \mathbb{E}_{q_{\phi}\left(Z \mid X_{i}, A\right)}\left(\frac{\left|X_{i}-\hat{X}_{i}\right|^{2}}{\sigma^{2}}\right)+\mathbb{D}_{K L}\left[q_{\phi}\left(Z \mid X_{i}, A\right) | p(Z)\right] $$ where $\sigma$ is a hyperparameter, $q_\phi$ is the encoder. Then, we generate counterfactuals as follows: $$ P\left(X_{A \leftarrow a^{\prime}}^{c} \mid F\right)=\int p_{\theta}\left(X^{c} \mid A=a^{\prime}, Z\right) q_{\phi}(Z \mid F) d Z $$ So, we perform 3 steps, paralleling Pearl’s steps:

Infer latent Z from observed features using encoder $q_\phi$
Perform intervention, changing sensitive conditional variable
Perform deduction using the Z and changed conditional with decoder $p_\theta$

We can use CVAE to audit for counterfactual fairness

We show that we can use the CVAE model to audit prediction models for counterfactual fairness. Note, how the CVAE generated counterfactuals provide auditing results very close to the oracle model $\mathcal{M}_*$, that is assumed to have the exact causal graph. The prior work FlipTest that uses GANs do not give as optimal results as the CVAE.

Auditing prediction models using CVAE counterfactuals.

We can use CVAE to train counterfactually fair predictors

We next show how we can actually use the latent Z of the CVAE to train prediction models that are counterfactually fair. Note, FlipTest has no immediate way to estimate latent Z from input data, so cannot be directly used here. CVAE-based Fair-Z is able to give very similar prediction results compared to the actual $U$ values from the true causal graphs.

Counterfactually fair predictors with CVAE

This project was done in collaboration with Preethi Lahoti, Junaid Ali, Till Speicher, Isabel Valera and Krishna Gummadi.

Bias in Generative Models

Sun, 01 Nov 2020 10:44:48 +0100

The project was carried out while I was a research assistant at the Max Planck Institute for Software Systems. The project explored the new direction of looking for potential bias in generative models. Models like variational autoencoders and GANs are becoming quite popular and also seeing deployment. The project aims to see if such models can exhibit any notion of bias. In particular, we explore variational autoenccoders and how the data that is generated or reconstructed by such models can be biased to certain demographics. The bias is attempted to be formally defined. The project also involves initial exploration for methods to de-bias such systems, e.g. using additional labels to supervise the training process of the generative models.

A. Exploring generated images of VAE

We train a standard VAE model on the CelebA dataset and use the model to generate new facial images. We explore the generated images for any potential bias, and perform this experiment by varying the population’s gender ratio. Note the original data has a female-male ratio of 52-48. We predict gender using the Clarifai API.

We use these only for research purposes. The CelebA dataset has several issues that have been discussed in the broader literature. Moreover, the moral implications of performing gender prediction from facial images should be looked into in more detail. We also do not assume perfect performance of the Clarifai API.

Note how with original data, there is a clear bias of the model towards generating feminine faces.
Even when Male to Female ratio in training data is equalized, there is a bias towards feminine faces.
The most equality for gender in the generated sample distribution is when we give more Male training photos (60%)!

So clearly, even with equalized data, a standard VAE can generate biased data.

Gender prediction of generated data

B. Auditing reconstructed data of VAE

Using the trained VAE again, we now use a curated auditing set created from the CelebA test data. The audit set is the intersection of gender (male-female) and race (black/african american-white/caucasian). For each face, we use the VAE to reconstruct the face again (feed it as input and analyze the output). We then use the Clarifai API to predict the demographics of the reconstructed faces.

The use of machine learning to predict demographics from facial data, and its implications need further study. We only use this here for an initial research insight.

Surprisingly here, male faces (white and black) are more likely to have the demography correctly predicted.
Black females suffer the most, where almost all the faces are predicted as Hispanic instead.
White females also suffer, but nowhere as poorly as black females.

So, the VAE clearly suffers from some bias in reconstructing data. The way that it reconstructs demographic information in the faces is very different for different groups.

Demography prediction of reconstructed data

C. Generating counterfactual images with Conditional VAE

We now start exploring some potential fixes. First, we assume that the facial data has gender labels available (it is the case for CelebA). Then, we train a Conditional VAE model, conditioned on gender. This actually allows us to generate counterfactual facial images!

Take an input image (say of a male) and feed it to the Encoder of the VAE.
Extract latent z.
Flip the conditional attribute from male to female at the Decoder of VAE.
Reconstruct the face image using the z and conditional Female.

We show some examples here, where we generate counterfactual female faces for some male face images. It is interesting to note what the VAE learns to transform in this counterfactual generation.

E.g., the VAE smoothens the skin, applies makeup to the faces whenever we make counterfactual females.

Counterfactual Male-to-Female images from VAE

D. Effect of conditioning VAE on gender prediction of reconstructed images

We test how a standard VAE and conditional VAE generate images, specifically with respect to the gender. So, we reconstruct input images using different VAE training procedures and use the Clarifai API to see how the gender errors look like. We also play around with the conditional gender values in the models to see how the image reconstructions are perceived by the API.

For the original CelebA images, the API makes more mistakes on Female faces than Male faces.
If we use a standard VAE, the error is significantly higher for Males than Females. So masculine faces are reconstructed to be less masculine.
Using a conditional VAE significantly reduces this bias. The error rates are much more equalized across genders.
If we multiply the conditional values during reconstruction (so instead of conditional Female=1 we say Female=2), the error rates become even lower.
So, multiplying the conditional can make the VAE generate more masculine/feminine faces (at least perceived by the API).
Flipping the gender value at the encoder has no effect, so it does not generate counterfactuals. This is as expected.
Flipping the gender value at the decoder however indeed seems to create proper counterfactuals. These images are almost always predicted to have the opposite gender of the input.
So, male input with flipped gender at decoder will always make the API think the reconstructed image is female.

Gender prediction and conditioning of VAE

This project was done in collaboration with Preethi Lahoti, Junaid Ali, Till Speicher and Krishna Gummadi.