The generative AI boom—especially in text-to-image (T2I) models like Stable Diffusion and DALL·E—has unlocked creative possibilities at scale. But behind every new model, a less-glamorous question shapes the technology's risks and opportunities: What exactly is in the data? This was the focus of our publication in IEEE Xplore (2025), a collaboration between Duke, UNC Charlotte, and Meta's GenAI Safety Team. Here's what we set out to do, how we did it, and what I learned as a researcher embedded in the safety trenches.
The Motivation: Why Dataset Analysis Matters
T2I safety research often starts "downstream"—at the level of filters or classifiers that try to catch harmful content. But after reading case studies of jailbroken diffusion models and AI "leakage," I became convinced that real safety starts upstream: with systematic auditing of datasets.
Most large T2I datasets are scraped en masse, poorly documented, and labeled inconsistently. This is not just a research problem—it's the root cause of failures in safety, bias mitigation, and regulatory compliance.
Our Methodology: How Do You Audit AI Datasets at Scale?
1. Dataset Compilation
We gathered all major open-source datasets used for T2I safety, including SafeGenExplicit56k, LatentGuardCoPro, ART, P4D, Adversarial Nibbler, MMA Diffusion, SneakyPrompt, and I2P. Each dataset had its own quirks—some adversarially generated, others crowdsourced, some paired with real images, others fully synthetic.
2. Standardized Harm Taxonomy (AIR 2024)
To cut through the labeling confusion, we mapped all prompt-image pairs to a single, granular harm taxonomy: the AIR (AI Risk) 2024 hierarchy, which has 16 Level-2 and 45 Level-3 harm categories. We automated much of the mapping using GPT-4o, then hand-labeled over 1,300 samples for validation, calculating Bayesian confidence intervals for labeling accuracy in each harm class.
3. Diversity Analysis: Not Just 'More Data', but 'Better Data'
We didn't stop at counting harms. We measured:
- Syntactic diversity (intra- and inter-distinctness of n-grams)
- Semantic diversity (average cosine distance of prompt embeddings)
- Linguistic diversity (language detection via XLM-RoBERTa)
- Topical diversity (tagging via prompt engineering with LLMs)
4. Quantitative & Qualitative Analysis
- Visualized distribution of prompt lengths, sentiment scores, word/bi-gram frequencies, and harm coverage per dataset.
- Mapped disagreements between original dataset labels and standardized AIR categories.
What We Discovered
1. Harm Coverage is Dangerously Imbalanced
~49% of "harmful" prompts were sexual content, while categories like misinformation, discrimination, and political manipulation were under 8%. Datasets reflected what was easiest to label, not what was most important to society.
2. Multilingual Gaps
Over 98% of prompts were in English. Less than 2% covered other languages, which is a huge blind spot for models deployed globally.
3. Labeling Inconsistencies and Ambiguities
Original dataset labels often disagreed with our standardized taxonomy—especially for nuanced classes like "fundamental rights" or "discrimination/bias." Even for well-known harms, labels drifted due to cultural differences or unclear definitions.
4. Dataset Quality Issues: Redundancy, Artifacts, and Synthetic Bias
Some datasets had high levels of redundancy (e.g., the same words/bi-grams repeated), or weird artifacts like "araffe" (nonsense tokens from synthetic image captioners). These issues hurt the generalizability of safety classifiers.
What I Learned (And What I'd Do Differently)
Upstream safety = more impact. All the downstream filtering in the world can't make up for weak, unrepresentative, or poorly labeled data. Dataset curation is AI safety.
Evaluation needs to be multidimensional. Looking at only the "types" of harms is not enough—you need to consider diversity in structure, language, and semantics. Metrics like cosine distance for prompts and inter-distinctness of n-grams became my new favorites.
Automation is necessary—but not sufficient. LLMs can help with large-scale relabeling, but human validation is essential, especially for ambiguous or low-frequency harm classes.
Working with industry partners lifts your game. Meta's team pushed us to think like product owners—balancing rigorous analysis with what actually moves the needle for deployed safety stacks.
Looking Forward
If T2I models are to be trusted in the wild, auditing and improving the upstream datasets should become standard practice—just like model evaluation or post-training safety layers. I hope our review pushes the field toward greater transparency and real-world robustness.
If you're building or using generative AI, start with the data. Everything else follows.