Evaluating the Social Impact of Generative AI Systems in Systems and Society
A structured social-impact evaluation framework for generative AI offers APS governance teams a ready-made taxonomy for risk and impact assessment work.
Key points
- MIT AI Risk Repository spotlights a 2023 framework for evaluating generative AI's broad social impacts across 11 categories.
- Framework covers both technical system evaluation and societal effects, including bias, privacy, inequality, and labor impacts.
- Useful reference for APS agencies developing AI impact assessment processes, though published in 2023 and not Australia-specific.
Summary
MIT's AI Risk Repository has highlighted a peer-reviewed framework by Solaiman et al. (2023) for evaluating the social impacts of generative AI systems. The framework spans two levels: technical base system evaluation (covering bias, cultural values, disparate performance, privacy, financial and environmental costs, and content moderation labor) and broader societal evaluation (covering trustworthiness, inequality, concentration of power, labor and creativity, and environmental ecosystems). Each category includes modality-specific guidance, discussion of evaluative limitations, and harm mitigation recommendations. The framework was developed through expert workshops and is included in the MIT AI Risk Repository as a structured reference for AI impact assessment.
Implications for Australian agencies
- Consider APS teams developing AI impact assessment or risk frameworks could draw on this taxonomy to ensure social impact categories are comprehensively scoped.
- Monitor Agencies tracking the MIT AI Risk Repository may want to follow subsequent spotlighted frameworks for additional reference material.
Implications are AI-generated. Starting points, not advice.
"Evaluating the Social Impact of Generative AI Systems in Systems and Society" Source: MIT AI Risk Repository – Blog Published: 27 February 2025 URL: https://airisk.mit.edu/blog/evaluating-the-social-impact-of-generative-ai-systems-in-systems-and-society MIT's AI Risk Repository has highlighted a peer-reviewed framework by Solaiman et al. (2023) for evaluating the social impacts of generative AI systems. The framework spans two levels: technical base system evaluation (covering bias, cultural values, disparate performance, privacy, financial and environmental costs, and content moderation labor) and broader societal evaluation (covering trustworthiness, inequality, concentration of power, labor and creativity, and environmental ecosystems). Each category includes modality-specific guidance, discussion of evaluative limitations, and harm mitigation recommendations. The framework was developed through expert workshops and is included in the MIT AI Risk Repository as a structured reference for AI impact assessment. Implications for Australian agencies: - [Consider] APS teams developing AI impact assessment or risk frameworks could draw on this taxonomy to ensure social impact categories are comprehensively scoped. - [Monitor] Agencies tracking the MIT AI Risk Repository may want to follow subsequent spotlighted frameworks for additional reference material. Retrieved from SIMS, 18 May 2026.