India's nodal cybersecurity agency CERT-In has proposed a new security-audit framework for artificial intelligence systems that could subject major AI companies, including OpenAI and Anthropic, to greater scrutiny over how their models are developed, trained and deployed.

The Artificial Intelligence Security Auditing Guidelines come amid growing concerns over cybersecurity risks linked to AI systems and prescribe a risk-based framework under which AI systems would be classified as low, moderate, high or critical risk.

The framework places particular emphasis on AI agents, which can independently perform tasks, access tools and interact with other systems.

Under the proposed guidelines, AI agents must operate on the principle of least privilege and should not automatically inherit permissions from users, tools or other agents.

High-risk agentic AI systems would also be required to have a “kill switch” mechanism that allows operators to stop the system when necessary.

The guidelines also propose greater scrutiny of an AI system's training data and development process, including whether the data is appropriately sourced, whether bias has been accounted for and whether adequate safeguards are in place throughout the model-development lifecycle.

AI systems classified as critical risk would face the most stringent requirements, including continuous monitoring and independent security audits conducted by organisations empanelled by CERT-In.

Audits would not be limited to conventional cybersecurity vulnerabilities. They would also examine whether systems have measures to mitigate AI hallucinations and whether their outputs could contain harmful, biased or copyright-protected material.

The framework further proposes scrutiny of AI-generated content, with companies expected to use labelling or watermarking to identify synthetic content, depending on the nature and context of its use.

Source link