More than 500,000 models can be found on the Hugging Face hub, but it’s not always clear to users how to choose the best model for them, notably on the security aspects. Developers might find a model that perfectly fits their use case, but have no systematic way to evaluate its security posture, privacy implications, or potential failure modes.

As models become more powerful and adoption accelerates, we need equally rapid progress in AI safety and security reporting. We're therefore excited to announce RiskRubric.ai, a novel initiative led by Cloud Security Alliance and Noma Security, with contributions by Haize Labs and Harmonic Security, for standardized and transparent risk assessment in the AI model ecosystem.

RiskRubric.ai provides consistent, comparable risk scores across the entire model landscape, by evaluating models across six pillars: transparency, reliability, security, privacy, safety, and reputation.

The platform's approach aligns perfectly with open-source values: rigorous, transparent, and reproducible. Using Noma Security capabilities to automate the effort, each model undergoes:

These assessments produce 0-100 scores for each risk pillar, rolling up to clear A-F letter grades. Each evaluation also includes specific vulnerabilities found, recommended mitigations, and suggestions for improvements.

RiskRubric also comes with filters to help developers and organizations make deployment decisions based on what’s important for them. Need a model with strong privacy guarantees for healthcare applications? Filter by privacy scores. Building a customer-facing application requiring consistent outputs? Prioritize reliability ratings.

Evaluating both open and closed models with the exact same standards highlighted some interesting results: many open models actually outperform their closed counterparts in specific risk dimensions (particularly transparency, where open development practices shine).

Source link