That’s why we developed FilBench: a comprehensive evaluation suite to assess the capabilities of LLMs for Tagalog, Filipino (the standardized form of Tagalog), and Cebuano, on fluency, linguistic and translation abilities, as well as specific cultural knowledge.

We used it to evaluate 20+ state-of-the-art LLMs on FilBench, providing a comprehensive assessment of their performance in Philippine languages:

The FilBench evaluation suite contains four major categories–Cultural Knowledge, Classical NLP, Reading Comprehension, and Generation–divided into 12 tasks. For example, the Classical NLP category includes tasks such as sentiment analysis, whereas Generation tasks include different aspects of translation. In order to ensure that these categories reflect the priorities and trends in NLP research and usage, we curate them based on a historical survey of NLP research on Philippine languages from 2006 to early 2024. (Most of these categories exclusively contain non-translated content to ensure faithfulness to the natural use of Philippine languages.)

Each of these categories provides an aggregated metric. To create a single representative score, we compute the weighted average based on the number of examples in each category, which we call the FilBench Score.

To simplify usage and set up, we built FilBench on top of Lighteval , an all-in-one framework for LLM evaluation. For language-specific evaluation, we first defined translation pairs from English to Tagalog (or Cebuano) for common terms used in evaluation such as "yes" ( oo ), "no" ( hindi ), and "true" ( totoo ) among others. Then, we used the provided templates to implement custom tasks for the capabilities we care about.

Source link