Kakao releases safety evaluation results for Kanana-2 AI models, topping global rivals

Scores surpass Google's Gemma and Alibaba's Qwen

Korean-language benchmark Assur AI used; Kanana-2 ranks first across all categories

Safety evaluations to be expanded to all future in-house AI models

A comparison chart of safety evaluation scores for Kakao's small language model Kanana-2 series [Kakao]

Kakao has released the safety evaluation results for its Kanana-2 series of small language models, which the company made available as open source.

Kakao said Tuesday it used its in-house AI Safety Evaluation Platform to assess two models — Kanana-2-1.3B-Instruct and Kanana-2-3B-Instruct. Both models were released as open source on Hugging Face on July 28.

Both models ranked first overall against comparable global models of similar parameter scale, Kakao said.

The evaluation used Assur AI, a Korean-language safety benchmark jointly developed in November last year by the Korea Information and Communication Technology Association (TTA), KAIST and Kakao as part of a Ministry of Science and ICT project. Assur AI is designed around Korea's social and cultural context and measures AI risk factors across a wide range of conditions — from multimodal inputs including text, images and audio, to real-world AI service scenarios and adversarial prompt attacks.

Assur AI covers 9,560 evaluation items across 35 risk categories. Kakao reorganized these into five broad classifications — social risk, sexual content and child protection, crime and illegal activity, violence, and rights violations — and applied an LLM-as-a-Judge method, in which an AI model scores responses based on predefined criteria. The company said this approach minimized subjective variation even in large-scale evaluations and enabled consistent cross-model comparisons.

Kakao benchmarked its models against Google's Gemma and Alibaba's Qwen, two globally recognized models of comparable parameter size. Kanana-2-1.3B-Instruct scored 0.70 overall, outperforming Gemma (0.68) and Qwen (0.58), while Kanana-2-3B-Instruct also scored 0.70, ahead of Gemma (0.66) and Qwen (0.62).

In category-level results, the 1.3B model showed a strong lead in the crime and illegal activity category, while the 3B model significantly outpaced its rivals in sexual content and child protection as well as rights violations.

Kakao plans to expand pre-release safety evaluations to all future in-house AI models, building on this assessment. The company also intends to broaden its evaluation scope beyond text-based language models to cover multimodal models and agentic AI risk assessment.

"This evaluation was an effort to proactively verify the safety of our open-source models through our own verification framework and to make those results public," said Kim Gyeong-hun, Kakao's AI safety leader. "Going forward, we will establish safety verification as a core step in the model release process, in line with our principles of responsible AI development."

chami@heraldcorp.com