Hello. I am Yeoun Yi, an ML engineer on the AI Services Lab team. Our team develops models to prevent harmful OpenChat rooms from being exposed to users. In this article, I describe the process of developing a model that determines whether an OpenChat room is harmful based on its name and description.
OpenChat monitoring introduction
When creating an OpenChat room, users must provide an OpenChat room name and may optionally leave a description for the OpenChat room. Operators need to review new or edited OpenChat room names and descriptions to check for expressions that violate the operation policy. As LINE is a global messenger, many OpenChat rooms are created and edited every day, and our team has developed monitoring models to reduce the amount of manual review required by people.
Existing monitoring models have already reduced manual review substantially in several countries. However, in some countries that require more granular judgment criteria, the existing models could not be used for automatic review. The goal of this project was to improve model performance to expand the scope of automatic review. We also ran various experiments to find ways to improve accuracy in countries that already use automatic review.
Data cleansing
We used previously manually reviewed OpenChat room names and descriptions for training. Considering that the review guidelines are updated over time, we only extracted data from the period where the current guidelines applied and used that as the training data.
When analyzing the data, we found many cases where identical OpenChat room names and descriptions received different penalty outcomes. To resolve this, when the same name and description had different penalties, we determined the final label using the following rules.
First, if the most severe penalty was assigned at least twice, we selected that penalty as the final label. If the highest-severity penalty appeared only once, we did not choose it as the final label because it may be noise. In other cases, to be conservative we chose the second-most severe penalty as the final label. If the penalty reasons differed, we selected the most frequent reason as the final label. If frequencies were tied, we preferred the reason that was rarer in the whole dataset. This approach is inspired by TF-IDF. We assumed that reasons that are not common in the overall dataset are more likely to describe the specific data precisely.
# Pseudocode for data cleansing
FOR EACH group IN GROUP_BY(records, key = (name, description)):
codes = SORT_DESC(group.penalty_codes, key = severity_rank)
top_code = FIRST(codes)
# Use the most severe penalty if it appears at least twice
IF COUNT(codes, top_code) >= 2:
final_code = top_code
# Otherwise, use the second-most severe penalty
ELSE:
final_code = SECOND_HIGHEST(codes, key = severity_rank)
reason_count = COUNT_BY_VALUE(group.penalty_reasons)
candidates = ARGMAX_ALL(reason_count)
# Use the most frequent reason
IF SIZE(candidates) = 1:
final_reason = FIRST(candidates)
# Break ties by choosing the globally rarer reason
ELSE:
final_reason = ARGMIN(candidates, key = global_reason_frequency)
EMIT(group.name, group.description, final_code, final_reason)OpenChat monitoring model
Pretrained model selection
We evaluated pretrained models using the following criteria.
- Decoder-only model
- Tuned for safety moderation tasks
- Around 2B in size
- Available under the Apache license
Recently, decoder models have shown strong performance even on classification tasks, which were traditionally considered more favorable for encoder models. Therefore, unlike prior encoder-only approaches, we decided to experiment with decoder models. We also believed that models already tuned for safety moderation would be advantageous for harmfulness detection even across languages and domains. Model size was an important consideration because LINE needs to monitor a large number of OpenChat rooms created or edited in real time, and very large models can be costly to serve and slow to respond. Therefore we considered relatively lightweight models around 2B. Finally, to avoid licensing issues we prioritized models under the Apache license that allow commercial use.
We ultimately selected the Granite Guardian 3.1 2B model. Granite Guardian is a model released by IBM Research that performs harmfulness classification by comparing the probabilities of predefined tokens corresponding to answers to the question "Is the input harmful?" such as "Yes" or "No". A typical generative model might produce long explanations or unexpected formats rather than the tokens "Yes" or "No". Granite Guardian evaluates harmfulness by comparing the generation probabilities of fixed candidate tokens, which lets us leverage the generative capabilities of the language model while avoiding output format variability. We also interpret token generation probability as model confidence and can tune a threshold to meet operational performance requirements.
Training
The OpenChat monitoring model follows Granite Guardian's basic approach but includes modifications tailored to OpenChat data and operational needs.
First, review results for OpenChat room names and descriptions cannot always be expressed as a simple "Yes" or "No" binary classification. Depending on the content of the name or description, the penalty level may be lower or higher. Also, when applying a penalty we must provide the reason for the penalty as well as whether it is harmful. Thus the model must predict both the penalty level and the penalty reason. We structured prompts in the following format.
<|start_of_role|>system<|end_of_role|>
You are a safety monitoring agent.<|end_of_text|>
<|start_of_role|>user<|end_of_role|>
{open chat name and description}<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>
Action: {Penalty code token}
Reason: {Penalty reason token}<|end_of_text|>We used cross entropy loss, commonly used for next-token prediction. However, when calculating the loss we computed it only over the model response region, not the entire prompt, because our goal was not to improve the model's generation of the input text but to better predict the penalty code and reason for the given OpenChat room.
To train efficiently we applied LoRA. LoRA keeps the original model parameters fixed and approximates the update as a product of two small matrices that are updated. This drastically reduces the number of parameters and memory required for training while preserving the pretrained knowledge.
Inference
At inference time we compared the generation probabilities of each penalty code token. Decoder models such as transformer-based decoders compute scores for all tokens when generating the next token. We extracted scores for tokens corresponding to penalty codes, converted them to probabilities, and interpreted those probabilities as the likelihood of each penalty code.
In production, penalty codes and reason codes are composed of arbitrary letters and numbers and may be split into multiple tokens by the tokenizer. For probability calculation it is convenient if each penalty code is represented as a single token. We therefore mapped each penalty code and penalty reason to natural language tokens that correspond to single tokens, which also helps learning compared with using arbitrary code strings directly.
The inference code is implemented as follows.
class OndfPredictor:
...
self.penalty_code_ids = [self.tokenizer.convert_tokens_to_ids(t) for t in PENALTY_CODE_TOKENS]
self.penalty_reason_ids = [self.tokenizer.convert_tokens_to_ids(t) for t in PENALTY_REASON_TOKENS]
# exclude <bos>
self.inner_template = self.tokenizer('\nReason:', add_special_tokens=False).input_ids
...
# (1) predict penalty_code
outputs = self.model(**inputs, use_cache=True)
logits = outputs.logits
past_key_values = outputs.past_key_values # cache for (2)
last_token_logits = logits[:, -1, :]
penalty_code_logits = last_token_logits[:, self.penalty_code_ids]
penalty_code_probs = torch.softmax(penalty_code_logits, dim=1)
pred_code_scores, pred_code_ids = torch.max(penalty_code_probs, dim=1)
# map subset indices back to original token IDs & add '\nReason:' tokens after penalty_code
pred_code_ids = torch.tensor([[self.penalty_code_ids[i]] + self.inner_template for i in pred_code_ids.cpu().tolist()]).to(self.device)
# (2) predict penalty_reason
outputs_step2 = self.model(
input_ids=pred_code_ids,
past_key_values=past_key_values,
use_cache=True # use KV-Caching
)
last_token_logits_2 = outputs_step2.logits[:, -1, :]
penalty_reason_logits = last_token_logits_2[:, self.penalty_reason_ids]
penalty_reason_probs = torch.softmax(penalty_reason_logits, dim=1)
pred_reason_scores, pred_reason_ids = torch.max(penalty_reason_probs, dim=1)
# map subset indices back to original token IDs
pred_reason_ids = [self.penalty_reason_ids[i] for i in pred_reason_ids.cpu().tolist()]
First we extract the probability for each penalty code. We then assume the penalty code with the highest probability was generated and generate the corresponding penalty reason. Since the input prompt is identical to the previous step except for the added penalty code, we applied key-value caching to avoid redundant computation and speed up inference.
Key-value caching (KV caching) reuses the computed keys and values for previous tokens. Decoder-only transformers compute attention scores between tokens when generating the next token. Each token needs query, key, and value vectors, and keys and values for existing tokens do not change as new tokens are added. Reusing them avoids recomputation and speeds up inference. After generating the penalty reason we extract probabilities for each reason in the same way as for penalty codes.
Performance evaluation
We conducted offline evaluations in three countries after training. Country A used about 10 days of data, and countries B and C used about one month of data each. The longer evaluation periods for countries B and C were because those countries already had automatic processing in place and fewer manual reviews remained.
We used F1 as the evaluation metric because it considers both precision and recall. The reason for choosing F1 is as follows.
Most OpenChat rooms are "no issue", so it is most important to correctly identify OpenChat rooms that have no problem. The normal-class precision measures the proportion of OpenChat rooms labeled "no issue" by the model that are indeed problem-free, indicating how accurate automatic "no issue" handling is. The normal-class recall measures the proportion of actually problem-free OpenChat rooms that the model labels as "no issue", indicating how many can be automatically processed as "no issue". If the model labels fewer OpenChat rooms as "no issue", normal-class precision increases but normal-class recall decreases. If the model labels more OpenChat rooms as "no issue", normal-class recall increases but precision decreases. Precision and recall involve a trade-off, so we used their harmonic mean, F1, to consider both at once.
LINE applies very strict standards for AI-based automatic processing and requires high precision before enabling automation. Therefore we introduced a threshold so that even if a penalty code has the highest probability, if that probability is below the threshold, the case is routed to manual review. We adjusted the threshold to maintain high precision in operation.
The performance evaluation graph is shown below (exact numbers cannot be disclosed for security reasons). The normal-class F1 improved substantially in all regions compared with the previous model. NG-class precision is the proportion of OpenChat rooms the model marked as problematic that are actually problematic, and NG-class recall is the proportion of actually problematic OpenChat rooms that the model marked as problematic. The NG-class F1, which considers both, also improved substantially in all regions compared with the previous model.
For reference, the reason countries B and C show lower previous-model normal-class F1 in the above graph is that, as mentioned earlier, we excluded data that was already processed automatically by the previous model from the evaluation. To reduce bias we also compared performance on the full dataset regardless of automatic processing, and in that case the newly developed model still outperformed the previous model as shown in the graph above.
The new model also performed well in online evaluation after offline testing, allowing us to expand the scope of automatic review.
Analysis of improvement factors
To determine how much each factor contributed to the performance improvement, we compared the same model using different classification methods and threshold settings.
First, classification method. In typical classification tasks we add a classification head on top of the final hidden state and let the head predict scores for each class. We compared this classification-head approach with the approach of classifying based on the probability of generating candidate tokens. As shown in the comparison graph below, both normal-class F1 and NG-class F1 improved when using the candidate-token-probability method.
Next we examined differences in precision (left graph) and recall (right graph) with and without thresholding.
The method that classifies as "no issue" whenever the normal-class token probability is highest without thresholding did not meet the precision required for automatic processing. However, the method that requires the normal-class token probability to be both highest and above a threshold met the precision requirement and could be used for automation.
After thresholding, recall decreased as shown in the right graph because precision and recall trade off. But if precision is not sufficiently high, even a high recall cannot be used for automatic processing, so improving precision was more effective for us.
Conclusion
We have introduced the development process of a model that determines whether OpenChat rooms are harmful based on their names and descriptions. We covered cleaning training data, selecting a pretrained model suitable for the task, designing inference to meet operational requirements, and tuning thresholds.
In the future we plan to set training weights according to data quality and to consider model cascading where cases that could not be automatically processed are re-inferred with a larger model.
We will continue improving performance so that all LINE users can enjoy a safe and healthy community. Thank you for reading to the end.
