This page is an exploratory draft. It records a working line of thought rather than a finished proposal. The argument below is retained in its original form.
Starting assumption
You could approach this as a language-specific safety system rather than trying to make an AI therapist understand everything that a general-purpose LLM understands.
Your first priority should be minimizing false negatives. Put simply, a false negative is when someone genuinely expresses something dangerous, but your system fails to recognize it. In a therapeutic setting, that is potentially far more serious than a false positive, where an innocent statement gets flagged for human review.
A false positive might mean that someone has to take a second look at a message. A false negative could mean that an important warning sign is missed entirely.
Because of that, your system should be deliberately cautious: when the meaning is ambiguous, it is generally safer to flag the statement for appropriate human review than to confidently dismiss it.
A language-specific semantic representation
The first thing you could build is a language-specific semantic representation. One possible starting point is a GloVe-style word-embedding model trained on a large corpus in the target low-resource language.
In layman's terms, the model learns which words tend to occur around which other words and gradually develops a mathematical representation of their relationships. If two words consistently appear in similar contexts, their vectors will tend to be closer together. This gives you a way of mapping the language's vocabulary without having to teach the system every linguistic rule manually.
However, you don't want to stop at individual words. Imagine that a particular expression in a low-resource language can literally be translated into something involving "death." That does not necessarily mean the person is expressing suicidal intent. It could be a joke, an idiom, a dramatic exaggeration, a quotation, or a completely ordinary culturally specific expression.
Conversely, someone might express genuine suicidal distress without using an obvious "suicide" or "death" word at all. That is exactly why a simple keyword detector would be inadequate.
A second, probabilistic layer
You could therefore build a second, probabilistic layer that considers the context surrounding an expression. Instead of asking, "Does this sentence contain a dangerous word?", the model would ask something closer to, "Given these words, their relationships, the surrounding conversation, the register being used, and the way this expression is normally used, how likely is it that the person is actually expressing harm?"
For example, imagine that an expression which literally translates roughly as "I could die laughing" occurs in your data. A keyword-based system might see "die" and flag it. A contextual model should instead learn that, in an appropriate context, this expression is associated with humour rather than an actual desire to die.
On the other hand, someone could communicate serious distress using indirect language that contains none of the obvious keywords — say "I want to go away forever." Your model should be capable of learning that distinction too.
Cultural and linguistic context
This is where cultural and linguistic context becomes particularly important. You could give the model information about register, slang, code-switching, dialectal variation, idioms, metaphor and other features of how the target language is actually used.
Someone writing formally, someone chatting with a friend, and someone using internet slang may communicate essentially the same idea in completely different ways. Likewise, code-switching between the target language and a global lingua franca could produce sentences that neither a monolingual system in either language handles particularly well.
This is one reason a language-specific model could be valuable even if an existing LLM technically "supports" a low-resource language. Supporting a language is not necessarily the same thing as understanding all of its sociolinguistic nuance. A general LLM may have comparatively limited exposure to particular forms of slang, informal speech, code-switching, regional variation or culturally specific expressions. You would instead be deliberately studying those phenomena and building them into the safety system.
What the training data should contain
The data you collect should therefore contain both actual safety signals and difficult non-safety examples.
For instance, you would want examples of literal expressions of suicidal thoughts or severe distress, but also jokes, metaphors, quotations, fictional writing, discussions about suicide, and ordinary expressions that happen to contain alarming vocabulary. These "hard negatives" are extremely important. Otherwise, the easiest way for your model to achieve high recall would simply be to flag almost everything.
You would also want examples where the meaning cannot be determined from one sentence alone. Consider a message such as "I can't do this anymore." On its own, that could mean anything from frustration with homework to a serious mental-health crisis. If the system has access to the preceding conversation, however, the distinction may become much clearer. For that reason, you should consider whether your classifier is evaluating individual sentences or conversation-level context. In a real therapeutic system, the latter is likely to be considerably more informative.
The examples should then be annotated by humans, preferably people who are both competent in the language and appropriately trained in mental health. The annotators should classify what the statement actually means rather than simply marking whether it contains a particular word. For example, two sentences might both contain a word associated with death, while one is an ordinary idiom and the other is a genuine safety concern. Your dataset needs to preserve that distinction.
A review interface, not a verdict
You could then expose the model through a human-review interface rather than simply giving a therapist an opaque "safe/unsafe" prediction.
A reviewer could see the original sentence, the model's estimated probabilities, and the factors or linguistic patterns that contributed to the classification. For example, the interface might indicate that the model considers a statement highly likely to be genuine distress, moderately likely to be metaphorical, or sufficiently ambiguous that human review is recommended.
The exact categories would depend on your safeguarding framework, but conceptually you could move away from a binary system:
no apparent safety signal → ambiguous → possible distress → possible self-harm → possible suicidality → high-confidence in suicidality
That doesn't mean the model should diagnose someone or determine whether they are actually in danger. Its role would be to identify language that deserves attention and make sure potentially serious cases aren't overlooked. Any actual intervention would remain governed by the company's safeguarding procedures and appropriately trained humans.
A semantic vector map
A particularly interesting second component would be a semantic vector map. You could visualize how different words and expressions occupy the model's semantic space. Instead of looking at a spreadsheet of thousands of words, researchers could actually see clusters of related expressions.
You might discover, for example, that several apparently unrelated slang expressions occur close together because they are used in similar emotional contexts, or that a formal expression and its informal equivalent occupy neighbouring regions of the semantic space.
That map could become useful beyond the immediate classifier. It would give the company a way to investigate how safety-relevant concepts are actually expressed in the target language. In a low-resource language, that foundational linguistic knowledge can itself be valuable.
Gradual development
You could then develop the system gradually. The earliest version might only perform relatively simple category marking: identifying expressions associated with distress, self-harm, suicidality, threats, or other safety-relevant concepts.
As you collect more annotated examples, you could move toward increasingly sophisticated abstraction. Instead of simply learning: "This word is associated with suicide." the model could eventually learn something closer to: "This expression is normally harmless, but in this particular register, conversational context and combination of surrounding expressions, it is being used in a way that is strongly associated with major distress."
That is the fundamental transition I am proposing, moving from word-level categorization to contextual meaning.
You don't need a giant model
You also don't necessarily need to build a giant model from scratch. The point of a specialized system is precisely that you have a much narrower objective than a general-purpose LLM. You aren't trying to teach a computer everything about language. You're trying to teach it one very specific and consequential distinction:
What does this person appear to be communicating, in this language and context, and does it warrant human attention?
What metrics should actually evaluate
Finally, you should evaluate the system using metrics that actually reflect that objective. Overall accuracy can be surprisingly misleading when truly dangerous statements are rare. If 99% of your dataset is non-dangerous, a model that simply predicts "no danger" could appear 99% accurate while being completely useless for safety.
You should therefore pay particular attention to recall and sensitivity and the false-negative rate, alongside precision, specificity and calibration. You should also test performance separately on slang, code-switching, different registers, metaphorical language, dialectal variation and other difficult cases.
The goal is not to build an AI that supposedly understands every low-resource language perfectly. It is to build a safety layer that understands enough of the target language's real-world nuance to notice when something may be wrong. Starting with the linguistic foundations and gradually moving toward contextual abstraction gives you a way to do that without assuming that a general-purpose LLM will automatically understand every language in the same way it understands English.
— end of exploratory draft —
This page is retained in its original form. Framing as exploration reflects that this is a working draft rather than a finished position.