Language testing for your AI assistant, by native linguists
Your AI assistant or chatbot answers users in languages your team may not read. Our native linguists test what it says in the languages and Arabic dialects we offer. Do its answers match your knowledge base, read naturally in your users’ dialect, use your approved terms, and fit the culture? Do they handle sensitive topics the way your own content policy says they should? Every finding comes with the exact conversation, a severity, and a category, so your team knows what to fix.
Gulf Arabic · Halora Bank assistantExample
User
هلا، أبي أحوّل فلوس لأهلي برّا. التحويل الدولي عليه رسوم؟
In English: Hi, I want to send money to my family abroad. Is there a fee for international transfers?
Assistant
هلا فيك! التحويل الدولي من التطبيق مجاني، بدون أي رسوم.
In English: Hi there! International transfers in the app are marked as the problem: free, with no fees at all.
- Finding
- LinguistThe answer leaves out a condition. Your knowledge base says only the first two international transfers each month are free.
- Category
- Accuracy
- Severity
- Critical
- Expected
- Say that the first two transfers each month are free and that a fee applies after that.
- Suggested fix
- Instruct the assistant to state the monthly limit whenever it answers a question about transfer fees.
- Retest
- Retest: passedAfter the fix, the answers to the same prompt and to new variants state the limit.
What we test
In each language and dialect we test, native linguists check your assistant’s answers for:
- Accuracy against your knowledge base. Answers checked against your help articles, product information, and policies. A wrong step, a wrong fact or number, a missing condition, or an answer to a different question is logged with the knowledge-base reference it breaks. We also ask questions your knowledge base does not answer, and check that the assistant says so or hands over, as your instructions say, instead of inventing an answer. A refusal to answer a question your knowledge base does answer is logged too.
- Conversations, not only single questions. Linguists run multi-turn conversations, the way users follow up, change their mind, or ask to speak to a person. They check that the assistant keeps the context, stays consistent, and hands over when your instructions say it should.
- Dialect and register. For Arabic, we test in Modern Standard Arabic and, on request, in Egyptian, Gulf, Levantine, and Maghrebi Arabic. Linguists write prompts the way users in each market write, including Arabic written in Latin letters and numbers (Arabizi) and messages that mix Arabic and English. They check that the assistant answers in the variant and tone you set, whether that is one fixed variant or the user’s own dialect, and keeps to them throughout a conversation.
- Terminology. Product names, feature names, and key terms, checked against your glossary. If you have no glossary yet, we check against your interface and knowledge base, and we can build a glossary with you.
- Cultural fit. Examples, idioms, forms of address, including addressing users with the right grammatical gender, and cultural references that read as natural and appropriate to users in the target market.
- Sensitive-content handling. How your assistant responds, in the target language, to prompts that are offensive or harmful, or that touch on religion, politics, or other sensitive topics, judged against your own content policy. You give us your content policy and your list of sensitive topics, or approve the list we propose, before testing starts.
A finding counts only when it names a concrete defect and points to a reference: your knowledge base, your glossary, your content policy, or your instructions. Preferences are logged separately from errors.
What we don’t test
We don’t run jailbreak, prompt-injection, data-leakage, or other security testing. Those are security tests, and they belong with a security-testing provider.
Our sensitive-content prompts are the kinds of messages real users send. We don’t write prompts designed to trick your assistant into breaking its rules, such as role-play tricks, encoded text, or instructions hidden in documents, and we don’t write prompts that ask for extreme material. If we see something that looks like a security problem while testing, we stop that line of testing and tell your named contact privately.
We test only assistants that your organization operates or is authorized to have tested, and only in a test environment you set up for us. We test written conversations, in chat or messaging. We don’t test speech recognition or voice output.
Our report is not a sign-off on your assistant. It tells you how your assistant responded to the prompts we ran, judged against your knowledge base and your own content policy.
How it works
Every score comes from a native linguist, and every step has a named owner.
- Access and scope. You set up a test environment for us, such as a staging link or a test account, and confirm that your organization operates the assistant or is authorized to have it tested. You also send your knowledge base, your content policy, and your glossary if you have one, and, if you can, a set of real user questions with personal data removed. We agree on the languages, the dialects, and the intents that matter most, such as billing questions or account recovery, and on how many prompts we run and how fast, so the traffic fits your test environment’s limits. If you want right-to-left display checked, give us access through the same chat interface your users see.
- Test plan. We write a test plan for each language: the intents to test, the user personas, the dialects for each, about how many prompts per intent, and the policy categories the sensitive prompts cover. You approve it before testing starts.
- Test prompts. Native linguists write test prompts in each dialect, the way real users phrase things, and our agents help generate variants from your knowledge base. Real user questions you share go into the set too, because they show how your users actually ask. We run the prompts in your test environment and log every response. Assistants can answer the same prompt differently from one run to the next, so we run key prompts more than once.
- Scoring. A native linguist reads the responses your service level covers against your knowledge base, glossary, and content policy, and gives every issue one severity and one category.
- Issue log. Each issue is logged with the conversation, the expected behavior, its category and severity, and a suggested fix. We then summarize the patterns behind the issues.
- Retest. After your team makes its fixes, we run the failed prompts again, with new variants, plus a sample of prompts that passed before, to catch anything the fixes broke. We report what changed. Your quote states how many retest rounds are included.
This is our AI cycle applied to your assistant: agents help with the volume work of generating prompt variants and flagging responses, and every score comes from a native linguist.
How findings are classified
Each issue gets one severity: critical, major, or minor. Severity depends on the likely impact on your users and your business: for example, a wrong fee, or a response that breaks your content policy on a sensitive topic, can be critical. Each issue also gets one category, from the same set we use for every quality report:
| Category | What it covers in your assistant’s answers |
|---|---|
| Accuracy | The answer does not match your knowledge base: a wrong step, a wrong fact or number, an omission, an addition, or an answer to a different question |
| Fluency | Grammar, spelling, and punctuation |
| Terminology | A term that does not match your glossary, or a term used inconsistently |
| Style | Tone, register, and dialect: an answer in the wrong variant of the language, a register your style guide does not allow, or phrasing that reads like a translation |
| Locale | Conventions of the target market: units, dates, number formats, regional vocabulary, and cultural references |
| Tags and format | Links, placeholders, variables, and formatting that break in the answer, including right-to-left display: mixed Arabic and English text, numbers, and punctuation that appear in the wrong order in your chat interface |
| Client requirements | Your content policy and instructions, including the language the assistant answers in and how it should respond to sensitive prompts |
What you get
- A test plan for each language: the intents, the user personas, the dialects, and the policy categories covered
- Scored results: the conversations we ran, your assistant’s responses, and the linguist’s assessment of each scored response
- Results by intent, language, and dialect, with issue counts by severity and category
- An issue log with the conversation, the expected behavior, the category, the severity, an explanation, and a suggested fix for each issue
- A pattern summary: the main issues by language and dialect, and where the fix most likely sits, such as your assistant’s instructions, your knowledge base, or your glossary, for your team to confirm
- A retest report after your fixes: which issues are fixed, which remain, and anything new the retest found
Every result records the date and time, the test environment, and the assistant version you tell us is under test, so your team can rerun the same prompts. Findings apply to that version: a later change to your assistant, its model, or its knowledge base can change its answers.
If a language has too few prompts for a reliable view, the report says so and states how many we ran. Scores appear only in your own report, never on this site.
How deep the scoring goes
Testing uses the names of our three service levels to set how many responses a native linguist scores. The translation steps on that page, such as editing and the extra agent checks at each level, don’t apply to testing. Your quote names the level.
| Service level | Which responses a native linguist scores |
|---|---|
| AI + Check | Every response our agents flag, plus a random sample |
| AI + Edit | Every response |
| AI + Edit + Review | Every response, plus a second linguist’s review of every score |
All responses from assistants that answer health, legal, financial, or other regulated questions are scored at AI + Edit + Review, or with no AI on our side, never at AI + Check. Responses to sensitive-content prompts are scored at AI + Edit or above. In both cases a native linguist scores every response.
Languages and dialects
We test in the languages we list, including Modern Standard Arabic and, on request, Egyptian, Gulf, Levantine, and Maghrebi Arabic. Other languages on request: we confirm we can run them before we quote.
Tell us which countries your users are in and which variant they write in for each language, and your quote names it.
Your access, content, and data
- We sign your NDA, and your data processing agreement (DPA) if you have one, before you share access or files. If a clause doesn’t match how we work, we agree on the change with you before work starts.
- We test only in the test environment or test accounts you set up for us. Please set them up with test data only, with no real customer data, because your assistant’s responses go into our findings. Please don’t put passwords or private test links in our forms: we agree with you by email on how to share access.
- Before testing starts, you confirm that your organization operates the assistant or is authorized to have it tested, and that the terms of the platform and AI services it runs on allow this testing.
- If your organization has vendor-assessment or data-location requirements, send them with your quote request, and we answer them before you share access.
- When we use AI on your content, we use only business AI services whose terms bar training on it. Your content here includes your knowledge base, your policies, and your assistant’s responses. We don’t use your content to train or tune our own AI tools.
- Prefer no AI on your content? Native linguists then write every prompt and score every response, with no AI on our side, and the quote says so.
- We delete your files and test results on request, unless the law requires us to keep them.
Start with a free spot check in one language
We run a small set of test prompts through your assistant in one language, or one Arabic dialect, on a few intents we agree with you by email, and send back the scored findings: each issue with its prompt, response, severity, and category. The findings cover only the prompts we ran. They are indicative, not a rating of your assistant. The spot check leaves out sensitive-content prompts: those run only in a full test, once we have agreed on your content policy.
On the quote form, choose “AI assistant or chatbot testing” and the free spot check, and confirm that your organization operates the assistant or is authorized to have it tested. Under Anything else, tell us the language or dialect. Please don’t include passwords or private test links: we arrange access to a test version of your assistant by email, after your NDA if you need one. We confirm the details by email before we start. One free spot check per company.
Questions about AI assistant language testing
Does it matter which platform or AI model our assistant runs on?
No. We test the text your assistant writes, so the method is the same whatever model it runs on. We need access to a test version of it, such as a staging link or a test account. We test written conversations, in chat or messaging. If your assistant runs only inside a messaging app, tell us, and we confirm how we can reach it before we quote.
Do you test for jailbreaks or prompt injection?
No. Those are security tests, and they belong with a security-testing provider. We test language, accuracy against your knowledge base, and how your assistant responds to sensitive prompts, judged against your own content policy.
Can you test Arabic dialects?
Yes. We test in Modern Standard Arabic and, on request, in Egyptian, Gulf, Levantine, and Maghrebi Arabic, including Arabizi and messages that mix Arabic and English. Tell us which countries your users are in, and we confirm the variant before we quote. Linguists write prompts the way users in each market write, and they check that the answers come back in the variant you set.
What do you need from us?
Access to a test version of your assistant, your knowledge base, your content policy, and your glossary if you have one. If you can, add a set of real user questions with personal data removed. Tell us the languages and dialects to test and the intents that matter most.
Can you fix what you find?
We don’t change your assistant; your team makes the fixes. Each issue comes with a suggested fix, and the pattern summary says where the fix most likely sits. If your knowledge base needs translating, see help-center and support content. If you need a glossary, see glossary, style guide, and translation memory. We quote either one separately.
How is this different from your AI translation quality review?
The quality review assesses translated text you already have, against its source. Assistant testing checks what your assistant writes in reply to users, against your knowledge base and your content policy. Both use the same severities and categories.