Is AI translation really usable? Increasing inquiries from companies asking to "evaluate translation quality"

The world of translation is beginning to change significantly due to the evolution of generative AI.
With the advent of generative AI such as ChatGPT, companies can now easily try out tasks that previously relied on specialized translation software or translation agencies.
"Couldn't we do this translation in-house?"
It is only natural that such voices are being heard.

In fact, we at Kodensha are receiving an increasing number of requests from client companies to evaluate the quality of AI translation.
For example, an inquiry from an electrical equipment manufacturer.

"Which is higher quality, the translation tool we currently use or the recently popular generative AI?"
"If we use AI translation for our product manuals, how much of it can actually be used for practical work?"
"Can we replace some of the work we outsource to translation agencies with AI?"

These are requests to compare translation results using actual documents and evaluate them from a professional perspective in response to such questions.
While it has become easy to try out AI translation itself, judging whether that translation is truly usable for business is surprisingly difficult.
In this article, we will consider the challenges companies face when considering the introduction of AI translation and the importance of evaluating translation quality.

1. AItranslation is evolving faster than you might imagine

A while ago, machine translation likely had a strong image of mechanically replacing words and grammar from the original text.
Many people have probably had the experience of looking at translation results and feeling, "I can sort of understand the meaning, but I can't use this as is."
However, with the spread of neural machine translation and the emergence of generative AI utilizing large language models, the situation has changed significantly.
The range of tasks AI can handle has expanded, from natural expressions that take context into account and paraphrasing complex sentences to even adjusting the tone and style of the text.
In particular, for business documents and general explanatory texts, it is not uncommon for generative AI to output more readable and natural translations compared to conventional machine translation.
We live in an era where such high-quality translations can be easily obtained with familiar tools.
It is perhaps natural for companies to consider "revisiting their translation operations."
However, what we want to be careful about here is that natural-sounding text and accurate translation are not necessarily the same thing.

2. It sounds natural, but is it wrong?

For example, let's consider a product manual for an electrical equipment manufacturer.
With general text, there are cases where slight differences in phrasing do not matter as long as the content is conveyed.
However, it is a different story for product manuals.
They contain a great deal of information where accuracy is required, such as operating procedures, safety precautions, component names, and technical specifications.

"Turn off the power" vs. "Disconnect the power"
"Connect" vs. "Is connected"
"Must be performed" vs. "Perform as necessary"

Even seemingly minor differences can significantly change the meaning depending on product specifications or work procedures.
Generative AI is adept at creating readable and natural-sounding text. On the other hand, it may infer and supplement the meaning of the original text or change expressions to fit the context.
While this can sometimes lead to better readability, it can also lead to changes in meaning that should be avoided in translation.
Furthermore, it is important to ensure that manufacturer-specific product names, technical terms, and company-standardized expressions are correctly reflected.
In other words, simply judging by the translation result and thinking "it's natural, so it's fine" is not enough.

3. AI translation evaluation is not just a simple score comparison

So, what should you do to evaluate the quality of AI translation?
Inputting the same text into multiple AI translation tools and comparing the output results is an effective first step.
However, that alone is not sufficient.
When evaluating, you need perspectives such as the following, for example.

• Accuracy: Whether the meaning and information of the original text are correctly reflected.
• Terminology: Are industry-specific terms and product names translated appropriately?
• Consistency: Are the same terms and expressions used uniformly throughout the document?
• Readability: Is the text natural and easy to understand for the target audience?
• Adherence to Instructions: Have the specified glossaries, translation rules, and writing styles been followed?
• Need for Revision: To what extent is manual correction required for practical use?

For example, in general business documents, readability may be prioritized.
On the other hand, in product manuals related to safety, it is important to convey the information in the original text accurately and without omissions.
Even with the same AI translation tool, the evaluation will change depending on the type and purpose of the document.
In addition, it is necessary to consider not only the quality of the translation but also the time and cost required for post-translation corrections.
Even if the AI translation output looks excellent at first glance, if it takes a lot of time for experts to correct it, it may not lead to overall operational efficiency.
The important thing is not simply to decide which AI is superior.
It is to clarify the scope and manner in which AI translation can be utilized in your company's operations, and to appropriately divide tasks between those left to AI and those to be checked and corrected by humans.

4. The significance of translation professionals evaluating AI translation

This is where the role of a professional translation company becomes clear.
In evaluating AI translation, it is necessary not only to compare the source and translated texts to point out mistranslations or omissions, but also to determine why a translation is problematic and what its practical impact might be.
For example, if there is an error in technical terminology, one must discern whether it is merely a variation in notation or a serious error that changes the technical meaning.
Furthermore, in evaluating translation quality, it is important to minimize discrepancies in judgment among evaluators.
This involves clarifying evaluation criteria and establishing a system where multiple evaluators can make judgments based on the same standards. As necessary, we compare the translated text before and after revisions to analyze what kind of corrections were required.
Through such work, the strengths and weaknesses of AI translation become concretely visible.
At Kodensha, we leverage our many years of experience in multilingual translation and translation support to engage in the evaluation of AI translation tailored to a company's actual documents and usage.
The scope of our evaluation is not limited to specific AI translation services.
It covers various systems under consideration for implementation, such as existing machine translation engines, translations utilizing generative AI, and translation environments built independently by companies.
We believe it is important to consider the content and perspectives of the evaluation in accordance with the unique translation needs of each company.

5. It is a waste to just evaluate and stop there

Conducting an evaluation of AI translation can sometimes lead to unexpected discoveries.

"It was of higher quality than expected"
"General text is fine, but there are issues with technical terminology"
"It excels with certain documents, but requires many corrections for others"

These results serve as important material for determining whether or not to introduce AI translation.
Furthermore, you can also consider how to utilize it beyond that.
For example, you could use AI for the translation of frequently occurring general documents, while combining it with human translator verification for highly specialized documents or important external documents.
Alternatively, you could organize past translation assets and glossaries, and utilize them to continuously improve the quality of AI translation.
Kodensha's "Asset Accumulation AI Translation" is a service based on this way of thinking.
Instead of letting translation results sit idle, you accumulate translation data that has been checked and corrected by humans and apply it to the next translation.
By combining AI with human knowledge and experience, we aim to cultivate translation quality unique to your company.
Evaluating AI translation is not just a task for choosing which tool to introduce.
It is also an opportunity to review your company's translation assets and think about the future of your translation operations.

6. AIWhy not verify the true capabilities of translation together?

With the evolution of AI, the options for translation methods are expanding rapidly.
On the other hand, it can also be said that because the options have increased, it has become more difficult to judge what is appropriate for your company.

"Can translation costs be reduced by using AI?"
"To what extent can current translation work be replaced by AI?"
"I want to compare multiple AI translation services."
"Can we utilize our company's technical terms and past translation data?"

Many companies likely have these questions.
The performance of AI translation is evolving every day. The evaluation results of today are not necessarily the same as those in six months or a year, and there is no guarantee that the same AI will produce the same translation results tomorrow as it did yesterday.
That is precisely why it is important to evaluate using documents used in actual work and to review them periodically.
At Kodensha, we accept consultations regarding AI translation quality evaluation, comparative verification, and streamlining translation work according to a company's translation tasks and implementation objectives.
Whether you are at the stage of considering the introduction of AI translation or feel that there are issues with the translation environment you are currently using, we are here to help.
First, please tell us about your current translation work.
Why not verify the true capabilities of AI translation against your actual work together with us?

Related Services

Asset Accumulation AI Translation | Kodensha