Document Digitization

Document digitization is the process of converting physical documents into digital formats, making them easily accessible and editable.

Layout Detection

Anuvaad is coupled with custom trained Layout detection models for Identifying and comprehending a document's structure, which involves the recognition of key elements, including headings, paragraphs, tables, and images. This process is essential not only for enhancing OCR accuracy but also for preserving the document's layout and structure in the translated version.

Document Translation

Document translation involves converting text from one language to another, facilitating cross-lingual communication and information access. Anuvaad supports using NMT models straight from Bhashini Dhruva or in-built plug and play type of models for domain specific use cases.

Document Structure Preservation

This feature ensures that the original formatting, layout, and structure of documents are maintained during the translation process, preserving the document's visual integrity.

Improve Translation from Speech

Speech to text technology converts spoken language into written text, enabling audio content to be transcribed for translation or other purposes.

Translation Memory

Translation memory stores and retrieves previously translated segments to ensure consistency across documents and reduce translation time.

Glossary Support

Glossary support provides access to defined terminology and specialised vocabulary, ensuring consistency and precision in translations, particularly in specialised fields.

Usage Analytics and Metrics

Usage analytics and metrics offer insights into how the platform is utilised, helping users track and optimise translation processes and workflows.

File Format Conversion

File format conversion simplifies the process of converting documents from one file format to another while preserving their content and structure, enhancing compatibility.

Transliteration Support

Transliteration support enables the conversion of text from one script or alphabet to another, aiding users in dealing with different writing systems and ensuring the correct pronunciation of words, especially in multilingual contexts.

Project Anuvaad is an open-sourced project funded by EkStep foundation.

It was bootstrapped by EkStep Foundation in late 2019 as a solution to enable easier translation of legal documents from English to Indic languages & vice-versa. Creating Anuvaad platform allowed legal entities to digitize & translate the Orders/Judgements using an easy to use user interface.

Anuvaad is an AI based open source Document Translation Platform to digitize and translate documents in Indic languages at scale. Anuvaad provides easy-to-edit capabilities on top the plug & play NMT models. Separate instances of Anuvaad are deployed to Diksha (NCERT), Supreme Court of India(SUVAS) and Supreme Court of Bangladesh (Amar Vasha).

Anuvaad leverages state of the art AI/ML models including NMT, OCR, Layout detection to provide high level of accuracy. Project Anuvaad was envisioned to be end to end open sourced solution for document translation across multiple domains.

Project Anuvaad is REST APIs driven and hence any third party system can use various features like sentence translation, layout detection etc.

NOTE: The documentation is still WIP. Feel free to contribute to it or raise issues if the desired info is not uptodate. Explore the KT videos if you would like to dive deep into each module.

Configs:

Parameters of a module that can be injected in and out of the system with zero to minimal code change in order to enable/disable/modify certain features of the system. The configs pertaining to modules of anuvaad data-flow pipeline can be broadly classified into 2 categories as:

1. Configs outside the build (docker image)

2. Configs within the build. (docker image)

Configs outside the build:

These are those configs which are injected to the system on the fly, the changes injected thereby can be incorporated into the system on runtime or on just a restart of the system without having to re-build or push a logical piece of code. For instance, WFM reads configs for identifying different workflows configured. In order to add/edit/delete a workflow, one needs to make changes to the config file as required and push the file, the changes will be incorporated into the system on restart or through a reload API on runtime. (https://raw.githubusercontent.com/project-anuvaad/anuvaad/master/anuvaad-etl/anuvaad-workflow-mgr/config/etl-wf-manager-config-dev.yml) These files will be saved in the ‘configs’ folder outside the source code of the system.

Configs within the build:

These configs travel with the build. Meaning, these configs will be a part of docker image. These configs can be controlled via an environment file during deployment or internally within the code. This also means that any changes in these parameters will need a rebuild and re-deployment of the system. However, no change in the logic or the code should be needed to incorporate the changes. Most of the hooks that are exposed for a given system fall under this category. These are mentioned in the ‘configs’ folder inside the source code. It is recommended to use just one ***config.py file inside the folder for all these configs for better maintainability. In case someone prefers to separate out config files based on concern, they can do so but bear the overhead of maintaining them. For convenience and readability, these configs are further divided as:

1. Cross module common configs

2. Module specific configs.

Cross module common configs:

These configs are used across all modules. Configs like Kafka host, Mongo host, File Upload URL etc fall under this category as these will not change from module to module. However, if there’s a case where modules chose to use different values for these parameters they can go ahead by using a different variable. The point of having this is ensuring we don’t create redundant variables in the environment file and use them from the same variables that are already defined.

Eg:

Module specific configs:

These configs are specific to the module, and will change for each module. This category includes both the common variables (Eg: https://github.com/project-anuvaad/anuvaad/blob/69b494224626d51a7baf0405603106a4a66a25c7/anuvaad-etl/anuvaad-extractor/aligner/etl-aligner/configs/alignerconfig.py#L10 ) inside the code which are used at multiple places in your project and the variables deriving value from the environment file (Eg: )

For convenience the second type of variables are categorised as:

1. Kafka Configs: Configs required for kafka like topics, consumer groups, partition keys etc. that are very specific to the module. Some other parameters required to customise your consumer and producer as per your requirement can be mentioned under this category.

2. Datastore Configs: Configs required for the datastore that is being used which is mostly Mongo in our use case. In case you’re using MySQL, Redis, Elasticsearch etc, mention the required parameters in this category.

3. Module Configs: All other configs required for your module can be mentioned here.

Note: It is recommended to have most of these parameters deriving values from the environment file only. In some cases, they can also be hard-coded within the code. It is mandatory for every file/class within the project to use these parameters from these variables of the ***config.py file only.

Having said that:

1. Please ensure the folder structure is perfectly maintained.

2. Never ever check-in sensitive data like AWS keys, passwords, PIIs etc in the config file, always erase/mask/encrypt them before pushing it to github.

3. You can check this file for reference:

The Tokenizer submodule in Anuvaad is designed to break down paragraphs into sentences or words, facilitating efficient preprocessing and accurate translations. This submodule is integral for preparing text data for subsequent processing and translation tasks.

Key Features

• Paragraph to Sentence Tokenization: Splits paragraphs into individual sentences, making the text easier to process.

• Sentence to Word Tokenization: Breaks down sentences into individual words for detailed analysis and translation.

• Document-Specific Handling: Manages document-specific symbols and special characters to ensure consistency and accuracy in tokenization.

• Flexible Integration: Can be invoked independently as a standalone service or as part of a larger workflow through the Workflow Manager.

Usage

The Tokenizer can be utilized in two main ways:

1. Independent Invocation:

   • As an independent service, the Tokenizer can be directly called to process text data. This is useful for isolated tasks where only tokenization is required.

2. Workflow Manager Integration:

Code Repository

You can find the code for the Tokenizer submodule in the Anuvaad repository at the following link:

API Contract

For detailed information about the API endpoints and their usage, refer to the API contract available at:

By employing the Tokenizer submodule, Anuvaad ensures that text data is meticulously prepared, contributing to the overall accuracy and efficiency of the document processing and translation workflow.

This microservice is intended to generate the final document after translation and digitization. This currently supports pdf, txt, xlsx document generation.

• API Contract:

• Code:

Follow these steps to set up the Anuvaad Web Application on your local machine:

Frontend

1. Clone the Repository:

2. Navigate to the Project Directory:

In order to make an API call this is the host URL and all mentioned API should be called against the mentioned point.

This is the very first step, users should register at https://users.anuvaad.org. Anuvaad will send a verification email, please verify the email before start making any APIs calls. Clearly registration is a one time activity.

Account Creation

Users shall onboard themselves on Anuvaad via the link below: Registration:

Once a user reaches the Sign Up page, they have to fill in the required details as shown below:

Upon successful submission, an E-Mail will be sent to the registered ID with a verification link

Clicking the verification link will redirect the user to the login page as shown below

For security purposes, Anuvaad follows an OTP-based login mechanism. You will be asked for the E-Mail to which a one-time password will be sent.

Upon confirming the ID, you will be asked to opt for one of the authentication methods.

Everything discussed above is a one-time process and must be applicable only on initial login to the application. Going forward, you will be redirected to the OTP verification page upon successful login.

Upon providing the correct OTP, the user will reach the below landing page of Anuvaad and we are good to go!

Translate Sentence

The translate sentence feature enables the user to input a text and instantaneously get its translation in another language. To use it, simply click on the Translate Sentence option on the landing page of Anuvaad.

The user will have to select the source language to which he provides the input and the target language to which the text must be translated.

If an Indic language is selected as the source, by default Transliteration feature will be enabled, assisting the user to type in that particular language from the normal keyboard with ease.

Upon clicking submit, the translated sentence will be displayed along with the model used to perform the translation. Using this feature, a stakeholder can quickly check the accuracy of the translation performed by Anuvaad.

Digitize Document

Digitize Document feature helps to convert scanned documents into editable digital format by preserving the structure. This process recognizes text in scanned (non hand-written for now) documents and converts it into searchable text.

To perform a document digitization, click on the Digitize Document option on the landing page of Anuvaad. You will be greeted with a screen as below

User may select a document/image and choose the appropriate source language and then trigger the digitization process. A pop-up window appears which shows the progress of the ongoing process. This is an async job and will happen in the background. The time taken will be dependent on the nature of the uploaded file

The status of all tasks given by user to Anuvaad can be viewed on the as below

If the status of a job is completed, you can view the result and make changes by clicking on the view document icon, which is second in the last column under the label Action.

Users can make changes, if any, by double-clicking on the word. once done, the digitized document can be downloaded by clicking on the download button in the top right corner in the desired format.

Translate Document

The translate document feature enables the user to upload a document and get its translated version. The key highlight of the feature is that Anuvaad tries to maintain the original structure of the document in the best possible manner.

To perform a document translation, click on the Translate Document option on the landing page of Anuvaad. You will be greeted with a screen as below

Here, the user will have the provision to upload a document (pdf,docx,pptx formats are supported as of now) and select a source and target language to perform translation. On successful upload of a supported file, the process begins and status will be shown

There are various stages happening behind the scenes of document translation and the status will be displayed on screen. The total time taken to complete the process depends on the number of pages and the structure of the input document. The translation is an async process and once initiated it will be performed in the background. Users can keep on adding more and more tasks to Anuvaad meanwhile.

After a certain time, by going to the , the user can access the translated document.

Users can make necessary changes to the document using Anuvaad's easy editor which is developed by seeing document translators forefront, and later on download the translated document back to their system in the desired format.

The blue icon in the bottom right corner is to use the merge feature. In case two or more text blocks need to be combined into a single unit, it could be used.

Glossary Support

Being an AI-assisted translation software, some occurrences can happen where a machine translation can give meaningful, yet non-contextual output to certain words/phrases. In order to work around this, Anuvaad offers user-level Glossary support.

A translator can store certain phrases and their predefined translation so that if there is an occurrence of a similar phrase in the document for translation, the system acts wisely based on the predefined criterions.

The list of default Glossaries and added ones can be viewed on page.

To delete a glossary, simply click on the bin icon under the Action column. To add an extra item, click on the button in the top right corner. You will be redirected to the following page where new words/phrases and their corresponding translation can be added. Once submitted, the new items will be visible on page.

Analytics

Anuvaad also offers a built-in Analytics feature to keep track of usage metrics. These Analytics are instance-specific. Bar graph-based representations offer quick insights into how well Anuvaad is utilized. This also helps stakeholders to keep track of the number of Documents processed, Languages used, sentences translated, and organization-level information. Furthermore, these data could be exported in the desired format for future reference.

The project Anuvaad repository serves as the primary codebase for the Anuvaad project, aimed at facilitating document processing and translation tasks efficiently.

Purpose of Folders

• anuvaad-api: Houses standalone APIs utilized within the project, such as login and analytics functionalities.

• anuvaad-fe: Contains frontend-related code, responsible for the user interface and interaction aspects of the application.

• chrome-extension: Hosts code relevant to the Anuvaad Chrome extension, offering additional features and integrations within the Chrome browser environment.

• anuvaad-nmt-inference [legacy]: Previously held legacy OpenNMT Python-based inference code. Deprecated and not actively utilized within the current project framework.

• anuvaad-etl: Comprises sub-modules dedicated to document processing tasks, enhancing the extraction, transformation, and loading capabilities within the Anuvaad ecosystem.

Microservice Structure

As an application, the Workflow Manager, in conjunction with independent APIs, forms the foundational architecture of Anuvaad. The Workflow Manager facilitates communication among various modules and orchestrates their interactions. However, Anuvaad's design accommodates diverse use cases, allowing each module to operate autonomously when necessary. For instance, the Tokenizer service can function independently to tokenize an Indic sentence without reliance on other modules.

Components of Each Microservice

Each microservice within Anuvaad adheres to a consistent structure, comprising the following common elements:

• Dockerfile: Provides instructions to build the individual microservice within a Docker container, ensuring portability and consistency across different environments.

• docs Folder: Contains documentation outlining the API contracts necessary for running and testing the module independently. This documentation serves as a reference for developers and users alike.

• config Folder: Stores module-specific configurations and secrets required for the proper functioning of the microservice. Centralizing configuration management simplifies deployment and maintenance tasks.

Overview

This document provides details about the translator service used in Anuvaad. Translator is a wrapper over the NMT and is used to send sentence by sentence to NMT for translation of the document.

Getting Started

Translator receives input from the tokeniser module, the input is a JSON file that contains tokenised sentences. These tokenised sentences are extracted from the JSON file and then sent to NMT over kafka for translation. NMT expects a batch of ‘n’ sentences in one request, Translator created ‘m’ no of batches of ‘n’ sentences each and pushes to the NMT input topic. In parallel it also listens to the NMT’s output topic to recieve the translation of the batches sent. Once, all the ‘m’ no of batches are received back from the NMT, the translation of the document is marked complete.

Next, Translator appends these translations back to the JSON file received from Tokeniser, The whole of this JSON which is now enriched with translations against every sentence is pushed to Content Handler via API, Content Handler then stores these translations.

TMX:

TMX is the system translation memory, A user can decide to override Anuvaad’s translation of a text/sentence by inserting ‘preferred translations’ into the system. TMX is backed by a redis store which hashes and stores user-specific translations for a text. This can be called as user’s personal cache of translations.

TMX provides three levels of cache-ing: Global, Org, User.

Global Level:

This is a global bucket of preferred translations where an ADMIN or a Global Level User can feed translations which will applied across users and orgs.

Org Level:

This a Org level bucket where Anuvaad translations are overridden by preferred translations to only those users who belong to a particular organisation. Any ADMIN or a Org Level user can feed these translations in to be overridden across users of his/her org.

User Level:

This a User level bucket where a User can feed in his/her preferred translations and the system will override Anuvaad translations only for this particular User.

TMX can be uploaded sentence by sentence or in bulk, both APIs are supported.

Example:

UTM:

UTM is User Translation Memory, This is slightly different from TMX, here there are no levels, This is totally and plainly a translation cache. Here, the system remembers the user's translation and applies it automatically when it encounters the same sentence for the same user.

Let’s say, We have a sentence ‘S1’ in a document ‘D1’, Anuvaad’s translation of this sentence is ‘T1’. Let’s say that the user on encountering this, changes the translation ‘T1’ of sentence ‘S1’ to ‘T2’. Now, Anuvaad remembers this such that, in any document, say ‘D2’ in this case, whenever there’s ‘S1’ and NMT translates it to ‘T1’, Anuvaad automatically overrides the translation to ‘T2’. However, let’s say NMT got better with time and now translates ‘S1’ to ‘T3’, in this case, Anuvaad dosen’t override it because the user context was ‘S1’ —> ‘T1’ and not ‘S1’ —> ‘T3’.

Modules

Source Code:

Before making an API call, the application should call this API first to receive authorization tokens. Once an application has valid token the same can be used to make all subsequent calls, please note token can expire and hence it is good practice to validate the token.

Analytics

When Anuvaad is implemented at an organizational level, analytics is crucial for tracking usage and metrics. A dedicated module exists to serve this purpose.

Code Repository: Anuvaad Metrics

API Contract: Metrics API Contract

For every X hours, a cron job creates a CSV file, and analytics are drawn based on it for time-consuming computations. For other metrics, data is fetched directly from the database. The following analytics are currently available, with room for more metrics to be visualized:

1. Total Documents Translated, Language-wise
2. Organization-wise Sentences Translated
3. Organization-wise Dashboard

1. Reviewer Metrics

The NMT module is responsible for the translation of sentences. It can be invoked directly or via the Workflow Manager. The NMT module works in correlation with the ETL Translator to enhance translation efficiency based on previous translations or pre-provided glossary and TMX support (refer to other sections). The module supports batch inferencing and provides APIs that return model details for language and other dropdown menus.

Initial Version

In the early days of Anuvaad, OpenNMT-py based models trained on Anuvaad's proprietary data were used. These models were primarily focused on judicial content. The inference code for this initial version is available here: OpenNMT-py Inference Code.

Intermediary Version

With the collaboration between Anuvaad and , data from Anuvaad and other sources were used to publish the Samanantar paper (https://arxiv.org/abs/2104.05596). Using the Samanantar dataset, IndicTrans, a more general domain model, was trained. This model performed well for legal use cases, leading to the replacement of OpenNMT with . The IndicTrans-based inferencing code is available here: .

Current Version

As the Sunbird ecosystem developed, the need for hosting multiple ML models independently became resource-intensive. This led to the development of , a centralized platform for hosting models. Applications can now utilize models from Dhruva using APIs. In Dhruva, models are wrapped with NVIDIA Triton, facilitating a scalable architecture. The IndicTrans model was moved to Dhruva, and currently, models are invoked from Dhruva via wrapper APIs from the NMT module rather than using dedicated inference. The Dhruva-ported code is available here: .

Anuvaad follows the standard feature-master type of branching strategy for code maintenance. The releases happen through the master branch via release tags.

Branches

It is recommended to translate batches of sentences in order to get high throughput.

Overview

This document explains how Netflix Zuul is used as an API Gateway in Anuvaad to perform Authentication, Authorization and API redirection to all inbound API calls. Zuul is an Open Source Project.

Getting Started

Zuul is an API Gateway developed as an Open Source Project by Netflix. Zuul provides various features to abstract out some of the common operations of the system and provide a strong layer for Authentication, Authorization, API Pre & Post Hooks, API Throttling, Session monitoring and much more. Zuul: Zuul is an edge service that proxies requests to multiple backing services. It provides a unified “front door” to your system, which allows a browser, mobile app, or other user interface to consume services from multiple hosts without managing cross-origin resource sharing (CORS) and authentication for each one.

Zuul in Anuvaad

Zuul is Anuvaad is a config driven implementation where APIs, Roles, Role Actions are read by Zuul through a file stored in a remote repository.

Roles

The set of Roles defined in the system, these will be attached to the Users and also mapped with the APIs to provide role based access control (RBAC).

Actions

Set of APIs exposed in the system by various microservices. Each action is an API which will be mapped against the roles. APIs are of 2 types: Open APIs and Closed APIs. Open APIs can be accessed without authentication, in other words: these APIs are whitelisted. Closed APIs can only be accessed after auth checks.

role-actions:

Mapping between the roles and actions, Zuul uses this to decide if the User should be allowed to access a particular API. These configs can be found here:

Authentication in Anuvaad:

Anuvaad uses JWT auth tokens for authentication and authorization purposes. The same token is also used as the session ID. These tokens are generated and stored securely by a UMS system. Example:

Source Code

Anuvaad Zuul uses 3 Pre Filters namely: Correlation, Auth, Rbac. Correlation: Filter to add a correlation ID to the inbound request. Auth: Filter to perform Authentication check on the inbound request. Rbac: Filter to perform Authorization check on the inbound request. API redirection configuration is provided in the

This microservice is served with multiple APIs to transform the data in the file to form JSON file and download the translated files of type DOCX, PPTX, and HTML.

Modules

Workflow Code

WF_A_FTTKTR

Steps:

1. Transformation Flow

   • Use the data in DOCX, PPTX, or HTML file to create a JSON file.

2. Tokenizer Flow

   • Read the JSON file created in the Transformation Flow and tokenize each paragraph.

WF_S_FT

Steps:

1. WF_A_FTTKTR Flow

   • This flow must be completed before calling the download flow.

2. Download Flow

   • Fetch content for the file, replace original sentences with translated ones, and download the file.

Through WF Manager

Transform Flow

Mandatory Params for File Translator:

• Path

• Type

• Locale

Actions:

1. Validate input parameters.

2. Generate a JSON file from the data of the given file.

3. Convert the given file to HTML, PDF, and push it to S3 (For showing it on UI).

4. Get the S3 link of the converted file and call content handler API to store the link.

Download Flow

Mandatory Params for File Translator:

• Path

• Type

• Locale

Actions:

1. Validate input parameters.

2. Call fetch-content to get the translation of the file passed in the param.

3. Replace the original text in the file with the translated text.

4. Return the path of the translated file.

The production environment of Anuvaad runs on top of translation models trained in the general domain, which will cover a good amount of scenarios. However, in case we need a separate instance of Anuvaad to translate domain-specific data, like financial, biomedical, etc: the existing model must be finetuned with more relevant data in a particular domain to improve the accuracy of translation. This page briefly summarises how it could be done

Data Collection

Bi-lingual, or parallel dataset is required for training the model. It is nothing but the same sentence pair in both source and target language. Example:

Source(en): India is my country

Target(hi): भारत मेरा देश है

The more the amount of available data, the more accurately the model could be trained. In short, data collection could be done in one of the following three approaches

Manual Annotation by linguistic experts

This is exhaustive but the best approach. At least a small sample size of the dataset must be manually curated and used for validation purposes

Creation of Corpus by web and document crawling

Certain websites will have the same data in multiple languages. The idea is to somehow find matching pairs of sentences from them. Scraping could be done using frameworks such as and sentence matching could be done by using techniques such as . This method if used properly can produce huge amounts of data without much manual effort. however random manual verification is recommended to ensure data accuracy

A lot of sample crawlers for reference are available in this

To do sentence matching of scraped sentences, Anuvaad aligner also could be used, which is implemented using LaBSE. The specs for is available

Purchasing or using an open-sourced dataset

Very often, datasets will be made available by certain research institutes or private vendors. This data also could be included to increase the quantity of training data

Data cleaning & formatting

The raw data that is purchased or web-scraped might have too much noise which could affect the training accuracy and thereby translation. Noise includes unwanted characters, blank spaces, bullets and numbering,html tags etc.

The basic script for sentence alignment, cleaning and formatting is available

However, more rules for cleaning could be applied based on context and manual verification of raw data as per the scenario.

Model retraining

The present default model of Anuvaad is Indictrans. The instructions to retrain and benchmark an Indicrans model is explained

The training repo of legacy openNMT-py models is available

The architecture is around 2 major blocks :

• Document Digitization

• Document Translation

The Aligner module is designed for “aligning” or finding similar sentence pairs from two lists of sentences, preferably in different languages. The Aligner is a standalone service that cannot be accessed from the UI as of now. The service is dependent on the file uploader and workflow manager (WFM) services.

The Aligner service is based on Google’s LaBSE model and FB’s FAISS algorithm. It accepts two files as inputs, from which two lists of sentences are collected. LaBSE Embeddings are calculated for each of the sentences in the list. Cosine similarity between embeddings is calculated to find meaningfully similar sentence pairs. The FAISS algorithm is used to dramatically speed up the whole process.

Simplified implementation:

The service accepts two text files, and the aligner module can ideally be invoked using WFM. It is time-consuming and hence an async service. Once the run is fully done, a WFM-based search can be conducted using the job ID to obtain the result.

This microservice is used to extract text from a digital document in a structured format (paragraph, image, table), which is then used for translation purposes.

Architecture

• It takes an image or pdf as an input.

This microservice is served with multiple APIs to handle and manipulate the digitized data from anuvaad-gv-document-digitize, which is part of the Anuvaad system. This service is functionally similar to the Content Handler service but differs since the output document (digitized doc) structure varies.

Modules

OCR Document Modules

DigitalDocumentSave

API to save translated documents. The JSON request object is generated from anuvaad-gv-document-digitizer and later updated by tokenizer. This API is being used internally.

Mandatory parameters: files, record_id

Actions:

• Validating input params as per the policies

• The document to be saved is converted into blocks of pages

• Each block contains regions such as line, word, table, etc.

• Every block is created with UUID

DigitalDocumentUpdateWord

API to update the text in the digitized doc. RBAC enabled.

Mandatory parameters: words, record_id, region_id, word_id, updated_word

Actions:

• Validating input params as per the policies

• Looping over the regions to locate the word to be updated

• Updating the word and setting a flag save=True

DigitalDocumentGet

API to fetch back the document. RBAC enabled.

Mandatory parameters: record_id, start_page, end_page

Actions:

• Validating input params as per the policies

• Returning back the document as an array of pages

NMT Inference

This module provides the NMT based translation service for various Indic language pairs. Currently the NMT models are trained using OpenNMT-py framework version 1 and the model binaries are generated using ctranslate2 module provided for OpenNMT-py and the same is used to generate model predictions.

Data preparation

NMT requires parallel corpus between languages. Typically the size of language corpus is in the millions. The language corpus must have enough examples to cover various situations. This is one of the most important portions of the system and a very time consuming work where quality of data has to be checked to ensure the accuracy of translation. At Anuvaad, we have collected data for 11 languages as parallel corpora. The corpus is available under MIT license.

Training and Retraining

Training and retraining is a continuous process and training is dependent on the quality of input dataset. We have to constantly monitor the quality of translation. The translation mistakes should be used to generate training examples and retraining exercises have to periodically be taken up. The training cycle is a costly affair as they need GPU infrastructure and long training hours.

Model Evaluation

The model output is evaluated on the per-selected sentences and BLEU score is calculated. BLEU score provides a score that helps as the guidance to provide feedback on the model quality. Translation output has to be evaluated by human translators as well before it can be used in a production environment.

Architecture

Anuvaad uses the current state-of-the-art Transformer model to achieve target sentence prediction or translation Supporting code and paper is in open source domain,

We are leveraging an open-source project called “openNMT” and also exploring “FairSeq”(IndicTrans) from the perspective of enhancement and usage. The deeplearning platform used is pytorch

MODEL TRAINING

• Vocabulary or dictionary generation

  1. Tokenizer (Detokenizer) or breaking of given sentence in word or sub-word. (Language specific) Moses or IndicNLP(for indian languages)

  2. Sentence Piece or subword-nmt Supporting code and paper is in open source domain,

Approaches used:

• BPE (Byte Pair Encoding)

• Unigram

• Tune model parameters and hyper parameters to improve accuracy.

Modules

• Opennmt-py based

Training Repo

API Details

aai4b-nmt-inference (indicTrans)

• Fairseq based

Training Repo

API Details

Prerequisites

• python 3.6

• ubuntu 16.04

Install various python libraries as mentioned in requirements.txt file

APIs and Documentation

Run app.py to start the service with all the packages installed

For more information about api documentation, please check @

License

A Python package that provides standardized logging and error handling for the Anuvaad dataflow pipeline. This package serves features like session tracing, job tracing, error debugging, and troubleshooting.

Installation

Prerequisites:

• Python 3.7

Source code: GitHub Repository

Command:

Logging/Auditing

This part of the library provides features for logging by exposing the following functions:

Import file:

Functions

log_info

Logs INFO level information.

log_debug

Logs DEBUG level information.

log_error

Logs ERROR level information. Should be used for logical errors like “File is not valid”, “File format not accepted” etc.

log_exception

Logs EXCEPTION level information. Should be used in case of exceptions like “TypeError”, “KeyError” etc.

Notes

• In all the functions, message and input-object are mandatory.

• These functions build an object using these parameters and index them to Elasticsearch for easy tracing.

• Ensure all major functions have a log_info call, all exceptions have log_exception calls, and all logical errors have log_error calls.

Error Handling

This part of the library provides features for standardizing and indexing the error objects of the pipeline.

Import file:

Functions

post_error

Returns a standard error object for replying back to the client during a SYNC call and indexes the error to an error index.

post_error_wf

Constructs a standard error object which will be indexed to a different error index and PUSHES THE ERROR TO WFM internally.

Usage Notes:

• Use post_error_wf for flows triggered via Kafka or REST through WFM.

• Ensure both log functions and error functions are used in case of exceptions or errors.

• Errors are indexed to two different indexes: Error index and Audit Index.

• Use post_error_wf

Example Usage

Current Version

anuvaad-auditor==0.1.1 - Please use this version.

ANUVAAD DATAFLOW PIPELINE WORKFLOW MANAGER

Workflow Manager is the orchestrator for the entire dataflow pipeline.

Overview

Integrating applications first have to fetch supported language pairs (source and target language) along with respective translation model identifiers. These two parameters are mandatory before calling the translation API.

In Anuvaad All the service are talked through Workflow-manager by respective workflow configs defined for each service and communicating between these service are done by means of Kafka. Each services has its own functionality and are not dependent to predecessor services outputs. Two Main Flows:

1. Translation - Single/Block Translation or Document (.pdf , .docx , .pptx) Translation.

2. Digitization - OCR on (.pdf or Images).

This pipeline is used to extract text from a digital/scanned document. Lines and layouts (header, footer, paragraph, table, cell, image) are detected by a custom-trained Prima layout model and OCR is done using the Anuvaad OCR model.

Github repo:

API contract:

How to Use

Feature Branch name: user-mangement_feature API Contract: here

UMS is the initial Anuvaad module that facilitates user login and other account-related functionalities. It features admin level login and user level login. Only super Admin has the authority to create new organizations or add new users to the system (if not for sign-up). Admin can assign roles to the new users as well.

Modules

User Modules

CreateUsers

Whitelisted bulk API to create/register users in the system.

Mandatory params: userName, email, password, roles

Actions:

• Validating input params as per the policies

• Storing user entry in the database and assigning a unique id (userID)

• Triggering verification email

VerifyUsers

Whitelisted API to verify and complete the registration process on Anuvaad.

Mandatory params: userName, userID

Actions:

• Validating input params as per the policies

• Activating the user

• Triggering registration successful email

UserLogin

Whitelisted API for login.

Mandatory params: userName, password

Actions:

• Validating input params as per the policies

• Issuing auth token (JWT token)

• Activating user session

UserLogout

Whitelisted API for logging out.

Mandatory params: userName

Actions:

• Validating input params as per the policies

• Turning off user session

AuthTokenSearch

API to validate auth tokens and fetch back user details.

Mandatory params: token

Actions:

• Validating the token

• Returning user records matching the token only when the token is active

• Same API is used for verifying a token generated on forgot-password as well.

UpdateUsers

Bulk API to update user details, RBAC enabled.

Mandatory params: userID

Updatable fields: orgID, roles, models, email

Actions:

• Validating input params as per the policies

• Updating DB records

ForgotPassword

API for forgot password.

Mandatory params: userName

Actions:

• Validating input params as per the policies

• Generating reset password link and sending it via email

ResetPassword

API to update password, RBAC enabled.

Mandatory params: userName, password

Actions:

• Validating input params as per the policies

• Generating reset password link and sending it via email

Admin Modules

(Only Admin has access)

OnboardUsers

Bulk API to onboard users to the Anuvaad system.

Mandatory params: userName, email, password, roles

Actions:

• Validating input params as per the policies

• Storing user entry in the database and assigning a unique userID

• User account is verified and activated by default

SearchUsers

API for bulk search with pagination property.

Actions:

• Validating input params as per the policies

• All user records are returned if skip_pagination is set to True

• When no offset and limit are provided, default values are set as per configs

• Only the records matching the search values are returned if skip_pagination

ActivateDeactivateUser

API to update the activation status of a user.

Mandatory params: userName, is_active

Actions:

• Validating input params as per the policies

• Updating the user activation status

SearchRoles

API to fetch active roles in Anuvaad.

Actions:

• Returning active role codes

Organization Modules (Currently only ADMIN has access)

CreateOrganization: Bulk API to upsert organizations.

Mandatory params: code, active

Actions:

• Validating input params as per the policies

• Creating or deactivating orgs as per active status on request

SearchOrganization: API to get organization details.

Actions:

• If org_code is given, searches for that organization alone; otherwise, all organizations are returned.

Extension (for Anuvaad web extension)

GenerateIdToken: Generating token for web extension user.

Mandatory params: id_token

Actions:

• Decrypting and validating the token

• If the token is valid, register the user and return auth token

Notes

• Add APIs with Zuul if they need external access.

• Rebuild and deploy UMS whenever a new role is added with Zuul.

• Email ID used for system notifications: [email protected]

• Email templates are available .

Setup Tips

• Run the docker container.

• Initialize the DB by creating a Super-Admin account directly in the DB.

• Additional users can be added from the UI by logging into the super admin account.

How to Initialize UMS without UI?

1. Create an account (Admin is preferred) using the API anuvaad/user-mgmt/v1/users/create.

2. Get the verification token from the email (2nd last ID on the ‘verify now’ link) or the userID from the user table.

3. Complete the registration process by calling the anuvaad/user-mgmt/v1/users/verify-user API.

This microservice is served with multiple APIs to handle and retrieve the contents (final result) of files translated in the Anuvaad system.

Modules

Common Information

Some common info that is applicable to save, update operations on translations.

Workflow Code

• WF_S_TR and WF_S_TKTR: Changes the sentence structure, hence s0 pair needs to be updated.

• DP_WFLOW_S_C: Doesn't change the sentence structure, hence no need to update the s0 pair.

Sentence Keys

• s0_src: Source sentence extracted from the file.

• s0_tgt: Sentence translation from NMT.

• tgt: Translation updated by the user (User translation). (Source may vary if the user edits the input document, or else it keeps the same as s0_src).

File Content Modules

SaveFileContent

API to save translated documents. The JSON request object is generated from block-merger and later updated by tokenizer and translator. This API is used internally.

Mandatory parameters: userid, pages, record_id, src_lang, tgt_lang

Actions:

• Validating input parameters as per the policies.

• The document to be saved is converted into blocks.

• Block can be of type images, lines, text_blocks, etc.

• Every block is created with UUID.

GetFileContent

API to fetch back the documents. The response object would be an array of pages, with pagination enabled. RBAC enabled.

Mandatory parameters: start_page, end_page, record_id

Actions:

• Validating input parameters as per the policies.

• Fetching back the blocks as per the page number requested.

UpdateFileContent

API to update the block content; triggered on split, merge, re-translate operations. Used internally.

Mandatory parameters: record_id, user_id, blocks, workflowCode

Actions:

• Validating input parameters as per the policies.

• Updating the list of blocks.

SaveFileContentReferences

Internal API to store S3 link references to translated documents (on docx flow).

Mandatory parameters: job_id, file_link

Actions:

• Validating input parameters as per the policies.

• Storing records in the database.

GetFileContentReferences

API to fetch back the S3 link for docx files. RBAC enabled.

Mandatory parameters: job_ids

Actions:

• Validating input parameters as per the policies.

• Fetching back the data from the database.

Sentence Modules

SaveSentence

API to store user translations. RBAC enabled.

Mandatory parameters: sentences, workflowCode, user_id

Actions:

• Validating input parameters as per the policies.

• Updating the sentence blocks.

• Saved sentences are always updated with "save": true flag.

• Saved sentences are also saved in the Redis store for Sentence Memory.

FetchSentence

Bulk API to fetch back sentences. RBAC enabled.

Mandatory parameters: sentences, record_id, block_identifier, s_id

Actions:

• Validating input parameters as per the policies.

• Returning back an array of sentences searched for.

Key API contract: API Contract

Service and KT Video Links