Hands-Free, Multilingual, Noise-Tolerant Voice Data Capture for Field Operations

Background and Problem Statement

A large-scale TB screening camp processes 50–200 beneficiaries per day. Each beneficiary requires: registration (30+ fields), anthropometry, nurse examination, TB screening (10 questions), and diagnostic referral documentation. The current workflow is entirely keyboard-and-dropdown-based, placing significant data entry burden on Registration Officers and Nurses particularly under the time pressure of a camp setting.

Common field outcomes include: fields left blank and completed retrospectively (data quality degradation), errors in name and age transcription from verbal communication, session slowdowns when typing in regional names, and Nurse attention divided between the beneficiary and the tablet screen. Voice-based data entry addresses all four simultaneously.

Current Challenges

• Typing on Android tablet in camp conditions (sunlight, noise, dust) is slow and error-prone.
• Names in tribal languages and regional dialects are frequently misspelled when typed by workers who are not native speakers.
• The nurse cannot maintain eye contact with the beneficiary while entering data.
• High camp volume (200 beneficiaries/day) means each registration has <5 minutes of field worker time.
• Offline constraint: cloud-based speech to text services (Google Speech, Azure) are not available without connectivity.

Proposed Solution

An on-device, offline-capable Speech-to-Text (STT) engine integrated into the Registration and Nurse workflows. The worker speaks and the system transcribes, maps to the correct form field, and pre-populates the value for worker review before saving. No data is committed without explicit worker confirmation.

To build an app localization feature for Indian languages that converts an English XML into multiple regional languages with industry-level accuracy and coverage, need is more than just translation rather the need of the hour is a structured pipeline with quality control, language rules, and tooling.

Key Functional Requirements

·       The STT engine shall support voice-to-field capture for: Name (First, Middle, Last), Age, Gender, Mobile Number, Symptoms (free text → mapped to dropdown options), Address / Village, Father's Name, Mother's Name, Occupation, Caste, Marital Status etc.

·       Supported languages for speech recognition: Hindi, English, Chhattisgarhi, Odia, Bengali and as per the requirement arises.

·       The STT engine shall include noise cancellation pre-processing suitable for outdoor camp environments: fan noise, crowd noise, wind etc.

·       The STT engine should have “Context-aware field mapping”: the system uses the currently active form field to constrain transcription output. Example: when the Age field is active, transcription is constrained to numerical output (0-99); when Gender is active, output is constrained to M/F/Transgender.

·       Confidence scoring: every transcription shall carry a confidence score (0–1.0). Scores below 0.75 shall display the transcription in amber with a warning: 'Low confidence- please verify'.

·       The STT engine should be optimized in Medical vocabulary ie the STT engine should be fine-tuned on a medical word list including TB symptom terminology, drug names (Rifampicin, Pyrazinamide, Bedaquiline, Isoniazid), comorbidity names (diabetes, HIV, anaemia), and tribal place names common in program geographies.

·       After transcription, the field worker reviews all pre-populated fields in a dedicated Review Screen before committing. Individual fields can be edited by typing. Confidence <0.75 fields should be highlighted and require explicit accept action.

Field Mapping Logic

The Field Mapper is the critical bridge between raw transcription and structured form data. The following mapping rules apply:

User Workflow

1. Registration Officer opens the Beneficiary Registration form.
2. A microphone icon is visible in the top bar of the form — tapping activates voice mode.
3. Voice mode activates: a waveform visualisation appears; the active field is highlighted with a blue border.
4. Registration Officer speaks: 'Ramesh Kumar Netam, forty-two saal, male, mobile number nine eight...'
5. After each pause (>800ms silence), the utterance is transcribed and the relevant fields are auto-populated.
6. Officer reviews the populated fields: green = high confidence (accepted); amber = low confidence (needs review).
7. Officer says 'Next field' to advance or taps the next field manually.
8. At form completion: a Review Screen shows all voice-filled fields with their confidence indicators. Officer accepts all, edits as needed, and taps Save.
9. Voice session log (fields captured by voice, confidence scores, edit events) is stored with the record for data quality audit.

Offline and Low-Connectivity Considerations

• The complete STT pipeline (audio capture, noise cancellation, acoustic model, field mapper) runs entirely on-device and offline mode.
• No audio data leaves the device, all processing is local.

Security and Privacy Considerations

• Audio is captured only when the worker explicitly activates voice mode (microphone icon tap or button press). No passive listening.
• Audio is processed in-memory and immediately discarded after transcription, no audio files are stored on device or transmitted.
• Voice session logs record only transcription outcomes and metadata, not raw audio.
• App requests microphone permission explicitly at first voice mode activation; permission can be revoked in device settings.

Expected Benefits and Impact

• 40–60% reduction in data entry time per beneficiary registration, based on comparable field STT deployments in public health settings.
• Reduction in transcription errors for names in regional languages- particularly important for Nikshay ID generation and beneficiary traceability.
• Nurse can maintain eye contact with beneficiary during history taking- improved clinical rapport and beneficiary experience.
• Camp efficiency increases: at 200 beneficiaries/day, a 2-minute-per-registration saving = 6+ hours of saved worker time daily.