LLM Based Voice Assistant
using ESP32
Mohammed Omama S 23MIS0010
Mohammad Nadeem A 23MIS0039
Adnan 23MIS0030
Field / Area of Project
- Embedded Systems
- Internet of Things (IoT)
- Voice Interaction Systems
- Edge Computing with Cloud Integration
- Artificial Intelligence (LLMs + TTS/STT APIs)
Background of the Invention
Existing virtual assistants like Alexa, Siri, or Google Assistant are proprietary, expensive,
and closed-source. They rely heavily on cloud infrastructure and specialized hardware,
which makes it difficult for students, researchers, and hobbyists to replicate or
experiment with such systems. Traditional microcontroller projects (Arduino/ESP32)
typically involve simple sensors and actuators, but they rarely integrate with state-of-the-
art Large Language Models (LLMs) for natural, human-like conversations.
The gap lies in creating a cost-effective, DIY-friendly, IoT-based conversational assistant
that:
1. Uses low-cost hardware (ESP32, microphone, speaker).
2. Integrates cloud-based LLMs (e.g., ChatGPT API) for intelligent responses.
3. Provides real-time voice interaction using Speech-to-Text (STT) and Text-to-Speech
(TTS).
4. Functions as a modular, open platform for IoT-based applications (e.g., smart homes,
educational tools, talking pets).
Discussion of Prior Art
Patent/Publication Description Relevance / Limitations
US20190118844A1 (2019) Smart speaker with cloud- Focuses on speech
based speech processing. recognition but lacks open-
source, low-cost DIY
implementation.
US10499754B2 (2019) System integrating sensors Relies on proprietary
with voice assistants for systems; not modular for
smart environments. educational use.
US20170186072A1 (2017) Outlines virtual assistant Cloud-dependent, no focus
with cloud-based on IoT-hobbyist scale
processing and mobile prototyping.
integration.
Summary of the Invention
The proposed system enables a low-cost, IoT-based conversational assistant using ESP32
as the core controller. It captures audio through a microphone, transmits it over WiFi to
cloud services for speech-to-text (STT) processing, sends the converted text to a Large
Language Model (LLM) for intelligent conversation, and then converts the response text
into speech (TTS) for playback through a speaker.
This modular design allows integration with IoT devices, meaning the assistant could be
extended to control lights, appliances, or even function as an interactive educational robot
or digital pet.
Novel aspects include:
- Low-cost, open-source conversational assistant using ESP32.
- Integration of LLM APIs (e.g., ChatGPT) with IoT hardware.
- Real-time speech input and audio output.
- Expandable framework for smart homes, mental health support, and hobbyist projects.
Objectives of the Project
1. To design and implement a low-cost, IoT-based conversational assistant using ESP32.
2. To integrate microphone input and speaker output with WiFi-enabled cloud
communication.
3. To utilize cloud-based Speech-to-Text (STT) and Text-to-Speech (TTS) services.
4. To connect with Large Language Models (LLMs) for intelligent conversational
responses.
5. To provide an extensible platform for future IoT applications (e.g., smart homes,
education, talking pet robots).
Potential Applications
Workplaces: Smart voice-enabled assistants that manage IoT devices (lighting,
temperature, reminders) for improved productivity and reduced fatigue.
Education: Personalized learning support, doubt-solving, and monitoring engagement
levels through conversational AI integrated with classroom IoT systems.
Healthcare: Voice-assisted therapy support for patients, medication reminders, and
integration with health IoT devices for real-time monitoring.
Smart Homes: Context-aware conversational assistant that controls appliances, lighting,
and entertainment systems based on user habits.
Vehicles: Voice assistant that interacts with in-car IoT systems to provide navigation,
stress-free driving aids, and personalized infotainment.
