Securing Confidential Data: Building a Fully Local AI Meeting Assistant

Hello, this is Tabuchi from the engineering team.
Recently, online meetings with team members, remote partners, and clients have become a daily staple of our workflow.

While increased communication is positive, the burden of creating meeting minutes has grown significantly. Documenting discussions, sharing them with stakeholders, and creating action lists is a time-consuming task that scales with meeting frequency.

While various external AI-powered services exist to assist with this, many companies, including ours, hesitate to use them due to security concerns—specifically, the risk of sending confidential data to external servers.

To address this, we have built a system that handles everything from recording and transcription to summarization and action item extraction entirely within our local machines. Today, I’d like to introduce the technical architecture and how we are using it.

System Architecture Overview

Our primary focus was on privacy and security. We designed a “Fully Local” architecture where no data is sent over the internet during the sensitive transcription and analysis phases.

System Diagram

System architecture

1. Cross-Platform Recording App (Node.js)

We developed a core recording application using Node.js. It is designed to be OS-agnostic, supporting Windows, macOS, and Linux to accommodate our diverse team environments.

The app runs stably in the background, capturing audio without interfering with the ongoing call.

2. Operation Support via Browser Extension

To streamline the process, we built a custom browser extension that integrates directly with meeting apps like Google Meet. The extension sends HTTP requests to the local recording app to start and stop the session with a single click.

Google Workspace (Standard plan and above) offers native recording features, but this tool provides similar convenience for any plan level.

Example of Google Meet with a custom extension adding a recording button.

3. Fast Transcription with Whisper CLI

Once a recording ends, Whisper CLI immediately starts the transcription process on the local machine. While it does consume some system resources, modern PCs can complete the task in the background without impacting other work.

4. Summarization and Action Item Extraction with Local LLM (Gemma)

The transcribed text is then passed to a local Large Language Model (LLM). Currently, we are seeing excellent results using lightweight yet powerful models like Gemma4. The AI generates a summary of who said what and extracts a clear list of action items.

Operations and Future Outlook

The final meeting minutes are automatically uploaded to a designated Google Drive folder, and a notification is sent via chat tools like Discord.

Our Stance on Google Drive Usage

From a strict perspective, uploading to Google Drive constitutes “sending data to an external server.” However, our company already utilizes Google Workspace for Business, managed under our corporate security policies.

The core of this project is avoiding “External AI services where the data usage policy is opaque (e.g., potential use for model training).” By completing the analysis process locally and only storing the final “output” in our standard corporate storage, we achieve an optimal balance between efficiency and security.

Critical Point: Prior Consent for Recording

Beyond the technical implementation, we prioritize “Prior Consent” in our operations. Before recording any online meeting, we make it a mandatory rule to inform all participants of the purpose and obtain their agreement. This ensures a transparent environment and maintains trust with all stakeholders.

Conclusion

For the challenge of “AI is useful, but data leakage is a concern,” an internal tool leveraging local LLMs and Whisper is a powerful solution. We will continue to explore technical approaches that maximize efficiency while upholding the highest security standards.

I hope this serves as a helpful reference for other engineers facing similar challenges!