Marc Roelofs

HAL — Personal AI Assistant

A self-built AI agent running 24/7 on my own server

While the world debates AI assistants, I built one. HAL is a fully autonomous AI agent connected to my entire digital life — mail, calendar, tasks, news, smart home, notes, voice — running proactively in the background without being asked. It communicates through Signal, the messaging app I already use every day.

It started with a teenage obsession with this video.

Apple Knowledge Navigator, 1987 — watch on YouTube. A vision of a proactive AI assistant, nearly 40 years before it was technically possible.

A professor. A tablet. An AI that rescheduled meetings, found people, made calls — without being asked. That demo was fiction in 1987. I'm building the real version now, not as a prototype, but as a daily tool I depend on.

What makes HAL different

• Contextual understanding. HAL knows my profile, my projects, my preferences, and past conversations. When I say "do it now," it references prior agreements rather than asking what I mean.
• Long-term memory. Every significant conversation, decision, and preference is stored as a vector embedding in Qdrant. HAL can reference something discussed months ago — because it searches by meaning, not keywords.
• Real ecosystem integration. Two email inboxes, two calendars, task manager, note vault, smart home, RSS feeds — all connected through orchestrated workflows that understand the relationships between these tools.
• Personalised filtering that learns. The daily news briefing learns from which articles I mark as interesting. Email prioritisation adapts to my behaviour. HAL gives me the right things, not just lists.

What HAL does — proactively

What HAL does — on request

• Answer questions about agenda, tasks, email — across both Google and Outlook
• Create, update, or close tasks in Todoist; set reminders in Apple Reminders
• Control Home Assistant devices — lights, heating, automations
• Search long-term memory for past decisions, conversations, preferences
• Read, write, or append to Obsidian notes; create new documents in the vault
• Send emails via Gmail with full draft-and-send capability
• Summarise PDFs from Nextcloud storage
• Analyse images — send a photo via Signal, HAL describes or reasons about it
• Respond to voice — record on the iPhone action button, Whisper transcribes, HAL responds
• Generate LinkedIn and marketing content using brand voice guidelines from memory
• Search the web in real time for current news, scores, prices, schedules

How it works

Real examples

The server

HAL runs on a Lenovo ThinkCentre M720Q — a fanless mini PC the size of a thick paperback. Six-core i5-9400T, 16.6 GB RAM, running Ubuntu 24.04.4 LTS. Small enough to sit behind a monitor, powerful enough to run 20 Docker containers simultaneously.

All containers are managed via Portainer on top of Docker Engine 29.1.3. Six Docker Compose stacks organise everything by function:

• homelab — the HAL core: n8n, Home Assistant, Qdrant, signal-cli, Puppeteer, Mosquitto MQTT, MCP server
• nextcloud — file storage and notes sync: Nextcloud + MariaDB
• caddy — reverse proxy with automatic HTTPS, routes all services to clean URLs
• tailscale — zero-config VPN mesh, makes the server reachable from anywhere without exposing ports
• pihole — network-wide DNS resolver and ad blocker
• dashboard — internal nginx overview of all running services

Home Assistant runs alongside Mosquitto (MQTT broker for IoT) and a Matter server for Thread/Matter smart home protocol support. All device states and automations are accessible to HAL via the Home Assistant REST API.

Memory — how HAL remembers

Standard LLMs have no memory between conversations. HAL solves this with Qdrant, a vector database running locally on the server.

When something meaningful happens — a conversation, a decision, a preference — n8n converts it into a vector embedding: ~1500 numbers encoding the semantic meaning of that text. Qdrant stores them. When HAL needs context, it runs a similarity search: "what do I know that's semantically related to this?" — retrieving the most relevant memories in milliseconds, regardless of how they were phrased originally. Not keyword search. Meaning search.

Long-term notes live in Obsidian, synced via Nextcloud WebDAV. HAL can read, write and append to any note — including updating its own knowledge base mid-conversation.

The interface — why Signal

Signal is end-to-end encrypted, runs on every device, and has a well-documented REST API via signal-cli. No new app, no new habit. HAL lives where I already communicate. Inbound messages pass a whitelist check, attach relevant memory context, and go to Claude. For voice: an iPhone Shortcut records audio, Whisper transcribes it, HAL responds — identical to typing.

The AI layer — Claude

HAL uses Anthropic's Claude Sonnet 4.6. Claude interprets intent, decides which tools to invoke, handles multi-step tasks and composes the final response. For real-time information, Claude uses built-in web search rather than cached data. Most interactions cost fractions of a cent. The bottleneck is latency, not cost.

n8n workflows

Stack

By the numbers

Interested in how this works, or want to explore something similar for your organisation? Send me a message or find me on LinkedIn.