Skip to content

TouchPilot

Local-first Android AI agent runtime for safe, observable phone control.

TouchPilot turns an Android device into a host for an AI agent that you can see, audit, and revoke. Every action goes through a typed, permissioned tool. Sensitive operations require your explicit approval. Every decision is logged.

Get the app

See Install for APK download and sideload instructions.

What it does

  • Chat interface on the device — describe what you want, watch the agent work.
  • Native Android tools — open apps, tap, type, scroll, swipe, press back/home, navigate Settings panels, wait for UI to settle.
  • Screen observation via Android AccessibilityService — the agent reads real on-screen text and metadata, not screenshots alone.
  • Skills — readable, editable Markdown files that describe how a task should be performed, what tools it may use, and what "done" looks like.
  • Local-first routing — simple actions run on a deterministic local router with no network and no API key.
  • Optional local model — LiteRT-backed command routing on capable devices.
  • Optional cloud fallback — OpenAI-compatible providers, with the API key stored in an Android Keystore-encrypted preference.
  • Risk-aware approvals — low-risk observation tools run without prompting; medium- and high-risk tools ask before they touch anything sensitive.
  • Full audit log — every model decision and every tool execution is persisted locally and exportable.

Why TouchPilot

Most mobile agents treat the phone like an API client. TouchPilot treats it like a workspace:

  • Visible. A pilot, not a hidden daemon — you see it in the notification shade, you see every action it proposes.
  • Explicit. No surprise taps, no background automation, no silent sharing.
  • Narrow. Skills declare which tools they may use. Tools enforce risk levels. Policy decides what needs approval.
  • Reversible. Most actions are easy to undo; the rest require consent.
  • Logged. Decisions, redacted arguments, and outcomes live in a local log you can copy out.

Current scope

The 1.0 release ships:

  • A local Android agent foundation (chat, screen context, tool execution).
  • A hybrid local AI core (deterministic router + LiteRT model + cloud fallback).
  • A Skills v2 registry with bundled skills for browser, settings, messages, Wi-Fi, volume, launcher, clipboard, and share.
  • A bounded multi-step agent loop with risk-aware approval UX.
  • Central policy decisions, app-aware rules, risk-specific approval copy, and expanded redaction in logs and traces.

See Roadmap for what's planned after 1.0.

Architecture at a glance

flowchart LR
    User([User]) --> Chat[Chat UI]
    Chat --> Loop[Bounded agent loop]
    Loop --> Router{Local router}
    Loop --> LiteRT[LiteRT model]
    Loop --> Cloud[OpenAI-compatible fallback]
    Router --> Tools[Android tools]
    LiteRT --> Tools
    Cloud --> Tools
    Tools --> A11y[AccessibilityService]
    Tools --> Intent[Android intents]
    Loop --> Audit[Local audit log]
    User -. approves .-> Loop

For the full picture see Architecture and Code Structure.

Project status

TouchPilot is approaching the Milestone 15: 1.0 Release line item in the roadmap. The Android agent foundation, hybrid local AI core, Skills v2, and bounded multi-step loop are complete. Active work focuses on safety policy v2, documentation, and validation.

Reference projects

TouchPilot draws inspiration from:

  • OpenClaw — sessions, skills, channels, and gateway design.
  • PicoClaw — lightweight runtime discipline and small-device deployment mindset.
  • MobileClaw — mobile GUI automation and chat-channel agent ideas.
  • ClawMobile — smartphone-native agent architecture and semantic Android control.
  • ExecuTorch — future on-device model inference.

Community

See Contributing if you want to help build it.