Each pass takes ten seconds. The roster has forty clients. You wouldn't call it a task worth automating. You'd call it Tuesday.
How it works
Three times by hand. Then it asks.
Nothing to record, describe, or configure. You just start, the way you always do.
-
You work.
Copy, switch windows, new folder, paste, return, next row. In Excel and Finder, Safari and Numbers, whatever you already use.
-
It notices.
A deterministic pattern finder spots the same sequence of actions on three different pieces of data. Grok confirms it is a real routine, not a coincidence. A card slides in from the right edge of the screen.
-
You say yes.
Grok writes a short plan from a fixed catalog of steps. You see the steps before anything runs. Click Do the rest and it finishes the remaining rows. Click Not now and it stays quiet.
The rhythm
It doesn't read your screen. It watches your hands.
A handful of signals is enough to see a loop. Plain typing is never recorded.
- Copies and pastes⌘C followed by a clipboard change, and ⌘V. The copied text is what tells passes apart.
- Command shortcuts⌘⇧N, ⌘S, ⌘E and a few others. Not the letters you type.
- ClicksResolved to the button, cell or field under the pointer, not to pixel coordinates.
- App switches and filesWhich app is in front, and new files landing in Downloads, Desktop or Documents.
Where it shows up
The work nobody automates, because it's only twenty minutes.
Each of these is a loop Handoff recognizes today. You do the first three. It builds the plan and does the rest.
-
Copy a client’s namenew folderpaste the name
A project manager setting up a workspace folder for every client on the roster. After the third folder: “Notice you’re creating folders from this list—want me to generate the rest?”
readCSV → forEach → createFolder Clients/{{row.first}} -
Copy an addressget directionstype the drive time back
A list of client sites and the minutes from the office. Handoff routes the remaining addresses itself with Apple Maps and writes a drive-times table next to the sheet.
readCSV → forEach → driveTime → appendCSV {{stem}}-drive-times.csv -
Open the weekly exportdrop, trim, sortsave a clean copy
The same cleanup on every sales CSV. The plan is checked against the file you produced by hand before it is offered, and it can run each time a new export lands.
readCSV → transformTable… → writeCSV {{stem}}-clean.csv -
Copy name and emailfill the formsubmit, next row
New hires into the HR form. Handoff fills the fields through the browser’s accessibility tree and asks you before every submit, because a submit cannot be undone.
readCSV → forEach → setValue Name, Email → ask → click Submit -
Download an invoicerename with the datemove it to Invoices
Filing by hand, three PDFs in a row. Afterwards it can file each new PDF as it arrives, and every move can be undone.
moveFile {{file}} → Invoices/{{today}}-{{stem}}.pdf -
Open a screenshotresize to 1280export a JPEG
Images for a blog post, one after another. The remaining screenshots are resized and converted without opening an editor.
resizeImage → convertImage → Web/{{stem}}.jpg
What it isn't
Nothing to set up. Nothing to describe.
-
Not a macro you record.
There is no record button. It does not replay your keystrokes; it rebuilds the task from what the steps meant.
-
Not an agent you prompt.
You don't describe the task. You've already shown it, by doing it three times.
-
Not a workflow you build.
Nothing to configure in advance. Keep the routine and let it run when a file lands, or throw it away when the batch is done.
It only speaks up about what you're doing right now, in a burst, and only after the third time.
Privacy and control
Yours, on your machine.
What it sees stays in a local database. What it does waits for your click.
- Local memory.Activity lives in a SQLite file on your Mac: fourteen days of actions, forty-eight hours of details, then gone. “Delete everything” wipes it.
- What reaches Grok.Only when a repeat is detected, and only that candidate: the sequence of actions plus the window titles, URLs and copied text that make up the three passes. Never your typing, never a screen recording.
- Never plain typing.Password fields, private browser windows and apps or sites you exclude produce nothing. Pause it from the menu bar at any time.
- Waits for you.Every run starts with the card and your click. Anything irreversible, like a submit button, asks first. Touch the keyboard or mouse during a run and it stops.
- Undo.Files it wrote, moved or renamed and rows it appended can be undone exactly, and it refuses to overwrite anything you edited afterwards.
- Your key.Plans are built with your own xAI API key, stored owner-only on disk. There is no Handoff account and no Handoff server.
Details
The questions people ask first.
- Which apps does it work with?
- It watches every app through macOS Accessibility, so there is no plugin per app. It reads the most from Finder, Safari, Excel, Numbers and Mail. The actions it can take come from a fixed catalog: folders and files, CSV and table transforms, images, drive times, filling and clicking in web pages, appending to Numbers or Excel, and running a Shortcut.
- How many times do I have to do something before it offers?
- Three. It needs the same sequence of actions with three different pieces of data moving through it. A few shapes, like cleaning a CSV or filling a form, are offered after two.
- What if it's wrong about the pattern?
- Click Not now, or just keep working; the card slides away and that suggestion is not repeated for a week. If a run goes wrong, touch the keyboard to stop it and use Undo for anything it wrote.
- Does it run in the background all day?
- It sits in the menu bar and watches while it is on, but it only speaks up during a burst, at most once every thirty seconds, and never while you are in a full-screen app. Pause it whenever you like.
- Does it need an internet connection?
- Detecting the pattern is local. Confirming it and writing the plan uses Grok, so that step needs a connection and your API key. Running the plan is local again, except drive times, which use Apple Maps.
Under the hood
Built in a weekend, at Carnegie Mellon.
- Swift, SwiftUI, AppKitA native Swift package. No Electron, no web view.
- Accessibility + FSEventsClicks resolve to real UI elements; new files are noticed as they land.
- Deterministic pattern finderFinds repeated windows of actions with distinct data. No model call until a candidate exists.
- Grok, fenced inStrict JSON schema, a whitelist of steps per pattern shape, validation and repair before anything runs.
Handoff started at HackCMU 2026, after one too many afternoons of copying things between windows. The code, the demo routines and the tests are on GitHub.
Do the first three. Hand off the rest.
Clone the repo, run scripts/build-app.sh, allow Accessibility, add a Grok key. Then go do something three times.