Inspiration
A sketch of a future
It's 2028. You turn off your screen and phone for the day and the real work begins. Your computer, phone, and glasses start shipping snapshots to your personal cloud and the H1000s in distant exa-scale compute centers get churning. The token stream is all-encompassing: 1st person video, audio, text you've typed and received, and paired mouse/keyboard plus screen data.
Your digital twin, named DigiClone, has already been trained on your entire digital footprint up to this point and only requires a small amount of fine-tuning (using the latest sample-efficient continual fine-tuning algorithm) to converge for the last day's data.
It's morning. You wake up and do your daily fine-tuning as you grab coffee.
(Italics denote human writing. Everything else is generated by the clone.)
Hypothetical 1, edit mode
Here, the model drafts initial reply to a fake message from a real coworker, John, and asks you to rewrite and explain why you changed what you did.
John: I'm getting this error when I run bigcorpcli command --all, \<long stack trace\>. Have you seen this before?
Initial response:
hm yeah I kind of remember seeing that previously in like April. Did you try pulling the latest main and rebuilding?
edited reply:
eh I kind of remember seeing that previously in April but I'm not sure. Did you try pulling the latest main and rebuilding?
justification for edits:
I'm honestly pretty uncertain about the answer to this, but the initial response sounded a bit overconfident. Also I'm trying to say “yeah” in fewer situations…
Hypothetical 2, open ended (via voice):
Here, the model is prompting you to actually fill in your reply and justify it without it drafting one ffirst.
Friend: hey dude, last minute, but are you free to grab dinner at 7 this Thursday?
You (digital): lemme check my calendar. No I already have something then
Friend: no worries, something important?
your reply:
uh nah just other plans. Hey I gotta run. Let's chat soon
justification:
you have a date you don't want to tell your friend about because last time you told them they told everyone
Hypothetical 3, open ended (text)
(A Q related to something you were reading yesterday)
What was the key insight that allowed continual low rank updates to quantized distilled models?
(answer based on your understanding of what you read)
Hypothetical 4, priority check
Here, the model provides you a list of messages/emails/texts you received yesterday and asks you to rate their priority in terms of a few priority categories you’ve set up, ranging from P0 (interrupt immediately) to P3 (fine for end-of-day).
The mix of work and personal questions are wide-ranging, but they’re all synthetic edge cases the model's generated to maximally reduce its uncertainty. The first one tunes the model through edits, whereas the others use a method based on completions that evolved from a mix of RLHF and standard supervised fine-tuning to learn from both actual text and your explanation.
You get ready and head out to the office. During your commute, you put on your glasses and start checking work messages. DigiClone has gone through a few hundred already and responded to a bunch, but there are a few where it's flagged that it thinks you need to reply yourself. You always wonder if the ones it flags are more likely to be written by an actual person on the other end but it's rude to ask too much. You go through them and DigiClone gives you a rundown of your calendar. You have one random internal meeting before lunch that you ask it to try and negotiate rescheduling. Right before you arrive at work, it gives you a notification that it's been moved to the afternoon.
Having pruned your message backlog and arrived at work, you sit down and open up your workspace. DigiClone presents you with a summary of where you left off yesterday before you signed off. You were debugging some weird behavior in a model your team owns by having 100 DigiClone copies mine for edge case examples, which you then work with one copy to dig into. You jump back into this and set yourself to "focus mode, P0 only". DigiClone continues to field inbound messages as you work, queueing up any messages it wants your feedback on for when you're done. None of them exceed the priority level you set, which is based on it prompting itself, "would I consider this a P0 to interrupt myself?"
Key idea
The key idea here is training a general, multimodal model to imitate you specifically rather than humanity in general. The digital twin wouldn’t be a perfect replica of you (yet), but it could be good enough to handle routine communication and tasks. It could learn and improve each day (overnight, probably not in real time unless we just have AGI) based on where its predicted actions diverged from your actual ones plus through new and improved versions of preference fine-tuning as depicted in the vignette.
Where might these systems work well vs. not as well assuming a continuation of current trends? Despite not learning in real time, these digital twins might be surprisingly good at navigating interpersonal dynamics for people with preexisting interaction history due to super human memory. On the other hand, they might struggle more with novel situations, long horizon tasks, and anything where learning in real time is necessary.
Why it’s probably not possible yet
Models probably aren’t smart enough to form the basis of autonomous digital twins today. Even with full penetration into our digital footprints, I doubt a GPT4-based tool could achieve the level of autonomy I described in the sketch. The multimodal features in particular seem out of reach for a GPT4V class model. To be clear, I remain bullish on tools like Lindy, but don’t expect them to achieve DigiClone status with current models alone.
This would all be prohibitively expensive today, especially since it relies on parameter-efficient fine-tuning being sufficient for not just text but a multimodal stream encompassing most of our daily inputs. I’m not even sure how well parameter-efficient techniques handle multimodal data right now (maybe decently well), but I suspect we’re at least a few OOMs of compute efficiency improvements away from being able to handle this. On the other hand, executives pay a lot for digital assistants! Would this cost more than 100k per year? I'm not sure without doing more inference arithmetic and projecting cost curves.
On top of that, continual learning using fine-tuning remains unsolved (as far as I know), let alone using parameter-efficient techniques. There are preliminary signs that scale helps a lot, but I don’t think anyone has evaluated whether that translates to such frequent (daily) fine-tuning. If this just worked, I’d expect to see OpenAI and other large model providers rolling out more continuous updates for their models, although cost explains the lower frequency equally as well.
Conventional wisdom and preliminary evidence both suggest that fine-tuning primarily locates pre-existing capabilities rather than instills new knowledge. For digital clones to work well, they will need to integrate new facts and ideally even learn new skills. I’m quite skeptical retrieval-augmented generation will be enough for this on its own because it won’t cover tacit knowledge, although it may be part of the solution.
Finally, there are major security issues with this, which is why RAG is so attractive for Enterprise tooling. If prompt injection doesn’t get solved, this could easily result in your digital twin leaking all your passwords and your social security number.
Why this might be closer than we think
Despite not being possible today, this vision may be closer than we think.
Distillation, transferring knowledge from larger, more capable models to smaller ones, is typically easier than starting from scratch. It may be surprisingly tractable to infuse a model with a person’s personality, idiosyncrasies, and even cognitive style. Additionally, as we integrate models with more tools and gather an abundance of preference and training data, we’ll have an increasingly large dataset we can use to bootstrap. From there, a flywheel effect could take over and incentivize power users to put in the work to improve their digital twins even more.
On top of that, parameter-efficient fine-tuning is improving quickly with LoRA and subsequent improvements enabling much cheaper fine-tuning at scale. If improvements continue, combined with compute improvements, it may become realistic to do the massive amount of fine-tuning this would require.
Assuming no massive disruptions to the industry, it seems likely we’ll have a model trained using 100x GPT4’s compute by 2028 (see here and here). While neither I nor anyone else knows what such a model will be capable of, if trends continue, it’ll be generally smarter, more coherent at longer horizon tasks, and even more efficient to fine-tune to act like specific people than GPT4 is. The former two together could provide enough capability to cross the inflection point where models become reliable enough to autonomously act in your stead and escalate when needed. The latter could help bridge the gap to making this cost effective. Under the conservative assumption that we have a model trained using 100x GPT4 compute by 2028, I put at least 50% chance on base model smartness not being the bottleneck.
How should you accelerate your personal trajectory given this future?
Maximize your token stream. Your own writing, your notes, your reading highlights, your dreams, your ideas, your thoughts in progress, put them all down and don’t lose them. Infuse the digital world with your essence. It doesn’t have to be social, it can all be in your favorite, local-first, encrypted app that never sees the light of day, but at least preserve it. You want your digital clone to know not just the current you, but the yous stretching back through time.
Practice delegating to people and machines. Eventually digital twins will probably be superhuman at understanding what you want anyway but there could be liminal period where your ability to work with a sub- or para-human digital assistant acts as major leverage for your effectiveness.
Acknowledgements
Matt Ritter, Willy C, Eryney Marrogi for feedback.