I've been daily driving Emacs again for a few months as my sole editor. Most things have improved to the point that I'm actually really happy, but I've spent much of my free time over the past few weeks struggling (mostly unsuccessfully) with TRAMP.

The promise of TRAMP

TRAMP (Transparent Remote Access, Multiple Protocols) is a built-in Emacs package that provides an abstraction layer for dealing with remote "things" transparently. The theory of TRAMP is that the buffer should feel more or less the same as if it were local to your computer.

I say "things" because it's not just limited to files. TRAMP is far more sophisticated than merely "opening" a file by scping a copy down and then sending it back when you save. Directory browsing, project-wide search, LSP integration, and even a shell all work exactly as if they were on your local machine!

TRAMP is, in my opinion, the coolest feature of Emacs. The idea is brilliant, and until very recently, it was the best "remote development" experience that I had ever seen. (I think that Zed and perhaps VS Code have a slight edge for SSH-based remote development now, but using those means you have to give up Emacs and the massive package ecosystem. You can't build Magit or org-mode as a Zed plugin currently.)

The design is also pretty novel. The buffers sit in regular Emacs windows (no weird "we'll replace everything with special, janky, less feature-ful 'remote' windows like JetBrains"). Almost all of your mode logic remains local (you don't need to install your modes on the remote system, or wait for a process over there to figure out the syntax table for a TOML file). And things that should be remote, like compilation, LSP, etc. all run on the remote.

And finally, that "multiple protocols" bit... Every other remote development mechanism I'm aware of has exactly one protocol: SSH (okay, maybe 3 if you count SFTP and SCP). TRAMP also comes with rsync (flexibility in file transfer mode); docker and podman (if you're cursed to work with containers during application development, this will help); su, sudo, doas, etc. (edit files as another user from the same session!), or even adb to connect to an Android phone.

You can even chain multiple protocols together! For example, chaining multiple SSH hops together for ad-hoc jumps, or an ssh chained to a sudo to edit a system configuration as root!

Brilliant, right? Well, it would be if it weren't janky as hell out of the box with the ssh protocol.

Making TRAMP go brr...

TRAMP's built-in ssh (or scp or similar) are fine if you just need to jump into a server to tweak a config real quick. But Emacs is a rich application platform, not a mere text editor. We have amazing tools like Magit, Projectile, eglot, and Flycheck.

Unfortunately, the ssh protocol totally falls over once you introduce these. Magit, for example, would take at least 3 seconds to build its overview, and staging a hunk took 5 seconds even on a relatively small repository.

In my frustrated wanderings around the Internet, I've seen many suggestions that offer some relief. But none of them made Magit feel anything like it does locally.

The best post I found suggested a bunch of tuning options (which DID have a positive impact), and using the new-ish direct async process feature of TRAMP. The latter caused significant trouble. By the time I got it working with my modest package set, I had over a hundred lines of hacks in my config just to get things working again.

That post alos concluded that if you want to work with Magit over TRAMP, you should probably just not use the overview, and that running shell commands was probably faster. This was depressing. I think that the git CLI is actually terrible (fight me), and that the Magit overview is amazing. In fact, I don't really see the point of Magit if I can't use this.

And it wasn't only speed either. Some things were janky enough to the point that I almost considered trying to run GUI Emacs on a remote system with X11 forwarding (this did not end well). I encountered far too many minibuffer messages yelling about "forbidden reentrant call of Tramp." This was not going very well...

It's hopeless...

I had several other seasoned Emacs users tell me that I was basically on a fool's errand, and should just give in to the standard "remote" Emacs setup using a terminal emulator, tmux, and mosh. But I can be extremely stubborn. I happen to think that windows (er... frames to Emacs users) are actually a great idea! And I like having a rich array of fonts, support for images, and other nice things, thank you!

Some of the reasons for all the jank are called out in excellent the article I linked above. Things like Magit, Projectile, and project-wide (rip)grep do a lot of file operations under the hood. Sometimes opening a Magit buffer will do over 100 file operations. Other flows get blocked easily because they rely on spawning lots of (usually short-lived) processes.

The reason TRAMP's ssh protocol implementation is a bad fit for "application" style workloads is because it runs everything over a shell pipe and parsing the output. This is slow and error-prone. The default synchronous mode of operation often introduces head-of-line blocking which can freeze the whole environment. And even in direct async mode, each operation is a round trip. Even if your latency is low, this becomes problematic.

Aside: I can sort of accept why a Magit stage operation might be slow, but I can't understand why something as basic as C-x f /ssh:myserver: will instantly create a 3-ish second pause. Even on the same LAN with a ping of 0.3ms! I don't understand how it is THAT slow getting the initial directory listing. This is literally instant if I do the same thing over plain terminal SSH, so TRAMP is adding some overhead in its ssh protocol implementation. If anyone can explain this to me, my curiosity is insatiable and you know where to find me!

Making TRAMP go 🚀

By this point I was pretty annoyed at how little I had to show for the amount of time I'd invested. I was more or less resigned to the fact that maybe remote development is still terrible. But I couldn't shake the feeling that, despite my limited knowledge of the TRAMP architecture, the flaw probably wasn't fatal.

As it turns out, I was not the first person to think of this. The author of TRAMP-RPC knows a bit more about this than me, and his solution neatly addresses all of the drawbacks of TRAMP's ssh implementation. It doesn't depend on shell parsing, and can batch multiple operations per request.

After I figured out how to get it to build on FreeBSD, suddenly everything just worked! Magit was instant. Ripgrep via C-x p g responded quickly to every keystroke, even in a large project. No more forbidden reentrant calls. And I just deleted those hundreds of lines of hacks from my Emacs config.

Now everything is fast enough that I honestly can't tell that I'm in a remote buffer. Bliss! The only tradeoff is that you do need to install a binary on the remote server. This is more or less automatic if you're connecting to Linux boxes. And I hope it will be for FreeBSD too soon!