Because I couldn't find any documentation on this at all, anywhere, here is how to make mailto: URLs open in Gmail in Fluid.app.

1. Open Mail.app. Open Preferences. On the "General" pane, set "Default email reader" to Gmail-Fluid.app.
2. In your Gmail Fluid.app instance, open Preferences. Open the "URL Handlers" pane.
3. Add a new handler by clicking the + button in the bottom left. Set URL scheme to mailto and set URL replacement to:
   https://mail.google.com/mail/?extsrc=mailto&url=mailto:
   Make sure to include the final colon.

Mailto links should start working immediately, so when you click on an email link on a website, it will open a compose window in your Fluid.app instance. This also provides a hint about how to use the URL handler feature in Fluid.app in general, though I haven’t tried anything else.</p>

Update: This changed in 1.7, I think, without any mention in the Changelog. Instead of following step 3, add a new pattern set in the URL Handler preference dialog. Change the pattern to match mailto:* and change the Javascript block to read:

function transform(inURLString) {
return 'https://mail.google.com/mail/?extsrc=mailto&url=' + inURLString;
}


And you're set!