Challenges in Developing a Screenshot Application with Tauri in Rust

Fullscreen on macOS

If you are using "fullscreen": true on Windows or Linux, it might not be an issue. However, when it comes to the screenshot functionality, macOS tends to have unexpected behavior.

As we discussed earlier, a screenshot application is essentially a transparent, always-on-top, and fullscreen application. Being always-on-top works fine on macOS. Transparency on macOS requires enabling the "macOSPrivateApi": true property and the following features in the crate:

tauri = { version = "2", features = ["macos-private-api"] }

However, fullscreen cannot use the "fullscreen": true property. This is because fullscreen on macOS will automatically go fullscreen in a separate desktop space.

Instead, you should use "maximized": true. After the application loads, we should set the application to maximize:

  // Maximize the window after the page loads
  useEffect(() => {
    const maximizeWindow = async () => {
      const currentWindow = getCurrentWebviewWindow();
      await currentWindow.maximize();
    };
    maximizeWindow();
  }, []);

Copying Screenshots to Clipboard

First, the frontend has the capability to copy images to the clipboard, and this API should be supported by most modern browsers, as shown below:

const blob = new Blob([new Uint8Array(image).buffer], { type: "image/png" });
const clipboardItem = new ClipboardItem({ 'image/png': blob })
await navigator.clipboard.write([clipboardItem])

If you use the above API, you might encounter the following error:

NotAllowedError: The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission

The reason is that the frontend API needs to be in a secure context (HTTPS or localhost) and requires explicit user permission when called.

The second method is to use a clipboard plugin clipboard-manager:

pnpm tauri add clipboard-manager

import { Image as TauriImage } from "@tauri-apps/api/image";
const image: Uint8Array = await invoke("capture", {
    x: selectionRect.x,
    y: selectionRect.y,
    width: selectionRect.width,
    height: selectionRect.height
  });
// Write to clipboard
await writeImage(image);

However, on macOS, you might also encounter a bug:

thread 'tokio-runtime-worker' panicked at /Users/xxx/.cargo/registry/src/rsproxy.cn-e3de039b2554c837/arboard-3.6.1/src/platform/osx.rs:88:6:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

clipboard-manager depends on the arboard crate to implement clipboard functionality, and arboard calls the Cocoa framework’s API on macOS. Unfortunately, arboard directly uses unwrap here:

    let cg_image = unsafe {
        CGImageCreate(
            width,
            height,
            8,
            32,
            4 * width,
            Some(&colorspace),
            CGBitmapInfo::ByteOrderDefault | CGBitmapInfo(CGImageAlphaInfo::Last.0),
            Some(&provider),
            ptr::null_mut(),
            false,
            CGColorRenderingIntent::RenderingIntentDefault,
        )
    }
    .unwrap();

Currently, arboard has modified this line of code to return an Error instead.

So how do we know the specific reason for this bug? The answer is the console.app (also known as Console in Chinese). By opening the console, clicking to start logging, and then reproducing the operation that causes the bug, we can obtain the error log mentioned above.

If you directly write the binary image to the clipboard:

// image is binary file data
const img_data = await TauriImage.new(image, selectionRect.width, selectionRect.height)
 await writeImage(img_data);
 // Or directly writeImage(image);

You might find an error log like this in the console:

CGImageCreate: invalid image data size: 165 (height) x 836 (bytesPerRow) data provider size 27295

On Windows, this error will be displayed directly.

According to the official example, we might also encounter this error:

Unhandled Promise Rejection: expected RGBA image data, found raw bytes

In fact, going through all this, this is merely an issue with the official documentation. It is not mentioned in the plugin section, only in the javascript api^[1] documentation, where it states that Tauri needs to add image-ico or image-png features:

[dependencies]
tauri = { version = "...", features = ["...", "image-png"] }

Saving Screenshots

There is not much to say, just use the dialog to prompt for saving to a file and obtain the file path, then pass it to the Rust layer for saving.

import { save } from '@tauri-apps/plugin-dialog';

const path = await save({
      filters: [
        {
          name: 'screenshot',
          extensions: ['png', 'jpeg'],
        },
      ],
    });
    
    try {
      await invoke("capture", {
        x: selectionRect.x,
        y: selectionRect.y,
        width: selectionRect.width,
        height: selectionRect.height,
        savePath: path,
      });

      const currentWindow = getCurrentWebviewWindow();
      await currentWindow.close();
    } catch (error) {
      console.error("Error closing window:", error);
    }

Hotkeys

The screenshot application should not always be on top, so we need to hide the main window after taking a screenshot and call it back when needed. This requires implementing a hotkey wake-up function.

pnpm tauri add global-shortcut

Note that the above command will automatically add the following line of code in lib.rs:

.plugin(tauri_plugin_global_shortcut::Builder::new().build())

Since we need to implement the hotkey wake-up in the Rust layer, we need to remove this line and re-implement it. For example, if we want to implement a ctrl + shift + s (mac: command + shift + s) screenshot function, we can implement showing the main window in the response callback.

.setup(|app| {
            #[cfg(desktop)]
            {
                use tauri::Manager;
                use tauri_plugin_global_shortcut::{Code, GlobalShortcutExt, Modifiers, Shortcut};
                let ctrl_shift_s_shortcut =
                    Shortcut::new(Some(Modifiers::SUPER | Modifiers::SHIFT), Code::KeyS);

                app.handle().plugin(
                    tauri_plugin_global_shortcut::Builder::new()
                        .with_handler(move |app, shortcut, _event| {
                            let app_handle = app.clone(); // Clone handle for closure
                            println!("{:?}", shortcut);
                            if shortcut == &ctrl_shift_s_shortcut {
                                println!("Ctrl-Shift-S Detected!");
                                app_handle
                                    .get_webview_window("main")
                                    .unwrap()
                                    .show()
                                    .unwrap();
                            }
                        })
                        .build(),
                )?;

                app.global_shortcut().register(ctrl_shift_s_shortcut)?;
            }
            Ok(())
        })

Others

Why does the screenshot preview flicker in Tauri?

This is due to using a timer to call the Rust layer to take a screenshot of the mouse preview area. The larger the timer interval, the more flickering this preview feature will have. If we can optimize the overall time of this feature to under 20ms per frame, we will generally not see flickering.

From my observation, on my device, a 100×100 size preview screenshot takes about 5ms-15ms. The time taken for this screenshot plus the transmission delay should be sufficient. This is also one of the reasons why Tauri is not particularly advantageous for writing screenshot applications; native frameworks tend to perform better.

If you are interested, the complete code is available here:Click here^[2]

Reference Links

[1] javascript api: https://tauri.app/zh-cn/reference/javascript/api/namespaceimage/[2] Click here: https://github.com/ximeiorg/jietu