Refuse to run under wayland

[Quaqqer]

I would like for picom to refuse to run under wayland. I faced a problem while running Gnome (Wayland) where picom broke xwayland. The effect was that no x-windows would show up. I suspect that the cause of this is that the autostart entry in /etc/xdg/autostart causes picom to start which messes with gnome.

Platform

NixOS 24.05 amd64

Environment

• Gnome Shell 46 with mutter/wayland
• GDM

picom version

v11

Diagnostics

Version: v11

Extensions:

• Shape: Yes
• RandR: Yes
• Present: Present

Misc:

• Use Overlay: No
  (Another compositor is already running)
• Config file used: None

Drivers (inaccurate):

Backend: glx

• Driver vendors:
• GLX: Mesa Project and SGI
• GL: AMD
• GL renderer: AMD Radeon RX 7800 XT (radeonsi, navi32, LLVM 17.0.6, DRM 3.57, 6.9.4)
• Accelerated: 1

Backend: egl

• Driver vendors:
• EGL: Mesa Project
• EGL driver: radeonsi
• GL: AMD
• GL renderer: AMD Radeon RX 7800 XT (radeonsi, navi32, LLVM 17.0.6, DRM 3.57, 6.9.4)

Steps of reproduction

1. Install Gnome, GDM, and picom
2. Start Gnome (wayland) from GDM
3. Try to open a x-program, xeyes for instance

Expected behavior

• Programs can open windows

Current Behavior

• No programs without native support for wayland open

[yshui]

Any reason why you still have picom installed when you are using wayland?

[Quaqqer]

It's mainly that I want Gnome alongside i3, if I need a more complete desktop environment. Although I can't speak for if the problem exists on other distros than NixOS, I feel like installing a package shouldn't break your desktop.

The main reason I want it to refuse to run under wayland is that it was really obscure to find out that picom was the culprit. Maybe refusing to run is a bit much, although I don't know of any reason to run under wayland. Warning the user would be sufficient.

[yshui]

I think picom isn't doing anything wrong here, in fact this sounds more like a gnome/mutter Xwayland support problem.

Also may I ask how did you determine picom was the culprit here?

[yshui]

And most importantly, I don't know if there even is a way to detect whether we are running under Xwayland or not. If you know, do tell.

[DarioDarko]

instead of
picom
put
[[ $XDG_SESSION_TYPE == "x11" ]] && picom
in your autostart file

that should prevent picom from launching in wayland

[jmanc3]

std::getenv("WAYLAND_DISPLAY") != nullptr is how I check if we're running in xwayland

[yshui]

@jmanc3 that only checks if you are running under wayland, which does often means you are also running under xwayland but is not a guarantee.

so, if we use that it would be a hack.

[yshui]

@DarioDarko yep I was also thinking along this line. problem is picom is started by xdg autostart, and I can't find a way to make it only start under a x11 session.

it is possible to use OnlyShowIn and NotShowIn to filter desktop environment, but that doesn't tell you if you are running under x11 or wayland.

[yshui]

I raised an issue to xdg-specs, hope they will give us something: https://gitlab.freedesktop.org/xdg/xdg-specs/-/issues/137