Audio portal

[GeorgesStavracas]

Right now, the almost ubiquitous way for sandboxed applications to access audio is through PulseAudio (even if the other end of the socket is PipeWire). On Flatpak, that translates to most apps having the pipewire socket always enabled as a static permission.

This is not ideal for multiple reasons: we're trying to move away from static permissions as much as possible; and this mechanism doesn't give much granularity over permissions to control audio.

We should discuss and explore introducing an Audio portal, that would operate much like the Camera portal, but focused on the various ways apps can use audio devices.

The D-Bus interface would look roughly like this:

<?xml version="1.0"?>
<node name="/" xmlns:doc="http://www.freedesktop.org/dbus/1.0/doc.dtd">
  <interface name="org.freedesktop.portal.Audio">

    <!--
        AccessAudio:
        @options: Vardict with optional further information
        @handle: Object path for the #org.freedesktop.portal.Request object representing this call

        Request to gain access to audio devices.

        Supported keys in the @options vardict include:
        <variablelist>
          <varlistentry>
            <term>sources as</term>
            <listitem><para>
              A list of audio sources to request access to. Acceped values are:

              <simplelist>
                <member>'microphone'</member>
                <member>'speakers'</member>
                <member>'camera'</member>
                <member>'applications'</member>
              </simplelist>

            </para></listitem>
          </varlistentry>
          <varlistentry>
            <term>handle_token s</term>
            <listitem><para>
              A string that will be used as the last element of the @handle. Must be a valid
              object path element. See the #org.freedesktop.portal.Request documentation for
              more information about the @handle.
            </para></listitem>
          </varlistentry>
        </variablelist>

        Following the #org.freedesktop.portal.Request::Response signal, if
        granted, org.freedesktop.portal.Audio.OpenPipeWireRemote() can be used to
        open a PipeWire remote.
    -->
    <method name="AccessAudio">
      <arg type="a{sv}" name="options" direction="in"/>
      <arg type="o" name="handle" direction="out"/>
    </method>

    <!--
        OpenPipeWireRemote:
        @options: Vardict with optional further information
        @fd: File descriptor of an open PipeWire remote.

        Open a file descriptor to the PipeWire remote where the audio nodes
        are available. The file descriptor should be used to create a
        <classname>pw_core</classname> object, by using
        <function>pw_context_connect_fd</function>.

        This method will only succeed if the application already has permission
        to access camera devices.

        Supported keys in the @options vardict include:
        <variablelist>
          <varlistentry>
            <term>sources as</term>
            <listitem><para>
              A list of audio sources to access. Acceped values are:

              <simplelist>
                <member>'microphone'</member>
                <member>'speakers'</member>
                <member>'camera'</member>
                <member>'applications'</member>
              </simplelist>

            </para></listitem>
          </varlistentry>
        </variablelist>
    -->
    <method name="OpenPipeWireRemote">
      <annotation name="org.gtk.GDBus.C.Name" value="open_pipewire_remote"/>
      <annotation name="org.gtk.GDBus.C.UnixFD" value="true"/>
      <arg type="a{sv}" name="options" direction="in"/>
      <arg type="h" name="fd" direction="out"/>
    </method>

    <property name="version" type="u" access="read"/>
  </interface>
</node>

Design Considerations

• Granting permissions for every app that wants to play some audio would be annoying, and that playing audio is not really security sensitive (at most, a malicious app can only be annoying), permission to output audio ("speakers") should probably be given without popping a dialog. Given that this is still a permission, people will be able to revoke it in e.g. GNOME Settings or Flatseal. The important bit is not to be annoying, but still use a permission to control access to it.
• Not sure if 'applications' should be in this list, or in a potential new Media Sharing portal. We can probably leave it out for now.

---

CC @hfiguiere @matthiasclasen @jadahl

[matthiasclasen]

One question here is: does it make to keep the camera separate?

Or could this just be an 'Pipewire access' portal?

I seem to remember that people have asked for screencast+audio before

[GeorgesStavracas]

I think the screencast+audio case would be entirely contained within the Screencast portal. Apps request a screencast with audio, get a PipeWire remote, and connect to whatever streams are available there.

@Obsessee per-device control sounds like a policy to implement on the media session (WirePlumber) level, not on portal level.

[Mikenux]

Since it's just audio, why not group camera and microphone as a request for audio input?

[GeorgesStavracas]

Since it's just audio, why not group camera and microphone as a request for audio input?

Sure

[Obsessee]

Differentiating between switches for input and output would be really terrific. I'd like to disallow badly behaving apps from capturing my microphone for voice input when I'm just using it to play music. Would especially help for fat fingering buttons on a Linux phone.

I feel like bringing this up now before there's an implementation because I don't think it'd be considered a big enough issue to revise the portal later down the line.

Not sure if 'applications' should be in this list, or in a potential new Media Sharing portal. We can probably leave it out for now.

I think application audio would probably be better fit for a Media Sharing portal, as it's not a (physical) device.

[GeorgesStavracas]

Another device category I did not consider: MIDI devices. They're managed by PipeWire as well.

[hfiguiere]

PulseAudio access is done with --socket=pulseaudio, which seems to grant ALSA (including MIDI), while other PipeWire access including JACK is done through the --filesystem=xdg-run/pipewire-0 permission.

About the sources:

• microphone: this should be something like audio_inputs.
• speakers: this should be something like audio_outputs.
• camera: this should be something like video_inputs, or does that also include video_outputs?
• applications: I don't even know what that is.
• midi: (see Audio portal #1142 (comment)) the MIDI bus, so this name seems to be unambiguous.

Also how do you specify the control of the graph? Example Helvum. Or the DAW that want to control all the audio, but do not need the video. Shall we have a permission for fuller access or shall grant it like it is currently done with --filesystem=xdg-run/pipewire-0?

[tytan652]

Also how do you specify the control of the graph? Example Helvum. Or the DAW that want to control all the audio, but do not need the video. Shall we have a permission for fuller access or shall grant it like it is currently done with --filesystem=xdg-run/pipewire-0?

I think apps like Helvum or qpwgraph falls under a category of apps that requires host PipeWire full access, so maybe those are out of scope for Portals, so they may have an exception to have --filesystem=xdg-run/pipewire-0.

For DAWs if they want more than what portal can simply provide (beside maybe needing their own audio PW server in the sandbox), it may mean that they are doing something wrong ? I never used a DAW before.

[Mikenux]

For DAW, things should be made such that they can built their own graph (which is about routing audio from what I understood) with the "devices" they have permission to access.

Also, I see that Audacity, with ALSA, let the user choose "devices" which correspond to physical computer ports. The questions are then:

• Does it make sense to allow apps to be set up without the devices being connected?
• Are there cases where it is possible, e.g., to have multiple speakers connected to the computer but using them individually (i.e. without a surround setup, controlling their volume individually)? Same for input devices.

[jadahl]

This sounds a bit like the devices portal that has been discussed as a "replacement" for the camera portal. Perhaps a less generic name is needed than "devices".

A "audio" portal that doesn't involve cameras could work too I guess, but it should ideally be easy to open a PipeWire remote that gets you both microphone and camera streams.

The "OpenPipeWireRemote" you add here adds different sources that would be exposed in the remote, including "camera", is that intentional?

microphone: this should be something like audio_inputs.
speakers: this should be something like audio_outputs.
applications: I don't even know what that is.

The point here is to create some predictable expectations. If an app gets "microphone" access, it shouldn't be able to "eaves drop" on every app's PipeWire source. In other words, applications can be audio input, but that doesn't mean it's a "microphone".

[matthiasclasen]

The point here is to create some predictable expectations. If an app gets "microphone" access, it shouldn't be able to "eaves drop" on every app's PipeWire source. In other words, applications can be audio input, but that doesn't mean it's a "microphone".

Yes indeed.

[GeorgesStavracas]

We seem to cyclicly float between "specific portals for independent categories of devices", and "generic portal covering pretty much all that PipeWire covers". I can see pros and cons for both approaches, and I haven't really made my mind about it.

I like the elegance and explicit semantic of specific portals. Want a camera? Camera portal; need audio? Audio portal; etc. But the downside is that we have to copy-paste an almost identical API in each one of them (AccessSomething, OpenPipeWireRemote). We also force apps to open different PipeWire remotes for each class of devices they need access to. It may or may not be a problem.

A catch-all PipeWire portal ("Devices" portal) would solve the multiple remotes problem, apps would be able to request multiple classes of devices in a single call (AccessDevices(['camera', 'speakers', 'microphone']), etc). But I wonder if we wouldn't be putting this portal in (ultimately) a dead end... what if we later find out that we need different options and flows for different devices?

I don't have a conclusive answer here, but I'm slightly happier with separate portals, even if that means apps having to open multiple PipeWire remotes. We can encode domain-specific knowledge in these portals - like the discussion above about audio device types, which wouldn't apply to camera video.

[jadahl]

I think a question that needs answering is how easy it is to turn multiple PipeWire remotes, and e.g. showing them into a single Gstreamer pipeline that spits out a video with video from a camera and audio from a microphone. If that is awkward and requires complicated plumbing, then I think our only reasonable choice is to go a "common" route, but if it's relatively trivial, our options gets more interesting, and we can explore per-device-class portals more freely.

[GeorgesStavracas]

As far as I've experimented in my Liveblast app, it's relatively trivial. You just create a gstpipewiresrc element with different file descriptors and nodes, and plug them together in the pipeline.

[ghost]

The Audio portal is still being discussed ?
In Flatseal I see that there is a Microphone toggle under the portals category.

[GeorgesStavracas]

There hasn't been any activity here, which means no discussions. The Flatseal toggle is not relevant to this discussion. Please do not write comments asking for updates, it is not helpful for the discussion.

[Be-ing]

How would this interact with the JACK, PulseAudio, and ALSA APIs? Would applications need to use PipeWire's APIs for audio I/O? If so, it will be an uphill battle getting applications to use the portal instead of continuing to use --socket=pulseaudio for PulseAudio (and ALSA... not sure why those are coupled 🤷 ) and --filesystem=xdg-run/pipewire-0 for JACK. Current advise from PipeWire upstream is for applications to continue using the JACK, PulseAudio, and ALSA APIs.

[tytan652]

How would this interact with the JACK, PulseAudio, and ALSA APIs?

None of them is compatible with a sandbox context this is why everything is converging toward PipeWire.
--socket=pulseaudio and --filesystem=xdg-run/pipewire-0 will end up being deprecated or made as exceptional as a per app basis by Flathub and/or Flatpak (it will take a while but it will happen).

This advise dates from 2020. PipeWire 1.0 is out since a little while, I think avoiding PipeWire API for this reason is no longer valid.

[Be-ing]

This advise dates from 2020. PipeWire 1.0 is out since a little while

AFAIK the advise is still current. That wiki page was last updated 4 months ago.

I think avoiding PipeWire API for this reason is no longer valid.

For what reason? Just because PipeWire 1.0 was released? The PipeWire 1.0 release was not accompanied by a recommendation from PipeWire upstream for applications to use the PipeWire API for audio if the JACK or PulseAudio APIs have been working for them already.

I do think a case could be made for suggesting application developers to switch to the PipeWire APIs for audio if it is required for fine grained Flatpak sandboxing with a new portal. I think this would need buy-in from PipeWire upstream though, and if there's a way to implement it without requiring existing JACK and PulseAudio applications to add new code, I think that would be preferable. @wtay what are your thoughts on this?

[tytan652]

and if there's a way to implement it without requiring existing JACK and PulseAudio applications to add new code

We can't avoid that, there is limit…
Maybe third-parties will create shims for JACK/PulseAudio for sandboxes, but this is out of scope of xdg-desktop-portal in my opinion.

[Be-ing]

this is out of scope of xdg-desktop-portal in my opinion

It needs to be considered if this is going to get uptake by application developers. It is one thing to add a bit more code to existing code to make a request to the portal. It is another to switch to a whole new API.

[tytan652]

ScreenCast and Camera portals are already existing example of API switch.

• OBS Studio does the switch progressively (waiting for new portals to be able to replace other features)
• Firefox allows to use the portal rather than V4L2
• Solution like xwaylandvideobridge (from third-parties) appeared but it is not perfect at all and is really a short-term solution, so asking upstream to switch API is preferred.

[Obsessee]

Now that the Pipewire Security Context is landing, would it technically be possible to start work on this, or is there more work needed for this to fall into place?

[tytan652]

I think it's unrelated, Portals mainly relies on PipeWire core created by the host server and given (with specific permissions) to the application.

[swick]

That was specifically created for flatpak and to be able to mount a pipewire socket to the usual location in the sandbox by default. The portal APIs would simply change what is visible on that socket.

[tytan652]

The actual APIs made the app only see what was requested per core which avoid to have to filter between type.

And now you are saying that you create a possibility of applications miss-filter and propose to the wrong choice (with all portals kind of nodes)…

That was specifically created for flatpak

xdg-desktop-portal should/must work the mostly the same between host apps and sandboxed apps, this new thing does not look like it would allow that…

[tytan652]

Maybe the fact that I found PipeWire documentation incomplete (having to read code all the time is a bad sign) gives me a bad opinion on exposing a contextual socket (with various kind of nodes) to an application.

[tytan652]

I try to restart discussion about audio source type so from I found here, we have:

microphone/audio_inputs

• Will show a permission prompt at the first access call (permission defaults to unset)
• Groups devices that sends audio to be recorded/sent by an application
• Must be revocable at any moment
• I think renaming it audio_inputs might lead to confusion when virtual speaker (App to App Media Sharing #1141) enter the equation.
  • Maybe I'm the only that found using input/output confusing, I forgot that the node may look like any microphone node

speakers/audio_outputs

• Will not show a permission prompt at the first access call (permission defaults to yes)
• Groups devices that receives audio from application (ignoring virtual microphone scenario)
• What about the monitor side ?, it must not be accessible or apps will be able to snoop audio from any apps sending audio to speakers.
• I think renaming it audio_outputs might lead to confusion when virtual microphone (App to App Media Sharing #1141) enter the equation.
  • Maybe I'm the only that found using input/output confusing, I forgot that the node may look like any speaker node

speakers-monitor ?

• Will show a permission prompt at the first access call (permission defaults to unset)
• Allows apps to access the monitor side of speakers devices
• Must be revocable at any moment

midi

• Will show a permission prompt at the first access call (permission defaults to unset)
• Like it was said "unambiguous", it will give access to MIDI devices.

camera

Device portal is no longer a thing and Cameras microphone falls under the microphone umbrella. Also video and audio are usually two separated devices.

I think this type is not needed at all.

applications

With #1141 in mind, any audio virtual devices might fall under the microphone and/or speaker umbrella (or even as ScreenCast audio). I think this type is not needed at all.

[Mikenux]

That's better understandable now, rather than you talk only about nodes. There is indeed no problem with that. My concern here is if the app indeed get, e.g., individual volume control for each speaker, if applicable?

Do you also think about more complex cases where e.g. are the inputs already linked to the outputs?

What do you mean ?

I mean if there are cases where input is routed to a specific output (custom config). (that's also a question)

[tytan652]

My concern here is if the app indeed get, e.g., individual volume control for each speaker, if applicable?

It depends on much PipeWire permissions we give between the node and the app, normally giving enough to interconnect nodes (device <-> application) should not allow volume control of the device.

I don't know how much we should give, I also don't want to make the UX over-complicated.

I mean if there are cases where input is routed to a specific output (custom config). (that's also a question)

You can connect multiple things to the same node, so this will be not an issue. (Sorry, if I didn't get it)

[Mikenux]

I opened a separate thread for sound control.

I mean if there are cases where input is routed to a specific output (custom config). (that's also a question)

You can connect multiple things to the same node, so this will be not an issue. (Sorry, if I didn't get it)

I don't know if you got it or not. Here is a perhaps unrealistic example (because I don't have such a case): on the system side, a user has defined that voice input from a microphone is to be listened to on a specific speaker (logically it would be on the app side, but let's imagine that it is on the system side; I hope that expresses the thing).

[tytan652]

a user has defined that voice input from a microphone is to be listened to on a specific speaker

Microphone will still be usable by other apps, there is no exclusivity.
Applications will normally not care or even see the connection between the two nodes (microphone and speaker).
This is not an issue.

[Mikenux]

I mean, if someone wants this configuration to be respected by apps.

[tytan652]

This thread is about how the flow portal should be:

Microphone, speaker, midi wording will be kept to avoid confusion. Wording of media roles is not part of this thread.

To took in account:

• Giving access to all microphone in one step is maybe not a good idea
• Default speaker access should not be complicated
• Speaker monitor access should require access prompt

If a new potential flow is proposed, I might add it in the following.

Full access to all device of the same type (actual proposition)

Like the Camera portal, the application get a PipeWire core with all Microphone/Speaker/Midi devices.
There is no portal implementation required (except Access portal).

Similar to the android way…

Fine-grained access

The android way has its limits and allowing some devices through a prompt may be better…

• The application gets a friction-less access (no UI/prompt) to the default Speaker (no monitor port)
• To access a microphone, midi device or another speaker, a prompt will be shown with a list of devices matching selected media roles.
• Restore session capability like ScreenCast and GlobalShortcut portals ?
  • Or something closer to the USB portal ?
• Enable modifing accessible nodes like GlobalShortcuts keybinds in the DE config panel ?

[Obsessee]

Disclaimer that I'm an end-user and thus have little idea of the technological possibilities and difficulties of each implementation.

I'm strongly in favour of a fine-grained access type of portal out of these two, and I think it's the general direction portals should head in for all device like portals. Allowing system default outputs (and probably also inputs) to be passed through by default would allow for fine access control, while avoiding the burdensome setup of an app that shows up permission dialogs for something as simple as entering a call.

Output monitors and cameras should probably always require a permission dialog upon first request.

[Mikenux]

@Obsessee: I don't think we will allow default input (the microphone interface on desktop computers, the microphone itself on laptops and phones) because it is privacy sensitive. We don't want apps to record (which is what you wanted in one of your comments), especially if it happens to be irrelevant to the app (at least if the user notices it is irrelevant).

---

• Restore session capability like ScreenCast and GlobalShortcut portals ?

  • Or something closer to the USB portal ?

Do you mean asking again to grant access versus not asking again? If so, I don't think anyone wants to be asked again.

• Enable modifing accessible nodes like GlobalShortcuts keybinds in the DE config panel ?

Yes. In the portal UI (app side), the user will see devices and (physical) interfaces. And we can't assume that an app will call a method that allows the user to modify them.

[Mikenux]

For session restore:

• Access "devices" for a one app session. This allows, for example, to prevent an app from automatically capturing the voice (if the microphone has been given) the next time the app is launched.
• Always restore (without asking), with automatic restoration of "devices" when connected (if not connected before launching the app). Users can choose to permanently give the "devices" if they want (but this does not protect against recording without consent when app is launched).
• Restore by asking by allowing to make a different selection.
• Here, restoration essentially in relation to recording. Need to think more about sound output.

[Mikenux]

Software Sound Control

• App sound volume level. Apps usually have a volume slider (e.g. a music player). The idea here is to allow apps to change their sound level. Given by default.
• Over-amplification. Is this risky for the device?
• Equalizer?
• Other things?

Hardware Sound Control

• Access to hardware sound control.

Are there any specific limitations that we want to apply if hardware volume control is available? (then, options to override limitations)

[tytan652]

Media player do have a volume slider but this is not the speaker volume, it's the application own volume slider (also accessible with tools like pavucontrol or some GNOME extensions) controling the volume sent by the app.
So the app volume is not related to the the device volume.

[Mikenux]

Of course, but these are still options to consider (at least if the app does not already have these features), since they are related to sound management, even if it is not hardware volume control.

That said, we can include hardware volume management.

[Mikenux]

I updated the main comment.

[Mikenux]

Earphones and Headsets (audio output + microphone)

• How to request headsets? (as one device, separating audio output from audio input)
• How to manage them? (e.g. Does switching between speakers and earphones/headset need to be reliable?)

[Mikenux]

For switching, I mean automatically switching between speakers and earphones, like if we plugged earphones into a smartphone. However, on desktop computers, I assume these devices remain connected to the system.

For this, the user can switch using system controls. However, this is probably limited to apps (possibly including fullscreen apps). For games, I suppose it is expected that the switch is handled from the game rather than the system.

So, I think we need to know whether we allow apps to get either speaker output AND earphones/headset output or just get a default output (where audio output switching is handled system-side, so even for games).

This is for basic cases. Are there any advanced cases (e.g. in pro audio?) where it is intended to produce sound (if that is even possible) in speakers and in a headset at the same time?

[tytan652]

For switching, I mean automatically switching between speakers and earphones, like if we plugged earphones into a smartphone. However, on desktop computers, I assume these devices remain connected to the system.

If it's default switching made by the PipeWire session manager, IIRC unless the application explicitly says it does not follow default when using the default, it happens. This is outside of the portal control and app control.

So, I think we need to know whether we allow apps to get either speaker output AND earphones/headset output or just get a default output (where audio output switching is handled system-side, so even for games).

It's been a while but I think I'm repeating myself, there is no distinction between device types beside being an output (speaker/sink) or an input (microphone/source). And I'm not saying that because I decided it was like this, ALSA will say there is those outputs (speaker/sink) and those inputs (microphone/source) to PipeWire and the concept of form factor is completely non-existent.

This is for basic cases. Are there any advanced cases (e.g. in pro audio?) where it is intended to produce sound (if that is even possible) in speakers and in a headset at the same time?

The user needs already to enable the Pro Audio mode of its audio device and then allow access to the device to the app.
Apps can use multiple speaker or microphone if the user allow them through the portal.

[Mikenux]

Of course, for automatic switching there is nothing to do in a portal.

For the rest, I'm talking about manual switching when the devices (speakers and earphones/headset) are permanently connected to the computer. The question is whether we default to manual switching (done via system controls by the user), unless, for example, we select a "Pro Audio" mode or not, while thinking that players would not be able to use this switching in-game. This may also be an option or this may possibly be made available only for fullscreen apps/games (if xdg-desktop-portal is able to detect this). The main thing here is to offer reliable switching from a public to a private output.

Note that even though PipeWire cannot distinguish device types, the user can and we can try to convey this in the UI.

[tytan652]

For the rest, I'm talking about manual switching when the devices (speakers and earphones/headset) are permanently connected to the computer. The question is whether we default to manual switching (done via system controls by the user)

Out of scope of the portal, we allow apps to see audio device nodes and apps do what they want with them, end of the story.

And it's up to the session manager to manage the scenarios where the user change devices through software like pavucontrol.
The session manager could check the given permission to allow the change to happen or not.

unless, for example, we select a "Pro Audio" mode or not, while thinking that players would not be able to use this switching in-game.

Switching device configuration (e.g. Stereo -> Pro Audio) is a edge-case in my opinion, you are changing the state of the device nodes. And IIRC, it's up to the session manager to deal with that.

This may also be an option or this may possibly be made available only for fullscreen apps/games (if xdg-desktop-portal is able to detect this). The main thing here is to offer reliable switching from a public to a private output.

Fullscreen game will have to deal portals UI/UX, if they only wants the default speaker there will be no UI, otherwise the portal can/will show a prompt.

Note that even though PipeWire cannot distinguish device types, the user can and we can try to convey this in the UI.

We can't, if PipeWire doesn't the portal doesn't too.

[Mikenux]

The main thing for an app is to either get the default output or select only one output device through the portal UI. In this case, no problem.

Switching between "public" and "private" output is only relevant if multiple devices are given to an app. In this case, the user, in the portal UI, will select multiple sound output devices, possibly including earphones/headset. Then, for example, the interface could inform the user of the functionality and ask them to select which sound output is headset/earphones (then the implementation returns it to the portal which are public and private outputs, and the portal informs PipeWire about it).

And I suggest this only in case user has to select multiple output devices. If so, having such functionality on the portal side allows for easy configuration for the user and having it cross-desktops (app developers also know they are limited with this cross-desktop, if it is relevant).