Musings on stream device and volume restore rules

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



'Twas brillig, and Lennart Poettering at 11/04/09 03:25 did gyre and gimble:
>> Currently module-stream-restore stores various metadata on streams it  
>> sees. This currently consists of:
>>  * volume
>>  * channel map
> 
> The channel map is only stored to make sense of the volume when
> restoring. It is not data that would be restored or could be
> restored. Channel maps is something applications have to decide about,
> not m-s-r.

OK, so these two can be collectively considered as "volume".

> Two more notes:
> 
> PA will only save volume/mute/device for streams where an explicit
> user-triggered volume change/mute change/device change was done. It
> won't create/changes rules for volume/mute/device changes that were
> done due to some automatic internal action, such as hot-unplug, ...
> 
> Also, if no rule is existant the stream volume is set to 100% relative
> to the current sink volume, the mute status is set to UNMUTED and the
> device is set to the default device that has been configured for the
> server or the first available sink (for whatever that means) of the server.

Noted.

>> Problems
>> ========
>>
>> The current system has a couple limitations.
>>  1. If a stream has a media role, it is impossible to save a volume for  
>> the specific application with that role as the key  
>> sink-input-by-media-role:xxxx will always take precidence over a  
>> sink-input-by-application-name:yyyy keyed rule.
> 
> No sure if that is a limitation or a feature.

Indeed. Personally, it smells like a limitation. I'm not sure I'd want 
to my two voip apps to have the same volume for example (they may do 
things internally with regards to levels etc, that mean I want to do 
things differently.

>>  2. If an application is modified to start providing new metadata (e.g.  
>> a role) the entries for that specific application that were used  
>> (sink-input-by-application-name) will no longer be matched and the data  
>> will be stale in the file. This is not really a major problem.
>>  3. The current system only allows for one device to be matched, not an  
>> ordered priority list.
> 
> Other limitations are:
> 
> It is a bit unclear whether it even makes sense to restore the volume
> on a different device than it was initially configured on. i.e. 100%
> on my hifi deck might be something completely different than 100% on
> my headset.

Very true.

> To remedy this we only restore the volume for rules where
> we also store the device. The result is that you can have rules for
> matching devices, and for matching devices+volumes, but not volumes
> alone. It's a bit weird. Sometimes you might not want to restore the
> device but still restore the volume if its the same device you end up
> using.

So would an application/device -> volume kinda table be wise here? It 
would store the volume of the app on that device? This isn't used for 
picking which device to restore the app to, that would be stored separately.

This has the arguably strange result that moving a stream to a different 
sink may change the volume of that stream which the user may not 
expect... e.g play some music quietly on USB and then pull the plug out 
and have it blare out noisily from the internal system as the volume 
jumps to max... (as that is what was stored previously).


> There is no sane way to say 'move *all* streams and rules to this new
> device'. Same for 'set *all* streams volume to x%".

I think there is a sane way to say 'move all streams to a new device' 
tho'. I don't think we should move the rules. If we store a app+device 
keyed volume table as mentioned above, we could use that info when we 
happen to restore app to device. e.g. that particular rule doesn't need 
to be moved.

The only rule we do need to change is the one that says "this app wants 
this device".

> We don't store any information about devices that are currently not
> available, except sometimes the name name as saved by m-d-r. But that
> we don't really have either. Icons, device descriptions, ... are never
> stored.

This could be changed tho' (and I think we do need this to create the 
kind of UI people will find useful).

> Also, there is no logic to move a stream to a newly plugged in device
> automatically. But there's some simple logic to move a stream away
> from a device that is being unplugged. Having this is highly
> desirable.

Again, I think this is fairly easy to implement... m-s-r (or another 
module) just needs to hook into the right events and re-process it's 
routing rules when a hotplug event occurs. Ultimately I think that this 
functionality will merge in the functionality of m-rescue-streams too.


> Also, sometimes a newly created audio streams might result in change
> of configuration. i.e. when a phone call stream is created it makes
> sense to switch a bt headset from a2dp to hsp mode, i.e. causing one
> sink to be replaced and an additional source to become available. That
> means we probably shouldn't be binding our rules to sinks/sources but
> to cards. Also, rules might trigger card profile changes.

Again, if the routing rules are re-run when a hotplug event occurs this 
shouldn't be a problem... I guess there could be circular loops created 
if rules trigger profile changes which causes a new sink which re-runs 
the rules which triggers a profile change which ..... etc. But I'm sure 
this can be protected against in some way.

> Then, some rules might be good to have as 'implicit'
> defaults. i.e. when the user didn't configure anything otherwise
> streams marked as 'phone call' should probably go to a device that is
> marked as 'headset' -- and not to the spdif output of the internal card.

Yeah that makes sense... I think this we have to show this somehow in 
the GUI tho' so users can see transparently what is going on.


> Finally, with the arrival of jack sensing things get even more
> complicated. i.e. music should go out via spdif when that is
> plugged in, but the bt headset otherwise with a last fallback of the
> internal speakers. Now, suddenly it's not enough to save/restore
> information about which device to pick, but about to which card
> to pick given a specific profile is set, or even wich card to pick and
> which profile to change to.

/me pretends not to notice this complication... ;)


> KDE users may be happy about that 'flexibility', but they are the ones
> who also think it is a good idea to ask for MySQL credentials when you
> start their media player. ;-) 

Not sure what you mean about that as I've never been asked for MySQL 
credentials using any KDE media player..., I can't really see it's 
relevant here either.

> People have requested a lot of things from PA, but not many have
> requested having such a publicly configurable priority list...

Well perhaps a priority list is not that important, but certainly some 
way to configure things with an active default nicely which restores 
itself and the streams etc. With this in mind a priority list in terms 
of UI presentation is a good solution to this implementation wise.

 From the feedback I've received, the actual layout and presentation of 
the KDE configuration UI is pretty good.

> Let's face it, 80% of all folks only have one audio device, the one
> built into their computer. And then there are a maybe 19% who might
> have one additional BT or USB headset/headphones or one additional USB
> sound card. And then there are 1% who have more than two audio
> devices. And for one or two sound cards having a priority list doesn't
> really make sense, does it? Also, don't forget that using multiple
> devices really *simultaneously* is very seldom.

True, but again, a list is quite a simple concept. It works IMO pretty 
well on OSX as a list too.

> Now, I am not saying that having such a priority list of devices is a
> bad idea. But I think the use case of folks with gazillions of devices
> is not realistic.

True, but I think with BT + USB + Built in + Network and you've already 
got a few different sinks there without trying too hard. While I agree 
it's going to be a minority of people that have such a setup, I think 
making configuration simple and easy in this scenario should be possible 
while maintaining a non-confusing UI for the majority, simpler case.

> (I) A better use case: the user plugs in his new audio device and now
> wants to play his music via that device. He wants to tell the computer
> that once in a simple way and wants that his music just moves to that
> device instantly and have the computer remember for the next time and
> that's it. He doesn't want a full screen dialog of lot's of options he
> doesn't understand and most importantly he doesn't want to understand
> the concept of 'priority lists')

Well, I don't think a priority list is that complex and like I say the 
feedback from people using the KDE priority list is pretty good. Ditto 
with the CoreAudio list of devices.

> (II) Or another use case: at work the user wants his music to be
> played on his BT headset. While travelling the user wants his music to
> be played on the internal speakers. While at home he wants his music
> to be played on his expensive USB sound card which connects to his
> stereo. For him it doesn't make sense to priorize the bt headset
> vs. the usb sound card, because he isn't at work and at home at the
> same time.

Sure, but dealing with that kind of scenario in a UI without it being 
very complex will be tricky. With a list, concepts are simple and the 
user will happily know that although he's prioritising one over the 
other that it doesn't really matter as he simply wont have a setup where 
the sinks are plugged at the same time - the logic of what will happen 
is still obvious to the user.


> (III) A third use case: the user wants music and events sounds and
> everything else go through his internal speakers. Phone calls however
> should go through his BT headset if available.

Should be OK with a priority list I should think.

> (IV) A fourth use case: some app played a very short sound. The user
> wants to adjust the sound's volume and device although it is already
> finished, for future streams.

Indeed. I think this should just be a matter of exposing the stream 
restore db in some capacity. Don't think this need will massively affect 
things here...

>> What to do
>> ==========
>>
>> With the above use cases in mind, it's clear that the current system  
>> needs to be revised to cope. In summary the problems we want to address 
>> are:
>>  1. Allow for an "active" default
> 
> Smells like an UI issue to me mostly.
> 
>>  2. Allow for a prioritised list of devices to be created (on a per-role 
>> basis - finer grained control would be too much).
> 
> We probably indeed want this. However I doubt that this should be user
> visible, or even controllable from outside of the PA server. It should
> be kept internally and learned internally. I don't think it makes
> sense to show the user devices that are not plugged in. A simple
> solution could be to use a list where the currently selected device if
> it was explicitly selected is moved to the top. i.e. just a simple
> history. And we pick the most recent item from the history list that
> would currently work.

I'd very much like to be able to query the server for devices that are 
not plugged in and show them to the user.

IMO this makes sense anyway (I kinda disagree that we should hide this 
completely). I really want this for KDE integration to work nicely. 
Their UI *does* expose this level of detail to the user (at a per-role 
level, not a per-app) and to get really good integration with PA 
(something I really want to see), it would be essential for me to be 
able to do this.

Also while the default tools etc. may present the user only simple 
options, I don't think we should discount the possibility of a more 
complex configuration app for power users.

That said, I do like the "history" idea. Perhaps the management of this 
history can be modular, so that in a KDE integrated mode, we leave it to 
the UI they provide to configure the lists, but for a "standard" 
install, the priority list is managed automatically by this new module? 
Would that work?


>>  3. Allow for applications which have the same role to have independant  
>> volumes saved.
> 
> Sure? Use case?

Two VoIP apps that have completely different levels coming out.

>> To this end, I think the current volume/map/mute/device database is  
>> incorrect. With the flat volume support, I think per-application volumes  
>> are more appropriate, rather than per-role.
> 
> Really? What's the difference between playing back music in rhythmbox
> and in banshee? The volume should be exactly the same. And what's the
> difference between getting an event sound from firefox or from
> nautilus? Or getting a phone stream from ekiga or telepathy?

Well rhythmbox may implement AGC and banshee may not. Perhaps one 
implements it's on soft volume and the other doesn't.

It's maybe not a massive problem practically tho', so I'm not sure if 
this is imporant really.


> I think storing things per-application is only a good idea if they
> don't supply us with the role they belong to or because they don't
> fit in any of the existing roles.

Perhaps. I'm still a bit undecided here personally. If this is the case, 
I'd argue that per-app volume control should be completely removed, and 
all "uncategorised" apps (e.g. those sans-role) be lumped into a 
"Uncategorised" role. It is this role that then has it's volume 
adjusted, not the app. This is clearly much simpler, but it also reduces 
the flexibility of pulseaudio by a big chunk. But I think if you're 
going to hide features under a certain condition (stream sans role), 
that it should be consistent across the board.

>> I think that a key->device->priority table is needed to allow for a  
>> given role identifier to have it's own rules written appropriately.  
> 
>> Pulse needs to remember the sinks (and sources) both past and present  
>> (including their "nice" name (description)) and provide a way for a UI  
>> app to purge a currently unavailable sink (e.g. remove it from the 
>> cache).
> 
> Again, I don't think it is a good idea to clobber things with audio
> devices that the user might have possessed at one time. For each
> role/active stream the user just should need to select one as the one
> to use from a list of actually available choices. If later on a
> different selection of devices is available it's PA's job to come up a
> with a good replacement choice by looking at the history of the user's
> choices.

I agree on the most part, but again, if this stuff is more or less 
stored internally, would it be a terrible idea to create a module 
capable of doing this. To integrate nicely into KDE I'd really need this 
functionality. I think it can be done in a modular way tho', so 
shouldn't get in the way of a default setup. WDYT?

>> When a stream is encountered, it's role should be checked and the  
>> prioritised list of devices loaded for that role. As in Phonon, if a  
>> stream does not have a role a "default" priority list should be used.  
>> (Optional) A subsequent check should be made to see if any specific  
>> rules state that this individual application has a preferred device to  
>> allow for fine grained control. If such a rule exists the device should  
>> be move to the top of the already loaded priority list or injected there  
>> if it does not exist (unlikely). The stream is then restored to the  
>> first available device. This priority list should be saved with the  
>> stream so it can be easily re-evaluated at any point should new sinks  
>> become available later.
>>
>> If any priority list is changed (e.g. by a client application) all  
>> streams should have their cached priority list invalidated and rebuilt  
>> (perhaps the rebuilding can be lazy - e.g. it's only loaded when
>> needed?)
> 
> I don't follow.

That's a pity as I still think this is right way forward.

Pseudo Logic:

1. New stream is created.
2. role = (empty(role) ? "default" : role);
3. Load role priority list and store with stream.
4. Check for per-app device preference, if exists, manipulate priority 
list accordingly.
5. Find first available device in priority list and use it.

(NB step 4 should be modular - e.g. modules should be able to hook into 
this in some way to manipulate the priority list as they need - this 
should give sufficient flexibility).


Event: New USB sink plugged in.
1. Go through all streams.
2. Find first available sink in the stream's priority list.
3. If the results of 2 != current, move the sink input.

(this allows the scenario that a user has chosen that their music should 
come out of USB speakers, they can plug/unplug their USB speakers as 
much as they want, and their currently playing track will automatically 
jump to the USB when it's there and fall back to some other device when 
it's not! Perfect!).


Event: USB sink pulled out.
* Do exactly the same as "New USB sink plugged in"

This replaces the functionality of module-rescue-streams.



Event: User changes a "priority list" via some UI.
1. Go through all streams.
2. Invalidate & Reload priority list.
3. Re-evaluate priority list (i.e. do the same as "New USB sink plugged in")


>> With regards to volumes etc., these should be saved against a given  
>> application and not a role. To do so at the role level is unlikely to be  
>> a useful end user feature.
> 
> Again, I doubt that.

Perhaps. I'm not really decided on this one as I mentioned above.

> 
>> With this system in place, a neat and effective UI can be produced and a  
>> good mapping into KDE can also exist which will be good for cross  
>> desktop accpetance of PulseAudio in the linux world.
> 
> Hmm, I honestly believe the KDE UI is too complex. Exposing this
> priority list to the user IMHO is overkill and we'd never implement
> something like that on GNOME.

That as it may be, but I'd very much like to integrate pulse into KDE 
properly, as do many others. Their UI is their UI, whether you or I like 
it is not really up for debate here. The feedback from users I've read 
is very good. People know how this system works and can do what they 
want with it, so it must be working in some capacity.

> Let's not limit ourselves by simply looking how KDE, or GNOME, or
> Windows, or MacOS do it right now. Let's do the design from
> scratch. It is not convincing to me to adopt a particular scheme just
> because KDE does it. And that's not because I hate KDE (I don't) but
> because I want to design a system that is as good and reasonable as I
> can come up with. And to make this clear: I don't look on how GNOME
> handles this right now either. 

That's fine, but just like you had to create an also plugin to make 
pulse work with what's out there, we may have to do something similar to 
make pulse work with various higher level stuff.

In KDE just now for example, I only display one thing in the "device" 
list, and that is a big PulseAudio "device". It would be nice to be able 
to see a list of sinks (past and present) here instead. This is the 
framework I need to fit into, whether it is the pinnacle of UI design or 
not. So while I agree with your "design from scratch" in principle, we 
do need to remain slightly flexible here.

> 
> Let me try to summarize what I have in mind:
> 
> Our rules have the following 'left-hand side', i.e. what is being matched on the streams:
> 
> A.1) the stream role, in absence we synthesize one by using the program name/identifier
> A.2) the stream role + already on a specific device
> A.3) the stream role + already on a specific card
> A.4) all streams
> 
> Additional conditions when rules may or may not apply:
> 
> B.1) when a stream is first created
> B.2) when a sink/source becomes available
> B.3) when a card becomes available
> B.4) when the user reconfigures things
> 
> We also need to match devices:
> 
> C.1) match device
> C.2) match card
> C.3) match device/card form factor
> C.4) match currently selected profile
> C.5) all devices
> 
> And on the 'right-hand side' (the effect of the rule):
> 
> D.1) move to a sink/source, possibly based on 'take first available from list' (aka history list)
> D.2) move to a sink/source of a specific card, possibly based on history list
> D.3) move to a sink/source of a specific card with history, and switch profile (profile switching with history list, too?)
> D.4) set volume
> D.5) set mute
> 
> (Did I miss anything?)
> 
> These are probably the most relevant features of a rule system for the
> desktop.  Our particular focus here is configurable rules,
> right? Hence I'd suggest ignoring for now static rules like "move
> phone calls to devices with form factor 'headsets'". That allows us to
> ignore C.3 and D.3 (I think...)
> 
> So the right hand side could be encoded in something like this:
> 
> struct device_history_item {
>        char *device;
>        char *card;       
>        pa_cvolume volume;  /* Volumes only make sense in the context of a specific device */
>        pa_channel_map map; /* To make sense of the volume */
> };
> struct right_hand_side {
>        pa_bool_t set_device_or_card;
>        pa_bool_t set_volume;
>        pa_bool_t set_mute;
>        pa_bool_t muted;
>        struct device_history_item *device_history;
> };
> 
> And the left hand side like this:
> 
> struct left_hand_size {
>        char *role;
>        char *device; 
>        char *card;
>        char *profile;
> 
>        pa_bool_t match_role;
>        pa_bool_t match_device_or_card;
>        pa_bool_t match_profile;
> 
>        pa_bool_t on_stream_new;
>        pa_bool_t on_device_or_card_new;
> };       
> 
> And when we have this most of the 'automatic' rules could be covered
> by this. Any use cases that are not?

Well, at a higher level than I'd like to see the implementation, I think 
this is OK.

As I said above, I'd still prefer to see pulse maintain a priority list 
internally, and have a module-history "manage" that list automatically 
to achieve the result you want. That way the routing logic can follow 
the process I outlined above ("Pseudo Logic").

I think the restoration of volume is slightly different tho' and could 
possibly be handled totally outwith the "priority list" approach (which 
is all about device restoration). I'm not really sure tho'.


We're possibly thinking about the problem differently and thus have 
different ways of visualising it, which probably isn't helping. 
Hopefully I've managed to get my ideas across a bit more clearly this time.

Col



-- 

Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
   Tribalogic Limited [http://www.tribalogic.net/]
Open Source:
   Mandriva Linux Contributor [http://www.mandriva.com/]
   PulseAudio Hacker [http://www.pulseaudio.org/]
   Trac Hacker [http://trac.edgewall.org/]




[Index of Archives]     [Linux Audio Users]     [AMD Graphics]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux