Add support for iOS AVAudioSessionCategory in MediaElement

[JamesBodi]

Is there an existing issue for this?

• I have searched the existing issues

Did you read the "Reporting a bug" section on Contributing file?

• I have read the "Reporting a bug" section on Contributing file: https://github.com/CommunityToolkit/Maui/blob/main/CONTRIBUTING.md#reporting-a-bug

Current Behavior

MediaElement plays audio on Android simulators and devices, and on IOS simulators, but does not work on IOS devices. Yes, I checked the "silent" switch. I've turned it on and off. I tested on 2 IOS Iphone versions and one IPad - None work.

MediaElement appears to utilize my logging mechanism (NLog) as I am getting the following error in my log:

2024-01-01 14:25:38.7042|WARN|MediaManager|NSConcreteNotification 0x284c84c80 {name = AVPlayerItemFailedToPlayToEndTimeNotification; object = <AVPlayerItem: 0x2888e8c20, asset = <AVURLAsset: 0x284ccd880, URL = file:///var/mobile/Containers/Data/Application/8B4B4CDD-4176-484D-9F8D-295331FDF6FD/Library/Recordings/2024-01-01_13-47-57/audio_data_00.wav>>; userInfo = { AVPlayerItemFailedToPlayToEndTimeErrorKey = "Error Domain=AVFoundationErrorDomain Code=-11800 \"The operation could not be completed\" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-66637), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x2805d6190 {Error Domain=NSOSStatusErrorDomain Code=-66637 \"(null)\"}}"; }}

I get this error despite never triggering MediaFailed or MediaEnded events. I get the exact same message when trying to play mp3s. What does this error mean?

I also tried exporting audio files to an external/public folder on an IPad to see if they would play - they played just fine. So there is no issue with the encoding of the audio files. I've also tried playing audio via url, via embedding, and audio files created by my app - none will play on IOS devices.

Expected Behavior

I expect audio files to play on actual IOS devices just as they do on IOS simulators.

Steps To Reproduce

See steps in ReadMe file of referenced project... MediaElement stops working once objects from PLugin.Maui.Audio are created.

public MainPage(IAudioManager audioManager)
{
InitializeComponent();
// TODO: MediaElement works until you uncomment the next 2 lines!!
//_audioManager = audioManager;
//_audioRecorder = _audioManager.CreateRecorder();
}

Link to public reproduction project repository

https://github.com/JamesBodi/AudioPlayerTest.git

Environment

- .NET MAUI CommunityToolkit: "7.0.1"
- CommunityToolkit.Maui.MediaElement: "3.0.1" 
- OS: IPhone 12 and 13, IPad 9th gen
- .NET MAUI: 8.0.3

Anything else?

No response

[ghost]

Hi @JamesBodi. We have added the "needs reproduction" label to this issue, which indicates that we cannot take further action. This issue will be closed automatically in 5 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[JamesBodi]

@brminnick I added a project to reproduce this issue.

[JamesBodi]

Based on my research, The error codes AVFoundationErrorDomain Code=-11800 and NSOSStatusErrorDomain Code=-66637 generally indicate an issue within the AVFoundation framework, used for processing and handling media content in iOS. While these error codes don't provide specific details about the cause of the issue, they do point to a failure in the media playback or processing operation.

The AVFoundationErrorDomain Code=-11800 error translates to AVErrorUnknown, which unfortunately means it's a kind of catch-all for various unknown errors within the AVFoundation framework. This lack of specificity makes it challenging to pinpoint the exact cause without further investigation.

In one instance discussed in the Apple Developer Forums, a user encountered a similar error when trying to export a video from an iPhone using AVAssetExportSession. The error was linked to issues with accessing or processing media files, especially when these files were located in shared iCloud albums or required specific handling due to their properties or the iOS version.

Another discussion on the Apple Developer Forums relates to an error NSOSStatusErrorDomain Code=-16154, which indicates a problem with an XPC connection, often related to media services being reset. This could result in errors like AVErrorMediaServicesWereReset, suggesting that the issue might stem from the underlying media services layer.

[JamesBodi]

I have continued to research this and eventually just wrote my own audio service in native code. In the process I think I discovered why MediaElement and Plugin.Maui.Audio fail when attempting to playback on any IOS device.

If your code looks like this it won't work:

        public void PlayAudio(string filePath)
        {
            if (_player == null)
            {
                _url = NSUrl.FromString(filePath); //  <<
                AVPlayerItem avPlayerItem = new AVPlayerItem(_url); //  <<
                _player = new AVPlayer(avPlayerItem);
                _player.AutomaticallyWaitsToMinimizeStalling = false;
                _player.Volume = 1;
                _player.Play();

But if your code looks like this it will work:

        public void PlayAudio(string filePath)
        {
            if (_player == null)
            {
                AVAsset asset = AVAsset.FromUrl(NSUrl.CreateFileUrl(new[] { filePath }));  // <<
                AVPlayerItem avPlayerItem = new AVPlayerItem(asset);  // <<
                _player = new AVPlayer(avPlayerItem);
                _player.AutomaticallyWaitsToMinimizeStalling = false;
                _player.Volume = 1;
                _player.Play();

I am guessing that MediaElement and Plugin.Maui.Audio are using the same base code as they both fail in the same way and have the same author. As my native code audio service is working perfectly I will not be switching back.

[bijington]

I'm intrigued to know why it's failing for you but not in the sample application, unless have we missed an example.

If you have a fix for this bug it would be amazing if you would submit it in via a PR.

[JamesBodi]

Playback fails only when you try to play on an IOS device. Did anyone ever test that? The fix for me is shown above. You have to create an AVAsset then create the AVPlayerItem using that asset. I found that some example projects out there have one version that uses the NSUrl and others that use AVAsset. I ended up having both versions with one app working and the other not. When I compared the code I found this one key difference.

[bijington]

I have audio playback working fine in some of my apps although they are loading from a resource rather than a file on disk.

[kablammyman]

I have videos loading from a server and the media player works fine in android and windows but does NOT work for me on an iphone (iphone 8 running ios 16.7.5). When I say it doesnt work, I mean there is no sound or video, just a black image that has a play button thats "crossed out"

However, I can see youtube videos thru the app tho (that doesnt use mediaplayer, but webview)

I really need this bug fixed since this is a show stopper for me. I can provide a video of both the android and ios version running at the same time on actual devices if that helps

Here are the packages I have:

[net8.0-ios17.2]: 
   Top-level Package                               Requested   Resolved
   > CommunityToolkit.Maui                         8.0.1       8.0.1   
   > CommunityToolkit.Maui.Maps                    2.0.1       2.0.1   
   > CommunityToolkit.Maui.MediaElement            3.1.0       3.1.0   
   > CommunityToolkit.Mvvm                         8.2.2       8.2.2   
   > CSCore                                        1.2.1.2     1.2.1.2 
   > Geocoding.Microsoft                           4.0.1       4.0.1   
   > Microsoft.Extensions.Logging.Console          8.0.0       8.0.0   
   > Microsoft.Extensions.Logging.Debug            8.0.0       8.0.0   
   > Microsoft.Maui.Controls                       8.0.21      8.0.21  
   > Microsoft.Maui.Controls.Compatibility         8.0.21      8.0.21  
   > Microsoft.Maui.Controls.Maps                  8.0.21      8.0.21  
   > Microsoft.Maui.Essentials                     8.0.21      8.0.21  
   > Microsoft.Maui.Extensions                     6.0.553     6.0.553 
   > Microsoft.NET.ILLink.Tasks              (A)   [8.0.2, )   8.0.2   
   > NAudio                                        2.2.1       2.2.1   
   > NAudio.Lame                                   2.1.0       2.1.0   
   > Newtonsoft.Json                               13.0.3      13.0.3  
   > OneSignalSDK.DotNet                           5.1.2       5.1.2   
   > Plugin.LocalNotification                      11.1.2      11.1.2  
   > Plugin.Maui.Audio                             2.1.0       2.1.0   

   [net8.0-maccatalyst17.2]: 
   Top-level Package                               Requested   Resolved
   > CommunityToolkit.Maui                         8.0.1       8.0.1   
   > CommunityToolkit.Maui.Maps                    2.0.1       2.0.1   
   > CommunityToolkit.Maui.MediaElement            3.1.0       3.1.0   
   > CommunityToolkit.Mvvm                         8.2.2       8.2.2   
   > CSCore                                        1.2.1.2     1.2.1.2 
   > Geocoding.Microsoft                           4.0.1       4.0.1   
   > Microsoft.Extensions.Logging.Console          8.0.0       8.0.0   
   > Microsoft.Extensions.Logging.Debug            8.0.0       8.0.0   
   > Microsoft.Maui.Controls                       8.0.21      8.0.21  
   > Microsoft.Maui.Controls.Compatibility         8.0.21      8.0.21  
   > Microsoft.Maui.Controls.Maps                  8.0.21      8.0.21  
   > Microsoft.Maui.Essentials                     8.0.21      8.0.21  
   > Microsoft.Maui.Extensions                     6.0.553     6.0.553 
   > Microsoft.NET.ILLink.Tasks              (A)   [8.0.2, )   8.0.2   
   > NAudio                                        2.2.1       2.2.1   
   > NAudio.Lame                                   2.1.0       2.1.0   
   > Newtonsoft.Json                               13.0.3      13.0.3  
   > OneSignalSDK.DotNet                           5.1.2       5.1.2   
   > Plugin.LocalNotification                      11.1.2      11.1.2  
   > Plugin.Maui.Audio                             2.1.0       2.1.0   

[ne0rrmatrix]

I have looked at how AVAudioSession works and it looks like we would need to implement additional features to support recording audio. ATM the way media element in initialized it sets AVAudioSessionCategory.Playback and for recording it needs to be set to AVAudioSessionCategory.Record. We will need to implement bindings and update the docs to support this use case. Maybe create a discussion about adding this? @vhugogarcia

[vhugogarcia]

Based on your statement @ne0rrmatrix, I believe this should be considered as a new proposal feature and maybe we should move this to a discussion, since we may need to discuss about it and if there are any implications.

@brminnick what do you think?