PnP -Core How to deal with List View Threshold

[faurepa]

Hello,
I have the following code that works for most instances for sharepoint access that I have come across so far.

var documentFolder = (await context.Web.Lists.GetByTitleAsync("Current", p => p.RootFolder));
var allFiles = (await documentFolder.RootFolder.GetAsync(f => f.Files)).Files; 
int counter = 0;

foreach (var file in allFiles.AsRequested())
{
    var extension = Path.GetExtension(file.Name);
    Console.WriteLine($"{counter} : Filename:{file.Name} - Path: https://honeywellprod.sharepoint.com{file.ServerRelativeUrl}");
    string documentUrl = $"{file.ServerRelativeUrl}";
    // Get a reference to the file
    IFile testDocument = await context.Web.GetFileByServerRelativeUrlAsync(documentUrl);

    // Download the file as stream
    Stream downloadedContentStream = await testDocument.GetContentAsync();

    using (FileStream localfs = File.OpenWrite(@$"c:\\temp\\{file.Name}"))
    {
        CopyStream(downloadedContentStream, localfs);
    }
}

The "AsRequested()" in the Foreach loop got around the 100 item limit.

When I try to run while trying to get a list of documents in the "Current" folder (currently has 16000+ documents). I get an error: 500:
Here is the console infromation:
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[100]
Start processing HTTP request GET https://honeywellprod.sharepoint.com/teams/aeroprocesscentral/_api/Web/Lists(guid'db9bce31-be74-4eaf-afce-7b52f8ed8257')/rootFolder?$select=UniqueId%2cFiles&$expand=Files
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[100]
Sending HTTP request GET https://honeywellprod.sharepoint.com/teams/aeroprocesscentral/_api/Web/Lists(guid'db9bce31-be74-4eaf-afce-7b52f8ed8257')/rootFolder?$select=UniqueId%2cFiles&$expand=Files
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[101]
Received HTTP response headers after 67.3409ms - 500
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[101]
End processing HTTP request after 73.6249ms - 500

When Looking at the Fiddler results for Error 500, Text View provides:
cd
{"odata.error":{"code":"-2147024860, Microsoft.SharePoint.SPQueryThrottledException","message":{"lang":"en-US","value":"The attempted operation is prohibited because it exceeds the list view threshold."}}}
0

Is there a way to deal with this, or do I need to change my implementation?
The current implementation reads all files in a given folder, and I compare them one at a time with another list. I download the files which have the correct file name. Using a find, I will need to use a regular expression as I don't always have an exact match for the file name.

Thoughts or suggestions would be appreciated.

Thank you

[faurepa]

Upon some digging, I found that there is the FindFiles() method available. (https://pnp.github.io/pnpcore/using-the-sdk/files-intro.html#finding-files) The issue I am having is that it doesn't return all possible solutions. The folder in question has 1400+ files. 20 or so start with "1004 - ". When I search for "1004", only 2 items are returned, not the 20. If I try to add any other filtering to the regular expression, I get 0 results. ie "1004EN.pdf*" I know that there is at least one file with that format.

Speed is not an issue for this application.

If I remove all filtering, I get 100 files, this means to me that the FindFiles method is running into the limit of returning only 100 results at a time. I know that I was able to use ".AsRequested()" on lists of documents before, but as the method is built into the program, I can't access it.

Is it possible to have the FindFiles Method updated so that it can handle any quantity of files?

I have also tried to use: https://pnp.github.io/pnpcore/using-the-sdk/listitems-intro.html#b-getting-list-items-via-paging-no-item-limit
but when I tried that, I keep getting an error stating that the "File name hasn't been loaded yet".
var myList = await context.Web.Lists.GetByTitleAsync("Policies", p=>p.Title);

await foreach (var listItem in myList.Items)
{
Console.WriteLine($"Filename:{listItem.File.Name} ");
}

[faurepa]

Hello, Thank you for your response, I have had a look at the throttling information. This is useful information. I understand why they put limits on responses. Now trying to get around them is the more challenging part. Essentially what I need to do is compare a list of document names from Sharepoint with a different list stored in a separate DB. The DB list is a subset of what is in Sharepoint. Only the files with the same name do I need to download, read (extract the released date/revision), and then move on to the next file (I can't count on the upload date/revision of the SP file as they don't match what is in the file). The problem is that I have no control over the location or structures of the SP files. I tried the FindFiles method, this runs into the 100 files returned limit. Finding a solution that allows me to get the list of files so that I can filter them is what is needed. Will try the enumeration example ( https://pnp.github.io/pnpcore/using-the-sdk/files-intro.html#enumerating-files ), Any other suggestions would be appreciated. As you can probably tell, I am new at working with SP, and tend to look for examples to learn from. Thank you for your time and guidance. Philippe

…

On Fri, 29 Dec 2023 at 05:21, KFC ***@***.***> wrote: Hi @faurepa <https://github.com/faurepa>, for the throttling issue, I'd suggest to go through this documentation first <https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online> to familiarize yourself with the different approaches of avoiding throttling and gaining the understanding of it, may be that will help you over come it. If I'm understanding the need of this action of reading all the files, to avoid a massive amount of request for files you can create your own kind of metadata/the actual text /whatever DB for the different files based on their modification/creation dates to reduce the amount of files that you're downloading directly from SP, maybe some sort of cache, and managing the amount of request being perform on a SC and so on... this is really depends on the type of compare you need, the frequency of the files being changed, the time you have for performing this action from the moment it's being triggered to completion and more.. hopes that helps. — Reply to this email directly, view it on GitHub <#1336 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGM2GLCHA5WGSCJ4CUDSJPLYL2KRZAVCNFSM6AAAAABA7IP6FWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TSNZRGIYDK> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[faurepa]

Trying to use the Enumerating-files doesn't provide the results that I am expecting.
IFolder folder = await context.Web.Folders.GetFirstOrDefaultAsync(f => f.Name == "Policies");
C# returns an error:
Severity Code Description Project File Line Suppression State
Error CS1061 'IFolderCollection' does not contain a definition for 'GetFirstOrDefaultAsync' and no accessible extension method 'GetFirstOrDefaultAsync' accepting a first argument of type 'IFolderCollection' could be found (are you missing a using directive or an assembly reference?) Demo C:\Users\H173957\Documents\Code\PnPCore-modified\Program.cs 400 Active

So trying the previous version:
IFolder folder = await context.Web.Folders.FirstOrDefaultAsync(f => f.Name == "Policies");
Returns an error 403 : Unauthorized access exception.

The "Policies" folder is a document Library, so based on the notes, this should be the correct solution.

I am not sure why the example doesn't work.