Sudden 404 after confusing situation

[Handrail9]

Hello, I am kind of at a loss for words at the moment as it seems something has happened to my nextcloud that I have no idea how to explain.

Today while I was out I got a notification on my phone that Davx5 was having trouble syncing, so when I got home I went to troubleshoot it, finding that the nextcloud containers were down. After bringing them back up there were more errors, almost every folder was missing from the /appdata_xxxx folder in my data directory, with the exception of an identity folder. I tried restoring a built in backup but after waiting for a few minutes the backup failed. I tried restoring a couple month old manual backup I had taken with rsync to another drive, and this has made things worse, as now nextcloud will not show me anything other than a 404 error when I try to navigate to it in the browser. I am not sure why /appdata_xxxx lost most of it's contents. I am not sure why the built in borg backup solution failed.

My system is an amd64 VM running locally with Fedora Server 41 (with SELinux disabled label added to the AIO container) with 8gb of ram, a 30gb qcow2 volume and a 16tb drive for the datadir and NC volumes. Auto-updates are enabled and I am using the :latest tag

I want to know what troubleshooting steps I can take from here. What logs can I provide to get things back on track? Is there a way to unencrypt the data if I had nextcloud data dir encryption turned on (NOT E2EE!) and start over with a new NCAIO?

sudo docker container logs nextcloud-aio-nextcloud:

 
Connection to nextcloud-aio-database (172.22.0.4) 5432 port [tcp/postgresql] succeeded!
              now              
-------------------------------
 2025-01-27 00:09:46.025518-05
(1 row)

+ '[' -f /dev-dri-group-was-added ']'
++ find /dev -maxdepth 1 -mindepth 1 -name dri
+ '[' -n '' ']'
+ set +x
Enabling Imagick...
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.21/main: No such file or directory
WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.21/community: No such file or directory
Connection to nextcloud-aio-redis (172.22.0.5) 6379 port [tcp/redis] succeeded!
Applying one-click-instance settings...
System config value one-click-instance set to boolean true
System config value one-click-instance.user-limit set to integer 100
System config value one-click-instance.link set to string https://nextcloud.com/all-in-one/
support already enabled
Adjusting log files...
System config value upgrade.cli-upgrade-link set to string https://github.com/nextcloud/all-in-one/discussions/2726
System config value logfile set to string /var/www/html/data/nextcloud.log
Config value were not updated
System config value updatedirectory set to string /nc-updater
System config value maintenance_window_start set to integer 100
Applying network settings...
System config value allow_local_remote_servers set to boolean true
System config value davstorage.request_timeout set to integer 3600
System config value trusted_domains => 1 set to string nextcloud.mydomain.tld
System config value overwrite.cli.url set to string https://nextcloud.mydomain.tld/
System config value documentation_url.server_logs set to string https://github.com/nextcloud/all-in-one/discussions/5425
System config value htaccess.RewriteBase set to string /
.htaccess has been updated
System config value dbpersistent set to boolean false
System config value auth.bruteforce.protection.enabled set to boolean true
System config value ratelimit.protection.enabled set to boolean true
System config value files_external_allow_create_new_local set to boolean true
notify_push is up-to-date or no updates could be found
System config value trusted_proxies => 0 set to string 127.0.0.1
System config value trusted_proxies => 1 set to string ::1
System config value trusted_proxies => 10 set to string 172.22.0.0/16
Config value were not updated
+ echo nextcloud-aio-collabora
+ grep -q 'nextcloud-.*-collabora'
+ COLLABORA_HOST=nextcloud.mydomain.tld
+ set +x
App "Nextcloud Office" cannot be installed because it is not compatible with this version of the server.
Config value were not updated
No ipv6-address found for nextcloud.mydomain.tld.
Config value were not updated
Connection to nextcloud-aio-clamav (172.22.0.6) 3310 port [tcp/*] succeeded!
files_antivirus is up-to-date or no updates could be found
Config value were not updated
Config value were not updated
Config value were not updated
Config value were not updated
Config value were not updated
Config value were not updated
System config value enabledPreviewProviders => 0 set to string OC\Preview\Imaginary
System config value enabledPreviewProviders => 23 set to string OC\Preview\ImaginaryPDF
System config value preview_imaginary_url set to string http://nextcloud-aio-imaginary:9000
System config value preview_imaginary_key set to string 79898876130123354e5a9f4fae3d9cbc8dcf5b2fbe6bf635
Connection to nextcloud-aio-fulltextsearch (172.22.0.7) 9200 port [tcp/*] succeeded!
App "Full text search" cannot be installed because it is not compatible with this version of the server.
App "Full text search - Elasticsearch Platform" cannot be installed because it is not compatible with this version of the server.
App "Full text search - Files" cannot be installed because it is not compatible with this version of the server.

                                                                    
  There are no commands defined in the "fulltextsearch" namespace.  
                                                                    


                                                                               
  There are no commands defined in the "fulltextsearch_elasticsearch" namespa  
  ce.                                                                          
                                                                               


                                                                          
  There are no commands defined in the "files_fulltextsearch" namespace.  
                                                                          

Error: Could not download app whiteboard
Config value were not updated
Config value were not updated
+ '[' true = true ']'
+ '[' 11000 = 443 ']'
+ set +x
Waiting for Apache to become available...
[27-Jan-2025 00:17:55] NOTICE: fpm is running, pid 406
[27-Jan-2025 00:17:55] NOTICE: ready to handle connections
Connection to nextcloud-aio-apache (172.22.0.13) 11000 port [tcp/*] succeeded!
Activating Collabora config...

                                                                   
  There are no commands defined in the "richdocuments" namespace.  
                                                                   

[27-Jan-2025 00:26:03] NOTICE: Terminating ...
[27-Jan-2025 00:26:03] NOTICE: exiting, bye-bye!
+++ pgrep -f /var/www/html/cron.php
++ '[' -n '' ']'
++ echo 'Cronjob successfully exited.'
++ exit
Cronjob successfully exited.

[szaimen]

I tried restoring a built in backup but after waiting for a few minutes the backup failed.

Hi, can you do this again and post the output of sudo docker logs nextcloud-aio-borgbackup here?

[Handrail9]

Hi, sorry for the late reply. I tried checking the integrity first and got these logs

2025-01-28T09:03:46.998590167Z Killed stale lock nextcloud-aio-borgbackup.37-0.
2025-01-28T09:03:47.099975277Z Removed stale exclusive roster lock for host nextcloud-aio-borgbackup pid 37 thread 0.
2025-01-28T09:03:47.198579710Z Killed stale lock nextcloud-aio-borgbackup.37-0.
2025-01-28T09:03:47.302054977Z Removed stale exclusive roster lock for host nextcloud-aio-borgbackup pid 37 thread 0.
2025-01-28T09:03:47.302923808Z Removed stale exclusive roster lock for host nextcloud-aio-borgbackup pid 37 thread 0.
2025-01-28T09:03:47.874084609Z Checking the backup integrity...
2025-01-28T09:03:48.182555509Z Starting repository check
2025-01-28T13:07:16.373373231Z finished segment check at segment 5894
2025-01-28T13:07:16.478414071Z Starting repository index check
2025-01-28T13:07:16.478577436Z Index object count match.
2025-01-28T13:07:16.971370803Z Finished full repository check, no problems found.
2025-01-28T13:07:16.973714870Z Starting archive consistency check...
2025-01-28T13:07:17.686535298Z Starting cryptographic data integrity verification...
2025-01-28T13:14:20.364216795Z chunk 1859fc42333b6d9b60808a3a391b497eca6fbb44bb2e536a4a067232f231cfac, integrity error: MAC Authentication failed
2025-01-28T19:33:31.440238973Z Found defect chunks. With --repair, they would get deleted, so affected files could get repaired then and maybe healed later.
2025-01-28T19:33:31.440269047Z Finished cryptographic data integrity verification, verified 1605557 chunks with 1 integrity errors.
2025-01-28T19:33:31.441985559Z Analyzing archive 20240731_070123-nextcloud-aio (1/16)
2025-01-28T19:33:53.152298692Z Analyzing archive 20240831_070123-nextcloud-aio (2/16)
2025-01-28T19:34:09.212448164Z Analyzing archive 20240928_070114-nextcloud-aio (3/16)
2025-01-28T19:34:29.624234958Z Analyzing archive 20241031_070117-nextcloud-aio (4/16)
2025-01-28T19:34:47.103411001Z Analyzing archive 20241121_164243-nextcloud-aio (5/16)
2025-01-28T19:35:04.148537327Z Analyzing archive 20241216_070124-nextcloud-aio (6/16)
2025-01-28T19:35:23.616240373Z Analyzing archive 20241229_070140-nextcloud-aio (7/16)
2025-01-28T19:35:24.441048996Z Analyzing archive 20241231_070147-nextcloud-aio (8/16)
2025-01-28T19:35:25.122539562Z Analyzing archive 20250104_070117-nextcloud-aio (9/16)
2025-01-28T19:35:25.843659930Z Analyzing archive 20250109_070142-nextcloud-aio (10/16)
2025-01-28T19:35:26.822502673Z Analyzing archive 20250121_070808-nextcloud-aio (11/16)
2025-01-28T19:35:27.622796879Z Analyzing archive 20250122_070233-nextcloud-aio (12/16)
2025-01-28T19:35:28.438676935Z Analyzing archive 20250123_070242-nextcloud-aio (13/16)
2025-01-28T19:35:29.407757839Z Analyzing archive 20250124_071430-nextcloud-aio (14/16)
2025-01-28T19:35:30.197336031Z Analyzing archive 20250125_070228-nextcloud-aio (15/16)
2025-01-28T19:35:31.032931707Z Analyzing archive 20250126_070152-nextcloud-aio (16/16)
2025-01-28T19:35:32.353371028Z Archive consistency check complete, problems found.
2025-01-28T19:35:32.582621578Z Some errors were found while checking the backup integrity!
2025-01-28T19:35:32.582647562Z Check the AIO interface for advices on how to proceed now!

I'm going to run the AIO repair and go from there and come back when I have errors from the backup again. I'm not sure if this is helpful information but I can actually view my files from the Android app, and am able to sync Davx5 and receive nextpush notifications. I may try connecting my laptop to the router via ethernet and downloading all my data when I get the chance if we can't figure this out, although I fear there has been some sort of data loss at this point as my NC data dir is about 150gb, when I remember it being over 400gb a few months ago, but like I said, ill run the repair and see what happens.

[Handrail9]

@szaimen Okay, finally got it to fail again. It takes a while for whatever reason. Here is the log (with some replaced text for privacy, although if you find something else feel free to hide the file)
logs.txt

[szaimen]

Hm... there are a lot of 2025-01-30T04:27:53.901036459Z rsync: [sender] opendir "/tmp/borg/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data/..." failed: Socket not connected (107) errors.

Can you run an backup integrity check from the AIO interface and see if it finds errors in your borg archive?

[Handrail9]

This is all it said, I let it run for about ~12 hours and I just saw it had "finished"

2025-01-31T10:12:35.478298325Z Checking the backup integrity...
2025-01-31T10:12:35.924848826Z Starting repository check

It doesn't look correct to me so I am going to rerun the check. I would imagine there has to be some sort of output.

[szaimen]

Yes, there should be more