Help me understand autorestic concurrency

[johnsturgeon]

I am a new autorestic user and I'm a bit confused. I set up several locations / backends in ~/.config/autorestic/.autorestic.yml

I run the backups from Kestra and I noticed that when two jobs overlap, the second one fails because of a lock.

I don't understand why since they're different repos. Also, I can just get around it by separating my locations / backends into different folders so that the lock file goes into different locations. They run concurrently just fine in that case.

Thoughts? Is this a bug? Is this intended behavior?

[Keisir]

Also, I can just get around it by separating my locations / backends into different folders so that the lock file goes into different locations.

So you have created several backends (restic respositories) in the same folder? That should not work.

I am guessing that you are talking about a restic lock or is some file locked by your system?

It would be good if you would share your autorestic configuration and the commands you are executing. Then it would be clearer what exactly you are trying to achieve. You should also include the stdout of the failing backup process.

[johnsturgeon]

My repositories are on BackBlaze, and no, I do not have several in the same folder.

I'm talking about the autorestic lock

I'm trying to achieve the ability to run two auto restic commands at the same time... If they are different locations, going to different backends, there is absolutely no reason that they should lock eachother out. I just don't understand autorestic's reasoning for locking whenever it's running based on the folder that the config file is located in.

Here's a concrete example

version: 2
extras:
  env: &b2env
    B2_ACCOUNT_ID: blah
    B2_ACCOUNT_KEY: blah
    RESTIC_PASSWORD: blah

global:
  backup:
    exclude:
      - '.Trash*'
      - '.Spotlight*'
      - '.DS_Store'
      - '@Recycle'
  forget:
    keep-hourly: 24
    keep-daily: 7
    keep-weekly: 5
    keep-monthly: 12
    keep-yearly: 10

locations:
  pictures:
    from: /Users/johnsturgeon/Pictures
    to: bb-pictures
    forget: prune
  mylio-photos:
    from: /Users/johnsturgeon/Mylio
    to: bb-mylio
    forget: prune
  usb-photos:
    from: /Volumes/Photos
    to: bb-usb-photos
    forget: prune

backends:
  bb-pictures:
    type: b2
    path: 'JHS-Photos:/Pictures'
    env:
      <<: *b2env
  bb-mylio:
    type: b2
    path: 'JHS-Photos:/Mylio'
    env:
      <<: *b2env
  bb-usb-photos:
    type: b2
    path: 'JHS-Photos:/iMac-USBDrive'
    env:
      <<: *b2env

Now suppose I run:
autorestic backup -l pictures
and WHILE THAT IS RUNNING I run
autorestic backup -l mylio-photos

I get the error:

Using lock:	 .autorestic.lock.yml
an instance is already running. exiting

That's odd.. since there's no reason that restic itself needs a lock in that instance.. why does autorestic use the same lock file for different locations / backends?

[Keisir]

Oh, I had previously misunderstood you.
I thought you meant a restic lock, as this leads to problems more often.

As far as I know, the autorestic.yml lock is technically only needed for the autorestic check command, as this command can change the configuration file itself.

I also have this problem on a system and solve it by copying the autorestic.yml to a temporary folder and deleting it again after the backup. This saves me having to split up and manage different configuration files.

When I considered this solution, I had a look at the code, but found no real indication that the creation of this lock is really necessary.

Why this lock is created can probably only be explained by the author himself.