v2: Metadata logging, custom attributes, inline files, and a major version bump

[geky]

This is the culmination of work that just started as adding custom attributes (#23). And now it's coming in with metadata logging, inline files, xor globals, independent cache sizes, and a bunch of other smaller features piggybacking on what is going to have to be a major version bump. It's been an unexpectedly crazy journey.

Note! There's still a lot to do, mostly around documentation and prepare for a major version bump. But I wanted to get this out earlier for feedback.

So! What's new?

Big things:

1. Metadata logging

   You heard it here folks, the metadata pairs can now be updated incrementally. This makes them act like little two-block logs.

   Logs have a few challenges, most notably, you either pay a O(n) RAM cost or you pay a O(n^2) for garbage collection (SPIFFS cleverly gets around this by twiddling previously bits, but this is limited to raw flash). But because these logs are limited to two blocks, the O(n^2) cost isn't that bad. And if your cache size is large enough, the number of disk reads is reduced to O(n). This is similar to the tradeoffs in nvstore I believe.

   There's also several other tricks that I need to document before this gets merged, such as only using one pass for name lookups and using xor-linked-lists to enable forward+backward iteration.

   This should improve low-level performance immensely, especially on storage with expensive erase operations (cough NOR flash cough). Additionally, entries are trivially resizable, which caused a bit of a hiccup for v1.

   Unfortunately, this ended up requiring a change to the structures stored on disk. So this will probably be the major bump to littlefs v2. One fortune is that the changes are only to metadata. And hey guess what, all metadata blocks are pairs where one block is redundant. So in theory it should be easy to write an upgrade function that runs in place, simply using the backup blocks in the metadata pairs. I'm going to try to implement this before merging v2 in.

2. Custom attributes

   It's now possible to get and set custom attributes on files, directories, and the superblock.

   This isn't quite the same as getxattr/setxattr, instead uses a byte identifier to lookup attributes. Additionally the API is a bit different to support atomic updates with file writes.

   Most notably, custom attributes take advantage of the config struct added for optional config per file (Added possibility to open multiple files with LFS_NO_MALLOC enabled #58).

   More info over here: Manage generic attributes #23

3. Inline files

   Now small files (<1KiB) can be inlined directly in the directory block instead of getting their own block, which can waste a lot of space on devices with larger block sizes or fewer blocks.

   Note: Because inline files must be entirely stored in a file's cache, a configurable inline_max attribute is written to the superblock at format time. If the device has less RAM and a smaller cache_size, the filesystem won't be able to mount. By default inline_max = cache_size.

   EDIT: This is has been fixed! Now littlefs can read filesystems with any inline size, it's only write time that is limited to cache_size.

   Combined with metadata logging, inline files have a lot of potential. The littlefs should no longer be a bad choice for internal flash and NAND as long as you make your inline_max large enough.

4. Xor globals, move problem solved

   So this kinda came out of nowhere. But while implementing inline files, I ran into a bit of a problem. The way littlefs handles moves depends on the uniqueness of an directory entry's contents. But inline file's aren't unique at all, and by making two inline files with the same data every thing broke.

   So I dusted an old idea I had about maintaining distributed global state and managed to get it working. This pushed everything back a bit, but I think the end result is a big improvement.

   Basically every metadata pair has a copy of global state, and during mount they all get xored together. This allows a commit to any metadata pair the opportunity to change global state for atomically.

   Toss on an entry for moves and some logic to fix them, and now littlefs can recover from moves O(1) worst case. This is a big improvement from the O(n) search in v1.

   As an added plus, by putting this logic in the dir commit function, littlefs was able to gracefully handle shortcut renames in the same directory by accident.

   This logic also has potential for future improvements (global free list?), but does have a big RAM cost, since each global element has a copy in every directory structure.

5. Independent cache sizes

   Before, the read_size and prog_size config options doubled as a way to increase the RAM used for read/prog operations. However with metadata logging, this presented a paradox. Log updates benefit from small prog sizes, but other operations benefit from large prog sizes.

   Additionally, there were some issues floating around with performance with large read sizes, since large read sizes previously required the full size get read, this made ctz-list traversal expensive.

   So now there's a new cache_size config option independent of read_size and prog_size. read_size and prog_size should be as small as allowed by the underling device, and cache_size should be as big as you're willing to give littlefs RAM.

6. Expanding superblocks

   A minor feature, but riding on the coattails of the breaking changes is the addition of expanding superblocks. Instead of permanently allocating blocks 0 and 1 and the superblock, littlefs now starts off by reusing blocks 0 and 1 as the root directory. However, as soon as it sees more than 8 erases on blocks 0 and 1 (currently an arbitrary number, but may be smarter in the future), littlefs splits out the root directory from the superblock.

   But this doesn't stop there, if littlefs again sees more than 8 erases, it will add another superblock. Because each new superblock requires a full lifetime before it causes a modification to travel to it's parent, the number of erases needed for the next superblock grows exponentially.

   This is a benefit for both small device, which no longer need the extra superblock, and large devices, which could exceed the number of erase cycles on the superblock if the root directory was written 10^10 times.

Small things:

3. Configurable name max

   I added an inline_max configuration option to the superblock, since knowing this is required for portability and there's not really a good value to assume. So why not other configurables? Now the superblock also tracks name max, which means you can push the RAM consumption of the lfs_stat struct down and still be portable.

4. Added lfs_fs_size

   A function to get a count of the used blocks on the filesystem. This is just a wrapper over lfs_traverse, but should be easier to use. Probably could have come in on a minor release, oh well.

5. Dropped global file buffer for local file buffers

   Thanks to @dpgeorge's patch for file-level config (Added possibility to open multiple files with LFS_NO_MALLOC enabled #58), we have a better way to provide files with buffers.

   Since we're bumping the major version, might as well remove the old way.

6. Better update tracking

   Metadata logging, with resizable entries, means that there's a lot of state changes flying around.

   This required a review and ultimately a rewrite of the way littlefs manages multiple open files and dirs. The end result is a pretty decent update system for managing linked data structures.

   It's an internal change, but notably now file's don't have to rescan their metadata pairs every write.

7. Renamed lfs_crc -> lfs_crc32

   I've been looking at CRC APIs and this was the most common naming pattern and arguments I found. I figured it would be a good idea to adopt the standard and avoid the indirect reference to crc.

Related issues

All of these changes are to tackle the biggest issues users have raised so far. A big thanks for all the feedback and for those who have had to wait.

• Add custom attributes, inline files, and resizable entries #48 - first attempt (superseded by this pr)
• implement generic attribute implementation #31 - time as custom attribute (alternative solution, superseded by this pr)
• Manage generic attributes #23 - custom attributes
• Small files / minimum allocation size issue #66 - small files are now not wasteful
• Is LFS_NAME_MAX 255 bytes a little larger? #49 - name_max is now configurable
• lfs_disk_entry #22 - alen questions
• Suitable for microcontroller onboard flash? #29 - nand devices (inline files can help)
• Discussion: littlefs on SPI NAND #11 - internal flash devices (inline files can help)
• Available space #45 - available space question

TODO

Biggest thing missing right now is documentation. And the commit history is a complete mess of terrible commit practice that I need to clean up. Also I intend to put together some strategy from upgrading devices with littlefs v1.

EDIT: All that remains now is 1. implement "migration" functionality and testing and 2. update documentation.

littlefs v2 is ready to merge!

• Fix big-endian support
• Fix littlefs-fuse support
• Pass testing
• Add new tests
• Create some performance measurements (optional)
• Create "migration" functionality to convert filesystems from v1
• Clean up commit history
• Update documentation with design changes

---

Thanks for everyone's help and input in getting here.

cc @guillaumerems, @dpgeorge, @rojer, @dannybenor, @davidsaada, @ARMmbed/mbed-os-storage, @kegilbert, @deepikabhavnani

[rojer]

@geky this is awesome! with inline files, it looks like lfs becomes viable for small fs-es. what about GC/wiping though?

[geky]

what about GC/wiping though?

I wasn't planning on it this release. Since we can add a wiping function on a minor release it's less of a priority. My goal is to get this set of changes out of the way before looking at the next steps.

[rojer]

@geky fair enough!

[FreddieChopin]

Now the superblock also tracks name max, which means you can push the RAM consumption of the lfs_stat struct down and still be portable.

If you configured (in your code) that max length of name is (say) 256 bytes, but the file system you mount has just (say) 100 stored in the superblock, would that work? Or do you have to have them equal? If they have to be equal, you would not be able to mount two different file systems in the same application.

[geky]

If you configured (in your code) that max length of name is (say) 256 bytes, but the file system you mount has just (say) 100 stored in the superblock, would that work?

Yep, this was the specific problem I was trying to solve. So now if you mount a filesystem with 100-byte filenames on a device that uses 256-byte filenames, littlefs will respect the 100-byte filenames and return LFS_ERR_NAMETOOLONG if you try to create a file that exceeds 100 bytes.

The only downside is you need to be careful about what settings you use to format. If you format with 256-byte filenames on a PC and then try to mount on the 100-byte filename device, mount will fail with LFS_ERR_INVAL (and a debug message saying file_max is too big).

[earlephilhower]

I've got a PR in the ESP8266 for a LittleFS filesystem extension, and it's been running great with v1 (master) but I'd like to move to V2 before we actually merge since inline small files are a big thing when you only have 512KB of flash for code + filesystem.

I upgraded the PR to v2-alpha, but I and am finding a failure on small file writes. The code runs fine on master, and it's really so trivial I don't understand what could be missing...

I just wrote a small MCVE (the only differences in v1 vs. v2 in the userland are in the cfg setup, as commented). Basically, it seems in V2 if I open a file in O_CREAT mode and write a small amount, then close it, the file is lost completely...

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>
#include "lfs.h"

uint32_t s_blockSize = 4096;
uint32_t s_flashSize = 128 *1024;
uint8_t s_flashmem[128 * 1024];

int lfs_flash_read(const struct lfs_config *c, lfs_block_t block, lfs_off_t off, void *buffer, lfs_size_t size)
{
  memcpy(buffer, &s_flashmem[0] + c->block_size * block + off, size);
  return 0;
}

int lfs_flash_prog(const struct lfs_config *c, lfs_block_t block, lfs_off_t off, const void *buffer, lfs_size_t size)
{
  memcpy(&s_flashmem[0] + block * c->block_size + off, buffer, size);
  return 0;
}

int lfs_flash_erase(const struct lfs_config *c, lfs_block_t block)
{
  memset(&s_flashmem[0] + block * c->block_size, 0, c->block_size);
  return 0;
}

int lfs_flash_sync(const struct lfs_config *c) {
  (void) c;
  return 0;
}

int main(void)
{
  lfs_t s_fs;
  struct lfs_config s_cfg;
  lfs_file_t file;
  char buff[5];

  memset(&s_fs, 0, sizeof(s_fs));
  memset(&s_cfg, 0, sizeof(s_cfg));
  s_cfg.read  = lfs_flash_read;
  s_cfg.prog  = lfs_flash_prog;
  s_cfg.erase = lfs_flash_erase;
  s_cfg.sync  = lfs_flash_sync;

  s_cfg.read_size = s_blockSize;
  s_cfg.prog_size = s_blockSize;
  s_cfg.block_size = s_blockSize;
  s_cfg.block_count = s_flashSize / s_blockSize;
  //s_cfg.lookahead = 128; // v1
  s_cfg.cache_size = s_blockSize; //v2
  s_cfg.lookahead_size = 128; // v2

  if (lfs_format(&s_fs, &s_cfg) < 0) { printf("format fail\n"); return 1; }
  if (lfs_mount(&s_fs, &s_cfg) < 0) { printf("mount fail\n"); return 2; }
  if (lfs_file_open(&s_fs, &file, "boot_count", LFS_O_RDWR | LFS_O_CREAT) < 0) { printf("open1 fail\n"); return 5; }
  if (lfs_file_write(&s_fs, &file, "test", 4) < 0) { printf("write fail\n"); return 6;}
  if (lfs_file_close(&s_fs, &file) < 0) { printf("close fail\n"); return 7; }
  if (lfs_file_open(&s_fs, &file, "boot_count", LFS_O_RDWR) < 0) { printf("open2 fail\n"); return 8; }
  if (lfs_file_read(&s_fs, &file, &buff, 4) < 4) { printf("read fail\n"); return 9;}
  buff[4]=0;
  printf("'%s'\n", buff);
  return 0;
}

When compiled with GCC I can read the data in V1:

$ git checkout master
Switched to branch 'master'
$ gcc -o test test.c lfs.c  lfs_util.c 
$ /test
'test'

and in V2-alpha the file reopen fails

$ git checkout v2-alpha
Switched to branch 'v2-alpha'
Your branch is up to date with 'origin/v2-alpha'.

$ gcc -o test test.c lfs.c  lfs_util.c 
$ ./test
open2 fail

Have others seem similar behavior, or am I doing something simply wrong here?

Thx
-EFP3

[geky]

Hi @apmorton, sorry about the late response, I didn't realize I never responded to this comment.

Only partially related, but after looking at the code I noticed that LFS_MKTAG is potentially unsafe if used with untrusted input.

#define LFS_MKTAG(type, id, size)
(((lfs_tag_t)(type) << 20) | ((lfs_tag_t)(id) << 10) | (lfs_tag_t)(size))

if the type/id/size values given to LFS_MKTAG are wider than their respective fields in the tag structure you can potentially produce an invalid tag value.

This is actually used (abused?) to create signed tags that can be summed with tags to modify specific fields. This was useful for changing ids in the tag be relative offsets.
https://github.com/ARMmbed/littlefs/blob/a32be1d875673c3c974fb96b57a1075acfa0ff0d/lfs.c#L546-L547

You're right this may be less safe. It may be better to have a sort of LFS_MKSTAG specifically for this special situation.

That being said, it did look like it would require a bit of work, so I'm going to consider it low priority until v2 is released. I may circle back around to look at it later. Feel free to create an issue or a PR for a safer LFS_MKTAG.

---

@earlephilhower, I'm looking into your issue now, sorry about the long delay. Thanks for the valuable MCVE!

[geky]

Thanks @earlephilhower for the bug report. I've pushed up a fix, but let me know if there's still issues.

Turns out the problem was prog_size > 1024 (the tag size limit). littlefs wasn't properly handling this and causing a crash.

I haven't heard of any storage devices that have program sizes that large, but it doesn't hurt to handle this case better. I've modified this situation so it will treat any prog_size > 1024 as prog_size = block_size. This isn't perfect, but allows littlefs to work with large prog_sizes.

It's possible to use multiple commits to pad the prog_size and avoid the tag size limit, but I will consider this up in the air for a future improvement if we start seeing many devices with >1024 prog_sizes.

[geky]

Sorry about the delays, my absence, and completely missing the target goal of this release. From what I've learned future releases will be very different.

Fortunately, I'm not dead, and hopefully I will have more time to work on littlefs, though I have quite a bit of backlog to work through. Sorry if I have yet to get to your open issues.

With that out of the way, littlefs v2 is ready to merge! Baring emergencies, I will be releasing v2 tomorrow.

Documentation is complete, with an updated DESIGN.md and SPEC.md.

As a part of this release, a best-effort migration function is available that can convert a v1 filesystem to v2 filesystem in most cases. If it fails, an error is returned and the v1 filesystem is left unmodified. More info here, and I will be documenting this as a part of the v2 release notes in GitHub.

Also, following the discussion on #127, releases now have three special branch/tags that are generated as a part of CI:

• vN.N.N (ie v1.7.2), a patch-specific tag
• vN (ie v1), a major version branch that is updated every patch release
• vN-prefix (ie v1-prefix), a major version branch updated every patch release that is modified to avoid name conflicts with other versions of littlefs.

I believe most v2 specific issues that have been raised are now fixed on the v2-alpha branch. Issues that are present in both v1 and v2 can wait until after merging v2.

As always, feel free to leave any feedback, feedback is always useful. And thanks for the support so far.

[geky]

I've thrown together a last minute comparison of the relative performance of littlefs v1 and v2. This was built by simulating both v1 and v2 locally and measuring the number of bytes read/prog/erased as well as the total size of the filesystem at the end. I then divided this result by the count*size of the files in the test to get a multiplicative cost, which is a bit easier to compare. Smaller numbers are better.

As expected, the performance does not change much for reading (and even gets worse on NAND). However, the prog/erase performance is much better. This is desired as erasing has a much higher runtime penalty than reading, sometimes even ~100x the cost (citation needed).

There's also a moderate improvement to storage consumption thanks to inline files, though note that the storage consumption converges as the file size increases.

Also note that these are on the relatively small scale of storage.

---

NOR flash

---

NAND flash

---

MCU internal flash

---

SD/eMMC

[geky]

Ok, v2 merge time!

[earlephilhower]

Thanks, @geky. Your fix worked wonders on the 8266 port and now we're back in action. Appreciate your help on it!

I will re-check if we might use a smaller program size since it seems I'm doing something a little different than your general use case.