rm file causes program to crash

[dyergan]

Using EN25QX256A nor flash with system of rt-thread and IC is blueturn(8952F3)

Operation steps:
1.create thirty files,file name (1~30)
2.rm file,one by one.Occasionally the program will crash at this point. Cause lfs_dir_traverse() entering an infinite loop, stack overflow.
3.If stack overflow hanppen, even reboot or burn new program, it will stack overflow.
4.Create a new file, now rm is normal.

Does anyone have a solution?

init :

struct lfs_config cfg = {
    // block device operations
    .read = awi_lfs_read,
    .prog = awi_lfs_prog,
    .erase = awi_lfs_erase,
    .sync = awi_lfs_sync,

    // block device configuration
    .read_size = READ_SIZE,                      //16
    .prog_size = PROG_SIZE,                    //16
    .block_size = BLOCK_SIZE,                 //16
    .block_count = BLOCK_COUNT,        //4096
    .cache_size = CACHE_SIZE,                 //128
    .lookahead_size = LOOKAHEAD_SIZE,  //16
    .block_cycles = 500,

    .lookahead_buffer = lookahead_buffer,
    .read_buffer = read_buffer,
    .prog_buffer = prog_buffer,
};

[dyergan]

add some printf in lfs_dir_traverse:

static int lfs_dir_traverse(lfs_t *lfs,
        const lfs_mdir_t *dir, lfs_off_t off, lfs_tag_t ptag,
        const struct lfs_mattr *attrs, int attrcount,
        lfs_tag_t tmask, lfs_tag_t ttag,
        uint16_t begin, uint16_t end, int16_t diff,
        int (*cb)(void *data, lfs_tag_t tag, const void *buffer), void *data) {
    // This function in inherently recursive, but bounded. To allow tool-based
    // analysis without unnecessary code-cost we use an explicit stack
    WDT_CLR();
    struct lfs_dir_traverse stack[LFS_DIR_TRAVERSE_DEPTH-1];
    unsigned sp = 0;
    int res;

    // iterate over directory and attrs
    lfs_tag_t tag;
    const void *buffer;
    struct lfs_diskoff disk = {0};
    dir_times++;
    printf("line:%d times:%d\n", __LINE__, dir_times);
    while (true) {
		WDT_CLR();
        {
            printf("line:%d off+lfs_tag_dsize(ptag):%d dir->off: %d\n", __LINE__, off+lfs_tag_dsize(ptag), dir->off);
            if (off+lfs_tag_dsize(ptag) < dir->off) {
                off += lfs_tag_dsize(ptag);
                int err = lfs_bd_read(lfs,
                        NULL, &lfs->rcache, sizeof(tag),
                        dir->pair[0], off, &tag, sizeof(tag));
                if (err) {
                    return err;
                }

                tag = (lfs_frombe32(tag) ^ ptag) | 0x80000000;
                disk.block = dir->pair[0];
                disk.off = off+sizeof(lfs_tag_t);
                buffer = &disk;
                ptag = tag;
                printf("line:%d tag:%d  disk.block :%d dir->off: %d buffer :%d\n", __LINE__, tag, disk.block, dir->off, buffer);
            } else if (attrcount > 0) {
                tag = attrs[0].tag;
                buffer = attrs[0].buffer;
                attrs += 1;
                attrcount -= 1;
            } else {
                // finished traversal, pop from stack?
                res = 0;
                break;
            }

            // do we need to filter?
            lfs_tag_t mask = LFS_MKTAG(0x7ff, 0, 0);
            if ((mask & tmask & tag) != (mask & tmask & ttag)) {
                continue;
            }
            printf("line:%d lfs_tag_id(tmask):%d sp: %d\n", __LINE__, lfs_tag_id(tmask), sp);
            if (lfs_tag_id(tmask) != 0) {
                LFS_ASSERT(sp < LFS_DIR_TRAVERSE_DEPTH);
                // recurse, scan for duplicates, and update tag based on
                // creates/deletes
                stack[sp] = (struct lfs_dir_traverse){
                    .dir        = dir,
                    .off        = off,
                    .ptag       = ptag,
                    .attrs      = attrs,
                    .attrcount  = attrcount,
                    .tmask      = tmask,
                    .ttag       = ttag,
                    .begin      = begin,
                    .end        = end,
                    .diff       = diff,
                    .cb         = cb,
                    .data       = data,
                    .tag        = tag,
                    .buffer     = buffer,
                    .disk       = disk,
                };
                sp += 1;

                tmask = 0;
                ttag = 0;
                begin = 0;
                end = 0;
                diff = 0;
                cb = lfs_dir_traverse_filter;
                data = &stack[sp-1].tag;
                continue;
            }
        }

popped:
        // in filter range?
        printf("line:%d lfs_tag_id(tmask):%d\n", __LINE__, lfs_tag_id(tmask));
        if (lfs_tag_id(tmask) != 0 &&
                !(lfs_tag_id(tag) >= begin && lfs_tag_id(tag) < end)) {
            continue;
        }
        printf("line:%d lfs_tag_type3(tag)%d sp: %d\n", __LINE__, lfs_tag_type3(tag), sp);
        // handle special cases for mcu-side operations
        if (lfs_tag_type3(tag) == LFS_FROM_NOOP) {
            // do nothing
        } else if (lfs_tag_type3(tag) == LFS_FROM_MOVE) {
            // Without this condition, lfs_dir_traverse can exhibit an
            // extremely expensive O(n^3) of nested loops when renaming.
            // This happens because lfs_dir_traverse tries to filter tags by
            // the tags in the source directory, triggering a second
            // lfs_dir_traverse with its own filter operation.
            //
            // traverse with commit
            // '-> traverse with filter
            //     '-> traverse with move
            //         '-> traverse with filter
            //
            // However we don't actually care about filtering the second set of
            // tags, since duplicate tags have no effect when filtering.
            //
            // This check skips this unnecessary recursive filtering explicitly,
            // reducing this runtime from O(n^3) to O(n^2).
            if (cb == lfs_dir_traverse_filter) {
                continue;
            }

            // recurse into move
            stack[sp] = (struct lfs_dir_traverse){
                .dir        = dir,
                .off        = off,
                .ptag       = ptag,
                .attrs      = attrs,
                .attrcount  = attrcount,
                .tmask      = tmask,
                .ttag       = ttag,
                .begin      = begin,
                .end        = end,
                .diff       = diff,
                .cb         = cb,
                .data       = data,
                .tag        = LFS_MKTAG(LFS_FROM_NOOP, 0, 0),
            };
            sp += 1;

            uint16_t fromid = lfs_tag_size(tag);
            uint16_t toid = lfs_tag_id(tag);
            dir = buffer;
            off = 0;
            ptag = 0xffffffff;
            attrs = NULL;
            attrcount = 0;
            tmask = LFS_MKTAG(0x600, 0x3ff, 0);
            ttag = LFS_MKTAG(LFS_TYPE_STRUCT, 0, 0);
            begin = fromid;
            end = fromid+1;
            diff = toid-fromid+diff;
        } else if (lfs_tag_type3(tag) == LFS_FROM_USERATTRS) {
            for (unsigned i = 0; i < lfs_tag_size(tag); i++) {
				WDT_CLR();
                const struct lfs_attr *a = buffer;
                res = cb(data, LFS_MKTAG(LFS_TYPE_USERATTR + a[i].type,
                        lfs_tag_id(tag) + diff, a[i].size), a[i].buffer);
                if (res < 0) {
                    return res;
                }

                if (res) {
                    break;
                }
            }
        } else {
            res = cb(data, tag + LFS_MKTAG(0, diff, 0), buffer);
            if (res < 0) {
                return res;
            }

            if (res) {
                break;
            }
        }
    }
    
    printf("line:%d sp: %d\n", __LINE__, sp);
    if (sp > 0) {
        // pop from the stack and return, fortunately all pops share
        // a destination
        dir         = stack[sp-1].dir;
        off         = stack[sp-1].off;
        ptag        = stack[sp-1].ptag;
        attrs       = stack[sp-1].attrs;
        attrcount   = stack[sp-1].attrcount;
        tmask       = stack[sp-1].tmask;
        ttag        = stack[sp-1].ttag;
        begin       = stack[sp-1].begin;
        end         = stack[sp-1].end;
        diff        = stack[sp-1].diff;
        cb          = stack[sp-1].cb;
        data        = stack[sp-1].data;
        tag         = stack[sp-1].tag;
        buffer      = stack[sp-1].buffer;
        disk        = stack[sp-1].disk;
        sp -= 1;
        goto popped;
    } else {
        return res;
    }
}

nomal print:

line:931 times:1
line:935 off+lfs_tag_dsize(ptag):224 dir->off: 208
line:967 lfs_tag_id(tmask):0 sp: 0
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)1279 sp: 0
line:935 off+lfs_tag_dsize(ptag):224 dir->off: 208
line:1090 sp: 0
index del : 6

error print :

line:931 times:1
line:935 off+lfs_tag_dsize(ptag):4 dir->off: 4096
line:950 tag:-1880096760 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):1023 sp: 0
line:935 off+lfs_tag_dsize(ptag):16 dir->off: 4096
line:950 tag:-1609564136 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)513 sp: 1
line:935 off+lfs_tag_dsize(ptag):44 dir->off: 4096
line:950 tag:-2146427903 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)1 sp: 1
line:935 off+lfs_tag_dsize(ptag):49 dir->off: 4096
line:950 tag:-1608508408 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)514 sp: 1
line:935 off+lfs_tag_dsize(ptag):61 dir->off: 4096
line:950 tag:-2146426879 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)1 sp: 1
line:935 off+lfs_tag_dsize(ptag):66 dir->off: 4096
line:950 tag:-1608507384 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)514 sp: 1
line:935 off+lfs_tag_dsize(ptag):78 dir->off: 4096
line:950 tag:-2146425855 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)1 sp: 1
line:935 off+lfs_tag_dsize(ptag):83 dir->off: 4096
line:950 tag:-1608506360 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)514 sp: 1
line:935 off+lfs_tag_dsize(ptag):95 dir->off: 4096
line:950 tag:-2146424831 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)1 sp: 1
line:935 off+lfs_tag_dsize(ptag):100 dir->off: 4096
line:950 tag:-1608505336 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)514 sp: 1
line:935 off+lfs_tag_dsize(ptag):112 dir->off: 4096
line:950 tag:-2146423807 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)1 sp: 1
line:935 off+lfs_tag_dsize(ptag):117 dir->off: 4096
line:950 tag:-1608504312 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)514 sp: 1
line:935 off+lfs_tag_dsize(ptag):129 dir->off: 4096
line:950 tag:-2146422783 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)1 sp: 1
line:935 off+lfs_tag_dsize(ptag):134 dir->off: 4096
line:950 tag:-1608503288 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)514 sp: 1
line:935 off+lfs_tag_dsize(ptag):146 dir->off: 4096
line:950 tag:-2146421759 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)1 sp: 1
line:935 off+lfs_tag_dsize(ptag):151 dir->off: 4096
line:950 tag:-1608502264 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)514 sp: 1
line:935 off+lfs_tag_dsize(ptag):163 dir->off: 4096
line:950 tag:-2146420735 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)1 sp: 1
line:935 off+lfs_tag_dsize(ptag):168 dir->off: 4096
line:950 tag:-1608501240 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)514 sp: 1
line:935 off+lfs_tag_dsize(ptag):180 dir->off: 4096
line:950 tag:-2146419711 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)1 sp: 1
line:935 off+lfs_tag_dsize(ptag):185 dir->off: 4096
line:950 tag:-1608500216 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)514 sp: 1
line:935 off+lfs_tag_dsize(ptag):197 dir->off: 4096
line:950 tag:-2146434046 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
line:1004 lfs_tag_id(tmask):0
line:1009 lfs_tag_type3(tag)1 sp: 1
line:935 off+lfs_tag_dsize(ptag):203 dir->off: 4096
line:950 tag:-1608514552 disk.block :1 dir->off: 4096 buffer :78772
line:967 lfs_tag_id(tmask):0 sp: 1
............

[dyergan]

In picture , when rm two file, step of 1 ptag > dir->off 4096,now rm is normal.But step 2 ptag < dir->off, and dir->off is 4096 ,now rm entering an infinite loop.

[geky]

Hi @dyergan, thanks for creating an issue, sorry for the late response.

If you're experiencing a stack overflow, is it possible there is just not enough stack for littlefs to operate?

littlefs uses a bounded RAM, but it can still be relatively large. Currently CI says it needs ~1.5 KiB of stack.

In picture , when rm two file, step of 1 ptag > dir->off 4096,now rm is normal.But step 2 ptag < dir->off, and dir->off is 4096 ,now rm entering an infinite loop.

If there was a stack overflow, the in-RAM filesystem state could be corrupted. Or, it's possible that it's not an infinite loop, but just a very long running loop due to performance issues with littlefs and metadata compaction.