Reference management questions

[SinisterRectus]

I'm trying to wrap my head around how userdata references are managed in luv.

Test script:

local uv = require('luv')

local gc = {}
local names = setmetatable({}, {__mode = 'k'})
local line = string.rep('-', 80)

local function check()
  print('gc start')
  collectgarbage()
  collectgarbage()
  print('gc stop')
  uv.walk(function(handle)
    if names[handle] then
      print('handle exists: ', handle, names[handle], 'closing:', handle:is_closing())
    end
  end)
end

local function wrap(meta)
  if gc[meta] then return end
  gc[meta] = meta.__gc
  function meta:__gc()
    print('collecting:', self, names[self], 'closing:', self:is_closing())
    return gc[meta](self)
  end
end

local function create(fn, name, close)
  local handle = fn()
  print('initializing:', handle, name)
  names[handle] = name
  wrap(getmetatable(handle))
  if close then
    print('closing handle:', handle, name)
    handle:close()
  end
  return handle
end

print(line)

check()

local a = create(uv.new_tcp, 'A', true)  -- A - closed; referenced by lua
local b = create(uv.new_tcp, 'B', false) -- B - not closed; referenced by lua
create(uv.new_tcp, 'C', true)            -- C - closed; not referenced by lua
create(uv.new_tcp, 'D', false)           -- D - not closed; not referenced by lua

check() -- nothing is collected; all handles remain referenced by uv

print(line)

local t = uv.new_timer() -- timer initialized
t:start(100, 0, function()
  print('timer start')
  check() -- C is collected; A and C are unreferenced by uv; B and D remain unchanged
  print('timer stop')
end)
uv.run()

print(line)

check() -- no changes

print(line)

print('exit') -- handles A, B, and D are gc'd exit

gc start
gc stop
initializing:   uv_tcp_t: 0x7fffc6084a80        A
closing handle: uv_tcp_t: 0x7fffc6084a80        A
initializing:   uv_tcp_t: 0x7fffc6084bb0        B
initializing:   uv_tcp_t: 0x7fffc6084ce0        C
closing handle: uv_tcp_t: 0x7fffc6084ce0        C
initializing:   uv_tcp_t: 0x7fffc6084e10        D
gc start
gc stop
handle exists:  uv_tcp_t: 0x7fffc6084a80        A       closing:        true
handle exists:  uv_tcp_t: 0x7fffc6084bb0        B       closing:        false
handle exists:  uv_tcp_t: 0x7fffc6084ce0        C       closing:        true
handle exists:  uv_tcp_t: 0x7fffc6084e10        D       closing:        false
--------------------------------------------------------------------------------
timer start
gc start
collecting:     uv_tcp_t: 0x7fffc6084ce0        C       closing:        true
gc stop
handle exists:  uv_tcp_t: 0x7fffc6084bb0        B       closing:        false
handle exists:  uv_tcp_t: 0x7fffc6084e10        D       closing:        false
timer stop
--------------------------------------------------------------------------------
gc start
gc stop
handle exists:  uv_tcp_t: 0x7fffc6084bb0        B       closing:        false
handle exists:  uv_tcp_t: 0x7fffc6084e10        D       closing:        false
--------------------------------------------------------------------------------
exit
collecting:     uv_tcp_t: 0x7fffc6084e10        D       closing:        false
collecting:     uv_tcp_t: 0x7fffc6084bb0        B       closing:        false
collecting:     uv_tcp_t: 0x7fffc6084a80        A       closing:        true

---

Questions:

In the first GC pass, after handle initialization, what is keeping each userdata and handle alive? I would expect at least C to be collected here because it is both closed and unreferenced by Lua.

In the timer's GC pass, C is finally collected, but what changed between the two GC runs to affect this behavior?

What happens at exit so that the remaining handles are collected?

(I have considered that my debug code might be keeping references alive, but I see the results with a reduced test.)

[rphillips]

The reference should be in the registry. IIRC there is only one reference called luv_context.

for k, v in pairs(debug.getregistry()) do
  print(k,v)
end

[SinisterRectus]

Looks like there is more than that.

At the start, there is:

FILE*   table: 0x7fa0821e5cd0
_LOADED table: 0x7fa0821e3510
_LOADLIB        table: 0x7fa0821e48b0
_PRELOAD        table: 0x7fa0821e4d18

Requiring luv adds:

uv_stream       table: 0x7f8f26efa6f8
uv_idle table: 0x7f8f26ef60e0
uv_check        table: 0x7f8f26ef4698
uv_loop.meta    table: 0x7f8f26ef3c60
LOADLIB: ./luv.so       userdata: 0x7f8f26eed280
uv_handle       table: 0x7f8f26ef41d0
uv_async        table: 0x7f8f26eed290
uv_process      table: 0x7f8f26ef42f0
uv_fs_poll      table: 0x7f8f26ef54e0
luv_work_ctx    table: 0x7f8f26efb888
uv_poll table: 0x7f8f26eeb270
uv_req  table: 0x7f8f26ef4118
uv_thread       table: 0x7f8f26efb578
uv_dir  table: 0x7f8f26eebbe0
uv_signal       table: 0x7f8f26efa3a0
uv_udp  table: 0x7f8f26eeb958
uv_tty  table: 0x7f8f26eec270
uv_timer        table: 0x7f8f26ef99b0
luv_context     userdata: 0x7f8f26eed3b8
uv_fs_event     table: 0x7f8f26ef4c30
uv_prepare      table: 0x7f8f26eeb6d8
uv_pipe table: 0x7f8f26ef6400
uv_tcp  table: 0x7f8f26ef9330

---

After the handle initialization, they are indeed in there:

initializing:   uv_tcp_t: 0x7fffbe53ba80        A
closing handle: uv_tcp_t: 0x7fffbe53ba80        A
initializing:   uv_tcp_t: 0x7fffbe53bbb0        B
initializing:   uv_tcp_t: 0x7fffbe53bce0        C
closing handle: uv_tcp_t: 0x7fffbe53bce0        C
initializing:   uv_tcp_t: 0x7fffbe53be10        D
gc start
gc stop
registry:       2       uv_tcp_t: 0x7fffbe53bbb0        B
registry:       1       uv_tcp_t: 0x7fffbe53ba80        A
registry:       4       uv_tcp_t: 0x7fffbe53be10        D
registry:       3       uv_tcp_t: 0x7fffbe53bce0        C

Somewhere during loop execution (the timer callback is irrelevant), A and C are removed from the registry (I assume because they are closed). Only C is collected because the local reference still exists for A.

uv run
gc start
collecting:     uv_tcp_t: 0x7fffbe53bce0        C       closing:        true
gc stop
registry:       2       uv_tcp_t: 0x7fffbe53bbb0        B
registry:       4       uv_tcp_t: 0x7fffbe53be10        D

---

Looking at the code, I see that (after uv_run), the luv_close callback calls luv_unref_handle, which removes the userdata from the registry.

static int luv_close(lua_State* L) {
  uv_handle_t* handle = luv_check_handle(L, 1);
  if (uv_is_closing(handle)) {
    luaL_error(L, "handle %p is already closing", handle);
  }
  if (!lua_isnoneornil(L, 2)) {
    luv_check_callback(L, (luv_handle_t*)handle->data, LUV_CLOSED, 2);
  }
  uv_close(handle, luv_close_cb);
  return 0;
}

static void luv_close_cb(uv_handle_t* handle) {
  lua_State* L;
  luv_handle_t* data = (luv_handle_t*)handle->data;
  if (!data) return;
  L = data->ctx->L;
  luv_call_callback(L, data, LUV_CLOSED, 0);
  luv_unref_handle(L, data);
}

static void luv_unref_handle(lua_State* L, luv_handle_t* data) {
  luaL_unref(L, LUA_REGISTRYINDEX, data->ref);
  luaL_unref(L, LUA_REGISTRYINDEX, data->callbacks[0]);
  luaL_unref(L, LUA_REGISTRYINDEX, data->callbacks[1]);
}

Is there a reason why the handle is not unreferenced in luv_close instead of luv_close_cb?

[squeek502]

Is there a reason why the handle is not unreferenced in luv_close instead of luv_close_cb?

Things can happen between luv_close and luv_close_cb. See point 2 here: #437

This whole thing is something I was trying to wrap my head around as well and didn't get very far. One conclusion I sort-of-came-to is that I don't think luv currently has the necessary separation between 'things that should outlive the Lua VM (cleaned up by libuv)' and 'things that should be cleaned up by Lua' (also related to #437). We have luv_newuserdata which I have commented locally with:

// Note: this allows the handle to outlive lua_close/gc?
// since only the pointer to the heap-allocated chunk
// is actually tracked by Lua
static void* luv_newuserdata(lua_State* L, size_t sz) {
  void* handle = malloc(sz);
  if (handle) {
    *(void**)lua_newuserdata(L, sizeof(void*)) = handle;
  }
  return handle;
}

but I think we should be thinking about this case more when allocating things.

[SinisterRectus]

Okay so it's not just me going down this rabbit hole. I got the answer to my questions though: the userdata isn't collected because it's still in the registry. That at least explains the behavior, even if it is janky. I'll close this and have a look at #437. Thanks guys.