circular import error

[jmlemetayer]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I am cross-compiling memray for an arm target using Yocto. The build went fine, but when I start memray I got an error:

ImportError: cannot import name 'FileFormat' from partially initialized module 'memray._memray' (most likely due to a circular import) (/usr/lib/python3.10/site-packages/memray/_memray.cpython-310-arm-linux-gnuea
bi.so)

The full stack is here:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/memray/__init__.py", line 2, in <module>
    from ._memray import AllocationRecord
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1184, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "src/memray/_memray.pyx", line 75, in init memray._memray
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/memray/_metadata.py", line 4, in <module>
    from ._memray import FileFormat
ImportError: cannot import name 'FileFormat' from partially initialized module 'memray._memray' (most likely due to a circular import) (/usr/lib/python3.10/site-packages/memray/_memray.cpython-310-arm-linux-gnuea
bi.so)

Expected Behavior

No ImportError.

Steps To Reproduce

1. My arm target is running Python 3.10.15
2. Then I just have to memray -h to raise the error

Memray Version

Tried 1.14.0 and 1.15.0

Python Version

3.10

Operating System

Linux

Anything else?

I updated the file _metadata.py:

 from dataclasses import dataclass
 from datetime import datetime

-from ._memray import FileFormat


 @dataclass 
 class Metadata:
     start_time: datetime
     end_time: datetime
     total_allocations: int
     total_frames: int
     peak_memory: int
     command_line: str
     pid: int
     main_thread_id: int
     python_allocator: str
     has_native_traces: bool
     trace_python_allocators: bool
-    file_format: FileFormat
+    file_format: "FileFormat"

This fix the issue on my device.

I don't know why but it is working on my PC without the fix (Python 3.12.7).

[pablogsal]

Unfortunately we cannot reproduce this in our side. Can you please give us detailed steps to get the error you are getting?

We build memray every day in our computers and in GitHub actions in many different platforms and this never appears so clearly there is something peculiar with your setup that's causing this. Without being able to do it on our side we won't be able to understand what's going on.

[jmlemetayer]

Yeah this is more an informational post than a real bug.

But the circular import can be seen easily in the code:

1. _memray.pyx line 75: from ._metadata import Metadata
2. _metadata.py line 4: from ._memray import FileFormat

I think it only works because the imports are “type hints”, but for some reason the python on my device doesn't recognize them as such.

[jmlemetayer]

Closing this issue for now. Feel free to reopen if necessary.

[godlygeek]

I'm gonna reopen this, actually. I'm not really sure why this is happening only for you, but it's probably a good idea to break this circular import anyway...

@jmlemetayer Can you confirm that the issue doesn't occur if you make this patch, instead?

diff --git a/src/memray/_metadata.py b/src/memray/_metadata.py
index 5d5913b4..f61af98f 100644
--- a/src/memray/_metadata.py
+++ b/src/memray/_metadata.py
@@ -1,7 +1,9 @@
+import typing
 from dataclasses import dataclass
 from datetime import datetime

-from ._memray import FileFormat
+if typing.TYPE_CHECKING:
+    from ._memray import FileFormat


 @dataclass

[jmlemetayer]

Unfortunately this is not working:

$  memray -h
Traceback (most recent call last):
  File "/usr/bin/memray", line 5, in <module>
    from memray.__main__ import main
  File "/usr/lib/python3.10/site-packages/memray/__init__.py", line 2, in <module>
    from ._memray import AllocationRecord
  File "src/memray/_memray.pyx", line 75, in init memray._memray
  File "/usr/lib/python3.10/site-packages/memray/_metadata.py", line 10, in <module>
    class Metadata:
  File "/usr/lib/python3.10/site-packages/memray/_metadata.py", line 22, in Metadata
    file_format: FileFormat
NameError: name 'FileFormat' is not defined

---

As I said, I am using Yocto to build the firmware for my device. The recipe can be seen here:
https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/python/python3_3.10.15.bb?h=kirkstone

I think this is due to a configuration problem. It seems that in my case, pgo and lto are disabled. I don't know if this is relevant or not...

[godlygeek]

Unfortunately this is not working:

Er, oops, right. Can you instead try:

diff --git a/src/memray/_metadata.py b/src/memray/_metadata.py
index 5d5913b4..96d6416b 100644
--- a/src/memray/_metadata.py
+++ b/src/memray/_metadata.py
@@ -1,7 +1,11 @@
+from __future__ import annotations
+
+import typing
 from dataclasses import dataclass
 from datetime import datetime

-from ._memray import FileFormat
+if typing.TYPE_CHECKING:
+    from ._memray import FileFormat


 @dataclass

Same as before, but with the addition of from __future__ import annotations at the top (we're already using that in other modules)

[godlygeek]

I think this is due to a configuration problem. It seems that in my case, pgo and lto are disabled. I don't know if this is relevant or not...

That should not make a difference, no...

[jmlemetayer]

Yes, the last fix is working 👍