Cross platform determinism issue (Windows - Linux) #1413
Replies: 3 comments 11 replies
-
|
Hello, These things are very hard to debug. What I do is to enable This will print out all float values in hexadecimal format so that it is easy to spot 1 bit differences. I then use a text diff application to compare the text streams between platforms. When I see a difference, I add more Specifically for |
Beta Was this translation helpful? Give feedback.
-
|
B.t.w. If you have a repro case that you can e.g. insert into the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for the help. I'm currently trying to enable I don't think Reproducing the issue in the Hello World example would be very interesting, but honestly, I find it strange that you haven't encountered a determinism issue before. Therefore, I tend to believe that the problem is somehow caused by my setup or configuration flags, and I believe it would work fine in the Hello World example. I'll keep trying to reproduce it at some point, but in the meantime, I wonder if you see anything unusual in my CMake configuration or Virtual Character processing. |
Beta Was this translation helpful? Give feedback.
-
|
CharacterVirtual is currently not being tested as part of the cross platform determinism checks, so I wouldn't be surprised if it has an issue. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @jrouwe, I have been able to collect some more information and verify that the determinism issue occurs on a single byte, specifically related to the contact "distance." Following is the initial state received from the linux server and before processing Here the State after processing As you can see, the VirtualCharacter state generated by the Linux server has the contact I'll keep investigating and reporting any updates. If anything comes to mind or you need anything from my side, let me know. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, let me investigate that, thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for recommending to dig deeper. I've actually found that the problem is not the cooker but the unit conversion from centimeters (Unreal Engine unit) to meters (Jolt). UnitScale::UnrealToJolt(ConvertR(InitialTransform.GetTranslation()));
// Pos: X: 0xc01ed261e0fd2dcd Y: 0xbff09172070ee60a Z: 0x403a0d6327f81d95
inline JPH::RVec3 ConvertR(const FVector& V)
{
return JPH::RVec3(V.X, V.Z, V.Y);
}
template <class T>
inline T UnrealToJolt(T D)
{
D /= 100.0;
return D;
}The function The interesting part is that the issue only occurs when everything is written in a single line, as shown above. When I split that line into three separate lines like below, the produced value is consistent with what I get on the in editor build and linux. UE_LOG(LogTemp, Error, TEXT("------------> X: %s, Y: %s, Z: %s"), UTF8_TO_TCHAR(FLASH::float_to_hex_string(v.X).c_str()), UTF8_TO_TCHAR(FLASH::float_to_hex_string(v.Y).c_str()), UTF8_TO_TCHAR(FLASH::float_to_hex_string(v.Z).c_str()));
const JPH::RVec3 v2 = ConvertR(v);
UE_LOG(LogTemp, Error, TEXT("------------> %s"), UTF8_TO_TCHAR(FLASH::to_hex_string(v2).c_str()));
const JPH::RVec3 v3 = UnitScaler::UnrealToJolt(v2);
UE_LOG(LogTemp, Error, TEXT("------------> %s"), UTF8_TO_TCHAR(FLASH::to_hex_string(v3).c_str()));
const JPH::RVec3 Position = UnitScaler::UnrealToJolt(ConvertR(InitialTransform.GetTranslation()));
// Pos: X: 0xc01ed261e0fd2dcd Y: 0xbff09172070ee60a Z: 0x403a0d6327f81d94I think that the optimization level used to compile the game during cooking on Windows might change the way the division is performed, leading to this discrepancy. I've tried wrapping the |
Beta Was this translation helpful? Give feedback.
-
|
Actually, Ported version of JPH_PRECISE_MATH_ON// Ported from Jolt
// Macro to enable floating point precise mode and to disable fused multiply add instructions
#if defined(JPH_COMPILER_GCC)
#define FLASH_PRECISE_MATH_ON \
_Pragma("GCC push_options") \
_Pragma("GCC optimize(\"-fno-fast-math\")") \
_Pragma("GCC optimize(\"-fno-fp-contract\")")
#define FLASH_PRECISE_MATH_OFF \
_Pragma("GCC pop_options")
#elif defined(JPH_COMPILER_CLANG)
// We compile without -ffast-math because pragma float_control(precise, on) doesn't seem to actually negate all of the -ffast-math effects and causes the unit tests to fail (even if the pragma is added to all files)
// On clang 14 and later we can turn off float contraction through a pragma (before it was buggy), so if FMA is on we can disable it through this macro
#if (defined(JPH_CPU_ARM) && !defined(JPH_PLATFORM_ANDROID) && __clang_major__ >= 16) || (defined(JPH_CPU_X86) && __clang_major__ >= 14)
#define FLASH_PRECISE_MATH_ON \
_Pragma("float_control(precise, on, push)") \
_Pragma("clang fp contract(off)")
#define FLASH_PRECISE_MATH_OFF \
_Pragma("float_control(pop)")
#elif __clang_major__ >= 14 && (defined(JPH_USE_FMADD) || defined(FP_FAST_FMA))
#define FLASH_PRECISE_MATH_ON \
_Pragma("clang fp contract(off)")
#define FLASH_PRECISE_MATH_OFF \
_Pragma("clang fp contract(on)")
#else
#define FLASH_PRECISE_MATH_ON
#define FLASH_PRECISE_MATH_OFF
#endif
#elif defined(JPH_COMPILER_MSVC)
// Unfortunately there is no way to push the state of fp_contract, so we have to assume it was turned on before FLASH_PRECISE_MATH_ON
#define FLASH_PRECISE_MATH_ON \
__pragma(float_control(precise, on, push)) \
__pragma(fp_contract(off))
#define FLASH_PRECISE_MATH_OFF \
__pragma(fp_contract(on)) \
__pragma(float_control(pop))
#else
#error Undefined
#endifI'm now conducting more tests to verify that everything else is working fine. Please let me know if you think there is a better way to handle this |
Beta Was this translation helpful? Give feedback.
-
|
I would expect that if you write your function as: that you don't need any of this. With |
Beta Was this translation helpful? Give feedback.
-
|
That's nice, thanks! |
Beta Was this translation helpful? Give feedback.
-
Hello,
I am encountering a problem in maintaining Jolt determinism between two different platforms, Windows and Linux, where the client runs the game on Windows and the server instead on Linux.
After several tests and having disabled all the game code to ensure that the determinism issue is not caused by the game, updating Jolt to the latest MASTER version, and making sure not to use SIN/COS/ATAN/sort, I have observed that Jolt does not behave deterministically. A small note: when both the client and the server use Windows as the platform, everything works perfectly.
To provide some context, the game is implemented in Unreal Engine, and Jolt is compiled as part of a CMake library that contains both the gameplay code and the state synchronization code alongside Jolt. This library is statically linked to various Unreal Engine modules (DLLs), which read states and positions to update the positions of the various objects on the screen. This setup ensures that all the gameplay code remains separate from Unreal Engine, allowing complete control over the execution flow as well as the compile flags, enabling the setting of various flags like /fp:precise, etc.
The synchronization problem appears somewhat everywhere, but in my test setup, I am focusing on the VirtualCharacter (Kinematic). What I do is process the character with hardcoded velocity and direction to ensure there are no external interferences. Nevertheless, consistently, the buffer produced by the client differs from the buffer produced by the server.
Have you had similar experiences?
What can I do to identify the problem?
Here is the CMake code I use to compile Jolt:
VirtualCharacter processing CPP
Beta Was this translation helpful? Give feedback.
All reactions