【Operator Mechanism】Pr68945 _C_ops.c_concat in dynamic graph bug fix #70870
Conversation
Function-Samuel
force-pushed
the
pr_69951
branch
2 times, most recently
from
305f5af to
cfe9a24
Compare
| @@ -517,6 +517,13 @@ def source_include(header_file_path, fw_header_file_path): | |||
| #include "paddle/phi/api/profiler/event_tracing.h" | |||
| #include "paddle/phi/api/profiler/supplement_tracing.h" | |||
| #include "paddle/phi/core/platform/device_context.h" | |||
There was a problem hiding this comment.
为什么需要引用paddle/phi/core/platform/device_context.h?
There was a problem hiding this comment.
为什么需要引用
paddle/phi/core/platform/device_context.h?
当时在dist_api_gen.py里用了device_context就下意识引进来了，刚刚看了看代码确实可以不引入，之后会把这里去掉重提
| @@ -517,6 +517,13 @@ def source_include(header_file_path, fw_header_file_path): | |||
| #include "paddle/phi/api/profiler/event_tracing.h" | |||
| #include "paddle/phi/api/profiler/supplement_tracing.h" | |||
| #include "paddle/phi/core/platform/device_context.h" | |||
| #include "paddle/phi/core/distributed/store/store_utils.h" | |||
There was a problem hiding this comment.
store_utils.h引用加到PADDLE_WITH_DISTRIBUTE宏下面
There was a problem hiding this comment.
store_utils.h引用加到PADDLE_WITH_DISTRIBUTE宏下面
done
| if self.kernel['func'][0] == 'c_concat': | ||
| if_condition_code = ( | ||
| if_condition_code | ||
| + '\n' | ||
| + self.generate_op_comm_init_code() | ||
| + '\n' | ||
| + self.generate_nccl_commcontext_init_code() | ||
| ) |
There was a problem hiding this comment.
为什么c_concat需要特判？是否可以做到不特判
There was a problem hiding this comment.
为什么
c_concat需要特判？
之前提过一个大范围的不加特判的pr，那个提上去以后说推理组那边会hang住，和ld讨论以后这次先提个范围小点的，推理那边没问题后再提大范围的
There was a problem hiding this comment.
这种特判需要尽可能避免，过多的特判逻辑会导致代码生成没法维护
There was a problem hiding this comment.
这种特判需要尽可能避免，过多的特判逻辑会导致代码生成没法维护
只有这一个，因为这个需求比较急，推理那边赶着用，那边用下来没问题的话会把这个特判删掉改成大范围的
There was a problem hiding this comment.
去掉特判后遇到的问题要怎么处理？
像pr70352这样#70352 ，只有ring_id属性的算子才加入初始化
| @@ -860,6 +896,15 @@ def process_data_type_args(args_item): | |||
| input_args=input_args, mesh=mesh, kernel_code=kernel_select_code | |||
| ) | |||
|
|
|||
| if self.kernel['func'][0] == 'c_concat': | |||
There was a problem hiding this comment.
这里删掉 if 判断，测试有问题吗？
test_margin_cross_entropy_op有问题，不过是环境变量未设置，正在修复
Function-Samuel
dismissed stale reviews from zyfncg and zhangbo9674
via
c2ab818
Function-Samuel
force-pushed
the
pr_69951
branch
from
cfe9a24 to
c2ab818
Compare
| nullptr, | ||
| common::errors::Unavailable( | ||
| "NCCLCommContext is nullptr, collective op should " | ||
| "has ring_id attr.")); |
| True, | ||
| ) | ||
| elif in_pir_mode(): | ||
| if in_dynamic_mode() or in_pir_mode(): |
There was a problem hiding this comment.
-> in_dynamic_or_pir_mode()
| @@ -81,6 +81,45 @@ | |||
| }} | |||
| """ | |||
|
|
|||
| OP_COMM_INIT = """ | |||
There was a problem hiding this comment.
OP_COMM_INIT -> NCCL_COMMCONTEXT_INIT
| @@ -1310,6 +1364,12 @@ def generate_kernel_selection_code(self) -> str: | |||
| self.api, self.kernel['func'][0], self.kernel['func'][0] | |||
| ) | |||
|
|
|||
| def generate_op_comm_init_code(self) -> str: | |||
Function-Samuel
force-pushed
the
pr_69951
branch
from
c2ab818 to
253a352
Compare
PR Category
Operator Mechanism
PR Types
Bug fixes
Description
修复了Pr68945新动态图模式下调用_C_ops.c_concat的通信初始化与精度错误
PCard-85979