Different kernels on different processors

[colinjcotter]

The following code, when executed on 2 MPI processes, produces different kernels on each process. This causes a hang, when PYOP2_DEBUG is not set to 1.

https://gist.github.com/anonymous/bdcb8f0e7f0d61c7cc26

[wence-]

Note that you can remove the from upwindDG_flux import DGstep line to actually run this code.

[colinjcotter]

I boiled it down a bit more:
https://gist.github.com/colinjcotter/976281a53aa0dd3688ad

[FabioLuporini]

Thanks. If I try to execute this now, I get a PETSc error:

Traceback (most recent call last):
  File "issue_50_association.py", line 30, in <module>
    solve(phi*p*detJ*dx == phi*Dexpr*detJ*dx, D0)
  File "/homes/fl1612/Projects/Firedrake/firedrake/firedrake/solving.py", line 119, in solve
    _solve_varproblem(*args, **kwargs)
  File "/homes/fl1612/Projects/Firedrake/firedrake/firedrake/solving.py", line 146, in _solve_varproblem
    solver.solve()
  File "<string>", line 2, in solve
  File "/homes/fl1612/Projects/Firedrake/PyOP2/pyop2/profiling.py", line 203, in wrapper
    return f(*args, **kwargs)
  File "/homes/fl1612/Projects/Firedrake/firedrake/firedrake/variational_solver.py", line 161, in solve
    self.snes.solve(None, v)
  File "PETSc/SNES.pyx", line 516, in petsc4py.PETSc.SNES.solve (src/petsc4py.PETSc.c:161864)
petsc4py.PETSc.Error: error code 71
[0] SNESSolve() line 3852 in /tmp/pip-qqyOTZ-build/src/snes/interface/snes.c
[0] SNESSolve_KSPONLY() line 43 in /tmp/pip-qqyOTZ-build/src/snes/impls/ksponly/ksponly.c
[0] KSPSolve() line 542 in /tmp/pip-qqyOTZ-build/src/ksp/ksp/interface/itfunc.c
[0] KSPSetUp() line 330 in /tmp/pip-qqyOTZ-build/src/ksp/ksp/interface/itfunc.c
[0] PCSetUp() line 918 in /tmp/pip-qqyOTZ-build/src/ksp/pc/interface/precon.c
[0] PCSetUp_ILU() line 232 in /tmp/pip-qqyOTZ-build/src/ksp/pc/impls/factor/ilu/ilu.c
[0] MatLUFactorNumeric() line 2945 in /tmp/pip-qqyOTZ-build/src/mat/interface/matrix.c
[0] MatLUFactorNumeric_SeqAIJ() line 566 in /tmp/pip-qqyOTZ-build/src/mat/impls/aij/seq/aijfact.c
[0] MatPivotCheck() line 664 in /tmp/pip-qqyOTZ-build/include/petsc-private/matimpl.h
[0] MatPivotCheck_none() line 645 in /tmp/pip-qqyOTZ-build/include/petsc-private/matimpl.h
[0] Zero pivot in LU factorization: http://www.mcs.anl.gov/petsc/documentation/faq.html#ZeroPivot
[0] Zero pivot row 7 value 0 tolerance 2.22045e-14

[FabioLuporini]

I guess this is now fixed?

[FabioLuporini]

I really guess so, so closing