add fixed receive source order in recvAndUnpack call
I have observed non-deterministic behaviour when using the parallel AMG preconditioner. I tracked down the source of this issue to IndicesSyncer, which is used to coarsen the index sets when setting up AMG. Because floating point operations are non-associative, one must expect some non-determinism in parallel (scalar products), but for my case the problem this MR tries to fix caused large variations in number of BiCGSTAB iterations (10%-20%) between runs. With the changes in this MR the variations disappear.
The reason for the non-deterministic behaviour is that the ordering of non-owned vertices (or matrix rows/DoFs) depend on which message is received first by each process in the IndicesSyncer. On each process a matrix on the following form is created for every level in the AMG hierarchy:
A_{local}=\left[ {\begin{array}{cc}
A_{OO} & A_{OG} \\
0 & 1 \\
\end{array} }
\right]
When the sparsity pattern of A_{local}
is created on each level and process, the parallel index set is used. The index set order of the non-owned vertices is set in the IndicesSyncer, and changing the order of the non-owned vertices changes the order of the entries in A_{OG}
, which is the part of the matrix relating owned unknowns to non-owned(=ghost) unknowns. Non-determinism is then caused by non-associative floating point operations when applying the smoother.
One way of making the non-owned unknowns having the same local ordering every run, is to force each process to receive its messages in a fixed order in the IndicesSyncer communication (recvAndUnpack method). This is what is done in this MR. Every process now receives its messages in the same order that it sends messages to other processes.