[petsc-users] Slow MatAssembly and MatMat Mult

[Bikash Kanungo]

Hi,

I have two small (2500x2500) matrices parallelized across 480 processors.
One of them is an MPIAIJ matrix while the other is an MPIDENSE matrix. I
perform a MatMatMult involving these two matrices. I tried these operations
on two machines, one is the local cluster at University of Michigan and the
other is the XSEDE Comet machine. The Comet machine takes 10-20 times more
time in steps involving MatAssembly and MatMatMult of the aforementioned
matrices. I have other Petsc MatMult operations in the same code involving
larger matrices (4 million x 4 million) which show similar timing on both
machines. It's just those small parallel matrices that are inconsistent in
terms of their timings. I used same the compilers and MPI libraries in both
the machines except that I have suppressed "avx2" flag in Comet. I believe
avx2 affects floating point operations and not communication. I would like
to know what might be causing these inconsistencies only in case of the
small matrices. Are there any network settings that I can look into and
compare?

Regards,
Bikash

-- 
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160217/283c47b0/attachment.html>