[petsc-users] Order of PetscRandom and SVD
Torquil Macdonald Sørensen
torquil at gmail.com
Wed Jul 1 03:32:36 CDT 2015
Thanks, using SPRNG seems to fix the problem, so I'll be using that
solution.
However, the fact that using SLEPc will cause an existing PetscRandom
object of the default type on PETSC_COMM_WORLD to start producing
identical numbers on all MPI ranks seems like a bug to me.
After invoking SVDSetFromOptions(svd), the chain of random numbers on
all ranks is the one that would previously occur only on rank 0, except
that it is skipping every N-th number, where N is the number of MPI
ranks. So to me it seems that the SLEPc function is leaving the
PetscRandom object in an inconsistent state.
Best regards,
Torquil Sørensen
On 01/07/15 02:28, Barry Smith wrote:
> It looks like all three random number generators that PETSc use keep their "state" in some global data structure which means that using the random number generator to get the second set of values will be affected by the first set of values. So the trick would be to use a different random number generator for your code then that SLEPc uses. My guess is that SLEPc uses PETSCRAND48 so you should run ./configure with --download-sprng and then after you create your PetscRandom object call PetscRandomSetType(r,PETSCSPRNG)
>
> Barry
>
>
>> On Jun 30, 2015, at 5:42 PM, Torquil Macdonald Sørensen <torquil at gmail.com> wrote:
>>
>> Thanks. So what can I do to create a PetscRandom object that is not
>> affected by the SLEPc-functions that I'm calling?
>>
>> Best regards,
>> Torquil Sørensen
>>
>> On 30/06/15 23:43, Jose E. Roman wrote:
>>> SLEPc solvers use a PetscRandom object internally, and may generate a different sequence of random values in each process.
>>> Jose
>>>
>>>
>>>
>>>> El 30/6/2015, a las 23:05, Torquil Macdonald Sørensen <torquil at gmail.com> escribió:
>>>>
>>>> Hi!
>>>>
>>>> I'm experiencing some problems using PetscRandom and SVD from SLEPc. For
>>>> some reason, the order of two seemingly unrelated parts of the program
>>>> below affects the random values in a vector. The test program below, run
>>>> with "mpiexec -n 2", will print a vector with two identical component
>>>> values. If I move the four lines of SVD code up above "PetscRandom r",
>>>> the two vector component are different, as expected. Why does this
>>>> matter, when the SVD code is unrelated to the PetscRandom r and Vec u?
>>>>
>>>> #include "petscdmda.h"
>>>> #include "slepcsvd.h"
>>>>
>>>> int main(int argc, char **argv)
>>>> {
>>>> PetscErrorCode ierr = SlepcInitialize(&argc, &argv, 0, 0);
>>>> CHKERRQ(ierr);
>>>>
>>>> DM da;
>>>> ierr = DMDACreate1d(PETSC_COMM_WORLD, DM_BOUNDARY_NONE, 2, 1, 1, 0,
>>>> &da); CHKERRQ(ierr);
>>>> ierr = DMSetFromOptions(da); CHKERRQ(ierr);
>>>>
>>>> Vec u;
>>>> ierr = DMCreateGlobalVector(da, &u); CHKERRQ(ierr);
>>>>
>>>> PetscRandom r;
>>>> ierr = PetscRandomCreate(PETSC_COMM_WORLD, &r); CHKERRQ(ierr);
>>>> ierr = PetscRandomSetFromOptions(r);
>>>>
>>>> SVD svd;
>>>> ierr = SVDCreate(PETSC_COMM_WORLD, &svd); CHKERRQ(ierr);
>>>> ierr = SVDSetFromOptions(svd); CHKERRQ(ierr);
>>>> ierr = SVDDestroy(&svd); CHKERRQ(ierr);
>>>>
>>>> ierr = VecSetRandom(u, r); CHKERRQ(ierr);
>>>> ierr = VecView(u, PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr);
>>>>
>>>> ierr = PetscRandomDestroy(&r); CHKERRQ(ierr);
>>>> ierr = VecDestroy(&u); CHKERRQ(ierr);
>>>> ierr = DMDestroy(&da); CHKERRQ(ierr);
>>>> ierr = SlepcFinalize(); CHKERRQ(ierr);
>>>>
>>>> return 0;
>>>> }
>>>>
>>>> Output of the program as written above:
>>>>
>>>> Vec Object: 2 MPI processes
>>>> type: mpi
>>>> Vec Object:Vec_0xd00a30_0 2 MPI processes
>>>> type: mpi
>>>> Process [0]
>>>> 0.720032 + 0.061794 i
>>>> Process [1]
>>>> 0.720032 + 0.061794 i
>>>>
>>>> Output of the program with the four lines of SVD code moved above the
>>>> line "PetscRandom r":
>>>>
>>>> Vec Object: 2 MPI processes
>>>> type: mpi
>>>> Vec Object:Vec_0x117aa30_0 2 MPI processes
>>>> type: mpi
>>>> Process [0]
>>>> 0.720032 + 0.061794 i
>>>> Process [1]
>>>> 0.541144 + 0.529699 i
>>>>
>>>> Other information:
>>>>
>>>> $ gcc --version
>>>> gcc (Debian 5.1.1-12) 5.1.1 20150622
>>>>
>>>> The PETSc version was obtained with the command:
>>>> git checkout 75ae60a9cdad23ddd49e9e39341b43db353bd940
>>>>
>>>> because the newest GIT version wouldn't compile. SLEPc is the newest
>>>> from the master branch. The -log_summary output of the program is:
>>>>
>>>> ---------------------------------------------- PETSc Performance
>>>> Summary: ----------------------------------------------
>>>>
>>>> ./randtest.exe on a arch-linux2-c-debug named lenovo with 2 processors,
>>>> by tmac Tue Jun 30 22:42:49 2015
>>>> Using Petsc Development GIT revision: v3.6-89-g75ae60a GIT Date:
>>>> 2015-06-29 18:37:23 -0500
>>>>
>>>> Max Max/Min Avg Total
>>>> Time (sec): 1.274e-02 1.00000 1.274e-02
>>>> Objects: 3.900e+01 1.00000 3.900e+01
>>>> Flops: 0.000e+00 0.00000 0.000e+00 0.000e+00
>>>> Flops/sec: 0.000e+00 0.00000 0.000e+00 0.000e+00
>>>> Memory: 1.273e+05 1.00000 2.546e+05
>>>> MPI Messages: 6.500e+00 1.00000 6.500e+00 1.300e+01
>>>> MPI Message Lengths: 3.600e+01 1.00000 5.538e+00 7.200e+01
>>>> MPI Reductions: 9.000e+01 1.00000
>>>>
>>>> Flop counting convention: 1 flop = 1 real number operation of type
>>>> (multiply/divide/add/subtract)
>>>> e.g., VecAXPY() for real vectors of length N
>>>> --> 2N flops
>>>> and VecAXPY() for complex vectors of length
>>>> N --> 8N flops
>>>>
>>>> Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages
>>>> --- -- Message Lengths -- -- Reductions --
>>>> Avg %Total Avg %Total counts
>>>> %Total Avg %Total counts %Total
>>>> 0: Main Stage: 1.2723e-02 99.9% 0.0000e+00 0.0% 1.300e+01
>>>> 100.0% 5.538e+00 100.0% 8.900e+01 98.9%
>>>>
>>>> ------------------------------------------------------------------------------------------------------------------------
>>>> See the 'Profiling' chapter of the users' manual for details on
>>>> interpreting output.
>>>> Phase summary info:
>>>> Count: number of times phase was executed
>>>> Time and Flops: Max - maximum over all processors
>>>> Ratio - ratio of maximum to minimum over all processors
>>>> Mess: number of messages sent
>>>> Avg. len: average message length (bytes)
>>>> Reduct: number of global reductions
>>>> Global: entire computation
>>>> Stage: stages of a computation. Set stages with PetscLogStagePush()
>>>> and PetscLogStagePop().
>>>> %T - percent time in this phase %F - percent flops in this
>>>> phase
>>>> %M - percent messages in this phase %L - percent message
>>>> lengths in this phase
>>>> %R - percent reductions in this phase
>>>> Total Mflop/s: 10e-6 * (sum of flops over all processors)/(max time
>>>> over all processors)
>>>> ------------------------------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> ##########################################################
>>>> # #
>>>> # WARNING!!! #
>>>> # #
>>>> # This code was compiled with a debugging option, #
>>>> # To get timing results run ./configure #
>>>> # using --with-debugging=no, the performance will #
>>>> # be generally two or three times faster. #
>>>> # #
>>>> ##########################################################
>>>>
>>>>
>>>>
>>>>
>>>> ##########################################################
>>>> # #
>>>> # WARNING!!! #
>>>> # #
>>>> # The code for various complex numbers numerical #
>>>> # kernels uses C++, which generally is not well #
>>>> # optimized. For performance that is about 4-5 times #
>>>> # faster, specify --with-fortran-kernels=1 #
>>>> # when running ./configure.py. #
>>>> # #
>>>> ##########################################################
>>>>
>>>>
>>>> Event Count Time (sec)
>>>> Flops --- Global --- --- Stage --- Total
>>>> Max Ratio Max Ratio Max Ratio Mess Avg len
>>>> Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
>>>> ------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> --- Event Stage 0: Main Stage
>>>>
>>>> VecView 1 1.0 1.0881e-03 1.2 0.00e+00 0.0 5.0e+00 8.0e+00
>>>> 1.5e+01 8 0 38 56 17 8 0 38 56 17 0
>>>> VecSet 1 1.0 1.5020e-05 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
>>>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
>>>> VecScatterBegin 1 1.0 2.1935e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
>>>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
>>>> VecScatterEnd 1 1.0 7.1526e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
>>>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
>>>> VecSetRandom 1 1.0 1.6928e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
>>>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
>>>> MatAssemblyBegin 1 1.0 8.4877e-05 1.6 0.00e+00 0.0 0.0e+00 0.0e+00
>>>> 2.0e+00 1 0 0 0 2 1 0 0 0 2 0
>>>> MatAssemblyEnd 1 1.0 6.7306e-04 1.0 0.00e+00 0.0 4.0e+00 4.0e+00
>>>> 1.4e+01 5 0 31 22 16 5 0 31 22 16 0
>>>> ------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> Memory usage is given in bytes:
>>>>
>>>> Object Type Creations Destructions Memory Descendants' Mem.
>>>> Reports information only for process 0.
>>>>
>>>> --- Event Stage 0: Main Stage
>>>>
>>>> Vector 7 7 11064 0
>>>> Vector Scatter 3 3 3280 0
>>>> Matrix 3 3 8064 0
>>>> Distributed Mesh 1 1 4972 0
>>>> Star Forest Bipartite Graph 2 2 1712 0
>>>> Discrete System 1 1 848 0
>>>> Index Set 6 6 4660 0
>>>> IS L to G Mapping 1 1 632 0
>>>> PetscRandom 3 3 1920 0
>>>> SVD Solver 1 1 984 0
>>>> EPS Solver 1 1 1272 0
>>>> Spectral Transform 1 1 784 0
>>>> Krylov Solver 1 1 1304 0
>>>> Preconditioner 1 1 848 0
>>>> Basis Vectors 3 3 3024 0
>>>> Direct Solver 2 2 2640 0
>>>> Region 1 1 648 0
>>>> Viewer 1 0 0 0
>>>> ========================================================================================================================
>>>> Average time to get PetscTime(): 9.53674e-08
>>>> Average time for MPI_Barrier(): 1.43051e-06
>>>> Average time for zero size MPI_Send(): 4.52995e-06
>>>> #PETSc Option Table entries:
>>>> -ksp_converged_reason
>>>> -log_summary
>>>> -objects_dump 0
>>>> -options_table
>>>> #End of PETSc Option Table entries
>>>> Compiled without FORTRAN kernels
>>>> Compiled with full precision matrices (default)
>>>> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8
>>>> sizeof(PetscScalar) 16 sizeof(PetscInt) 4
>>>> Configure options: --prefix=/home/tmac/usr/stow/petsc
>>>> --with-scalar-type=complex --with-shared-libraries=yes
>>>> --with-debugging=yes --download-superlu=yes --download-ptscotch=yes
>>>> --with-sowing=no --download-sowing=no
>>>> -----------------------------------------
>>>> Libraries compiled on Tue Jun 30 20:18:19 2015 on lenovo
>>>> Machine characteristics: Linux-4.0.0-2-amd64-x86_64-with-debian-stretch-sid
>>>> Using PETSc directory: /home/tmac/tmp/src/petsc
>>>> Using PETSc arch: arch-linux2-c-debug
>>>> -----------------------------------------
>>>>
>>>> Using C compiler: mpicc -fPIC -Wall -Wwrite-strings
>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 ${COPTFLAGS} ${CFLAGS}
>>>> Using Fortran compiler: mpif90 -fPIC -Wall -Wno-unused-variable
>>>> -ffree-line-length-0 -g -O0 ${FOPTFLAGS} ${FFLAGS}
>>>> -----------------------------------------
>>>>
>>>> Using include paths:
>>>> -I/home/tmac/tmp/src/petsc/arch-linux2-c-debug/include
>>>> -I/home/tmac/tmp/src/petsc/include -I/home/tmac/tmp/src/petsc/include
>>>> -I/home/tmac/tmp/src/petsc/arch-linux2-c-debug/include
>>>> -I/home/tmac/usr/stow/petsc/include -I/usr/lib/openmpi/include
>>>> -I/usr/lib/openmpi/include/openmpi
>>>> -----------------------------------------
>>>>
>>>> Using C linker: mpicc
>>>> Using Fortran linker: mpif90
>>>> Using libraries:
>>>> -Wl,-rpath,/home/tmac/tmp/src/petsc/arch-linux2-c-debug/lib
>>>> -L/home/tmac/tmp/src/petsc/arch-linux2-c-debug/lib -lpetsc
>>>> -Wl,-rpath,/home/tmac/usr/stow/petsc/lib -L/home/tmac/usr/stow/petsc/lib
>>>> -lsuperlu_4.3 -llapack -lblas -lptesmumps -lptscotch -lptscotcherr
>>>> -lscotch -lscotcherr -lX11 -lssl -lcrypto -lm
>>>> -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib
>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/5
>>>> -L/usr/lib/gcc/x86_64-linux-gnu/5 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
>>>> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
>>>> -L/lib/x86_64-linux-gnu -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran
>>>> -lm -lquadmath -lm -lmpi_cxx -lstdc++ -lrt -lm -lz
>>>> -Wl,-rpath,/usr/lib/openmpi/lib -L/usr/lib/openmpi/lib
>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/5
>>>> -L/usr/lib/gcc/x86_64-linux-gnu/5 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
>>>> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
>>>> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu
>>>> -L/usr/lib/x86_64-linux-gnu -ldl -lmpi -lhwloc -lgcc_s -lpthread -ldl
>>>> -----------------------------------------
>>>>
>>>> Best regards
>>>> Torquil Sørensen
More information about the petsc-users
mailing list