[petsc-users] eps_target malloc issue

[hong at aspiritech.org]

jifeng :

> "The matrices come from Jifeng Zhao, Northwestern University,
> zhaojf06 at gmail.com. The stiffness and mass matrices arise from a extended
> finite element model to predict phononic band gap of a three-dimensional
> two-phase composite with periodic SCC crystal structure. In addition,
> periodic boundary conditions are applied on the primary unit cell by
> employing Bloch wave analysis."

Thanks. I used your description, but removed your email address:
=============================
The matrices come from Jifeng Zhao, Northwestern University. The
stiffness and mass matrices arise from a extended finite element model
to predict phononic band gap of a three-dimensional two-phase
composite with periodic SCC crystal structure. In addition, periodic
boundary conditions are applied on the primary unit cell by employing
Bloch wave analysis.

Compute first 40 eigensolutions
K x = lamda M x
K and M are real and symmetric (Aug. 2014)
-----------------------------------------------
petsc_stiff20.dat:
  type: seqaij
  rows=1591812, cols=1591812, total: nonzeros=6.93869e+07
petsc_mass20.dat
  type: seqaij
  rows=1591812, cols=1591812, total: nonzeros=4.09661e+07,
---------------------------------
petsc_stiff33.dat:
  type: seqaij
  rows=1591812, cols=1591812, total: nonzeros=7.05468e+07,
petsc_mass33.dat:
  type: seqaij
  rows=1591812, cols=1591812, total: nonzeros=4.13649e+07
==============================

As you see, K and M have different data structures (different non
zeros) which makes
MatAXPY() less efficient than the same data structure. You may create
K and M with same data structure, i.e., inserting additional zeros to
K or M. Then tells slepc that your matrices K and M have same non-zero
structure.
>
> ----------------------------
>
> In addition, do want to let you know that I tried the runtime command you
> gave me using PETSC/3.4.4 (3.5 is not ready yet for me), it failed to
> converge.
>
> PETSC ERROR: !
> PETSC ERROR: KSP did not converge (reason=DIVERGED_ITS)!

I used minres/bjacobi/ icc with np=32 CPUs and got very slow
convergence. How many CPUs do you use?
When I run sequential minres/ icc, I got better convergence
(use '-st_ksp_monitor to monitor the convergence). If you use more
than 32 CPUs, then your bjacobi has more blocks than 32, thus a weaker
preconditioner than the one I used. Finding an efficient
preconditioner seems the key for your application.
>
> I also tried to different preconditioners:
> minres/bjacobi/ icc
> minres/bjacobi/ asm

This should be minres/asm/icc
Use  '-st_ksp_monitor', do you see improvement of asm over bjacobi?
You may also increase levels of icc.

> minres/bjacobi/ gamg
Do not do this.
>
> All of them failed by the same error. I can try to increase the maximum
> iteration number to remove this error message, but it will converge very
> slow for sure.
>
> Since I am already using -aij command, I am not sure if these slow
> convergence performance will disappear if I upgrade to PETSC/3.5. Is there
> anything else I can try on?

The bug in sbaij format is irrelevant to the convergence problems. I
would suggest staying with aij format until we patch the petsc-3.5.

When I find time, I may help you explore better preconditioners. Which
matrices are you interested,
petsc_mass20.dat and petsc_stiff20.dat, or something more challenging?

I would suggest you sending email to petsc-maint instead of
petsc-users. I guess Jose also on the mailing list of petsc-maint.

Hong

>
> On Mon, Aug 4, 2014 at 11:18 PM, hong at aspiritech.org <hong at aspiritech.org>
> wrote:
>>
>> jifeng :
>> >
>> > I just want to check with you if there is any updates on why the code
>> > crashes under -sbaij?
>>
>> It is a bug in petsc. I've fixed it in the brach hzhang/mataxpy-bugfix:
>> https://bitbucket.org/petsc/petsc/branch/hzhang/mataxpy-bugfix
>>
>> which will be merged to 'next' branch for tests, then patch to petsc-3.5.
>> You can use this branch if you like.
>>
>> Again, I like to add your matrices into petsc matrix collection.
>> Can you give me a short sentence describing your matrices, e.g.,
>> "The following matrices come from Rick Dean, Arco E&P Technology,
>> rhd at arco.com.  They arise from multiphase flow modeling of oil
>> reservoirs,..."
>>
>> Thanks,
>> Hong
>>
>>
>> >
>> > At the meanwhile, unfortunately PETSC-3.5 hasn't been installed on my
>> > server
>> > due to some technical problems. I hope technicians can get it done as
>> > soon
>> > as possible. I will update you when I have any new results. Thanks so
>> > much.
>> >
>> > Best regards,
>> > Jifeng
>> >
>> >
>> > On Wed, Jul 30, 2014 at 2:53 PM, jifeng zhao
>> > <jifengzhao2015 at u.northwestern.edu> wrote:
>> >>
>> >> Hello Hong,
>> >>
>> >> Thanks a lot. The result makes sense, but I think I will need higher
>> >> accuracy (smaller tolerance). I will take a try myself at the
>> >> meanwhile.
>> >>
>> >> Best regards,
>> >> Jifeng Zhao
>> >>
>> >>
>> >> On Wed, Jul 30, 2014 at 1:16 PM, hong at aspiritech.org
>> >> <hong at aspiritech.org>
>> >> wrote:
>> >>>
>> >>> Jifeng,
>> >>> Using petsc-3.5 and slepc-3.5 (maint branch), I tested your matrices
>> >>>
>> >>> mpiexec -n 32 ./ex7 -f1 $data/zhao/petsc_stiff20.dat -f2
>> >>> $data/zhao/petsc_mass20.dat -mat_view ascii::ascii_info -mat_type aij
>> >>> -eps_gen_hermitian -eps_type krylovschur -eps_target -0.1 -st_ksp_type
>> >>> minres -st_pc_type bjacobi -st_sub_pc_type icc -st_ksp_rtol 1.e-2
>> >>> -eps_tol 1.e-2 -eps_nev 40 -st_type sinvert -log_summary |& tee
>> >>> log.mat20.ex7.np32.tol-2
>> >>>
>> >>> The output is attached.
>> >>>
>> >>> When using '-mat_type sbaij', I got same error as reported, which
>> >>> might be a bug in our libraries. I'll investigate it.
>> >>> Switching to '-mat_type aij' works (I'm not sure if the computed
>> >>> eigenvalues are meaningful though), but the linear solver
>> >>> minres/bjacobi/icc converges very slow, takes around 500 - 1000 linear
>> >>> iterations for each eps iteration. You may experiment with stronger
>> >>> pc_type, e.g., '-pc_type asm' etc.
>> >>>
>> >>> I'll let you know why sbaij crashes.
>> >>>
>> >>> Hong
>> >>>
>> >>> On Mon, Jul 28, 2014 at 4:52 PM, Jose E. Roman <jroman at dsic.upv.es>
>> >>> wrote:
>> >>> >
>> >>> > El 28/07/2014, a las 23:48, jifeng zhao escribió:
>> >>> >
>> >>> >> Hi Barry,
>> >>> >>
>> >>> >> Thanks for the reply. I see. I am still testing it and looking for
>> >>> >> the
>> >>> >> bug.
>> >>> >>
>> >>> >> It is pretty weird that the error appears, since the code I am
>> >>> >> running
>> >>> >> is ex7 and I didn't change anything.
>> >>> >>
>> >>> >> My matrices are assembled in binary files separately. As I
>> >>> >> explained
>> >>> >> earlier, I have a series of matrices to solve, each of them have
>> >>> >> the same
>> >>> >> patterns but with different values. The way I assembly the matrix
>> >>> >> has proven
>> >>> >> to be correct for all the other cases. Only for this one, when the
>> >>> >> matrix
>> >>> >> has an zero eigenvalue, ex7 failed to solve it.
>> >>> >>
>> >>> >> In fact, I don't understand why this "Out of range" error could
>> >>> >> possibly appear? 1. my matrices have the correct parrellel lay up.
>> >>> >> 2. In
>> >>> >> ex7, I am not explicitly accessing to any elements at all! All I
>> >>> >> did is
>> >>> >> EPSSolve();
>> >>> >>
>> >>> >> Any ideas?
>> >>> >>
>> >>> >> Best regards,
>> >>> >> Jifeng Zhao
>> >>> >>
>> >>> >
>> >>> > Tomorrow we will release slepc-3.5. I would suggest trying with the
>> >>> > new
>> >>> > version. If the problem persist we can help to track down the
>> >>> > problem.
>> >>> >
>> >>> > Jose
>> >>> >
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Jifeng Zhao
>> >> PhD candidate at Northwestern University, US
>> >> Theoretical and Applied Mechanics Program
>> >
>> >
>> >
>> >
>> > --
>> > Jifeng Zhao
>> > PhD candidate at Northwestern University, US
>> > Theoretical and Applied Mechanics Program
>
>
>
>
> --
> Jifeng Zhao
> PhD candidate at Northwestern University, US
> Theoretical and Applied Mechanics Program