[petsc-users] eps_target malloc issue

Tue Aug 5 11:37:30 CDT 2014

Hello Hong,

Thank you so much.

I was using 24 processors currently and may considering using more. For
this application, the fewer processors, the more efficient bjacobi will be,
Is that right?

As you suggest, these days I will start trying these combinations: minres/icc
or minres/asm/icc or minres/icc/others ...

What is more, how about jacobi? the most simplest preconditioner? It gives
me the best performance actually when I was trying to use JD solver before.

petsc_mass20.dat and petsc_stiff20.dat are the most challenging one for
now. I have another set of matrices which have very similar eigenvalues
distribution with ***20.dat. I think they should work under the same
conditions. It will be fantastic, if you will have time to work out a
better preconditioner for them. I deeply appreciate your kind help.

Best regards,
Jifeng Zhao

On Tue, Aug 5, 2014 at 11:15 AM, hong at aspiritech.org <hong at aspiritech.org>
wrote:

> jifeng :
>
> > "The matrices come from Jifeng Zhao, Northwestern University,
> > zhaojf06 at gmail.com. The stiffness and mass matrices arise from a
> extended
> > finite element model to predict phononic band gap of a three-dimensional
> > two-phase composite with periodic SCC crystal structure. In addition,
> > periodic boundary conditions are applied on the primary unit cell by
> > employing Bloch wave analysis."
>
> Thanks. I used your description, but removed your email address:
> =============================
> The matrices come from Jifeng Zhao, Northwestern University. The
> stiffness and mass matrices arise from a extended finite element model
> to predict phononic band gap of a three-dimensional two-phase
> composite with periodic SCC crystal structure. In addition, periodic
> boundary conditions are applied on the primary unit cell by employing
> Bloch wave analysis.
>
> Compute first 40 eigensolutions
> K x = lamda M x
> K and M are real and symmetric (Aug. 2014)
> -----------------------------------------------
> petsc_stiff20.dat:
>   type: seqaij
>   rows=1591812, cols=1591812, total: nonzeros=6.93869e+07
> petsc_mass20.dat
>   type: seqaij
>   rows=1591812, cols=1591812, total: nonzeros=4.09661e+07,
> ---------------------------------
> petsc_stiff33.dat:
>   type: seqaij
>   rows=1591812, cols=1591812, total: nonzeros=7.05468e+07,
> petsc_mass33.dat:
>   type: seqaij
>   rows=1591812, cols=1591812, total: nonzeros=4.13649e+07
> ==============================
>
> As you see, K and M have different data structures (different non
> zeros) which makes
> MatAXPY() less efficient than the same data structure. You may create
> K and M with same data structure, i.e., inserting additional zeros to
> K or M. Then tells slepc that your matrices K and M have same non-zero
> structure.
> >
> > ----------------------------
> >
> > In addition, do want to let you know that I tried the runtime command you
> > gave me using PETSC/3.4.4 (3.5 is not ready yet for me), it failed to
> > converge.
> >
> > PETSC ERROR: !
> > PETSC ERROR: KSP did not converge (reason=DIVERGED_ITS)!
>
> I used minres/bjacobi/ icc with np=32 CPUs and got very slow
> convergence. How many CPUs do you use?
> When I run sequential minres/ icc, I got better convergence
> (use '-st_ksp_monitor to monitor the convergence). If you use more
> than 32 CPUs, then your bjacobi has more blocks than 32, thus a weaker
> preconditioner than the one I used. Finding an efficient
> preconditioner seems the key for your application.
> >
> > I also tried to different preconditioners:
> > minres/bjacobi/ icc
> > minres/bjacobi/ asm
>
> This should be minres/asm/icc
> Use  '-st_ksp_monitor', do you see improvement of asm over bjacobi?
> You may also increase levels of icc.
>
> > minres/bjacobi/ gamg
> Do not do this.
> >
> > All of them failed by the same error. I can try to increase the maximum
> > iteration number to remove this error message, but it will converge very
> > slow for sure.
> >
> > Since I am already using -aij command, I am not sure if these slow
> > convergence performance will disappear if I upgrade to PETSC/3.5. Is
> there
> > anything else I can try on?
>
> The bug in sbaij format is irrelevant to the convergence problems. I
> would suggest staying with aij format until we patch the petsc-3.5.
>
> When I find time, I may help you explore better preconditioners. Which
> matrices are you interested,
> petsc_mass20.dat and petsc_stiff20.dat, or something more challenging?
>
> I would suggest you sending email to petsc-maint instead of
> petsc-users. I guess Jose also on the mailing list of petsc-maint.
>
> Hong
>
> >
> > On Mon, Aug 4, 2014 at 11:18 PM, hong at aspiritech.org <
> hong at aspiritech.org>
> > wrote:
> >>
> >> jifeng :
> >> >
> >> > I just want to check with you if there is any updates on why the code
> >> > crashes under -sbaij?
> >>
> >> It is a bug in petsc. I've fixed it in the brach hzhang/mataxpy-bugfix:
> >> https://bitbucket.org/petsc/petsc/branch/hzhang/mataxpy-bugfix
> >>
> >> which will be merged to 'next' branch for tests, then patch to
> petsc-3.5.
> >> You can use this branch if you like.
> >>
> >> Again, I like to add your matrices into petsc matrix collection.
> >> Can you give me a short sentence describing your matrices, e.g.,
> >> "The following matrices come from Rick Dean, Arco E&P Technology,
> >> rhd at arco.com.  They arise from multiphase flow modeling of oil
> >> reservoirs,..."
> >>
> >> Thanks,
> >> Hong
> >>
> >>
> >> >
> >> > At the meanwhile, unfortunately PETSC-3.5 hasn't been installed on my
> >> > server
> >> > due to some technical problems. I hope technicians can get it done as
> >> > soon
> >> > as possible. I will update you when I have any new results. Thanks so
> >> > much.
> >> >
> >> > Best regards,
> >> > Jifeng
> >> >
> >> >
> >> > On Wed, Jul 30, 2014 at 2:53 PM, jifeng zhao
> >> > <jifengzhao2015 at u.northwestern.edu> wrote:
> >> >>
> >> >> Hello Hong,
> >> >>
> >> >> Thanks a lot. The result makes sense, but I think I will need higher
> >> >> accuracy (smaller tolerance). I will take a try myself at the
> >> >> meanwhile.
> >> >>
> >> >> Best regards,
> >> >> Jifeng Zhao
> >> >>
> >> >>
> >> >> On Wed, Jul 30, 2014 at 1:16 PM, hong at aspiritech.org
> >> >> <hong at aspiritech.org>
> >> >> wrote:
> >> >>>
> >> >>> Jifeng,
> >> >>> Using petsc-3.5 and slepc-3.5 (maint branch), I tested your matrices
> >> >>>
> >> >>> mpiexec -n 32 ./ex7 -f1 $data/zhao/petsc_stiff20.dat -f2
> >> >>> $data/zhao/petsc_mass20.dat -mat_view ascii::ascii_info -mat_type
> aij
> >> >>> -eps_gen_hermitian -eps_type krylovschur -eps_target -0.1
> -st_ksp_type
> >> >>> minres -st_pc_type bjacobi -st_sub_pc_type icc -st_ksp_rtol 1.e-2
> >> >>> -eps_tol 1.e-2 -eps_nev 40 -st_type sinvert -log_summary |& tee
> >> >>> log.mat20.ex7.np32.tol-2
> >> >>>
> >> >>> The output is attached.
> >> >>>
> >> >>> When using '-mat_type sbaij', I got same error as reported, which
> >> >>> might be a bug in our libraries. I'll investigate it.
> >> >>> Switching to '-mat_type aij' works (I'm not sure if the computed
> >> >>> eigenvalues are meaningful though), but the linear solver
> >> >>> minres/bjacobi/icc converges very slow, takes around 500 - 1000
> linear
> >> >>> iterations for each eps iteration. You may experiment with stronger
> >> >>> pc_type, e.g., '-pc_type asm' etc.
> >> >>>
> >> >>> I'll let you know why sbaij crashes.
> >> >>>
> >> >>> Hong
> >> >>>
> >> >>> On Mon, Jul 28, 2014 at 4:52 PM, Jose E. Roman <jroman at dsic.upv.es>
> >> >>> wrote:
> >> >>> >
> >> >>> > El 28/07/2014, a las 23:48, jifeng zhao escribió:
> >> >>> >
> >> >>> >> Hi Barry,
> >> >>> >>
> >> >>> >> Thanks for the reply. I see. I am still testing it and looking
> for
> >> >>> >> the
> >> >>> >> bug.
> >> >>> >>
> >> >>> >> It is pretty weird that the error appears, since the code I am
> >> >>> >> running
> >> >>> >> is ex7 and I didn't change anything.
> >> >>> >>
> >> >>> >> My matrices are assembled in binary files separately. As I
> >> >>> >> explained
> >> >>> >> earlier, I have a series of matrices to solve, each of them have
> >> >>> >> the same
> >> >>> >> patterns but with different values. The way I assembly the matrix
> >> >>> >> has proven
> >> >>> >> to be correct for all the other cases. Only for this one, when
> the
> >> >>> >> matrix
> >> >>> >> has an zero eigenvalue, ex7 failed to solve it.
> >> >>> >>
> >> >>> >> In fact, I don't understand why this "Out of range" error could
> >> >>> >> possibly appear? 1. my matrices have the correct parrellel lay
> up.
> >> >>> >> 2. In
> >> >>> >> ex7, I am not explicitly accessing to any elements at all! All I
> >> >>> >> did is
> >> >>> >> EPSSolve();
> >> >>> >>
> >> >>> >> Any ideas?
> >> >>> >>
> >> >>> >> Best regards,
> >> >>> >> Jifeng Zhao
> >> >>> >>
> >> >>> >
> >> >>> > Tomorrow we will release slepc-3.5. I would suggest trying with
> the
> >> >>> > new
> >> >>> > version. If the problem persist we can help to track down the
> >> >>> > problem.
> >> >>> >
> >> >>> > Jose
> >> >>> >
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Jifeng Zhao
> >> >> PhD candidate at Northwestern University, US
> >> >> Theoretical and Applied Mechanics Program
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Jifeng Zhao
> >> > PhD candidate at Northwestern University, US
> >> > Theoretical and Applied Mechanics Program
> >
> >
> >
> >
> > --
> > Jifeng Zhao
> > PhD candidate at Northwestern University, US
> > Theoretical and Applied Mechanics Program
>

-- 
Jifeng Zhao
PhD candidate at Northwestern University, US
Theoretical and Applied Mechanics Program
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20140805/41b6718b/attachment.html>