From hzhang at mcs.anl.gov  Tue May  1 09:24:50 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Tue, 1 May 2012 09:24:50 -0500
Subject: [petsc-users] Solving a very simple time step problem:
In-Reply-To: <ECCB1A7D-4604-4CF0-A583-EDE8060A4682@gmail.com>
References: <ECCB1A7D-4604-4CF0-A583-EDE8060A4682@gmail.com>
Message-ID: <CAGCphBueBVMGJ0adO_5dPDbOgEie5e3M4CMcWRcSxDcdk5JECw@mail.gmail.com>

Andrew :
We have TS examples under
~/src/ts/examples/tutorials/
/src/ts/examples/tests

Hong

> I want to solve a very simple equation:
>
> u_t = F(t) u
>
> Where F(t) = H_0 + a(t) H' (H_0 and H' are constant matrices, and a(t) is
> a time dependent scalar).
>
> But I'm not sure how to go about doing this using the TS context.
>
> I don't have a Jacobian that I need to be worried about, so should I be
> doing:
>
> TSSetRHSFunction(ts,PETSC_NULL,myRHSFunction,&appctx);
> TSSetRHSJacobian(ts,A,A,TSComputeRHSJacobianConstant,&appctx);
>
> Where:
> myRHSFunction(TS ts,PetscReal t,Vec u,Vec F,void *ctx)
> {
>        //Create temporary matrix A = H_0 + a(t) H'
>        //then do F = A u
> }
>
> Or should I be doing something else?
>
> Thanks for the help, unfortunately, it looks like the documentation on TS
> in the manual isn't accurate.
>
> -Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120501/10f83278/attachment.htm>

From knepley at gmail.com  Tue May  1 09:36:42 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 May 2012 10:36:42 -0400
Subject: [petsc-users] Solving a very simple time step problem:
In-Reply-To: <CAGCphBueBVMGJ0adO_5dPDbOgEie5e3M4CMcWRcSxDcdk5JECw@mail.gmail.com>
References: <ECCB1A7D-4604-4CF0-A583-EDE8060A4682@gmail.com>
	<CAGCphBueBVMGJ0adO_5dPDbOgEie5e3M4CMcWRcSxDcdk5JECw@mail.gmail.com>
Message-ID: <CAMYG4G=TcLpx1ZM-J1EeVynSAQ4eRzChpMqSMs_EsOxH5OqcEQ@mail.gmail.com>

On Tue, May 1, 2012 at 10:24 AM, Hong Zhang <hzhang at mcs.anl.gov> wrote:

> Andrew :
> We have TS examples under
> ~/src/ts/examples/tutorials/
> /src/ts/examples/tests
>
> Hong
>
>> I want to solve a very simple equation:
>>
>> u_t = F(t) u
>>
>> Where F(t) = H_0 + a(t) H' (H_0 and H' are constant matrices, and a(t) is
>> a time dependent scalar).
>>
>> But I'm not sure how to go about doing this using the TS context.
>>
>> I don't have a Jacobian that I need to be worried about, so should I be
>> doing:
>>
>
I am not sure I understand what you mean. Your Jacobian above is F(t), so
it does change. You can
of course do this MF since it will be exact, just probably more expensive.

   Matt


> TSSetRHSFunction(ts,PETSC_NULL,myRHSFunction,&appctx);
>> TSSetRHSJacobian(ts,A,A,TSComputeRHSJacobianConstant,&appctx);
>>
>> Where:
>> myRHSFunction(TS ts,PetscReal t,Vec u,Vec F,void *ctx)
>> {
>>        //Create temporary matrix A = H_0 + a(t) H'
>>        //then do F = A u
>> }
>>
>> Or should I be doing something else?
>>
>> Thanks for the help, unfortunately, it looks like the documentation on TS
>> in the manual isn't accurate.
>>
>> -Andrew
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120501/3e2ade59/attachment-0001.htm>

From andrew.spott at gmail.com  Tue May  1 13:27:24 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Tue, 1 May 2012 12:27:24 -0600
Subject: [petsc-users] Solving a very simple time step problem:
In-Reply-To: <CAMYG4G=TcLpx1ZM-J1EeVynSAQ4eRzChpMqSMs_EsOxH5OqcEQ@mail.gmail.com>
References: <ECCB1A7D-4604-4CF0-A583-EDE8060A4682@gmail.com>
	<CAGCphBueBVMGJ0adO_5dPDbOgEie5e3M4CMcWRcSxDcdk5JECw@mail.gmail.com>
	<CAMYG4G=TcLpx1ZM-J1EeVynSAQ4eRzChpMqSMs_EsOxH5OqcEQ@mail.gmail.com>
Message-ID: <D76545BB-18EC-448A-A9F1-72ADE2C95E1B@gmail.com>

This is actually what I figured, but I haven't seen the jacobian used in this context before, and wanted to make sure before I bugged you guys with a more complicated problem.

I am attempting to do a very basic linear time stepping problem:

u_t = H(t) u

Where H(t) = H0 + a(t) * H'

Where a(t) is just a scalar function of time, and H' has nothing on the diagonal (but is symmetric) and H0 is diagonal

I'm following ex4.c because my problem is very similar to that one, using a linear rhs.

I create my time step context, and the matrix to use:

    //creating wave function, creating 

    MatCreate(world, &A);
    MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,evalues.size(),evalues.size());
    MatSetType(A,MATMPIAIJ);
    MatSetFromOptions(A);
    MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);     // I don't know why I need these lines, but when I don't have them, the program complains about the matrix being in the wrong state.
    MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);

    TSCreate(world, &timeStepContext);
    TSSetProblemType(timeStepContext,TS_LINEAR);
    TSMonitorSet(timeStepContext,Monitor,&cntx,PETSC_NULL);
    TSSetType(timeStepContext,TSCN);

    TSSetRHSFunction(timeStepContext,PETSC_NULL,TSComputeRHSFunctionLinear,&cntx);
    TSSetRHSJacobian(timeStepContext,A,A,*Hamiltonian,&cntx);
    TSSetInitialTimeStep(timeStepContext,0.0,cntx.params->dt());
    TSSetSolution(timeStepContext,wavefunction);

Then, my hamiltonian function is:

PetscErrorCode Hamiltonian(TS timeStepContext, PetscReal t, Vec u, Mat *A, Mat *B, MatStructure *flag, void* context)
{
    Mat h = *A;
    PetscErrorCode  err;
    Context         *cntx = (Context*)context;
    PetscScalar     ef = eField(t, cntx->params);
    
    
    MatDuplicate(cntx->energy_eigenstates,MAT_COPY_VALUES,&h);
    MatAXPY(h, ef, cntx->dipole_matrix, DIFFERENT_NONZERO_PATTERN);
    
    MatAssemblyBegin(h, MAT_FINAL_ASSEMBLY);
    MatAssemblyEnd(h, MAT_FINAL_ASSEMBLY);

    //If I test the matrix 'h' here, against a vector (1,0,0?.), I get the correct results that I would expect.
}
 

But this doesn't work.  my "Monitor" function writes the wave function to disk, but it remains (1,0,0,0?,0,0)

When I do a test on h inside the Hamiltonian function, (create a vector that is (1,0,0,0?0,0), and multiply it by h), I get the expected results, so the matrices are correct, and h is being created, but when I write out the vector as it is time stepped, (in the monitor function), nothing changes (I get (1,0,0,0?,0,0) for each time step)

My monitor code for reference:

PetscErrorCode Monitor(TS ts, PetscInt step, PetscReal t, Vec u, void *cntx)
{
    Context         *context = (Context*) cntx;
    PetscErrorCode  ierr;
    PetscScalar     ef = eField(t, context->params);
    if (step < 30)
    {
        std::stringstream s("");
        s << step;
        ierr = PetscViewerASCIIOpen(context->world, context->params->getWaveFunctionFilename().append(s.str()).append(".dat").c_str(),&context->view);
        ierr = PetscViewerSetFormat(context->view,	 PETSC_VIEWER_ASCII_SYMMODU);
        ierr = VecView(u, context->view);
    }
    return ierr;
}

I'm sorry to bug you guys with this, but I'm a little lost on where even to start looking for an error?

Thanks as always for the help,

-Andrew


On May 1, 2012, at 8:36 AM, Matthew Knepley wrote:

> On Tue, May 1, 2012 at 10:24 AM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
> Andrew :
> We have TS examples under
> ~/src/ts/examples/tutorials/
> /src/ts/examples/tests
> 
> Hong
> I want to solve a very simple equation:
> 
> u_t = F(t) u
> 
> Where F(t) = H_0 + a(t) H' (H_0 and H' are constant matrices, and a(t) is a time dependent scalar).
> 
> But I'm not sure how to go about doing this using the TS context.
> 
> I don't have a Jacobian that I need to be worried about, so should I be doing:
> 
> I am not sure I understand what you mean. Your Jacobian above is F(t), so it does change. You can
> of course do this MF since it will be exact, just probably more expensive.
> 
>    Matt
>  
> TSSetRHSFunction(ts,PETSC_NULL,myRHSFunction,&appctx);
> TSSetRHSJacobian(ts,A,A,TSComputeRHSJacobianConstant,&appctx);
> 
> Where:
> myRHSFunction(TS ts,PetscReal t,Vec u,Vec F,void *ctx)
> {
>        //Create temporary matrix A = H_0 + a(t) H'
>        //then do F = A u
> }
> 
> Or should I be doing something else?
> 
> Thanks for the help, unfortunately, it looks like the documentation on TS in the manual isn't accurate.
> 
> -Andrew
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120501/f4a4adca/attachment.htm>

From knepley at gmail.com  Tue May  1 15:12:07 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 May 2012 16:12:07 -0400
Subject: [petsc-users] Solving a very simple time step problem:
In-Reply-To: <D76545BB-18EC-448A-A9F1-72ADE2C95E1B@gmail.com>
References: <ECCB1A7D-4604-4CF0-A583-EDE8060A4682@gmail.com>
	<CAGCphBueBVMGJ0adO_5dPDbOgEie5e3M4CMcWRcSxDcdk5JECw@mail.gmail.com>
	<CAMYG4G=TcLpx1ZM-J1EeVynSAQ4eRzChpMqSMs_EsOxH5OqcEQ@mail.gmail.com>
	<D76545BB-18EC-448A-A9F1-72ADE2C95E1B@gmail.com>
Message-ID: <CAMYG4G=UKx88bjc2ndfSDe=Qr92zybkVGrVXUL+7KSONQNPSJA@mail.gmail.com>

On Tue, May 1, 2012 at 2:27 PM, Andrew Spott <andrew.spott at gmail.com> wrote:

> This is actually what I figured, but I haven't seen the jacobian used in
> this context before, and wanted to make sure before I bugged you guys with
> a more complicated problem.
>
> I am attempting to do a very basic linear time stepping problem:
>
> u_t = H(t) u
>
> Where H(t) = H0 + a(t) * H'
>
> Where a(t) is just a scalar function of time, and H' has nothing on the
> diagonal (but is symmetric) and H0 is diagonal
>
> I'm following ex4.c because my problem is very similar to that one, using
> a linear rhs.
>
> I create my time step context, and the matrix to use:
>
>     //creating wave function, creating
>
>     MatCreate(world, &A);
>     MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,evalues.size(),evalues.size());
>     MatSetType(A,MATMPIAIJ);
>     MatSetFromOptions(A);
>     MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);     // I don't know why I
> need these lines, but when I don't have them, the program complains about
> the matrix being in the wrong state.
>     MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);
>
>     TSCreate(world, &timeStepContext);
>     TSSetProblemType(timeStepContext,TS_LINEAR);
>     TSMonitorSet(timeStepContext,Monitor,&cntx,PETSC_NULL);
>     TSSetType(timeStepContext,TSCN);
>
>
> TSSetRHSFunction(timeStepContext,PETSC_NULL,TSComputeRHSFunctionLinear,&cntx);
>     TSSetRHSJacobian(timeStepContext,A,A,*Hamiltonian,&cntx);
>     TSSetInitialTimeStep(timeStepContext,0.0,cntx.params->dt());
>     TSSetSolution(timeStepContext,wavefunction);
>
> Then, my hamiltonian function is:
>
> PetscErrorCode Hamiltonian(TS timeStepContext, PetscReal t, Vec u, Mat *A,
> Mat *B, MatStructure *flag, void* context)
> {
>     Mat h = *A;
>     PetscErrorCode  err;
>     Context         *cntx = (Context*)context;
>     PetscScalar     ef = eField(t, cntx->params);
>
>
>     MatDuplicate(cntx->energy_eigenstates,MAT_COPY_VALUES,&h);
>

This is a problem. You are overwriting 'h' here, so it is never written
back to *A.


>     MatAXPY(h, ef, cntx->dipole_matrix, DIFFERENT_NONZERO_PATTERN);
>
>     MatAssemblyBegin(h, MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(h, MAT_FINAL_ASSEMBLY);
>
>     //If I test the matrix 'h' here, against a vector (1,0,0?.), I get the
> correct results that I would expect.
>

You need

     *A = h;

in order to update it.

   Matt


> }
>
>
> But this doesn't work.  my "Monitor" function writes the wave function to
> disk, but it remains (1,0,0,0?,0,0)
>
> When I do a test on h inside the Hamiltonian function, (create a vector
> that is (1,0,0,0?0,0), and multiply it by h), I get the expected results,
> so the matrices are correct, and h is being created, but when I write out
> the vector as it is time stepped, (in the monitor function), nothing
> changes (I get (1,0,0,0?,0,0) for each time step)
>
> My monitor code for reference:
>
> PetscErrorCode Monitor(TS ts, PetscInt step, PetscReal t, Vec u, void
> *cntx)
> {
>     Context         *context = (Context*) cntx;
>     PetscErrorCode  ierr;
>     PetscScalar     ef = eField(t, context->params);
>     if (step < 30)
>     {
>         std::stringstream s("");
>         s << step;
>         ierr = PetscViewerASCIIOpen(context->world,
> context->params->getWaveFunctionFilename().append(s.str()).append(".dat").c_str(),&context->view);
>         ierr = PetscViewerSetFormat(context->view,
>  PETSC_VIEWER_ASCII_SYMMODU);
>         ierr = VecView(u, context->view);
>     }
>     return ierr;
> }
>
> I'm sorry to bug you guys with this, but I'm a little lost on where even
> to start looking for an error?
>
> Thanks as always for the help,
>
> -Andrew
>
>
> On May 1, 2012, at 8:36 AM, Matthew Knepley wrote:
>
> On Tue, May 1, 2012 at 10:24 AM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
>
>> Andrew :
>> We have TS examples under
>> ~/src/ts/examples/tutorials/
>> /src/ts/examples/tests
>>
>> Hong
>>
>>> I want to solve a very simple equation:
>>>
>>> u_t = F(t) u
>>>
>>> Where F(t) = H_0 + a(t) H' (H_0 and H' are constant matrices, and a(t)
>>> is a time dependent scalar).
>>>
>>> But I'm not sure how to go about doing this using the TS context.
>>>
>>> I don't have a Jacobian that I need to be worried about, so should I be
>>> doing:
>>>
>>
> I am not sure I understand what you mean. Your Jacobian above is F(t), so
> it does change. You can
> of course do this MF since it will be exact, just probably more expensive.
>
>    Matt
>
>
>>  TSSetRHSFunction(ts,PETSC_NULL,myRHSFunction,&appctx);
>>> TSSetRHSJacobian(ts,A,A,TSComputeRHSJacobianConstant,&appctx);
>>>
>>> Where:
>>> myRHSFunction(TS ts,PetscReal t,Vec u,Vec F,void *ctx)
>>> {
>>>        //Create temporary matrix A = H_0 + a(t) H'
>>>        //then do F = A u
>>> }
>>>
>>> Or should I be doing something else?
>>>
>>> Thanks for the help, unfortunately, it looks like the documentation on
>>> TS in the manual isn't accurate.
>>>
>>> -Andrew
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120501/b630f4cf/attachment.htm>

From andrew.spott at gmail.com  Tue May  1 15:17:05 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Tue, 1 May 2012 14:17:05 -0600
Subject: [petsc-users] Solving a very simple time step problem:
In-Reply-To: <CAMYG4G=UKx88bjc2ndfSDe=Qr92zybkVGrVXUL+7KSONQNPSJA@mail.gmail.com>
References: <ECCB1A7D-4604-4CF0-A583-EDE8060A4682@gmail.com>
	<CAGCphBueBVMGJ0adO_5dPDbOgEie5e3M4CMcWRcSxDcdk5JECw@mail.gmail.com>
	<CAMYG4G=TcLpx1ZM-J1EeVynSAQ4eRzChpMqSMs_EsOxH5OqcEQ@mail.gmail.com>
	<D76545BB-18EC-448A-A9F1-72ADE2C95E1B@gmail.com>
	<CAMYG4G=UKx88bjc2ndfSDe=Qr92zybkVGrVXUL+7KSONQNPSJA@mail.gmail.com>
Message-ID: <7DEC05AB-BECC-46DB-A77F-813768857E9F@gmail.com>

Thanks!

-Andrew


On May 1, 2012, at 2:12 PM, Matthew Knepley wrote:

> On Tue, May 1, 2012 at 2:27 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
> This is actually what I figured, but I haven't seen the jacobian used in this context before, and wanted to make sure before I bugged you guys with a more complicated problem.
> 
> I am attempting to do a very basic linear time stepping problem:
> 
> u_t = H(t) u
> 
> Where H(t) = H0 + a(t) * H'
> 
> Where a(t) is just a scalar function of time, and H' has nothing on the diagonal (but is symmetric) and H0 is diagonal
> 
> I'm following ex4.c because my problem is very similar to that one, using a linear rhs.
> 
> I create my time step context, and the matrix to use:
> 
>     //creating wave function, creating 
> 
>     MatCreate(world, &A);
>     MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,evalues.size(),evalues.size());
>     MatSetType(A,MATMPIAIJ);
>     MatSetFromOptions(A);
>     MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);     // I don't know why I need these lines, but when I don't have them, the program complains about the matrix being in the wrong state.
>     MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);
> 
>     TSCreate(world, &timeStepContext);
>     TSSetProblemType(timeStepContext,TS_LINEAR);
>     TSMonitorSet(timeStepContext,Monitor,&cntx,PETSC_NULL);
>     TSSetType(timeStepContext,TSCN);
> 
>     TSSetRHSFunction(timeStepContext,PETSC_NULL,TSComputeRHSFunctionLinear,&cntx);
>     TSSetRHSJacobian(timeStepContext,A,A,*Hamiltonian,&cntx);
>     TSSetInitialTimeStep(timeStepContext,0.0,cntx.params->dt());
>     TSSetSolution(timeStepContext,wavefunction);
> 
> Then, my hamiltonian function is:
> 
> PetscErrorCode Hamiltonian(TS timeStepContext, PetscReal t, Vec u, Mat *A, Mat *B, MatStructure *flag, void* context)
> {
>     Mat h = *A;
>     PetscErrorCode  err;
>     Context         *cntx = (Context*)context;
>     PetscScalar     ef = eField(t, cntx->params);
>     
>     
>     MatDuplicate(cntx->energy_eigenstates,MAT_COPY_VALUES,&h);
> 
> This is a problem. You are overwriting 'h' here, so it is never written back to *A.
>  
>     MatAXPY(h, ef, cntx->dipole_matrix, DIFFERENT_NONZERO_PATTERN);
>     
>     MatAssemblyBegin(h, MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(h, MAT_FINAL_ASSEMBLY);
> 
>     //If I test the matrix 'h' here, against a vector (1,0,0?.), I get the correct results that I would expect.
> 
> You need
> 
>      *A = h;
> 
> in order to update it.
> 
>    Matt
>  
> }
>  
> 
> But this doesn't work.  my "Monitor" function writes the wave function to disk, but it remains (1,0,0,0?,0,0)
> 
> When I do a test on h inside the Hamiltonian function, (create a vector that is (1,0,0,0?0,0), and multiply it by h), I get the expected results, so the matrices are correct, and h is being created, but when I write out the vector as it is time stepped, (in the monitor function), nothing changes (I get (1,0,0,0?,0,0) for each time step)
> 
> My monitor code for reference:
> 
> PetscErrorCode Monitor(TS ts, PetscInt step, PetscReal t, Vec u, void *cntx)
> {
>     Context         *context = (Context*) cntx;
>     PetscErrorCode  ierr;
>     PetscScalar     ef = eField(t, context->params);
>     if (step < 30)
>     {
>         std::stringstream s("");
>         s << step;
>         ierr = PetscViewerASCIIOpen(context->world, context->params->getWaveFunctionFilename().append(s.str()).append(".dat").c_str(),&context->view);
>         ierr = PetscViewerSetFormat(context->view,	 PETSC_VIEWER_ASCII_SYMMODU);
>         ierr = VecView(u, context->view);
>     }
>     return ierr;
> }
> 
> I'm sorry to bug you guys with this, but I'm a little lost on where even to start looking for an error?
> 
> Thanks as always for the help,
> 
> -Andrew
> 
> 
> On May 1, 2012, at 8:36 AM, Matthew Knepley wrote:
> 
>> On Tue, May 1, 2012 at 10:24 AM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
>> Andrew :
>> We have TS examples under
>> ~/src/ts/examples/tutorials/
>> /src/ts/examples/tests
>> 
>> Hong
>> I want to solve a very simple equation:
>> 
>> u_t = F(t) u
>> 
>> Where F(t) = H_0 + a(t) H' (H_0 and H' are constant matrices, and a(t) is a time dependent scalar).
>> 
>> But I'm not sure how to go about doing this using the TS context.
>> 
>> I don't have a Jacobian that I need to be worried about, so should I be doing:
>> 
>> I am not sure I understand what you mean. Your Jacobian above is F(t), so it does change. You can
>> of course do this MF since it will be exact, just probably more expensive.
>> 
>>    Matt
>>  
>> TSSetRHSFunction(ts,PETSC_NULL,myRHSFunction,&appctx);
>> TSSetRHSJacobian(ts,A,A,TSComputeRHSJacobianConstant,&appctx);
>> 
>> Where:
>> myRHSFunction(TS ts,PetscReal t,Vec u,Vec F,void *ctx)
>> {
>>        //Create temporary matrix A = H_0 + a(t) H'
>>        //then do F = A u
>> }
>> 
>> Or should I be doing something else?
>> 
>> Thanks for the help, unfortunately, it looks like the documentation on TS in the manual isn't accurate.
>> 
>> -Andrew
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120501/d637b23e/attachment-0001.htm>

From dkarthik at stanford.edu  Tue May  1 15:37:00 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Tue, 1 May 2012 13:37:00 -0700 (PDT)
Subject: [petsc-users] Multigrid
In-Reply-To: <7DEC05AB-BECC-46DB-A77F-813768857E9F@gmail.com>
Message-ID: <1475542904.99693159.1335904620372.JavaMail.root@zm09.stanford.edu>

Hello,

        I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 

        What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any help in this regard will be appreciated.

        Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results.

Thanks,
Karthik



-- 

=======================================
Karthik Duraisamy
Assistant Professor (Consulting)
Durand Building Rm 357
Dept of Aeronautics and Astronautics
Stanford University
Stanford CA 94305

Phone: 650-721-2835
Web:     www.stanford.edu/~dkarthik
=======================================

From bsmith at mcs.anl.gov  Tue May  1 15:39:26 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 1 May 2012 15:39:26 -0500
Subject: [petsc-users] Multigrid
In-Reply-To: <1475542904.99693159.1335904620372.JavaMail.root@zm09.stanford.edu>
References: <1475542904.99693159.1335904620372.JavaMail.root@zm09.stanford.edu>
Message-ID: <CB42609E-F0F9-4CC8-B78A-9C39421C45E7@mcs.anl.gov>


On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote:

> Hello,
> 
>        I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
> 
>        What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any
> help in this regard will be appreciated.

   First run with -ksp_view to see what solver it is actually using.

   Barry

> 
>        Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results.
> 
> Thanks,
> Karthik
> 
> 
> 
> -- 
> 
> =======================================
> Karthik Duraisamy
> Assistant Professor (Consulting)
> Durand Building Rm 357
> Dept of Aeronautics and Astronautics
> Stanford University
> Stanford CA 94305
> 
> Phone: 650-721-2835
> Web:     www.stanford.edu/~dkarthik
> =======================================


From zonexo at gmail.com  Tue May  1 16:48:19 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Tue, 01 May 2012 23:48:19 +0200
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <C92D4D80-A685-4196-BFF6-85E4CB795D04@mcs.anl.gov>
References: <4F97E8F2.5050803@gmail.com> <4F993F1D.9020105@gmail.com>
	<CAMYG4GnHyb1VKSY3gwpTxyU=xe1ZcEhitKCu5+CpK9a1v0ymuA@mail.gmail.com>
	<4F99A15B.1010500@gmail.com>
	<CAMYG4Gmzkpt+UyYLbzhit9Jr1G6SO99C9qKp+qg6Nyd__rTo3g@mail.gmail.com>
	<4F99CA4C.1080609@gmail.com>
	<C92D4D80-A685-4196-BFF6-85E4CB795D04@mcs.anl.gov>
Message-ID: <4FA05A23.7080209@gmail.com>

Hi,

Do you mean my method is wrong?

I am following the template of ex22f,

where the variables are declared as :

PetscScalar  v(5)

MatStencil   row(4),col(4,5)

Hence,

for the neumann BC

num = 1

                 if (j/=0) then

                     v(num) = -rho*HxdHy

                     col(MatStencil_i,num) = i

                     col(MatStencil_j,num) = j-1

                     num = num + 1

                 end if

                 if (i/=0) then

                     v(num) = -rho*HydHx

                     col(MatStencil_i,num) = i-1

                     col(MatStencil_j,num) = j

                     num = num + 1

                 end if

                 if (i/=mx-1) then

                     v(num) = -rho*HydHx

                     col(MatStencil_i,num) = i+1

                     col(MatStencil_j,num) = j

                     num = num + 1

                 end if

                 if (j/=my-1) then

                     v(num) = -rho*HxdHy

                     col(MatStencil_i,num) = i

                     col(MatStencil_j,num) = j+1

                     num = num + 1

                 end if

                 v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)

                 print *, v

                 col(MatStencil_i,num) = i

                 col(MatStencil_j,num) = j

                 !num = num + 1

                 call 
MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)

I do not get any more out of range error. However,my ans is still 
different from that of ex29 in C.

Yours sincerely,

TAY wee-beng


On 27/4/2012 1:22 AM, Barry Smith wrote:
>     From the manual page for MatSetValuesStencil():
>
>     MatSetValuesStencil() uses 0-based row and column numbers in Fortran
>     as well as in C.
>
>
> On Apr 26, 2012, at 5:21 PM, TAY wee-beng wrote:
>
>> Hi,
>>
>> I managed to remove the out of range error by removing
>>
>>     num = num + 1 from
>>
>> col(MatStencil_i,num) = i
>>
>>                  col(MatStencil_j,num) = j+1
>>
>>                  num = num + 1
>>
>>                  call MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>
>> In C, num will give the number 5 since num start from 0.
>>
>> In fortran, num starts from 1 so num = num + 1 before MatSetValuesStencil must be removed.
>>
>> I still can't get the same ans. I will try to figure it out. Just to confirm, this subroutine:
>>
>> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>
>> is not added in fortran, although it is used in C.
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>> On 26/4/2012 9:29 PM, Matthew Knepley wrote:
>>> On Thu, Apr 26, 2012 at 3:26 PM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>> Hi,
>>>
>>> Thanks for pointing out my mistake. I'm still learning about the code. I corrected but I still get the error:
>>>
>>> I am sure its another simple bug. Go in with the debugger. When it says you are inserting a new nonzero,
>>> back up the stack to your routine and see what indices you are tying to insert. If its not part of the stencil,
>>> you have found the bug.
>>>
>>>    Matt
>>>
>>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>>> --------
>>> [0]PETSC ERROR: Argument out of range!
>>> [0]PETSC ERROR: New nonzero at (2,4) caused a malloc!
>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>
>>> boundary condition region:
>>>
>>> num = 1
>>>
>>>                  if (j/=0) then
>>>
>>>                      v(num) = -rho*HxdHy
>>>
>>>                      col(MatStencil_i,num) = i
>>>
>>>                      col(MatStencil_j,num) = j-1
>>>
>>>                      num = num + 1
>>>
>>>                  end if
>>>
>>>                  if (i/=0) then
>>>
>>>                      v(num) = -rho*HydHx
>>>
>>>                      col(MatStencil_i,num) = i-1
>>>
>>>                      col(MatStencil_j,num) = j
>>>
>>>                      num = num + 1
>>>
>>>                  end if
>>>
>>>                  if (i/=mx-1) then
>>>
>>>                      v(num) = -rho*HydHx
>>>
>>>                      col(MatStencil_i,num) = i+1
>>>
>>>                      col(MatStencil_j,num) = j
>>>
>>>                      num = num + 1
>>>
>>>                  end if
>>>
>>>                  if (j/=my-1) then
>>>
>>>                      v(num) = -rho*HxdHy
>>>
>>>                      col(MatStencil_i,num) = i
>>>
>>>                      col(MatStencil_j,num) = j+1
>>>
>>>                      num = num + 1
>>>
>>>                  end if
>>>
>>>                  v(num) = (num/2.0)*rho*(HxdHy + HydHx)
>>>
>>>                  col(MatStencil_i,num) = i
>>>
>>>                  col(MatStencil_j,num) = j+1
>>>
>>>                  num = num + 1
>>>
>>>                  call MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>> Yours sincerely,
>>>
>>> TAY wee-beng
>>>
>>>
>>> On 26/4/2012 2:45 PM, Matthew Knepley wrote:
>>>> On Thu, Apr 26, 2012 at 8:27 AM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>> Hi,
>>>>
>>>> Is there an answer to the question below?
>>>>   I still can't get it working.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> Hi,
>>>>
>>>> There's no error now but I didn't get the same ans as ex29. I believe the error lies in the boundary evaluation of v.
>>>>
>>>> In c, the code is :
>>>>
>>>> else if (user->bcType == NEUMANN) {
>>>>            num = 0;
>>>>            if (j!=0) {
>>>>              v[num] = -rho*HxdHy;              col[num].i = i;   col[num].j = j-1;
>>>>                                                                    ^^^ See this?
>>>>              num++;
>>>>            }
>>>>            if (i!=0) {
>>>>              v[num] = -rho*HydHx;              col[num].i = i-1; col[num].j = j;
>>>>              num++;
>>>>            }
>>>>            if (i!=mx-1) {
>>>>              v[num] = -rho*HydHx;              col[num].i = i+1; col[num].j = j;
>>>>              num++;
>>>>            }
>>>>            if (j!=my-1) {
>>>>              v[num] = -rho*HxdHy;              col[num].i = i;   col[num].j = j+1;
>>>>              num++;
>>>>            }
>>>>            v[num]   = (num/2.0)*rho*(HxdHy + HydHx); col[num].i = i;   col[num].j = j;
>>>>            num++;
>>>>            ierr = MatSetValuesStencil(jac,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);
>>>>
>>>> In fortran, I have changed to :
>>>>
>>>> else if (bc_cond == 2) then
>>>>
>>>>                  num = 1
>>>>
>>>>                  if (j/=0) then
>>>>
>>>>                      v(num) = -rho*HxdHy
>>>>
>>>>                      col(MatStencil_i,1) = i
>>>>                                                 ^^ Why is this 1 in stead of 'num'?
>>>>
>>>>     Matt
>>>>
>>>>
>>>>                      col(MatStencil_j,2) = j-1
>>>>
>>>>                      num = num + 1
>>>>
>>>>                  end if
>>>>
>>>>                  if (i/=0) then
>>>>
>>>>                      v(num) = -rho*HydHx
>>>>
>>>>                      col(MatStencil_i,1) = i-1
>>>>
>>>>                      col(MatStencil_j,2) = j
>>>>
>>>>                      num = num + 1
>>>>
>>>>                  end if
>>>>
>>>>                  if (i/=mx-1) then
>>>>
>>>>                      v(num) = -rho*HydHx
>>>>
>>>>                      col(MatStencil_i,1) = i+1
>>>>
>>>>                      col(MatStencil_j,2) = j
>>>>
>>>>                      num = num + 1
>>>>
>>>>                  end if
>>>>
>>>>                  if (j/=my-1) then
>>>>
>>>>                      v(num) = -rho*HxdHy
>>>>
>>>>                      col(MatStencil_i,1) = i
>>>>
>>>>                      col(MatStencil_j,2) = j+1
>>>>
>>>>                      num = num + 1
>>>>
>>>>                  end if
>>>>
>>>>                  v(num) = (num/2.0)*rho*(HxdHy + HydHx)
>>>>
>>>>                  col(MatStencil_i,1) = i
>>>>
>>>>                  col(MatStencil_j,2) = j+1
>>>>
>>>>                  num = num + 1
>>>>
>>>>                  call MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>
>>>> However, I got the error:
>>>>
>>>> [0]PETSC ERROR: --------------------- Error Message ---------------------------
>>>> --------
>>>> [0]PETSC ERROR: Argument out of range!
>>>> [0]PETSC ERROR: New nonzero at (1,3) caused a malloc!
>>>> [0]PETSC ERROR: ---------------------------------------------------------------
>>>>
>>>> How such is be defined in Fortran?
>>>>
>>>> Thank you
>>>> Yours sincerely,
>>>>
>>>> TAY wee-beng
>>>>
>>>>
>>>> On 25/4/2012 12:06 AM, Matthew Knepley wrote:
>>>>> On Tue, Apr 24, 2012 at 4:57 PM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>>> Hi,
>>>>>
>>>>> I still got the same error the moment this subroutine is called, after changing to :
>>>>>
>>>>> call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>>
>>>>> Argument 3 should be 0.
>>>>>
>>>>>    Matt
>>>>>
>>>>> [0]PETSC ERROR: Null argument, when expecting valid pointer!
>>>>> [0]PETSC ERROR: Null Object: Parameter # 4!
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: Petsc Development HG revision: fc934c1d84bc6ba8e2686702a8a99539d
>>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named USER-PC by
>>>>>   User Tue Apr 24 22:54:50 2012
>>>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>> /lib
>>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 --with-debuggin
>>>>> g=1 --useThreads=0
>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>> --------
>>>>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in src/mat/interface/C:\temp\PETSC
>>>>> -~1\src\mat\INTERF~1\matnull.c
>>>>> Yours sincerely,
>>>>>
>>>>> TAY wee-beng
>>>>>
>>>>>
>>>>> On 24/4/2012 10:37 PM, Matthew Knepley wrote:
>>>>>> On Tue, Apr 24, 2012 at 4:32 PM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'm trying to rewrite ex29 (KSP^Laplacian, 2d) from c to fortran version. The fortran version is based on the template of ex22f, which is in 3d.
>>>>>>
>>>>>> I have attached the code below. I have 2 problems.
>>>>>>
>>>>>> 1. When i3 = -3, in dirichlet BC, I got the same answer as ex29 in c version.
>>>>>>
>>>>>> However, when I change i3 to -4, I got the following error:
>>>>>>
>>>>>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: Argument out of range!
>>>>>> [0]PETSC ERROR: New nonzero at (9,1) caused a malloc!
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: Petsc Development HG revision: fc934c1d84bc6ba8e2686702a8a99539d
>>>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named USER-PC by
>>>>>>   User Tue Apr 24 22:08:46 2012
>>>>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>>> /lib
>>>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 --with-debuggin
>>>>>> g=1 --useThreads=0
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: MatSetValues_SeqAIJ() line 331 in src/mat/impls/aij/seq/C:\temp\
>>>>>> PETSC-~1\src\mat\impls\aij\seq\aij.c
>>>>>> [0]PETSC ERROR: MatSetValues() line 1148 in src/mat/interface/C:\temp\PETSC-~1\s
>>>>>> rc\mat\INTERF~1\matrix.c
>>>>>> [0]PETSC ERROR: MatSetValuesLocal() line 2003 in src/mat/interface/C:\temp\PETSC
>>>>>> -~1\src\mat\INTERF~1\matrix.c
>>>>>> [0]PETSC ERROR: MatSetValuesStencil() line 1379 in src/mat/interface/C:\temp\PET
>>>>>> SC-~1\src\mat\INTERF~1\matrix.c
>>>>>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: Argument out of range!
>>>>>> [0]PETSC ERROR: New nonzero at (10,2) caused a malloc!
>>>>>>
>>>>>> ...
>>>>>>
>>>>>> What did I do wrong?
>>>>>>
>>>>>>
>>>>>> 2. When I wanted to use the neumann BC, following ex29, I added :
>>>>>>
>>>>>> In subroutine ComputeRHS
>>>>>>
>>>>>> call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>>>
>>>>>> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>>>
>>>>>> call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>
>>>>>> and in subroutine ComputeMatrix
>>>>>>
>>>>>> call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>>>
>>>>>> call MatSetNullSpace(jac,nullspace,ierr)
>>>>>>
>>>>>> call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>
>>>>>> When I compile, it says unresolved external symbol MatNullSpaceRemove. Removing MatNullSpaceRemove, I can compile but running gives the error:
>>>>>>
>>>>>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: Corrupt argument:
>>>>>> seehttp://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>>>>> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: Petsc Development HG revision: fc934c1d84bc6ba8e2686702a8a99539d
>>>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named USER-PC by
>>>>>>   User Tue Apr 24 22:30:31 2012
>>>>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>>> /lib
>>>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 --with-debuggin
>>>>>> g=1 --useThreads=0
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in src/mat/interface/C:\temp\PETSC
>>>>>> -~1\src\mat\INTERF~1\matnull.c
>>>>>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: Invalid argument!
>>>>>> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: Petsc Development HG revision: fc934c1d84bc6ba8e2686702a8a99539d
>>>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named USER-PC by
>>>>>>   User Tue Apr 24 22:30:31 2012
>>>>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>>> /lib
>>>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 --with-debuggin
>>>>>> g=1 --useThreads=0
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in src/mat/interface/C:\temp\PETS
>>>>>> C-~1\src\mat\INTERF~1\matnull.c
>>>>>> [0]PETSC ERROR: ourdmfunction() line 30 in src/dm/interface/ftn-custom/C:\temp\P
>>>>>> ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>>>>> [0]PETSC ERROR: DMComputeFunction() line 1783 in src/dm/interface/C:\temp\PETSC-
>>>>>> ~1\src\dm\INTERF~1\dm.c
>>>>>> [0]PETSC ERROR: KSPSetUp() line 218 in src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>>>> c\ksp\ksp\INTERF~1\itfunc.c
>>>>>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: Corrupt argument:
>>>>>> seehttp://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind!
>>>>>> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 4!
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: Petsc Development HG revision: fc934c1d84bc6ba8e2686702a8a99539d
>>>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named USER-PC by
>>>>>>   User Tue Apr 24 22:30:31 2012
>>>>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>>> /lib
>>>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 --with-debuggin
>>>>>> g=1 --useThreads=0
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: MatNullSpaceCreate() line 250 in src/mat/interface/C:\temp\PETSC
>>>>>> -~1\src\mat\INTERF~1\matnull.c
>>>>>> [0]PETSC ERROR: --------------------- Error Message ----------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: Invalid argument!
>>>>>> [0]PETSC ERROR: Wrong type of object: Parameter # 1!
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: Petsc Development HG revision: fc934c1d84bc6ba8e2686702a8a99539d
>>>>>> 50af122  HG Date: Mon Apr 23 11:39:32 2012 -0500
>>>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>>>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: c:\obj_tmp\ex29f\Debug\ex29f.exe on a petsc-3.2 named USER-PC by
>>>>>>   User Tue Apr 24 22:30:31 2012
>>>>>> [0]PETSC ERROR: Libraries linked from /cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008
>>>>>> /lib
>>>>>> [0]PETSC ERROR: Configure run at Mon Apr 23 21:45:20 2012
>>>>>> [0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
>>>>>> t" --with-cxx="win32fe cl" --with-mpi-dir=/cygdrive/d/Lib/MPICH2/ --download-f-b
>>>>>> las-lapack=1 --prefix=/cygdrive/d/Lib/petsc-3.2-dev_win32_vs2008 --with-debuggin
>>>>>> g=1 --useThreads=0
>>>>>> [0]PETSC ERROR: ----------------------------------------------------------------
>>>>>> --------
>>>>>> [0]PETSC ERROR: MatNullSpaceDestroy() line 305 in src/mat/interface/C:\temp\PETS
>>>>>> C-~1\src\mat\INTERF~1\matnull.c
>>>>>> [0]PETSC ERROR: ourdmfunction() line 30 in src/dm/interface/ftn-custom/C:\temp\P
>>>>>> ETSC-~1\src\dm\INTERF~1\FTN-CU~1\zdmf.c
>>>>>> [0]PETSC ERROR: DMComputeFunction() line 1783 in src/dm/interface/C:\temp\PETSC-
>>>>>> ~1\src\dm\INTERF~1\dm.c
>>>>>> [0]PETSC ERROR: KSPSetUp() line 218 in src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>>>> c\ksp\ksp\INTERF~1\itfunc.c
>>>>>> [0]PETSC ERROR: KSPSolve() line 402 in src/ksp/ksp/interface/C:\temp\PETSC-~1\sr
>>>>>> c\ksp\ksp\INTERF~1\itfunc.c
>>>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>>>    type: mpi
>>>>>> Process [0]
>>>>>>
>>>>>> What's wrong?
>>>>>>
>>>>>>
>>>>>> program ex29f
>>>>>>
>>>>>> implicit none
>>>>>>
>>>>>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>>> !                    Include files
>>>>>> ! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>>>>> !
>>>>>> !     petscsys.h  - base PETSc routines      petscvec.h - vectors
>>>>>> !     petscmat.h - matrices
>>>>>> !     petscksp.h    - Krylov subspace methods  petscpc.h  - preconditioners
>>>>>>
>>>>>> #include "finclude/petsc.h90"
>>>>>>
>>>>>> PetscErrorCode   ierr
>>>>>>
>>>>>> DM               da
>>>>>>
>>>>>> KSP              ksp
>>>>>>
>>>>>> Vec              x
>>>>>>
>>>>>> external         ComputeRHS,ComputeMatrix
>>>>>>
>>>>>> PetscInt i1,i3
>>>>>>
>>>>>> call  PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>>>>>>
>>>>>> i3 = -4
>>>>>>
>>>>>> i1 = 1
>>>>>>
>>>>>> call KSPCreate(MPI_COMM_WORLD,ksp,ierr)
>>>>>>
>>>>>> call DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>>>>>> call DMSetFunction(da,ComputeRHS,ierr)
>>>>>> call DMSetJacobian(da,ComputeMatrix,ierr)
>>>>>> call KSPSetDM(ksp,da,ierr)
>>>>>>
>>>>>> call KSPSetFromOptions(ksp,ierr)
>>>>>> call KSPSetUp(ksp,ierr)
>>>>>> call KSPSolve(ksp,PETSC_NULL_OBJECT,PETSC_NULL_OBJECT,ierr)
>>>>>> call KSPGetSolution(ksp,x,ierr)
>>>>>> call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>>>>>> call KSPDestroy(ksp,ierr)
>>>>>> call DMDestroy(da,ierr)
>>>>>> call PetscFinalize(ierr)
>>>>>>
>>>>>> end program ex29f
>>>>>>
>>>>>> subroutine ComputeRHS(da,x,b,ierr)
>>>>>>
>>>>>> implicit none
>>>>>>
>>>>>> #include "finclude/petsc.h90"
>>>>>>
>>>>>> PetscErrorCode  ierr
>>>>>> PetscInt i,j,mx,my,xm,ym,xs,ys
>>>>>> PetscScalar  h,nu,rho
>>>>>> PetscScalar Hx,Hy
>>>>>> PetscScalar,pointer :: array(:,:)
>>>>>> Vec          x,b
>>>>>> DM           da
>>>>>> MatNullSpace nullspace    !>neumann BC
>>>>>>
>>>>>> nu = 0.1
>>>>>>
>>>>>> call DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>>>>>
>>>>>> Hx = 1.d0 / (mx-1)
>>>>>>
>>>>>> Hy = 1.d0 / (my-1)
>>>>>>
>>>>>> call DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>>>>>
>>>>>> call DMDAVecGetArrayF90(da,b,array,ierr)
>>>>>>
>>>>>> do j = ys,ys+ym-1
>>>>>>
>>>>>>      do i = xs,xs+xm-1
>>>>>>
>>>>>>          array(i,j) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>>>>>>
>>>>>>      end do
>>>>>>
>>>>>> end do
>>>>>>
>>>>>> call DMDAVecRestoreArrayF90(da,b,array,ierr)
>>>>>>
>>>>>> call VecAssemblyBegin(b,ierr)
>>>>>>
>>>>>> call VecAssemblyEnd(b,ierr)
>>>>>>
>>>>>> !call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>>>
>>>>>> 4th argument should be PETSC_NULL_OBJECT.
>>>>>>
>>>>>>     Matt
>>>>>>
>>>>>> !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>>>
>>>>>> !call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>
>>>>>> end subroutine ComputeRHS
>>>>>>
>>>>>>
>>>>>> subroutine ComputeMatrix(da,x,JJ,jac,str,ierr)
>>>>>>
>>>>>> implicit none
>>>>>>
>>>>>> #include "finclude/petsc.h90"
>>>>>>
>>>>>> Mat          jac,JJ
>>>>>>
>>>>>> PetscErrorCode    ierr
>>>>>>
>>>>>> DM           da
>>>>>>
>>>>>> PetscInt    i,j,k,mx,my,xm
>>>>>>
>>>>>> PetscInt    ym,xs,ys,i1,i5
>>>>>>
>>>>>> PetscScalar  v(5),Hx,Hy,rho,centerRho
>>>>>>
>>>>>> PetscScalar  HydHx
>>>>>>
>>>>>> PetscScalar  HxdHy
>>>>>>
>>>>>> MatStencil   row(4),col(4,5)
>>>>>>
>>>>>> Vec          x
>>>>>>
>>>>>> MatStructure str
>>>>>>
>>>>>> MatNullSpace nullspace    !>neumann BC
>>>>>>
>>>>>> rho = 1.0
>>>>>>
>>>>>> i1 = 1
>>>>>>
>>>>>> i5 = 5
>>>>>>
>>>>>> centerRho = rho
>>>>>>
>>>>>> call DMDAGetInfo(da,PETSC_NULL_INTEGER,mx,my,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,ierr)
>>>>>>
>>>>>> Hx = 1.d0 / (mx-1)
>>>>>>
>>>>>> Hy = 1.d0 / (my-1)
>>>>>>
>>>>>> HxdHy = Hx/Hy
>>>>>>
>>>>>> HydHx = Hy/Hx
>>>>>>
>>>>>> call DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>>>>>
>>>>>> do j=ys,ys+ym-1
>>>>>>
>>>>>>      do i=xs,xs+xm-1
>>>>>>
>>>>>>          row(MatStencil_i) = i
>>>>>>
>>>>>>          row(MatStencil_j) = j
>>>>>>
>>>>>>          call ComputeRho(i,j,mx,my,centerRho,rho)
>>>>>>
>>>>>>          if (i.eq.0 .or. j.eq.0 .or. i.eq.mx-1 .or. j.eq.my-1) then
>>>>>>
>>>>>>              v(1) = 2.0*rho*(HxdHy + HydHx)
>>>>>>
>>>>>>              call MatSetValuesStencil(jac,i1,row,i1,row,v,INSERT_VALUES,ierr)
>>>>>>
>>>>>>          else
>>>>>>
>>>>>>              v(1) = -rho*HxdHy
>>>>>>
>>>>>>              col(MatStencil_i,1) = i
>>>>>>
>>>>>>              col(MatStencil_j,2) = j-1
>>>>>>
>>>>>> Cut and paste error above.
>>>>>>
>>>>>>    Matt
>>>>>>
>>>>>>              v(2) = -rho*HydHx
>>>>>>
>>>>>>              col(MatStencil_i,2) = i-1
>>>>>>
>>>>>>              col(MatStencil_j,2) = j
>>>>>>
>>>>>>              v(3) = 2.0*rho*(HxdHy + HydHx)
>>>>>>
>>>>>>              col(MatStencil_i,3) = i
>>>>>>
>>>>>>              col(MatStencil_j,3) = j
>>>>>>
>>>>>>              v(4) = -rho*HydHx
>>>>>>
>>>>>>              col(MatStencil_i,4) = i+1
>>>>>>
>>>>>>              col(MatStencil_j,4) = j
>>>>>>
>>>>>>              v(5) = -rho*HxdHy
>>>>>>
>>>>>>              col(MatStencil_i,5) = i
>>>>>>
>>>>>>              col(MatStencil_j,5) = j+1
>>>>>>
>>>>>>              call MatSetValuesStencil(jac,i1,row,i5,col,v,INSERT_VALUES,ierr)
>>>>>>
>>>>>>          end if
>>>>>>
>>>>>>      end do
>>>>>>
>>>>>> end do
>>>>>>
>>>>>> call MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY,ierr)
>>>>>>
>>>>>> call MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY,ierr)
>>>>>>
>>>>>> !call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,nullspace,ierr)
>>>>>>
>>>>>> !call MatSetNullSpace(jac,nullspace,ierr)
>>>>>>
>>>>>> !call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>
>>>>>> end subroutine ComputeMatrix
>>>>>>
>>>>>> subroutine ComputeRho(i,j,mx,my,centerRho,rho)
>>>>>>
>>>>>> PetscInt    i,j,mx,my
>>>>>>
>>>>>> PetscScalar  rho,centerRho
>>>>>>
>>>>>> if ((i>  mx/3.0) .and. (i<  2.0*mx/3.0) .and. (j>  my/3.0) .and. (j<  2.0*my/3.0)) then
>>>>>>
>>>>>>      rho = centerRho
>>>>>>
>>>>>> else
>>>>>>
>>>>>>      rho = 1.0
>>>>>>
>>>>>> end if
>>>>>>
>>>>>>
>>>>>> end subroutine ComputeRho
>>>>>> -- 
>>>>>> Yours sincerely,
>>>>>>
>>>>>> TAY wee-beng
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>> -- Norbert Wiener
>>>>>
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener
>>>>
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>>
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener

From knepley at gmail.com  Tue May  1 16:54:56 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 May 2012 17:54:56 -0400
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <4FA05A23.7080209@gmail.com>
References: <4F97E8F2.5050803@gmail.com> <4F993F1D.9020105@gmail.com>
	<CAMYG4GnHyb1VKSY3gwpTxyU=xe1ZcEhitKCu5+CpK9a1v0ymuA@mail.gmail.com>
	<4F99A15B.1010500@gmail.com>
	<CAMYG4Gmzkpt+UyYLbzhit9Jr1G6SO99C9qKp+qg6Nyd__rTo3g@mail.gmail.com>
	<4F99CA4C.1080609@gmail.com>
	<C92D4D80-A685-4196-BFF6-85E4CB795D04@mcs.anl.gov>
	<4FA05A23.7080209@gmail.com>
Message-ID: <CAMYG4Gk-vSa_kwRnp58up8cu7YPDN53Lb03JQviB+nduJSy9KQ@mail.gmail.com>

On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng <zonexo at gmail.com> wrote:

> Hi,
>
> Do you mean my method is wrong?
>
> I am following the template of ex22f,
>
> where the variables are declared as :
>
> PetscScalar  v(5)
>
> MatStencil   row(4),col(4,5)
>
> Hence,
>
> for the neumann BC
>
> num = 1
>
>                if (j/=0) then
>
>                    v(num) = -rho*HxdHy
>
>                    col(MatStencil_i,num) = i
>
>                    col(MatStencil_j,num) = j-1
>
>                    num = num + 1
>
>                end if
>
>                if (i/=0) then
>
>                    v(num) = -rho*HydHx
>
>                    col(MatStencil_i,num) = i-1
>
>                    col(MatStencil_j,num) = j
>
>                    num = num + 1
>
>                end if
>
>                if (i/=mx-1) then
>
>                    v(num) = -rho*HydHx
>
>                    col(MatStencil_i,num) = i+1
>
>                    col(MatStencil_j,num) = j
>
>                    num = num + 1
>
>                end if
>
>                if (j/=my-1) then
>
>                    v(num) = -rho*HxdHy
>
>                    col(MatStencil_i,num) = i
>
>                    col(MatStencil_j,num) = j+1
>
>                    num = num + 1
>
>                end if
>
>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>
>                print *, v
>
>                col(MatStencil_i,num) = i
>
>                col(MatStencil_j,num) = j
>
>                !num = num + 1
>
>                call MatSetValuesStencil(jac,i1,**
> row,num,col,v,INSERT_VALUES,**ierr)
>
> I do not get any more out of range error. However,my ans is still
> different from that of ex29 in C.
>

This is very simple. You have an error in your code. Checking it is very
simple: run the code and
break in MatSetValues(). Make sure ex29 makes calls with exactly the same
indices as your ex29f.

    Matt


> Yours sincerely,
>
> TAY wee-beng
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120501/4167ac1b/attachment.htm>

From dkarthik at stanford.edu  Tue May  1 17:12:36 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Tue, 1 May 2012 15:12:36 -0700 (PDT)
Subject: [petsc-users] Multigrid
In-Reply-To: <CB42609E-F0F9-4CC8-B78A-9C39421C45E7@mcs.anl.gov>
Message-ID: <905748401.99886844.1335910356334.JavaMail.root@zm09.stanford.edu>

Hello Barry,

    Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on?

    Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ?

Best regards,
Karthik.

 type: mg
    MG: type is MULTIPLICATIVE, levels=1 cycles=v
      Cycles per PCApply=1
      Not using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (mg_levels_0_)     8 MPI processes
      type: gmres
        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
        GMRES: happy breakdown tolerance 1e-30
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using PRECONDITIONED norm type for convergence test
    PC Object:    (mg_levels_0_)     8 MPI processes
      type: bjacobi
        block Jacobi: number of blocks = 8
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object:      (mg_levels_0_sub_)       1 MPI processes
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_levels_0_sub_)       1 MPI processes
        type: ilu
          ILU: out-of-place factorization
          0 levels of fill
          tolerance for zero pivot 1e-12
          using diagonal shift to prevent zero pivot
          matrix ordering: natural
          factor fill ratio given 1, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes
                type: seqaij
                rows=9015, cols=9015
                package used to perform factorization: petsc
                total: nonzeros=517777, allocated nonzeros=517777
                total number of mallocs used during MatSetValues calls =0
                  using I-node routines: found 3476 nodes, limit used is 5
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes
          type: seqaij
          rows=9015, cols=9015
          total: nonzeros=517777, allocated nonzeros=517777
          total number of mallocs used during MatSetValues calls =0
            using I-node routines: found 3476 nodes, limit used is 5
      linear system matrix = precond matrix:
      Matrix Object:       8 MPI processes
        type: mpiaij
        rows=75000, cols=75000
        total: nonzeros=4427800, allocated nonzeros=4427800
         total number of mallocs used during MatSetValues calls =0
          using I-node (on process 0) routines: found 3476 nodes, limit used is 5
  linear system matrix = precond matrix:
  Matrix Object:   8 MPI processes
    type: mpiaij
    rows=75000, cols=75000
    total: nonzeros=4427800, allocated nonzeros=4427800
    total number of mallocs used during MatSetValues calls =0
      using I-node (on process 0) routines: found 3476 nodes, limit used is 5



----- Original Message -----
From: "Barry Smith" <bsmith at mcs.anl.gov>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Tuesday, May 1, 2012 1:39:26 PM
Subject: Re: [petsc-users] Multigrid


On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote:

> Hello,
> 
>        I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
> 
>        What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any
> help in this regard will be appreciated.

   First run with -ksp_view to see what solver it is actually using.

   Barry

> 
>        Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results.
> 
> Thanks,
> Karthik
> 
> 
> 
> -- 
> 
> =======================================
> Karthik Duraisamy
> Assistant Professor (Consulting)
> Durand Building Rm 357
> Dept of Aeronautics and Astronautics
> Stanford University
> Stanford CA 94305
> 
> Phone: 650-721-2835
> Web:     www.stanford.edu/~dkarthik
> =======================================


From knepley at gmail.com  Tue May  1 17:15:14 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 May 2012 18:15:14 -0400
Subject: [petsc-users] Multigrid
In-Reply-To: <905748401.99886844.1335910356334.JavaMail.root@zm09.stanford.edu>
References: <CB42609E-F0F9-4CC8-B78A-9C39421C45E7@mcs.anl.gov>
	<905748401.99886844.1335910356334.JavaMail.root@zm09.stanford.edu>
Message-ID: <CAMYG4G=wg5jPaz66m8srP9NCoBtYK-TAgbSmvTFg0OncNi9mdw@mail.gmail.com>

On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy <dkarthik at stanford.edu>wrote:

> Hello Barry,
>
>    Thank you for your super quick response. I have attached the output of
> ksp_view and it is practically the same as that when I don't use PCMG. The
> part I don't understand is how PCMG able to function at the zero grid level
> and still produce a much better convergence than when using the default PC.
> Is there any additional smoothing or interpolation going on?
>

You only included one output, so I have no way of knowing what you used
before. However, this is running GMRES/ILU.


>    Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ?
>

They are different algorithms. Its not possible to say generally that one
is better. Try them both.

   Matt


> Best regards,
> Karthik.
>
>  type: mg
>    MG: type is MULTIPLICATIVE, levels=1 cycles=v
>      Cycles per PCApply=1
>      Not using Galerkin computed coarse grid matrices
>  Coarse grid solver -- level -------------------------------
>    KSP Object:    (mg_levels_0_)     8 MPI processes
>      type: gmres
>        GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>        GMRES: happy breakdown tolerance 1e-30
>      maximum iterations=1, initial guess is zero
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using PRECONDITIONED norm type for convergence test
>    PC Object:    (mg_levels_0_)     8 MPI processes
>      type: bjacobi
>        block Jacobi: number of blocks = 8
>        Local solve is same for all blocks, in the following KSP and PC
> objects:
>      KSP Object:      (mg_levels_0_sub_)       1 MPI processes
>        type: preonly
>        maximum iterations=10000, initial guess is zero
>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>        left preconditioning
>        using NONE norm type for convergence test
>      PC Object:      (mg_levels_0_sub_)       1 MPI processes
>        type: ilu
>          ILU: out-of-place factorization
>          0 levels of fill
>          tolerance for zero pivot 1e-12
>          using diagonal shift to prevent zero pivot
>          matrix ordering: natural
>          factor fill ratio given 1, needed 1
>            Factored matrix follows:
>              Matrix Object:               1 MPI processes
>                type: seqaij
>                rows=9015, cols=9015
>                package used to perform factorization: petsc
>                total: nonzeros=517777, allocated nonzeros=517777
>                total number of mallocs used during MatSetValues calls =0
>                  using I-node routines: found 3476 nodes, limit used is 5
>        linear system matrix = precond matrix:
>        Matrix Object:         1 MPI processes
>          type: seqaij
>          rows=9015, cols=9015
>          total: nonzeros=517777, allocated nonzeros=517777
>          total number of mallocs used during MatSetValues calls =0
>            using I-node routines: found 3476 nodes, limit used is 5
>      linear system matrix = precond matrix:
>      Matrix Object:       8 MPI processes
>        type: mpiaij
>        rows=75000, cols=75000
>        total: nonzeros=4427800, allocated nonzeros=4427800
>         total number of mallocs used during MatSetValues calls =0
>          using I-node (on process 0) routines: found 3476 nodes, limit
> used is 5
>  linear system matrix = precond matrix:
>  Matrix Object:   8 MPI processes
>    type: mpiaij
>    rows=75000, cols=75000
>    total: nonzeros=4427800, allocated nonzeros=4427800
>    total number of mallocs used during MatSetValues calls =0
>      using I-node (on process 0) routines: found 3476 nodes, limit used is
> 5
>
>
>
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Tuesday, May 1, 2012 1:39:26 PM
> Subject: Re: [petsc-users] Multigrid
>
>
> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote:
>
> > Hello,
> >
> >        I have been using PETSc for a couple of years with good success,
> but lately as my linear problems have become stiffer (condition numbers of
> the order of 1.e20), I am looking to use better preconditioners. I tried
> using PCMG with all the default options (i.e., I just specified my
> preconditioner as PCMG and did not add any options to it) and I am
> immediately seeing better convergence.
> >
> >        What I am not sure of is why? I would like to know more about the
> default parameters (the manual is not very explicit) and more importantly,
> want to know why it is working even when I haven't specified any grid
> levels and coarse grid operators. Any
> > help in this regard will be appreciated.
>
>   First run with -ksp_view to see what solver it is actually using.
>
>   Barry
>
> >
> >        Also, ultimately I want to use algebraic multigrid so is PCML a
> better option than BoomerAMG? I tried BoomerAMG with mixed results.
> >
> > Thanks,
> > Karthik
> >
> >
> >
> > --
> >
> > =======================================
> > Karthik Duraisamy
> > Assistant Professor (Consulting)
> > Durand Building Rm 357
> > Dept of Aeronautics and Astronautics
> > Stanford University
> > Stanford CA 94305
> >
> > Phone: 650-721-2835
> > Web:     www.stanford.edu/~dkarthik
> > =======================================
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120501/c6ac9a02/attachment.htm>

From dkarthik at stanford.edu  Tue May  1 17:18:34 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Tue, 1 May 2012 15:18:34 -0700 (PDT)
Subject: [petsc-users] Multigrid
In-Reply-To: <CAMYG4G=wg5jPaz66m8srP9NCoBtYK-TAgbSmvTFg0OncNi9mdw@mail.gmail.com>
Message-ID: <1257743495.99895161.1335910714903.JavaMail.root@zm09.stanford.edu>

Hello,

   Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same).

Regards,
Karthik

KSP Object: 8 MPI processes
  type: gmres
    GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=1
  using preconditioner applied to right hand side for initial guess
  tolerances:  relative=1e-05, absolute=1e-50, divergence=1e+10
  left preconditioning
  using nonzero initial guess
  using PRECONDITIONED norm type for convergence test
PC Object: 8 MPI processes
  type: bjacobi
    block Jacobi: number of blocks = 8
    Local solve is same for all blocks, in the following KSP and PC objects:
  KSP Object:  (sub_)   1 MPI processes  
    type: preonly
    maximum iterations=10000, initial guess is zero
    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
    left preconditioning
    using NONE norm type for convergence test
  PC Object:  (sub_)   1 MPI processes  
    type: ilu
      ILU: out-of-place factorization
      0 levels of fill
      tolerance for zero pivot 1e-12
      using diagonal shift to prevent zero pivot
      matrix ordering: natural
      factor fill ratio given 1, needed 1
        Factored matrix follows:
          Matrix Object:           1 MPI processes          
            type: seqaij
            rows=9015, cols=9015
            package used to perform factorization: petsc
            total: nonzeros=517777, allocated nonzeros=517777
            total number of mallocs used during MatSetValues calls =0
              using I-node routines: found 3476 nodes, limit used is 5
    linear system matrix = precond matrix:
    Matrix Object:     1 MPI processes    
      type: seqaij
      rows=9015, cols=9015
      total: nonzeros=517777, allocated nonzeros=517777
      total number of mallocs used during MatSetValues calls =0
        using I-node routines: found 3476 nodes, limit used is 5
  linear system matrix = precond matrix:
  Matrix Object:   8 MPI processes  
    type: mpiaij
    rows=75000, cols=75000
    total: nonzeros=4427800, allocated nonzeros=4427800
    total number of mallocs used during MatSetValues calls =0
      using I-node (on process 0) routines: found 3476 nodes, limit used is 5


----- Original Message -----
From: "Matthew Knepley" <knepley at gmail.com>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Tuesday, May 1, 2012 3:15:14 PM
Subject: Re: [petsc-users] Multigrid


On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 



Hello Barry, 

Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 



You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 


Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 



They are different algorithms. Its not possible to say generally that one is better. Try them both. 


Matt 


Best regards, 
Karthik. 

type: mg 
MG: type is MULTIPLICATIVE, levels=1 cycles=v 
Cycles per PCApply=1 
Not using Galerkin computed coarse grid matrices 
Coarse grid solver -- level ------------------------------- 
KSP Object: (mg_levels_0_) 8 MPI processes 
type: gmres 
GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
GMRES: happy breakdown tolerance 1e-30 
maximum iterations=1, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using PRECONDITIONED norm type for convergence test 
PC Object: (mg_levels_0_) 8 MPI processes 
type: bjacobi 
block Jacobi: number of blocks = 8 
Local solve is same for all blocks, in the following KSP and PC objects: 
KSP Object: (mg_levels_0_sub_) 1 MPI processes 
type: preonly 
maximum iterations=10000, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using NONE norm type for convergence test 
PC Object: (mg_levels_0_sub_) 1 MPI processes 
type: ilu 
ILU: out-of-place factorization 
0 levels of fill 
tolerance for zero pivot 1e-12 
using diagonal shift to prevent zero pivot 
matrix ordering: natural 
factor fill ratio given 1, needed 1 
Factored matrix follows: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
package used to perform factorization: petsc 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 



----- Original Message ----- 
From: "Barry Smith" < bsmith at mcs.anl.gov > 
To: "PETSc users list" < petsc-users at mcs.anl.gov > 
Sent: Tuesday, May 1, 2012 1:39:26 PM 
Subject: Re: [petsc-users] Multigrid 


On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 

> Hello, 
> 
> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
> 
> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
> help in this regard will be appreciated. 

First run with -ksp_view to see what solver it is actually using. 

Barry 

> 
> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
> 
> Thanks, 
> Karthik 
> 
> 
> 
> -- 
> 
> ======================================= 
> Karthik Duraisamy 
> Assistant Professor (Consulting) 
> Durand Building Rm 357 
> Dept of Aeronautics and Astronautics 
> Stanford University 
> Stanford CA 94305 
> 
> Phone: 650-721-2835 
> Web: www.stanford.edu/~dkarthik 
> ======================================= 





-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 

From tibo at berkeley.edu  Tue May  1 17:20:38 2012
From: tibo at berkeley.edu (tibo at berkeley.edu)
Date: Tue, 1 May 2012 15:20:38 -0700
Subject: [petsc-users] Interations increase with parallelism
Message-ID: <df2fb81bb3854463c3483575416e79b8.squirrel@calmail.berkeley.edu>

Dear petsc users,

I am solving large systems of non linear PDEs. To do so, the most
expensive operation is to solve linear systems Ax=b, where A is block
tridiagonal. To do so, I am using petsc.

A is created using MatCreateMPIAIJ and x and b using VecCreateMPI, and I
do not modify default parameters of the KSP context for now (ie the Krylov
method is GMRES and the preconditioner method is ILU(0) if I use 1
processor - sequential matrix - and Block Jacobi with ILU(0) for each
sub-block if I use more than 1 processor).

For any number n of processors used, I do get the correct result. However,
it seems that the more processors I have, the more iterations are done on
each linear solve (n = 1 gives about 1-2 iterations per solve, n = 2 gives
8-12 iterations per solve, n = 4 gives 15-20 iterations per solve).

While I can understand the difference between n = 1 and n = 2, since the
preconditioning method changes from ILU(0) to Block Jacobi, I don't
understand why this is the case from n = 2 to 4 for example, since it
seems to me that the method used to solve Ax=b will be the same (although
the partitioning is different) and so the operations will be the same,
even though there will be more communication.

My first question is then: Is this normal behavior or am I probably wrong
somewhere ?

Also, since the increase in the number of iterations more than offsets the
decrease in time spent solving the system when n increase, my program runs
slower with an increasing number of processors, which is the opposite of
what is desired...Would you have suggestions as what I could change to
correct this ?

I would be happy to provide more details about the problem/datastructures
used if needed,

thank you for your help,

Tibo


From dkarthik at stanford.edu  Tue May  1 17:21:15 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Tue, 1 May 2012 15:21:15 -0700 (PDT)
Subject: [petsc-users] Multigrid
In-Reply-To: <1257743495.99895161.1335910714903.JavaMail.root@zm09.stanford.edu>
Message-ID: <791625170.99899464.1335910875888.JavaMail.root@zm09.stanford.edu>

ps: I tried it with and without the non-zero initial guess to check the results, BTW.

----- Original Message -----
From: "Karthik Duraisamy" <dkarthik at stanford.edu>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Tuesday, May 1, 2012 3:18:34 PM
Subject: Re: [petsc-users] Multigrid

Hello,

   Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same).

Regards,
Karthik

KSP Object: 8 MPI processes
  type: gmres
    GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=1
  using preconditioner applied to right hand side for initial guess
  tolerances:  relative=1e-05, absolute=1e-50, divergence=1e+10
  left preconditioning
  using nonzero initial guess
  using PRECONDITIONED norm type for convergence test
PC Object: 8 MPI processes
  type: bjacobi
    block Jacobi: number of blocks = 8
    Local solve is same for all blocks, in the following KSP and PC objects:
  KSP Object:  (sub_)   1 MPI processes  
    type: preonly
    maximum iterations=10000, initial guess is zero
    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
    left preconditioning
    using NONE norm type for convergence test
  PC Object:  (sub_)   1 MPI processes  
    type: ilu
      ILU: out-of-place factorization
      0 levels of fill
      tolerance for zero pivot 1e-12
      using diagonal shift to prevent zero pivot
      matrix ordering: natural
      factor fill ratio given 1, needed 1
        Factored matrix follows:
          Matrix Object:           1 MPI processes          
            type: seqaij
            rows=9015, cols=9015
            package used to perform factorization: petsc
            total: nonzeros=517777, allocated nonzeros=517777
            total number of mallocs used during MatSetValues calls =0
              using I-node routines: found 3476 nodes, limit used is 5
    linear system matrix = precond matrix:
    Matrix Object:     1 MPI processes    
      type: seqaij
      rows=9015, cols=9015
      total: nonzeros=517777, allocated nonzeros=517777
      total number of mallocs used during MatSetValues calls =0
        using I-node routines: found 3476 nodes, limit used is 5
  linear system matrix = precond matrix:
  Matrix Object:   8 MPI processes  
    type: mpiaij
    rows=75000, cols=75000
    total: nonzeros=4427800, allocated nonzeros=4427800
    total number of mallocs used during MatSetValues calls =0
      using I-node (on process 0) routines: found 3476 nodes, limit used is 5


----- Original Message -----
From: "Matthew Knepley" <knepley at gmail.com>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Tuesday, May 1, 2012 3:15:14 PM
Subject: Re: [petsc-users] Multigrid


On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 



Hello Barry, 

Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 



You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 


Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 



They are different algorithms. Its not possible to say generally that one is better. Try them both. 


Matt 


Best regards, 
Karthik. 

type: mg 
MG: type is MULTIPLICATIVE, levels=1 cycles=v 
Cycles per PCApply=1 
Not using Galerkin computed coarse grid matrices 
Coarse grid solver -- level ------------------------------- 
KSP Object: (mg_levels_0_) 8 MPI processes 
type: gmres 
GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
GMRES: happy breakdown tolerance 1e-30 
maximum iterations=1, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using PRECONDITIONED norm type for convergence test 
PC Object: (mg_levels_0_) 8 MPI processes 
type: bjacobi 
block Jacobi: number of blocks = 8 
Local solve is same for all blocks, in the following KSP and PC objects: 
KSP Object: (mg_levels_0_sub_) 1 MPI processes 
type: preonly 
maximum iterations=10000, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using NONE norm type for convergence test 
PC Object: (mg_levels_0_sub_) 1 MPI processes 
type: ilu 
ILU: out-of-place factorization 
0 levels of fill 
tolerance for zero pivot 1e-12 
using diagonal shift to prevent zero pivot 
matrix ordering: natural 
factor fill ratio given 1, needed 1 
Factored matrix follows: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
package used to perform factorization: petsc 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 



----- Original Message ----- 
From: "Barry Smith" < bsmith at mcs.anl.gov > 
To: "PETSc users list" < petsc-users at mcs.anl.gov > 
Sent: Tuesday, May 1, 2012 1:39:26 PM 
Subject: Re: [petsc-users] Multigrid 


On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 

> Hello, 
> 
> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
> 
> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
> help in this regard will be appreciated. 

First run with -ksp_view to see what solver it is actually using. 

Barry 

> 
> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
> 
> Thanks, 
> Karthik 
> 
> 
> 
> -- 
> 
> ======================================= 
> Karthik Duraisamy 
> Assistant Professor (Consulting) 
> Durand Building Rm 357 
> Dept of Aeronautics and Astronautics 
> Stanford University 
> Stanford CA 94305 
> 
> Phone: 650-721-2835 
> Web: www.stanford.edu/~dkarthik 
> ======================================= 





-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 

From knepley at gmail.com  Tue May  1 17:22:56 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 May 2012 18:22:56 -0400
Subject: [petsc-users] Multigrid
In-Reply-To: <1257743495.99895161.1335910714903.JavaMail.root@zm09.stanford.edu>
References: <CAMYG4G=wg5jPaz66m8srP9NCoBtYK-TAgbSmvTFg0OncNi9mdw@mail.gmail.com>
	<1257743495.99895161.1335910714903.JavaMail.root@zm09.stanford.edu>
Message-ID: <CAMYG4GmMg3piKcJWfAfg9YpcgEpcjwsV1KsTe2Tmr=d3xAzrqQ@mail.gmail.com>

On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy <dkarthik at stanford.edu>wrote:

> Hello,
>
>   Sorry (and thanks for the reply). I've attached the no multigrid case. I
> didn't include it because (at least to the untrained eye, everything looks
> the same).
>

Did you send all the output from the MG case? There must be a PC around it.
By default its GMRES, so there would be
an extra GMRES loop compared to the case without MG.

   Matt


> Regards,
> Karthik
>
> KSP Object: 8 MPI processes
>  type: gmres
>    GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>    GMRES: happy breakdown tolerance 1e-30
>  maximum iterations=1
>  using preconditioner applied to right hand side for initial guess
>  tolerances:  relative=1e-05, absolute=1e-50, divergence=1e+10
>  left preconditioning
>  using nonzero initial guess
>  using PRECONDITIONED norm type for convergence test
> PC Object: 8 MPI processes
>  type: bjacobi
>    block Jacobi: number of blocks = 8
>    Local solve is same for all blocks, in the following KSP and PC objects:
>  KSP Object:  (sub_)   1 MPI processes
>    type: preonly
>    maximum iterations=10000, initial guess is zero
>    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>    left preconditioning
>    using NONE norm type for convergence test
>  PC Object:  (sub_)   1 MPI processes
>    type: ilu
>      ILU: out-of-place factorization
>      0 levels of fill
>      tolerance for zero pivot 1e-12
>      using diagonal shift to prevent zero pivot
>      matrix ordering: natural
>      factor fill ratio given 1, needed 1
>        Factored matrix follows:
>          Matrix Object:           1 MPI processes
>            type: seqaij
>            rows=9015, cols=9015
>            package used to perform factorization: petsc
>            total: nonzeros=517777, allocated nonzeros=517777
>            total number of mallocs used during MatSetValues calls =0
>              using I-node routines: found 3476 nodes, limit used is 5
>    linear system matrix = precond matrix:
>    Matrix Object:     1 MPI processes
>      type: seqaij
>      rows=9015, cols=9015
>      total: nonzeros=517777, allocated nonzeros=517777
>      total number of mallocs used during MatSetValues calls =0
>        using I-node routines: found 3476 nodes, limit used is 5
>  linear system matrix = precond matrix:
>  Matrix Object:   8 MPI processes
>    type: mpiaij
>    rows=75000, cols=75000
>    total: nonzeros=4427800, allocated nonzeros=4427800
>    total number of mallocs used during MatSetValues calls =0
>      using I-node (on process 0) routines: found 3476 nodes, limit used is
> 5
>
>
> ----- Original Message -----
> From: "Matthew Knepley" <knepley at gmail.com>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Tuesday, May 1, 2012 3:15:14 PM
> Subject: Re: [petsc-users] Multigrid
>
>
> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu> wrote:
>
>
>
> Hello Barry,
>
> Thank you for your super quick response. I have attached the output of
> ksp_view and it is practically the same as that when I don't use PCMG. The
> part I don't understand is how PCMG able to function at the zero grid level
> and still produce a much better convergence than when using the default PC.
> Is there any additional smoothing or interpolation going on?
>
>
>
> You only included one output, so I have no way of knowing what you used
> before. However, this is running GMRES/ILU.
>
>
> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ?
>
>
>
> They are different algorithms. Its not possible to say generally that one
> is better. Try them both.
>
>
> Matt
>
>
> Best regards,
> Karthik.
>
> type: mg
> MG: type is MULTIPLICATIVE, levels=1 cycles=v
> Cycles per PCApply=1
> Not using Galerkin computed coarse grid matrices
> Coarse grid solver -- level -------------------------------
> KSP Object: (mg_levels_0_) 8 MPI processes
> type: gmres
> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
> maximum iterations=1, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using PRECONDITIONED norm type for convergence test
> PC Object: (mg_levels_0_) 8 MPI processes
> type: bjacobi
> block Jacobi: number of blocks = 8
> Local solve is same for all blocks, in the following KSP and PC objects:
> KSP Object: (mg_levels_0_sub_) 1 MPI processes
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using NONE norm type for convergence test
> PC Object: (mg_levels_0_sub_) 1 MPI processes
> type: ilu
> ILU: out-of-place factorization
> 0 levels of fill
> tolerance for zero pivot 1e-12
> using diagonal shift to prevent zero pivot
> matrix ordering: natural
> factor fill ratio given 1, needed 1
> Factored matrix follows:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> package used to perform factorization: petsc
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 8 MPI processes
> type: mpiaij
> rows=75000, cols=75000
> total: nonzeros=4427800, allocated nonzeros=4427800
> total number of mallocs used during MatSetValues calls =0
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 8 MPI processes
> type: mpiaij
> rows=75000, cols=75000
> total: nonzeros=4427800, allocated nonzeros=4427800
> total number of mallocs used during MatSetValues calls =0
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5
>
>
>
> ----- Original Message -----
> From: "Barry Smith" < bsmith at mcs.anl.gov >
> To: "PETSc users list" < petsc-users at mcs.anl.gov >
> Sent: Tuesday, May 1, 2012 1:39:26 PM
> Subject: Re: [petsc-users] Multigrid
>
>
> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote:
>
> > Hello,
> >
> > I have been using PETSc for a couple of years with good success, but
> lately as my linear problems have become stiffer (condition numbers of the
> order of 1.e20), I am looking to use better preconditioners. I tried using
> PCMG with all the default options (i.e., I just specified my preconditioner
> as PCMG and did not add any options to it) and I am immediately seeing
> better convergence.
> >
> > What I am not sure of is why? I would like to know more about the
> default parameters (the manual is not very explicit) and more importantly,
> want to know why it is working even when I haven't specified any grid
> levels and coarse grid operators. Any
> > help in this regard will be appreciated.
>
> First run with -ksp_view to see what solver it is actually using.
>
> Barry
>
> >
> > Also, ultimately I want to use algebraic multigrid so is PCML a better
> option than BoomerAMG? I tried BoomerAMG with mixed results.
> >
> > Thanks,
> > Karthik
> >
> >
> >
> > --
> >
> > =======================================
> > Karthik Duraisamy
> > Assistant Professor (Consulting)
> > Durand Building Rm 357
> > Dept of Aeronautics and Astronautics
> > Stanford University
> > Stanford CA 94305
> >
> > Phone: 650-721-2835
> > Web: www.stanford.edu/~dkarthik
> > =======================================
>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120501/7fa99aba/attachment.htm>

From knepley at gmail.com  Tue May  1 17:26:15 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 May 2012 18:26:15 -0400
Subject: [petsc-users] Interations increase with parallelism
In-Reply-To: <df2fb81bb3854463c3483575416e79b8.squirrel@calmail.berkeley.edu>
References: <df2fb81bb3854463c3483575416e79b8.squirrel@calmail.berkeley.edu>
Message-ID: <CAMYG4GkkcD8EC8Wa58o_=76MAtGGHgnCCEe8tzK-8gkevvhXoA@mail.gmail.com>

On Tue, May 1, 2012 at 6:20 PM, <tibo at berkeley.edu> wrote:

> Dear petsc users,
>
> I am solving large systems of non linear PDEs. To do so, the most
> expensive operation is to solve linear systems Ax=b, where A is block
> tridiagonal. To do so, I am using petsc.
>
> A is created using MatCreateMPIAIJ and x and b using VecCreateMPI, and I
> do not modify default parameters of the KSP context for now (ie the Krylov
> method is GMRES and the preconditioner method is ILU(0) if I use 1
> processor - sequential matrix - and Block Jacobi with ILU(0) for each
> sub-block if I use more than 1 processor).
>
> For any number n of processors used, I do get the correct result. However,
> it seems that the more processors I have, the more iterations are done on
> each linear solve (n = 1 gives about 1-2 iterations per solve, n = 2 gives
> 8-12 iterations per solve, n = 4 gives 15-20 iterations per solve).
>
> While I can understand the difference between n = 1 and n = 2, since the
> preconditioning method changes from ILU(0) to Block Jacobi, I don't
> understand why this is the case from n = 2 to 4 for example, since it
> seems to me that the method used to solve Ax=b will be the same (although
> the partitioning is different) and so the operations will be the same,
> even though there will be more communication.
>
> My first question is then: Is this normal behavior or am I probably wrong
> somewhere ?
>

Its not the same from 2-4. You have 4 smaller blocks in BJACOBI and less
coupling.

Also, since the increase in the number of iterations more than offsets the
> decrease in time spent solving the system when n increase, my program runs
> slower with an increasing number of processors, which is the opposite of
> what is desired...Would you have suggestions as what I could change to
> correct this ?
>

Scalable preconditioning is a research topic. However, the next thing to
try is AMG.


> I would be happy to provide more details about the problem/datastructures
> used if needed,
>

The best thing to try usually is a literature search for people using
scalable preconditioners
for your kind of physics.

    Matt


> thank you for your help,
>
> Tibo
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120501/e244893b/attachment-0001.htm>

From dkarthik at stanford.edu  Tue May  1 17:27:41 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Tue, 1 May 2012 15:27:41 -0700 (PDT)
Subject: [petsc-users] Multigrid
In-Reply-To: <CAMYG4GmMg3piKcJWfAfg9YpcgEpcjwsV1KsTe2Tmr=d3xAzrqQ@mail.gmail.com>
Message-ID: <1357435407.100000640.1335911261741.JavaMail.root@zm09.stanford.edu>

The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow)



KSP Object: 8 MPI processes
  type: gmres
    GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=1, initial guess is zero
  using preconditioner applied to right hand side for initial guess
  tolerances:  relative=0.01, absolute=1e-08, divergence=1e+10
  left preconditioning
  using DEFAULT norm type for convergence test
PC Object: 8 MPI processes
  type: mg
    MG: type is MULTIPLICATIVE, levels=1 cycles=v
      Cycles per PCApply=1
      Not using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (mg_levels_0_)     8 MPI processes
      type not yet set
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning   
      using DEFAULT norm type for convergence test
    PC Object:    (mg_levels_0_)     8 MPI processes
      type not yet set
  linear system matrix = precond matrix:
  Matrix Object:   8 MPI processes  
    type: mpiaij
    rows=75000, cols=75000
    total: nonzeros=4427800, allocated nonzeros=4427800
    total number of mallocs used during MatSetValues calls =0
      using I-node (on process 0) routines: found 3476 nodes, limit used is 5


----- Original Message -----
From: "Matthew Knepley" <knepley at gmail.com>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Tuesday, May 1, 2012 3:22:56 PM
Subject: Re: [petsc-users] Multigrid


On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 



Hello, 

Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same). 



Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be 
an extra GMRES loop compared to the case without MG. 


Matt 


Regards, 
Karthik 

KSP Object: 8 MPI processes 
type: gmres 
GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
GMRES: happy breakdown tolerance 1e-30 
maximum iterations=1 
using preconditioner applied to right hand side for initial guess 
tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 
left preconditioning 
using nonzero initial guess 
using PRECONDITIONED norm type for convergence test 
PC Object: 8 MPI processes 
type: bjacobi 
block Jacobi: number of blocks = 8 
Local solve is same for all blocks, in the following KSP and PC objects: 
KSP Object: (sub_) 1 MPI processes 
type: preonly 
maximum iterations=10000, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using NONE norm type for convergence test 
PC Object: (sub_) 1 MPI processes 
type: ilu 
ILU: out-of-place factorization 
0 levels of fill 
tolerance for zero pivot 1e-12 
using diagonal shift to prevent zero pivot 
matrix ordering: natural 
factor fill ratio given 1, needed 1 
Factored matrix follows: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
package used to perform factorization: petsc 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 


----- Original Message ----- 
From: "Matthew Knepley" < knepley at gmail.com > 
To: "PETSc users list" < petsc-users at mcs.anl.gov > 
Sent: Tuesday, May 1, 2012 3:15:14 PM 
Subject: Re: [petsc-users] Multigrid 


On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 



Hello Barry, 

Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 



You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 


Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 



They are different algorithms. Its not possible to say generally that one is better. Try them both. 


Matt 


Best regards, 
Karthik. 

type: mg 
MG: type is MULTIPLICATIVE, levels=1 cycles=v 
Cycles per PCApply=1 
Not using Galerkin computed coarse grid matrices 
Coarse grid solver -- level ------------------------------- 
KSP Object: (mg_levels_0_) 8 MPI processes 
type: gmres 
GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
GMRES: happy breakdown tolerance 1e-30 
maximum iterations=1, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using PRECONDITIONED norm type for convergence test 
PC Object: (mg_levels_0_) 8 MPI processes 
type: bjacobi 
block Jacobi: number of blocks = 8 
Local solve is same for all blocks, in the following KSP and PC objects: 
KSP Object: (mg_levels_0_sub_) 1 MPI processes 
type: preonly 
maximum iterations=10000, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using NONE norm type for convergence test 
PC Object: (mg_levels_0_sub_) 1 MPI processes 
type: ilu 
ILU: out-of-place factorization 
0 levels of fill 
tolerance for zero pivot 1e-12 
using diagonal shift to prevent zero pivot 
matrix ordering: natural 
factor fill ratio given 1, needed 1 
Factored matrix follows: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
package used to perform factorization: petsc 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 



----- Original Message ----- 
From: "Barry Smith" < bsmith at mcs.anl.gov > 
To: "PETSc users list" < petsc-users at mcs.anl.gov > 
Sent: Tuesday, May 1, 2012 1:39:26 PM 
Subject: Re: [petsc-users] Multigrid 


On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 

> Hello, 
> 
> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
> 
> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
> help in this regard will be appreciated. 

First run with -ksp_view to see what solver it is actually using. 

Barry 

> 
> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
> 
> Thanks, 
> Karthik 
> 
> 
> 
> -- 
> 
> ======================================= 
> Karthik Duraisamy 
> Assistant Professor (Consulting) 
> Durand Building Rm 357 
> Dept of Aeronautics and Astronautics 
> Stanford University 
> Stanford CA 94305 
> 
> Phone: 650-721-2835 
> Web: www.stanford.edu/~dkarthik 
> ======================================= 





-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 




-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 

From tibo at berkeley.edu  Tue May  1 17:29:16 2012
From: tibo at berkeley.edu (tibo at berkeley.edu)
Date: Tue, 1 May 2012 15:29:16 -0700
Subject: [petsc-users] Interations increase with parallelism
In-Reply-To: <CAMYG4GkkcD8EC8Wa58o_=76MAtGGHgnCCEe8tzK-8gkevvhXoA@mail.gmail.com>
References: <df2fb81bb3854463c3483575416e79b8.squirrel@calmail.berkeley.edu>
	<CAMYG4GkkcD8EC8Wa58o_=76MAtGGHgnCCEe8tzK-8gkevvhXoA@mail.gmail.com>
Message-ID: <7e5ce7ee8784e03813d0d1dbe644430f.squirrel@calmail.berkeley.edu>

Thank you for your answer,
I will then investigate more the topic of scalable preconditioners, as
well as try different preconditioners.

> On Tue, May 1, 2012 at 6:20 PM, <tibo at berkeley.edu> wrote:
>
>> Dear petsc users,
>>
>> I am solving large systems of non linear PDEs. To do so, the most
>> expensive operation is to solve linear systems Ax=b, where A is block
>> tridiagonal. To do so, I am using petsc.
>>
>> A is created using MatCreateMPIAIJ and x and b using VecCreateMPI, and I
>> do not modify default parameters of the KSP context for now (ie the
>> Krylov
>> method is GMRES and the preconditioner method is ILU(0) if I use 1
>> processor - sequential matrix - and Block Jacobi with ILU(0) for each
>> sub-block if I use more than 1 processor).
>>
>> For any number n of processors used, I do get the correct result.
>> However,
>> it seems that the more processors I have, the more iterations are done
>> on
>> each linear solve (n = 1 gives about 1-2 iterations per solve, n = 2
>> gives
>> 8-12 iterations per solve, n = 4 gives 15-20 iterations per solve).
>>
>> While I can understand the difference between n = 1 and n = 2, since the
>> preconditioning method changes from ILU(0) to Block Jacobi, I don't
>> understand why this is the case from n = 2 to 4 for example, since it
>> seems to me that the method used to solve Ax=b will be the same
>> (although
>> the partitioning is different) and so the operations will be the same,
>> even though there will be more communication.
>>
>> My first question is then: Is this normal behavior or am I probably
>> wrong
>> somewhere ?
>>
>
> Its not the same from 2-4. You have 4 smaller blocks in BJACOBI and less
> coupling.
>
> Also, since the increase in the number of iterations more than offsets the
>> decrease in time spent solving the system when n increase, my program
>> runs
>> slower with an increasing number of processors, which is the opposite of
>> what is desired...Would you have suggestions as what I could change to
>> correct this ?
>>
>
> Scalable preconditioning is a research topic. However, the next thing to
> try is AMG.
>
>
>> I would be happy to provide more details about the
>> problem/datastructures
>> used if needed,
>>
>
> The best thing to try usually is a literature search for people using
> scalable preconditioners
> for your kind of physics.
>
>     Matt
>
>
>> thank you for your help,
>>
>> Tibo
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



From knepley at gmail.com  Tue May  1 17:31:05 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 May 2012 18:31:05 -0400
Subject: [petsc-users] Multigrid
In-Reply-To: <1357435407.100000640.1335911261741.JavaMail.root@zm09.stanford.edu>
References: <CAMYG4GmMg3piKcJWfAfg9YpcgEpcjwsV1KsTe2Tmr=d3xAzrqQ@mail.gmail.com>
	<1357435407.100000640.1335911261741.JavaMail.root@zm09.stanford.edu>
Message-ID: <CAMYG4Gn7dmN05jC3ybQnw9WCV07KCVuEo+n6t1pOKDOr57a9=Q@mail.gmail.com>

On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy <dkarthik at stanford.edu>wrote:

> The following was output for the very first iteration whereas what I had
> attached earlier was output every iteration. I am still a bit perplexed
> because PCMG drops the residual like a rock (after the first few iterations
> whereas with no PCMG, it is very slow)
>

Because the smoother IS the solver you were using before. Just like I said
last time, what you are doing is
wrapping up the same solver you used before, sticking it in another GMRES
loop, and only looking at the
outer loop. This has nothing to do with MG.

   Matt


> KSP Object: 8 MPI processes
>  type: gmres
>    GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>    GMRES: happy breakdown tolerance 1e-30
>  maximum iterations=1, initial guess is zero
>  using preconditioner applied to right hand side for initial guess
>  tolerances:  relative=0.01, absolute=1e-08, divergence=1e+10
>  left preconditioning
>  using DEFAULT norm type for convergence test
> PC Object: 8 MPI processes
>  type: mg
>    MG: type is MULTIPLICATIVE, levels=1 cycles=v
>      Cycles per PCApply=1
>      Not using Galerkin computed coarse grid matrices
>  Coarse grid solver -- level -------------------------------
>    KSP Object:    (mg_levels_0_)     8 MPI processes
>      type not yet set
>      maximum iterations=1, initial guess is zero
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using DEFAULT norm type for convergence test
>    PC Object:    (mg_levels_0_)     8 MPI processes
>      type not yet set
>  linear system matrix = precond matrix:
>  Matrix Object:   8 MPI processes
>    type: mpiaij
>    rows=75000, cols=75000
>    total: nonzeros=4427800, allocated nonzeros=4427800
>    total number of mallocs used during MatSetValues calls =0
>      using I-node (on process 0) routines: found 3476 nodes, limit used is
> 5
>
>
> ----- Original Message -----
> From: "Matthew Knepley" <knepley at gmail.com>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Tuesday, May 1, 2012 3:22:56 PM
> Subject: Re: [petsc-users] Multigrid
>
>
> On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu> wrote:
>
>
>
> Hello,
>
> Sorry (and thanks for the reply). I've attached the no multigrid case. I
> didn't include it because (at least to the untrained eye, everything looks
> the same).
>
>
>
> Did you send all the output from the MG case? There must be a PC around
> it. By default its GMRES, so there would be
> an extra GMRES loop compared to the case without MG.
>
>
> Matt
>
>
> Regards,
> Karthik
>
> KSP Object: 8 MPI processes
> type: gmres
> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
> maximum iterations=1
> using preconditioner applied to right hand side for initial guess
> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10
> left preconditioning
> using nonzero initial guess
> using PRECONDITIONED norm type for convergence test
> PC Object: 8 MPI processes
> type: bjacobi
> block Jacobi: number of blocks = 8
> Local solve is same for all blocks, in the following KSP and PC objects:
> KSP Object: (sub_) 1 MPI processes
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using NONE norm type for convergence test
> PC Object: (sub_) 1 MPI processes
> type: ilu
> ILU: out-of-place factorization
> 0 levels of fill
> tolerance for zero pivot 1e-12
> using diagonal shift to prevent zero pivot
> matrix ordering: natural
> factor fill ratio given 1, needed 1
> Factored matrix follows:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> package used to perform factorization: petsc
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 8 MPI processes
> type: mpiaij
> rows=75000, cols=75000
> total: nonzeros=4427800, allocated nonzeros=4427800
> total number of mallocs used during MatSetValues calls =0
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5
>
>
> ----- Original Message -----
> From: "Matthew Knepley" < knepley at gmail.com >
> To: "PETSc users list" < petsc-users at mcs.anl.gov >
> Sent: Tuesday, May 1, 2012 3:15:14 PM
> Subject: Re: [petsc-users] Multigrid
>
>
> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu> wrote:
>
>
>
> Hello Barry,
>
> Thank you for your super quick response. I have attached the output of
> ksp_view and it is practically the same as that when I don't use PCMG. The
> part I don't understand is how PCMG able to function at the zero grid level
> and still produce a much better convergence than when using the default PC.
> Is there any additional smoothing or interpolation going on?
>
>
>
> You only included one output, so I have no way of knowing what you used
> before. However, this is running GMRES/ILU.
>
>
> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ?
>
>
>
> They are different algorithms. Its not possible to say generally that one
> is better. Try them both.
>
>
> Matt
>
>
> Best regards,
> Karthik.
>
> type: mg
> MG: type is MULTIPLICATIVE, levels=1 cycles=v
> Cycles per PCApply=1
> Not using Galerkin computed coarse grid matrices
> Coarse grid solver -- level -------------------------------
> KSP Object: (mg_levels_0_) 8 MPI processes
> type: gmres
> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
> maximum iterations=1, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using PRECONDITIONED norm type for convergence test
> PC Object: (mg_levels_0_) 8 MPI processes
> type: bjacobi
> block Jacobi: number of blocks = 8
> Local solve is same for all blocks, in the following KSP and PC objects:
> KSP Object: (mg_levels_0_sub_) 1 MPI processes
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using NONE norm type for convergence test
> PC Object: (mg_levels_0_sub_) 1 MPI processes
> type: ilu
> ILU: out-of-place factorization
> 0 levels of fill
> tolerance for zero pivot 1e-12
> using diagonal shift to prevent zero pivot
> matrix ordering: natural
> factor fill ratio given 1, needed 1
> Factored matrix follows:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> package used to perform factorization: petsc
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 8 MPI processes
> type: mpiaij
> rows=75000, cols=75000
> total: nonzeros=4427800, allocated nonzeros=4427800
> total number of mallocs used during MatSetValues calls =0
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 8 MPI processes
> type: mpiaij
> rows=75000, cols=75000
> total: nonzeros=4427800, allocated nonzeros=4427800
> total number of mallocs used during MatSetValues calls =0
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5
>
>
>
> ----- Original Message -----
> From: "Barry Smith" < bsmith at mcs.anl.gov >
> To: "PETSc users list" < petsc-users at mcs.anl.gov >
> Sent: Tuesday, May 1, 2012 1:39:26 PM
> Subject: Re: [petsc-users] Multigrid
>
>
> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote:
>
> > Hello,
> >
> > I have been using PETSc for a couple of years with good success, but
> lately as my linear problems have become stiffer (condition numbers of the
> order of 1.e20), I am looking to use better preconditioners. I tried using
> PCMG with all the default options (i.e., I just specified my preconditioner
> as PCMG and did not add any options to it) and I am immediately seeing
> better convergence.
> >
> > What I am not sure of is why? I would like to know more about the
> default parameters (the manual is not very explicit) and more importantly,
> want to know why it is working even when I haven't specified any grid
> levels and coarse grid operators. Any
> > help in this regard will be appreciated.
>
> First run with -ksp_view to see what solver it is actually using.
>
> Barry
>
> >
> > Also, ultimately I want to use algebraic multigrid so is PCML a better
> option than BoomerAMG? I tried BoomerAMG with mixed results.
> >
> > Thanks,
> > Karthik
> >
> >
> >
> > --
> >
> > =======================================
> > Karthik Duraisamy
> > Assistant Professor (Consulting)
> > Durand Building Rm 357
> > Dept of Aeronautics and Astronautics
> > Stanford University
> > Stanford CA 94305
> >
> > Phone: 650-721-2835
> > Web: www.stanford.edu/~dkarthik
> > =======================================
>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120501/2a26fb6b/attachment-0001.htm>

From mark.adams at columbia.edu  Tue May  1 17:50:31 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Tue, 1 May 2012 18:50:31 -0400
Subject: [petsc-users] Multigrid
In-Reply-To: <CAMYG4Gn7dmN05jC3ybQnw9WCV07KCVuEo+n6t1pOKDOr57a9=Q@mail.gmail.com>
References: <CAMYG4GmMg3piKcJWfAfg9YpcgEpcjwsV1KsTe2Tmr=d3xAzrqQ@mail.gmail.com>
	<1357435407.100000640.1335911261741.JavaMail.root@zm09.stanford.edu>
	<CAMYG4Gn7dmN05jC3ybQnw9WCV07KCVuEo+n6t1pOKDOr57a9=Q@mail.gmail.com>
Message-ID: <C088813F-92A4-4523-A9EB-565659AA9203@columbia.edu>

Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ matrix.  If you use regular grids (DA?) then PETSc can construct geometric multigrid coarse grid spaces, although I don't know if PCMG will construct these for you (I don't think it will and I can see from your output that PCMG just used one grid).  'ml', hypre' and 'gamg' (a native AMG solver) will do real AMG solvers for you.  All three can work on a similar class of problems.

Also, you mention that you have a condition number of 1.e20.  That is astronomical for such a small problem.  How did you compute that number?  Do you know where the ill-conditioning comes from? Is this an elliptic operator?

Mark


On May 1, 2012, at 6:31 PM, Matthew Knepley wrote:

> On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy <dkarthik at stanford.edu> wrote:
> The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow)
> 
> Because the smoother IS the solver you were using before. Just like I said last time, what you are doing is
> wrapping up the same solver you used before, sticking it in another GMRES loop, and only looking at the
> outer loop. This has nothing to do with MG.
> 
>    Matt
>  
> KSP Object: 8 MPI processes
>  type: gmres
>    GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>    GMRES: happy breakdown tolerance 1e-30
>  maximum iterations=1, initial guess is zero
>  using preconditioner applied to right hand side for initial guess
>  tolerances:  relative=0.01, absolute=1e-08, divergence=1e+10
>  left preconditioning
>  using DEFAULT norm type for convergence test
> PC Object: 8 MPI processes
>  type: mg
>    MG: type is MULTIPLICATIVE, levels=1 cycles=v
>      Cycles per PCApply=1
>      Not using Galerkin computed coarse grid matrices
>  Coarse grid solver -- level -------------------------------
>    KSP Object:    (mg_levels_0_)     8 MPI processes
>      type not yet set
>      maximum iterations=1, initial guess is zero
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>      left preconditioning
>      using DEFAULT norm type for convergence test
>    PC Object:    (mg_levels_0_)     8 MPI processes
>      type not yet set
>  linear system matrix = precond matrix:
>  Matrix Object:   8 MPI processes
>    type: mpiaij
>    rows=75000, cols=75000
>    total: nonzeros=4427800, allocated nonzeros=4427800
>    total number of mallocs used during MatSetValues calls =0
>      using I-node (on process 0) routines: found 3476 nodes, limit used is 5
> 
> 
> ----- Original Message -----
> From: "Matthew Knepley" <knepley at gmail.com>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Tuesday, May 1, 2012 3:22:56 PM
> Subject: Re: [petsc-users] Multigrid
> 
> 
> On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote:
> 
> 
> 
> Hello,
> 
> Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same).
> 
> 
> 
> Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be
> an extra GMRES loop compared to the case without MG.
> 
> 
> Matt
> 
> 
> Regards,
> Karthik
> 
> KSP Object: 8 MPI processes
> type: gmres
> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
> maximum iterations=1
> using preconditioner applied to right hand side for initial guess
> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10
> left preconditioning
> using nonzero initial guess
> using PRECONDITIONED norm type for convergence test
> PC Object: 8 MPI processes
> type: bjacobi
> block Jacobi: number of blocks = 8
> Local solve is same for all blocks, in the following KSP and PC objects:
> KSP Object: (sub_) 1 MPI processes
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using NONE norm type for convergence test
> PC Object: (sub_) 1 MPI processes
> type: ilu
> ILU: out-of-place factorization
> 0 levels of fill
> tolerance for zero pivot 1e-12
> using diagonal shift to prevent zero pivot
> matrix ordering: natural
> factor fill ratio given 1, needed 1
> Factored matrix follows:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> package used to perform factorization: petsc
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 8 MPI processes
> type: mpiaij
> rows=75000, cols=75000
> total: nonzeros=4427800, allocated nonzeros=4427800
> total number of mallocs used during MatSetValues calls =0
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5
> 
> 
> ----- Original Message -----
> From: "Matthew Knepley" < knepley at gmail.com >
> To: "PETSc users list" < petsc-users at mcs.anl.gov >
> Sent: Tuesday, May 1, 2012 3:15:14 PM
> Subject: Re: [petsc-users] Multigrid
> 
> 
> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote:
> 
> 
> 
> Hello Barry,
> 
> Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on?
> 
> 
> 
> You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU.
> 
> 
> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ?
> 
> 
> 
> They are different algorithms. Its not possible to say generally that one is better. Try them both.
> 
> 
> Matt
> 
> 
> Best regards,
> Karthik.
> 
> type: mg
> MG: type is MULTIPLICATIVE, levels=1 cycles=v
> Cycles per PCApply=1
> Not using Galerkin computed coarse grid matrices
> Coarse grid solver -- level -------------------------------
> KSP Object: (mg_levels_0_) 8 MPI processes
> type: gmres
> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
> maximum iterations=1, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using PRECONDITIONED norm type for convergence test
> PC Object: (mg_levels_0_) 8 MPI processes
> type: bjacobi
> block Jacobi: number of blocks = 8
> Local solve is same for all blocks, in the following KSP and PC objects:
> KSP Object: (mg_levels_0_sub_) 1 MPI processes
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using NONE norm type for convergence test
> PC Object: (mg_levels_0_sub_) 1 MPI processes
> type: ilu
> ILU: out-of-place factorization
> 0 levels of fill
> tolerance for zero pivot 1e-12
> using diagonal shift to prevent zero pivot
> matrix ordering: natural
> factor fill ratio given 1, needed 1
> Factored matrix follows:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> package used to perform factorization: petsc
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 8 MPI processes
> type: mpiaij
> rows=75000, cols=75000
> total: nonzeros=4427800, allocated nonzeros=4427800
> total number of mallocs used during MatSetValues calls =0
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 8 MPI processes
> type: mpiaij
> rows=75000, cols=75000
> total: nonzeros=4427800, allocated nonzeros=4427800
> total number of mallocs used during MatSetValues calls =0
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5
> 
> 
> 
> ----- Original Message -----
> From: "Barry Smith" < bsmith at mcs.anl.gov >
> To: "PETSc users list" < petsc-users at mcs.anl.gov >
> Sent: Tuesday, May 1, 2012 1:39:26 PM
> Subject: Re: [petsc-users] Multigrid
> 
> 
> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote:
> 
> > Hello,
> >
> > I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence.
> >
> > What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any
> > help in this regard will be appreciated.
> 
> First run with -ksp_view to see what solver it is actually using.
> 
> Barry
> 
> >
> > Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results.
> >
> > Thanks,
> > Karthik
> >
> >
> >
> > --
> >
> > =======================================
> > Karthik Duraisamy
> > Assistant Professor (Consulting)
> > Durand Building Rm 357
> > Dept of Aeronautics and Astronautics
> > Stanford University
> > Stanford CA 94305
> >
> > Phone: 650-721-2835
> > Web: www.stanford.edu/~dkarthik
> > =======================================
> 
> 
> 
> 
> 
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 
> 
> 
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120501/cd3b38d0/attachment.htm>

From dkarthik at stanford.edu  Tue May  1 18:00:14 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Tue, 1 May 2012 16:00:14 -0700 (PDT)
Subject: [petsc-users] Multigrid
In-Reply-To: <C088813F-92A4-4523-A9EB-565659AA9203@columbia.edu>
Message-ID: <638595201.100045964.1335913214427.JavaMail.root@zm09.stanford.edu>

So as I understand it, GMRES is used as a preconditioner and as a solver when I use PCMG with defaults. If this is the case, I should be able to recreate this set up without the PCMG. Any pointers as to how this can be done?

Also, yes indeed, my mesh is completely unstructured, so I will have to use ml or boomeramg. 

The problems that I am attempting involve RANS of compressible turbulent combustion (finite volume, steady). The high condition numbers are because of the extreme grid stretching and stiff source terms (in the turbulence and combustion model). I have been trying a reduced problem in these 2D test cases, in which the condition number is only 1e7.

Thanks,
Karthik.


----- Original Message -----
From: "Mark F. Adams" <mark.adams at columbia.edu>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Tuesday, May 1, 2012 3:50:31 PM
Subject: Re: [petsc-users] Multigrid


Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ matrix. If you use regular grids (DA?) then PETSc can construct geometric multigrid coarse grid spaces, although I don't know if PCMG will construct these for you (I don't think it will and I can see from your output that PCMG just used one grid). 'ml', hypre' and 'gamg' (a native AMG solver) will do real AMG solvers for you. All three can work on a similar class of problems. 


Also, you mention that you have a condition number of 1.e20. That is astronomical for such a small problem. How did you compute that number? Do you know where the ill-conditioning comes from? Is this an elliptic operator? 


Mark 




On May 1, 2012, at 6:31 PM, Matthew Knepley wrote: 



On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 



The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow) 



Because the smoother IS the solver you were using before. Just like I said last time, what you are doing is 
wrapping up the same solver you used before, sticking it in another GMRES loop, and only looking at the 
outer loop. This has nothing to do with MG. 


Matt 


KSP Object: 8 MPI processes 
type: gmres 
GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
GMRES: happy breakdown tolerance 1e-30 
maximum iterations=1, initial guess is zero 
using preconditioner applied to right hand side for initial guess 
tolerances: relative=0.01, absolute=1e-08, divergence=1e+10 
left preconditioning 
using DEFAULT norm type for convergence test 
PC Object: 8 MPI processes 
type: mg 
MG: type is MULTIPLICATIVE, levels=1 cycles=v 
Cycles per PCApply=1 
Not using Galerkin computed coarse grid matrices 
Coarse grid solver -- level ------------------------------- 
KSP Object: (mg_levels_0_) 8 MPI processes 
type not yet set 
maximum iterations=1, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using DEFAULT norm type for convergence test 
PC Object: (mg_levels_0_) 8 MPI processes 
type not yet set 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 


----- Original Message ----- 
From: "Matthew Knepley" < knepley at gmail.com > 
To: "PETSc users list" < petsc-users at mcs.anl.gov > 
Sent: Tuesday, May 1, 2012 3:22:56 PM 
Subject: Re: [petsc-users] Multigrid 


On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 



Hello, 

Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same). 



Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be 
an extra GMRES loop compared to the case without MG. 


Matt 


Regards, 
Karthik 

KSP Object: 8 MPI processes 
type: gmres 
GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
GMRES: happy breakdown tolerance 1e-30 
maximum iterations=1 
using preconditioner applied to right hand side for initial guess 
tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 
left preconditioning 
using nonzero initial guess 
using PRECONDITIONED norm type for convergence test 
PC Object: 8 MPI processes 
type: bjacobi 
block Jacobi: number of blocks = 8 
Local solve is same for all blocks, in the following KSP and PC objects: 
KSP Object: (sub_) 1 MPI processes 
type: preonly 
maximum iterations=10000, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using NONE norm type for convergence test 
PC Object: (sub_) 1 MPI processes 
type: ilu 
ILU: out-of-place factorization 
0 levels of fill 
tolerance for zero pivot 1e-12 
using diagonal shift to prevent zero pivot 
matrix ordering: natural 
factor fill ratio given 1, needed 1 
Factored matrix follows: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
package used to perform factorization: petsc 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 


----- Original Message ----- 
From: "Matthew Knepley" < knepley at gmail.com > 
To: "PETSc users list" < petsc-users at mcs.anl.gov > 
Sent: Tuesday, May 1, 2012 3:15:14 PM 
Subject: Re: [petsc-users] Multigrid 


On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 



Hello Barry, 

Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 



You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 


Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 



They are different algorithms. Its not possible to say generally that one is better. Try them both. 


Matt 


Best regards, 
Karthik. 

type: mg 
MG: type is MULTIPLICATIVE, levels=1 cycles=v 
Cycles per PCApply=1 
Not using Galerkin computed coarse grid matrices 
Coarse grid solver -- level ------------------------------- 
KSP Object: (mg_levels_0_) 8 MPI processes 
type: gmres 
GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
GMRES: happy breakdown tolerance 1e-30 
maximum iterations=1, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using PRECONDITIONED norm type for convergence test 
PC Object: (mg_levels_0_) 8 MPI processes 
type: bjacobi 
block Jacobi: number of blocks = 8 
Local solve is same for all blocks, in the following KSP and PC objects: 
KSP Object: (mg_levels_0_sub_) 1 MPI processes 
type: preonly 
maximum iterations=10000, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using NONE norm type for convergence test 
PC Object: (mg_levels_0_sub_) 1 MPI processes 
type: ilu 
ILU: out-of-place factorization 
0 levels of fill 
tolerance for zero pivot 1e-12 
using diagonal shift to prevent zero pivot 
matrix ordering: natural 
factor fill ratio given 1, needed 1 
Factored matrix follows: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
package used to perform factorization: petsc 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 



----- Original Message ----- 
From: "Barry Smith" < bsmith at mcs.anl.gov > 
To: "PETSc users list" < petsc-users at mcs.anl.gov > 
Sent: Tuesday, May 1, 2012 1:39:26 PM 
Subject: Re: [petsc-users] Multigrid 


On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 

> Hello, 
> 
> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
> 
> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
> help in this regard will be appreciated. 

First run with -ksp_view to see what solver it is actually using. 

Barry 

> 
> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
> 
> Thanks, 
> Karthik 
> 
> 
> 
> -- 
> 
> ======================================= 
> Karthik Duraisamy 
> Assistant Professor (Consulting) 
> Durand Building Rm 357 
> Dept of Aeronautics and Astronautics 
> Stanford University 
> Stanford CA 94305 
> 
> Phone: 650-721-2835 
> Web: www.stanford.edu/~dkarthik 
> ======================================= 





-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 




-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 




-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 


From knepley at gmail.com  Tue May  1 18:04:36 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 1 May 2012 19:04:36 -0400
Subject: [petsc-users] Multigrid
In-Reply-To: <638595201.100045964.1335913214427.JavaMail.root@zm09.stanford.edu>
References: <C088813F-92A4-4523-A9EB-565659AA9203@columbia.edu>
	<638595201.100045964.1335913214427.JavaMail.root@zm09.stanford.edu>
Message-ID: <CAMYG4GnKHPgRo3rJWvoXR9dfYi2Q=tQ8Dzrj8G5=xcqq_qN4Bg@mail.gmail.com>

On Tue, May 1, 2012 at 7:00 PM, Karthik Duraisamy <dkarthik at stanford.edu>wrote:

> So as I understand it, GMRES is used as a preconditioner and as a solver
> when I use PCMG with defaults. If this is the case, I should be able to
> recreate this set up without the PCMG. Any pointers as to how this can be
> done?
>

You can use
http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCKSP.html to
stick your initial
solver inside another GMRES, however this is unlikely to be faster than
running more iterates.

   Matt


> Also, yes indeed, my mesh is completely unstructured, so I will have to
> use ml or boomeramg.
>
> The problems that I am attempting involve RANS of compressible turbulent
> combustion (finite volume, steady). The high condition numbers are because
> of the extreme grid stretching and stiff source terms (in the turbulence
> and combustion model). I have been trying a reduced problem in these 2D
> test cases, in which the condition number is only 1e7.
>
> Thanks,
> Karthik.
>
>
> ----- Original Message -----
> From: "Mark F. Adams" <mark.adams at columbia.edu>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Tuesday, May 1, 2012 3:50:31 PM
> Subject: Re: [petsc-users] Multigrid
>
>
> Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ
> matrix. If you use regular grids (DA?) then PETSc can construct geometric
> multigrid coarse grid spaces, although I don't know if PCMG will construct
> these for you (I don't think it will and I can see from your output that
> PCMG just used one grid). 'ml', hypre' and 'gamg' (a native AMG solver)
> will do real AMG solvers for you. All three can work on a similar class of
> problems.
>
>
> Also, you mention that you have a condition number of 1.e20. That is
> astronomical for such a small problem. How did you compute that number? Do
> you know where the ill-conditioning comes from? Is this an elliptic
> operator?
>
>
> Mark
>
>
>
>
> On May 1, 2012, at 6:31 PM, Matthew Knepley wrote:
>
>
>
> On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy < dkarthik at stanford.edu> wrote:
>
>
>
> The following was output for the very first iteration whereas what I had
> attached earlier was output every iteration. I am still a bit perplexed
> because PCMG drops the residual like a rock (after the first few iterations
> whereas with no PCMG, it is very slow)
>
>
>
> Because the smoother IS the solver you were using before. Just like I said
> last time, what you are doing is
> wrapping up the same solver you used before, sticking it in another GMRES
> loop, and only looking at the
> outer loop. This has nothing to do with MG.
>
>
> Matt
>
>
> KSP Object: 8 MPI processes
> type: gmres
> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
> maximum iterations=1, initial guess is zero
> using preconditioner applied to right hand side for initial guess
> tolerances: relative=0.01, absolute=1e-08, divergence=1e+10
> left preconditioning
> using DEFAULT norm type for convergence test
> PC Object: 8 MPI processes
> type: mg
> MG: type is MULTIPLICATIVE, levels=1 cycles=v
> Cycles per PCApply=1
> Not using Galerkin computed coarse grid matrices
> Coarse grid solver -- level -------------------------------
> KSP Object: (mg_levels_0_) 8 MPI processes
> type not yet set
> maximum iterations=1, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using DEFAULT norm type for convergence test
> PC Object: (mg_levels_0_) 8 MPI processes
> type not yet set
> linear system matrix = precond matrix:
> Matrix Object: 8 MPI processes
> type: mpiaij
> rows=75000, cols=75000
> total: nonzeros=4427800, allocated nonzeros=4427800
> total number of mallocs used during MatSetValues calls =0
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5
>
>
> ----- Original Message -----
> From: "Matthew Knepley" < knepley at gmail.com >
> To: "PETSc users list" < petsc-users at mcs.anl.gov >
> Sent: Tuesday, May 1, 2012 3:22:56 PM
> Subject: Re: [petsc-users] Multigrid
>
>
> On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu> wrote:
>
>
>
> Hello,
>
> Sorry (and thanks for the reply). I've attached the no multigrid case. I
> didn't include it because (at least to the untrained eye, everything looks
> the same).
>
>
>
> Did you send all the output from the MG case? There must be a PC around
> it. By default its GMRES, so there would be
> an extra GMRES loop compared to the case without MG.
>
>
> Matt
>
>
> Regards,
> Karthik
>
> KSP Object: 8 MPI processes
> type: gmres
> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
> maximum iterations=1
> using preconditioner applied to right hand side for initial guess
> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10
> left preconditioning
> using nonzero initial guess
> using PRECONDITIONED norm type for convergence test
> PC Object: 8 MPI processes
> type: bjacobi
> block Jacobi: number of blocks = 8
> Local solve is same for all blocks, in the following KSP and PC objects:
> KSP Object: (sub_) 1 MPI processes
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using NONE norm type for convergence test
> PC Object: (sub_) 1 MPI processes
> type: ilu
> ILU: out-of-place factorization
> 0 levels of fill
> tolerance for zero pivot 1e-12
> using diagonal shift to prevent zero pivot
> matrix ordering: natural
> factor fill ratio given 1, needed 1
> Factored matrix follows:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> package used to perform factorization: petsc
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 8 MPI processes
> type: mpiaij
> rows=75000, cols=75000
> total: nonzeros=4427800, allocated nonzeros=4427800
> total number of mallocs used during MatSetValues calls =0
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5
>
>
> ----- Original Message -----
> From: "Matthew Knepley" < knepley at gmail.com >
> To: "PETSc users list" < petsc-users at mcs.anl.gov >
> Sent: Tuesday, May 1, 2012 3:15:14 PM
> Subject: Re: [petsc-users] Multigrid
>
>
> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu> wrote:
>
>
>
> Hello Barry,
>
> Thank you for your super quick response. I have attached the output of
> ksp_view and it is practically the same as that when I don't use PCMG. The
> part I don't understand is how PCMG able to function at the zero grid level
> and still produce a much better convergence than when using the default PC.
> Is there any additional smoothing or interpolation going on?
>
>
>
> You only included one output, so I have no way of knowing what you used
> before. However, this is running GMRES/ILU.
>
>
> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ?
>
>
>
> They are different algorithms. Its not possible to say generally that one
> is better. Try them both.
>
>
> Matt
>
>
> Best regards,
> Karthik.
>
> type: mg
> MG: type is MULTIPLICATIVE, levels=1 cycles=v
> Cycles per PCApply=1
> Not using Galerkin computed coarse grid matrices
> Coarse grid solver -- level -------------------------------
> KSP Object: (mg_levels_0_) 8 MPI processes
> type: gmres
> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> GMRES: happy breakdown tolerance 1e-30
> maximum iterations=1, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using PRECONDITIONED norm type for convergence test
> PC Object: (mg_levels_0_) 8 MPI processes
> type: bjacobi
> block Jacobi: number of blocks = 8
> Local solve is same for all blocks, in the following KSP and PC objects:
> KSP Object: (mg_levels_0_sub_) 1 MPI processes
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
> left preconditioning
> using NONE norm type for convergence test
> PC Object: (mg_levels_0_sub_) 1 MPI processes
> type: ilu
> ILU: out-of-place factorization
> 0 levels of fill
> tolerance for zero pivot 1e-12
> using diagonal shift to prevent zero pivot
> matrix ordering: natural
> factor fill ratio given 1, needed 1
> Factored matrix follows:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> package used to perform factorization: petsc
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 1 MPI processes
> type: seqaij
> rows=9015, cols=9015
> total: nonzeros=517777, allocated nonzeros=517777
> total number of mallocs used during MatSetValues calls =0
> using I-node routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 8 MPI processes
> type: mpiaij
> rows=75000, cols=75000
> total: nonzeros=4427800, allocated nonzeros=4427800
> total number of mallocs used during MatSetValues calls =0
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5
> linear system matrix = precond matrix:
> Matrix Object: 8 MPI processes
> type: mpiaij
> rows=75000, cols=75000
> total: nonzeros=4427800, allocated nonzeros=4427800
> total number of mallocs used during MatSetValues calls =0
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5
>
>
>
> ----- Original Message -----
> From: "Barry Smith" < bsmith at mcs.anl.gov >
> To: "PETSc users list" < petsc-users at mcs.anl.gov >
> Sent: Tuesday, May 1, 2012 1:39:26 PM
> Subject: Re: [petsc-users] Multigrid
>
>
> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote:
>
> > Hello,
> >
> > I have been using PETSc for a couple of years with good success, but
> lately as my linear problems have become stiffer (condition numbers of the
> order of 1.e20), I am looking to use better preconditioners. I tried using
> PCMG with all the default options (i.e., I just specified my preconditioner
> as PCMG and did not add any options to it) and I am immediately seeing
> better convergence.
> >
> > What I am not sure of is why? I would like to know more about the
> default parameters (the manual is not very explicit) and more importantly,
> want to know why it is working even when I haven't specified any grid
> levels and coarse grid operators. Any
> > help in this regard will be appreciated.
>
> First run with -ksp_view to see what solver it is actually using.
>
> Barry
>
> >
> > Also, ultimately I want to use algebraic multigrid so is PCML a better
> option than BoomerAMG? I tried BoomerAMG with mixed results.
> >
> > Thanks,
> > Karthik
> >
> >
> >
> > --
> >
> > =======================================
> > Karthik Duraisamy
> > Assistant Professor (Consulting)
> > Durand Building Rm 357
> > Dept of Aeronautics and Astronautics
> > Stanford University
> > Stanford CA 94305
> >
> > Phone: 650-721-2835
> > Web: www.stanford.edu/~dkarthik
> > =======================================
>
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120501/9cb8c2b7/attachment.htm>

From dkarthik at stanford.edu  Tue May  1 18:18:02 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Tue, 1 May 2012 16:18:02 -0700 (PDT)
Subject: [petsc-users] Multigrid
In-Reply-To: <CAMYG4GnKHPgRo3rJWvoXR9dfYi2Q=tQ8Dzrj8G5=xcqq_qN4Bg@mail.gmail.com>
Message-ID: <1438204112.100071417.1335914282810.JavaMail.root@zm09.stanford.edu>

Matt, Mark and Barry, Thanks for all your replies. There is still something that I am missing (running more GMRES iterations without the PCMG flag is not as beneficial - that is the first thing I tried after Mathew suggested that I was using GMRES as a smoother). 

Instead of bugging you guys, I will take some time tonight to figure this out and get back to you. I will also try PCGAMG (I wasn't aware of this).

----- Original Message -----
From: "Matthew Knepley" <knepley at gmail.com>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Tuesday, May 1, 2012 4:04:36 PM
Subject: Re: [petsc-users] Multigrid


On Tue, May 1, 2012 at 7:00 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 



So as I understand it, GMRES is used as a preconditioner and as a solver when I use PCMG with defaults. If this is the case, I should be able to recreate this set up without the PCMG. Any pointers as to how this can be done? 



You can use http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCKSP.html to stick your initial 
solver inside another GMRES, however this is unlikely to be faster than running more iterates. 


Matt 


Also, yes indeed, my mesh is completely unstructured, so I will have to use ml or boomeramg. 

The problems that I am attempting involve RANS of compressible turbulent combustion (finite volume, steady). The high condition numbers are because of the extreme grid stretching and stiff source terms (in the turbulence and combustion model). I have been trying a reduced problem in these 2D test cases, in which the condition number is only 1e7. 

Thanks, 
Karthik. 



----- Original Message ----- 
From: "Mark F. Adams" < mark.adams at columbia.edu > 
To: "PETSc users list" < petsc-users at mcs.anl.gov > 


Sent: Tuesday, May 1, 2012 3:50:31 PM 
Subject: Re: [petsc-users] Multigrid 


Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ matrix. If you use regular grids (DA?) then PETSc can construct geometric multigrid coarse grid spaces, although I don't know if PCMG will construct these for you (I don't think it will and I can see from your output that PCMG just used one grid). 'ml', hypre' and 'gamg' (a native AMG solver) will do real AMG solvers for you. All three can work on a similar class of problems. 


Also, you mention that you have a condition number of 1.e20. That is astronomical for such a small problem. How did you compute that number? Do you know where the ill-conditioning comes from? Is this an elliptic operator? 


Mark 




On May 1, 2012, at 6:31 PM, Matthew Knepley wrote: 



On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 



The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow) 



Because the smoother IS the solver you were using before. Just like I said last time, what you are doing is 
wrapping up the same solver you used before, sticking it in another GMRES loop, and only looking at the 
outer loop. This has nothing to do with MG. 


Matt 


KSP Object: 8 MPI processes 
type: gmres 
GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
GMRES: happy breakdown tolerance 1e-30 
maximum iterations=1, initial guess is zero 
using preconditioner applied to right hand side for initial guess 
tolerances: relative=0.01, absolute=1e-08, divergence=1e+10 
left preconditioning 
using DEFAULT norm type for convergence test 
PC Object: 8 MPI processes 
type: mg 
MG: type is MULTIPLICATIVE, levels=1 cycles=v 
Cycles per PCApply=1 
Not using Galerkin computed coarse grid matrices 
Coarse grid solver -- level ------------------------------- 
KSP Object: (mg_levels_0_) 8 MPI processes 
type not yet set 
maximum iterations=1, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using DEFAULT norm type for convergence test 
PC Object: (mg_levels_0_) 8 MPI processes 
type not yet set 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 


----- Original Message ----- 
From: "Matthew Knepley" < knepley at gmail.com > 
To: "PETSc users list" < petsc-users at mcs.anl.gov > 
Sent: Tuesday, May 1, 2012 3:22:56 PM 
Subject: Re: [petsc-users] Multigrid 


On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 



Hello, 

Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same). 



Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be 
an extra GMRES loop compared to the case without MG. 


Matt 


Regards, 
Karthik 

KSP Object: 8 MPI processes 
type: gmres 
GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
GMRES: happy breakdown tolerance 1e-30 
maximum iterations=1 
using preconditioner applied to right hand side for initial guess 
tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 
left preconditioning 
using nonzero initial guess 
using PRECONDITIONED norm type for convergence test 
PC Object: 8 MPI processes 
type: bjacobi 
block Jacobi: number of blocks = 8 
Local solve is same for all blocks, in the following KSP and PC objects: 
KSP Object: (sub_) 1 MPI processes 
type: preonly 
maximum iterations=10000, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using NONE norm type for convergence test 
PC Object: (sub_) 1 MPI processes 
type: ilu 
ILU: out-of-place factorization 
0 levels of fill 
tolerance for zero pivot 1e-12 
using diagonal shift to prevent zero pivot 
matrix ordering: natural 
factor fill ratio given 1, needed 1 
Factored matrix follows: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
package used to perform factorization: petsc 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 


----- Original Message ----- 
From: "Matthew Knepley" < knepley at gmail.com > 
To: "PETSc users list" < petsc-users at mcs.anl.gov > 
Sent: Tuesday, May 1, 2012 3:15:14 PM 
Subject: Re: [petsc-users] Multigrid 


On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 



Hello Barry, 

Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 



You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 


Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 



They are different algorithms. Its not possible to say generally that one is better. Try them both. 


Matt 


Best regards, 
Karthik. 

type: mg 
MG: type is MULTIPLICATIVE, levels=1 cycles=v 
Cycles per PCApply=1 
Not using Galerkin computed coarse grid matrices 
Coarse grid solver -- level ------------------------------- 
KSP Object: (mg_levels_0_) 8 MPI processes 
type: gmres 
GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
GMRES: happy breakdown tolerance 1e-30 
maximum iterations=1, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using PRECONDITIONED norm type for convergence test 
PC Object: (mg_levels_0_) 8 MPI processes 
type: bjacobi 
block Jacobi: number of blocks = 8 
Local solve is same for all blocks, in the following KSP and PC objects: 
KSP Object: (mg_levels_0_sub_) 1 MPI processes 
type: preonly 
maximum iterations=10000, initial guess is zero 
tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
left preconditioning 
using NONE norm type for convergence test 
PC Object: (mg_levels_0_sub_) 1 MPI processes 
type: ilu 
ILU: out-of-place factorization 
0 levels of fill 
tolerance for zero pivot 1e-12 
using diagonal shift to prevent zero pivot 
matrix ordering: natural 
factor fill ratio given 1, needed 1 
Factored matrix follows: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
package used to perform factorization: petsc 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 1 MPI processes 
type: seqaij 
rows=9015, cols=9015 
total: nonzeros=517777, allocated nonzeros=517777 
total number of mallocs used during MatSetValues calls =0 
using I-node routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
linear system matrix = precond matrix: 
Matrix Object: 8 MPI processes 
type: mpiaij 
rows=75000, cols=75000 
total: nonzeros=4427800, allocated nonzeros=4427800 
total number of mallocs used during MatSetValues calls =0 
using I-node (on process 0) routines: found 3476 nodes, limit used is 5 



----- Original Message ----- 
From: "Barry Smith" < bsmith at mcs.anl.gov > 
To: "PETSc users list" < petsc-users at mcs.anl.gov > 
Sent: Tuesday, May 1, 2012 1:39:26 PM 
Subject: Re: [petsc-users] Multigrid 


On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 

> Hello, 
> 
> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
> 
> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
> help in this regard will be appreciated. 

First run with -ksp_view to see what solver it is actually using. 

Barry 

> 
> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
> 
> Thanks, 
> Karthik 
> 
> 
> 
> -- 
> 
> ======================================= 
> Karthik Duraisamy 
> Assistant Professor (Consulting) 
> Durand Building Rm 357 
> Dept of Aeronautics and Astronautics 
> Stanford University 
> Stanford CA 94305 
> 
> Phone: 650-721-2835 
> Web: www.stanford.edu/~dkarthik 
> ======================================= 





-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 




-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 




-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 





-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
-- Norbert Wiener 

From bsmith at mcs.anl.gov  Tue May  1 23:14:05 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 1 May 2012 23:14:05 -0500
Subject: [petsc-users] Multigrid
In-Reply-To: <638595201.100045964.1335913214427.JavaMail.root@zm09.stanford.edu>
References: <638595201.100045964.1335913214427.JavaMail.root@zm09.stanford.edu>
Message-ID: <CAB6D953-49ED-44D9-88BD-9A5C87AD5061@mcs.anl.gov>


On May 1, 2012, at 6:00 PM, Karthik Duraisamy wrote:

> So as I understand it, GMRES is used as a preconditioner and as a solver when I use PCMG with defaults. If this is the case, I should be able to recreate this set up without the PCMG. Any pointers as to how this can be done?
> 
> Also, yes indeed, my mesh is completely unstructured, so I will have to use ml or boomeramg. 
> 
> The problems that I am attempting involve RANS of compressible turbulent combustion (finite volume, steady). The high condition numbers are because of the extreme grid stretching and stiff source terms (in the turbulence and combustion model).

   With a condition number like that I wouldn't even consider using double precision, I think you would just be computing meaningless noise. With petsc-dev you can use quad precision ./configure --with-precision=__float128 using recent GNU compilers
(no we haven't tested the gfortran sides of things). 

   Barry

   
> I have been trying a reduced problem in these 2D test cases, in which the condition number is only 1e7.
> 
> Thanks,
> Karthik.
> 
> 
> ----- Original Message -----
> From: "Mark F. Adams" <mark.adams at columbia.edu>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Tuesday, May 1, 2012 3:50:31 PM
> Subject: Re: [petsc-users] Multigrid
> 
> 
> Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ matrix. If you use regular grids (DA?) then PETSc can construct geometric multigrid coarse grid spaces, although I don't know if PCMG will construct these for you (I don't think it will and I can see from your output that PCMG just used one grid). 'ml', hypre' and 'gamg' (a native AMG solver) will do real AMG solvers for you. All three can work on a similar class of problems. 
> 
> 
> Also, you mention that you have a condition number of 1.e20. That is astronomical for such a small problem. How did you compute that number? Do you know where the ill-conditioning comes from? Is this an elliptic operator? 
> 
> 
> Mark 
> 
> 
> 
> 
> On May 1, 2012, at 6:31 PM, Matthew Knepley wrote: 
> 
> 
> 
> On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
> 
> 
> 
> The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow) 
> 
> 
> 
> Because the smoother IS the solver you were using before. Just like I said last time, what you are doing is 
> wrapping up the same solver you used before, sticking it in another GMRES loop, and only looking at the 
> outer loop. This has nothing to do with MG. 
> 
> 
> Matt 
> 
> 
> KSP Object: 8 MPI processes 
> type: gmres 
> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
> GMRES: happy breakdown tolerance 1e-30 
> maximum iterations=1, initial guess is zero 
> using preconditioner applied to right hand side for initial guess 
> tolerances: relative=0.01, absolute=1e-08, divergence=1e+10 
> left preconditioning 
> using DEFAULT norm type for convergence test 
> PC Object: 8 MPI processes 
> type: mg 
> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
> Cycles per PCApply=1 
> Not using Galerkin computed coarse grid matrices 
> Coarse grid solver -- level ------------------------------- 
> KSP Object: (mg_levels_0_) 8 MPI processes 
> type not yet set 
> maximum iterations=1, initial guess is zero 
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
> left preconditioning 
> using DEFAULT norm type for convergence test 
> PC Object: (mg_levels_0_) 8 MPI processes 
> type not yet set 
> linear system matrix = precond matrix: 
> Matrix Object: 8 MPI processes 
> type: mpiaij 
> rows=75000, cols=75000 
> total: nonzeros=4427800, allocated nonzeros=4427800 
> total number of mallocs used during MatSetValues calls =0 
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
> 
> 
> ----- Original Message ----- 
> From: "Matthew Knepley" < knepley at gmail.com > 
> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
> Sent: Tuesday, May 1, 2012 3:22:56 PM 
> Subject: Re: [petsc-users] Multigrid 
> 
> 
> On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
> 
> 
> 
> Hello, 
> 
> Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same). 
> 
> 
> 
> Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be 
> an extra GMRES loop compared to the case without MG. 
> 
> 
> Matt 
> 
> 
> Regards, 
> Karthik 
> 
> KSP Object: 8 MPI processes 
> type: gmres 
> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
> GMRES: happy breakdown tolerance 1e-30 
> maximum iterations=1 
> using preconditioner applied to right hand side for initial guess 
> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 
> left preconditioning 
> using nonzero initial guess 
> using PRECONDITIONED norm type for convergence test 
> PC Object: 8 MPI processes 
> type: bjacobi 
> block Jacobi: number of blocks = 8 
> Local solve is same for all blocks, in the following KSP and PC objects: 
> KSP Object: (sub_) 1 MPI processes 
> type: preonly 
> maximum iterations=10000, initial guess is zero 
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
> left preconditioning 
> using NONE norm type for convergence test 
> PC Object: (sub_) 1 MPI processes 
> type: ilu 
> ILU: out-of-place factorization 
> 0 levels of fill 
> tolerance for zero pivot 1e-12 
> using diagonal shift to prevent zero pivot 
> matrix ordering: natural 
> factor fill ratio given 1, needed 1 
> Factored matrix follows: 
> Matrix Object: 1 MPI processes 
> type: seqaij 
> rows=9015, cols=9015 
> package used to perform factorization: petsc 
> total: nonzeros=517777, allocated nonzeros=517777 
> total number of mallocs used during MatSetValues calls =0 
> using I-node routines: found 3476 nodes, limit used is 5 
> linear system matrix = precond matrix: 
> Matrix Object: 1 MPI processes 
> type: seqaij 
> rows=9015, cols=9015 
> total: nonzeros=517777, allocated nonzeros=517777 
> total number of mallocs used during MatSetValues calls =0 
> using I-node routines: found 3476 nodes, limit used is 5 
> linear system matrix = precond matrix: 
> Matrix Object: 8 MPI processes 
> type: mpiaij 
> rows=75000, cols=75000 
> total: nonzeros=4427800, allocated nonzeros=4427800 
> total number of mallocs used during MatSetValues calls =0 
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
> 
> 
> ----- Original Message ----- 
> From: "Matthew Knepley" < knepley at gmail.com > 
> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
> Sent: Tuesday, May 1, 2012 3:15:14 PM 
> Subject: Re: [petsc-users] Multigrid 
> 
> 
> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
> 
> 
> 
> Hello Barry, 
> 
> Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 
> 
> 
> 
> You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 
> 
> 
> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 
> 
> 
> 
> They are different algorithms. Its not possible to say generally that one is better. Try them both. 
> 
> 
> Matt 
> 
> 
> Best regards, 
> Karthik. 
> 
> type: mg 
> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
> Cycles per PCApply=1 
> Not using Galerkin computed coarse grid matrices 
> Coarse grid solver -- level ------------------------------- 
> KSP Object: (mg_levels_0_) 8 MPI processes 
> type: gmres 
> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
> GMRES: happy breakdown tolerance 1e-30 
> maximum iterations=1, initial guess is zero 
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
> left preconditioning 
> using PRECONDITIONED norm type for convergence test 
> PC Object: (mg_levels_0_) 8 MPI processes 
> type: bjacobi 
> block Jacobi: number of blocks = 8 
> Local solve is same for all blocks, in the following KSP and PC objects: 
> KSP Object: (mg_levels_0_sub_) 1 MPI processes 
> type: preonly 
> maximum iterations=10000, initial guess is zero 
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
> left preconditioning 
> using NONE norm type for convergence test 
> PC Object: (mg_levels_0_sub_) 1 MPI processes 
> type: ilu 
> ILU: out-of-place factorization 
> 0 levels of fill 
> tolerance for zero pivot 1e-12 
> using diagonal shift to prevent zero pivot 
> matrix ordering: natural 
> factor fill ratio given 1, needed 1 
> Factored matrix follows: 
> Matrix Object: 1 MPI processes 
> type: seqaij 
> rows=9015, cols=9015 
> package used to perform factorization: petsc 
> total: nonzeros=517777, allocated nonzeros=517777 
> total number of mallocs used during MatSetValues calls =0 
> using I-node routines: found 3476 nodes, limit used is 5 
> linear system matrix = precond matrix: 
> Matrix Object: 1 MPI processes 
> type: seqaij 
> rows=9015, cols=9015 
> total: nonzeros=517777, allocated nonzeros=517777 
> total number of mallocs used during MatSetValues calls =0 
> using I-node routines: found 3476 nodes, limit used is 5 
> linear system matrix = precond matrix: 
> Matrix Object: 8 MPI processes 
> type: mpiaij 
> rows=75000, cols=75000 
> total: nonzeros=4427800, allocated nonzeros=4427800 
> total number of mallocs used during MatSetValues calls =0 
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
> linear system matrix = precond matrix: 
> Matrix Object: 8 MPI processes 
> type: mpiaij 
> rows=75000, cols=75000 
> total: nonzeros=4427800, allocated nonzeros=4427800 
> total number of mallocs used during MatSetValues calls =0 
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
> 
> 
> 
> ----- Original Message ----- 
> From: "Barry Smith" < bsmith at mcs.anl.gov > 
> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
> Sent: Tuesday, May 1, 2012 1:39:26 PM 
> Subject: Re: [petsc-users] Multigrid 
> 
> 
> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 
> 
>> Hello, 
>> 
>> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
>> 
>> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
>> help in this regard will be appreciated. 
> 
> First run with -ksp_view to see what solver it is actually using. 
> 
> Barry 
> 
>> 
>> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
>> 
>> Thanks, 
>> Karthik 
>> 
>> 
>> 
>> -- 
>> 
>> ======================================= 
>> Karthik Duraisamy 
>> Assistant Professor (Consulting) 
>> Durand Building Rm 357 
>> Dept of Aeronautics and Astronautics 
>> Stanford University 
>> Stanford CA 94305 
>> 
>> Phone: 650-721-2835 
>> Web: www.stanford.edu/~dkarthik 
>> ======================================= 
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
> -- Norbert Wiener 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
> -- Norbert Wiener 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
> -- Norbert Wiener 
> 


From dkarthik at stanford.edu  Wed May  2 01:33:35 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Tue, 1 May 2012 23:33:35 -0700 (PDT)
Subject: [petsc-users] Multigrid
In-Reply-To: <CAB6D953-49ED-44D9-88BD-9A5C87AD5061@mcs.anl.gov>
Message-ID: <629146396.100456667.1335940415297.JavaMail.root@zm09.stanford.edu>

Hello again,

      I played with BoomerAMG and ML (for the 2D problem with a reasonable condition number), and so far, they haven't worked (the residual eventually diverges). The options that I am using are (for instance)

    PC pc; 
    KSPGetPC(ksp,&pc);
    PetscOptionsSetValue("-ksp_type", "gmres");
    KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
    PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
    PCSetType(pc, PCML);
    PetscOptionsSetValue("-pc_ml_maxNlevels", "3");
    PetscOptionsSetValue("-mg_coarse_ksp_type","richardson");
    PetscOptionsSetValue("-mg_coarse_pc_type","sor");
    PetscOptionsSetValue("-mg_coarse_pc_sor_its","8");
    PetscOptionsSetValue("-ksp_monitor","1");
    KSPSetFromOptions(ksp);

I have replaced KSP with bcgsl, fgmres, etc.

As I mentioned earlier, the following works like a charm:

    PC pc; 
    KSPGetPC(ksp,&pc);
    PCSetType(pc, PCMG);
    KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
    PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
    KSPSetType(ksp, KSPGMRES);
    KSPGMRESSetRestart(ksp, 100);
    KSPSetFromOptions(ksp);


Regards,
Karthik.

 

----- Original Message -----
From: "Barry Smith" <bsmith at mcs.anl.gov>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Tuesday, May 1, 2012 9:14:05 PM
Subject: Re: [petsc-users] Multigrid


On May 1, 2012, at 6:00 PM, Karthik Duraisamy wrote:

> So as I understand it, GMRES is used as a preconditioner and as a solver when I use PCMG with defaults. If this is the case, I should be able to recreate this set up without the PCMG. Any pointers as to how this can be done?
> 
> Also, yes indeed, my mesh is completely unstructured, so I will have to use ml or boomeramg. 
> 
> The problems that I am attempting involve RANS of compressible turbulent combustion (finite volume, steady). The high condition numbers are because of the extreme grid stretching and stiff source terms (in the turbulence and combustion model).

   With a condition number like that I wouldn't even consider using double precision, I think you would just be computing meaningless noise. With petsc-dev you can use quad precision ./configure --with-precision=__float128 using recent GNU compilers
(no we haven't tested the gfortran sides of things). 

   Barry

   
> I have been trying a reduced problem in these 2D test cases, in which the condition number is only 1e7.
> 
> Thanks,
> Karthik.
> 
> 
> ----- Original Message -----
> From: "Mark F. Adams" <mark.adams at columbia.edu>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Tuesday, May 1, 2012 3:50:31 PM
> Subject: Re: [petsc-users] Multigrid
> 
> 
> Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ matrix. If you use regular grids (DA?) then PETSc can construct geometric multigrid coarse grid spaces, although I don't know if PCMG will construct these for you (I don't think it will and I can see from your output that PCMG just used one grid). 'ml', hypre' and 'gamg' (a native AMG solver) will do real AMG solvers for you. All three can work on a similar class of problems. 
> 
> 
> Also, you mention that you have a condition number of 1.e20. That is astronomical for such a small problem. How did you compute that number? Do you know where the ill-conditioning comes from? Is this an elliptic operator? 
> 
> 
> Mark 
> 
> 
> 
> 
> On May 1, 2012, at 6:31 PM, Matthew Knepley wrote: 
> 
> 
> 
> On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
> 
> 
> 
> The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow) 
> 
> 
> 
> Because the smoother IS the solver you were using before. Just like I said last time, what you are doing is 
> wrapping up the same solver you used before, sticking it in another GMRES loop, and only looking at the 
> outer loop. This has nothing to do with MG. 
> 
> 
> Matt 
> 
> 
> KSP Object: 8 MPI processes 
> type: gmres 
> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
> GMRES: happy breakdown tolerance 1e-30 
> maximum iterations=1, initial guess is zero 
> using preconditioner applied to right hand side for initial guess 
> tolerances: relative=0.01, absolute=1e-08, divergence=1e+10 
> left preconditioning 
> using DEFAULT norm type for convergence test 
> PC Object: 8 MPI processes 
> type: mg 
> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
> Cycles per PCApply=1 
> Not using Galerkin computed coarse grid matrices 
> Coarse grid solver -- level ------------------------------- 
> KSP Object: (mg_levels_0_) 8 MPI processes 
> type not yet set 
> maximum iterations=1, initial guess is zero 
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
> left preconditioning 
> using DEFAULT norm type for convergence test 
> PC Object: (mg_levels_0_) 8 MPI processes 
> type not yet set 
> linear system matrix = precond matrix: 
> Matrix Object: 8 MPI processes 
> type: mpiaij 
> rows=75000, cols=75000 
> total: nonzeros=4427800, allocated nonzeros=4427800 
> total number of mallocs used during MatSetValues calls =0 
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
> 
> 
> ----- Original Message ----- 
> From: "Matthew Knepley" < knepley at gmail.com > 
> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
> Sent: Tuesday, May 1, 2012 3:22:56 PM 
> Subject: Re: [petsc-users] Multigrid 
> 
> 
> On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
> 
> 
> 
> Hello, 
> 
> Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same). 
> 
> 
> 
> Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be 
> an extra GMRES loop compared to the case without MG. 
> 
> 
> Matt 
> 
> 
> Regards, 
> Karthik 
> 
> KSP Object: 8 MPI processes 
> type: gmres 
> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
> GMRES: happy breakdown tolerance 1e-30 
> maximum iterations=1 
> using preconditioner applied to right hand side for initial guess 
> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 
> left preconditioning 
> using nonzero initial guess 
> using PRECONDITIONED norm type for convergence test 
> PC Object: 8 MPI processes 
> type: bjacobi 
> block Jacobi: number of blocks = 8 
> Local solve is same for all blocks, in the following KSP and PC objects: 
> KSP Object: (sub_) 1 MPI processes 
> type: preonly 
> maximum iterations=10000, initial guess is zero 
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
> left preconditioning 
> using NONE norm type for convergence test 
> PC Object: (sub_) 1 MPI processes 
> type: ilu 
> ILU: out-of-place factorization 
> 0 levels of fill 
> tolerance for zero pivot 1e-12 
> using diagonal shift to prevent zero pivot 
> matrix ordering: natural 
> factor fill ratio given 1, needed 1 
> Factored matrix follows: 
> Matrix Object: 1 MPI processes 
> type: seqaij 
> rows=9015, cols=9015 
> package used to perform factorization: petsc 
> total: nonzeros=517777, allocated nonzeros=517777 
> total number of mallocs used during MatSetValues calls =0 
> using I-node routines: found 3476 nodes, limit used is 5 
> linear system matrix = precond matrix: 
> Matrix Object: 1 MPI processes 
> type: seqaij 
> rows=9015, cols=9015 
> total: nonzeros=517777, allocated nonzeros=517777 
> total number of mallocs used during MatSetValues calls =0 
> using I-node routines: found 3476 nodes, limit used is 5 
> linear system matrix = precond matrix: 
> Matrix Object: 8 MPI processes 
> type: mpiaij 
> rows=75000, cols=75000 
> total: nonzeros=4427800, allocated nonzeros=4427800 
> total number of mallocs used during MatSetValues calls =0 
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
> 
> 
> ----- Original Message ----- 
> From: "Matthew Knepley" < knepley at gmail.com > 
> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
> Sent: Tuesday, May 1, 2012 3:15:14 PM 
> Subject: Re: [petsc-users] Multigrid 
> 
> 
> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
> 
> 
> 
> Hello Barry, 
> 
> Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 
> 
> 
> 
> You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 
> 
> 
> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 
> 
> 
> 
> They are different algorithms. Its not possible to say generally that one is better. Try them both. 
> 
> 
> Matt 
> 
> 
> Best regards, 
> Karthik. 
> 
> type: mg 
> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
> Cycles per PCApply=1 
> Not using Galerkin computed coarse grid matrices 
> Coarse grid solver -- level ------------------------------- 
> KSP Object: (mg_levels_0_) 8 MPI processes 
> type: gmres 
> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
> GMRES: happy breakdown tolerance 1e-30 
> maximum iterations=1, initial guess is zero 
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
> left preconditioning 
> using PRECONDITIONED norm type for convergence test 
> PC Object: (mg_levels_0_) 8 MPI processes 
> type: bjacobi 
> block Jacobi: number of blocks = 8 
> Local solve is same for all blocks, in the following KSP and PC objects: 
> KSP Object: (mg_levels_0_sub_) 1 MPI processes 
> type: preonly 
> maximum iterations=10000, initial guess is zero 
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
> left preconditioning 
> using NONE norm type for convergence test 
> PC Object: (mg_levels_0_sub_) 1 MPI processes 
> type: ilu 
> ILU: out-of-place factorization 
> 0 levels of fill 
> tolerance for zero pivot 1e-12 
> using diagonal shift to prevent zero pivot 
> matrix ordering: natural 
> factor fill ratio given 1, needed 1 
> Factored matrix follows: 
> Matrix Object: 1 MPI processes 
> type: seqaij 
> rows=9015, cols=9015 
> package used to perform factorization: petsc 
> total: nonzeros=517777, allocated nonzeros=517777 
> total number of mallocs used during MatSetValues calls =0 
> using I-node routines: found 3476 nodes, limit used is 5 
> linear system matrix = precond matrix: 
> Matrix Object: 1 MPI processes 
> type: seqaij 
> rows=9015, cols=9015 
> total: nonzeros=517777, allocated nonzeros=517777 
> total number of mallocs used during MatSetValues calls =0 
> using I-node routines: found 3476 nodes, limit used is 5 
> linear system matrix = precond matrix: 
> Matrix Object: 8 MPI processes 
> type: mpiaij 
> rows=75000, cols=75000 
> total: nonzeros=4427800, allocated nonzeros=4427800 
> total number of mallocs used during MatSetValues calls =0 
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
> linear system matrix = precond matrix: 
> Matrix Object: 8 MPI processes 
> type: mpiaij 
> rows=75000, cols=75000 
> total: nonzeros=4427800, allocated nonzeros=4427800 
> total number of mallocs used during MatSetValues calls =0 
> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
> 
> 
> 
> ----- Original Message ----- 
> From: "Barry Smith" < bsmith at mcs.anl.gov > 
> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
> Sent: Tuesday, May 1, 2012 1:39:26 PM 
> Subject: Re: [petsc-users] Multigrid 
> 
> 
> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 
> 
>> Hello, 
>> 
>> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
>> 
>> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
>> help in this regard will be appreciated. 
> 
> First run with -ksp_view to see what solver it is actually using. 
> 
> Barry 
> 
>> 
>> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
>> 
>> Thanks, 
>> Karthik 
>> 
>> 
>> 
>> -- 
>> 
>> ======================================= 
>> Karthik Duraisamy 
>> Assistant Professor (Consulting) 
>> Durand Building Rm 357 
>> Dept of Aeronautics and Astronautics 
>> Stanford University 
>> Stanford CA 94305 
>> 
>> Phone: 650-721-2835 
>> Web: www.stanford.edu/~dkarthik 
>> ======================================= 
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
> -- Norbert Wiener 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
> -- Norbert Wiener 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
> -- Norbert Wiener 
> 


From behzad.baghapour at gmail.com  Wed May  2 03:58:04 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Wed, 2 May 2012 13:28:04 +0430
Subject: [petsc-users] Some Questions about Inexact Newton Solver
Message-ID: <CA+jOtDJWG+06-w5NpYQ280WsPf1KCSmeL6NGnB0vd1yua4s3tA@mail.gmail.com>

Dear developers,

I have some questions to set up the inexact Newton solver in SNES:

1- Is there any other method other than EW?

2- What are the default values for EW when using -snes_ksp_ew?

3- Is the following procedure enough to implement user-defined inexact
method?

    "reset the linear tolerance (rtol) in a monitor routine defined for
SNES"

Thanks a lot,
BehZad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120502/c8490878/attachment.htm>

From bsmith at mcs.anl.gov  Wed May  2 08:02:06 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 2 May 2012 08:02:06 -0500
Subject: [petsc-users] Multigrid
In-Reply-To: <629146396.100456667.1335940415297.JavaMail.root@zm09.stanford.edu>
References: <629146396.100456667.1335940415297.JavaMail.root@zm09.stanford.edu>
Message-ID: <2BAD4E6D-E17D-42C2-9501-664DCAAEA90A@mcs.anl.gov>


   This is likely helping a great deal:   

  KSPGMRESSetRestart(ksp, 100);

   You can try this with ml also

On May 2, 2012, at 1:33 AM, Karthik Duraisamy wrote:

> Hello again,
> 
>      I played with BoomerAMG and ML (for the 2D problem with a reasonable condition number), and so far, they haven't worked (the residual eventually diverges). The options that I am using are (for instance)
> 
>    PC pc; 
>    KSPGetPC(ksp,&pc);
>    PetscOptionsSetValue("-ksp_type", "gmres");

   Here you NEED  fgmres since the smoother is gmres by default. You can see this with -ksp_view

>    KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>    PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>    PCSetType(pc, PCML);
>    PetscOptionsSetValue("-pc_ml_maxNlevels", "3");
>    PetscOptionsSetValue("-mg_coarse_ksp_type","richardson");
>    PetscOptionsSetValue("-mg_coarse_pc_type","sor");
>    PetscOptionsSetValue("-mg_coarse_pc_sor_its","8");

Just use -mg_coarse_pc_type  redundant -mg_coarse_ksp_type preonly instead of the above.


   Barry

>    PetscOptionsSetValue("-ksp_monitor","1");
>    KSPSetFromOptions(ksp);


> 
> I have replaced KSP with bcgsl, fgmres, etc.
> 
> As I mentioned earlier, the following works like a charm:
> 
>    PC pc; 
>    KSPGetPC(ksp,&pc);
>    PCSetType(pc, PCMG);
>    KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>    PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>    KSPSetType(ksp, KSPGMRES);
>    KSPGMRESSetRestart(ksp, 100);
>    KSPSetFromOptions(ksp);
> 
> 
> Regards,
> Karthik.
> 
> 
> 
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Tuesday, May 1, 2012 9:14:05 PM
> Subject: Re: [petsc-users] Multigrid
> 
> 
> On May 1, 2012, at 6:00 PM, Karthik Duraisamy wrote:
> 
>> So as I understand it, GMRES is used as a preconditioner and as a solver when I use PCMG with defaults. If this is the case, I should be able to recreate this set up without the PCMG. Any pointers as to how this can be done?
>> 
>> Also, yes indeed, my mesh is completely unstructured, so I will have to use ml or boomeramg. 
>> 
>> The problems that I am attempting involve RANS of compressible turbulent combustion (finite volume, steady). The high condition numbers are because of the extreme grid stretching and stiff source terms (in the turbulence and combustion model).
> 
>   With a condition number like that I wouldn't even consider using double precision, I think you would just be computing meaningless noise. With petsc-dev you can use quad precision ./configure --with-precision=__float128 using recent GNU compilers
> (no we haven't tested the gfortran sides of things). 
> 
>   Barry
> 
> 
>> I have been trying a reduced problem in these 2D test cases, in which the condition number is only 1e7.
>> 
>> Thanks,
>> Karthik.
>> 
>> 
>> ----- Original Message -----
>> From: "Mark F. Adams" <mark.adams at columbia.edu>
>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>> Sent: Tuesday, May 1, 2012 3:50:31 PM
>> Subject: Re: [petsc-users] Multigrid
>> 
>> 
>> Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ matrix. If you use regular grids (DA?) then PETSc can construct geometric multigrid coarse grid spaces, although I don't know if PCMG will construct these for you (I don't think it will and I can see from your output that PCMG just used one grid). 'ml', hypre' and 'gamg' (a native AMG solver) will do real AMG solvers for you. All three can work on a similar class of problems. 
>> 
>> 
>> Also, you mention that you have a condition number of 1.e20. That is astronomical for such a small problem. How did you compute that number? Do you know where the ill-conditioning comes from? Is this an elliptic operator? 
>> 
>> 
>> Mark 
>> 
>> 
>> 
>> 
>> On May 1, 2012, at 6:31 PM, Matthew Knepley wrote: 
>> 
>> 
>> 
>> On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>> 
>> 
>> 
>> The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow) 
>> 
>> 
>> 
>> Because the smoother IS the solver you were using before. Just like I said last time, what you are doing is 
>> wrapping up the same solver you used before, sticking it in another GMRES loop, and only looking at the 
>> outer loop. This has nothing to do with MG. 
>> 
>> 
>> Matt 
>> 
>> 
>> KSP Object: 8 MPI processes 
>> type: gmres 
>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>> GMRES: happy breakdown tolerance 1e-30 
>> maximum iterations=1, initial guess is zero 
>> using preconditioner applied to right hand side for initial guess 
>> tolerances: relative=0.01, absolute=1e-08, divergence=1e+10 
>> left preconditioning 
>> using DEFAULT norm type for convergence test 
>> PC Object: 8 MPI processes 
>> type: mg 
>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>> Cycles per PCApply=1 
>> Not using Galerkin computed coarse grid matrices 
>> Coarse grid solver -- level ------------------------------- 
>> KSP Object: (mg_levels_0_) 8 MPI processes 
>> type not yet set 
>> maximum iterations=1, initial guess is zero 
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>> left preconditioning 
>> using DEFAULT norm type for convergence test 
>> PC Object: (mg_levels_0_) 8 MPI processes 
>> type not yet set 
>> linear system matrix = precond matrix: 
>> Matrix Object: 8 MPI processes 
>> type: mpiaij 
>> rows=75000, cols=75000 
>> total: nonzeros=4427800, allocated nonzeros=4427800 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>> 
>> 
>> ----- Original Message ----- 
>> From: "Matthew Knepley" < knepley at gmail.com > 
>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>> Sent: Tuesday, May 1, 2012 3:22:56 PM 
>> Subject: Re: [petsc-users] Multigrid 
>> 
>> 
>> On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>> 
>> 
>> 
>> Hello, 
>> 
>> Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same). 
>> 
>> 
>> 
>> Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be 
>> an extra GMRES loop compared to the case without MG. 
>> 
>> 
>> Matt 
>> 
>> 
>> Regards, 
>> Karthik 
>> 
>> KSP Object: 8 MPI processes 
>> type: gmres 
>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>> GMRES: happy breakdown tolerance 1e-30 
>> maximum iterations=1 
>> using preconditioner applied to right hand side for initial guess 
>> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 
>> left preconditioning 
>> using nonzero initial guess 
>> using PRECONDITIONED norm type for convergence test 
>> PC Object: 8 MPI processes 
>> type: bjacobi 
>> block Jacobi: number of blocks = 8 
>> Local solve is same for all blocks, in the following KSP and PC objects: 
>> KSP Object: (sub_) 1 MPI processes 
>> type: preonly 
>> maximum iterations=10000, initial guess is zero 
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>> left preconditioning 
>> using NONE norm type for convergence test 
>> PC Object: (sub_) 1 MPI processes 
>> type: ilu 
>> ILU: out-of-place factorization 
>> 0 levels of fill 
>> tolerance for zero pivot 1e-12 
>> using diagonal shift to prevent zero pivot 
>> matrix ordering: natural 
>> factor fill ratio given 1, needed 1 
>> Factored matrix follows: 
>> Matrix Object: 1 MPI processes 
>> type: seqaij 
>> rows=9015, cols=9015 
>> package used to perform factorization: petsc 
>> total: nonzeros=517777, allocated nonzeros=517777 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node routines: found 3476 nodes, limit used is 5 
>> linear system matrix = precond matrix: 
>> Matrix Object: 1 MPI processes 
>> type: seqaij 
>> rows=9015, cols=9015 
>> total: nonzeros=517777, allocated nonzeros=517777 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node routines: found 3476 nodes, limit used is 5 
>> linear system matrix = precond matrix: 
>> Matrix Object: 8 MPI processes 
>> type: mpiaij 
>> rows=75000, cols=75000 
>> total: nonzeros=4427800, allocated nonzeros=4427800 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>> 
>> 
>> ----- Original Message ----- 
>> From: "Matthew Knepley" < knepley at gmail.com > 
>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>> Sent: Tuesday, May 1, 2012 3:15:14 PM 
>> Subject: Re: [petsc-users] Multigrid 
>> 
>> 
>> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>> 
>> 
>> 
>> Hello Barry, 
>> 
>> Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 
>> 
>> 
>> 
>> You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 
>> 
>> 
>> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 
>> 
>> 
>> 
>> They are different algorithms. Its not possible to say generally that one is better. Try them both. 
>> 
>> 
>> Matt 
>> 
>> 
>> Best regards, 
>> Karthik. 
>> 
>> type: mg 
>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>> Cycles per PCApply=1 
>> Not using Galerkin computed coarse grid matrices 
>> Coarse grid solver -- level ------------------------------- 
>> KSP Object: (mg_levels_0_) 8 MPI processes 
>> type: gmres 
>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>> GMRES: happy breakdown tolerance 1e-30 
>> maximum iterations=1, initial guess is zero 
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>> left preconditioning 
>> using PRECONDITIONED norm type for convergence test 
>> PC Object: (mg_levels_0_) 8 MPI processes 
>> type: bjacobi 
>> block Jacobi: number of blocks = 8 
>> Local solve is same for all blocks, in the following KSP and PC objects: 
>> KSP Object: (mg_levels_0_sub_) 1 MPI processes 
>> type: preonly 
>> maximum iterations=10000, initial guess is zero 
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>> left preconditioning 
>> using NONE norm type for convergence test 
>> PC Object: (mg_levels_0_sub_) 1 MPI processes 
>> type: ilu 
>> ILU: out-of-place factorization 
>> 0 levels of fill 
>> tolerance for zero pivot 1e-12 
>> using diagonal shift to prevent zero pivot 
>> matrix ordering: natural 
>> factor fill ratio given 1, needed 1 
>> Factored matrix follows: 
>> Matrix Object: 1 MPI processes 
>> type: seqaij 
>> rows=9015, cols=9015 
>> package used to perform factorization: petsc 
>> total: nonzeros=517777, allocated nonzeros=517777 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node routines: found 3476 nodes, limit used is 5 
>> linear system matrix = precond matrix: 
>> Matrix Object: 1 MPI processes 
>> type: seqaij 
>> rows=9015, cols=9015 
>> total: nonzeros=517777, allocated nonzeros=517777 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node routines: found 3476 nodes, limit used is 5 
>> linear system matrix = precond matrix: 
>> Matrix Object: 8 MPI processes 
>> type: mpiaij 
>> rows=75000, cols=75000 
>> total: nonzeros=4427800, allocated nonzeros=4427800 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>> linear system matrix = precond matrix: 
>> Matrix Object: 8 MPI processes 
>> type: mpiaij 
>> rows=75000, cols=75000 
>> total: nonzeros=4427800, allocated nonzeros=4427800 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>> 
>> 
>> 
>> ----- Original Message ----- 
>> From: "Barry Smith" < bsmith at mcs.anl.gov > 
>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>> Sent: Tuesday, May 1, 2012 1:39:26 PM 
>> Subject: Re: [petsc-users] Multigrid 
>> 
>> 
>> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 
>> 
>>> Hello, 
>>> 
>>> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
>>> 
>>> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
>>> help in this regard will be appreciated. 
>> 
>> First run with -ksp_view to see what solver it is actually using. 
>> 
>> Barry 
>> 
>>> 
>>> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
>>> 
>>> Thanks, 
>>> Karthik 
>>> 
>>> 
>>> 
>>> -- 
>>> 
>>> ======================================= 
>>> Karthik Duraisamy 
>>> Assistant Professor (Consulting) 
>>> Durand Building Rm 357 
>>> Dept of Aeronautics and Astronautics 
>>> Stanford University 
>>> Stanford CA 94305 
>>> 
>>> Phone: 650-721-2835 
>>> Web: www.stanford.edu/~dkarthik 
>>> ======================================= 
>> 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>> -- Norbert Wiener 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>> -- Norbert Wiener 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>> -- Norbert Wiener 
>> 
> 


From jfresno at infor.uva.es  Wed May  2 11:01:13 2012
From: jfresno at infor.uva.es (Javier Fresno)
Date: Wed, 02 May 2012 18:01:13 +0200
Subject: [petsc-users] MatMult scalability problem
Message-ID: <4FA15A49.8040101@infor.uva.es>



I have a very simple Petsc program that multiplies a matrix and a vector 
several times. It works fine but it has some scalability issues. I 
execute it in a shared memory machine with 16 processors and it only 
runs 5 or 6 times faster (only taking into account the MatMult call). I 
have programmed the same algorithm with C and MPI and it shows a proper 
speedup (around 14 or 15). The matrices I use have millions of non zero 
elements, so I think they are big enough.

What can I do to get the same speedup that in the manual C version?

I enclose an except of the code. Thank you in advance.

Javier



/**
  * Main function
  */
int main(int argc, char ** argv){

     // Initialize Petsc
     PetscInitialize(&argc, &argv, (char *) 0, NULL);

     // Timers
     PetscLogDouble t_start, t_end;

     // File Viewer
     PetscViewer fd;
    
  PetscViewerBinaryOpen(PETSC_COMM_WORLD,"matrix_file",FILE_MODE_READ,&fd);

     // M matrix
     Mat M;
     MatCreate(PETSC_COMM_WORLD,&M);
     MatSetFromOptions(M);
     MatLoad(M,fd);
     PetscViewerDestroy(&fd);
     MatAssemblyBegin(M,MAT_FINAL_ASSEMBLY);
     MatAssemblyEnd(M,MAT_FINAL_ASSEMBLY);

     PetscInt n, m, local_n, local_m;
     MatGetSize(M,&n,&m);
     MatGetLocalSize(M,&local_n,&local_m);

     // b and c vectors
     Vec b,c;
     VecCreate(PETSC_COMM_WORLD,&b);
     VecSetFromOptions(b);
     VecSetSizes(b,local_n,n);

     VecCreate(PETSC_COMM_WORLD,&c);
     VecSetFromOptions(c);
     VecSetSizes(c,local_n,n);

     init_vector_values(b);

     VecAssemblyBegin(b);
     VecAssemblyEnd(b);


     // Main computation
     PetscGetTime(&t_start);
     int i;
     for(i=0; i<iter/2; i++){
         MatMult(M,b,c);
         MatMult(M,c,b);
     }
     PetscGetTime(&t_end);

     PetscPrintf(PETSC_COMM_WORLD,"Comp time: %lf\n",t_end-t_start);

     PetscFinalize();

     return 0;
}



From knepley at gmail.com  Wed May  2 11:13:56 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 2 May 2012 12:13:56 -0400
Subject: [petsc-users] Some Questions about Inexact Newton Solver
In-Reply-To: <CA+jOtDJWG+06-w5NpYQ280WsPf1KCSmeL6NGnB0vd1yua4s3tA@mail.gmail.com>
References: <CA+jOtDJWG+06-w5NpYQ280WsPf1KCSmeL6NGnB0vd1yua4s3tA@mail.gmail.com>
Message-ID: <CAMYG4GnuWgXAR14VrP0QgVEn5MMBho8kuJ1QUX2D2Oc-qqNkaw@mail.gmail.com>

On Wed, May 2, 2012 at 4:58 AM, behzad baghapour <behzad.baghapour at gmail.com
> wrote:

> Dear developers,
>
> I have some questions to set up the inexact Newton solver in SNES:
>
> 1- Is there any other method other than EW?
>

No, do you know of one?


> 2- What are the default values for EW when using -snes_ksp_ew?
>

-help


> 3- Is the following procedure enough to implement user-defined inexact
> method?
>
>     "reset the linear tolerance (rtol) in a monitor routine defined for
> SNES"
>

Yes, although I think I would use the Update() method.

   Matt


> Thanks a lot,
> BehZad
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120502/40d6f4a3/attachment.htm>

From mark.adams at columbia.edu  Wed May  2 11:24:43 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Wed, 2 May 2012 12:24:43 -0400
Subject: [petsc-users] MatMult scalability problem
In-Reply-To: <4FA15A49.8040101@infor.uva.es>
References: <4FA15A49.8040101@infor.uva.es>
Message-ID: <562D85F9-4F7E-43E8-A04E-EA4BC650D5B3@columbia.edu>

THings to check are:

1) run on a dedicated machine

2) are the matrices partitioned in the same way for both tests.

3) MatVec is memory bandwidth limited.  14 or 15 speedup on 16 shared memory cores is great.  5 or 6 is bad.  My experience with old IBS SPs was getting a speed of 12 and SPs have a great (and expensive) memory system.  I don't know what state-of-the-art is now but memory bandwidth has been going down generally wrt processor speed so I'd take a look at the number from your code again.

Mark

On May 2, 2012, at 12:01 PM, Javier Fresno wrote:

> 
> 
> I have a very simple Petsc program that multiplies a matrix and a vector several times. It works fine but it has some scalability issues. I execute it in a shared memory machine with 16 processors and it only runs 5 or 6 times faster (only taking into account the MatMult call). I have programmed the same algorithm with C and MPI and it shows a proper speedup (around 14 or 15). The matrices I use have millions of non zero elements, so I think they are big enough.
> 
> What can I do to get the same speedup that in the manual C version?
> 
> I enclose an except of the code. Thank you in advance.
> 
> Javier
> 
> 
> 
> /**
> * Main function
> */
> int main(int argc, char ** argv){
> 
>    // Initialize Petsc
>    PetscInitialize(&argc, &argv, (char *) 0, NULL);
> 
>    // Timers
>    PetscLogDouble t_start, t_end;
> 
>    // File Viewer
>    PetscViewer fd;
>    PetscViewerBinaryOpen(PETSC_COMM_WORLD,"matrix_file",FILE_MODE_READ,&fd);
> 
>    // M matrix
>    Mat M;
>    MatCreate(PETSC_COMM_WORLD,&M);
>    MatSetFromOptions(M);
>    MatLoad(M,fd);
>    PetscViewerDestroy(&fd);
>    MatAssemblyBegin(M,MAT_FINAL_ASSEMBLY);
>    MatAssemblyEnd(M,MAT_FINAL_ASSEMBLY);
> 
>    PetscInt n, m, local_n, local_m;
>    MatGetSize(M,&n,&m);
>    MatGetLocalSize(M,&local_n,&local_m);
> 
>    // b and c vectors
>    Vec b,c;
>    VecCreate(PETSC_COMM_WORLD,&b);
>    VecSetFromOptions(b);
>    VecSetSizes(b,local_n,n);
> 
>    VecCreate(PETSC_COMM_WORLD,&c);
>    VecSetFromOptions(c);
>    VecSetSizes(c,local_n,n);
> 
>    init_vector_values(b);
> 
>    VecAssemblyBegin(b);
>    VecAssemblyEnd(b);
> 
> 
>    // Main computation
>    PetscGetTime(&t_start);
>    int i;
>    for(i=0; i<iter/2; i++){
>        MatMult(M,b,c);
>        MatMult(M,c,b);
>    }
>    PetscGetTime(&t_end);
> 
>    PetscPrintf(PETSC_COMM_WORLD,"Comp time: %lf\n",t_end-t_start);
> 
>    PetscFinalize();
> 
>    return 0;
> }
> 
> 
> 


From knepley at gmail.com  Wed May  2 11:36:29 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 2 May 2012 12:36:29 -0400
Subject: [petsc-users] MatMult scalability problem
In-Reply-To: <4FA15A49.8040101@infor.uva.es>
References: <4FA15A49.8040101@infor.uva.es>
Message-ID: <CAMYG4GmEnd=oMjK4YYQNb=XQHuBoBLzat7ahzayGsEJF7V_WQA@mail.gmail.com>

On Wed, May 2, 2012 at 12:01 PM, Javier Fresno <jfresno at infor.uva.es> wrote:

>
>
> I have a very simple Petsc program that multiplies a matrix and a vector
> several times. It works fine but it has some scalability issues. I execute
> it in a shared memory machine with 16 processors and it only runs 5 or 6
> times faster (only taking into account the MatMult call). I have programmed
> the same algorithm with C and MPI and it shows a proper speedup (around 14
> or 15). The matrices I use have millions of non zero elements, so I think
> they are big enough.
>
> What can I do to get the same speedup that in the manual C version?
>
> I enclose an except of the code. Thank you in advance.
>

We need the output of -log_summary to say anything about performance. Also,
we need to know what the
machine architecture is.

    Matt


> Javier
>
>
>
> /**
>  * Main function
>  */
> int main(int argc, char ** argv){
>
>    // Initialize Petsc
>    PetscInitialize(&argc, &argv, (char *) 0, NULL);
>
>    // Timers
>    PetscLogDouble t_start, t_end;
>
>    // File Viewer
>    PetscViewer fd;
>    PetscViewerBinaryOpen(PETSC_**COMM_WORLD,"matrix_file",FILE_**
> MODE_READ,&fd);
>
>    // M matrix
>    Mat M;
>    MatCreate(PETSC_COMM_WORLD,&M)**;
>    MatSetFromOptions(M);
>    MatLoad(M,fd);
>    PetscViewerDestroy(&fd);
>    MatAssemblyBegin(M,MAT_FINAL_**ASSEMBLY);
>    MatAssemblyEnd(M,MAT_FINAL_**ASSEMBLY);
>
>    PetscInt n, m, local_n, local_m;
>    MatGetSize(M,&n,&m);
>    MatGetLocalSize(M,&local_n,&**local_m);
>
>    // b and c vectors
>    Vec b,c;
>    VecCreate(PETSC_COMM_WORLD,&b)**;
>    VecSetFromOptions(b);
>    VecSetSizes(b,local_n,n);
>
>    VecCreate(PETSC_COMM_WORLD,&c)**;
>    VecSetFromOptions(c);
>    VecSetSizes(c,local_n,n);
>
>    init_vector_values(b);
>
>    VecAssemblyBegin(b);
>    VecAssemblyEnd(b);
>
>
>    // Main computation
>    PetscGetTime(&t_start);
>    int i;
>    for(i=0; i<iter/2; i++){
>        MatMult(M,b,c);
>        MatMult(M,c,b);
>    }
>    PetscGetTime(&t_end);
>
>    PetscPrintf(PETSC_COMM_WORLD,"**Comp time: %lf\n",t_end-t_start);
>
>    PetscFinalize();
>
>    return 0;
> }
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120502/710c6b2a/attachment.htm>

From dkarthik at stanford.edu  Wed May  2 12:27:21 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Wed, 2 May 2012 10:27:21 -0700 (PDT)
Subject: [petsc-users] Multigrid
In-Reply-To: <2BAD4E6D-E17D-42C2-9501-664DCAAEA90A@mcs.anl.gov>
Message-ID: <2135394531.100982489.1335979641652.JavaMail.root@zm09.stanford.edu>

Hello,
       
    With PCML, I tried 
 
    -mg_coarse_pc_type  redundant ; -mg_coarse_ksp_type preonly ;   KSPSetType(ksp, KSPGMRES);
 
and the residual still diverges after a while. Strangely, if I use PCMG (with 0 levels), it works OK, but not with PCML.

This is definitely strange. Here is the entire piece of relevant code.

When initializing Petsc, I do the following: (let's call this the first bit and I do this only once) 

    KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
    KSPSetInitialGuessKnoll(ksp, PETSC_TRUE);
    KSPSetType(ksp, KSPFGMRES);
    KSPGMRESSetRestart(ksp, 100);
    KSPSetFromOptions(ksp);


Then for each outer solver iteration: (let's call this the second bit and I do this every outer iteration)

    PC pc;
    KSPGetPC(ksp,&pc);
    PCSetType(pc, PCMG);
    KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
    PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
    KSPSetType(ksp, KSPGMRES);
    KSPGMRESSetRestart(ksp, 100);
    KSPSetFromOptions(ksp);


I have tried various combinations of preconditioners and smoothers (including fgmres, bcgsl, etc) but the only combination that works is the above. Any of the PCML options that I have tried failed. 

Notes:
- If I keep both of the above bits, it converges very well

- If I keep both of the above bits, but replace PCMG with PCML, it diverges quickly

- If I removed the first bit and kept the second bit, the residuals diverge even with PCMG 

- If I keep the first bit and remove the second bit, the residuals diverge eventually, but at a slow rate


I am very confident with the rest of the code as I've been using PETSc (with just the first bit) for 3 years on 100s of problems.

Regards,
Karthik
















----- Original Message -----
From: "Barry Smith" <bsmith at mcs.anl.gov>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Wednesday, May 2, 2012 6:02:06 AM
Subject: Re: [petsc-users] Multigrid


   This is likely helping a great deal:   

  KSPGMRESSetRestart(ksp, 100);

   You can try this with ml also

On May 2, 2012, at 1:33 AM, Karthik Duraisamy wrote:

> Hello again,
> 
>      I played with BoomerAMG and ML (for the 2D problem with a reasonable condition number), and so far, they haven't worked (the residual eventually diverges). The options that I am using are (for instance)
> 
>    PC pc; 
>    KSPGetPC(ksp,&pc);
>    PetscOptionsSetValue("-ksp_type", "gmres");

   Here you NEED  fgmres since the smoother is gmres by default. You can see this with -ksp_view

>    KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>    PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>    PCSetType(pc, PCML);
>    PetscOptionsSetValue("-pc_ml_maxNlevels", "3");
>    PetscOptionsSetValue("-mg_coarse_ksp_type","richardson");
>    PetscOptionsSetValue("-mg_coarse_pc_type","sor");
>    PetscOptionsSetValue("-mg_coarse_pc_sor_its","8");

Just use -mg_coarse_pc_type  redundant -mg_coarse_ksp_type preonly instead of the above.


   Barry

>    PetscOptionsSetValue("-ksp_monitor","1");
>    KSPSetFromOptions(ksp);


> 
> I have replaced KSP with bcgsl, fgmres, etc.
> 
> As I mentioned earlier, the following works like a charm:
> 
>    PC pc; 
>    KSPGetPC(ksp,&pc);
>    PCSetType(pc, PCMG);
>    KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>    PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>    KSPSetType(ksp, KSPGMRES);
>    KSPGMRESSetRestart(ksp, 100);
>    KSPSetFromOptions(ksp);
> 
> 
> Regards,
> Karthik.
> 
> 
> 
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Tuesday, May 1, 2012 9:14:05 PM
> Subject: Re: [petsc-users] Multigrid
> 
> 
> On May 1, 2012, at 6:00 PM, Karthik Duraisamy wrote:
> 
>> So as I understand it, GMRES is used as a preconditioner and as a solver when I use PCMG with defaults. If this is the case, I should be able to recreate this set up without the PCMG. Any pointers as to how this can be done?
>> 
>> Also, yes indeed, my mesh is completely unstructured, so I will have to use ml or boomeramg. 
>> 
>> The problems that I am attempting involve RANS of compressible turbulent combustion (finite volume, steady). The high condition numbers are because of the extreme grid stretching and stiff source terms (in the turbulence and combustion model).
> 
>   With a condition number like that I wouldn't even consider using double precision, I think you would just be computing meaningless noise. With petsc-dev you can use quad precision ./configure --with-precision=__float128 using recent GNU compilers
> (no we haven't tested the gfortran sides of things). 
> 
>   Barry
> 
> 
>> I have been trying a reduced problem in these 2D test cases, in which the condition number is only 1e7.
>> 
>> Thanks,
>> Karthik.
>> 
>> 
>> ----- Original Message -----
>> From: "Mark F. Adams" <mark.adams at columbia.edu>
>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>> Sent: Tuesday, May 1, 2012 3:50:31 PM
>> Subject: Re: [petsc-users] Multigrid
>> 
>> 
>> Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ matrix. If you use regular grids (DA?) then PETSc can construct geometric multigrid coarse grid spaces, although I don't know if PCMG will construct these for you (I don't think it will and I can see from your output that PCMG just used one grid). 'ml', hypre' and 'gamg' (a native AMG solver) will do real AMG solvers for you. All three can work on a similar class of problems. 
>> 
>> 
>> Also, you mention that you have a condition number of 1.e20. That is astronomical for such a small problem. How did you compute that number? Do you know where the ill-conditioning comes from? Is this an elliptic operator? 
>> 
>> 
>> Mark 
>> 
>> 
>> 
>> 
>> On May 1, 2012, at 6:31 PM, Matthew Knepley wrote: 
>> 
>> 
>> 
>> On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>> 
>> 
>> 
>> The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow) 
>> 
>> 
>> 
>> Because the smoother IS the solver you were using before. Just like I said last time, what you are doing is 
>> wrapping up the same solver you used before, sticking it in another GMRES loop, and only looking at the 
>> outer loop. This has nothing to do with MG. 
>> 
>> 
>> Matt 
>> 
>> 
>> KSP Object: 8 MPI processes 
>> type: gmres 
>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>> GMRES: happy breakdown tolerance 1e-30 
>> maximum iterations=1, initial guess is zero 
>> using preconditioner applied to right hand side for initial guess 
>> tolerances: relative=0.01, absolute=1e-08, divergence=1e+10 
>> left preconditioning 
>> using DEFAULT norm type for convergence test 
>> PC Object: 8 MPI processes 
>> type: mg 
>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>> Cycles per PCApply=1 
>> Not using Galerkin computed coarse grid matrices 
>> Coarse grid solver -- level ------------------------------- 
>> KSP Object: (mg_levels_0_) 8 MPI processes 
>> type not yet set 
>> maximum iterations=1, initial guess is zero 
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>> left preconditioning 
>> using DEFAULT norm type for convergence test 
>> PC Object: (mg_levels_0_) 8 MPI processes 
>> type not yet set 
>> linear system matrix = precond matrix: 
>> Matrix Object: 8 MPI processes 
>> type: mpiaij 
>> rows=75000, cols=75000 
>> total: nonzeros=4427800, allocated nonzeros=4427800 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>> 
>> 
>> ----- Original Message ----- 
>> From: "Matthew Knepley" < knepley at gmail.com > 
>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>> Sent: Tuesday, May 1, 2012 3:22:56 PM 
>> Subject: Re: [petsc-users] Multigrid 
>> 
>> 
>> On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>> 
>> 
>> 
>> Hello, 
>> 
>> Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same). 
>> 
>> 
>> 
>> Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be 
>> an extra GMRES loop compared to the case without MG. 
>> 
>> 
>> Matt 
>> 
>> 
>> Regards, 
>> Karthik 
>> 
>> KSP Object: 8 MPI processes 
>> type: gmres 
>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>> GMRES: happy breakdown tolerance 1e-30 
>> maximum iterations=1 
>> using preconditioner applied to right hand side for initial guess 
>> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 
>> left preconditioning 
>> using nonzero initial guess 
>> using PRECONDITIONED norm type for convergence test 
>> PC Object: 8 MPI processes 
>> type: bjacobi 
>> block Jacobi: number of blocks = 8 
>> Local solve is same for all blocks, in the following KSP and PC objects: 
>> KSP Object: (sub_) 1 MPI processes 
>> type: preonly 
>> maximum iterations=10000, initial guess is zero 
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>> left preconditioning 
>> using NONE norm type for convergence test 
>> PC Object: (sub_) 1 MPI processes 
>> type: ilu 
>> ILU: out-of-place factorization 
>> 0 levels of fill 
>> tolerance for zero pivot 1e-12 
>> using diagonal shift to prevent zero pivot 
>> matrix ordering: natural 
>> factor fill ratio given 1, needed 1 
>> Factored matrix follows: 
>> Matrix Object: 1 MPI processes 
>> type: seqaij 
>> rows=9015, cols=9015 
>> package used to perform factorization: petsc 
>> total: nonzeros=517777, allocated nonzeros=517777 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node routines: found 3476 nodes, limit used is 5 
>> linear system matrix = precond matrix: 
>> Matrix Object: 1 MPI processes 
>> type: seqaij 
>> rows=9015, cols=9015 
>> total: nonzeros=517777, allocated nonzeros=517777 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node routines: found 3476 nodes, limit used is 5 
>> linear system matrix = precond matrix: 
>> Matrix Object: 8 MPI processes 
>> type: mpiaij 
>> rows=75000, cols=75000 
>> total: nonzeros=4427800, allocated nonzeros=4427800 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>> 
>> 
>> ----- Original Message ----- 
>> From: "Matthew Knepley" < knepley at gmail.com > 
>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>> Sent: Tuesday, May 1, 2012 3:15:14 PM 
>> Subject: Re: [petsc-users] Multigrid 
>> 
>> 
>> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>> 
>> 
>> 
>> Hello Barry, 
>> 
>> Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 
>> 
>> 
>> 
>> You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 
>> 
>> 
>> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 
>> 
>> 
>> 
>> They are different algorithms. Its not possible to say generally that one is better. Try them both. 
>> 
>> 
>> Matt 
>> 
>> 
>> Best regards, 
>> Karthik. 
>> 
>> type: mg 
>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>> Cycles per PCApply=1 
>> Not using Galerkin computed coarse grid matrices 
>> Coarse grid solver -- level ------------------------------- 
>> KSP Object: (mg_levels_0_) 8 MPI processes 
>> type: gmres 
>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>> GMRES: happy breakdown tolerance 1e-30 
>> maximum iterations=1, initial guess is zero 
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>> left preconditioning 
>> using PRECONDITIONED norm type for convergence test 
>> PC Object: (mg_levels_0_) 8 MPI processes 
>> type: bjacobi 
>> block Jacobi: number of blocks = 8 
>> Local solve is same for all blocks, in the following KSP and PC objects: 
>> KSP Object: (mg_levels_0_sub_) 1 MPI processes 
>> type: preonly 
>> maximum iterations=10000, initial guess is zero 
>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>> left preconditioning 
>> using NONE norm type for convergence test 
>> PC Object: (mg_levels_0_sub_) 1 MPI processes 
>> type: ilu 
>> ILU: out-of-place factorization 
>> 0 levels of fill 
>> tolerance for zero pivot 1e-12 
>> using diagonal shift to prevent zero pivot 
>> matrix ordering: natural 
>> factor fill ratio given 1, needed 1 
>> Factored matrix follows: 
>> Matrix Object: 1 MPI processes 
>> type: seqaij 
>> rows=9015, cols=9015 
>> package used to perform factorization: petsc 
>> total: nonzeros=517777, allocated nonzeros=517777 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node routines: found 3476 nodes, limit used is 5 
>> linear system matrix = precond matrix: 
>> Matrix Object: 1 MPI processes 
>> type: seqaij 
>> rows=9015, cols=9015 
>> total: nonzeros=517777, allocated nonzeros=517777 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node routines: found 3476 nodes, limit used is 5 
>> linear system matrix = precond matrix: 
>> Matrix Object: 8 MPI processes 
>> type: mpiaij 
>> rows=75000, cols=75000 
>> total: nonzeros=4427800, allocated nonzeros=4427800 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>> linear system matrix = precond matrix: 
>> Matrix Object: 8 MPI processes 
>> type: mpiaij 
>> rows=75000, cols=75000 
>> total: nonzeros=4427800, allocated nonzeros=4427800 
>> total number of mallocs used during MatSetValues calls =0 
>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>> 
>> 
>> 
>> ----- Original Message ----- 
>> From: "Barry Smith" < bsmith at mcs.anl.gov > 
>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>> Sent: Tuesday, May 1, 2012 1:39:26 PM 
>> Subject: Re: [petsc-users] Multigrid 
>> 
>> 
>> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 
>> 
>>> Hello, 
>>> 
>>> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
>>> 
>>> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
>>> help in this regard will be appreciated. 
>> 
>> First run with -ksp_view to see what solver it is actually using. 
>> 
>> Barry 
>> 
>>> 
>>> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
>>> 
>>> Thanks, 
>>> Karthik 
>>> 
>>> 
>>> 
>>> -- 
>>> 
>>> ======================================= 
>>> Karthik Duraisamy 
>>> Assistant Professor (Consulting) 
>>> Durand Building Rm 357 
>>> Dept of Aeronautics and Astronautics 
>>> Stanford University 
>>> Stanford CA 94305 
>>> 
>>> Phone: 650-721-2835 
>>> Web: www.stanford.edu/~dkarthik 
>>> ======================================= 
>> 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>> -- Norbert Wiener 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>> -- Norbert Wiener 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>> -- Norbert Wiener 
>> 
> 


From bsmith at mcs.anl.gov  Wed May  2 12:34:17 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 2 May 2012 12:34:17 -0500
Subject: [petsc-users] Multigrid
In-Reply-To: <2135394531.100982489.1335979641652.JavaMail.root@zm09.stanford.edu>
References: <2135394531.100982489.1335979641652.JavaMail.root@zm09.stanford.edu>
Message-ID: <23ADD932-4C45-421A-B401-F929C8B0CEB5@mcs.anl.gov>


   Run with -pc_type ml -ksp_type fgmres   -ksp_max_it 100 -ksp_view   and NOTHING else. Don'ts set any KSP or PC options in the code!  Then send us ALL the output.

    Barry

  I am getting all confused with "I ran with these options and this happened, I ran with these options and this happened, I ran with these options and this happened". I am getting information overload with too much information and yet not enough information.


On May 2, 2012, at 12:27 PM, Karthik Duraisamy wrote:

> Hello,
> 
>    With PCML, I tried 
> 
>    -mg_coarse_pc_type  redundant ; -mg_coarse_ksp_type preonly ;   KSPSetType(ksp, KSPGMRES);
> 
> and the residual still diverges after a while. Strangely, if I use PCMG (with 0 levels), it works OK, but not with PCML.
> 
> This is definitely strange. Here is the entire piece of relevant code.
> 
> When initializing Petsc, I do the following: (let's call this the first bit and I do this only once) 
> 
>    KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>    KSPSetInitialGuessKnoll(ksp, PETSC_TRUE);
>    KSPSetType(ksp, KSPFGMRES);
>    KSPGMRESSetRestart(ksp, 100);
>    KSPSetFromOptions(ksp);
> 
> 
> Then for each outer solver iteration: (let's call this the second bit and I do this every outer iteration)
> 
>    PC pc;
>    KSPGetPC(ksp,&pc);
>    PCSetType(pc, PCMG);
>    KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>    PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>    KSPSetType(ksp, KSPGMRES);
>    KSPGMRESSetRestart(ksp, 100);
>    KSPSetFromOptions(ksp);
> 
> 
> I have tried various combinations of preconditioners and smoothers (including fgmres, bcgsl, etc) but the only combination that works is the above. Any of the PCML options that I have tried failed. 
> 
> Notes:
> - If I keep both of the above bits, it converges very well
> 
> - If I keep both of the above bits, but replace PCMG with PCML, it diverges quickly
> 
> - If I removed the first bit and kept the second bit, the residuals diverge even with PCMG 
> 
> - If I keep the first bit and remove the second bit, the residuals diverge eventually, but at a slow rate
> 
> 
> I am very confident with the rest of the code as I've been using PETSc (with just the first bit) for 3 years on 100s of problems.
> 
> Regards,
> Karthik
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Wednesday, May 2, 2012 6:02:06 AM
> Subject: Re: [petsc-users] Multigrid
> 
> 
>   This is likely helping a great deal:   
> 
>  KSPGMRESSetRestart(ksp, 100);
> 
>   You can try this with ml also
> 
> On May 2, 2012, at 1:33 AM, Karthik Duraisamy wrote:
> 
>> Hello again,
>> 
>>     I played with BoomerAMG and ML (for the 2D problem with a reasonable condition number), and so far, they haven't worked (the residual eventually diverges). The options that I am using are (for instance)
>> 
>>   PC pc; 
>>   KSPGetPC(ksp,&pc);
>>   PetscOptionsSetValue("-ksp_type", "gmres");
> 
>   Here you NEED  fgmres since the smoother is gmres by default. You can see this with -ksp_view
> 
>>   KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>   PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>   PCSetType(pc, PCML);
>>   PetscOptionsSetValue("-pc_ml_maxNlevels", "3");
>>   PetscOptionsSetValue("-mg_coarse_ksp_type","richardson");
>>   PetscOptionsSetValue("-mg_coarse_pc_type","sor");
>>   PetscOptionsSetValue("-mg_coarse_pc_sor_its","8");
> 
> Just use -mg_coarse_pc_type  redundant -mg_coarse_ksp_type preonly instead of the above.
> 
> 
>   Barry
> 
>>   PetscOptionsSetValue("-ksp_monitor","1");
>>   KSPSetFromOptions(ksp);
> 
> 
>> 
>> I have replaced KSP with bcgsl, fgmres, etc.
>> 
>> As I mentioned earlier, the following works like a charm:
>> 
>>   PC pc; 
>>   KSPGetPC(ksp,&pc);
>>   PCSetType(pc, PCMG);
>>   KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>   PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>   KSPSetType(ksp, KSPGMRES);
>>   KSPGMRESSetRestart(ksp, 100);
>>   KSPSetFromOptions(ksp);
>> 
>> 
>> Regards,
>> Karthik.
>> 
>> 
>> 
>> ----- Original Message -----
>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>> Sent: Tuesday, May 1, 2012 9:14:05 PM
>> Subject: Re: [petsc-users] Multigrid
>> 
>> 
>> On May 1, 2012, at 6:00 PM, Karthik Duraisamy wrote:
>> 
>>> So as I understand it, GMRES is used as a preconditioner and as a solver when I use PCMG with defaults. If this is the case, I should be able to recreate this set up without the PCMG. Any pointers as to how this can be done?
>>> 
>>> Also, yes indeed, my mesh is completely unstructured, so I will have to use ml or boomeramg. 
>>> 
>>> The problems that I am attempting involve RANS of compressible turbulent combustion (finite volume, steady). The high condition numbers are because of the extreme grid stretching and stiff source terms (in the turbulence and combustion model).
>> 
>>  With a condition number like that I wouldn't even consider using double precision, I think you would just be computing meaningless noise. With petsc-dev you can use quad precision ./configure --with-precision=__float128 using recent GNU compilers
>> (no we haven't tested the gfortran sides of things). 
>> 
>>  Barry
>> 
>> 
>>> I have been trying a reduced problem in these 2D test cases, in which the condition number is only 1e7.
>>> 
>>> Thanks,
>>> Karthik.
>>> 
>>> 
>>> ----- Original Message -----
>>> From: "Mark F. Adams" <mark.adams at columbia.edu>
>>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>>> Sent: Tuesday, May 1, 2012 3:50:31 PM
>>> Subject: Re: [petsc-users] Multigrid
>>> 
>>> 
>>> Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ matrix. If you use regular grids (DA?) then PETSc can construct geometric multigrid coarse grid spaces, although I don't know if PCMG will construct these for you (I don't think it will and I can see from your output that PCMG just used one grid). 'ml', hypre' and 'gamg' (a native AMG solver) will do real AMG solvers for you. All three can work on a similar class of problems. 
>>> 
>>> 
>>> Also, you mention that you have a condition number of 1.e20. That is astronomical for such a small problem. How did you compute that number? Do you know where the ill-conditioning comes from? Is this an elliptic operator? 
>>> 
>>> 
>>> Mark 
>>> 
>>> 
>>> 
>>> 
>>> On May 1, 2012, at 6:31 PM, Matthew Knepley wrote: 
>>> 
>>> 
>>> 
>>> On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>> 
>>> 
>>> 
>>> The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow) 
>>> 
>>> 
>>> 
>>> Because the smoother IS the solver you were using before. Just like I said last time, what you are doing is 
>>> wrapping up the same solver you used before, sticking it in another GMRES loop, and only looking at the 
>>> outer loop. This has nothing to do with MG. 
>>> 
>>> 
>>> Matt 
>>> 
>>> 
>>> KSP Object: 8 MPI processes 
>>> type: gmres 
>>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>> GMRES: happy breakdown tolerance 1e-30 
>>> maximum iterations=1, initial guess is zero 
>>> using preconditioner applied to right hand side for initial guess 
>>> tolerances: relative=0.01, absolute=1e-08, divergence=1e+10 
>>> left preconditioning 
>>> using DEFAULT norm type for convergence test 
>>> PC Object: 8 MPI processes 
>>> type: mg 
>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>>> Cycles per PCApply=1 
>>> Not using Galerkin computed coarse grid matrices 
>>> Coarse grid solver -- level ------------------------------- 
>>> KSP Object: (mg_levels_0_) 8 MPI processes 
>>> type not yet set 
>>> maximum iterations=1, initial guess is zero 
>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>> left preconditioning 
>>> using DEFAULT norm type for convergence test 
>>> PC Object: (mg_levels_0_) 8 MPI processes 
>>> type not yet set 
>>> linear system matrix = precond matrix: 
>>> Matrix Object: 8 MPI processes 
>>> type: mpiaij 
>>> rows=75000, cols=75000 
>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>> 
>>> 
>>> ----- Original Message ----- 
>>> From: "Matthew Knepley" < knepley at gmail.com > 
>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>> Sent: Tuesday, May 1, 2012 3:22:56 PM 
>>> Subject: Re: [petsc-users] Multigrid 
>>> 
>>> 
>>> On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>> 
>>> 
>>> 
>>> Hello, 
>>> 
>>> Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same). 
>>> 
>>> 
>>> 
>>> Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be 
>>> an extra GMRES loop compared to the case without MG. 
>>> 
>>> 
>>> Matt 
>>> 
>>> 
>>> Regards, 
>>> Karthik 
>>> 
>>> KSP Object: 8 MPI processes 
>>> type: gmres 
>>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>> GMRES: happy breakdown tolerance 1e-30 
>>> maximum iterations=1 
>>> using preconditioner applied to right hand side for initial guess 
>>> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 
>>> left preconditioning 
>>> using nonzero initial guess 
>>> using PRECONDITIONED norm type for convergence test 
>>> PC Object: 8 MPI processes 
>>> type: bjacobi 
>>> block Jacobi: number of blocks = 8 
>>> Local solve is same for all blocks, in the following KSP and PC objects: 
>>> KSP Object: (sub_) 1 MPI processes 
>>> type: preonly 
>>> maximum iterations=10000, initial guess is zero 
>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>> left preconditioning 
>>> using NONE norm type for convergence test 
>>> PC Object: (sub_) 1 MPI processes 
>>> type: ilu 
>>> ILU: out-of-place factorization 
>>> 0 levels of fill 
>>> tolerance for zero pivot 1e-12 
>>> using diagonal shift to prevent zero pivot 
>>> matrix ordering: natural 
>>> factor fill ratio given 1, needed 1 
>>> Factored matrix follows: 
>>> Matrix Object: 1 MPI processes 
>>> type: seqaij 
>>> rows=9015, cols=9015 
>>> package used to perform factorization: petsc 
>>> total: nonzeros=517777, allocated nonzeros=517777 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node routines: found 3476 nodes, limit used is 5 
>>> linear system matrix = precond matrix: 
>>> Matrix Object: 1 MPI processes 
>>> type: seqaij 
>>> rows=9015, cols=9015 
>>> total: nonzeros=517777, allocated nonzeros=517777 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node routines: found 3476 nodes, limit used is 5 
>>> linear system matrix = precond matrix: 
>>> Matrix Object: 8 MPI processes 
>>> type: mpiaij 
>>> rows=75000, cols=75000 
>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>> 
>>> 
>>> ----- Original Message ----- 
>>> From: "Matthew Knepley" < knepley at gmail.com > 
>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>> Sent: Tuesday, May 1, 2012 3:15:14 PM 
>>> Subject: Re: [petsc-users] Multigrid 
>>> 
>>> 
>>> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>> 
>>> 
>>> 
>>> Hello Barry, 
>>> 
>>> Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 
>>> 
>>> 
>>> 
>>> You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 
>>> 
>>> 
>>> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 
>>> 
>>> 
>>> 
>>> They are different algorithms. Its not possible to say generally that one is better. Try them both. 
>>> 
>>> 
>>> Matt 
>>> 
>>> 
>>> Best regards, 
>>> Karthik. 
>>> 
>>> type: mg 
>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>>> Cycles per PCApply=1 
>>> Not using Galerkin computed coarse grid matrices 
>>> Coarse grid solver -- level ------------------------------- 
>>> KSP Object: (mg_levels_0_) 8 MPI processes 
>>> type: gmres 
>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>> GMRES: happy breakdown tolerance 1e-30 
>>> maximum iterations=1, initial guess is zero 
>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>> left preconditioning 
>>> using PRECONDITIONED norm type for convergence test 
>>> PC Object: (mg_levels_0_) 8 MPI processes 
>>> type: bjacobi 
>>> block Jacobi: number of blocks = 8 
>>> Local solve is same for all blocks, in the following KSP and PC objects: 
>>> KSP Object: (mg_levels_0_sub_) 1 MPI processes 
>>> type: preonly 
>>> maximum iterations=10000, initial guess is zero 
>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>> left preconditioning 
>>> using NONE norm type for convergence test 
>>> PC Object: (mg_levels_0_sub_) 1 MPI processes 
>>> type: ilu 
>>> ILU: out-of-place factorization 
>>> 0 levels of fill 
>>> tolerance for zero pivot 1e-12 
>>> using diagonal shift to prevent zero pivot 
>>> matrix ordering: natural 
>>> factor fill ratio given 1, needed 1 
>>> Factored matrix follows: 
>>> Matrix Object: 1 MPI processes 
>>> type: seqaij 
>>> rows=9015, cols=9015 
>>> package used to perform factorization: petsc 
>>> total: nonzeros=517777, allocated nonzeros=517777 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node routines: found 3476 nodes, limit used is 5 
>>> linear system matrix = precond matrix: 
>>> Matrix Object: 1 MPI processes 
>>> type: seqaij 
>>> rows=9015, cols=9015 
>>> total: nonzeros=517777, allocated nonzeros=517777 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node routines: found 3476 nodes, limit used is 5 
>>> linear system matrix = precond matrix: 
>>> Matrix Object: 8 MPI processes 
>>> type: mpiaij 
>>> rows=75000, cols=75000 
>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>> linear system matrix = precond matrix: 
>>> Matrix Object: 8 MPI processes 
>>> type: mpiaij 
>>> rows=75000, cols=75000 
>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>> 
>>> 
>>> 
>>> ----- Original Message ----- 
>>> From: "Barry Smith" < bsmith at mcs.anl.gov > 
>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>> Sent: Tuesday, May 1, 2012 1:39:26 PM 
>>> Subject: Re: [petsc-users] Multigrid 
>>> 
>>> 
>>> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 
>>> 
>>>> Hello, 
>>>> 
>>>> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
>>>> 
>>>> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
>>>> help in this regard will be appreciated. 
>>> 
>>> First run with -ksp_view to see what solver it is actually using. 
>>> 
>>> Barry 
>>> 
>>>> 
>>>> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
>>>> 
>>>> Thanks, 
>>>> Karthik 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> 
>>>> ======================================= 
>>>> Karthik Duraisamy 
>>>> Assistant Professor (Consulting) 
>>>> Durand Building Rm 357 
>>>> Dept of Aeronautics and Astronautics 
>>>> Stanford University 
>>>> Stanford CA 94305 
>>>> 
>>>> Phone: 650-721-2835 
>>>> Web: www.stanford.edu/~dkarthik 
>>>> ======================================= 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>> -- Norbert Wiener 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>> -- Norbert Wiener 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>> -- Norbert Wiener 
>>> 
>> 
> 


From behzad.baghapour at gmail.com  Wed May  2 12:38:04 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Wed, 2 May 2012 21:08:04 +0330
Subject: [petsc-users] Some Questions about Inexact Newton Solver
In-Reply-To: <CAMYG4GnuWgXAR14VrP0QgVEn5MMBho8kuJ1QUX2D2Oc-qqNkaw@mail.gmail.com>
References: <CA+jOtDJWG+06-w5NpYQ280WsPf1KCSmeL6NGnB0vd1yua4s3tA@mail.gmail.com>
	<CAMYG4GnuWgXAR14VrP0QgVEn5MMBho8kuJ1QUX2D2Oc-qqNkaw@mail.gmail.com>
Message-ID: <CA+jOtDKHYdyCRybbPoRno=tEbgV9zWfsPRMAnXfLfrfJZFwdvw@mail.gmail.com>

Thanks. Actually I couldn't find a different noticeable method in this area
in the literature yet. I am searching to found out the ability of EW in
non-smooth nonlinear problems especially in fluid flows like shock waves
and stagnation points.

Regards,
BehZad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120502/3cc3f6f4/attachment.htm>

From zonexo at gmail.com  Wed May  2 12:55:51 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Wed, 02 May 2012 19:55:51 +0200
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <CAMYG4Gk-vSa_kwRnp58up8cu7YPDN53Lb03JQviB+nduJSy9KQ@mail.gmail.com>
References: <4F97E8F2.5050803@gmail.com> <4F993F1D.9020105@gmail.com>
	<CAMYG4GnHyb1VKSY3gwpTxyU=xe1ZcEhitKCu5+CpK9a1v0ymuA@mail.gmail.com>
	<4F99A15B.1010500@gmail.com>
	<CAMYG4Gmzkpt+UyYLbzhit9Jr1G6SO99C9qKp+qg6Nyd__rTo3g@mail.gmail.com>
	<4F99CA4C.1080609@gmail.com>
	<C92D4D80-A685-4196-BFF6-85E4CB795D04@mcs.anl.gov>
	<4FA05A23.7080209@gmail.com>
	<CAMYG4Gk-vSa_kwRnp58up8cu7YPDN53Lb03JQviB+nduJSy9KQ@mail.gmail.com>
Message-ID: <4FA17527.6030100@gmail.com>

Hi,

I did a MatView and VecView on both C and Fortran, right after Mat and 
Vec assembly. I have attached the printout below. They are exactly the 
same, but yet the result is different in Neumann condition. However, the 
dirichlet condition gives the correct ans. Is there anything else that 
could be wrong even if the Mat and Vec are the same?

Thanks!

Fortran:

Matrix Object: 1 MPI processes
   type: seqaij
row 0: (0, 2)  (1, -1)  (3, -1)
row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
row 2: (1, -1)  (2, 2)  (5, -1)
row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
row 6: (3, -1)  (6, 2)  (7, -1)
row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
row 8: (5, -1)  (7, -1)  (8, 2)
Vector Object:Vec_0000000084000000_0 1 MPI processes
   type: mpi
Process [0]
0.25
0.0205213
1.135e-005
0.0205213
0.00168449
9.31663e-007
1.135e-005
9.31663e-007
5.15289e-010
Vector Object:Vec_0000000084000000_1 1 MPI processes
   type: mpi
Process [0]
0.14924
0.0242397
-0.0260347
0.0242397
-0.0256192
-0.0400102
-0.0260347
-0.0400102
-0.0400102
Press any key to continue . . .

C:

Matrix Object: 1 MPI processes
   type: seqaij
row 0: (0, 2)  (1, -1)  (3, -1)
row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
row 2: (1, -1)  (2, 2)  (5, -1)
row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
row 6: (3, -1)  (6, 2)  (7, -1)
row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
row 8: (5, -1)  (7, -1)  (8, 2)
Vector Object:Vec_0x1d3b000_0 1 MPI processes
   type: mpi
Process [0]
0.25
0.0205212
1.135e-05
0.0205212
0.00168449
9.31663e-07
1.135e-05
9.31663e-07
5.15288e-10
Vector Object:Vec_0x1d3b000_1 1 MPI processes
   type: mpi
Process [0]
0.139311
0.0305751
-0.0220633
0.0305751
-0.0135158
-0.042185
-0.0220633
-0.042185
-0.058449



Yours sincerely,

TAY wee-beng


On 1/5/2012 11:54 PM, Matthew Knepley wrote:
> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     Do you mean my method is wrong?
>
>     I am following the template of ex22f,
>
>     where the variables are declared as :
>
>     PetscScalar  v(5)
>
>     MatStencil   row(4),col(4,5)
>
>     Hence,
>
>     for the neumann BC
>
>     num = 1
>
>                    if (j/=0) then
>
>                        v(num) = -rho*HxdHy
>
>                        col(MatStencil_i,num) = i
>
>                        col(MatStencil_j,num) = j-1
>
>                        num = num + 1
>
>                    end if
>
>                    if (i/=0) then
>
>                        v(num) = -rho*HydHx
>
>                        col(MatStencil_i,num) = i-1
>
>                        col(MatStencil_j,num) = j
>
>                        num = num + 1
>
>                    end if
>
>                    if (i/=mx-1) then
>
>                        v(num) = -rho*HydHx
>
>                        col(MatStencil_i,num) = i+1
>
>                        col(MatStencil_j,num) = j
>
>                        num = num + 1
>
>                    end if
>
>                    if (j/=my-1) then
>
>                        v(num) = -rho*HxdHy
>
>                        col(MatStencil_i,num) = i
>
>                        col(MatStencil_j,num) = j+1
>
>                        num = num + 1
>
>                    end if
>
>                    v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>
>                    print *, v
>
>                    col(MatStencil_i,num) = i
>
>                    col(MatStencil_j,num) = j
>
>                    !num = num + 1
>
>                    call
>     MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>
>     I do not get any more out of range error. However,my ans is still
>     different from that of ex29 in C.
>
>
> This is very simple. You have an error in your code. Checking it is 
> very simple: run the code and
> break in MatSetValues(). Make sure ex29 makes calls with exactly the 
> same indices as your ex29f.
>
>     Matt
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120502/9011a365/attachment.htm>

From knepley at gmail.com  Wed May  2 15:06:20 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 2 May 2012 16:06:20 -0400
Subject: [petsc-users] Some Questions about Inexact Newton Solver
In-Reply-To: <CA+jOtDKHYdyCRybbPoRno=tEbgV9zWfsPRMAnXfLfrfJZFwdvw@mail.gmail.com>
References: <CA+jOtDJWG+06-w5NpYQ280WsPf1KCSmeL6NGnB0vd1yua4s3tA@mail.gmail.com>
	<CAMYG4GnuWgXAR14VrP0QgVEn5MMBho8kuJ1QUX2D2Oc-qqNkaw@mail.gmail.com>
	<CA+jOtDKHYdyCRybbPoRno=tEbgV9zWfsPRMAnXfLfrfJZFwdvw@mail.gmail.com>
Message-ID: <CAMYG4Gn3p0WgyOuTud4B4+f=XGGKYyi9qjAs3w+ArWVO6wD-yA@mail.gmail.com>

On Wed, May 2, 2012 at 1:38 PM, behzad baghapour <behzad.baghapour at gmail.com
> wrote:

> Thanks. Actually I couldn't find a different noticeable method in this
> area in the literature yet. I am searching to found out the ability of EW
> in non-smooth nonlinear problems especially in fluid flows like shock waves
> and stagnation points.
>

I suspect something like ASPIN would be much better, but we don't have that
working yet.

   Matt


> Regards,
> BehZad

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120502/f4062d7f/attachment.htm>

From knepley at gmail.com  Wed May  2 15:11:51 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 2 May 2012 16:11:51 -0400
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <4FA17527.6030100@gmail.com>
References: <4F97E8F2.5050803@gmail.com> <4F993F1D.9020105@gmail.com>
	<CAMYG4GnHyb1VKSY3gwpTxyU=xe1ZcEhitKCu5+CpK9a1v0ymuA@mail.gmail.com>
	<4F99A15B.1010500@gmail.com>
	<CAMYG4Gmzkpt+UyYLbzhit9Jr1G6SO99C9qKp+qg6Nyd__rTo3g@mail.gmail.com>
	<4F99CA4C.1080609@gmail.com>
	<C92D4D80-A685-4196-BFF6-85E4CB795D04@mcs.anl.gov>
	<4FA05A23.7080209@gmail.com>
	<CAMYG4Gk-vSa_kwRnp58up8cu7YPDN53Lb03JQviB+nduJSy9KQ@mail.gmail.com>
	<4FA17527.6030100@gmail.com>
Message-ID: <CAMYG4GkSc4dzoavG8=jM7GNo53c0Vdmumn4NEj2JacxxdLtw7Q@mail.gmail.com>

On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng <zonexo at gmail.com> wrote:

>  Hi,
>
> I did a MatView and VecView on both C and Fortran, right after Mat and Vec
> assembly. I have attached the printout below. They are exactly the same,
> but yet the result is different in Neumann condition. However, the
> dirichlet condition gives the correct ans. Is there anything else that
> could be wrong even if the Mat and Vec are the same?
>

Did you set the null space for the matrix when you have Neumann conditions?

   Matt


> Thanks!
>
> Fortran:
>
> Matrix Object: 1 MPI processes
>   type: seqaij
> row 0: (0, 2)  (1, -1)  (3, -1)
> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
> row 2: (1, -1)  (2, 2)  (5, -1)
> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
> row 6: (3, -1)  (6, 2)  (7, -1)
> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
> row 8: (5, -1)  (7, -1)  (8, 2)
> Vector Object:Vec_0000000084000000_0 1 MPI processes
>   type: mpi
> Process [0]
> 0.25
> 0.0205213
> 1.135e-005
> 0.0205213
> 0.00168449
> 9.31663e-007
> 1.135e-005
> 9.31663e-007
> 5.15289e-010
> Vector Object:Vec_0000000084000000_1 1 MPI processes
>   type: mpi
> Process [0]
> 0.14924
> 0.0242397
> -0.0260347
> 0.0242397
> -0.0256192
> -0.0400102
> -0.0260347
> -0.0400102
> -0.0400102
> Press any key to continue . . .
>
> C:
>
> Matrix Object: 1 MPI processes
>   type: seqaij
> row 0: (0, 2)  (1, -1)  (3, -1)
> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
> row 2: (1, -1)  (2, 2)  (5, -1)
> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
> row 6: (3, -1)  (6, 2)  (7, -1)
> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
> row 8: (5, -1)  (7, -1)  (8, 2)
> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>   type: mpi
> Process [0]
> 0.25
> 0.0205212
> 1.135e-05
> 0.0205212
> 0.00168449
> 9.31663e-07
> 1.135e-05
> 9.31663e-07
> 5.15288e-10
> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>   type: mpi
> Process [0]
> 0.139311
> 0.0305751
> -0.0220633
> 0.0305751
> -0.0135158
> -0.042185
> -0.0220633
> -0.042185
> -0.058449
>
>
>
> Yours sincerely,
>
> TAY wee-beng
>
>
> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>
> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>> Hi,
>>
>> Do you mean my method is wrong?
>>
>> I am following the template of ex22f,
>>
>> where the variables are declared as :
>>
>> PetscScalar  v(5)
>>
>> MatStencil   row(4),col(4,5)
>>
>> Hence,
>>
>> for the neumann BC
>>
>> num = 1
>>
>>                if (j/=0) then
>>
>>                    v(num) = -rho*HxdHy
>>
>>                    col(MatStencil_i,num) = i
>>
>>                    col(MatStencil_j,num) = j-1
>>
>>                    num = num + 1
>>
>>                end if
>>
>>                if (i/=0) then
>>
>>                    v(num) = -rho*HydHx
>>
>>                    col(MatStencil_i,num) = i-1
>>
>>                    col(MatStencil_j,num) = j
>>
>>                    num = num + 1
>>
>>                end if
>>
>>                if (i/=mx-1) then
>>
>>                    v(num) = -rho*HydHx
>>
>>                    col(MatStencil_i,num) = i+1
>>
>>                    col(MatStencil_j,num) = j
>>
>>                    num = num + 1
>>
>>                end if
>>
>>                if (j/=my-1) then
>>
>>                    v(num) = -rho*HxdHy
>>
>>                    col(MatStencil_i,num) = i
>>
>>                    col(MatStencil_j,num) = j+1
>>
>>                    num = num + 1
>>
>>                end if
>>
>>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>
>>                print *, v
>>
>>                col(MatStencil_i,num) = i
>>
>>                col(MatStencil_j,num) = j
>>
>>                !num = num + 1
>>
>>                call
>> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>
>> I do not get any more out of range error. However,my ans is still
>> different from that of ex29 in C.
>>
>
>  This is very simple. You have an error in your code. Checking it is very
> simple: run the code and
> break in MatSetValues(). Make sure ex29 makes calls with exactly the same
> indices as your ex29f.
>
>      Matt
>
>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120502/4f7a1ff0/attachment-0001.htm>

From zonexo at gmail.com  Wed May  2 15:42:39 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Wed, 02 May 2012 22:42:39 +0200
Subject: [petsc-users] Problem with fortran version of ex29 in ksp
In-Reply-To: <CAMYG4GkSc4dzoavG8=jM7GNo53c0Vdmumn4NEj2JacxxdLtw7Q@mail.gmail.com>
References: <4F97E8F2.5050803@gmail.com> <4F993F1D.9020105@gmail.com>
	<CAMYG4GnHyb1VKSY3gwpTxyU=xe1ZcEhitKCu5+CpK9a1v0ymuA@mail.gmail.com>
	<4F99A15B.1010500@gmail.com>
	<CAMYG4Gmzkpt+UyYLbzhit9Jr1G6SO99C9qKp+qg6Nyd__rTo3g@mail.gmail.com>
	<4F99CA4C.1080609@gmail.com>
	<C92D4D80-A685-4196-BFF6-85E4CB795D04@mcs.anl.gov>
	<4FA05A23.7080209@gmail.com>
	<CAMYG4Gk-vSa_kwRnp58up8cu7YPDN53Lb03JQviB+nduJSy9KQ@mail.gmail.com>
	<4FA17527.6030100@gmail.com>
	<CAMYG4GkSc4dzoavG8=jM7GNo53c0Vdmumn4NEj2JacxxdLtw7Q@mail.gmail.com>
Message-ID: <4FA19C3F.5090604@gmail.com>


On 2/5/2012 10:11 PM, Matthew Knepley wrote:
> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I did a MatView and VecView on both C and Fortran, right after Mat
>     and Vec assembly. I have attached the printout below. They are
>     exactly the same, but yet the result is different in Neumann
>     condition. However, the dirichlet condition gives the correct ans.
>     Is there anything else that could be wrong even if the Mat and Vec
>     are the same?
>
>
> Did you set the null space for the matrix when you have Neumann 
> conditions?
Yes, for the matrix, I set as:

call 
MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)

     call MatSetNullSpace(jac,nullspace,ierr)

     call MatNullSpaceDestroy(nullspace,ierr)

for the Vec,

call 
MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)

*!call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)*

     call MatNullSpaceDestroy(nullspace,ierr)

MatNullSpaceRemove was comment out because there's error during linking


>
>    Matt
>
>     Thanks!
>
>     Fortran:
>
>     Matrix Object: 1 MPI processes
>       type: seqaij
>     row 0: (0, 2)  (1, -1)  (3, -1)
>     row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>     row 2: (1, -1)  (2, 2)  (5, -1)
>     row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>     row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>     row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>     row 6: (3, -1)  (6, 2)  (7, -1)
>     row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>     row 8: (5, -1)  (7, -1)  (8, 2)
>     Vector Object:Vec_0000000084000000_0 1 MPI processes
>       type: mpi
>     Process [0]
>     0.25
>     0.0205213
>     1.135e-005
>     0.0205213
>     0.00168449
>     9.31663e-007
>     1.135e-005
>     9.31663e-007
>     5.15289e-010
>     Vector Object:Vec_0000000084000000_1 1 MPI processes
>       type: mpi
>     Process [0]
>     0.14924
>     0.0242397
>     -0.0260347
>     0.0242397
>     -0.0256192
>     -0.0400102
>     -0.0260347
>     -0.0400102
>     -0.0400102
>     Press any key to continue . . .
>
>     C:
>
>     Matrix Object: 1 MPI processes
>       type: seqaij
>     row 0: (0, 2)  (1, -1)  (3, -1)
>     row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>     row 2: (1, -1)  (2, 2)  (5, -1)
>     row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>     row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>     row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>     row 6: (3, -1)  (6, 2)  (7, -1)
>     row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>     row 8: (5, -1)  (7, -1)  (8, 2)
>     Vector Object:Vec_0x1d3b000_0 1 MPI processes
>       type: mpi
>     Process [0]
>     0.25
>     0.0205212
>     1.135e-05
>     0.0205212
>     0.00168449
>     9.31663e-07
>     1.135e-05
>     9.31663e-07
>     5.15288e-10
>     Vector Object:Vec_0x1d3b000_1 1 MPI processes
>       type: mpi
>     Process [0]
>     0.139311
>     0.0305751
>     -0.0220633
>     0.0305751
>     -0.0135158
>     -0.042185
>     -0.0220633
>     -0.042185
>     -0.058449
>
>
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>     On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>     On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         Do you mean my method is wrong?
>>
>>         I am following the template of ex22f,
>>
>>         where the variables are declared as :
>>
>>         PetscScalar  v(5)
>>
>>         MatStencil   row(4),col(4,5)
>>
>>         Hence,
>>
>>         for the neumann BC
>>
>>         num = 1
>>
>>                        if (j/=0) then
>>
>>                            v(num) = -rho*HxdHy
>>
>>                            col(MatStencil_i,num) = i
>>
>>                            col(MatStencil_j,num) = j-1
>>
>>                            num = num + 1
>>
>>                        end if
>>
>>                        if (i/=0) then
>>
>>                            v(num) = -rho*HydHx
>>
>>                            col(MatStencil_i,num) = i-1
>>
>>                            col(MatStencil_j,num) = j
>>
>>                            num = num + 1
>>
>>                        end if
>>
>>                        if (i/=mx-1) then
>>
>>                            v(num) = -rho*HydHx
>>
>>                            col(MatStencil_i,num) = i+1
>>
>>                            col(MatStencil_j,num) = j
>>
>>                            num = num + 1
>>
>>                        end if
>>
>>                        if (j/=my-1) then
>>
>>                            v(num) = -rho*HxdHy
>>
>>                            col(MatStencil_i,num) = i
>>
>>                            col(MatStencil_j,num) = j+1
>>
>>                            num = num + 1
>>
>>                        end if
>>
>>                        v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>
>>                        print *, v
>>
>>                        col(MatStencil_i,num) = i
>>
>>                        col(MatStencil_j,num) = j
>>
>>                        !num = num + 1
>>
>>                        call
>>         MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>
>>         I do not get any more out of range error. However,my ans is
>>         still different from that of ex29 in C.
>>
>>
>>     This is very simple. You have an error in your code. Checking it
>>     is very simple: run the code and
>>     break in MatSetValues(). Make sure ex29 makes calls with exactly
>>     the same indices as your ex29f.
>>
>>         Matt
>>
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120502/a845553f/attachment.htm>

From bin.gao at uit.no  Wed May  2 18:55:32 2012
From: bin.gao at uit.no (Gao Bin)
Date: Wed, 2 May 2012 23:55:32 +0000
Subject: [petsc-users] fill matrix using dynamic load balancing code
Message-ID: <CCA7D73E69B27E4AB6075A96C35D44394777B787@MBS04.uit.no>

Hi, all

(1) I am using PETSc to perform, for instance matrix-matrix multiplication. But the numerical values of the matrix are calculated from my integral code. I guess, for matrix-matrix multiplication, it might be better to distribute the rows of matrix uniformly on processors, right?

(2) If yes, I have a problem in my integral code. Since I need dynamic load balancing to calculate the matrix elements and then put them into correct places by using MatSetValues. But I have two questions regarding MatSetValues:

(2.1) If one processor calculated a lot of elements belong to another remote processor, will this become a problem for the performance? Should I make all processors calculate as many local elements as possible?

(2.2) From the manual, I see MatSetValues only caches the values, but MatAssemblyBegin() and MatAssemblyEnd() assembles the matrix. Therefore, is there any memory limit for MatSetValues, or in other words, will it cost a lot of memory before assembling?

Thank you in advance.

Cheers

Gao
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120502/786b9c56/attachment.htm>

From knepley at gmail.com  Wed May  2 19:56:30 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 2 May 2012 20:56:30 -0400
Subject: [petsc-users] fill matrix using dynamic load balancing code
In-Reply-To: <CCA7D73E69B27E4AB6075A96C35D44394777B787@MBS04.uit.no>
References: <CCA7D73E69B27E4AB6075A96C35D44394777B787@MBS04.uit.no>
Message-ID: <CAMYG4Gm7d+woogxnjDUMigiY13FYT1R3qcX6iikabkQ7L13zAw@mail.gmail.com>

On Wed, May 2, 2012 at 7:55 PM, Gao Bin <bin.gao at uit.no> wrote:

>  Hi, all
>
> (1) I am using PETSc to perform, for instance matrix-matrix
> multiplication. But the numerical values of the matrix are calculated from
> my integral code. I guess, for matrix-matrix multiplication, it might be
> better to distribute the rows of matrix uniformly on processors, right?
>

It depends on the matrix.


> (2) If yes, I have a problem in my integral code. Since I need dynamic
> load balancing to calculate the matrix elements and then put them into
> correct places by using MatSetValues. But I have two questions regarding
> MatSetValues:
>
> (2.1) If one processor calculated a lot of elements belong to another
> remote processor, will this become a problem for the performance? Should I
> make all processors calculate as many local elements as possible?
>

It will raise the communication cost. This, of course, must be weighed
against the penalty for load imbalance in your case.


> (2.2) From the manual, I see MatSetValues only caches the values, but
> MatAssemblyBegin() and MatAssemblyEnd() assembles the matrix. Therefore, is
> there any memory limit for MatSetValues, or in other words, will it cost a
> lot of memory before assembling?
>

Yes, there is a limit to the stash space:

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatStashSetInitialSize.html

    Matt


> Thank you in advance.
>
> Cheers
>
> Gao
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120502/5c729061/attachment.htm>

From ansayre at sbcglobal.net  Wed May  2 21:45:59 2012
From: ansayre at sbcglobal.net (Alan Sayre)
Date: Wed, 2 May 2012 19:45:59 -0700 (PDT)
Subject: [petsc-users] application of petsc to psr
Message-ID: <1336013159.64482.YahooMailRC@web181501.mail.ne1.yahoo.com>

I'm new to using PETSc. I need to model perfectly stirred reactors for chemical 
kinetics (10-1000 species), I welcome recommendations on how to get 
"up-to-speed". I also need to incorprate solution of the energy (temperature) 
equation and moments of soot distributions.

Thanks for any suggestions.

Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120502/948a6a37/attachment-0001.htm>

From bsmith at mcs.anl.gov  Wed May  2 21:54:14 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 2 May 2012 21:54:14 -0500
Subject: [petsc-users] application of petsc to psr
In-Reply-To: <1336013159.64482.YahooMailRC@web181501.mail.ne1.yahoo.com>
References: <1336013159.64482.YahooMailRC@web181501.mail.ne1.yahoo.com>
Message-ID: <DF2472D2-33FE-4876-825E-118EC9BB031A@mcs.anl.gov>


http://www.sandia.gov/chemkin/workshop/proceedings/Meeks_CHEMKINWorkshop2006.pdf  ?


On May 2, 2012, at 9:45 PM, Alan Sayre wrote:

> I'm new to using PETSc. I need to model perfectly stirred reactors for chemical kinetics (10-1000 species), I welcome recommendations on how to get "up-to-speed". I also need to incorprate solution of the energy (temperature) equation and moments of soot distributions.
>  
> Thanks for any suggestions.
>  
> Alan


From dkarthik at stanford.edu  Wed May  2 22:21:57 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Wed, 2 May 2012 20:21:57 -0700 (PDT)
Subject: [petsc-users] Multigrid
In-Reply-To: <23ADD932-4C45-421A-B401-F929C8B0CEB5@mcs.anl.gov>
Message-ID: <1758429860.101898369.1336015317357.JavaMail.root@zm09.stanford.edu>

Fair enough. I wiped out all references to petsc in my code and started from scratch and used just the options you mentioned.. and I get good convergence as first reported in the PCMG reference! I have attached the output. Now that it is behaving, could you recommend some options to exercise the multigrid?

Thanks a lot for your time (and patience!)

----------------------

KSP Object: 8 MPI processes
  type: gmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=1, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=1e+10
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 8 MPI processes
  type: bjacobi
    block Jacobi: number of blocks = 8
    Local solve is same for all blocks, in the following KSP and PC objects:
  KSP Object:  (sub_)   1 MPI processes  
    type: preonly
    maximum iterations=10000, initial guess is zero
    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
    left preconditioning
    using NONE norm type for convergence test
  PC Object:  (sub_)   1 MPI processes  
    type: ilu
      ILU: out-of-place factorization
      0 levels of fill
      tolerance for zero pivot 1e-12
      using diagonal shift to prevent zero pivot
      matrix ordering: natural
      factor fill ratio given 1, needed 1
        Factored matrix follows:
          Matrix Object:           1 MPI processes          
            type: seqaij
            rows=9015, cols=9015
            package used to perform factorization: petsc
            total: nonzeros=517777, allocated nonzeros=517777
            total number of mallocs used during MatSetValues calls =0
              using I-node routines: found 3476 nodes, limit used is 5
    linear system matrix = precond matrix:
    Matrix Object:     1 MPI processes    
      type: seqaij
      rows=9015, cols=9015
      total: nonzeros=517777, allocated nonzeros=517777
      total number of mallocs used during MatSetValues calls =0
        using I-node routines: found 3476 nodes, limit used is 5
  linear system matrix = precond matrix:
  Matrix Object:   8 MPI processes  
    type: mpiaij
    rows=75000, cols=75000
    total: nonzeros=4427800, allocated nonzeros=4427800
    total number of mallocs used during MatSetValues calls =0
      using I-node (on process 0) routines: found 3476 nodes, limit used is 5




----- Original Message -----
From: "Barry Smith" <bsmith at mcs.anl.gov>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Wednesday, May 2, 2012 10:34:17 AM
Subject: Re: [petsc-users] Multigrid


   Run with -pc_type ml -ksp_type fgmres   -ksp_max_it 100 -ksp_view   and NOTHING else. Don'ts set any KSP or PC options in the code!  Then send us ALL the output.

    Barry

  I am getting all confused with "I ran with these options and this happened, I ran with these options and this happened, I ran with these options and this happened". I am getting information overload with too much information and yet not enough information.


On May 2, 2012, at 12:27 PM, Karthik Duraisamy wrote:

> Hello,
> 
>    With PCML, I tried 
> 
>    -mg_coarse_pc_type  redundant ; -mg_coarse_ksp_type preonly ;   KSPSetType(ksp, KSPGMRES);
> 
> and the residual still diverges after a while. Strangely, if I use PCMG (with 0 levels), it works OK, but not with PCML.
> 
> This is definitely strange. Here is the entire piece of relevant code.
> 
> When initializing Petsc, I do the following: (let's call this the first bit and I do this only once) 
> 
>    KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>    KSPSetInitialGuessKnoll(ksp, PETSC_TRUE);
>    KSPSetType(ksp, KSPFGMRES);
>    KSPGMRESSetRestart(ksp, 100);
>    KSPSetFromOptions(ksp);
> 
> 
> Then for each outer solver iteration: (let's call this the second bit and I do this every outer iteration)
> 
>    PC pc;
>    KSPGetPC(ksp,&pc);
>    PCSetType(pc, PCMG);
>    KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>    PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>    KSPSetType(ksp, KSPGMRES);
>    KSPGMRESSetRestart(ksp, 100);
>    KSPSetFromOptions(ksp);
> 
> 
> I have tried various combinations of preconditioners and smoothers (including fgmres, bcgsl, etc) but the only combination that works is the above. Any of the PCML options that I have tried failed. 
> 
> Notes:
> - If I keep both of the above bits, it converges very well
> 
> - If I keep both of the above bits, but replace PCMG with PCML, it diverges quickly
> 
> - If I removed the first bit and kept the second bit, the residuals diverge even with PCMG 
> 
> - If I keep the first bit and remove the second bit, the residuals diverge eventually, but at a slow rate
> 
> 
> I am very confident with the rest of the code as I've been using PETSc (with just the first bit) for 3 years on 100s of problems.
> 
> Regards,
> Karthik
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Wednesday, May 2, 2012 6:02:06 AM
> Subject: Re: [petsc-users] Multigrid
> 
> 
>   This is likely helping a great deal:   
> 
>  KSPGMRESSetRestart(ksp, 100);
> 
>   You can try this with ml also
> 
> On May 2, 2012, at 1:33 AM, Karthik Duraisamy wrote:
> 
>> Hello again,
>> 
>>     I played with BoomerAMG and ML (for the 2D problem with a reasonable condition number), and so far, they haven't worked (the residual eventually diverges). The options that I am using are (for instance)
>> 
>>   PC pc; 
>>   KSPGetPC(ksp,&pc);
>>   PetscOptionsSetValue("-ksp_type", "gmres");
> 
>   Here you NEED  fgmres since the smoother is gmres by default. You can see this with -ksp_view
> 
>>   KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>   PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>   PCSetType(pc, PCML);
>>   PetscOptionsSetValue("-pc_ml_maxNlevels", "3");
>>   PetscOptionsSetValue("-mg_coarse_ksp_type","richardson");
>>   PetscOptionsSetValue("-mg_coarse_pc_type","sor");
>>   PetscOptionsSetValue("-mg_coarse_pc_sor_its","8");
> 
> Just use -mg_coarse_pc_type  redundant -mg_coarse_ksp_type preonly instead of the above.
> 
> 
>   Barry
> 
>>   PetscOptionsSetValue("-ksp_monitor","1");
>>   KSPSetFromOptions(ksp);
> 
> 
>> 
>> I have replaced KSP with bcgsl, fgmres, etc.
>> 
>> As I mentioned earlier, the following works like a charm:
>> 
>>   PC pc; 
>>   KSPGetPC(ksp,&pc);
>>   PCSetType(pc, PCMG);
>>   KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>   PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>   KSPSetType(ksp, KSPGMRES);
>>   KSPGMRESSetRestart(ksp, 100);
>>   KSPSetFromOptions(ksp);
>> 
>> 
>> Regards,
>> Karthik.
>> 
>> 
>> 
>> ----- Original Message -----
>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>> Sent: Tuesday, May 1, 2012 9:14:05 PM
>> Subject: Re: [petsc-users] Multigrid
>> 
>> 
>> On May 1, 2012, at 6:00 PM, Karthik Duraisamy wrote:
>> 
>>> So as I understand it, GMRES is used as a preconditioner and as a solver when I use PCMG with defaults. If this is the case, I should be able to recreate this set up without the PCMG. Any pointers as to how this can be done?
>>> 
>>> Also, yes indeed, my mesh is completely unstructured, so I will have to use ml or boomeramg. 
>>> 
>>> The problems that I am attempting involve RANS of compressible turbulent combustion (finite volume, steady). The high condition numbers are because of the extreme grid stretching and stiff source terms (in the turbulence and combustion model).
>> 
>>  With a condition number like that I wouldn't even consider using double precision, I think you would just be computing meaningless noise. With petsc-dev you can use quad precision ./configure --with-precision=__float128 using recent GNU compilers
>> (no we haven't tested the gfortran sides of things). 
>> 
>>  Barry
>> 
>> 
>>> I have been trying a reduced problem in these 2D test cases, in which the condition number is only 1e7.
>>> 
>>> Thanks,
>>> Karthik.
>>> 
>>> 
>>> ----- Original Message -----
>>> From: "Mark F. Adams" <mark.adams at columbia.edu>
>>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>>> Sent: Tuesday, May 1, 2012 3:50:31 PM
>>> Subject: Re: [petsc-users] Multigrid
>>> 
>>> 
>>> Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ matrix. If you use regular grids (DA?) then PETSc can construct geometric multigrid coarse grid spaces, although I don't know if PCMG will construct these for you (I don't think it will and I can see from your output that PCMG just used one grid). 'ml', hypre' and 'gamg' (a native AMG solver) will do real AMG solvers for you. All three can work on a similar class of problems. 
>>> 
>>> 
>>> Also, you mention that you have a condition number of 1.e20. That is astronomical for such a small problem. How did you compute that number? Do you know where the ill-conditioning comes from? Is this an elliptic operator? 
>>> 
>>> 
>>> Mark 
>>> 
>>> 
>>> 
>>> 
>>> On May 1, 2012, at 6:31 PM, Matthew Knepley wrote: 
>>> 
>>> 
>>> 
>>> On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>> 
>>> 
>>> 
>>> The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow) 
>>> 
>>> 
>>> 
>>> Because the smoother IS the solver you were using before. Just like I said last time, what you are doing is 
>>> wrapping up the same solver you used before, sticking it in another GMRES loop, and only looking at the 
>>> outer loop. This has nothing to do with MG. 
>>> 
>>> 
>>> Matt 
>>> 
>>> 
>>> KSP Object: 8 MPI processes 
>>> type: gmres 
>>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>> GMRES: happy breakdown tolerance 1e-30 
>>> maximum iterations=1, initial guess is zero 
>>> using preconditioner applied to right hand side for initial guess 
>>> tolerances: relative=0.01, absolute=1e-08, divergence=1e+10 
>>> left preconditioning 
>>> using DEFAULT norm type for convergence test 
>>> PC Object: 8 MPI processes 
>>> type: mg 
>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>>> Cycles per PCApply=1 
>>> Not using Galerkin computed coarse grid matrices 
>>> Coarse grid solver -- level ------------------------------- 
>>> KSP Object: (mg_levels_0_) 8 MPI processes 
>>> type not yet set 
>>> maximum iterations=1, initial guess is zero 
>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>> left preconditioning 
>>> using DEFAULT norm type for convergence test 
>>> PC Object: (mg_levels_0_) 8 MPI processes 
>>> type not yet set 
>>> linear system matrix = precond matrix: 
>>> Matrix Object: 8 MPI processes 
>>> type: mpiaij 
>>> rows=75000, cols=75000 
>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>> 
>>> 
>>> ----- Original Message ----- 
>>> From: "Matthew Knepley" < knepley at gmail.com > 
>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>> Sent: Tuesday, May 1, 2012 3:22:56 PM 
>>> Subject: Re: [petsc-users] Multigrid 
>>> 
>>> 
>>> On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>> 
>>> 
>>> 
>>> Hello, 
>>> 
>>> Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same). 
>>> 
>>> 
>>> 
>>> Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be 
>>> an extra GMRES loop compared to the case without MG. 
>>> 
>>> 
>>> Matt 
>>> 
>>> 
>>> Regards, 
>>> Karthik 
>>> 
>>> KSP Object: 8 MPI processes 
>>> type: gmres 
>>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>> GMRES: happy breakdown tolerance 1e-30 
>>> maximum iterations=1 
>>> using preconditioner applied to right hand side for initial guess 
>>> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 
>>> left preconditioning 
>>> using nonzero initial guess 
>>> using PRECONDITIONED norm type for convergence test 
>>> PC Object: 8 MPI processes 
>>> type: bjacobi 
>>> block Jacobi: number of blocks = 8 
>>> Local solve is same for all blocks, in the following KSP and PC objects: 
>>> KSP Object: (sub_) 1 MPI processes 
>>> type: preonly 
>>> maximum iterations=10000, initial guess is zero 
>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>> left preconditioning 
>>> using NONE norm type for convergence test 
>>> PC Object: (sub_) 1 MPI processes 
>>> type: ilu 
>>> ILU: out-of-place factorization 
>>> 0 levels of fill 
>>> tolerance for zero pivot 1e-12 
>>> using diagonal shift to prevent zero pivot 
>>> matrix ordering: natural 
>>> factor fill ratio given 1, needed 1 
>>> Factored matrix follows: 
>>> Matrix Object: 1 MPI processes 
>>> type: seqaij 
>>> rows=9015, cols=9015 
>>> package used to perform factorization: petsc 
>>> total: nonzeros=517777, allocated nonzeros=517777 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node routines: found 3476 nodes, limit used is 5 
>>> linear system matrix = precond matrix: 
>>> Matrix Object: 1 MPI processes 
>>> type: seqaij 
>>> rows=9015, cols=9015 
>>> total: nonzeros=517777, allocated nonzeros=517777 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node routines: found 3476 nodes, limit used is 5 
>>> linear system matrix = precond matrix: 
>>> Matrix Object: 8 MPI processes 
>>> type: mpiaij 
>>> rows=75000, cols=75000 
>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>> 
>>> 
>>> ----- Original Message ----- 
>>> From: "Matthew Knepley" < knepley at gmail.com > 
>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>> Sent: Tuesday, May 1, 2012 3:15:14 PM 
>>> Subject: Re: [petsc-users] Multigrid 
>>> 
>>> 
>>> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>> 
>>> 
>>> 
>>> Hello Barry, 
>>> 
>>> Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 
>>> 
>>> 
>>> 
>>> You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 
>>> 
>>> 
>>> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 
>>> 
>>> 
>>> 
>>> They are different algorithms. Its not possible to say generally that one is better. Try them both. 
>>> 
>>> 
>>> Matt 
>>> 
>>> 
>>> Best regards, 
>>> Karthik. 
>>> 
>>> type: mg 
>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>>> Cycles per PCApply=1 
>>> Not using Galerkin computed coarse grid matrices 
>>> Coarse grid solver -- level ------------------------------- 
>>> KSP Object: (mg_levels_0_) 8 MPI processes 
>>> type: gmres 
>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>> GMRES: happy breakdown tolerance 1e-30 
>>> maximum iterations=1, initial guess is zero 
>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>> left preconditioning 
>>> using PRECONDITIONED norm type for convergence test 
>>> PC Object: (mg_levels_0_) 8 MPI processes 
>>> type: bjacobi 
>>> block Jacobi: number of blocks = 8 
>>> Local solve is same for all blocks, in the following KSP and PC objects: 
>>> KSP Object: (mg_levels_0_sub_) 1 MPI processes 
>>> type: preonly 
>>> maximum iterations=10000, initial guess is zero 
>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>> left preconditioning 
>>> using NONE norm type for convergence test 
>>> PC Object: (mg_levels_0_sub_) 1 MPI processes 
>>> type: ilu 
>>> ILU: out-of-place factorization 
>>> 0 levels of fill 
>>> tolerance for zero pivot 1e-12 
>>> using diagonal shift to prevent zero pivot 
>>> matrix ordering: natural 
>>> factor fill ratio given 1, needed 1 
>>> Factored matrix follows: 
>>> Matrix Object: 1 MPI processes 
>>> type: seqaij 
>>> rows=9015, cols=9015 
>>> package used to perform factorization: petsc 
>>> total: nonzeros=517777, allocated nonzeros=517777 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node routines: found 3476 nodes, limit used is 5 
>>> linear system matrix = precond matrix: 
>>> Matrix Object: 1 MPI processes 
>>> type: seqaij 
>>> rows=9015, cols=9015 
>>> total: nonzeros=517777, allocated nonzeros=517777 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node routines: found 3476 nodes, limit used is 5 
>>> linear system matrix = precond matrix: 
>>> Matrix Object: 8 MPI processes 
>>> type: mpiaij 
>>> rows=75000, cols=75000 
>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>> linear system matrix = precond matrix: 
>>> Matrix Object: 8 MPI processes 
>>> type: mpiaij 
>>> rows=75000, cols=75000 
>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>> total number of mallocs used during MatSetValues calls =0 
>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>> 
>>> 
>>> 
>>> ----- Original Message ----- 
>>> From: "Barry Smith" < bsmith at mcs.anl.gov > 
>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>> Sent: Tuesday, May 1, 2012 1:39:26 PM 
>>> Subject: Re: [petsc-users] Multigrid 
>>> 
>>> 
>>> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 
>>> 
>>>> Hello, 
>>>> 
>>>> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
>>>> 
>>>> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
>>>> help in this regard will be appreciated. 
>>> 
>>> First run with -ksp_view to see what solver it is actually using. 
>>> 
>>> Barry 
>>> 
>>>> 
>>>> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
>>>> 
>>>> Thanks, 
>>>> Karthik 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> 
>>>> ======================================= 
>>>> Karthik Duraisamy 
>>>> Assistant Professor (Consulting) 
>>>> Durand Building Rm 357 
>>>> Dept of Aeronautics and Astronautics 
>>>> Stanford University 
>>>> Stanford CA 94305 
>>>> 
>>>> Phone: 650-721-2835 
>>>> Web: www.stanford.edu/~dkarthik 
>>>> ======================================= 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>> -- Norbert Wiener 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>> -- Norbert Wiener 
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>> -- Norbert Wiener 
>>> 
>> 
> 


From sciam at iastate.edu  Wed May  2 23:32:48 2012
From: sciam at iastate.edu (Bo Sun)
Date: Wed, 2 May 2012 22:32:48 -0600
Subject: [petsc-users] extract data from vector in petsc into fortran array
Message-ID: <CA+5XtAUjevavzs5LT0WAZrvYQYK5xUZFbhZRknMvmKtRdh2SEA@mail.gmail.com>

Hi there,
I am coding using Fortran 90 with petsc. I extract the results from vector
in petsc as follows,

   PetscScalar, pointer :: xx_v(:)
   Vec pet_u

   ..............
   call VecGetArrayF90(pet_u,xx_v,ierr)
    data_for_fortran = xx_v
    call VecRestoreArrayF90(pet_u,xx_v,ierr)
   ...................


For small size of vector, there is no problem. But when the size of vector
is above million, some strange data appear. Could you recommend another way
to output the data of vector in petsc into fortran array ?


Thanks,
Bo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120502/e81782e5/attachment.htm>

From bsmith at mcs.anl.gov  Wed May  2 23:56:05 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 2 May 2012 23:56:05 -0500
Subject: [petsc-users] Multigrid
In-Reply-To: <1758429860.101898369.1336015317357.JavaMail.root@zm09.stanford.edu>
References: <1758429860.101898369.1336015317357.JavaMail.root@zm09.stanford.edu>
Message-ID: <2C2DF8CD-7A12-451E-B50E-94A83ABFEC65@mcs.anl.gov>


   Make the run I suggested -pc_type ml -ksp_type fgmres   -ksp_max_it 100 -ksp_view   

  Barry

On May 2, 2012, at 10:21 PM, Karthik Duraisamy wrote:

> Fair enough. I wiped out all references to petsc in my code and started from scratch and used just the options you mentioned.. and I get good convergence as first reported in the PCMG reference! I have attached the output. Now that it is behaving, could you recommend some options to exercise the multigrid?
> 
> Thanks a lot for your time (and patience!)
> 
> ----------------------
> 
> KSP Object: 8 MPI processes
>  type: gmres
>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>    GMRES: happy breakdown tolerance 1e-30
>  maximum iterations=1, initial guess is zero
>  tolerances:  relative=1e-05, absolute=1e-50, divergence=1e+10
>  left preconditioning
>  using PRECONDITIONED norm type for convergence test
> PC Object: 8 MPI processes
>  type: bjacobi
>    block Jacobi: number of blocks = 8
>    Local solve is same for all blocks, in the following KSP and PC objects:
>  KSP Object:  (sub_)   1 MPI processes  
>    type: preonly
>    maximum iterations=10000, initial guess is zero
>    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>    left preconditioning
>    using NONE norm type for convergence test
>  PC Object:  (sub_)   1 MPI processes  
>    type: ilu
>      ILU: out-of-place factorization
>      0 levels of fill
>      tolerance for zero pivot 1e-12
>      using diagonal shift to prevent zero pivot
>      matrix ordering: natural
>      factor fill ratio given 1, needed 1
>        Factored matrix follows:
>          Matrix Object:           1 MPI processes          
>            type: seqaij
>            rows=9015, cols=9015
>            package used to perform factorization: petsc
>            total: nonzeros=517777, allocated nonzeros=517777
>            total number of mallocs used during MatSetValues calls =0
>              using I-node routines: found 3476 nodes, limit used is 5
>    linear system matrix = precond matrix:
>    Matrix Object:     1 MPI processes    
>      type: seqaij
>      rows=9015, cols=9015
>      total: nonzeros=517777, allocated nonzeros=517777
>      total number of mallocs used during MatSetValues calls =0
>        using I-node routines: found 3476 nodes, limit used is 5
>  linear system matrix = precond matrix:
>  Matrix Object:   8 MPI processes  
>    type: mpiaij
>    rows=75000, cols=75000
>    total: nonzeros=4427800, allocated nonzeros=4427800
>    total number of mallocs used during MatSetValues calls =0
>      using I-node (on process 0) routines: found 3476 nodes, limit used is 5
> 
> 
> 
> 
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Wednesday, May 2, 2012 10:34:17 AM
> Subject: Re: [petsc-users] Multigrid
> 
> 
>   Run with -pc_type ml -ksp_type fgmres   -ksp_max_it 100 -ksp_view   and NOTHING else. Don'ts set any KSP or PC options in the code!  Then send us ALL the output.
> 
>    Barry
> 
>  I am getting all confused with "I ran with these options and this happened, I ran with these options and this happened, I ran with these options and this happened". I am getting information overload with too much information and yet not enough information.
> 
> 
> On May 2, 2012, at 12:27 PM, Karthik Duraisamy wrote:
> 
>> Hello,
>> 
>>   With PCML, I tried 
>> 
>>   -mg_coarse_pc_type  redundant ; -mg_coarse_ksp_type preonly ;   KSPSetType(ksp, KSPGMRES);
>> 
>> and the residual still diverges after a while. Strangely, if I use PCMG (with 0 levels), it works OK, but not with PCML.
>> 
>> This is definitely strange. Here is the entire piece of relevant code.
>> 
>> When initializing Petsc, I do the following: (let's call this the first bit and I do this only once) 
>> 
>>   KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>   KSPSetInitialGuessKnoll(ksp, PETSC_TRUE);
>>   KSPSetType(ksp, KSPFGMRES);
>>   KSPGMRESSetRestart(ksp, 100);
>>   KSPSetFromOptions(ksp);
>> 
>> 
>> Then for each outer solver iteration: (let's call this the second bit and I do this every outer iteration)
>> 
>>   PC pc;
>>   KSPGetPC(ksp,&pc);
>>   PCSetType(pc, PCMG);
>>   KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>   PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>   KSPSetType(ksp, KSPGMRES);
>>   KSPGMRESSetRestart(ksp, 100);
>>   KSPSetFromOptions(ksp);
>> 
>> 
>> I have tried various combinations of preconditioners and smoothers (including fgmres, bcgsl, etc) but the only combination that works is the above. Any of the PCML options that I have tried failed. 
>> 
>> Notes:
>> - If I keep both of the above bits, it converges very well
>> 
>> - If I keep both of the above bits, but replace PCMG with PCML, it diverges quickly
>> 
>> - If I removed the first bit and kept the second bit, the residuals diverge even with PCMG 
>> 
>> - If I keep the first bit and remove the second bit, the residuals diverge eventually, but at a slow rate
>> 
>> 
>> I am very confident with the rest of the code as I've been using PETSc (with just the first bit) for 3 years on 100s of problems.
>> 
>> Regards,
>> Karthik
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> ----- Original Message -----
>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>> Sent: Wednesday, May 2, 2012 6:02:06 AM
>> Subject: Re: [petsc-users] Multigrid
>> 
>> 
>>  This is likely helping a great deal:   
>> 
>> KSPGMRESSetRestart(ksp, 100);
>> 
>>  You can try this with ml also
>> 
>> On May 2, 2012, at 1:33 AM, Karthik Duraisamy wrote:
>> 
>>> Hello again,
>>> 
>>>    I played with BoomerAMG and ML (for the 2D problem with a reasonable condition number), and so far, they haven't worked (the residual eventually diverges). The options that I am using are (for instance)
>>> 
>>>  PC pc; 
>>>  KSPGetPC(ksp,&pc);
>>>  PetscOptionsSetValue("-ksp_type", "gmres");
>> 
>>  Here you NEED  fgmres since the smoother is gmres by default. You can see this with -ksp_view
>> 
>>>  KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>>  PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>>  PCSetType(pc, PCML);
>>>  PetscOptionsSetValue("-pc_ml_maxNlevels", "3");
>>>  PetscOptionsSetValue("-mg_coarse_ksp_type","richardson");
>>>  PetscOptionsSetValue("-mg_coarse_pc_type","sor");
>>>  PetscOptionsSetValue("-mg_coarse_pc_sor_its","8");
>> 
>> Just use -mg_coarse_pc_type  redundant -mg_coarse_ksp_type preonly instead of the above.
>> 
>> 
>>  Barry
>> 
>>>  PetscOptionsSetValue("-ksp_monitor","1");
>>>  KSPSetFromOptions(ksp);
>> 
>> 
>>> 
>>> I have replaced KSP with bcgsl, fgmres, etc.
>>> 
>>> As I mentioned earlier, the following works like a charm:
>>> 
>>>  PC pc; 
>>>  KSPGetPC(ksp,&pc);
>>>  PCSetType(pc, PCMG);
>>>  KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>>  PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>>  KSPSetType(ksp, KSPGMRES);
>>>  KSPGMRESSetRestart(ksp, 100);
>>>  KSPSetFromOptions(ksp);
>>> 
>>> 
>>> Regards,
>>> Karthik.
>>> 
>>> 
>>> 
>>> ----- Original Message -----
>>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>>> Sent: Tuesday, May 1, 2012 9:14:05 PM
>>> Subject: Re: [petsc-users] Multigrid
>>> 
>>> 
>>> On May 1, 2012, at 6:00 PM, Karthik Duraisamy wrote:
>>> 
>>>> So as I understand it, GMRES is used as a preconditioner and as a solver when I use PCMG with defaults. If this is the case, I should be able to recreate this set up without the PCMG. Any pointers as to how this can be done?
>>>> 
>>>> Also, yes indeed, my mesh is completely unstructured, so I will have to use ml or boomeramg. 
>>>> 
>>>> The problems that I am attempting involve RANS of compressible turbulent combustion (finite volume, steady). The high condition numbers are because of the extreme grid stretching and stiff source terms (in the turbulence and combustion model).
>>> 
>>> With a condition number like that I wouldn't even consider using double precision, I think you would just be computing meaningless noise. With petsc-dev you can use quad precision ./configure --with-precision=__float128 using recent GNU compilers
>>> (no we haven't tested the gfortran sides of things). 
>>> 
>>> Barry
>>> 
>>> 
>>>> I have been trying a reduced problem in these 2D test cases, in which the condition number is only 1e7.
>>>> 
>>>> Thanks,
>>>> Karthik.
>>>> 
>>>> 
>>>> ----- Original Message -----
>>>> From: "Mark F. Adams" <mark.adams at columbia.edu>
>>>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>>>> Sent: Tuesday, May 1, 2012 3:50:31 PM
>>>> Subject: Re: [petsc-users] Multigrid
>>>> 
>>>> 
>>>> Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ matrix. If you use regular grids (DA?) then PETSc can construct geometric multigrid coarse grid spaces, although I don't know if PCMG will construct these for you (I don't think it will and I can see from your output that PCMG just used one grid). 'ml', hypre' and 'gamg' (a native AMG solver) will do real AMG solvers for you. All three can work on a similar class of problems. 
>>>> 
>>>> 
>>>> Also, you mention that you have a condition number of 1.e20. That is astronomical for such a small problem. How did you compute that number? Do you know where the ill-conditioning comes from? Is this an elliptic operator? 
>>>> 
>>>> 
>>>> Mark 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On May 1, 2012, at 6:31 PM, Matthew Knepley wrote: 
>>>> 
>>>> 
>>>> 
>>>> On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>>> 
>>>> 
>>>> 
>>>> The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow) 
>>>> 
>>>> 
>>>> 
>>>> Because the smoother IS the solver you were using before. Just like I said last time, what you are doing is 
>>>> wrapping up the same solver you used before, sticking it in another GMRES loop, and only looking at the 
>>>> outer loop. This has nothing to do with MG. 
>>>> 
>>>> 
>>>> Matt 
>>>> 
>>>> 
>>>> KSP Object: 8 MPI processes 
>>>> type: gmres 
>>>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>>> GMRES: happy breakdown tolerance 1e-30 
>>>> maximum iterations=1, initial guess is zero 
>>>> using preconditioner applied to right hand side for initial guess 
>>>> tolerances: relative=0.01, absolute=1e-08, divergence=1e+10 
>>>> left preconditioning 
>>>> using DEFAULT norm type for convergence test 
>>>> PC Object: 8 MPI processes 
>>>> type: mg 
>>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>>>> Cycles per PCApply=1 
>>>> Not using Galerkin computed coarse grid matrices 
>>>> Coarse grid solver -- level ------------------------------- 
>>>> KSP Object: (mg_levels_0_) 8 MPI processes 
>>>> type not yet set 
>>>> maximum iterations=1, initial guess is zero 
>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>> left preconditioning 
>>>> using DEFAULT norm type for convergence test 
>>>> PC Object: (mg_levels_0_) 8 MPI processes 
>>>> type not yet set 
>>>> linear system matrix = precond matrix: 
>>>> Matrix Object: 8 MPI processes 
>>>> type: mpiaij 
>>>> rows=75000, cols=75000 
>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>> 
>>>> 
>>>> ----- Original Message ----- 
>>>> From: "Matthew Knepley" < knepley at gmail.com > 
>>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>>> Sent: Tuesday, May 1, 2012 3:22:56 PM 
>>>> Subject: Re: [petsc-users] Multigrid 
>>>> 
>>>> 
>>>> On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>>> 
>>>> 
>>>> 
>>>> Hello, 
>>>> 
>>>> Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same). 
>>>> 
>>>> 
>>>> 
>>>> Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be 
>>>> an extra GMRES loop compared to the case without MG. 
>>>> 
>>>> 
>>>> Matt 
>>>> 
>>>> 
>>>> Regards, 
>>>> Karthik 
>>>> 
>>>> KSP Object: 8 MPI processes 
>>>> type: gmres 
>>>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>>> GMRES: happy breakdown tolerance 1e-30 
>>>> maximum iterations=1 
>>>> using preconditioner applied to right hand side for initial guess 
>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 
>>>> left preconditioning 
>>>> using nonzero initial guess 
>>>> using PRECONDITIONED norm type for convergence test 
>>>> PC Object: 8 MPI processes 
>>>> type: bjacobi 
>>>> block Jacobi: number of blocks = 8 
>>>> Local solve is same for all blocks, in the following KSP and PC objects: 
>>>> KSP Object: (sub_) 1 MPI processes 
>>>> type: preonly 
>>>> maximum iterations=10000, initial guess is zero 
>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>> left preconditioning 
>>>> using NONE norm type for convergence test 
>>>> PC Object: (sub_) 1 MPI processes 
>>>> type: ilu 
>>>> ILU: out-of-place factorization 
>>>> 0 levels of fill 
>>>> tolerance for zero pivot 1e-12 
>>>> using diagonal shift to prevent zero pivot 
>>>> matrix ordering: natural 
>>>> factor fill ratio given 1, needed 1 
>>>> Factored matrix follows: 
>>>> Matrix Object: 1 MPI processes 
>>>> type: seqaij 
>>>> rows=9015, cols=9015 
>>>> package used to perform factorization: petsc 
>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>> linear system matrix = precond matrix: 
>>>> Matrix Object: 1 MPI processes 
>>>> type: seqaij 
>>>> rows=9015, cols=9015 
>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>> linear system matrix = precond matrix: 
>>>> Matrix Object: 8 MPI processes 
>>>> type: mpiaij 
>>>> rows=75000, cols=75000 
>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>> 
>>>> 
>>>> ----- Original Message ----- 
>>>> From: "Matthew Knepley" < knepley at gmail.com > 
>>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>>> Sent: Tuesday, May 1, 2012 3:15:14 PM 
>>>> Subject: Re: [petsc-users] Multigrid 
>>>> 
>>>> 
>>>> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>>> 
>>>> 
>>>> 
>>>> Hello Barry, 
>>>> 
>>>> Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 
>>>> 
>>>> 
>>>> 
>>>> You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 
>>>> 
>>>> 
>>>> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 
>>>> 
>>>> 
>>>> 
>>>> They are different algorithms. Its not possible to say generally that one is better. Try them both. 
>>>> 
>>>> 
>>>> Matt 
>>>> 
>>>> 
>>>> Best regards, 
>>>> Karthik. 
>>>> 
>>>> type: mg 
>>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>>>> Cycles per PCApply=1 
>>>> Not using Galerkin computed coarse grid matrices 
>>>> Coarse grid solver -- level ------------------------------- 
>>>> KSP Object: (mg_levels_0_) 8 MPI processes 
>>>> type: gmres 
>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>>> GMRES: happy breakdown tolerance 1e-30 
>>>> maximum iterations=1, initial guess is zero 
>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>> left preconditioning 
>>>> using PRECONDITIONED norm type for convergence test 
>>>> PC Object: (mg_levels_0_) 8 MPI processes 
>>>> type: bjacobi 
>>>> block Jacobi: number of blocks = 8 
>>>> Local solve is same for all blocks, in the following KSP and PC objects: 
>>>> KSP Object: (mg_levels_0_sub_) 1 MPI processes 
>>>> type: preonly 
>>>> maximum iterations=10000, initial guess is zero 
>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>> left preconditioning 
>>>> using NONE norm type for convergence test 
>>>> PC Object: (mg_levels_0_sub_) 1 MPI processes 
>>>> type: ilu 
>>>> ILU: out-of-place factorization 
>>>> 0 levels of fill 
>>>> tolerance for zero pivot 1e-12 
>>>> using diagonal shift to prevent zero pivot 
>>>> matrix ordering: natural 
>>>> factor fill ratio given 1, needed 1 
>>>> Factored matrix follows: 
>>>> Matrix Object: 1 MPI processes 
>>>> type: seqaij 
>>>> rows=9015, cols=9015 
>>>> package used to perform factorization: petsc 
>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>> linear system matrix = precond matrix: 
>>>> Matrix Object: 1 MPI processes 
>>>> type: seqaij 
>>>> rows=9015, cols=9015 
>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>> linear system matrix = precond matrix: 
>>>> Matrix Object: 8 MPI processes 
>>>> type: mpiaij 
>>>> rows=75000, cols=75000 
>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>> linear system matrix = precond matrix: 
>>>> Matrix Object: 8 MPI processes 
>>>> type: mpiaij 
>>>> rows=75000, cols=75000 
>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>> 
>>>> 
>>>> 
>>>> ----- Original Message ----- 
>>>> From: "Barry Smith" < bsmith at mcs.anl.gov > 
>>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>>> Sent: Tuesday, May 1, 2012 1:39:26 PM 
>>>> Subject: Re: [petsc-users] Multigrid 
>>>> 
>>>> 
>>>> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 
>>>> 
>>>>> Hello, 
>>>>> 
>>>>> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
>>>>> 
>>>>> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
>>>>> help in this regard will be appreciated. 
>>>> 
>>>> First run with -ksp_view to see what solver it is actually using. 
>>>> 
>>>> Barry 
>>>> 
>>>>> 
>>>>> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
>>>>> 
>>>>> Thanks, 
>>>>> Karthik 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> 
>>>>> ======================================= 
>>>>> Karthik Duraisamy 
>>>>> Assistant Professor (Consulting) 
>>>>> Durand Building Rm 357 
>>>>> Dept of Aeronautics and Astronautics 
>>>>> Stanford University 
>>>>> Stanford CA 94305 
>>>>> 
>>>>> Phone: 650-721-2835 
>>>>> Web: www.stanford.edu/~dkarthik 
>>>>> ======================================= 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>>> -- Norbert Wiener 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>>> -- Norbert Wiener 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>>> -- Norbert Wiener 
>>>> 
>>> 
>> 
> 


From bsmith at mcs.anl.gov  Thu May  3 00:01:14 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 3 May 2012 00:01:14 -0500
Subject: [petsc-users] extract data from vector in petsc into fortran
	array
In-Reply-To: <CA+5XtAUjevavzs5LT0WAZrvYQYK5xUZFbhZRknMvmKtRdh2SEA@mail.gmail.com>
References: <CA+5XtAUjevavzs5LT0WAZrvYQYK5xUZFbhZRknMvmKtRdh2SEA@mail.gmail.com>
Message-ID: <0A58C6E9-DD55-42B6-AB7F-768A07E5E7F8@mcs.anl.gov>


On May 2, 2012, at 11:32 PM, Bo Sun wrote:

> Hi there,
> I am coding using Fortran 90 with petsc. I extract the results from vector in petsc as follows,
>  
>    PetscScalar, pointer :: xx_v(:)
>    Vec pet_u 
>   
>    ..............
>    call VecGetArrayF90(pet_u,xx_v,ierr)  
>     data_for_fortran = xx_v
>     call VecRestoreArrayF90(pet_u,xx_v,ierr)
>    ...................
>    
> 
> For small size of vector, there is no problem. But when the size of vector is above million, some strange data appear.

   What do you mean strange data? Is this reproducable? Do two different Fortran compilers give the same problem? I you can send us a simple Fortran code that demonstrates the problem we can try to reproduce it. Please send such a bug report to petsc-maint at mcs.anl.gov

   VecGetArrayF90() is generally the way to go, other things are more cumbersome.
> 

  
> Could you recommend another way to output the data of vector in petsc into fortran array ? 
> 
> 
> Thanks,
> Bo  


From dkarthik at stanford.edu  Thu May  3 00:01:47 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Wed, 2 May 2012 22:01:47 -0700 (PDT)
Subject: [petsc-users] Multigrid
In-Reply-To: <2C2DF8CD-7A12-451E-B50E-94A83ABFEC65@mcs.anl.gov>
Message-ID: <604010753.102000523.1336021307180.JavaMail.root@zm09.stanford.edu>

Sorry.. I forgot to add that I tried just those options 

-pc_type ml -ksp_type fgmres   -ksp_max_it 100 -ksp_view 

to get the result that I got. I was asking for additional options. In any case, now I can try a few things

----- Original Message -----
From: "Barry Smith" <bsmith at mcs.anl.gov>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Wednesday, May 2, 2012 9:56:05 PM
Subject: Re: [petsc-users] Multigrid


   Make the run I suggested -pc_type ml -ksp_type fgmres   -ksp_max_it 100 -ksp_view   

  Barry

On May 2, 2012, at 10:21 PM, Karthik Duraisamy wrote:

> Fair enough. I wiped out all references to petsc in my code and started from scratch and used just the options you mentioned.. and I get good convergence as first reported in the PCMG reference! I have attached the output. Now that it is behaving, could you recommend some options to exercise the multigrid?
> 
> Thanks a lot for your time (and patience!)
> 
> ----------------------
> 
> KSP Object: 8 MPI processes
>  type: gmres
>    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>    GMRES: happy breakdown tolerance 1e-30
>  maximum iterations=1, initial guess is zero
>  tolerances:  relative=1e-05, absolute=1e-50, divergence=1e+10
>  left preconditioning
>  using PRECONDITIONED norm type for convergence test
> PC Object: 8 MPI processes
>  type: bjacobi
>    block Jacobi: number of blocks = 8
>    Local solve is same for all blocks, in the following KSP and PC objects:
>  KSP Object:  (sub_)   1 MPI processes  
>    type: preonly
>    maximum iterations=10000, initial guess is zero
>    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>    left preconditioning
>    using NONE norm type for convergence test
>  PC Object:  (sub_)   1 MPI processes  
>    type: ilu
>      ILU: out-of-place factorization
>      0 levels of fill
>      tolerance for zero pivot 1e-12
>      using diagonal shift to prevent zero pivot
>      matrix ordering: natural
>      factor fill ratio given 1, needed 1
>        Factored matrix follows:
>          Matrix Object:           1 MPI processes          
>            type: seqaij
>            rows=9015, cols=9015
>            package used to perform factorization: petsc
>            total: nonzeros=517777, allocated nonzeros=517777
>            total number of mallocs used during MatSetValues calls =0
>              using I-node routines: found 3476 nodes, limit used is 5
>    linear system matrix = precond matrix:
>    Matrix Object:     1 MPI processes    
>      type: seqaij
>      rows=9015, cols=9015
>      total: nonzeros=517777, allocated nonzeros=517777
>      total number of mallocs used during MatSetValues calls =0
>        using I-node routines: found 3476 nodes, limit used is 5
>  linear system matrix = precond matrix:
>  Matrix Object:   8 MPI processes  
>    type: mpiaij
>    rows=75000, cols=75000
>    total: nonzeros=4427800, allocated nonzeros=4427800
>    total number of mallocs used during MatSetValues calls =0
>      using I-node (on process 0) routines: found 3476 nodes, limit used is 5
> 
> 
> 
> 
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Wednesday, May 2, 2012 10:34:17 AM
> Subject: Re: [petsc-users] Multigrid
> 
> 
>   Run with -pc_type ml -ksp_type fgmres   -ksp_max_it 100 -ksp_view   and NOTHING else. Don'ts set any KSP or PC options in the code!  Then send us ALL the output.
> 
>    Barry
> 
>  I am getting all confused with "I ran with these options and this happened, I ran with these options and this happened, I ran with these options and this happened". I am getting information overload with too much information and yet not enough information.
> 
> 
> On May 2, 2012, at 12:27 PM, Karthik Duraisamy wrote:
> 
>> Hello,
>> 
>>   With PCML, I tried 
>> 
>>   -mg_coarse_pc_type  redundant ; -mg_coarse_ksp_type preonly ;   KSPSetType(ksp, KSPGMRES);
>> 
>> and the residual still diverges after a while. Strangely, if I use PCMG (with 0 levels), it works OK, but not with PCML.
>> 
>> This is definitely strange. Here is the entire piece of relevant code.
>> 
>> When initializing Petsc, I do the following: (let's call this the first bit and I do this only once) 
>> 
>>   KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>   KSPSetInitialGuessKnoll(ksp, PETSC_TRUE);
>>   KSPSetType(ksp, KSPFGMRES);
>>   KSPGMRESSetRestart(ksp, 100);
>>   KSPSetFromOptions(ksp);
>> 
>> 
>> Then for each outer solver iteration: (let's call this the second bit and I do this every outer iteration)
>> 
>>   PC pc;
>>   KSPGetPC(ksp,&pc);
>>   PCSetType(pc, PCMG);
>>   KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>   PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>   KSPSetType(ksp, KSPGMRES);
>>   KSPGMRESSetRestart(ksp, 100);
>>   KSPSetFromOptions(ksp);
>> 
>> 
>> I have tried various combinations of preconditioners and smoothers (including fgmres, bcgsl, etc) but the only combination that works is the above. Any of the PCML options that I have tried failed. 
>> 
>> Notes:
>> - If I keep both of the above bits, it converges very well
>> 
>> - If I keep both of the above bits, but replace PCMG with PCML, it diverges quickly
>> 
>> - If I removed the first bit and kept the second bit, the residuals diverge even with PCMG 
>> 
>> - If I keep the first bit and remove the second bit, the residuals diverge eventually, but at a slow rate
>> 
>> 
>> I am very confident with the rest of the code as I've been using PETSc (with just the first bit) for 3 years on 100s of problems.
>> 
>> Regards,
>> Karthik
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> ----- Original Message -----
>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>> Sent: Wednesday, May 2, 2012 6:02:06 AM
>> Subject: Re: [petsc-users] Multigrid
>> 
>> 
>>  This is likely helping a great deal:   
>> 
>> KSPGMRESSetRestart(ksp, 100);
>> 
>>  You can try this with ml also
>> 
>> On May 2, 2012, at 1:33 AM, Karthik Duraisamy wrote:
>> 
>>> Hello again,
>>> 
>>>    I played with BoomerAMG and ML (for the 2D problem with a reasonable condition number), and so far, they haven't worked (the residual eventually diverges). The options that I am using are (for instance)
>>> 
>>>  PC pc; 
>>>  KSPGetPC(ksp,&pc);
>>>  PetscOptionsSetValue("-ksp_type", "gmres");
>> 
>>  Here you NEED  fgmres since the smoother is gmres by default. You can see this with -ksp_view
>> 
>>>  KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>>  PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>>  PCSetType(pc, PCML);
>>>  PetscOptionsSetValue("-pc_ml_maxNlevels", "3");
>>>  PetscOptionsSetValue("-mg_coarse_ksp_type","richardson");
>>>  PetscOptionsSetValue("-mg_coarse_pc_type","sor");
>>>  PetscOptionsSetValue("-mg_coarse_pc_sor_its","8");
>> 
>> Just use -mg_coarse_pc_type  redundant -mg_coarse_ksp_type preonly instead of the above.
>> 
>> 
>>  Barry
>> 
>>>  PetscOptionsSetValue("-ksp_monitor","1");
>>>  KSPSetFromOptions(ksp);
>> 
>> 
>>> 
>>> I have replaced KSP with bcgsl, fgmres, etc.
>>> 
>>> As I mentioned earlier, the following works like a charm:
>>> 
>>>  PC pc; 
>>>  KSPGetPC(ksp,&pc);
>>>  PCSetType(pc, PCMG);
>>>  KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>>  PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>>  KSPSetType(ksp, KSPGMRES);
>>>  KSPGMRESSetRestart(ksp, 100);
>>>  KSPSetFromOptions(ksp);
>>> 
>>> 
>>> Regards,
>>> Karthik.
>>> 
>>> 
>>> 
>>> ----- Original Message -----
>>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>>> Sent: Tuesday, May 1, 2012 9:14:05 PM
>>> Subject: Re: [petsc-users] Multigrid
>>> 
>>> 
>>> On May 1, 2012, at 6:00 PM, Karthik Duraisamy wrote:
>>> 
>>>> So as I understand it, GMRES is used as a preconditioner and as a solver when I use PCMG with defaults. If this is the case, I should be able to recreate this set up without the PCMG. Any pointers as to how this can be done?
>>>> 
>>>> Also, yes indeed, my mesh is completely unstructured, so I will have to use ml or boomeramg. 
>>>> 
>>>> The problems that I am attempting involve RANS of compressible turbulent combustion (finite volume, steady). The high condition numbers are because of the extreme grid stretching and stiff source terms (in the turbulence and combustion model).
>>> 
>>> With a condition number like that I wouldn't even consider using double precision, I think you would just be computing meaningless noise. With petsc-dev you can use quad precision ./configure --with-precision=__float128 using recent GNU compilers
>>> (no we haven't tested the gfortran sides of things). 
>>> 
>>> Barry
>>> 
>>> 
>>>> I have been trying a reduced problem in these 2D test cases, in which the condition number is only 1e7.
>>>> 
>>>> Thanks,
>>>> Karthik.
>>>> 
>>>> 
>>>> ----- Original Message -----
>>>> From: "Mark F. Adams" <mark.adams at columbia.edu>
>>>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>>>> Sent: Tuesday, May 1, 2012 3:50:31 PM
>>>> Subject: Re: [petsc-users] Multigrid
>>>> 
>>>> 
>>>> Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ matrix. If you use regular grids (DA?) then PETSc can construct geometric multigrid coarse grid spaces, although I don't know if PCMG will construct these for you (I don't think it will and I can see from your output that PCMG just used one grid). 'ml', hypre' and 'gamg' (a native AMG solver) will do real AMG solvers for you. All three can work on a similar class of problems. 
>>>> 
>>>> 
>>>> Also, you mention that you have a condition number of 1.e20. That is astronomical for such a small problem. How did you compute that number? Do you know where the ill-conditioning comes from? Is this an elliptic operator? 
>>>> 
>>>> 
>>>> Mark 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On May 1, 2012, at 6:31 PM, Matthew Knepley wrote: 
>>>> 
>>>> 
>>>> 
>>>> On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>>> 
>>>> 
>>>> 
>>>> The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow) 
>>>> 
>>>> 
>>>> 
>>>> Because the smoother IS the solver you were using before. Just like I said last time, what you are doing is 
>>>> wrapping up the same solver you used before, sticking it in another GMRES loop, and only looking at the 
>>>> outer loop. This has nothing to do with MG. 
>>>> 
>>>> 
>>>> Matt 
>>>> 
>>>> 
>>>> KSP Object: 8 MPI processes 
>>>> type: gmres 
>>>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>>> GMRES: happy breakdown tolerance 1e-30 
>>>> maximum iterations=1, initial guess is zero 
>>>> using preconditioner applied to right hand side for initial guess 
>>>> tolerances: relative=0.01, absolute=1e-08, divergence=1e+10 
>>>> left preconditioning 
>>>> using DEFAULT norm type for convergence test 
>>>> PC Object: 8 MPI processes 
>>>> type: mg 
>>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>>>> Cycles per PCApply=1 
>>>> Not using Galerkin computed coarse grid matrices 
>>>> Coarse grid solver -- level ------------------------------- 
>>>> KSP Object: (mg_levels_0_) 8 MPI processes 
>>>> type not yet set 
>>>> maximum iterations=1, initial guess is zero 
>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>> left preconditioning 
>>>> using DEFAULT norm type for convergence test 
>>>> PC Object: (mg_levels_0_) 8 MPI processes 
>>>> type not yet set 
>>>> linear system matrix = precond matrix: 
>>>> Matrix Object: 8 MPI processes 
>>>> type: mpiaij 
>>>> rows=75000, cols=75000 
>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>> 
>>>> 
>>>> ----- Original Message ----- 
>>>> From: "Matthew Knepley" < knepley at gmail.com > 
>>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>>> Sent: Tuesday, May 1, 2012 3:22:56 PM 
>>>> Subject: Re: [petsc-users] Multigrid 
>>>> 
>>>> 
>>>> On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>>> 
>>>> 
>>>> 
>>>> Hello, 
>>>> 
>>>> Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same). 
>>>> 
>>>> 
>>>> 
>>>> Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be 
>>>> an extra GMRES loop compared to the case without MG. 
>>>> 
>>>> 
>>>> Matt 
>>>> 
>>>> 
>>>> Regards, 
>>>> Karthik 
>>>> 
>>>> KSP Object: 8 MPI processes 
>>>> type: gmres 
>>>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>>> GMRES: happy breakdown tolerance 1e-30 
>>>> maximum iterations=1 
>>>> using preconditioner applied to right hand side for initial guess 
>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 
>>>> left preconditioning 
>>>> using nonzero initial guess 
>>>> using PRECONDITIONED norm type for convergence test 
>>>> PC Object: 8 MPI processes 
>>>> type: bjacobi 
>>>> block Jacobi: number of blocks = 8 
>>>> Local solve is same for all blocks, in the following KSP and PC objects: 
>>>> KSP Object: (sub_) 1 MPI processes 
>>>> type: preonly 
>>>> maximum iterations=10000, initial guess is zero 
>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>> left preconditioning 
>>>> using NONE norm type for convergence test 
>>>> PC Object: (sub_) 1 MPI processes 
>>>> type: ilu 
>>>> ILU: out-of-place factorization 
>>>> 0 levels of fill 
>>>> tolerance for zero pivot 1e-12 
>>>> using diagonal shift to prevent zero pivot 
>>>> matrix ordering: natural 
>>>> factor fill ratio given 1, needed 1 
>>>> Factored matrix follows: 
>>>> Matrix Object: 1 MPI processes 
>>>> type: seqaij 
>>>> rows=9015, cols=9015 
>>>> package used to perform factorization: petsc 
>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>> linear system matrix = precond matrix: 
>>>> Matrix Object: 1 MPI processes 
>>>> type: seqaij 
>>>> rows=9015, cols=9015 
>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>> linear system matrix = precond matrix: 
>>>> Matrix Object: 8 MPI processes 
>>>> type: mpiaij 
>>>> rows=75000, cols=75000 
>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>> 
>>>> 
>>>> ----- Original Message ----- 
>>>> From: "Matthew Knepley" < knepley at gmail.com > 
>>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>>> Sent: Tuesday, May 1, 2012 3:15:14 PM 
>>>> Subject: Re: [petsc-users] Multigrid 
>>>> 
>>>> 
>>>> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>>> 
>>>> 
>>>> 
>>>> Hello Barry, 
>>>> 
>>>> Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 
>>>> 
>>>> 
>>>> 
>>>> You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 
>>>> 
>>>> 
>>>> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 
>>>> 
>>>> 
>>>> 
>>>> They are different algorithms. Its not possible to say generally that one is better. Try them both. 
>>>> 
>>>> 
>>>> Matt 
>>>> 
>>>> 
>>>> Best regards, 
>>>> Karthik. 
>>>> 
>>>> type: mg 
>>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>>>> Cycles per PCApply=1 
>>>> Not using Galerkin computed coarse grid matrices 
>>>> Coarse grid solver -- level ------------------------------- 
>>>> KSP Object: (mg_levels_0_) 8 MPI processes 
>>>> type: gmres 
>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>>> GMRES: happy breakdown tolerance 1e-30 
>>>> maximum iterations=1, initial guess is zero 
>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>> left preconditioning 
>>>> using PRECONDITIONED norm type for convergence test 
>>>> PC Object: (mg_levels_0_) 8 MPI processes 
>>>> type: bjacobi 
>>>> block Jacobi: number of blocks = 8 
>>>> Local solve is same for all blocks, in the following KSP and PC objects: 
>>>> KSP Object: (mg_levels_0_sub_) 1 MPI processes 
>>>> type: preonly 
>>>> maximum iterations=10000, initial guess is zero 
>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>> left preconditioning 
>>>> using NONE norm type for convergence test 
>>>> PC Object: (mg_levels_0_sub_) 1 MPI processes 
>>>> type: ilu 
>>>> ILU: out-of-place factorization 
>>>> 0 levels of fill 
>>>> tolerance for zero pivot 1e-12 
>>>> using diagonal shift to prevent zero pivot 
>>>> matrix ordering: natural 
>>>> factor fill ratio given 1, needed 1 
>>>> Factored matrix follows: 
>>>> Matrix Object: 1 MPI processes 
>>>> type: seqaij 
>>>> rows=9015, cols=9015 
>>>> package used to perform factorization: petsc 
>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>> linear system matrix = precond matrix: 
>>>> Matrix Object: 1 MPI processes 
>>>> type: seqaij 
>>>> rows=9015, cols=9015 
>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>> linear system matrix = precond matrix: 
>>>> Matrix Object: 8 MPI processes 
>>>> type: mpiaij 
>>>> rows=75000, cols=75000 
>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>> linear system matrix = precond matrix: 
>>>> Matrix Object: 8 MPI processes 
>>>> type: mpiaij 
>>>> rows=75000, cols=75000 
>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>> total number of mallocs used during MatSetValues calls =0 
>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>> 
>>>> 
>>>> 
>>>> ----- Original Message ----- 
>>>> From: "Barry Smith" < bsmith at mcs.anl.gov > 
>>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>>> Sent: Tuesday, May 1, 2012 1:39:26 PM 
>>>> Subject: Re: [petsc-users] Multigrid 
>>>> 
>>>> 
>>>> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 
>>>> 
>>>>> Hello, 
>>>>> 
>>>>> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
>>>>> 
>>>>> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
>>>>> help in this regard will be appreciated. 
>>>> 
>>>> First run with -ksp_view to see what solver it is actually using. 
>>>> 
>>>> Barry 
>>>> 
>>>>> 
>>>>> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
>>>>> 
>>>>> Thanks, 
>>>>> Karthik 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> 
>>>>> ======================================= 
>>>>> Karthik Duraisamy 
>>>>> Assistant Professor (Consulting) 
>>>>> Durand Building Rm 357 
>>>>> Dept of Aeronautics and Astronautics 
>>>>> Stanford University 
>>>>> Stanford CA 94305 
>>>>> 
>>>>> Phone: 650-721-2835 
>>>>> Web: www.stanford.edu/~dkarthik 
>>>>> ======================================= 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>>> -- Norbert Wiener 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>>> -- Norbert Wiener 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>>> -- Norbert Wiener 
>>>> 
>>> 
>> 
> 


From bsmith at mcs.anl.gov  Thu May  3 00:04:07 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 3 May 2012 00:04:07 -0500
Subject: [petsc-users] Multigrid
In-Reply-To: <604010753.102000523.1336021307180.JavaMail.root@zm09.stanford.edu>
References: <604010753.102000523.1336021307180.JavaMail.root@zm09.stanford.edu>
Message-ID: <954EB0D4-EBB3-491B-B862-74504CF5064F@mcs.anl.gov>


On May 3, 2012, at 12:01 AM, Karthik Duraisamy wrote:

> Sorry.. I forgot to add that I tried just those options 
> 
> -pc_type ml -ksp_type fgmres   -ksp_max_it 100 -ksp_view 
> 
> to get the result that I got.

    The output you sent below was NOT obtained with these options. 

>> type: gmres
>> type: bjacobi

   Are you sure you have KSPSetFromOptions() in your code?

  Barry


> I was asking for additional options. In any case, now I can try a few things
> 
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Wednesday, May 2, 2012 9:56:05 PM
> Subject: Re: [petsc-users] Multigrid
> 
> 
>   Make the run I suggested -pc_type ml -ksp_type fgmres   -ksp_max_it 100 -ksp_view   
> 
>  Barry
> 
> On May 2, 2012, at 10:21 PM, Karthik Duraisamy wrote:
> 
>> Fair enough. I wiped out all references to petsc in my code and started from scratch and used just the options you mentioned.. and I get good convergence as first reported in the PCMG reference! I have attached the output. Now that it is behaving, could you recommend some options to exercise the multigrid?
>> 
>> Thanks a lot for your time (and patience!)
>> 
>> ----------------------
>> 
>> KSP Object: 8 MPI processes
>> type: gmres
>>   GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>   GMRES: happy breakdown tolerance 1e-30
>> maximum iterations=1, initial guess is zero
>> tolerances:  relative=1e-05, absolute=1e-50, divergence=1e+10
>> left preconditioning
>> using PRECONDITIONED norm type for convergence test
>> PC Object: 8 MPI processes
>> type: bjacobi
>>   block Jacobi: number of blocks = 8
>>   Local solve is same for all blocks, in the following KSP and PC objects:
>> KSP Object:  (sub_)   1 MPI processes  
>>   type: preonly
>>   maximum iterations=10000, initial guess is zero
>>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>   left preconditioning
>>   using NONE norm type for convergence test
>> PC Object:  (sub_)   1 MPI processes  
>>   type: ilu
>>     ILU: out-of-place factorization
>>     0 levels of fill
>>     tolerance for zero pivot 1e-12
>>     using diagonal shift to prevent zero pivot
>>     matrix ordering: natural
>>     factor fill ratio given 1, needed 1
>>       Factored matrix follows:
>>         Matrix Object:           1 MPI processes          
>>           type: seqaij
>>           rows=9015, cols=9015
>>           package used to perform factorization: petsc
>>           total: nonzeros=517777, allocated nonzeros=517777
>>           total number of mallocs used during MatSetValues calls =0
>>             using I-node routines: found 3476 nodes, limit used is 5
>>   linear system matrix = precond matrix:
>>   Matrix Object:     1 MPI processes    
>>     type: seqaij
>>     rows=9015, cols=9015
>>     total: nonzeros=517777, allocated nonzeros=517777
>>     total number of mallocs used during MatSetValues calls =0
>>       using I-node routines: found 3476 nodes, limit used is 5
>> linear system matrix = precond matrix:
>> Matrix Object:   8 MPI processes  
>>   type: mpiaij
>>   rows=75000, cols=75000
>>   total: nonzeros=4427800, allocated nonzeros=4427800
>>   total number of mallocs used during MatSetValues calls =0
>>     using I-node (on process 0) routines: found 3476 nodes, limit used is 5
>> 
>> 
>> 
>> 
>> ----- Original Message -----
>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>> Sent: Wednesday, May 2, 2012 10:34:17 AM
>> Subject: Re: [petsc-users] Multigrid
>> 
>> 
>>  Run with -pc_type ml -ksp_type fgmres   -ksp_max_it 100 -ksp_view   and NOTHING else. Don'ts set any KSP or PC options in the code!  Then send us ALL the output.
>> 
>>   Barry
>> 
>> I am getting all confused with "I ran with these options and this happened, I ran with these options and this happened, I ran with these options and this happened". I am getting information overload with too much information and yet not enough information.
>> 
>> 
>> On May 2, 2012, at 12:27 PM, Karthik Duraisamy wrote:
>> 
>>> Hello,
>>> 
>>>  With PCML, I tried 
>>> 
>>>  -mg_coarse_pc_type  redundant ; -mg_coarse_ksp_type preonly ;   KSPSetType(ksp, KSPGMRES);
>>> 
>>> and the residual still diverges after a while. Strangely, if I use PCMG (with 0 levels), it works OK, but not with PCML.
>>> 
>>> This is definitely strange. Here is the entire piece of relevant code.
>>> 
>>> When initializing Petsc, I do the following: (let's call this the first bit and I do this only once) 
>>> 
>>>  KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>>  KSPSetInitialGuessKnoll(ksp, PETSC_TRUE);
>>>  KSPSetType(ksp, KSPFGMRES);
>>>  KSPGMRESSetRestart(ksp, 100);
>>>  KSPSetFromOptions(ksp);
>>> 
>>> 
>>> Then for each outer solver iteration: (let's call this the second bit and I do this every outer iteration)
>>> 
>>>  PC pc;
>>>  KSPGetPC(ksp,&pc);
>>>  PCSetType(pc, PCMG);
>>>  KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>>  PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>>  KSPSetType(ksp, KSPGMRES);
>>>  KSPGMRESSetRestart(ksp, 100);
>>>  KSPSetFromOptions(ksp);
>>> 
>>> 
>>> I have tried various combinations of preconditioners and smoothers (including fgmres, bcgsl, etc) but the only combination that works is the above. Any of the PCML options that I have tried failed. 
>>> 
>>> Notes:
>>> - If I keep both of the above bits, it converges very well
>>> 
>>> - If I keep both of the above bits, but replace PCMG with PCML, it diverges quickly
>>> 
>>> - If I removed the first bit and kept the second bit, the residuals diverge even with PCMG 
>>> 
>>> - If I keep the first bit and remove the second bit, the residuals diverge eventually, but at a slow rate
>>> 
>>> 
>>> I am very confident with the rest of the code as I've been using PETSc (with just the first bit) for 3 years on 100s of problems.
>>> 
>>> Regards,
>>> Karthik
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> ----- Original Message -----
>>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>>> Sent: Wednesday, May 2, 2012 6:02:06 AM
>>> Subject: Re: [petsc-users] Multigrid
>>> 
>>> 
>>> This is likely helping a great deal:   
>>> 
>>> KSPGMRESSetRestart(ksp, 100);
>>> 
>>> You can try this with ml also
>>> 
>>> On May 2, 2012, at 1:33 AM, Karthik Duraisamy wrote:
>>> 
>>>> Hello again,
>>>> 
>>>>   I played with BoomerAMG and ML (for the 2D problem with a reasonable condition number), and so far, they haven't worked (the residual eventually diverges). The options that I am using are (for instance)
>>>> 
>>>> PC pc; 
>>>> KSPGetPC(ksp,&pc);
>>>> PetscOptionsSetValue("-ksp_type", "gmres");
>>> 
>>> Here you NEED  fgmres since the smoother is gmres by default. You can see this with -ksp_view
>>> 
>>>> KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>>> PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>>> PCSetType(pc, PCML);
>>>> PetscOptionsSetValue("-pc_ml_maxNlevels", "3");
>>>> PetscOptionsSetValue("-mg_coarse_ksp_type","richardson");
>>>> PetscOptionsSetValue("-mg_coarse_pc_type","sor");
>>>> PetscOptionsSetValue("-mg_coarse_pc_sor_its","8");
>>> 
>>> Just use -mg_coarse_pc_type  redundant -mg_coarse_ksp_type preonly instead of the above.
>>> 
>>> 
>>> Barry
>>> 
>>>> PetscOptionsSetValue("-ksp_monitor","1");
>>>> KSPSetFromOptions(ksp);
>>> 
>>> 
>>>> 
>>>> I have replaced KSP with bcgsl, fgmres, etc.
>>>> 
>>>> As I mentioned earlier, the following works like a charm:
>>>> 
>>>> PC pc; 
>>>> KSPGetPC(ksp,&pc);
>>>> PCSetType(pc, PCMG);
>>>> KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>>> PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>>> KSPSetType(ksp, KSPGMRES);
>>>> KSPGMRESSetRestart(ksp, 100);
>>>> KSPSetFromOptions(ksp);
>>>> 
>>>> 
>>>> Regards,
>>>> Karthik.
>>>> 
>>>> 
>>>> 
>>>> ----- Original Message -----
>>>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>>>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>>>> Sent: Tuesday, May 1, 2012 9:14:05 PM
>>>> Subject: Re: [petsc-users] Multigrid
>>>> 
>>>> 
>>>> On May 1, 2012, at 6:00 PM, Karthik Duraisamy wrote:
>>>> 
>>>>> So as I understand it, GMRES is used as a preconditioner and as a solver when I use PCMG with defaults. If this is the case, I should be able to recreate this set up without the PCMG. Any pointers as to how this can be done?
>>>>> 
>>>>> Also, yes indeed, my mesh is completely unstructured, so I will have to use ml or boomeramg. 
>>>>> 
>>>>> The problems that I am attempting involve RANS of compressible turbulent combustion (finite volume, steady). The high condition numbers are because of the extreme grid stretching and stiff source terms (in the turbulence and combustion model).
>>>> 
>>>> With a condition number like that I wouldn't even consider using double precision, I think you would just be computing meaningless noise. With petsc-dev you can use quad precision ./configure --with-precision=__float128 using recent GNU compilers
>>>> (no we haven't tested the gfortran sides of things). 
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>>> I have been trying a reduced problem in these 2D test cases, in which the condition number is only 1e7.
>>>>> 
>>>>> Thanks,
>>>>> Karthik.
>>>>> 
>>>>> 
>>>>> ----- Original Message -----
>>>>> From: "Mark F. Adams" <mark.adams at columbia.edu>
>>>>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>>>>> Sent: Tuesday, May 1, 2012 3:50:31 PM
>>>>> Subject: Re: [petsc-users] Multigrid
>>>>> 
>>>>> 
>>>>> Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ matrix. If you use regular grids (DA?) then PETSc can construct geometric multigrid coarse grid spaces, although I don't know if PCMG will construct these for you (I don't think it will and I can see from your output that PCMG just used one grid). 'ml', hypre' and 'gamg' (a native AMG solver) will do real AMG solvers for you. All three can work on a similar class of problems. 
>>>>> 
>>>>> 
>>>>> Also, you mention that you have a condition number of 1.e20. That is astronomical for such a small problem. How did you compute that number? Do you know where the ill-conditioning comes from? Is this an elliptic operator? 
>>>>> 
>>>>> 
>>>>> Mark 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On May 1, 2012, at 6:31 PM, Matthew Knepley wrote: 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>>>> 
>>>>> 
>>>>> 
>>>>> The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow) 
>>>>> 
>>>>> 
>>>>> 
>>>>> Because the smoother IS the solver you were using before. Just like I said last time, what you are doing is 
>>>>> wrapping up the same solver you used before, sticking it in another GMRES loop, and only looking at the 
>>>>> outer loop. This has nothing to do with MG. 
>>>>> 
>>>>> 
>>>>> Matt 
>>>>> 
>>>>> 
>>>>> KSP Object: 8 MPI processes 
>>>>> type: gmres 
>>>>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>>>> GMRES: happy breakdown tolerance 1e-30 
>>>>> maximum iterations=1, initial guess is zero 
>>>>> using preconditioner applied to right hand side for initial guess 
>>>>> tolerances: relative=0.01, absolute=1e-08, divergence=1e+10 
>>>>> left preconditioning 
>>>>> using DEFAULT norm type for convergence test 
>>>>> PC Object: 8 MPI processes 
>>>>> type: mg 
>>>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>>>>> Cycles per PCApply=1 
>>>>> Not using Galerkin computed coarse grid matrices 
>>>>> Coarse grid solver -- level ------------------------------- 
>>>>> KSP Object: (mg_levels_0_) 8 MPI processes 
>>>>> type not yet set 
>>>>> maximum iterations=1, initial guess is zero 
>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>>> left preconditioning 
>>>>> using DEFAULT norm type for convergence test 
>>>>> PC Object: (mg_levels_0_) 8 MPI processes 
>>>>> type not yet set 
>>>>> linear system matrix = precond matrix: 
>>>>> Matrix Object: 8 MPI processes 
>>>>> type: mpiaij 
>>>>> rows=75000, cols=75000 
>>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>>> 
>>>>> 
>>>>> ----- Original Message ----- 
>>>>> From: "Matthew Knepley" < knepley at gmail.com > 
>>>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>>>> Sent: Tuesday, May 1, 2012 3:22:56 PM 
>>>>> Subject: Re: [petsc-users] Multigrid 
>>>>> 
>>>>> 
>>>>> On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>>>> 
>>>>> 
>>>>> 
>>>>> Hello, 
>>>>> 
>>>>> Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same). 
>>>>> 
>>>>> 
>>>>> 
>>>>> Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be 
>>>>> an extra GMRES loop compared to the case without MG. 
>>>>> 
>>>>> 
>>>>> Matt 
>>>>> 
>>>>> 
>>>>> Regards, 
>>>>> Karthik 
>>>>> 
>>>>> KSP Object: 8 MPI processes 
>>>>> type: gmres 
>>>>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>>>> GMRES: happy breakdown tolerance 1e-30 
>>>>> maximum iterations=1 
>>>>> using preconditioner applied to right hand side for initial guess 
>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 
>>>>> left preconditioning 
>>>>> using nonzero initial guess 
>>>>> using PRECONDITIONED norm type for convergence test 
>>>>> PC Object: 8 MPI processes 
>>>>> type: bjacobi 
>>>>> block Jacobi: number of blocks = 8 
>>>>> Local solve is same for all blocks, in the following KSP and PC objects: 
>>>>> KSP Object: (sub_) 1 MPI processes 
>>>>> type: preonly 
>>>>> maximum iterations=10000, initial guess is zero 
>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>>> left preconditioning 
>>>>> using NONE norm type for convergence test 
>>>>> PC Object: (sub_) 1 MPI processes 
>>>>> type: ilu 
>>>>> ILU: out-of-place factorization 
>>>>> 0 levels of fill 
>>>>> tolerance for zero pivot 1e-12 
>>>>> using diagonal shift to prevent zero pivot 
>>>>> matrix ordering: natural 
>>>>> factor fill ratio given 1, needed 1 
>>>>> Factored matrix follows: 
>>>>> Matrix Object: 1 MPI processes 
>>>>> type: seqaij 
>>>>> rows=9015, cols=9015 
>>>>> package used to perform factorization: petsc 
>>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>>> linear system matrix = precond matrix: 
>>>>> Matrix Object: 1 MPI processes 
>>>>> type: seqaij 
>>>>> rows=9015, cols=9015 
>>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>>> linear system matrix = precond matrix: 
>>>>> Matrix Object: 8 MPI processes 
>>>>> type: mpiaij 
>>>>> rows=75000, cols=75000 
>>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>>> 
>>>>> 
>>>>> ----- Original Message ----- 
>>>>> From: "Matthew Knepley" < knepley at gmail.com > 
>>>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>>>> Sent: Tuesday, May 1, 2012 3:15:14 PM 
>>>>> Subject: Re: [petsc-users] Multigrid 
>>>>> 
>>>>> 
>>>>> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>>>> 
>>>>> 
>>>>> 
>>>>> Hello Barry, 
>>>>> 
>>>>> Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 
>>>>> 
>>>>> 
>>>>> 
>>>>> You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 
>>>>> 
>>>>> 
>>>>> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 
>>>>> 
>>>>> 
>>>>> 
>>>>> They are different algorithms. Its not possible to say generally that one is better. Try them both. 
>>>>> 
>>>>> 
>>>>> Matt 
>>>>> 
>>>>> 
>>>>> Best regards, 
>>>>> Karthik. 
>>>>> 
>>>>> type: mg 
>>>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>>>>> Cycles per PCApply=1 
>>>>> Not using Galerkin computed coarse grid matrices 
>>>>> Coarse grid solver -- level ------------------------------- 
>>>>> KSP Object: (mg_levels_0_) 8 MPI processes 
>>>>> type: gmres 
>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>>>> GMRES: happy breakdown tolerance 1e-30 
>>>>> maximum iterations=1, initial guess is zero 
>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>>> left preconditioning 
>>>>> using PRECONDITIONED norm type for convergence test 
>>>>> PC Object: (mg_levels_0_) 8 MPI processes 
>>>>> type: bjacobi 
>>>>> block Jacobi: number of blocks = 8 
>>>>> Local solve is same for all blocks, in the following KSP and PC objects: 
>>>>> KSP Object: (mg_levels_0_sub_) 1 MPI processes 
>>>>> type: preonly 
>>>>> maximum iterations=10000, initial guess is zero 
>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>>> left preconditioning 
>>>>> using NONE norm type for convergence test 
>>>>> PC Object: (mg_levels_0_sub_) 1 MPI processes 
>>>>> type: ilu 
>>>>> ILU: out-of-place factorization 
>>>>> 0 levels of fill 
>>>>> tolerance for zero pivot 1e-12 
>>>>> using diagonal shift to prevent zero pivot 
>>>>> matrix ordering: natural 
>>>>> factor fill ratio given 1, needed 1 
>>>>> Factored matrix follows: 
>>>>> Matrix Object: 1 MPI processes 
>>>>> type: seqaij 
>>>>> rows=9015, cols=9015 
>>>>> package used to perform factorization: petsc 
>>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>>> linear system matrix = precond matrix: 
>>>>> Matrix Object: 1 MPI processes 
>>>>> type: seqaij 
>>>>> rows=9015, cols=9015 
>>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>>> linear system matrix = precond matrix: 
>>>>> Matrix Object: 8 MPI processes 
>>>>> type: mpiaij 
>>>>> rows=75000, cols=75000 
>>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>>> linear system matrix = precond matrix: 
>>>>> Matrix Object: 8 MPI processes 
>>>>> type: mpiaij 
>>>>> rows=75000, cols=75000 
>>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>>> 
>>>>> 
>>>>> 
>>>>> ----- Original Message ----- 
>>>>> From: "Barry Smith" < bsmith at mcs.anl.gov > 
>>>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>>>> Sent: Tuesday, May 1, 2012 1:39:26 PM 
>>>>> Subject: Re: [petsc-users] Multigrid 
>>>>> 
>>>>> 
>>>>> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 
>>>>> 
>>>>>> Hello, 
>>>>>> 
>>>>>> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
>>>>>> 
>>>>>> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
>>>>>> help in this regard will be appreciated. 
>>>>> 
>>>>> First run with -ksp_view to see what solver it is actually using. 
>>>>> 
>>>>> Barry 
>>>>> 
>>>>>> 
>>>>>> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
>>>>>> 
>>>>>> Thanks, 
>>>>>> Karthik 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> 
>>>>>> ======================================= 
>>>>>> Karthik Duraisamy 
>>>>>> Assistant Professor (Consulting) 
>>>>>> Durand Building Rm 357 
>>>>>> Dept of Aeronautics and Astronautics 
>>>>>> Stanford University 
>>>>>> Stanford CA 94305 
>>>>>> 
>>>>>> Phone: 650-721-2835 
>>>>>> Web: www.stanford.edu/~dkarthik 
>>>>>> ======================================= 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>>>> -- Norbert Wiener 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>>>> -- Norbert Wiener 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>>>> -- Norbert Wiener 
>>>>> 
>>>> 
>>> 
>> 
> 


From dkarthik at stanford.edu  Thu May  3 00:10:36 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Wed, 2 May 2012 22:10:36 -0700 (PDT)
Subject: [petsc-users] Multigrid
In-Reply-To: <954EB0D4-EBB3-491B-B862-74504CF5064F@mcs.anl.gov>
Message-ID: <490781821.102009281.1336021836522.JavaMail.root@zm09.stanford.edu>

oops.. I am sorry.. I attached the wrong output (I need some sleep). This is what I got from your suggestion and it worked well.

--------------------------------------
KSP Object: 8 MPI processes
  type: fgmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=1, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=1e+10
  right preconditioning
  using UNPRECONDITIONED norm type for convergence test
PC Object: 8 MPI processes
  type: ml
    MG: type is MULTIPLICATIVE, levels=5 cycles=v
      Cycles per PCApply=1
      Using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (mg_coarse_)     8 MPI processes    
      type: preonly
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test
    PC Object:    (mg_coarse_)     8 MPI processes    
      type: redundant
        Redundant preconditioner: First (color=0) of 8 PCs follows
      KSP Object:      (mg_coarse_redundant_)       1 MPI processes      
        type: preonly
        maximum iterations=10000, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
        left preconditioning
        using NONE norm type for convergence test
      PC Object:      (mg_coarse_redundant_)       1 MPI processes      
        type: lu
          LU: out-of-place factorization
          tolerance for zero pivot 1e-12
          matrix ordering: nd
          factor fill ratio given 5, needed 1
            Factored matrix follows:
              Matrix Object:               1 MPI processes              
                type: seqaij
                rows=8, cols=8
                package used to perform factorization: petsc
                total: nonzeros=64, allocated nonzeros=64
                total number of mallocs used during MatSetValues calls =0
                  using I-node routines: found 2 nodes, limit used is 5
        linear system matrix = precond matrix:
        Matrix Object:         1 MPI processes        
          type: seqaij
          rows=8, cols=8
          total: nonzeros=64, allocated nonzeros=64
          total number of mallocs used during MatSetValues calls =0
            using I-node routines: found 2 nodes, limit used is 5
      linear system matrix = precond matrix:
      Matrix Object:       8 MPI processes      
        type: mpiaij
        rows=8, cols=8
        total: nonzeros=64, allocated nonzeros=64
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object:    (mg_levels_1_)     8 MPI processes    
      type: richardson
        Richardson: damping factor=1
      maximum iterations=1
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using PRECONDITIONED norm type for convergence test
    PC Object:    (mg_levels_1_)     8 MPI processes    
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       8 MPI processes      
        type: mpiaij
        rows=9, cols=9
        total: nonzeros=77, allocated nonzeros=77
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object:    (mg_levels_2_)     8 MPI processes    
      type: richardson
        Richardson: damping factor=1
      maximum iterations=1
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using PRECONDITIONED norm type for convergence test
    PC Object:    (mg_levels_2_)     8 MPI processes    
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       8 MPI processes      
        type: mpiaij
        rows=98, cols=98
        total: nonzeros=1589, allocated nonzeros=1589
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 3 -------------------------------
    KSP Object:    (mg_levels_3_)     8 MPI processes    
      type: richardson
        Richardson: damping factor=1
      maximum iterations=1
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using PRECONDITIONED norm type for convergence test
    PC Object:    (mg_levels_3_)     8 MPI processes    
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       8 MPI processes      
        type: mpiaij
        rows=1362, cols=1362
        total: nonzeros=21940, allocated nonzeros=21940
        total number of mallocs used during MatSetValues calls =0
          not using I-node (on process 0) routines
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 4 -------------------------------
    KSP Object:    (mg_levels_4_)     8 MPI processes    
      type: richardson
        Richardson: damping factor=1
      maximum iterations=1
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using nonzero initial guess
      using PRECONDITIONED norm type for convergence test
    PC Object:    (mg_levels_4_)     8 MPI processes    
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Matrix Object:       8 MPI processes      
        type: mpiaij
        rows=75000, cols=75000
        total: nonzeros=4427800, allocated nonzeros=4427800
        total number of mallocs used during MatSetValues calls =0
          using I-node (on process 0) routines: found 3476 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Matrix Object:   8 MPI processes  
    type: mpiaij
    rows=75000, cols=75000
    total: nonzeros=4427800, allocated nonzeros=4427800
    total number of mallocs used during MatSetValues calls =0
      using I-node (on process 0) routines: found 3476 nodes, limit used is 5
ITER, DTMIN, CFL = 9 0.459314 10


----- Original Message -----
From: "Barry Smith" <bsmith at mcs.anl.gov>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Wednesday, May 2, 2012 10:04:07 PM
Subject: Re: [petsc-users] Multigrid


On May 3, 2012, at 12:01 AM, Karthik Duraisamy wrote:

> Sorry.. I forgot to add that I tried just those options 
> 
> -pc_type ml -ksp_type fgmres   -ksp_max_it 100 -ksp_view 
> 
> to get the result that I got.

    The output you sent below was NOT obtained with these options. 

>> type: gmres
>> type: bjacobi

   Are you sure you have KSPSetFromOptions() in your code?

  Barry


> I was asking for additional options. In any case, now I can try a few things
> 
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Wednesday, May 2, 2012 9:56:05 PM
> Subject: Re: [petsc-users] Multigrid
> 
> 
>   Make the run I suggested -pc_type ml -ksp_type fgmres   -ksp_max_it 100 -ksp_view   
> 
>  Barry
> 
> On May 2, 2012, at 10:21 PM, Karthik Duraisamy wrote:
> 
>> Fair enough. I wiped out all references to petsc in my code and started from scratch and used just the options you mentioned.. and I get good convergence as first reported in the PCMG reference! I have attached the output. Now that it is behaving, could you recommend some options to exercise the multigrid?
>> 
>> Thanks a lot for your time (and patience!)
>> 
>> ----------------------
>> 
>> KSP Object: 8 MPI processes
>> type: gmres
>>   GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>   GMRES: happy breakdown tolerance 1e-30
>> maximum iterations=1, initial guess is zero
>> tolerances:  relative=1e-05, absolute=1e-50, divergence=1e+10
>> left preconditioning
>> using PRECONDITIONED norm type for convergence test
>> PC Object: 8 MPI processes
>> type: bjacobi
>>   block Jacobi: number of blocks = 8
>>   Local solve is same for all blocks, in the following KSP and PC objects:
>> KSP Object:  (sub_)   1 MPI processes  
>>   type: preonly
>>   maximum iterations=10000, initial guess is zero
>>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>   left preconditioning
>>   using NONE norm type for convergence test
>> PC Object:  (sub_)   1 MPI processes  
>>   type: ilu
>>     ILU: out-of-place factorization
>>     0 levels of fill
>>     tolerance for zero pivot 1e-12
>>     using diagonal shift to prevent zero pivot
>>     matrix ordering: natural
>>     factor fill ratio given 1, needed 1
>>       Factored matrix follows:
>>         Matrix Object:           1 MPI processes          
>>           type: seqaij
>>           rows=9015, cols=9015
>>           package used to perform factorization: petsc
>>           total: nonzeros=517777, allocated nonzeros=517777
>>           total number of mallocs used during MatSetValues calls =0
>>             using I-node routines: found 3476 nodes, limit used is 5
>>   linear system matrix = precond matrix:
>>   Matrix Object:     1 MPI processes    
>>     type: seqaij
>>     rows=9015, cols=9015
>>     total: nonzeros=517777, allocated nonzeros=517777
>>     total number of mallocs used during MatSetValues calls =0
>>       using I-node routines: found 3476 nodes, limit used is 5
>> linear system matrix = precond matrix:
>> Matrix Object:   8 MPI processes  
>>   type: mpiaij
>>   rows=75000, cols=75000
>>   total: nonzeros=4427800, allocated nonzeros=4427800
>>   total number of mallocs used during MatSetValues calls =0
>>     using I-node (on process 0) routines: found 3476 nodes, limit used is 5
>> 
>> 
>> 
>> 
>> ----- Original Message -----
>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>> Sent: Wednesday, May 2, 2012 10:34:17 AM
>> Subject: Re: [petsc-users] Multigrid
>> 
>> 
>>  Run with -pc_type ml -ksp_type fgmres   -ksp_max_it 100 -ksp_view   and NOTHING else. Don'ts set any KSP or PC options in the code!  Then send us ALL the output.
>> 
>>   Barry
>> 
>> I am getting all confused with "I ran with these options and this happened, I ran with these options and this happened, I ran with these options and this happened". I am getting information overload with too much information and yet not enough information.
>> 
>> 
>> On May 2, 2012, at 12:27 PM, Karthik Duraisamy wrote:
>> 
>>> Hello,
>>> 
>>>  With PCML, I tried 
>>> 
>>>  -mg_coarse_pc_type  redundant ; -mg_coarse_ksp_type preonly ;   KSPSetType(ksp, KSPGMRES);
>>> 
>>> and the residual still diverges after a while. Strangely, if I use PCMG (with 0 levels), it works OK, but not with PCML.
>>> 
>>> This is definitely strange. Here is the entire piece of relevant code.
>>> 
>>> When initializing Petsc, I do the following: (let's call this the first bit and I do this only once) 
>>> 
>>>  KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>>  KSPSetInitialGuessKnoll(ksp, PETSC_TRUE);
>>>  KSPSetType(ksp, KSPFGMRES);
>>>  KSPGMRESSetRestart(ksp, 100);
>>>  KSPSetFromOptions(ksp);
>>> 
>>> 
>>> Then for each outer solver iteration: (let's call this the second bit and I do this every outer iteration)
>>> 
>>>  PC pc;
>>>  KSPGetPC(ksp,&pc);
>>>  PCSetType(pc, PCMG);
>>>  KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>>  PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>>  KSPSetType(ksp, KSPGMRES);
>>>  KSPGMRESSetRestart(ksp, 100);
>>>  KSPSetFromOptions(ksp);
>>> 
>>> 
>>> I have tried various combinations of preconditioners and smoothers (including fgmres, bcgsl, etc) but the only combination that works is the above. Any of the PCML options that I have tried failed. 
>>> 
>>> Notes:
>>> - If I keep both of the above bits, it converges very well
>>> 
>>> - If I keep both of the above bits, but replace PCMG with PCML, it diverges quickly
>>> 
>>> - If I removed the first bit and kept the second bit, the residuals diverge even with PCMG 
>>> 
>>> - If I keep the first bit and remove the second bit, the residuals diverge eventually, but at a slow rate
>>> 
>>> 
>>> I am very confident with the rest of the code as I've been using PETSc (with just the first bit) for 3 years on 100s of problems.
>>> 
>>> Regards,
>>> Karthik
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> ----- Original Message -----
>>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>>> Sent: Wednesday, May 2, 2012 6:02:06 AM
>>> Subject: Re: [petsc-users] Multigrid
>>> 
>>> 
>>> This is likely helping a great deal:   
>>> 
>>> KSPGMRESSetRestart(ksp, 100);
>>> 
>>> You can try this with ml also
>>> 
>>> On May 2, 2012, at 1:33 AM, Karthik Duraisamy wrote:
>>> 
>>>> Hello again,
>>>> 
>>>>   I played with BoomerAMG and ML (for the 2D problem with a reasonable condition number), and so far, they haven't worked (the residual eventually diverges). The options that I am using are (for instance)
>>>> 
>>>> PC pc; 
>>>> KSPGetPC(ksp,&pc);
>>>> PetscOptionsSetValue("-ksp_type", "gmres");
>>> 
>>> Here you NEED  fgmres since the smoother is gmres by default. You can see this with -ksp_view
>>> 
>>>> KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>>> PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>>> PCSetType(pc, PCML);
>>>> PetscOptionsSetValue("-pc_ml_maxNlevels", "3");
>>>> PetscOptionsSetValue("-mg_coarse_ksp_type","richardson");
>>>> PetscOptionsSetValue("-mg_coarse_pc_type","sor");
>>>> PetscOptionsSetValue("-mg_coarse_pc_sor_its","8");
>>> 
>>> Just use -mg_coarse_pc_type  redundant -mg_coarse_ksp_type preonly instead of the above.
>>> 
>>> 
>>> Barry
>>> 
>>>> PetscOptionsSetValue("-ksp_monitor","1");
>>>> KSPSetFromOptions(ksp);
>>> 
>>> 
>>>> 
>>>> I have replaced KSP with bcgsl, fgmres, etc.
>>>> 
>>>> As I mentioned earlier, the following works like a charm:
>>>> 
>>>> PC pc; 
>>>> KSPGetPC(ksp,&pc);
>>>> PCSetType(pc, PCMG);
>>>> KSPSetOperators(ksp, A_, A_, SAME_NONZERO_PATTERN);
>>>> PetscOptionsSetValue("-ksp_initial_guess_nonzero", "true");
>>>> KSPSetType(ksp, KSPGMRES);
>>>> KSPGMRESSetRestart(ksp, 100);
>>>> KSPSetFromOptions(ksp);
>>>> 
>>>> 
>>>> Regards,
>>>> Karthik.
>>>> 
>>>> 
>>>> 
>>>> ----- Original Message -----
>>>> From: "Barry Smith" <bsmith at mcs.anl.gov>
>>>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>>>> Sent: Tuesday, May 1, 2012 9:14:05 PM
>>>> Subject: Re: [petsc-users] Multigrid
>>>> 
>>>> 
>>>> On May 1, 2012, at 6:00 PM, Karthik Duraisamy wrote:
>>>> 
>>>>> So as I understand it, GMRES is used as a preconditioner and as a solver when I use PCMG with defaults. If this is the case, I should be able to recreate this set up without the PCMG. Any pointers as to how this can be done?
>>>>> 
>>>>> Also, yes indeed, my mesh is completely unstructured, so I will have to use ml or boomeramg. 
>>>>> 
>>>>> The problems that I am attempting involve RANS of compressible turbulent combustion (finite volume, steady). The high condition numbers are because of the extreme grid stretching and stiff source terms (in the turbulence and combustion model).
>>>> 
>>>> With a condition number like that I wouldn't even consider using double precision, I think you would just be computing meaningless noise. With petsc-dev you can use quad precision ./configure --with-precision=__float128 using recent GNU compilers
>>>> (no we haven't tested the gfortran sides of things). 
>>>> 
>>>> Barry
>>>> 
>>>> 
>>>>> I have been trying a reduced problem in these 2D test cases, in which the condition number is only 1e7.
>>>>> 
>>>>> Thanks,
>>>>> Karthik.
>>>>> 
>>>>> 
>>>>> ----- Original Message -----
>>>>> From: "Mark F. Adams" <mark.adams at columbia.edu>
>>>>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>>>>> Sent: Tuesday, May 1, 2012 3:50:31 PM
>>>>> Subject: Re: [petsc-users] Multigrid
>>>>> 
>>>>> 
>>>>> Also, note that PCMG can not create coarse grid spaces for an (MPI)AIJ matrix. If you use regular grids (DA?) then PETSc can construct geometric multigrid coarse grid spaces, although I don't know if PCMG will construct these for you (I don't think it will and I can see from your output that PCMG just used one grid). 'ml', hypre' and 'gamg' (a native AMG solver) will do real AMG solvers for you. All three can work on a similar class of problems. 
>>>>> 
>>>>> 
>>>>> Also, you mention that you have a condition number of 1.e20. That is astronomical for such a small problem. How did you compute that number? Do you know where the ill-conditioning comes from? Is this an elliptic operator? 
>>>>> 
>>>>> 
>>>>> Mark 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On May 1, 2012, at 6:31 PM, Matthew Knepley wrote: 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, May 1, 2012 at 6:27 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>>>> 
>>>>> 
>>>>> 
>>>>> The following was output for the very first iteration whereas what I had attached earlier was output every iteration. I am still a bit perplexed because PCMG drops the residual like a rock (after the first few iterations whereas with no PCMG, it is very slow) 
>>>>> 
>>>>> 
>>>>> 
>>>>> Because the smoother IS the solver you were using before. Just like I said last time, what you are doing is 
>>>>> wrapping up the same solver you used before, sticking it in another GMRES loop, and only looking at the 
>>>>> outer loop. This has nothing to do with MG. 
>>>>> 
>>>>> 
>>>>> Matt 
>>>>> 
>>>>> 
>>>>> KSP Object: 8 MPI processes 
>>>>> type: gmres 
>>>>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>>>> GMRES: happy breakdown tolerance 1e-30 
>>>>> maximum iterations=1, initial guess is zero 
>>>>> using preconditioner applied to right hand side for initial guess 
>>>>> tolerances: relative=0.01, absolute=1e-08, divergence=1e+10 
>>>>> left preconditioning 
>>>>> using DEFAULT norm type for convergence test 
>>>>> PC Object: 8 MPI processes 
>>>>> type: mg 
>>>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>>>>> Cycles per PCApply=1 
>>>>> Not using Galerkin computed coarse grid matrices 
>>>>> Coarse grid solver -- level ------------------------------- 
>>>>> KSP Object: (mg_levels_0_) 8 MPI processes 
>>>>> type not yet set 
>>>>> maximum iterations=1, initial guess is zero 
>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>>> left preconditioning 
>>>>> using DEFAULT norm type for convergence test 
>>>>> PC Object: (mg_levels_0_) 8 MPI processes 
>>>>> type not yet set 
>>>>> linear system matrix = precond matrix: 
>>>>> Matrix Object: 8 MPI processes 
>>>>> type: mpiaij 
>>>>> rows=75000, cols=75000 
>>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>>> 
>>>>> 
>>>>> ----- Original Message ----- 
>>>>> From: "Matthew Knepley" < knepley at gmail.com > 
>>>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>>>> Sent: Tuesday, May 1, 2012 3:22:56 PM 
>>>>> Subject: Re: [petsc-users] Multigrid 
>>>>> 
>>>>> 
>>>>> On Tue, May 1, 2012 at 6:18 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>>>> 
>>>>> 
>>>>> 
>>>>> Hello, 
>>>>> 
>>>>> Sorry (and thanks for the reply). I've attached the no multigrid case. I didn't include it because (at least to the untrained eye, everything looks the same). 
>>>>> 
>>>>> 
>>>>> 
>>>>> Did you send all the output from the MG case? There must be a PC around it. By default its GMRES, so there would be 
>>>>> an extra GMRES loop compared to the case without MG. 
>>>>> 
>>>>> 
>>>>> Matt 
>>>>> 
>>>>> 
>>>>> Regards, 
>>>>> Karthik 
>>>>> 
>>>>> KSP Object: 8 MPI processes 
>>>>> type: gmres 
>>>>> GMRES: restart=100, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>>>> GMRES: happy breakdown tolerance 1e-30 
>>>>> maximum iterations=1 
>>>>> using preconditioner applied to right hand side for initial guess 
>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=1e+10 
>>>>> left preconditioning 
>>>>> using nonzero initial guess 
>>>>> using PRECONDITIONED norm type for convergence test 
>>>>> PC Object: 8 MPI processes 
>>>>> type: bjacobi 
>>>>> block Jacobi: number of blocks = 8 
>>>>> Local solve is same for all blocks, in the following KSP and PC objects: 
>>>>> KSP Object: (sub_) 1 MPI processes 
>>>>> type: preonly 
>>>>> maximum iterations=10000, initial guess is zero 
>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>>> left preconditioning 
>>>>> using NONE norm type for convergence test 
>>>>> PC Object: (sub_) 1 MPI processes 
>>>>> type: ilu 
>>>>> ILU: out-of-place factorization 
>>>>> 0 levels of fill 
>>>>> tolerance for zero pivot 1e-12 
>>>>> using diagonal shift to prevent zero pivot 
>>>>> matrix ordering: natural 
>>>>> factor fill ratio given 1, needed 1 
>>>>> Factored matrix follows: 
>>>>> Matrix Object: 1 MPI processes 
>>>>> type: seqaij 
>>>>> rows=9015, cols=9015 
>>>>> package used to perform factorization: petsc 
>>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>>> linear system matrix = precond matrix: 
>>>>> Matrix Object: 1 MPI processes 
>>>>> type: seqaij 
>>>>> rows=9015, cols=9015 
>>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>>> linear system matrix = precond matrix: 
>>>>> Matrix Object: 8 MPI processes 
>>>>> type: mpiaij 
>>>>> rows=75000, cols=75000 
>>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>>> 
>>>>> 
>>>>> ----- Original Message ----- 
>>>>> From: "Matthew Knepley" < knepley at gmail.com > 
>>>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>>>> Sent: Tuesday, May 1, 2012 3:15:14 PM 
>>>>> Subject: Re: [petsc-users] Multigrid 
>>>>> 
>>>>> 
>>>>> On Tue, May 1, 2012 at 6:12 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 
>>>>> 
>>>>> 
>>>>> 
>>>>> Hello Barry, 
>>>>> 
>>>>> Thank you for your super quick response. I have attached the output of ksp_view and it is practically the same as that when I don't use PCMG. The part I don't understand is how PCMG able to function at the zero grid level and still produce a much better convergence than when using the default PC. Is there any additional smoothing or interpolation going on? 
>>>>> 
>>>>> 
>>>>> 
>>>>> You only included one output, so I have no way of knowing what you used before. However, this is running GMRES/ILU. 
>>>>> 
>>>>> 
>>>>> Also, for Algebraic Multigrid, would you recommend BoomerAMG or ML ? 
>>>>> 
>>>>> 
>>>>> 
>>>>> They are different algorithms. Its not possible to say generally that one is better. Try them both. 
>>>>> 
>>>>> 
>>>>> Matt 
>>>>> 
>>>>> 
>>>>> Best regards, 
>>>>> Karthik. 
>>>>> 
>>>>> type: mg 
>>>>> MG: type is MULTIPLICATIVE, levels=1 cycles=v 
>>>>> Cycles per PCApply=1 
>>>>> Not using Galerkin computed coarse grid matrices 
>>>>> Coarse grid solver -- level ------------------------------- 
>>>>> KSP Object: (mg_levels_0_) 8 MPI processes 
>>>>> type: gmres 
>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement 
>>>>> GMRES: happy breakdown tolerance 1e-30 
>>>>> maximum iterations=1, initial guess is zero 
>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>>> left preconditioning 
>>>>> using PRECONDITIONED norm type for convergence test 
>>>>> PC Object: (mg_levels_0_) 8 MPI processes 
>>>>> type: bjacobi 
>>>>> block Jacobi: number of blocks = 8 
>>>>> Local solve is same for all blocks, in the following KSP and PC objects: 
>>>>> KSP Object: (mg_levels_0_sub_) 1 MPI processes 
>>>>> type: preonly 
>>>>> maximum iterations=10000, initial guess is zero 
>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000 
>>>>> left preconditioning 
>>>>> using NONE norm type for convergence test 
>>>>> PC Object: (mg_levels_0_sub_) 1 MPI processes 
>>>>> type: ilu 
>>>>> ILU: out-of-place factorization 
>>>>> 0 levels of fill 
>>>>> tolerance for zero pivot 1e-12 
>>>>> using diagonal shift to prevent zero pivot 
>>>>> matrix ordering: natural 
>>>>> factor fill ratio given 1, needed 1 
>>>>> Factored matrix follows: 
>>>>> Matrix Object: 1 MPI processes 
>>>>> type: seqaij 
>>>>> rows=9015, cols=9015 
>>>>> package used to perform factorization: petsc 
>>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>>> linear system matrix = precond matrix: 
>>>>> Matrix Object: 1 MPI processes 
>>>>> type: seqaij 
>>>>> rows=9015, cols=9015 
>>>>> total: nonzeros=517777, allocated nonzeros=517777 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node routines: found 3476 nodes, limit used is 5 
>>>>> linear system matrix = precond matrix: 
>>>>> Matrix Object: 8 MPI processes 
>>>>> type: mpiaij 
>>>>> rows=75000, cols=75000 
>>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>>> linear system matrix = precond matrix: 
>>>>> Matrix Object: 8 MPI processes 
>>>>> type: mpiaij 
>>>>> rows=75000, cols=75000 
>>>>> total: nonzeros=4427800, allocated nonzeros=4427800 
>>>>> total number of mallocs used during MatSetValues calls =0 
>>>>> using I-node (on process 0) routines: found 3476 nodes, limit used is 5 
>>>>> 
>>>>> 
>>>>> 
>>>>> ----- Original Message ----- 
>>>>> From: "Barry Smith" < bsmith at mcs.anl.gov > 
>>>>> To: "PETSc users list" < petsc-users at mcs.anl.gov > 
>>>>> Sent: Tuesday, May 1, 2012 1:39:26 PM 
>>>>> Subject: Re: [petsc-users] Multigrid 
>>>>> 
>>>>> 
>>>>> On May 1, 2012, at 3:37 PM, Karthik Duraisamy wrote: 
>>>>> 
>>>>>> Hello, 
>>>>>> 
>>>>>> I have been using PETSc for a couple of years with good success, but lately as my linear problems have become stiffer (condition numbers of the order of 1.e20), I am looking to use better preconditioners. I tried using PCMG with all the default options (i.e., I just specified my preconditioner as PCMG and did not add any options to it) and I am immediately seeing better convergence. 
>>>>>> 
>>>>>> What I am not sure of is why? I would like to know more about the default parameters (the manual is not very explicit) and more importantly, want to know why it is working even when I haven't specified any grid levels and coarse grid operators. Any 
>>>>>> help in this regard will be appreciated. 
>>>>> 
>>>>> First run with -ksp_view to see what solver it is actually using. 
>>>>> 
>>>>> Barry 
>>>>> 
>>>>>> 
>>>>>> Also, ultimately I want to use algebraic multigrid so is PCML a better option than BoomerAMG? I tried BoomerAMG with mixed results. 
>>>>>> 
>>>>>> Thanks, 
>>>>>> Karthik 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> 
>>>>>> ======================================= 
>>>>>> Karthik Duraisamy 
>>>>>> Assistant Professor (Consulting) 
>>>>>> Durand Building Rm 357 
>>>>>> Dept of Aeronautics and Astronautics 
>>>>>> Stanford University 
>>>>>> Stanford CA 94305 
>>>>>> 
>>>>>> Phone: 650-721-2835 
>>>>>> Web: www.stanford.edu/~dkarthik 
>>>>>> ======================================= 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>>>> -- Norbert Wiener 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>>>> -- Norbert Wiener 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. 
>>>>> -- Norbert Wiener 
>>>>> 
>>>> 
>>> 
>> 
> 


From m.pavanakumar at gmail.com  Thu May  3 02:44:29 2012
From: m.pavanakumar at gmail.com (Pavanakumar Mohanamuraly)
Date: Thu, 3 May 2012 13:14:29 +0530
Subject: [petsc-users] Krylov for Implicit CFD
Message-ID: <CAAS9=SWq3BycZ5sVtp1fZ0zu6WUhjcTKa84diB2ZWxt+=StnPw@mail.gmail.com>

Hi,

Currently I am planning to use Petsc Krylov solver for implicit time
integration of my CFD code. I need some pointer on the implementation. I am
providing a detailed list of things I am not very clear with in both the
documentation and the mailing list archives.

--> The equations to be solved is of the form :  [ I / \delta t + J ] \delta
U^n = - R[ U^n ]

--> The Jacobian J is hard to evaluate exactly and hence we use the finite
difference method to evaluate J. I want to use a matrix-free approach so
all I need is the action of this J on some vector v.

--> Jv = \frac{\partial R}{\partial U}v = ( R[ U^n + hv ] - R[ U^] ) / h, h
is some parameter. Since I is identity matrix and \delta t is fixed this
diagonal matrix is trivial to evaluate. Thus [ I / \delta t + J ] is
available as a user define function in my solver.

1) Vector U^n is from an unstructured node distribution and is partitioned
outside of Petsc. I have both the global indexing and local indexing
information along with the ghost node information.
2) I have already defined functions to move data across the ghost nodes
given the local variable address and memory stride/offset information.
3) In Jacobian-vector Jv evaluation, one has to exchange the ghost cell
values of vector v with adjacent processor before the calculation is done.
4) I can write the Jacobian-vector Jv evaluation function to perform ghost
node exchange before evaluation but I am not clear as to how I can
interface my local arrays and local / global indexing information into
Petsc.
5) I understand that I have to create a Matrix Shell for the matrix free
method and define my user-defined function to evaluate the matrix-vector
product given an input vector. But it is not clear as to how Petsc
understands how I order my vectors locally and how that corresponds to the
global indexes.
6) I cannot use the DA object as it is for structured/contiguous
partitioning .i.e., the local indices don't correspond to contiguous global
ordering/indexing.
7) I also don't want to use the unstructured mesh and decomposition
routines in Petsc and I already have my own.

Can this be done in Petsc ?

Thanking you in anticipation.

Regards,


-- 

Pavanakumar Mohanamuraly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/306c7f6f/attachment.htm>

From behzad.baghapour at gmail.com  Thu May  3 03:18:58 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Thu, 3 May 2012 12:48:58 +0430
Subject: [petsc-users] Some Questions about Inexact Newton Solver
In-Reply-To: <CAMYG4Gn3p0WgyOuTud4B4+f=XGGKYyi9qjAs3w+ArWVO6wD-yA@mail.gmail.com>
References: <CA+jOtDJWG+06-w5NpYQ280WsPf1KCSmeL6NGnB0vd1yua4s3tA@mail.gmail.com>
	<CAMYG4GnuWgXAR14VrP0QgVEn5MMBho8kuJ1QUX2D2Oc-qqNkaw@mail.gmail.com>
	<CA+jOtDKHYdyCRybbPoRno=tEbgV9zWfsPRMAnXfLfrfJZFwdvw@mail.gmail.com>
	<CAMYG4Gn3p0WgyOuTud4B4+f=XGGKYyi9qjAs3w+ArWVO6wD-yA@mail.gmail.com>
Message-ID: <CA+jOtDKxNYTz7Ame36JDbYS+t3jiWwmid4L6WiJDvXF9PxoL5A@mail.gmail.com>

Thanks a lot. I will search about ASPIN.

BehZad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/269080eb/attachment-0001.htm>

From bin.gao at uit.no  Thu May  3 04:04:44 2012
From: bin.gao at uit.no (Gao Bin)
Date: Thu, 3 May 2012 09:04:44 +0000
Subject: [petsc-users] fill matrix using dynamic load balancing code
In-Reply-To: <CAMYG4Gm7d+woogxnjDUMigiY13FYT1R3qcX6iikabkQ7L13zAw@mail.gmail.com>
References: <CCA7D73E69B27E4AB6075A96C35D44394777B787@MBS04.uit.no>,
	<CAMYG4Gm7d+woogxnjDUMigiY13FYT1R3qcX6iikabkQ7L13zAw@mail.gmail.com>
Message-ID: <CCA7D73E69B27E4AB6075A96C35D44394777BA6F@MBS04.uit.no>

Hi, Matt

Thank you very much for your detailed reply.

Cheers

Gao
________________________________
From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] on behalf of Matthew Knepley [knepley at gmail.com]
Sent: Thursday, May 03, 2012 2:56 AM
To: PETSc users list
Subject: Re: [petsc-users] fill matrix using dynamic load balancing code

On Wed, May 2, 2012 at 7:55 PM, Gao Bin <bin.gao at uit.no<mailto:bin.gao at uit.no>> wrote:
Hi, all

(1) I am using PETSc to perform, for instance matrix-matrix multiplication. But the numerical values of the matrix are calculated from my integral code. I guess, for matrix-matrix multiplication, it might be better to distribute the rows of matrix uniformly on processors, right?

It depends on the matrix.

(2) If yes, I have a problem in my integral code. Since I need dynamic load balancing to calculate the matrix elements and then put them into correct places by using MatSetValues. But I have two questions regarding MatSetValues:

(2.1) If one processor calculated a lot of elements belong to another remote processor, will this become a problem for the performance? Should I make all processors calculate as many local elements as possible?

It will raise the communication cost. This, of course, must be weighed against the penalty for load imbalance in your case.

(2.2) From the manual, I see MatSetValues only caches the values, but MatAssemblyBegin() and MatAssemblyEnd() assembles the matrix. Therefore, is there any memory limit for MatSetValues, or in other words, will it cost a lot of memory before assembling?

Yes, there is a limit to the stash space:

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatStashSetInitialSize.html

    Matt

Thank you in advance.

Cheers

Gao



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/ab2f0426/attachment.htm>

From pham_ha at yahoo.com  Thu May  3 05:27:22 2012
From: pham_ha at yahoo.com (Pham Van)
Date: Thu, 3 May 2012 03:27:22 -0700 (PDT)
Subject: [petsc-users] (no subject)
Message-ID: <1336040842.17802.YahooMailNeo@web163106.mail.bf1.yahoo.com>

Dear Developers,

I found the out of core parameter in the external solver MUMPS is important when using with cygwin because the default directory /tmp/ simply does not exist in windows. So you may consider putting:

? ierr = PetscOptionsString("-ooc_tmpdir", "The Out of Core directory", "None", lu->id.ooc_tmpdir, lu->id.ooc_tmpdir, 256, PETSC_NULL);


?in the function:

PetscErrorCode PetscSetMUMPSOptions(Mat F, Mat A)

in the file: mumps.c

So that the users can enter the temporary directory of their choice.


Kind regards,
Pham Van Ha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/92b3a8eb/attachment.htm>

From knepley at gmail.com  Thu May  3 09:11:47 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 May 2012 10:11:47 -0400
Subject: [petsc-users] Krylov for Implicit CFD
In-Reply-To: <CAAS9=SWq3BycZ5sVtp1fZ0zu6WUhjcTKa84diB2ZWxt+=StnPw@mail.gmail.com>
References: <CAAS9=SWq3BycZ5sVtp1fZ0zu6WUhjcTKa84diB2ZWxt+=StnPw@mail.gmail.com>
Message-ID: <CAMYG4G=thFKcEJ5UKQfX-cK0P0yrOTkr9CfaUh+whUjHKr1qKQ@mail.gmail.com>

On Thu, May 3, 2012 at 3:44 AM, Pavanakumar Mohanamuraly <
m.pavanakumar at gmail.com> wrote:

> Hi,
>
> Currently I am planning to use Petsc Krylov solver for implicit time
> integration of my CFD code. I need some pointer on the implementation. I am
> providing a detailed list of things I am not very clear with in both the
> documentation and the mailing list archives.
>
> --> The equations to be solved is of the form :  [ I / \delta t + J ] \delta
> U^n = - R[ U^n ]
>
> --> The Jacobian J is hard to evaluate exactly and hence we use the finite
> difference method to evaluate J. I want to use a matrix-free approach so
> all I need is the action of this J on some vector v.
>
> --> Jv = \frac{\partial R}{\partial U}v = ( R[ U^n + hv ] - R[ U^] ) / h,
> h is some parameter. Since I is identity matrix and \delta t is fixed this
> diagonal matrix is trivial to evaluate. Thus [ I / \delta t + J ] is
> available as a user define function in my solver.
>
> 1) Vector U^n is from an unstructured node distribution and is partitioned
> outside of Petsc. I have both the global indexing and local indexing
> information along with the ghost node information.
> 2) I have already defined functions to move data across the ghost nodes
> given the local variable address and memory stride/offset information.
> 3) In Jacobian-vector Jv evaluation, one has to exchange the ghost cell
> values of vector v with adjacent processor before the calculation is done.
> 4) I can write the Jacobian-vector Jv evaluation function to perform ghost
> node exchange before evaluation but I am not clear as to how I can
> interface my local arrays and local / global indexing information into
> Petsc.
> 5) I understand that I have to create a Matrix Shell for the matrix free
> method and define my user-defined function to evaluate the matrix-vector
> product given an input vector. But it is not clear as to how Petsc
> understands how I order my vectors locally and how that corresponds to the
> global indexes.
> 6) I cannot use the DA object as it is for structured/contiguous
> partitioning .i.e., the local indices don't correspond to contiguous global
> ordering/indexing.
> 7) I also don't want to use the unstructured mesh and decomposition
> routines in Petsc and I already have my own.
>
> Can this be done in Petsc ?
>

Yes, there are ??? parts:

  1) You want to manage unstructured data yourself. The easiest way to do
this now is to implement a DM. You just need to provide
      routines for making global and local vectors (you have this), and
routines for mapping between them (you have this). You will have
      to use petsc-dev, but this definitely looks like the easiest thing to
do.

     a) If you consider using part of our infrastructure, you can build a
PetscSection and a PetscSF, and everything else will be automatically
         calculated. These are very new things, so there is not a lot of
documentation, and thus you might prefer to do the whole DM yourself.

  2) You want to do matrix-free action of your Jacobian. This is easy using
MatShell, however the real question is How will you precondition that
       equation?

   Matt


>
> Thanking you in anticipation.
>
> Regards,
>
>
> --
>
> Pavanakumar Mohanamuraly
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/1aeeacbc/attachment.htm>

From hzhang at mcs.anl.gov  Thu May  3 10:06:40 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Thu, 3 May 2012 10:06:40 -0500
Subject: [petsc-users] (no subject)
In-Reply-To: <1336040842.17802.YahooMailNeo@web163106.mail.bf1.yahoo.com>
References: <1336040842.17802.YahooMailNeo@web163106.mail.bf1.yahoo.com>
Message-ID: <CAGCphBtcBLzPDnk5YKDEUU7pu+jbsM72FF8CiWWJBrS0RYJAqg@mail.gmail.com>

Pham Van:
Added to petsc-dev
http://petsc.cs.iit.edu/petsc/petsc-dev/rev/c3403225fb48

Let us know if there is any problem with it.
Thanks for your contribution!
Hong

Dear Developers,
>
> I found the out of core parameter in the external solver MUMPS is
> important when using with cygwin because the default directory /tmp/ simply
> does not exist in windows. So you may consider putting:
>
>   ierr = PetscOptionsString("-ooc_tmpdir", "The Out of Core directory",
> "None", lu->id.ooc_tmpdir, lu->id.ooc_tmpdir, 256, PETSC_NULL);
>
>  in the function:
>
> PetscErrorCode PetscSetMUMPSOptions(Mat F, Mat A)
>
> in the file: mumps.c
>
> So that the users can enter the temporary directory of their choice.
>
> Kind regards,
> Pham Van Ha
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/013d134a/attachment.htm>

From niko.karin at gmail.com  Thu May  3 11:20:45 2012
From: niko.karin at gmail.com (Karin&NiKo)
Date: Thu, 3 May 2012 18:20:45 +0200
Subject: [petsc-users] ML tuning in Fortran
Message-ID: <CA+gX-L8GfM-uiZMdin4N=y6AGEXXhtpBz97WUfu9n8qeH6fnKA@mail.gmail.com>

Dear PETSc team,

I would like to use the ML preconditioner from a fortran source.
I would like to know if there is another way to set the coarsening scheme
in other way than :

      call PetscOptionsSetValue('-pc_ml_CoarsenScheme','Uncoupled',ierr)
      call KSPSetFromOptions(ksp,ierr)

Thanks in advance,
Nicolas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/7e52993f/attachment.htm>

From m.pavanakumar at gmail.com  Thu May  3 12:28:26 2012
From: m.pavanakumar at gmail.com (Pavanakumar Mohanamuraly)
Date: Thu, 3 May 2012 22:58:26 +0530
Subject: [petsc-users] petsc-users Digest, Vol 41, Issue 17
In-Reply-To: <mailman.53.1336064419.24081.petsc-users@mcs.anl.gov>
References: <mailman.53.1336064419.24081.petsc-users@mcs.anl.gov>
Message-ID: <CAAS9=SXpW+b5ZX4Kwpko7-vDmKgRUq8C2jx+O4gOYwUMJn1WrQ@mail.gmail.com>

Dear Matt,



> > Currently I am planning to use Petsc Krylov solver for implicit time
> > integration of my CFD code. I need some pointer on the implementation. I
> am
> > providing a detailed list of things I am not very clear with in both the
> > documentation and the mailing list archives.
> >
> > --> The equations to be solved is of the form :  [ I / \delta t + J ]
> \delta
> > U^n = - R[ U^n ]
> >
> > --> The Jacobian J is hard to evaluate exactly and hence we use the
> finite
> > difference method to evaluate J. I want to use a matrix-free approach so
> > all I need is the action of this J on some vector v.
> >
> > --> Jv = \frac{\partial R}{\partial U}v = ( R[ U^n + hv ] - R[ U^] ) / h,
> > h is some parameter. Since I is identity matrix and \delta t is fixed
> this
> > diagonal matrix is trivial to evaluate. Thus [ I / \delta t + J ] is
> > available as a user define function in my solver.
> >
> > 1) Vector U^n is from an unstructured node distribution and is
> partitioned
> > outside of Petsc. I have both the global indexing and local indexing
> > information along with the ghost node information.
> > 2) I have already defined functions to move data across the ghost nodes
> > given the local variable address and memory stride/offset information.
> > 3) In Jacobian-vector Jv evaluation, one has to exchange the ghost cell
> > values of vector v with adjacent processor before the calculation is
> done.
> > 4) I can write the Jacobian-vector Jv evaluation function to perform
> ghost
> > node exchange before evaluation but I am not clear as to how I can
> > interface my local arrays and local / global indexing information into
> > Petsc.
> > 5) I understand that I have to create a Matrix Shell for the matrix free
> > method and define my user-defined function to evaluate the matrix-vector
> > product given an input vector. But it is not clear as to how Petsc
> > understands how I order my vectors locally and how that corresponds to
> the
> > global indexes.
> > 6) I cannot use the DA object as it is for structured/contiguous
> > partitioning .i.e., the local indices don't correspond to contiguous
> global
> > ordering/indexing.
> > 7) I also don't want to use the unstructured mesh and decomposition
> > routines in Petsc and I already have my own.
> >
> > Can this be done in Petsc ?
> >
>
> Yes, there are ??? parts:
>
>  1) You want to manage unstructured data yourself. The easiest way to do
> this now is to implement a DM. You just need to provide
>      routines for making global and local vectors (you have this), and
> routines for mapping between them (you have this). You will have
>      to use petsc-dev, but this definitely looks like the easiest thing to
> do.
>
>

Thanks for the reply.



>     a) If you consider using part of our infrastructure, you can build a
> PetscSection and a PetscSF, and everything else will be automatically
>         calculated. These are very new things, so there is not a lot of
> documentation, and thus you might prefer to do the whole DM yourself.
>
>
Most of the tutorial slides point out that DM is for structured arrays
(maybe this mislead me). Can you point me to the functions that i would
have to use to create

1) DM array ghost cells
2) Local/Global indexing
3) DM array exchange

It would be nice you can share some code snippet on writing our own DM's.

 2) You want to do matrix-free action of your Jacobian. This is easy using
> MatShell, however the real question is How will you precondition that
>       equation?


Currently I use the Jacobi diagonal preconditioner. I use the largest
spectral radius of my Euler equations \lambda = U + a. BTW I actually have
implemented my own parallel Krylov solver for my CFD code using the
algorithm in Yousuf Saad's book. But my advisor urged me to use Petsc so
that I would be able to use the various pre-conditioners in Petsc and the
code would run more faster/optimal. I have not written my Krylov solver
optimally.

So is there is no way I can use preconditioning in matrix-free context in
Petsc ?

Regards,

--

Pavanakumar Mohanamuraly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/2f9ab156/attachment.htm>

From knepley at gmail.com  Thu May  3 12:38:35 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 May 2012 13:38:35 -0400
Subject: [petsc-users] petsc-users Digest, Vol 41, Issue 17
In-Reply-To: <CAAS9=SXpW+b5ZX4Kwpko7-vDmKgRUq8C2jx+O4gOYwUMJn1WrQ@mail.gmail.com>
References: <mailman.53.1336064419.24081.petsc-users@mcs.anl.gov>
	<CAAS9=SXpW+b5ZX4Kwpko7-vDmKgRUq8C2jx+O4gOYwUMJn1WrQ@mail.gmail.com>
Message-ID: <CAMYG4G=BEFG0+BxDuPvhdHUDmJZZQ4-gZigZU6DH90AGiLnkUg@mail.gmail.com>

On Thu, May 3, 2012 at 1:28 PM, Pavanakumar Mohanamuraly <
m.pavanakumar at gmail.com> wrote:

> Dear Matt,
>
>
>
>> > Currently I am planning to use Petsc Krylov solver for implicit time
>> > integration of my CFD code. I need some pointer on the implementation.
>> I am
>> > providing a detailed list of things I am not very clear with in both the
>> > documentation and the mailing list archives.
>> >
>> > --> The equations to be solved is of the form :  [ I / \delta t + J ]
>> \delta
>> > U^n = - R[ U^n ]
>> >
>> > --> The Jacobian J is hard to evaluate exactly and hence we use the
>> finite
>> > difference method to evaluate J. I want to use a matrix-free approach so
>> > all I need is the action of this J on some vector v.
>> >
>> > --> Jv = \frac{\partial R}{\partial U}v = ( R[ U^n + hv ] - R[ U^] ) /
>> h,
>> > h is some parameter. Since I is identity matrix and \delta t is fixed
>> this
>> > diagonal matrix is trivial to evaluate. Thus [ I / \delta t + J ] is
>> > available as a user define function in my solver.
>> >
>> > 1) Vector U^n is from an unstructured node distribution and is
>> partitioned
>> > outside of Petsc. I have both the global indexing and local indexing
>> > information along with the ghost node information.
>> > 2) I have already defined functions to move data across the ghost nodes
>> > given the local variable address and memory stride/offset information.
>> > 3) In Jacobian-vector Jv evaluation, one has to exchange the ghost cell
>> > values of vector v with adjacent processor before the calculation is
>> done.
>> > 4) I can write the Jacobian-vector Jv evaluation function to perform
>> ghost
>> > node exchange before evaluation but I am not clear as to how I can
>> > interface my local arrays and local / global indexing information into
>> > Petsc.
>> > 5) I understand that I have to create a Matrix Shell for the matrix free
>> > method and define my user-defined function to evaluate the matrix-vector
>> > product given an input vector. But it is not clear as to how Petsc
>> > understands how I order my vectors locally and how that corresponds to
>> the
>> > global indexes.
>> > 6) I cannot use the DA object as it is for structured/contiguous
>> > partitioning .i.e., the local indices don't correspond to contiguous
>> global
>> > ordering/indexing.
>> > 7) I also don't want to use the unstructured mesh and decomposition
>> > routines in Petsc and I already have my own.
>> >
>> > Can this be done in Petsc ?
>> >
>>
>> Yes, there are ??? parts:
>>
>>  1) You want to manage unstructured data yourself. The easiest way to do
>> this now is to implement a DM. You just need to provide
>>      routines for making global and local vectors (you have this), and
>> routines for mapping between them (you have this). You will have
>>      to use petsc-dev, but this definitely looks like the easiest thing to
>> do.
>>
>>
>
> Thanks for the reply.
>
>
>
>>     a) If you consider using part of our infrastructure, you can build a
>> PetscSection and a PetscSF, and everything else will be automatically
>>         calculated. These are very new things, so there is not a lot of
>> documentation, and thus you might prefer to do the whole DM yourself.
>>
>>
> Most of the tutorial slides point out that DM is for structured arrays
> (maybe this mislead me). Can you point me to the functions that i would
> have to use to create
>

No, the DA is for structured grids. It is a subclass of DM. There is also a
DM for unstructured grids.


> 1) DM array ghost cells
>

http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMCreateGlobalVector.html
http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMCreateLocalVector.html


> 2) Local/Global indexing
>
http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMGetLocalToGlobalMapping.html


> 3) DM array exchange
>


http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMGlobalToLocalBegin.html
http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMGlobalToLocalEnd.html
http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMLocalToGlobalBegin.html
http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMLocalToGlobalEnd.html



>
> It would be nice you can share some code snippet on writing our own DM's.
>

You can look at any of the existing DMs in src/dm/impls


>  2) You want to do matrix-free action of your Jacobian. This is easy using
>> MatShell, however the real question is How will you precondition that
>>       equation?
>
>
> Currently I use the Jacobi diagonal preconditioner. I use the largest
> spectral radius of my Euler equations \lambda = U + a. BTW I actually have
> implemented my own parallel Krylov solver for my CFD code using the
> algorithm in Yousuf Saad's book. But my advisor urged me to use Petsc so
> that I would be able to use the various pre-conditioners in Petsc and the
> code would run more faster/optimal. I have not written my Krylov solver
> optimally.
>
> So is there is no way I can use preconditioning in matrix-free context in
> Petsc ?
>

It depends what kind of preconditioner you want to use. Not even Jacobi is
matrix-free. You need the diagonal. You
can basically use Chebychev. What you should consider is providing a
simplified matrix to use for building a
preconditioner.

   Matt


> Regards,
>
> --
>
> Pavanakumar Mohanamuraly
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/41ee1280/attachment.htm>

From bsmith at mcs.anl.gov  Thu May  3 12:58:35 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 3 May 2012 12:58:35 -0500
Subject: [petsc-users] ML tuning in Fortran
In-Reply-To: <CA+gX-L8GfM-uiZMdin4N=y6AGEXXhtpBz97WUfu9n8qeH6fnKA@mail.gmail.com>
References: <CA+gX-L8GfM-uiZMdin4N=y6AGEXXhtpBz97WUfu9n8qeH6fnKA@mail.gmail.com>
Message-ID: <4998B47F-8FED-475A-B168-ACDC311EF79D@mcs.anl.gov>


  No, nor is there in C.


On May 3, 2012, at 11:20 AM, Karin&NiKo wrote:

> Dear PETSc team,
> 
> I would like to use the ML preconditioner from a fortran source.
> I would like to know if there is another way to set the coarsening scheme in other way than :
> 
>       call PetscOptionsSetValue('-pc_ml_CoarsenScheme','Uncoupled',ierr)
>       call KSPSetFromOptions(ksp,ierr)
> 
> Thanks in advance,
> Nicolas
> 


From jkozdon at stanford.edu  Thu May  3 13:05:25 2012
From: jkozdon at stanford.edu (Jeremy Kozdon)
Date: Thu, 3 May 2012 11:05:25 -0700
Subject: [petsc-users] multiblock example
In-Reply-To: <CAMYG4GnRChRRn2885Vxx5bRRkhwdE2=kyPJsGZ-8nPFYjdd9Qw@mail.gmail.com>
References: <CADXDPriEf8B+unjVbcEKMKiinVya2Dodmz-wk__VvLNjV1ArSQ@mail.gmail.com>
	<CAMYG4GnRChRRn2885Vxx5bRRkhwdE2=kyPJsGZ-8nPFYjdd9Qw@mail.gmail.com>
Message-ID: <CADXDPriYBEfc=fRoYbDeaUNbPVb54-i_m_HgQ0eZT4BHiGC58Q@mail.gmail.com>

Thanks for the response Matt. I am clearly a little slow to respond as I
have been working on other things.

Right now there is nothing in particular that isn't working for us, since
we have really started things yet.

The interaction between our blocks can be complicated, but we don't use
ghost cells at all and interaction is done weakly through interfaces so I
would think that this will simplify things (we use summation-by-parts
finite difference methods with the simultaneous approximation term method
if that happens to mean anything to you). Haven't really thought through
our Jacobians yet, but the nonlinearities in our problems do arise through
the coupling terms.

On Sun, Apr 15, 2012 at 2:54 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sun, Apr 15, 2012 at 4:24 PM, Jeremy Kozdon <jkozdon at stanford.edu>wrote:
>
>> Hello All,
>>
>> We are considering using petsc for a new code we are developing which
>> involves multiblock grids and finite difference methods. I am still pretty
>> new to petsc (still reading the manual), so wondering where a good place to
>> look for examples on this is (since I assume that we are not the first to
>> consider this). Not sure if there is a good place in the petsc source to
>> consider this, one of the libraries examples perhaps, or another opensource
>> package.
>>
>> One of the things we want to be able to do is assign different numbers of
>> processors to different blocks based on the work load we estimate for that
>> block. Currently we do this by decomposing the global communicator into a
>> block communicators, but if I understand correctly this is not the correct
>> approach to take with petsc since we will need to use DMComposite which
>> relies on collective operations.
>>
>
> The Cartesian grid interface is the DMDA class in PETSc. You can prescribe
> whatever processor layout you want. The key
> question is how will the blocks interact. The DMComposite is fine for
> putting blocks together to generate a combined residual,
> but the Jacobian interface is not as fleshed out, for example it does not
> automatically allocate the off-diagonal blocks from
> block interaction. I don't think there are any problems with collectives
> in DMComposite. Do you have a specific thing that
> does not work for you?
>
>
>> Currently everything we are doing is with structured cartesian grids, but
>> in the future we will also need to be able to handle unstructured grids
>> which are coupled to the structured grids.
>>
>
> This can also be done with DMComposite, using DMComplex for the
> unstructured grids.
>
>    Matt
>
>
>> Thanks!
>> Jeremy
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/74937f9e/attachment-0001.htm>

From ansayre at sbcglobal.net  Thu May  3 20:02:58 2012
From: ansayre at sbcglobal.net (Alan Sayre)
Date: Thu, 3 May 2012 18:02:58 -0700 (PDT)
Subject: [petsc-users] application of petsc to psr
In-Reply-To: <DF2472D2-33FE-4876-825E-118EC9BB031A@mcs.anl.gov>
References: <1336013159.64482.YahooMailRC@web181501.mail.ne1.yahoo.com>
	<DF2472D2-33FE-4876-825E-118EC9BB031A@mcs.anl.gov>
Message-ID: <1336093378.96379.YahooMailRC@web181511.mail.ne1.yahoo.com>

Barry,

Thanks but that is not quite what I have in mind. The existing PSR model in my 
CFD code is not converging for PSR models containing the Method of Moments using 
the ABF mechanism described as shown in the Chemkin 2006 workshop proceedings. I 
have tried KINSOL and have still not been able to get it to converge. Any 
assistance in developing a PETSC based solution algorithm is appreciated.

thanks,

Alan




________________________________
From: Barry Smith <bsmith at mcs.anl.gov>
To: PETSc users list <petsc-users at mcs.anl.gov>
Sent: Wed, May 2, 2012 10:54:21 PM
Subject: Re: [petsc-users] application of petsc to psr


http://www.sandia.gov/chemkin/workshop/proceedings/Meeks_CHEMKINWorkshop2006.pdf?
 ?


On May 2, 2012, at 9:45 PM, Alan Sayre wrote:

> I'm new to using PETSc. I need to model perfectly stirred reactors for chemical 
>kinetics (10-1000 species), I welcome recommendations on how to get 
>"up-to-speed". I also need to incorprate solution of the energy (temperature) 
>equation and moments of soot distributions.
>? 
> Thanks for any suggestions.
>? 
> Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/4c5866ee/attachment.htm>

From s_g at berkeley.edu  Thu May  3 20:20:33 2012
From: s_g at berkeley.edu (Sanjay Govindjee)
Date: Thu, 03 May 2012 18:20:33 -0700
Subject: [petsc-users] VecValid
Message-ID: <4FA32EE1.6080601@berkeley.edu>

We have some code that has worked up through petsc-3.1 and we are in the 
process of updating it to petsc-3.2.

One thing that I can not find mentioned in the change logs (looking all 
the way back to petsc-2.1.0) or the FAQs is the elimination of the 
VecValid and MatValid tests.

We used to perform operations like:

       call VecValid(xvec, chk, ierr)

       if(chk .eqv. PETSC_FALSE) then
                 call VecCreate        (PETSC_COMM_WORLD, xvec, ierr)
                 call VecSetSizes      (xvec, numpeq, PETSC_DECIDE, ierr)
                 call VecSetFromOptions(xvec, ierr)
       else
         ....
       endif

But it seems VecValid and MatValid have been eliminated.  Is there a 
reason for this?  or have
I made a mistake?  Is there a replacement test for invalid Vec and Mat 
pointers?

-sanjay


From knepley at gmail.com  Thu May  3 21:08:04 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 3 May 2012 22:08:04 -0400
Subject: [petsc-users] VecValid
In-Reply-To: <4FA32EE1.6080601@berkeley.edu>
References: <4FA32EE1.6080601@berkeley.edu>
Message-ID: <CAMYG4Gm_q5d7G8iYAuOSZdZtCTnncTqUkeg8gG105whZuCt+OA@mail.gmail.com>

On Thu, May 3, 2012 at 9:20 PM, Sanjay Govindjee <s_g at berkeley.edu> wrote:

> We have some code that has worked up through petsc-3.1 and we are in the
> process of updating it to petsc-3.2.
>
> One thing that I can not find mentioned in the change logs (looking all
> the way back to petsc-2.1.0) or the FAQs is the elimination of the VecValid
> and MatValid tests.
>

In C, this is now the macro PetscValidHeaderSpecific(). This check now
happens in every function call. What
logic would need to look for a corrupt Vec beyond CHKERRQ?

  Thanks,

      Matt


> We used to perform operations like:
>
>      call VecValid(xvec, chk, ierr)
>
>      if(chk .eqv. PETSC_FALSE) then
>                call VecCreate        (PETSC_COMM_WORLD, xvec, ierr)
>                call VecSetSizes      (xvec, numpeq, PETSC_DECIDE, ierr)
>                call VecSetFromOptions(xvec, ierr)
>      else
>        ....
>      endif
>
> But it seems VecValid and MatValid have been eliminated.  Is there a
> reason for this?  or have
> I made a mistake?  Is there a replacement test for invalid Vec and Mat
> pointers?
>
> -sanjay
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/5023cfe7/attachment.htm>

From s_g at berkeley.edu  Thu May  3 21:17:16 2012
From: s_g at berkeley.edu (Sanjay Govindjee)
Date: Thu, 03 May 2012 19:17:16 -0700
Subject: [petsc-users] VecValid
In-Reply-To: <CAMYG4Gm_q5d7G8iYAuOSZdZtCTnncTqUkeg8gG105whZuCt+OA@mail.gmail.com>
References: <4FA32EE1.6080601@berkeley.edu>
	<CAMYG4Gm_q5d7G8iYAuOSZdZtCTnncTqUkeg8gG105whZuCt+OA@mail.gmail.com>
Message-ID: <4FA33C2C.6050104@berkeley.edu>

The entry points for our code can vary.  Thus a vector for example can 
get created in a handful
of routines.  But it is possible that the vector has already been 
created in another routine.  So
we perform a VecValid check on it.  If it is false we create it, else we 
go on and use it.  Also at
the end when we exit the program we clean up with a series of VecDestroy 
calls but we check first if the vector is valid before calling 
VecDestroy on it.  Same for MatValid/MatDestroy.

Obviously we can store a logical flag in a common block or the like upon 
creation; but VecValid and
MatValid were handy utility functions for us.

Is is feasible to call PetscValidHeaderSpecific( ) out of fortran?  Or 
should should I just set up my own
collection of allocate/deallocate flags.

-sanjay

On 5/3/12 7:08 PM, Matthew Knepley wrote:
> On Thu, May 3, 2012 at 9:20 PM, Sanjay Govindjee <s_g at berkeley.edu 
> <mailto:s_g at berkeley.edu>> wrote:
>
>     We have some code that has worked up through petsc-3.1 and we are
>     in the process of updating it to petsc-3.2.
>
>     One thing that I can not find mentioned in the change logs
>     (looking all the way back to petsc-2.1.0) or the FAQs is the
>     elimination of the VecValid and MatValid tests.
>
>
> In C, this is now the macro PetscValidHeaderSpecific(). This check now 
> happens in every function call. What
> logic would need to look for a corrupt Vec beyond CHKERRQ?
>
>   Thanks,
>
>       Matt
>
>     We used to perform operations like:
>
>          call VecValid(xvec, chk, ierr)
>
>          if(chk .eqv. PETSC_FALSE) then
>                    call VecCreate        (PETSC_COMM_WORLD, xvec, ierr)
>                    call VecSetSizes      (xvec, numpeq, PETSC_DECIDE,
>     ierr)
>                    call VecSetFromOptions(xvec, ierr)
>          else
>            ....
>          endif
>
>     But it seems VecValid and MatValid have been eliminated.  Is there
>     a reason for this?  or have
>     I made a mistake?  Is there a replacement test for invalid Vec and
>     Mat pointers?
>
>     -sanjay
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-- 
-----------------------------------------------
Sanjay Govindjee, PhD, PE
Professor of Civil Engineering

779 Davis Hall
Structural Engineering, Mechanics and Materials
Department of Civil Engineering
University of California
Berkeley, CA 94720-1710

Voice:  +1 510 642 6060
FAX:    +1 510 643 5264
s_g at berkeley.edu
http://www.ce.berkeley.edu/~sanjay
-----------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/6ae92626/attachment.htm>

From bsmith at mcs.anl.gov  Thu May  3 21:22:17 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 3 May 2012 21:22:17 -0500
Subject: [petsc-users] VecValid
In-Reply-To: <4FA33C2C.6050104@berkeley.edu>
References: <4FA32EE1.6080601@berkeley.edu>
	<CAMYG4Gm_q5d7G8iYAuOSZdZtCTnncTqUkeg8gG105whZuCt+OA@mail.gmail.com>
	<4FA33C2C.6050104@berkeley.edu>
Message-ID: <186A3A5C-168F-4993-9D0D-74715F33970E@mcs.anl.gov>


  VecValid was unsafe because it looks inside a data structure that may or not be valid. 

   In Fortran the safe way to do this is to set the value to 0 right after declaration and then check if not zero. So

   Mat    A 

   A  = 0

    .......

    if (A .eq. 0) then 
         call VecCreate(   A,ierr)
    endif

   Barry

On May 3, 2012, at 9:17 PM, Sanjay Govindjee wrote:

> The entry points for our code can vary.  Thus a vector for example can get created in a handful
> of routines.  But it is possible that the vector has already been created in another routine.  So
> we perform a VecValid check on it.  If it is false we create it, else we go on and use it.  Also at
> the end when we exit the program we clean up with a series of VecDestroy calls but we check first if the vector is valid before calling VecDestroy on it.  Same for MatValid/MatDestroy.
> 
> Obviously we can store a logical flag in a common block or the like upon creation; but VecValid and
> MatValid were handy utility functions for us.
> 
> Is is feasible to call PetscValidHeaderSpecific( ) out of fortran?  Or should should I just set up my own
> collection of allocate/deallocate flags.
> 
> -sanjay
> 
> On 5/3/12 7:08 PM, Matthew Knepley wrote:
>> On Thu, May 3, 2012 at 9:20 PM, Sanjay Govindjee <s_g at berkeley.edu> wrote:
>> We have some code that has worked up through petsc-3.1 and we are in the process of updating it to petsc-3.2.
>> 
>> One thing that I can not find mentioned in the change logs (looking all the way back to petsc-2.1.0) or the FAQs is the elimination of the VecValid and MatValid tests.
>> 
>> In C, this is now the macro PetscValidHeaderSpecific(). This check now happens in every function call. What
>> logic would need to look for a corrupt Vec beyond CHKERRQ?
>> 
>>   Thanks,
>> 
>>       Matt
>>  
>> We used to perform operations like:
>> 
>>      call VecValid(xvec, chk, ierr)
>> 
>>      if(chk .eqv. PETSC_FALSE) then
>>                call VecCreate        (PETSC_COMM_WORLD, xvec, ierr)
>>                call VecSetSizes      (xvec, numpeq, PETSC_DECIDE, ierr)
>>                call VecSetFromOptions(xvec, ierr)
>>      else
>>        ....
>>      endif
>> 
>> But it seems VecValid and MatValid have been eliminated.  Is there a reason for this?  or have
>> I made a mistake?  Is there a replacement test for invalid Vec and Mat pointers?
>> 
>> -sanjay
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> -- 
> -----------------------------------------------
> Sanjay Govindjee, PhD, PE
> Professor of Civil Engineering
> 
> 779 Davis Hall
> Structural Engineering, Mechanics and Materials
> Department of Civil Engineering
> University of California
> Berkeley, CA 94720-1710
> 
> Voice:  +1 510 642 6060
> FAX:    +1 510 643 5264
> 
> s_g at berkeley.edu
> http://www.ce.berkeley.edu/~sanjay
> 
> -----------------------------------------------
> 


From bourdin at lsu.edu  Thu May  3 21:39:52 2012
From: bourdin at lsu.edu (Blaise Bourdin)
Date: Thu, 3 May 2012 21:39:52 -0500
Subject: [petsc-users] VecValid
In-Reply-To: <186A3A5C-168F-4993-9D0D-74715F33970E@mcs.anl.gov>
References: <4FA32EE1.6080601@berkeley.edu>
	<CAMYG4Gm_q5d7G8iYAuOSZdZtCTnncTqUkeg8gG105whZuCt+OA@mail.gmail.com>
	<4FA33C2C.6050104@berkeley.edu>
	<186A3A5C-168F-4993-9D0D-74715F33970E@mcs.anl.gov>
Message-ID: <1771606E-950F-4112-9A10-8F41B12E1C45@lsu.edu>

Hi,

As a side note: this is actually a problem I have met in one of the TS examples. When using fortran datatypes, the statements "A=0" and "if (A .eq. 0)" are not valid. They would require overloading the assignment and comparison operators.
Can anybody see an alternative?

Blaise

> 
>  VecValid was unsafe because it looks inside a data structure that may or not be valid. 
> 
>   In Fortran the safe way to do this is to set the value to 0 right after declaration and then check if not zero. So
> 
>   Mat    A 
> 
>   A  = 0
> 
>    .......
> 
>    if (A .eq. 0) then 
>         call VecCreate(   A,ierr)
>    endif
> 
>   Barry
> 
> On May 3, 2012, at 9:17 PM, Sanjay Govindjee wrote:
> 
>> The entry points for our code can vary.  Thus a vector for example can get created in a handful
>> of routines.  But it is possible that the vector has already been created in another routine.  So
>> we perform a VecValid check on it.  If it is false we create it, else we go on and use it.  Also at
>> the end when we exit the program we clean up with a series of VecDestroy calls but we check first if the vector is valid before calling VecDestroy on it.  Same for MatValid/MatDestroy.
>> 
>> Obviously we can store a logical flag in a common block or the like upon creation; but VecValid and
>> MatValid were handy utility functions for us.
>> 
>> Is is feasible to call PetscValidHeaderSpecific( ) out of fortran?  Or should should I just set up my own
>> collection of allocate/deallocate flags.
>> 
>> -sanjay
>> 
>> On 5/3/12 7:08 PM, Matthew Knepley wrote:
>>> On Thu, May 3, 2012 at 9:20 PM, Sanjay Govindjee <s_g at berkeley.edu> wrote:
>>> We have some code that has worked up through petsc-3.1 and we are in the process of updating it to petsc-3.2.
>>> 
>>> One thing that I can not find mentioned in the change logs (looking all the way back to petsc-2.1.0) or the FAQs is the elimination of the VecValid and MatValid tests.
>>> 
>>> In C, this is now the macro PetscValidHeaderSpecific(). This check now happens in every function call. What
>>> logic would need to look for a corrupt Vec beyond CHKERRQ?
>>> 
>>>  Thanks,
>>> 
>>>      Matt
>>> 
>>> We used to perform operations like:
>>> 
>>>     call VecValid(xvec, chk, ierr)
>>> 
>>>     if(chk .eqv. PETSC_FALSE) then
>>>               call VecCreate        (PETSC_COMM_WORLD, xvec, ierr)
>>>               call VecSetSizes      (xvec, numpeq, PETSC_DECIDE, ierr)
>>>               call VecSetFromOptions(xvec, ierr)
>>>     else
>>>       ....
>>>     endif
>>> 
>>> But it seems VecValid and MatValid have been eliminated.  Is there a reason for this?  or have
>>> I made a mistake?  Is there a replacement test for invalid Vec and Mat pointers?
>>> 
>>> -sanjay
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
>> -- 
>> -----------------------------------------------
>> Sanjay Govindjee, PhD, PE
>> Professor of Civil Engineering
>> 
>> 779 Davis Hall
>> Structural Engineering, Mechanics and Materials
>> Department of Civil Engineering
>> University of California
>> Berkeley, CA 94720-1710
>> 
>> Voice:  +1 510 642 6060
>> FAX:    +1 510 643 5264
>> 
>> s_g at berkeley.edu
>> http://www.ce.berkeley.edu/~sanjay
>> 
>> -----------------------------------------------
>> 
> 

-- 
Department of Mathematics and Center for Computation & Technology
Louisiana State University, Baton Rouge, LA 70803, USA
Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin








From bsmith at mcs.anl.gov  Thu May  3 21:53:06 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 3 May 2012 21:53:06 -0500
Subject: [petsc-users] VecValid
In-Reply-To: <1771606E-950F-4112-9A10-8F41B12E1C45@lsu.edu>
References: <4FA32EE1.6080601@berkeley.edu>
	<CAMYG4Gm_q5d7G8iYAuOSZdZtCTnncTqUkeg8gG105whZuCt+OA@mail.gmail.com>
	<4FA33C2C.6050104@berkeley.edu>
	<186A3A5C-168F-4993-9D0D-74715F33970E@mcs.anl.gov>
	<1771606E-950F-4112-9A10-8F41B12E1C45@lsu.edu>
Message-ID: <276A213C-3CD9-4899-AED4-9613E1E6F855@mcs.anl.gov>


On May 3, 2012, at 9:39 PM, Blaise Bourdin wrote:

> Hi,
> 
> As a side note: this is actually a problem I have met in one of the TS examples. When using fortran datatypes, the statements "A=0" and "if (A .eq. 0)" are not valid. They would require overloading the assignment and comparison operators.

   You actually use the type(Mat) A   stuff? Wow? You really have a Fortran fetish.

    Try A%v = 0 at declaration and if (A%v == 0) at check

   Barry



> Can anybody see an alternative?
> 
> Blaise
> 
>> 
>> VecValid was unsafe because it looks inside a data structure that may or not be valid. 
>> 
>>  In Fortran the safe way to do this is to set the value to 0 right after declaration and then check if not zero. So
>> 
>>  Mat    A 
>> 
>>  A  = 0
>> 
>>   .......
>> 
>>   if (A .eq. 0) then 
>>        call VecCreate(   A,ierr)
>>   endif
>> 
>>  Barry
>> 
>> On May 3, 2012, at 9:17 PM, Sanjay Govindjee wrote:
>> 
>>> The entry points for our code can vary.  Thus a vector for example can get created in a handful
>>> of routines.  But it is possible that the vector has already been created in another routine.  So
>>> we perform a VecValid check on it.  If it is false we create it, else we go on and use it.  Also at
>>> the end when we exit the program we clean up with a series of VecDestroy calls but we check first if the vector is valid before calling VecDestroy on it.  Same for MatValid/MatDestroy.
>>> 
>>> Obviously we can store a logical flag in a common block or the like upon creation; but VecValid and
>>> MatValid were handy utility functions for us.
>>> 
>>> Is is feasible to call PetscValidHeaderSpecific( ) out of fortran?  Or should should I just set up my own
>>> collection of allocate/deallocate flags.
>>> 
>>> -sanjay
>>> 
>>> On 5/3/12 7:08 PM, Matthew Knepley wrote:
>>>> On Thu, May 3, 2012 at 9:20 PM, Sanjay Govindjee <s_g at berkeley.edu> wrote:
>>>> We have some code that has worked up through petsc-3.1 and we are in the process of updating it to petsc-3.2.
>>>> 
>>>> One thing that I can not find mentioned in the change logs (looking all the way back to petsc-2.1.0) or the FAQs is the elimination of the VecValid and MatValid tests.
>>>> 
>>>> In C, this is now the macro PetscValidHeaderSpecific(). This check now happens in every function call. What
>>>> logic would need to look for a corrupt Vec beyond CHKERRQ?
>>>> 
>>>> Thanks,
>>>> 
>>>>     Matt
>>>> 
>>>> We used to perform operations like:
>>>> 
>>>>    call VecValid(xvec, chk, ierr)
>>>> 
>>>>    if(chk .eqv. PETSC_FALSE) then
>>>>              call VecCreate        (PETSC_COMM_WORLD, xvec, ierr)
>>>>              call VecSetSizes      (xvec, numpeq, PETSC_DECIDE, ierr)
>>>>              call VecSetFromOptions(xvec, ierr)
>>>>    else
>>>>      ....
>>>>    endif
>>>> 
>>>> But it seems VecValid and MatValid have been eliminated.  Is there a reason for this?  or have
>>>> I made a mistake?  Is there a replacement test for invalid Vec and Mat pointers?
>>>> 
>>>> -sanjay
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>>> -- 
>>> -----------------------------------------------
>>> Sanjay Govindjee, PhD, PE
>>> Professor of Civil Engineering
>>> 
>>> 779 Davis Hall
>>> Structural Engineering, Mechanics and Materials
>>> Department of Civil Engineering
>>> University of California
>>> Berkeley, CA 94720-1710
>>> 
>>> Voice:  +1 510 642 6060
>>> FAX:    +1 510 643 5264
>>> 
>>> s_g at berkeley.edu
>>> http://www.ce.berkeley.edu/~sanjay
>>> 
>>> -----------------------------------------------
>>> 
>> 
> 
> -- 
> Department of Mathematics and Center for Computation & Technology
> Louisiana State University, Baton Rouge, LA 70803, USA
> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin
> 
> 
> 
> 
> 
> 
> 


From s_g at berkeley.edu  Thu May  3 22:09:29 2012
From: s_g at berkeley.edu (Sanjay Govindjee)
Date: Thu, 03 May 2012 20:09:29 -0700
Subject: [petsc-users] VecValid
In-Reply-To: <186A3A5C-168F-4993-9D0D-74715F33970E@mcs.anl.gov>
References: <4FA32EE1.6080601@berkeley.edu>
	<CAMYG4Gm_q5d7G8iYAuOSZdZtCTnncTqUkeg8gG105whZuCt+OA@mail.gmail.com>
	<4FA33C2C.6050104@berkeley.edu>
	<186A3A5C-168F-4993-9D0D-74715F33970E@mcs.anl.gov>
Message-ID: <4FA34869.8030404@berkeley.edu>

Thanks.  I think this will work for us.
-sanjay

On 5/3/12 7:22 PM, Barry Smith wrote:
>    VecValid was unsafe because it looks inside a data structure that may or not be valid.
>
>     In Fortran the safe way to do this is to set the value to 0 right after declaration and then check if not zero. So
>
>     Mat    A
>
>     A  = 0
>
>      .......
>
>      if (A .eq. 0) then
>           call VecCreate(   A,ierr)
>      endif
>
>     Barry
>
> On May 3, 2012, at 9:17 PM, Sanjay Govindjee wrote:
>
>> The entry points for our code can vary.  Thus a vector for example can get created in a handful
>> of routines.  But it is possible that the vector has already been created in another routine.  So
>> we perform a VecValid check on it.  If it is false we create it, else we go on and use it.  Also at
>> the end when we exit the program we clean up with a series of VecDestroy calls but we check first if the vector is valid before calling VecDestroy on it.  Same for MatValid/MatDestroy.
>>
>> Obviously we can store a logical flag in a common block or the like upon creation; but VecValid and
>> MatValid were handy utility functions for us.
>>
>> Is is feasible to call PetscValidHeaderSpecific( ) out of fortran?  Or should should I just set up my own
>> collection of allocate/deallocate flags.
>>
>> -sanjay
>>
>> On 5/3/12 7:08 PM, Matthew Knepley wrote:
>>> On Thu, May 3, 2012 at 9:20 PM, Sanjay Govindjee<s_g at berkeley.edu>  wrote:
>>> We have some code that has worked up through petsc-3.1 and we are in the process of updating it to petsc-3.2.
>>>
>>> One thing that I can not find mentioned in the change logs (looking all the way back to petsc-2.1.0) or the FAQs is the elimination of the VecValid and MatValid tests.
>>>
>>> In C, this is now the macro PetscValidHeaderSpecific(). This check now happens in every function call. What
>>> logic would need to look for a corrupt Vec beyond CHKERRQ?
>>>
>>>    Thanks,
>>>
>>>        Matt
>>>
>>> We used to perform operations like:
>>>
>>>       call VecValid(xvec, chk, ierr)
>>>
>>>       if(chk .eqv. PETSC_FALSE) then
>>>                 call VecCreate        (PETSC_COMM_WORLD, xvec, ierr)
>>>                 call VecSetSizes      (xvec, numpeq, PETSC_DECIDE, ierr)
>>>                 call VecSetFromOptions(xvec, ierr)
>>>       else
>>>         ....
>>>       endif
>>>
>>> But it seems VecValid and MatValid have been eliminated.  Is there a reason for this?  or have
>>> I made a mistake?  Is there a replacement test for invalid Vec and Mat pointers?
>>>
>>> -sanjay
>>>
>>>
>>>
>>>
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> -- 
>> -----------------------------------------------
>> Sanjay Govindjee, PhD, PE
>> Professor of Civil Engineering
>>
>> 779 Davis Hall
>> Structural Engineering, Mechanics and Materials
>> Department of Civil Engineering
>> University of California
>> Berkeley, CA 94720-1710
>>
>> Voice:  +1 510 642 6060
>> FAX:    +1 510 643 5264
>>
>> s_g at berkeley.edu
>> http://www.ce.berkeley.edu/~sanjay
>>
>> -----------------------------------------------
>>

-- 
-----------------------------------------------
Sanjay Govindjee, PhD, PE
Professor of Civil Engineering

779 Davis Hall
Structural Engineering, Mechanics and Materials
Department of Civil Engineering
University of California
Berkeley, CA 94720-1710

Voice:  +1 510 642 6060
FAX:    +1 510 643 5264
s_g at berkeley.edu
http://www.ce.berkeley.edu/~sanjay
-----------------------------------------------


From sciam at iastate.edu  Thu May  3 22:38:40 2012
From: sciam at iastate.edu (Bo Sun)
Date: Thu, 3 May 2012 21:38:40 -0600
Subject: [petsc-users] extract data from vector in petsc into fortran
	array
In-Reply-To: <0A58C6E9-DD55-42B6-AB7F-768A07E5E7F8@mcs.anl.gov>
References: <CA+5XtAUjevavzs5LT0WAZrvYQYK5xUZFbhZRknMvmKtRdh2SEA@mail.gmail.com>
	<0A58C6E9-DD55-42B6-AB7F-768A07E5E7F8@mcs.anl.gov>
Message-ID: <CA+5XtAV48x2u-H2pJpDWvn=cHscAv=HGWfXgYgai9GytVeb6Fw@mail.gmail.com>

Hi Barry,
Thank you for your reply.  Sorry to confuse you. The strange data means
that when I run the case using only one cpu, after some steps, the case
blows up or give wrong data. I think that is because the local memory is
not enough. And some case gives the following error message "

[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Out of memory. This could be due to allocating
[0]PETSC ERROR: too large an object or bleeding by not properly
[0]PETSC ERROR: destroying unneeded objects.
[0]PETSC ERROR: Memory allocated 0 Memory used by process 1882513408
[0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[0]PETSC ERROR: Memory requested 2628072!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45
CST 2012
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./main on a arch-linu named lightningsmp.its.iastate.edu by
sciam Thu May  3 15:58:24 2012
[0]PETSC ERROR: Libraries linked from
/work/doe_ar1/sciam/petsc-3.2-p6/arch-linux2-c-opt/lib
[0]PETSC ERROR: Configure run at Wed Apr  4 12:45:08 2012
[0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90
--with-debugging=0 FOPTFLAGS="-O3 -qarch=p4 -qtune=p4"
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: PetscMallocAlign() line 49 in src/sys/memory/mal.c
[0]PETSC ERROR: VecCreate_Seq() line 40 in src/vec/vec/impls/seq/bvec3.c
[0]PETSC ERROR: VecSetType() line 53 in src/vec/vec/interface/vecreg.c
[0]PETSC ERROR: VecDuplicate_Seq() line 800 in src/vec/vec/impls/seq/bvec2.c
[0]PETSC ERROR: VecDuplicate() line 541 in src/vec/vec/interface/vector.c
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSCERROR:
or try
http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
run
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45
CST 2012
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./main on a arch-linu named lightningsmp.its.iastate.edu by
sciam Thu May  3 15:58:24 2012
[0]PETSC ERROR: Libraries linked from
/work/doe_ar1/sciam/petsc-3.2-p6/arch-linux2-c-opt/lib
[0]PETSC ERROR: Configure run at Wed Apr  4 12:45:08 2012
[0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90
--with-debugging=0 FOPTFLAGS="-O3 -qarch=p4 -qtune=p4"
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory
unknown file
------------------------------------------------------------------------------------------------------------------------------

The input from fortran array to petsc vector and output from petsc vector
to fortran array happens in each loop. I guess that I didn't do
appropriate  destroy to the vector in petsc.

Regards,
Bo

On Wed, May 2, 2012 at 11:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On May 2, 2012, at 11:32 PM, Bo Sun wrote:
>
> > Hi there,
> > I am coding using Fortran 90 with petsc. I extract the results from
> vector in petsc as follows,
> >
> >    PetscScalar, pointer :: xx_v(:)
> >    Vec pet_u
> >
> >    ..............
> >    call VecGetArrayF90(pet_u,xx_v,ierr)
> >     data_for_fortran = xx_v
> >     call VecRestoreArrayF90(pet_u,xx_v,ierr)
> >    ...................
> >
> >
> > For small size of vector, there is no problem. But when the size of
> vector is above million, some strange data appear.
>
>    What do you mean strange data? Is this reproducable? Do two different
> Fortran compilers give the same problem? I you can send us a simple Fortran
> code that demonstrates the problem we can try to reproduce it. Please send
> such a bug report to petsc-maint at mcs.anl.gov
>
>   VecGetArrayF90() is generally the way to go, other things are more
> cumbersome.
> >
>
>
> > Could you recommend another way to output the data of vector in petsc
> into fortran array ?
> >
> >
> > Thanks,
> > Bo
>
>


-- 
--------------------------------------------------------------------
CFD
Mechanical Engineering
Iowa State University
50011
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120503/89d42b75/attachment.htm>

From bsmith at mcs.anl.gov  Thu May  3 22:46:21 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 3 May 2012 22:46:21 -0500
Subject: [petsc-users] extract data from vector in petsc into fortran
	array
In-Reply-To: <CA+5XtAV48x2u-H2pJpDWvn=cHscAv=HGWfXgYgai9GytVeb6Fw@mail.gmail.com>
References: <CA+5XtAUjevavzs5LT0WAZrvYQYK5xUZFbhZRknMvmKtRdh2SEA@mail.gmail.com>
	<0A58C6E9-DD55-42B6-AB7F-768A07E5E7F8@mcs.anl.gov>
	<CA+5XtAV48x2u-H2pJpDWvn=cHscAv=HGWfXgYgai9GytVeb6Fw@mail.gmail.com>
Message-ID: <6E0C4A02-80C5-48D6-8DC7-B9A53BE50357@mcs.anl.gov>


  You are creating or duplicating vectors and not destroying them so they will swallow up more and more memory. Make sure every create/duplicate has a corresponding destroy.

   Barry

On May 3, 2012, at 10:38 PM, Bo Sun wrote:

> Hi Barry,
> Thank you for your reply.  Sorry to confuse you. The strange data means that when I run the case using only one cpu, after some steps, the case blows up or give wrong data. I think that is because the local memory is not enough. And some case gives the following error message "
>  
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Out of memory. This could be due to allocating
> [0]PETSC ERROR: too large an object or bleeding by not properly
> [0]PETSC ERROR: destroying unneeded objects.
> [0]PETSC ERROR: Memory allocated 0 Memory used by process 1882513408
> [0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [0]PETSC ERROR: Memory requested 2628072!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./main on a arch-linu named lightningsmp.its.iastate.edu by sciam Thu May  3 15:58:24 2012
> [0]PETSC ERROR: Libraries linked from /work/doe_ar1/sciam/petsc-3.2-p6/arch-linux2-c-opt/lib
> [0]PETSC ERROR: Configure run at Wed Apr  4 12:45:08 2012
> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-debugging=0 FOPTFLAGS="-O3 -qarch=p4 -qtune=p4"
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: PetscMallocAlign() line 49 in src/sys/memory/mal.c
> [0]PETSC ERROR: VecCreate_Seq() line 40 in src/vec/vec/impls/seq/bvec3.c
> [0]PETSC ERROR: VecSetType() line 53 in src/vec/vec/interface/vecreg.c
> [0]PETSC ERROR: VecDuplicate_Seq() line 800 in src/vec/vec/impls/seq/bvec2.c
> [0]PETSC ERROR: VecDuplicate() line 541 in src/vec/vec/interface/vector.c
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./main on a arch-linu named lightningsmp.its.iastate.edu by sciam Thu May  3 15:58:24 2012
> [0]PETSC ERROR: Libraries linked from /work/doe_ar1/sciam/petsc-3.2-p6/arch-linux2-c-opt/lib
> [0]PETSC ERROR: Configure run at Wed Apr  4 12:45:08 2012
> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-debugging=0 FOPTFLAGS="-O3 -qarch=p4 -qtune=p4"
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> ------------------------------------------------------------------------------------------------------------------------------
> 
> The input from fortran array to petsc vector and output from petsc vector to fortran array happens in each loop. I guess that I didn't do appropriate  destroy to the vector in petsc. 
> 
> Regards,
> Bo
> 
> On Wed, May 2, 2012 at 11:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> On May 2, 2012, at 11:32 PM, Bo Sun wrote:
> 
> > Hi there,
> > I am coding using Fortran 90 with petsc. I extract the results from vector in petsc as follows,
> >
> >    PetscScalar, pointer :: xx_v(:)
> >    Vec pet_u
> >
> >    ..............
> >    call VecGetArrayF90(pet_u,xx_v,ierr)
> >     data_for_fortran = xx_v
> >     call VecRestoreArrayF90(pet_u,xx_v,ierr)
> >    ...................
> >
> >
> > For small size of vector, there is no problem. But when the size of vector is above million, some strange data appear.
> 
>   What do you mean strange data? Is this reproducable? Do two different Fortran compilers give the same problem? I you can send us a simple Fortran code that demonstrates the problem we can try to reproduce it. Please send such a bug report to petsc-maint at mcs.anl.gov
> 
>   VecGetArrayF90() is generally the way to go, other things are more cumbersome.
> >
> 
> 
> > Could you recommend another way to output the data of vector in petsc into fortran array ?
> >
> >
> > Thanks,
> > Bo
> 
> 
> 
> 
> -- 
> --------------------------------------------------------------------
> CFD
> Mechanical Engineering
> Iowa State University
> 50011


From juhaj at iki.fi  Fri May  4 06:01:40 2012
From: juhaj at iki.fi (Juha =?iso-8859-15?q?J=E4ykk=E4?=)
Date: Fri, 4 May 2012 13:01:40 +0200
Subject: [petsc-users] determining dof at runtime
In-Reply-To: <CAMYG4G=E8vDOioF7=rC6y4hc6QQ4o=CivBJxoQE9=4i-TVzDsA@mail.gmail.com>
References: <201204162311.16383.juhaj@iki.fi>
	<CAMYG4G=E8vDOioF7=rC6y4hc6QQ4o=CivBJxoQE9=4i-TVzDsA@mail.gmail.com>
Message-ID: <201205041301.51595.juhaj@iki.fi>

> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMDAVecGetArrayD
> OF.html

Thanks! I has managed to overlook his. Switching to use this solved my 
problem. Sorry for taking a while to reply, though.

Cheers,
Juha
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 230 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/25d22095/attachment-0001.pgp>

From zonexo at gmail.com  Fri May  4 07:59:00 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Fri, 04 May 2012 14:59:00 +0200
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in ksp
In-Reply-To: <4FA19C3F.5090604@gmail.com>
References: <4FA19C3F.5090604@gmail.com>
Message-ID: <4FA3D294.5050202@gmail.com>


Hi,

Is there anything else I can try to get it working right?

Thanks!


On 2/5/2012 10:11 PM, Matthew Knepley wrote:
> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I did a MatView and VecView on both C and Fortran, right after Mat
>     and Vec assembly. I have attached the printout below. They are
>     exactly the same, but yet the result is different in Neumann
>     condition. However, the dirichlet condition gives the correct ans.
>     Is there anything else that could be wrong even if the Mat and Vec
>     are the same?
>
>
> Did you set the null space for the matrix when you have Neumann 
> conditions?
Yes, for the matrix, I set as:

call 
MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)

     call MatSetNullSpace(jac,nullspace,ierr)

     call MatNullSpaceDestroy(nullspace,ierr)

for the Vec,

call 
MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)

*!call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)*

     call MatNullSpaceDestroy(nullspace,ierr)

MatNullSpaceRemove was comment out because there's error during linking


>
>    Matt
>
>     Thanks!
>
>     Fortran:
>
>     Matrix Object: 1 MPI processes
>       type: seqaij
>     row 0: (0, 2)  (1, -1)  (3, -1)
>     row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>     row 2: (1, -1)  (2, 2)  (5, -1)
>     row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>     row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>     row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>     row 6: (3, -1)  (6, 2)  (7, -1)
>     row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>     row 8: (5, -1)  (7, -1)  (8, 2)
>     Vector Object:Vec_0000000084000000_0 1 MPI processes
>       type: mpi
>     Process [0]
>     0.25
>     0.0205213
>     1.135e-005
>     0.0205213
>     0.00168449
>     9.31663e-007
>     1.135e-005
>     9.31663e-007
>     5.15289e-010
>     Vector Object:Vec_0000000084000000_1 1 MPI processes
>       type: mpi
>     Process [0]
>     0.14924
>     0.0242397
>     -0.0260347
>     0.0242397
>     -0.0256192
>     -0.0400102
>     -0.0260347
>     -0.0400102
>     -0.0400102
>     Press any key to continue . . .
>
>     C:
>
>     Matrix Object: 1 MPI processes
>       type: seqaij
>     row 0: (0, 2)  (1, -1)  (3, -1)
>     row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>     row 2: (1, -1)  (2, 2)  (5, -1)
>     row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>     row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>     row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>     row 6: (3, -1)  (6, 2)  (7, -1)
>     row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>     row 8: (5, -1)  (7, -1)  (8, 2)
>     Vector Object:Vec_0x1d3b000_0 1 MPI processes
>       type: mpi
>     Process [0]
>     0.25
>     0.0205212
>     1.135e-05
>     0.0205212
>     0.00168449
>     9.31663e-07
>     1.135e-05
>     9.31663e-07
>     5.15288e-10
>     Vector Object:Vec_0x1d3b000_1 1 MPI processes
>       type: mpi
>     Process [0]
>     0.139311
>     0.0305751
>     -0.0220633
>     0.0305751
>     -0.0135158
>     -0.042185
>     -0.0220633
>     -0.042185
>     -0.058449
>
>
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>     On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>     On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         Do you mean my method is wrong?
>>
>>         I am following the template of ex22f,
>>
>>         where the variables are declared as :
>>
>>         PetscScalar  v(5)
>>
>>         MatStencil   row(4),col(4,5)
>>
>>         Hence,
>>
>>         for the neumann BC
>>
>>         num = 1
>>
>>                        if (j/=0) then
>>
>>                            v(num) = -rho*HxdHy
>>
>>                            col(MatStencil_i,num) = i
>>
>>                            col(MatStencil_j,num) = j-1
>>
>>                            num = num + 1
>>
>>                        end if
>>
>>                        if (i/=0) then
>>
>>                            v(num) = -rho*HydHx
>>
>>                            col(MatStencil_i,num) = i-1
>>
>>                            col(MatStencil_j,num) = j
>>
>>                            num = num + 1
>>
>>                        end if
>>
>>                        if (i/=mx-1) then
>>
>>                            v(num) = -rho*HydHx
>>
>>                            col(MatStencil_i,num) = i+1
>>
>>                            col(MatStencil_j,num) = j
>>
>>                            num = num + 1
>>
>>                        end if
>>
>>                        if (j/=my-1) then
>>
>>                            v(num) = -rho*HxdHy
>>
>>                            col(MatStencil_i,num) = i
>>
>>                            col(MatStencil_j,num) = j+1
>>
>>                            num = num + 1
>>
>>                        end if
>>
>>                        v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>
>>                        print *, v
>>
>>                        col(MatStencil_i,num) = i
>>
>>                        col(MatStencil_j,num) = j
>>
>>                        !num = num + 1
>>
>>                        call
>>         MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>
>>         I do not get any more out of range error. However,my ans is
>>         still different from that of ex29 in C.
>>
>>
>>     This is very simple. You have an error in your code. Checking it
>>     is very simple: run the code and
>>     break in MatSetValues(). Make sure ex29 makes calls with exactly
>>     the same indices as your ex29f.
>>
>>         Matt
>>
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/51e90ef3/attachment.htm>

From knepley at gmail.com  Fri May  4 08:05:00 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 May 2012 09:05:00 -0400
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
	ksp
In-Reply-To: <4FA3D294.5050202@gmail.com>
References: <4FA19C3F.5090604@gmail.com>
	<4FA3D294.5050202@gmail.com>
Message-ID: <CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>

On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng <zonexo at gmail.com> wrote:

>
> Hi,
>
> Is there anything else I can try to get it working right?
>

The MatGetNullSpaceRemove() is missing.

   Matt


> Thanks?
>
>
> On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>
> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>  Hi,
>>
>> I did a MatView and VecView on both C and Fortran, right after Mat and
>> Vec assembly. I have attached the printout below. They are exactly the
>> same, but yet the result is different in Neumann condition. However, the
>> dirichlet condition gives the correct ans. Is there anything else that
>> could be wrong even if the Mat and Vec are the same?
>>
>
>  Did you set the null space for the matrix when you have Neumann
> conditions?
>
> Yes, for the matrix, I set as:
>
> call
> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>
>     call MatSetNullSpace(jac,nullspace,ierr)
>
>     call MatNullSpaceDestroy(nullspace,ierr)
>
> for the Vec,
>
> call
> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>
>     *!call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)*
>
>     call MatNullSpaceDestroy(nullspace,ierr)
>
> MatNullSpaceRemove was comment out because there's error during linking
>
>
>
>     Matt
>
>
>>  Thanks!
>>
>> Fortran:
>>
>> Matrix Object: 1 MPI processes
>>   type: seqaij
>> row 0: (0, 2)  (1, -1)  (3, -1)
>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>> row 2: (1, -1)  (2, 2)  (5, -1)
>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>> row 6: (3, -1)  (6, 2)  (7, -1)
>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>> row 8: (5, -1)  (7, -1)  (8, 2)
>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>   type: mpi
>> Process [0]
>> 0.25
>> 0.0205213
>> 1.135e-005
>> 0.0205213
>> 0.00168449
>> 9.31663e-007
>> 1.135e-005
>> 9.31663e-007
>> 5.15289e-010
>> Vector Object:Vec_0000000084000000_1 1 MPI processes
>>   type: mpi
>> Process [0]
>> 0.14924
>> 0.0242397
>> -0.0260347
>> 0.0242397
>> -0.0256192
>> -0.0400102
>> -0.0260347
>> -0.0400102
>> -0.0400102
>> Press any key to continue . . .
>>
>> C:
>>
>> Matrix Object: 1 MPI processes
>>   type: seqaij
>> row 0: (0, 2)  (1, -1)  (3, -1)
>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>> row 2: (1, -1)  (2, 2)  (5, -1)
>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>> row 6: (3, -1)  (6, 2)  (7, -1)
>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>> row 8: (5, -1)  (7, -1)  (8, 2)
>> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>   type: mpi
>> Process [0]
>> 0.25
>> 0.0205212
>> 1.135e-05
>> 0.0205212
>> 0.00168449
>> 9.31663e-07
>> 1.135e-05
>> 9.31663e-07
>> 5.15288e-10
>> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>   type: mpi
>> Process [0]
>> 0.139311
>> 0.0305751
>> -0.0220633
>> 0.0305751
>> -0.0135158
>> -0.042185
>> -0.0220633
>> -0.042185
>> -0.058449
>>
>>
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>
>> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Do you mean my method is wrong?
>>>
>>> I am following the template of ex22f,
>>>
>>> where the variables are declared as :
>>>
>>> PetscScalar  v(5)
>>>
>>> MatStencil   row(4),col(4,5)
>>>
>>> Hence,
>>>
>>> for the neumann BC
>>>
>>> num = 1
>>>
>>>                if (j/=0) then
>>>
>>>                    v(num) = -rho*HxdHy
>>>
>>>                    col(MatStencil_i,num) = i
>>>
>>>                    col(MatStencil_j,num) = j-1
>>>
>>>                    num = num + 1
>>>
>>>                end if
>>>
>>>                if (i/=0) then
>>>
>>>                    v(num) = -rho*HydHx
>>>
>>>                    col(MatStencil_i,num) = i-1
>>>
>>>                    col(MatStencil_j,num) = j
>>>
>>>                    num = num + 1
>>>
>>>                end if
>>>
>>>                if (i/=mx-1) then
>>>
>>>                    v(num) = -rho*HydHx
>>>
>>>                    col(MatStencil_i,num) = i+1
>>>
>>>                    col(MatStencil_j,num) = j
>>>
>>>                    num = num + 1
>>>
>>>                end if
>>>
>>>                if (j/=my-1) then
>>>
>>>                    v(num) = -rho*HxdHy
>>>
>>>                    col(MatStencil_i,num) = i
>>>
>>>                    col(MatStencil_j,num) = j+1
>>>
>>>                    num = num + 1
>>>
>>>                end if
>>>
>>>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>
>>>                print *, v
>>>
>>>                col(MatStencil_i,num) = i
>>>
>>>                col(MatStencil_j,num) = j
>>>
>>>                !num = num + 1
>>>
>>>                call
>>> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>
>>> I do not get any more out of range error. However,my ans is still
>>> different from that of ex29 in C.
>>>
>>
>>  This is very simple. You have an error in your code. Checking it is
>> very simple: run the code and
>> break in MatSetValues(). Make sure ex29 makes calls with exactly the same
>> indices as your ex29f.
>>
>>      Matt
>>
>>
>>> Yours sincerely,
>>>
>>> TAY wee-beng
>>>
>>
>>   --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/da7c7fc7/attachment-0001.htm>

From zonexo at gmail.com  Fri May  4 10:05:58 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Fri, 04 May 2012 17:05:58 +0200
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
 ksp
In-Reply-To: <CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
Message-ID: <4FA3F056.4090406@gmail.com>


On 4/5/2012 3:05 PM, Matthew Knepley wrote:
> On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>
>     Hi,
>
>     Is there anything else I can try to get it working right?
>
>
> The MatGetNullSpaceRemove() is missing.
Where should I add MatGetNullSpaceRemove and what are its syntax? I 
googled but there's no results.

Thanks.
>
>    Matt
>
>     Thanks?
>
>
>     On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>     On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         I did a MatView and VecView on both C and Fortran, right
>>         after Mat and Vec assembly. I have attached the printout
>>         below. They are exactly the same, but yet the result is
>>         different in Neumann condition. However, the dirichlet
>>         condition gives the correct ans. Is there anything else that
>>         could be wrong even if the Mat and Vec are the same?
>>
>>
>>     Did you set the null space for the matrix when you have Neumann
>>     conditions?
>     Yes, for the matrix, I set as:
>
>     call
>     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>
>         call MatSetNullSpace(jac,nullspace,ierr)
>
>         call MatNullSpaceDestroy(nullspace,ierr)
>
>     for the Vec,
>
>     call
>     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>
>     *!call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)*
>
>         call MatNullSpaceDestroy(nullspace,ierr)
>
>     MatNullSpaceRemove was comment out because there's error during
>     linking
>
>
>>
>>        Matt
>>
>>         Thanks!
>>
>>         Fortran:
>>
>>         Matrix Object: 1 MPI processes
>>           type: seqaij
>>         row 0: (0, 2)  (1, -1)  (3, -1)
>>         row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>         row 2: (1, -1)  (2, 2)  (5, -1)
>>         row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>         row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>         row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>         row 6: (3, -1)  (6, 2)  (7, -1)
>>         row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>         row 8: (5, -1)  (7, -1)  (8, 2)
>>         Vector Object:Vec_0000000084000000_0 1 MPI processes
>>           type: mpi
>>         Process [0]
>>         0.25
>>         0.0205213
>>         1.135e-005
>>         0.0205213
>>         0.00168449
>>         9.31663e-007
>>         1.135e-005
>>         9.31663e-007
>>         5.15289e-010
>>         Vector Object:Vec_0000000084000000_1 1 MPI processes
>>           type: mpi
>>         Process [0]
>>         0.14924
>>         0.0242397
>>         -0.0260347
>>         0.0242397
>>         -0.0256192
>>         -0.0400102
>>         -0.0260347
>>         -0.0400102
>>         -0.0400102
>>         Press any key to continue . . .
>>
>>         C:
>>
>>         Matrix Object: 1 MPI processes
>>           type: seqaij
>>         row 0: (0, 2)  (1, -1)  (3, -1)
>>         row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>         row 2: (1, -1)  (2, 2)  (5, -1)
>>         row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>         row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>         row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>         row 6: (3, -1)  (6, 2)  (7, -1)
>>         row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>         row 8: (5, -1)  (7, -1)  (8, 2)
>>         Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>           type: mpi
>>         Process [0]
>>         0.25
>>         0.0205212
>>         1.135e-05
>>         0.0205212
>>         0.00168449
>>         9.31663e-07
>>         1.135e-05
>>         9.31663e-07
>>         5.15288e-10
>>         Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>           type: mpi
>>         Process [0]
>>         0.139311
>>         0.0305751
>>         -0.0220633
>>         0.0305751
>>         -0.0135158
>>         -0.042185
>>         -0.0220633
>>         -0.042185
>>         -0.058449
>>
>>
>>
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>
>>         On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>         On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng
>>>         <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>
>>>             Hi,
>>>
>>>             Do you mean my method is wrong?
>>>
>>>             I am following the template of ex22f,
>>>
>>>             where the variables are declared as :
>>>
>>>             PetscScalar  v(5)
>>>
>>>             MatStencil   row(4),col(4,5)
>>>
>>>             Hence,
>>>
>>>             for the neumann BC
>>>
>>>             num = 1
>>>
>>>                            if (j/=0) then
>>>
>>>                                v(num) = -rho*HxdHy
>>>
>>>                                col(MatStencil_i,num) = i
>>>
>>>                                col(MatStencil_j,num) = j-1
>>>
>>>                                num = num + 1
>>>
>>>                            end if
>>>
>>>                            if (i/=0) then
>>>
>>>                                v(num) = -rho*HydHx
>>>
>>>                                col(MatStencil_i,num) = i-1
>>>
>>>                                col(MatStencil_j,num) = j
>>>
>>>                                num = num + 1
>>>
>>>                            end if
>>>
>>>                            if (i/=mx-1) then
>>>
>>>                                v(num) = -rho*HydHx
>>>
>>>                                col(MatStencil_i,num) = i+1
>>>
>>>                                col(MatStencil_j,num) = j
>>>
>>>                                num = num + 1
>>>
>>>                            end if
>>>
>>>                            if (j/=my-1) then
>>>
>>>                                v(num) = -rho*HxdHy
>>>
>>>                                col(MatStencil_i,num) = i
>>>
>>>                                col(MatStencil_j,num) = j+1
>>>
>>>                                num = num + 1
>>>
>>>                            end if
>>>
>>>                            v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>
>>>                            print *, v
>>>
>>>                            col(MatStencil_i,num) = i
>>>
>>>                            col(MatStencil_j,num) = j
>>>
>>>                            !num = num + 1
>>>
>>>                            call
>>>             MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>
>>>             I do not get any more out of range error. However,my ans
>>>             is still different from that of ex29 in C.
>>>
>>>
>>>         This is very simple. You have an error in your code.
>>>         Checking it is very simple: run the code and
>>>         break in MatSetValues(). Make sure ex29 makes calls with
>>>         exactly the same indices as your ex29f.
>>>
>>>             Matt
>>>
>>>             Yours sincerely,
>>>
>>>             TAY wee-beng
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/ed0eb2a7/attachment.htm>

From knepley at gmail.com  Fri May  4 10:17:21 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 May 2012 11:17:21 -0400
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
	ksp
In-Reply-To: <4FA3F056.4090406@gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
Message-ID: <CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>

On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng <zonexo at gmail.com> wrote:

>
> On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>
> On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>
>> Hi,
>>
>> Is there anything else I can try to get it working right?
>>
>
>  The MatGetNullSpaceRemove() is missing.
>
> Where should I add MatGetNullSpaceRemove and what are its syntax? I
> googled but there's no results.
>

Fixed in p;etsc-dev:


http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html


     Matt


> Thanks.
>
>
>     Matt
>
>
>>  Thanks?
>>
>>
>> On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>
>> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>>  Hi,
>>>
>>> I did a MatView and VecView on both C and Fortran, right after Mat and
>>> Vec assembly. I have attached the printout below. They are exactly the
>>> same, but yet the result is different in Neumann condition. However, the
>>> dirichlet condition gives the correct ans. Is there anything else that
>>> could be wrong even if the Mat and Vec are the same?
>>>
>>
>>  Did you set the null space for the matrix when you have Neumann
>> conditions?
>>
>> Yes, for the matrix, I set as:
>>
>> call
>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>
>>     call MatSetNullSpace(jac,nullspace,ierr)
>>
>>     call MatNullSpaceDestroy(nullspace,ierr)
>>
>> for the Vec,
>>
>> call
>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>
>>     *!call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)*
>>
>>     call MatNullSpaceDestroy(nullspace,ierr)
>>
>> MatNullSpaceRemove was comment out because there's error during linking
>>
>>
>>
>>     Matt
>>
>>
>>>  Thanks!
>>>
>>> Fortran:
>>>
>>> Matrix Object: 1 MPI processes
>>>   type: seqaij
>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>   type: mpi
>>> Process [0]
>>> 0.25
>>> 0.0205213
>>> 1.135e-005
>>> 0.0205213
>>> 0.00168449
>>> 9.31663e-007
>>> 1.135e-005
>>> 9.31663e-007
>>> 5.15289e-010
>>> Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>   type: mpi
>>> Process [0]
>>> 0.14924
>>> 0.0242397
>>> -0.0260347
>>> 0.0242397
>>> -0.0256192
>>> -0.0400102
>>> -0.0260347
>>> -0.0400102
>>> -0.0400102
>>> Press any key to continue . . .
>>>
>>> C:
>>>
>>> Matrix Object: 1 MPI processes
>>>   type: seqaij
>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>   type: mpi
>>> Process [0]
>>> 0.25
>>> 0.0205212
>>> 1.135e-05
>>> 0.0205212
>>> 0.00168449
>>> 9.31663e-07
>>> 1.135e-05
>>> 9.31663e-07
>>> 5.15288e-10
>>> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>   type: mpi
>>> Process [0]
>>> 0.139311
>>> 0.0305751
>>> -0.0220633
>>> 0.0305751
>>> -0.0135158
>>> -0.042185
>>> -0.0220633
>>> -0.042185
>>> -0.058449
>>>
>>>
>>>
>>> Yours sincerely,
>>>
>>> TAY wee-beng
>>>
>>>
>>> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>
>>> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Do you mean my method is wrong?
>>>>
>>>> I am following the template of ex22f,
>>>>
>>>> where the variables are declared as :
>>>>
>>>> PetscScalar  v(5)
>>>>
>>>> MatStencil   row(4),col(4,5)
>>>>
>>>> Hence,
>>>>
>>>> for the neumann BC
>>>>
>>>> num = 1
>>>>
>>>>                if (j/=0) then
>>>>
>>>>                    v(num) = -rho*HxdHy
>>>>
>>>>                    col(MatStencil_i,num) = i
>>>>
>>>>                    col(MatStencil_j,num) = j-1
>>>>
>>>>                    num = num + 1
>>>>
>>>>                end if
>>>>
>>>>                if (i/=0) then
>>>>
>>>>                    v(num) = -rho*HydHx
>>>>
>>>>                    col(MatStencil_i,num) = i-1
>>>>
>>>>                    col(MatStencil_j,num) = j
>>>>
>>>>                    num = num + 1
>>>>
>>>>                end if
>>>>
>>>>                if (i/=mx-1) then
>>>>
>>>>                    v(num) = -rho*HydHx
>>>>
>>>>                    col(MatStencil_i,num) = i+1
>>>>
>>>>                    col(MatStencil_j,num) = j
>>>>
>>>>                    num = num + 1
>>>>
>>>>                end if
>>>>
>>>>                if (j/=my-1) then
>>>>
>>>>                    v(num) = -rho*HxdHy
>>>>
>>>>                    col(MatStencil_i,num) = i
>>>>
>>>>                    col(MatStencil_j,num) = j+1
>>>>
>>>>                    num = num + 1
>>>>
>>>>                end if
>>>>
>>>>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>
>>>>                print *, v
>>>>
>>>>                col(MatStencil_i,num) = i
>>>>
>>>>                col(MatStencil_j,num) = j
>>>>
>>>>                !num = num + 1
>>>>
>>>>                call
>>>> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>
>>>> I do not get any more out of range error. However,my ans is still
>>>> different from that of ex29 in C.
>>>>
>>>
>>>  This is very simple. You have an error in your code. Checking it is
>>> very simple: run the code and
>>> break in MatSetValues(). Make sure ex29 makes calls with exactly the
>>> same indices as your ex29f.
>>>
>>>      Matt
>>>
>>>
>>>> Yours sincerely,
>>>>
>>>> TAY wee-beng
>>>>
>>>
>>>    --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/6320fe37/attachment-0001.htm>

From zonexo at gmail.com  Fri May  4 14:01:51 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Fri, 04 May 2012 21:01:51 +0200
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
 ksp
In-Reply-To: <CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
Message-ID: <4FA4279F.1030204@gmail.com>


On 4/5/2012 5:17 PM, Matthew Knepley wrote:
> On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>
>     On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>     On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>
>>         Hi,
>>
>>         Is there anything else I can try to get it working right?
>>
>>
>>     The MatGetNullSpaceRemove() is missing.
>     Where should I add MatGetNullSpaceRemove and what are its syntax?
>     I googled but there's no results.
>
>
> Fixed in p;etsc-dev:
>
> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html

I just compiled the updated petsc-dev but I got the same error msg when 
I use:

call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)

error LNK2019: unresolved external symbol MATNULLSPACEREMOVE referenced 
in function COMPUTERHS
1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120: 1 unresolved 
externals
>
>
>      Matt
>
>     Thanks.
>>
>>        Matt
>>
>>         Thanks?
>>
>>
>>         On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>         On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng
>>>         <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>
>>>             Hi,
>>>
>>>             I did a MatView and VecView on both C and Fortran, right
>>>             after Mat and Vec assembly. I have attached the printout
>>>             below. They are exactly the same, but yet the result is
>>>             different in Neumann condition. However, the dirichlet
>>>             condition gives the correct ans. Is there anything else
>>>             that could be wrong even if the Mat and Vec are the same?
>>>
>>>
>>>         Did you set the null space for the matrix when you have
>>>         Neumann conditions?
>>         Yes, for the matrix, I set as:
>>
>>         call
>>         MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>
>>             call MatSetNullSpace(jac,nullspace,ierr)
>>
>>             call MatNullSpaceDestroy(nullspace,ierr)
>>
>>         for the Vec,
>>
>>         call
>>         MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>
>>         *!call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)*
>>
>>             call MatNullSpaceDestroy(nullspace,ierr)
>>
>>         MatNullSpaceRemove was comment out because there's error
>>         during linking
>>
>>
>>>
>>>            Matt
>>>
>>>             Thanks!
>>>
>>>             Fortran:
>>>
>>>             Matrix Object: 1 MPI processes
>>>               type: seqaij
>>>             row 0: (0, 2)  (1, -1)  (3, -1)
>>>             row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>             row 2: (1, -1)  (2, 2)  (5, -1)
>>>             row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>             row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>             row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>             row 6: (3, -1)  (6, 2)  (7, -1)
>>>             row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>             row 8: (5, -1)  (7, -1)  (8, 2)
>>>             Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>               type: mpi
>>>             Process [0]
>>>             0.25
>>>             0.0205213
>>>             1.135e-005
>>>             0.0205213
>>>             0.00168449
>>>             9.31663e-007
>>>             1.135e-005
>>>             9.31663e-007
>>>             5.15289e-010
>>>             Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>               type: mpi
>>>             Process [0]
>>>             0.14924
>>>             0.0242397
>>>             -0.0260347
>>>             0.0242397
>>>             -0.0256192
>>>             -0.0400102
>>>             -0.0260347
>>>             -0.0400102
>>>             -0.0400102
>>>             Press any key to continue . . .
>>>
>>>             C:
>>>
>>>             Matrix Object: 1 MPI processes
>>>               type: seqaij
>>>             row 0: (0, 2)  (1, -1)  (3, -1)
>>>             row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>             row 2: (1, -1)  (2, 2)  (5, -1)
>>>             row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>             row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>             row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>             row 6: (3, -1)  (6, 2)  (7, -1)
>>>             row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>             row 8: (5, -1)  (7, -1)  (8, 2)
>>>             Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>               type: mpi
>>>             Process [0]
>>>             0.25
>>>             0.0205212
>>>             1.135e-05
>>>             0.0205212
>>>             0.00168449
>>>             9.31663e-07
>>>             1.135e-05
>>>             9.31663e-07
>>>             5.15288e-10
>>>             Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>               type: mpi
>>>             Process [0]
>>>             0.139311
>>>             0.0305751
>>>             -0.0220633
>>>             0.0305751
>>>             -0.0135158
>>>             -0.042185
>>>             -0.0220633
>>>             -0.042185
>>>             -0.058449
>>>
>>>
>>>
>>>             Yours sincerely,
>>>
>>>             TAY wee-beng
>>>
>>>
>>>             On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>             On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng
>>>>             <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>>
>>>>                 Hi,
>>>>
>>>>                 Do you mean my method is wrong?
>>>>
>>>>                 I am following the template of ex22f,
>>>>
>>>>                 where the variables are declared as :
>>>>
>>>>                 PetscScalar  v(5)
>>>>
>>>>                 MatStencil   row(4),col(4,5)
>>>>
>>>>                 Hence,
>>>>
>>>>                 for the neumann BC
>>>>
>>>>                 num = 1
>>>>
>>>>                                if (j/=0) then
>>>>
>>>>                                    v(num) = -rho*HxdHy
>>>>
>>>>                                    col(MatStencil_i,num) = i
>>>>
>>>>                                    col(MatStencil_j,num) = j-1
>>>>
>>>>                                    num = num + 1
>>>>
>>>>                                end if
>>>>
>>>>                                if (i/=0) then
>>>>
>>>>                                    v(num) = -rho*HydHx
>>>>
>>>>                                    col(MatStencil_i,num) = i-1
>>>>
>>>>                                    col(MatStencil_j,num) = j
>>>>
>>>>                                    num = num + 1
>>>>
>>>>                                end if
>>>>
>>>>                                if (i/=mx-1) then
>>>>
>>>>                                    v(num) = -rho*HydHx
>>>>
>>>>                                    col(MatStencil_i,num) = i+1
>>>>
>>>>                                    col(MatStencil_j,num) = j
>>>>
>>>>                                    num = num + 1
>>>>
>>>>                                end if
>>>>
>>>>                                if (j/=my-1) then
>>>>
>>>>                                    v(num) = -rho*HxdHy
>>>>
>>>>                                    col(MatStencil_i,num) = i
>>>>
>>>>                                    col(MatStencil_j,num) = j+1
>>>>
>>>>                                    num = num + 1
>>>>
>>>>                                end if
>>>>
>>>>                                v(num) = ((num-1)/2.0)*rho*(HxdHy +
>>>>                 HydHx)
>>>>
>>>>                                print *, v
>>>>
>>>>                                col(MatStencil_i,num) = i
>>>>
>>>>                                col(MatStencil_j,num) = j
>>>>
>>>>                                !num = num + 1
>>>>
>>>>                                call
>>>>                 MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>
>>>>                 I do not get any more out of range error.
>>>>                 However,my ans is still different from that of ex29
>>>>                 in C.
>>>>
>>>>
>>>>             This is very simple. You have an error in your code.
>>>>             Checking it is very simple: run the code and
>>>>             break in MatSetValues(). Make sure ex29 makes calls
>>>>             with exactly the same indices as your ex29f.
>>>>
>>>>                 Matt
>>>>
>>>>                 Yours sincerely,
>>>>
>>>>                 TAY wee-beng
>>>>
>>>>
>>>>             -- 
>>>>             What most experimenters take for granted before they
>>>>             begin their experiments is infinitely more interesting
>>>>             than any results to which their experiments lead.
>>>>             -- Norbert Wiener
>>>
>>>
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/26d06297/attachment.htm>

From john.mousel at gmail.com  Fri May  4 14:09:02 2012
From: john.mousel at gmail.com (John Mousel)
Date: Fri, 4 May 2012 14:09:02 -0500
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
	ksp
In-Reply-To: <4FA4279F.1030204@gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
Message-ID: <CA+zegp=aszZ28auQMG9h9XOf+WNXDKKxMBJg9+ekeeDiczG=YQ@mail.gmail.com>

You have to pass PETSC_NULL_OBJECT instead of PETSC_NULL in Fortran.


>From the man pages:

"This macro does not exist in Fortran; you must use PETSC_NULL_INTEGER,
PETSC_NULL_DOUBLE_PRECISION, PETSC_NULL_FUNCTION, PETSC_NULL_OBJECT etc "


http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PETSC_NULL.html#PETSC_NULL


John


On Fri, May 4, 2012 at 2:01 PM, TAY wee-beng <zonexo at gmail.com> wrote:

>
> On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>
> On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>
>> On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>
>> On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>>
>>> Hi,
>>>
>>> Is there anything else I can try to get it working right?
>>>
>>
>>  The MatGetNullSpaceRemove() is missing.
>>
>> Where should I add MatGetNullSpaceRemove and what are its syntax? I
>> googled but there's no results.
>>
>
>  Fixed in p;etsc-dev:
>
>
> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>
>
> I just compiled the updated petsc-dev but I got the same error msg when I
> use:
>
> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>
> error LNK2019: unresolved external symbol MATNULLSPACEREMOVE referenced in
> function COMPUTERHS
> 1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120: 1 unresolved
> externals
>
>
>
>       Matt
>
>
>>  Thanks.
>>
>>
>>     Matt
>>
>>
>>>  Thanks?
>>>
>>>
>>> On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>
>>> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>
>>>>  Hi,
>>>>
>>>> I did a MatView and VecView on both C and Fortran, right after Mat and
>>>> Vec assembly. I have attached the printout below. They are exactly the
>>>> same, but yet the result is different in Neumann condition. However, the
>>>> dirichlet condition gives the correct ans. Is there anything else that
>>>> could be wrong even if the Mat and Vec are the same?
>>>>
>>>
>>>  Did you set the null space for the matrix when you have Neumann
>>> conditions?
>>>
>>> Yes, for the matrix, I set as:
>>>
>>> call
>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>
>>>     call MatSetNullSpace(jac,nullspace,ierr)
>>>
>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>> for the Vec,
>>>
>>> call
>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>
>>>     *!call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)*
>>>
>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>> MatNullSpaceRemove was comment out because there's error during linking
>>>
>>>
>>>
>>>     Matt
>>>
>>>
>>>>  Thanks!
>>>>
>>>> Fortran:
>>>>
>>>> Matrix Object: 1 MPI processes
>>>>   type: seqaij
>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>   type: mpi
>>>> Process [0]
>>>> 0.25
>>>> 0.0205213
>>>> 1.135e-005
>>>> 0.0205213
>>>> 0.00168449
>>>> 9.31663e-007
>>>> 1.135e-005
>>>> 9.31663e-007
>>>> 5.15289e-010
>>>> Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>>   type: mpi
>>>> Process [0]
>>>> 0.14924
>>>> 0.0242397
>>>> -0.0260347
>>>> 0.0242397
>>>> -0.0256192
>>>> -0.0400102
>>>> -0.0260347
>>>> -0.0400102
>>>> -0.0400102
>>>> Press any key to continue . . .
>>>>
>>>> C:
>>>>
>>>> Matrix Object: 1 MPI processes
>>>>   type: seqaij
>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>>   type: mpi
>>>> Process [0]
>>>> 0.25
>>>> 0.0205212
>>>> 1.135e-05
>>>> 0.0205212
>>>> 0.00168449
>>>> 9.31663e-07
>>>> 1.135e-05
>>>> 9.31663e-07
>>>> 5.15288e-10
>>>> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>>   type: mpi
>>>> Process [0]
>>>> 0.139311
>>>> 0.0305751
>>>> -0.0220633
>>>> 0.0305751
>>>> -0.0135158
>>>> -0.042185
>>>> -0.0220633
>>>> -0.042185
>>>> -0.058449
>>>>
>>>>
>>>>
>>>> Yours sincerely,
>>>>
>>>> TAY wee-beng
>>>>
>>>>
>>>> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>
>>>> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Do you mean my method is wrong?
>>>>>
>>>>> I am following the template of ex22f,
>>>>>
>>>>> where the variables are declared as :
>>>>>
>>>>> PetscScalar  v(5)
>>>>>
>>>>> MatStencil   row(4),col(4,5)
>>>>>
>>>>> Hence,
>>>>>
>>>>> for the neumann BC
>>>>>
>>>>> num = 1
>>>>>
>>>>>                if (j/=0) then
>>>>>
>>>>>                    v(num) = -rho*HxdHy
>>>>>
>>>>>                    col(MatStencil_i,num) = i
>>>>>
>>>>>                    col(MatStencil_j,num) = j-1
>>>>>
>>>>>                    num = num + 1
>>>>>
>>>>>                end if
>>>>>
>>>>>                if (i/=0) then
>>>>>
>>>>>                    v(num) = -rho*HydHx
>>>>>
>>>>>                    col(MatStencil_i,num) = i-1
>>>>>
>>>>>                    col(MatStencil_j,num) = j
>>>>>
>>>>>                    num = num + 1
>>>>>
>>>>>                end if
>>>>>
>>>>>                if (i/=mx-1) then
>>>>>
>>>>>                    v(num) = -rho*HydHx
>>>>>
>>>>>                    col(MatStencil_i,num) = i+1
>>>>>
>>>>>                    col(MatStencil_j,num) = j
>>>>>
>>>>>                    num = num + 1
>>>>>
>>>>>                end if
>>>>>
>>>>>                if (j/=my-1) then
>>>>>
>>>>>                    v(num) = -rho*HxdHy
>>>>>
>>>>>                    col(MatStencil_i,num) = i
>>>>>
>>>>>                    col(MatStencil_j,num) = j+1
>>>>>
>>>>>                    num = num + 1
>>>>>
>>>>>                end if
>>>>>
>>>>>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>>
>>>>>                print *, v
>>>>>
>>>>>                col(MatStencil_i,num) = i
>>>>>
>>>>>                col(MatStencil_j,num) = j
>>>>>
>>>>>                !num = num + 1
>>>>>
>>>>>                call
>>>>> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>>
>>>>> I do not get any more out of range error. However,my ans is still
>>>>> different from that of ex29 in C.
>>>>>
>>>>
>>>>  This is very simple. You have an error in your code. Checking it is
>>>> very simple: run the code and
>>>> break in MatSetValues(). Make sure ex29 makes calls with exactly the
>>>> same indices as your ex29f.
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Yours sincerely,
>>>>>
>>>>> TAY wee-beng
>>>>>
>>>>
>>>>    --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>>
>>>
>>>
>>>  --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/a2bebe2b/attachment-0001.htm>

From knepley at gmail.com  Fri May  4 14:11:55 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 May 2012 15:11:55 -0400
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
	ksp
In-Reply-To: <4FA4279F.1030204@gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
Message-ID: <CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>

On Fri, May 4, 2012 at 3:01 PM, TAY wee-beng <zonexo at gmail.com> wrote:

>
> On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>
> On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>
>> On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>
>> On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>>
>>> Hi,
>>>
>>> Is there anything else I can try to get it working right?
>>>
>>
>>  The MatGetNullSpaceRemove() is missing.
>>
>> Where should I add MatGetNullSpaceRemove and what are its syntax? I
>> googled but there's no results.
>>
>
>  Fixed in p;etsc-dev:
>
>
> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>
>
> I just compiled the updated petsc-dev but I got the same error msg when I
> use:
>
> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>
> error LNK2019: unresolved external symbol MATNULLSPACEREMOVE referenced in
> function COMPUTERHS
> 1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120: 1 unresolved
> externals
>

That function is in:

  src/mat/interface/ftn-custom/zmatrixf.c

Did that compile? Can you see the symbol in libpetsc.a?

    Matt


>       Matt
>
>
>>  Thanks.
>>
>>
>>     Matt
>>
>>
>>>  Thanks?
>>>
>>>
>>> On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>
>>> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>
>>>>  Hi,
>>>>
>>>> I did a MatView and VecView on both C and Fortran, right after Mat and
>>>> Vec assembly. I have attached the printout below. They are exactly the
>>>> same, but yet the result is different in Neumann condition. However, the
>>>> dirichlet condition gives the correct ans. Is there anything else that
>>>> could be wrong even if the Mat and Vec are the same?
>>>>
>>>
>>>  Did you set the null space for the matrix when you have Neumann
>>> conditions?
>>>
>>> Yes, for the matrix, I set as:
>>>
>>> call
>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>
>>>     call MatSetNullSpace(jac,nullspace,ierr)
>>>
>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>> for the Vec,
>>>
>>> call
>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>
>>>     *!call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)*
>>>
>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>> MatNullSpaceRemove was comment out because there's error during linking
>>>
>>>
>>>
>>>     Matt
>>>
>>>
>>>>  Thanks!
>>>>
>>>> Fortran:
>>>>
>>>> Matrix Object: 1 MPI processes
>>>>   type: seqaij
>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>   type: mpi
>>>> Process [0]
>>>> 0.25
>>>> 0.0205213
>>>> 1.135e-005
>>>> 0.0205213
>>>> 0.00168449
>>>> 9.31663e-007
>>>> 1.135e-005
>>>> 9.31663e-007
>>>> 5.15289e-010
>>>> Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>>   type: mpi
>>>> Process [0]
>>>> 0.14924
>>>> 0.0242397
>>>> -0.0260347
>>>> 0.0242397
>>>> -0.0256192
>>>> -0.0400102
>>>> -0.0260347
>>>> -0.0400102
>>>> -0.0400102
>>>> Press any key to continue . . .
>>>>
>>>> C:
>>>>
>>>> Matrix Object: 1 MPI processes
>>>>   type: seqaij
>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>>   type: mpi
>>>> Process [0]
>>>> 0.25
>>>> 0.0205212
>>>> 1.135e-05
>>>> 0.0205212
>>>> 0.00168449
>>>> 9.31663e-07
>>>> 1.135e-05
>>>> 9.31663e-07
>>>> 5.15288e-10
>>>> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>>   type: mpi
>>>> Process [0]
>>>> 0.139311
>>>> 0.0305751
>>>> -0.0220633
>>>> 0.0305751
>>>> -0.0135158
>>>> -0.042185
>>>> -0.0220633
>>>> -0.042185
>>>> -0.058449
>>>>
>>>>
>>>>
>>>> Yours sincerely,
>>>>
>>>> TAY wee-beng
>>>>
>>>>
>>>> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>
>>>> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Do you mean my method is wrong?
>>>>>
>>>>> I am following the template of ex22f,
>>>>>
>>>>> where the variables are declared as :
>>>>>
>>>>> PetscScalar  v(5)
>>>>>
>>>>> MatStencil   row(4),col(4,5)
>>>>>
>>>>> Hence,
>>>>>
>>>>> for the neumann BC
>>>>>
>>>>> num = 1
>>>>>
>>>>>                if (j/=0) then
>>>>>
>>>>>                    v(num) = -rho*HxdHy
>>>>>
>>>>>                    col(MatStencil_i,num) = i
>>>>>
>>>>>                    col(MatStencil_j,num) = j-1
>>>>>
>>>>>                    num = num + 1
>>>>>
>>>>>                end if
>>>>>
>>>>>                if (i/=0) then
>>>>>
>>>>>                    v(num) = -rho*HydHx
>>>>>
>>>>>                    col(MatStencil_i,num) = i-1
>>>>>
>>>>>                    col(MatStencil_j,num) = j
>>>>>
>>>>>                    num = num + 1
>>>>>
>>>>>                end if
>>>>>
>>>>>                if (i/=mx-1) then
>>>>>
>>>>>                    v(num) = -rho*HydHx
>>>>>
>>>>>                    col(MatStencil_i,num) = i+1
>>>>>
>>>>>                    col(MatStencil_j,num) = j
>>>>>
>>>>>                    num = num + 1
>>>>>
>>>>>                end if
>>>>>
>>>>>                if (j/=my-1) then
>>>>>
>>>>>                    v(num) = -rho*HxdHy
>>>>>
>>>>>                    col(MatStencil_i,num) = i
>>>>>
>>>>>                    col(MatStencil_j,num) = j+1
>>>>>
>>>>>                    num = num + 1
>>>>>
>>>>>                end if
>>>>>
>>>>>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>>
>>>>>                print *, v
>>>>>
>>>>>                col(MatStencil_i,num) = i
>>>>>
>>>>>                col(MatStencil_j,num) = j
>>>>>
>>>>>                !num = num + 1
>>>>>
>>>>>                call
>>>>> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>>
>>>>> I do not get any more out of range error. However,my ans is still
>>>>> different from that of ex29 in C.
>>>>>
>>>>
>>>>  This is very simple. You have an error in your code. Checking it is
>>>> very simple: run the code and
>>>> break in MatSetValues(). Make sure ex29 makes calls with exactly the
>>>> same indices as your ex29f.
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Yours sincerely,
>>>>>
>>>>> TAY wee-beng
>>>>>
>>>>
>>>>    --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>>
>>>
>>>
>>>  --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/5ac800ed/attachment.htm>

From zonexo at gmail.com  Fri May  4 14:13:00 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Fri, 04 May 2012 21:13:00 +0200
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
 ksp
In-Reply-To: <CA+zegp=aszZ28auQMG9h9XOf+WNXDKKxMBJg9+ekeeDiczG=YQ@mail.gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CA+zegp=aszZ28auQMG9h9XOf+WNXDKKxMBJg9+ekeeDiczG=YQ@mail.gmail.com>
Message-ID: <4FA42A3C.2030507@gmail.com>


On 4/5/2012 9:09 PM, John Mousel wrote:
> You have to pass PETSC_NULL_OBJECT instead of PETSC_NULL in Fortran.
>
>
> From the man pages:
>
> "This macro does not exist in Fortran; you must use 
> PETSC_NULL_INTEGER, PETSC_NULL_DOUBLE_PRECISION, PETSC_NULL_FUNCTION, 
> PETSC_NULL_OBJECT etc "
>
>
> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PETSC_NULL.html#PETSC_NULL
>
>
> John
>
Thanks for the suggestion, but they all didn't work.
>
> On Fri, May 4, 2012 at 2:01 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>
>     On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>>     On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>
>>         On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>>         On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng
>>>         <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>
>>>
>>>             Hi,
>>>
>>>             Is there anything else I can try to get it working right?
>>>
>>>
>>>         The MatGetNullSpaceRemove() is missing.
>>         Where should I add MatGetNullSpaceRemove and what are its
>>         syntax? I googled but there's no results.
>>
>>
>>     Fixed in p;etsc-dev:
>>
>>     http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>
>     I just compiled the updated petsc-dev but I got the same error msg
>     when I use:
>
>     call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>
>     error LNK2019: unresolved external symbol MATNULLSPACEREMOVE
>     referenced in function COMPUTERHS
>     1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120: 1
>     unresolved externals
>
>>
>>
>>          Matt
>>
>>         Thanks.
>>>
>>>            Matt
>>>
>>>             Thanks?
>>>
>>>
>>>             On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>>             On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng
>>>>             <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>>
>>>>                 Hi,
>>>>
>>>>                 I did a MatView and VecView on both C and Fortran,
>>>>                 right after Mat and Vec assembly. I have attached
>>>>                 the printout below. They are exactly the same, but
>>>>                 yet the result is different in Neumann condition.
>>>>                 However, the dirichlet condition gives the correct
>>>>                 ans. Is there anything else that could be wrong
>>>>                 even if the Mat and Vec are the same?
>>>>
>>>>
>>>>             Did you set the null space for the matrix when you have
>>>>             Neumann conditions?
>>>             Yes, for the matrix, I set as:
>>>
>>>             call
>>>             MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>
>>>                 call MatSetNullSpace(jac,nullspace,ierr)
>>>
>>>                 call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>>             for the Vec,
>>>
>>>             call
>>>             MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>
>>>             *!call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)*
>>>
>>>                 call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>>             MatNullSpaceRemove was comment out because there's error
>>>             during linking
>>>
>>>
>>>>
>>>>                Matt
>>>>
>>>>                 Thanks!
>>>>
>>>>                 Fortran:
>>>>
>>>>                 Matrix Object: 1 MPI processes
>>>>                   type: seqaij
>>>>                 row 0: (0, 2)  (1, -1)  (3, -1)
>>>>                 row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>                 row 2: (1, -1)  (2, 2)  (5, -1)
>>>>                 row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>                 row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>                 row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>                 row 6: (3, -1)  (6, 2)  (7, -1)
>>>>                 row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>                 row 8: (5, -1)  (7, -1)  (8, 2)
>>>>                 Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>                   type: mpi
>>>>                 Process [0]
>>>>                 0.25
>>>>                 0.0205213
>>>>                 1.135e-005
>>>>                 0.0205213
>>>>                 0.00168449
>>>>                 9.31663e-007
>>>>                 1.135e-005
>>>>                 9.31663e-007
>>>>                 5.15289e-010
>>>>                 Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>>                   type: mpi
>>>>                 Process [0]
>>>>                 0.14924
>>>>                 0.0242397
>>>>                 -0.0260347
>>>>                 0.0242397
>>>>                 -0.0256192
>>>>                 -0.0400102
>>>>                 -0.0260347
>>>>                 -0.0400102
>>>>                 -0.0400102
>>>>                 Press any key to continue . . .
>>>>
>>>>                 C:
>>>>
>>>>                 Matrix Object: 1 MPI processes
>>>>                   type: seqaij
>>>>                 row 0: (0, 2)  (1, -1)  (3, -1)
>>>>                 row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>                 row 2: (1, -1)  (2, 2)  (5, -1)
>>>>                 row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>                 row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>                 row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>                 row 6: (3, -1)  (6, 2)  (7, -1)
>>>>                 row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>                 row 8: (5, -1)  (7, -1)  (8, 2)
>>>>                 Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>>                   type: mpi
>>>>                 Process [0]
>>>>                 0.25
>>>>                 0.0205212
>>>>                 1.135e-05
>>>>                 0.0205212
>>>>                 0.00168449
>>>>                 9.31663e-07
>>>>                 1.135e-05
>>>>                 9.31663e-07
>>>>                 5.15288e-10
>>>>                 Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>>                   type: mpi
>>>>                 Process [0]
>>>>                 0.139311
>>>>                 0.0305751
>>>>                 -0.0220633
>>>>                 0.0305751
>>>>                 -0.0135158
>>>>                 -0.042185
>>>>                 -0.0220633
>>>>                 -0.042185
>>>>                 -0.058449
>>>>
>>>>
>>>>
>>>>                 Yours sincerely,
>>>>
>>>>                 TAY wee-beng
>>>>
>>>>
>>>>                 On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>>                 On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng
>>>>>                 <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>>>
>>>>>                     Hi,
>>>>>
>>>>>                     Do you mean my method is wrong?
>>>>>
>>>>>                     I am following the template of ex22f,
>>>>>
>>>>>                     where the variables are declared as :
>>>>>
>>>>>                     PetscScalar  v(5)
>>>>>
>>>>>                     MatStencil   row(4),col(4,5)
>>>>>
>>>>>                     Hence,
>>>>>
>>>>>                     for the neumann BC
>>>>>
>>>>>                     num = 1
>>>>>
>>>>>                                    if (j/=0) then
>>>>>
>>>>>                                        v(num) = -rho*HxdHy
>>>>>
>>>>>                                        col(MatStencil_i,num) = i
>>>>>
>>>>>                                        col(MatStencil_j,num) = j-1
>>>>>
>>>>>                                        num = num + 1
>>>>>
>>>>>                                    end if
>>>>>
>>>>>                                    if (i/=0) then
>>>>>
>>>>>                                        v(num) = -rho*HydHx
>>>>>
>>>>>                                        col(MatStencil_i,num) = i-1
>>>>>
>>>>>                                        col(MatStencil_j,num) = j
>>>>>
>>>>>                                        num = num + 1
>>>>>
>>>>>                                    end if
>>>>>
>>>>>                                    if (i/=mx-1) then
>>>>>
>>>>>                                        v(num) = -rho*HydHx
>>>>>
>>>>>                                        col(MatStencil_i,num) = i+1
>>>>>
>>>>>                                        col(MatStencil_j,num) = j
>>>>>
>>>>>                                        num = num + 1
>>>>>
>>>>>                                    end if
>>>>>
>>>>>                                    if (j/=my-1) then
>>>>>
>>>>>                                        v(num) = -rho*HxdHy
>>>>>
>>>>>                                        col(MatStencil_i,num) = i
>>>>>
>>>>>                                        col(MatStencil_j,num) = j+1
>>>>>
>>>>>                                        num = num + 1
>>>>>
>>>>>                                    end if
>>>>>
>>>>>                                    v(num) =
>>>>>                     ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>>
>>>>>                                    print *, v
>>>>>
>>>>>                                    col(MatStencil_i,num) = i
>>>>>
>>>>>                                    col(MatStencil_j,num) = j
>>>>>
>>>>>                                    !num = num + 1
>>>>>
>>>>>                                    call
>>>>>                     MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>>
>>>>>                     I do not get any more out of range error.
>>>>>                     However,my ans is still different from that of
>>>>>                     ex29 in C.
>>>>>
>>>>>
>>>>>                 This is very simple. You have an error in your
>>>>>                 code. Checking it is very simple: run the code and
>>>>>                 break in MatSetValues(). Make sure ex29 makes
>>>>>                 calls with exactly the same indices as your ex29f.
>>>>>
>>>>>                     Matt
>>>>>
>>>>>                     Yours sincerely,
>>>>>
>>>>>                     TAY wee-beng
>>>>>
>>>>>
>>>>>                 -- 
>>>>>                 What most experimenters take for granted before
>>>>>                 they begin their experiments is infinitely more
>>>>>                 interesting than any results to which their
>>>>>                 experiments lead.
>>>>>                 -- Norbert Wiener
>>>>
>>>>
>>>>
>>>>
>>>>             -- 
>>>>             What most experimenters take for granted before they
>>>>             begin their experiments is infinitely more interesting
>>>>             than any results to which their experiments lead.
>>>>             -- Norbert Wiener
>>>
>>>
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/16879c44/attachment-0001.htm>

From bsmith at mcs.anl.gov  Fri May  4 14:16:27 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 4 May 2012 14:16:27 -0500
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
	ksp
In-Reply-To: <CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
Message-ID: <5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>


   Do an hg pull and then run make in src/mat/interface/ftn-custom/

    Then it should link.

   Barry

   There was a E missing from the all caps name of the function.


On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:

> On Fri, May 4, 2012 at 3:01 PM, TAY wee-beng <zonexo at gmail.com> wrote:
> 
> On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>> On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>> 
>> On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>> On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> Is there anything else I can try to get it working right?
>>> 
>>> The MatGetNullSpaceRemove() is missing.
>> Where should I add MatGetNullSpaceRemove and what are its syntax? I googled but there's no results.
>> 
>> Fixed in p;etsc-dev:
>> 
>>   http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
> 
> I just compiled the updated petsc-dev but I got the same error msg when I use:
> 
> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
> 
> error LNK2019: unresolved external symbol MATNULLSPACEREMOVE referenced in function COMPUTERHS
> 1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120: 1 unresolved externals
> 
> That function is in:
> 
>   src/mat/interface/ftn-custom/zmatrixf.c
> 
> Did that compile? Can you see the symbol in libpetsc.a?
> 
>     Matt
>  
>>      Matt
>>  
>> Thanks.
>>> 
>>>    Matt
>>>  
>>> Thanks?
>>> 
>>> 
>>> On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>> Hi,
>>>> 
>>>> I did a MatView and VecView on both C and Fortran, right after Mat and Vec assembly. I have attached the printout below. They are exactly the same, but yet the result is different in Neumann condition. However, the dirichlet condition gives the correct ans. Is there anything else that could be wrong even if the Mat and Vec are the same?
>>>> 
>>>> Did you set the null space for the matrix when you have Neumann conditions?
>>> Yes, for the matrix, I set as:
>>> 
>>> call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>> 
>>>     call MatSetNullSpace(jac,nullspace,ierr)
>>> 
>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>> 
>>> for the Vec, 
>>> 
>>> call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>> 
>>>     !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>> 
>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>> 
>>> MatNullSpaceRemove was comment out because there's error during linking
>>> 
>>> 
>>>> 
>>>>    Matt
>>>>  
>>>> Thanks!
>>>> 
>>>> Fortran:
>>>> 
>>>> Matrix Object: 1 MPI processes
>>>>   type: seqaij
>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>   type: mpi
>>>> Process [0]
>>>> 0.25
>>>> 0.0205213
>>>> 1.135e-005
>>>> 0.0205213
>>>> 0.00168449
>>>> 9.31663e-007
>>>> 1.135e-005
>>>> 9.31663e-007
>>>> 5.15289e-010
>>>> Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>>   type: mpi
>>>> Process [0]
>>>> 0.14924
>>>> 0.0242397
>>>> -0.0260347
>>>> 0.0242397
>>>> -0.0256192
>>>> -0.0400102
>>>> -0.0260347
>>>> -0.0400102
>>>> -0.0400102
>>>> Press any key to continue . . .
>>>> 
>>>> C:
>>>> 
>>>> Matrix Object: 1 MPI processes
>>>>   type: seqaij
>>>> row 0: (0, 2)  (1, -1)  (3, -1) 
>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1) 
>>>> row 2: (1, -1)  (2, 2)  (5, -1) 
>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1) 
>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1) 
>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1) 
>>>> row 6: (3, -1)  (6, 2)  (7, -1) 
>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1) 
>>>> row 8: (5, -1)  (7, -1)  (8, 2) 
>>>> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>>   type: mpi
>>>> Process [0]
>>>> 0.25
>>>> 0.0205212
>>>> 1.135e-05
>>>> 0.0205212
>>>> 0.00168449
>>>> 9.31663e-07
>>>> 1.135e-05
>>>> 9.31663e-07
>>>> 5.15288e-10
>>>> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>>   type: mpi
>>>> Process [0]
>>>> 0.139311
>>>> 0.0305751
>>>> -0.0220633
>>>> 0.0305751
>>>> -0.0135158
>>>> -0.042185
>>>> -0.0220633
>>>> -0.042185
>>>> -0.058449
>>>> 
>>>> 
>>>> 
>>>> Yours sincerely,
>>>> 
>>>> TAY wee-beng
>>>> 
>>>> 
>>>> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>> Hi,
>>>>> 
>>>>> Do you mean my method is wrong?
>>>>> 
>>>>> I am following the template of ex22f,
>>>>> 
>>>>> where the variables are declared as :
>>>>> 
>>>>> PetscScalar  v(5)
>>>>> 
>>>>> MatStencil   row(4),col(4,5)
>>>>> 
>>>>> Hence,
>>>>> 
>>>>> for the neumann BC
>>>>> 
>>>>> num = 1
>>>>> 
>>>>>                if (j/=0) then
>>>>> 
>>>>>                    v(num) = -rho*HxdHy
>>>>> 
>>>>>                    col(MatStencil_i,num) = i
>>>>> 
>>>>>                    col(MatStencil_j,num) = j-1
>>>>> 
>>>>>                    num = num + 1
>>>>> 
>>>>>                end if
>>>>> 
>>>>>                if (i/=0) then
>>>>> 
>>>>>                    v(num) = -rho*HydHx
>>>>> 
>>>>>                    col(MatStencil_i,num) = i-1
>>>>> 
>>>>>                    col(MatStencil_j,num) = j
>>>>> 
>>>>>                    num = num + 1
>>>>> 
>>>>>                end if
>>>>> 
>>>>>                if (i/=mx-1) then
>>>>> 
>>>>>                    v(num) = -rho*HydHx
>>>>> 
>>>>>                    col(MatStencil_i,num) = i+1
>>>>> 
>>>>>                    col(MatStencil_j,num) = j
>>>>> 
>>>>>                    num = num + 1
>>>>> 
>>>>>                end if
>>>>> 
>>>>>                if (j/=my-1) then
>>>>> 
>>>>>                    v(num) = -rho*HxdHy
>>>>> 
>>>>>                    col(MatStencil_i,num) = i
>>>>> 
>>>>>                    col(MatStencil_j,num) = j+1
>>>>> 
>>>>>                    num = num + 1
>>>>> 
>>>>>                end if
>>>>> 
>>>>>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>> 
>>>>>                print *, v
>>>>> 
>>>>>                col(MatStencil_i,num) = i
>>>>> 
>>>>>                col(MatStencil_j,num) = j
>>>>> 
>>>>>                !num = num + 1
>>>>> 
>>>>>                call MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>> 
>>>>> I do not get any more out of range error. However,my ans is still different from that of ex29 in C.
>>>>> 
>>>>> This is very simple. You have an error in your code. Checking it is very simple: run the code and
>>>>> break in MatSetValues(). Make sure ex29 makes calls with exactly the same indices as your ex29f.
>>>>> 
>>>>>     Matt
>>>>>  
>>>>> Yours sincerely,
>>>>> 
>>>>> TAY wee-beng
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From zonexo at gmail.com  Fri May  4 14:18:25 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Fri, 04 May 2012 21:18:25 +0200
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
 ksp
In-Reply-To: <CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
Message-ID: <4FA42B81.40708@gmail.com>


On 4/5/2012 9:11 PM, Matthew Knepley wrote:
> On Fri, May 4, 2012 at 3:01 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>
>     On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>>     On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>
>>         On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>>         On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng
>>>         <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>
>>>
>>>             Hi,
>>>
>>>             Is there anything else I can try to get it working right?
>>>
>>>
>>>         The MatGetNullSpaceRemove() is missing.
>>         Where should I add MatGetNullSpaceRemove and what are its
>>         syntax? I googled but there's no results.
>>
>>
>>     Fixed in p;etsc-dev:
>>
>>     http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>
>     I just compiled the updated petsc-dev but I got the same error msg
>     when I use:
>
>     call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>
>     error LNK2019: unresolved external symbol MATNULLSPACEREMOVE
>     referenced in function COMPUTERHS
>     1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120: 1
>     unresolved externals
>
>
> That function is in:
>
>   src/mat/interface/ftn-custom/zmatrixf.c
>
> Did that compile? Can you see the symbol in libpetsc.a?
>
>     Matt

How can I check for the symbol in libpetsc.a?

Thanks.
>
>>          Matt
>>
>>         Thanks.
>>>
>>>            Matt
>>>
>>>             Thanks?
>>>
>>>
>>>             On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>>             On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng
>>>>             <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>>
>>>>                 Hi,
>>>>
>>>>                 I did a MatView and VecView on both C and Fortran,
>>>>                 right after Mat and Vec assembly. I have attached
>>>>                 the printout below. They are exactly the same, but
>>>>                 yet the result is different in Neumann condition.
>>>>                 However, the dirichlet condition gives the correct
>>>>                 ans. Is there anything else that could be wrong
>>>>                 even if the Mat and Vec are the same?
>>>>
>>>>
>>>>             Did you set the null space for the matrix when you have
>>>>             Neumann conditions?
>>>             Yes, for the matrix, I set as:
>>>
>>>             call
>>>             MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>
>>>                 call MatSetNullSpace(jac,nullspace,ierr)
>>>
>>>                 call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>>             for the Vec,
>>>
>>>             call
>>>             MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>
>>>             *!call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)*
>>>
>>>                 call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>>             MatNullSpaceRemove was comment out because there's error
>>>             during linking
>>>
>>>
>>>>
>>>>                Matt
>>>>
>>>>                 Thanks!
>>>>
>>>>                 Fortran:
>>>>
>>>>                 Matrix Object: 1 MPI processes
>>>>                   type: seqaij
>>>>                 row 0: (0, 2)  (1, -1)  (3, -1)
>>>>                 row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>                 row 2: (1, -1)  (2, 2)  (5, -1)
>>>>                 row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>                 row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>                 row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>                 row 6: (3, -1)  (6, 2)  (7, -1)
>>>>                 row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>                 row 8: (5, -1)  (7, -1)  (8, 2)
>>>>                 Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>                   type: mpi
>>>>                 Process [0]
>>>>                 0.25
>>>>                 0.0205213
>>>>                 1.135e-005
>>>>                 0.0205213
>>>>                 0.00168449
>>>>                 9.31663e-007
>>>>                 1.135e-005
>>>>                 9.31663e-007
>>>>                 5.15289e-010
>>>>                 Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>>                   type: mpi
>>>>                 Process [0]
>>>>                 0.14924
>>>>                 0.0242397
>>>>                 -0.0260347
>>>>                 0.0242397
>>>>                 -0.0256192
>>>>                 -0.0400102
>>>>                 -0.0260347
>>>>                 -0.0400102
>>>>                 -0.0400102
>>>>                 Press any key to continue . . .
>>>>
>>>>                 C:
>>>>
>>>>                 Matrix Object: 1 MPI processes
>>>>                   type: seqaij
>>>>                 row 0: (0, 2)  (1, -1)  (3, -1)
>>>>                 row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>                 row 2: (1, -1)  (2, 2)  (5, -1)
>>>>                 row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>                 row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>                 row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>                 row 6: (3, -1)  (6, 2)  (7, -1)
>>>>                 row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>                 row 8: (5, -1)  (7, -1)  (8, 2)
>>>>                 Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>>                   type: mpi
>>>>                 Process [0]
>>>>                 0.25
>>>>                 0.0205212
>>>>                 1.135e-05
>>>>                 0.0205212
>>>>                 0.00168449
>>>>                 9.31663e-07
>>>>                 1.135e-05
>>>>                 9.31663e-07
>>>>                 5.15288e-10
>>>>                 Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>>                   type: mpi
>>>>                 Process [0]
>>>>                 0.139311
>>>>                 0.0305751
>>>>                 -0.0220633
>>>>                 0.0305751
>>>>                 -0.0135158
>>>>                 -0.042185
>>>>                 -0.0220633
>>>>                 -0.042185
>>>>                 -0.058449
>>>>
>>>>
>>>>
>>>>                 Yours sincerely,
>>>>
>>>>                 TAY wee-beng
>>>>
>>>>
>>>>                 On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>>                 On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng
>>>>>                 <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>>>
>>>>>                     Hi,
>>>>>
>>>>>                     Do you mean my method is wrong?
>>>>>
>>>>>                     I am following the template of ex22f,
>>>>>
>>>>>                     where the variables are declared as :
>>>>>
>>>>>                     PetscScalar  v(5)
>>>>>
>>>>>                     MatStencil   row(4),col(4,5)
>>>>>
>>>>>                     Hence,
>>>>>
>>>>>                     for the neumann BC
>>>>>
>>>>>                     num = 1
>>>>>
>>>>>                                    if (j/=0) then
>>>>>
>>>>>                                        v(num) = -rho*HxdHy
>>>>>
>>>>>                                        col(MatStencil_i,num) = i
>>>>>
>>>>>                                        col(MatStencil_j,num) = j-1
>>>>>
>>>>>                                        num = num + 1
>>>>>
>>>>>                                    end if
>>>>>
>>>>>                                    if (i/=0) then
>>>>>
>>>>>                                        v(num) = -rho*HydHx
>>>>>
>>>>>                                        col(MatStencil_i,num) = i-1
>>>>>
>>>>>                                        col(MatStencil_j,num) = j
>>>>>
>>>>>                                        num = num + 1
>>>>>
>>>>>                                    end if
>>>>>
>>>>>                                    if (i/=mx-1) then
>>>>>
>>>>>                                        v(num) = -rho*HydHx
>>>>>
>>>>>                                        col(MatStencil_i,num) = i+1
>>>>>
>>>>>                                        col(MatStencil_j,num) = j
>>>>>
>>>>>                                        num = num + 1
>>>>>
>>>>>                                    end if
>>>>>
>>>>>                                    if (j/=my-1) then
>>>>>
>>>>>                                        v(num) = -rho*HxdHy
>>>>>
>>>>>                                        col(MatStencil_i,num) = i
>>>>>
>>>>>                                        col(MatStencil_j,num) = j+1
>>>>>
>>>>>                                        num = num + 1
>>>>>
>>>>>                                    end if
>>>>>
>>>>>                                    v(num) =
>>>>>                     ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>>
>>>>>                                    print *, v
>>>>>
>>>>>                                    col(MatStencil_i,num) = i
>>>>>
>>>>>                                    col(MatStencil_j,num) = j
>>>>>
>>>>>                                    !num = num + 1
>>>>>
>>>>>                                    call
>>>>>                     MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>>
>>>>>                     I do not get any more out of range error.
>>>>>                     However,my ans is still different from that of
>>>>>                     ex29 in C.
>>>>>
>>>>>
>>>>>                 This is very simple. You have an error in your
>>>>>                 code. Checking it is very simple: run the code and
>>>>>                 break in MatSetValues(). Make sure ex29 makes
>>>>>                 calls with exactly the same indices as your ex29f.
>>>>>
>>>>>                     Matt
>>>>>
>>>>>                     Yours sincerely,
>>>>>
>>>>>                     TAY wee-beng
>>>>>
>>>>>
>>>>>                 -- 
>>>>>                 What most experimenters take for granted before
>>>>>                 they begin their experiments is infinitely more
>>>>>                 interesting than any results to which their
>>>>>                 experiments lead.
>>>>>                 -- Norbert Wiener
>>>>
>>>>
>>>>
>>>>
>>>>             -- 
>>>>             What most experimenters take for granted before they
>>>>             begin their experiments is infinitely more interesting
>>>>             than any results to which their experiments lead.
>>>>             -- Norbert Wiener
>>>
>>>
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/a43ed2fa/attachment-0001.htm>

From zonexo at gmail.com  Fri May  4 14:24:44 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Fri, 04 May 2012 21:24:44 +0200
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
 ksp
In-Reply-To: <5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
Message-ID: <4FA42CFC.8090002@gmail.com>


On 4/5/2012 9:16 PM, Barry Smith wrote:
>     Do an hg pull and then run make in src/mat/interface/ftn-custom/
>
>      Then it should link.
>
>     Barry
>
>     There was a E missing from the all caps name of the function.
After hg pull, I did:

cd src//mat/interface/ftn-custom/

User at User-PC /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom
$ make
make[1]: Warning: File 
`/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)' 
has modification time 787 s in the future
make[1]: Nothing to be done for `libc'.
make[1]: warning:  Clock skew detected.  Your build may be incomplete.

But it still can't work.

>
>
> On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:
>
>> On Fri, May 4, 2012 at 3:01 PM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>
>> On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>>> On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>
>>> On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>>> On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>>
>>>> Hi,
>>>>
>>>> Is there anything else I can try to get it working right?
>>>>
>>>> The MatGetNullSpaceRemove() is missing.
>>> Where should I add MatGetNullSpaceRemove and what are its syntax? I googled but there's no results.
>>>
>>> Fixed in p;etsc-dev:
>>>
>>>    http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>> I just compiled the updated petsc-dev but I got the same error msg when I use:
>>
>> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>
>> error LNK2019: unresolved external symbol MATNULLSPACEREMOVE referenced in function COMPUTERHS
>> 1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120: 1 unresolved externals
>>
>> That function is in:
>>
>>    src/mat/interface/ftn-custom/zmatrixf.c
>>
>> Did that compile? Can you see the symbol in libpetsc.a?
>>
>>      Matt
>>
>>>       Matt
>>>
>>> Thanks.
>>>>     Matt
>>>>
>>>> Thanks?
>>>>
>>>>
>>>> On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>>> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>>> Hi,
>>>>>
>>>>> I did a MatView and VecView on both C and Fortran, right after Mat and Vec assembly. I have attached the printout below. They are exactly the same, but yet the result is different in Neumann condition. However, the dirichlet condition gives the correct ans. Is there anything else that could be wrong even if the Mat and Vec are the same?
>>>>>
>>>>> Did you set the null space for the matrix when you have Neumann conditions?
>>>> Yes, for the matrix, I set as:
>>>>
>>>> call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>
>>>>      call MatSetNullSpace(jac,nullspace,ierr)
>>>>
>>>>      call MatNullSpaceDestroy(nullspace,ierr)
>>>>
>>>> for the Vec,
>>>>
>>>> call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>
>>>>      !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>
>>>>      call MatNullSpaceDestroy(nullspace,ierr)
>>>>
>>>> MatNullSpaceRemove was comment out because there's error during linking
>>>>
>>>>
>>>>>     Matt
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Fortran:
>>>>>
>>>>> Matrix Object: 1 MPI processes
>>>>>    type: seqaij
>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>>    type: mpi
>>>>> Process [0]
>>>>> 0.25
>>>>> 0.0205213
>>>>> 1.135e-005
>>>>> 0.0205213
>>>>> 0.00168449
>>>>> 9.31663e-007
>>>>> 1.135e-005
>>>>> 9.31663e-007
>>>>> 5.15289e-010
>>>>> Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>>>    type: mpi
>>>>> Process [0]
>>>>> 0.14924
>>>>> 0.0242397
>>>>> -0.0260347
>>>>> 0.0242397
>>>>> -0.0256192
>>>>> -0.0400102
>>>>> -0.0260347
>>>>> -0.0400102
>>>>> -0.0400102
>>>>> Press any key to continue . . .
>>>>>
>>>>> C:
>>>>>
>>>>> Matrix Object: 1 MPI processes
>>>>>    type: seqaij
>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>>>    type: mpi
>>>>> Process [0]
>>>>> 0.25
>>>>> 0.0205212
>>>>> 1.135e-05
>>>>> 0.0205212
>>>>> 0.00168449
>>>>> 9.31663e-07
>>>>> 1.135e-05
>>>>> 9.31663e-07
>>>>> 5.15288e-10
>>>>> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>>>    type: mpi
>>>>> Process [0]
>>>>> 0.139311
>>>>> 0.0305751
>>>>> -0.0220633
>>>>> 0.0305751
>>>>> -0.0135158
>>>>> -0.042185
>>>>> -0.0220633
>>>>> -0.042185
>>>>> -0.058449
>>>>>
>>>>>
>>>>>
>>>>> Yours sincerely,
>>>>>
>>>>> TAY wee-beng
>>>>>
>>>>>
>>>>> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>>> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Do you mean my method is wrong?
>>>>>>
>>>>>> I am following the template of ex22f,
>>>>>>
>>>>>> where the variables are declared as :
>>>>>>
>>>>>> PetscScalar  v(5)
>>>>>>
>>>>>> MatStencil   row(4),col(4,5)
>>>>>>
>>>>>> Hence,
>>>>>>
>>>>>> for the neumann BC
>>>>>>
>>>>>> num = 1
>>>>>>
>>>>>>                 if (j/=0) then
>>>>>>
>>>>>>                     v(num) = -rho*HxdHy
>>>>>>
>>>>>>                     col(MatStencil_i,num) = i
>>>>>>
>>>>>>                     col(MatStencil_j,num) = j-1
>>>>>>
>>>>>>                     num = num + 1
>>>>>>
>>>>>>                 end if
>>>>>>
>>>>>>                 if (i/=0) then
>>>>>>
>>>>>>                     v(num) = -rho*HydHx
>>>>>>
>>>>>>                     col(MatStencil_i,num) = i-1
>>>>>>
>>>>>>                     col(MatStencil_j,num) = j
>>>>>>
>>>>>>                     num = num + 1
>>>>>>
>>>>>>                 end if
>>>>>>
>>>>>>                 if (i/=mx-1) then
>>>>>>
>>>>>>                     v(num) = -rho*HydHx
>>>>>>
>>>>>>                     col(MatStencil_i,num) = i+1
>>>>>>
>>>>>>                     col(MatStencil_j,num) = j
>>>>>>
>>>>>>                     num = num + 1
>>>>>>
>>>>>>                 end if
>>>>>>
>>>>>>                 if (j/=my-1) then
>>>>>>
>>>>>>                     v(num) = -rho*HxdHy
>>>>>>
>>>>>>                     col(MatStencil_i,num) = i
>>>>>>
>>>>>>                     col(MatStencil_j,num) = j+1
>>>>>>
>>>>>>                     num = num + 1
>>>>>>
>>>>>>                 end if
>>>>>>
>>>>>>                 v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>>>
>>>>>>                 print *, v
>>>>>>
>>>>>>                 col(MatStencil_i,num) = i
>>>>>>
>>>>>>                 col(MatStencil_j,num) = j
>>>>>>
>>>>>>                 !num = num + 1
>>>>>>
>>>>>>                 call MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>>>
>>>>>> I do not get any more out of range error. However,my ans is still different from that of ex29 in C.
>>>>>>
>>>>>> This is very simple. You have an error in your code. Checking it is very simple: run the code and
>>>>>> break in MatSetValues(). Make sure ex29 makes calls with exactly the same indices as your ex29f.
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>> Yours sincerely,
>>>>>>
>>>>>> TAY wee-beng
>>>>>>
>>>>>> -- 
>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>> -- Norbert Wiener
>>>>>
>>>>>
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener
>>>>
>>>>
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>>
>>>
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>>
>>
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener

From bsmith at mcs.anl.gov  Fri May  4 14:27:31 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 4 May 2012 14:27:31 -0500
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
	ksp
In-Reply-To: <4FA42CFC.8090002@gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
	<4FA42CFC.8090002@gmail.com>
Message-ID: <7C2E476F-20A9-43BF-967E-BA6FE2A404DC@mcs.anl.gov>


On May 4, 2012, at 2:24 PM, TAY wee-beng wrote:

> 
> On 4/5/2012 9:16 PM, Barry Smith wrote:
>>    Do an hg pull and then run make in src/mat/interface/ftn-custom/
>> 
>>     Then it should link.
>> 
>>    Barry
>> 
>>    There was a E missing from the all caps name of the function.
> After hg pull, I did:
> 
> cd src//mat/interface/ftn-custom/
> 
> User at User-PC /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom
> $ make
> make[1]: Warning: File `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)' has modification time 787 s in the future
> make[1]: Nothing to be done for `libc'.
> make[1]: warning:  Clock skew detected.  Your build may be incomplete.

   It did not compile the fixed code because something is messed up on your machine with regard to the time. So you did not get my fix.

   I guess you need to delete /cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib and run make again in the root petsc-dev directory.

   Barry

> 
> But it still can't work.
> 
>> 
>> 
>> On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:
>> 
>>> On Fri, May 4, 2012 at 3:01 PM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>> 
>>> On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>>>> On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>> 
>>>> On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>>>> On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> Is there anything else I can try to get it working right?
>>>>> 
>>>>> The MatGetNullSpaceRemove() is missing.
>>>> Where should I add MatGetNullSpaceRemove and what are its syntax? I googled but there's no results.
>>>> 
>>>> Fixed in p;etsc-dev:
>>>> 
>>>>   http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>>> I just compiled the updated petsc-dev but I got the same error msg when I use:
>>> 
>>> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>> 
>>> error LNK2019: unresolved external symbol MATNULLSPACEREMOVE referenced in function COMPUTERHS
>>> 1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120: 1 unresolved externals
>>> 
>>> That function is in:
>>> 
>>>   src/mat/interface/ftn-custom/zmatrixf.c
>>> 
>>> Did that compile? Can you see the symbol in libpetsc.a?
>>> 
>>>     Matt
>>> 
>>>>      Matt
>>>> 
>>>> Thanks.
>>>>>    Matt
>>>>> 
>>>>> Thanks?
>>>>> 
>>>>> 
>>>>> On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>>>> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I did a MatView and VecView on both C and Fortran, right after Mat and Vec assembly. I have attached the printout below. They are exactly the same, but yet the result is different in Neumann condition. However, the dirichlet condition gives the correct ans. Is there anything else that could be wrong even if the Mat and Vec are the same?
>>>>>> 
>>>>>> Did you set the null space for the matrix when you have Neumann conditions?
>>>>> Yes, for the matrix, I set as:
>>>>> 
>>>>> call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>> 
>>>>>     call MatSetNullSpace(jac,nullspace,ierr)
>>>>> 
>>>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>>> 
>>>>> for the Vec,
>>>>> 
>>>>> call MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>> 
>>>>>     !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>> 
>>>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>>> 
>>>>> MatNullSpaceRemove was comment out because there's error during linking
>>>>> 
>>>>> 
>>>>>>    Matt
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>> Fortran:
>>>>>> 
>>>>>> Matrix Object: 1 MPI processes
>>>>>>   type: seqaij
>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>>>   type: mpi
>>>>>> Process [0]
>>>>>> 0.25
>>>>>> 0.0205213
>>>>>> 1.135e-005
>>>>>> 0.0205213
>>>>>> 0.00168449
>>>>>> 9.31663e-007
>>>>>> 1.135e-005
>>>>>> 9.31663e-007
>>>>>> 5.15289e-010
>>>>>> Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>>>>   type: mpi
>>>>>> Process [0]
>>>>>> 0.14924
>>>>>> 0.0242397
>>>>>> -0.0260347
>>>>>> 0.0242397
>>>>>> -0.0256192
>>>>>> -0.0400102
>>>>>> -0.0260347
>>>>>> -0.0400102
>>>>>> -0.0400102
>>>>>> Press any key to continue . . .
>>>>>> 
>>>>>> C:
>>>>>> 
>>>>>> Matrix Object: 1 MPI processes
>>>>>>   type: seqaij
>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>>>>   type: mpi
>>>>>> Process [0]
>>>>>> 0.25
>>>>>> 0.0205212
>>>>>> 1.135e-05
>>>>>> 0.0205212
>>>>>> 0.00168449
>>>>>> 9.31663e-07
>>>>>> 1.135e-05
>>>>>> 9.31663e-07
>>>>>> 5.15288e-10
>>>>>> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>>>>   type: mpi
>>>>>> Process [0]
>>>>>> 0.139311
>>>>>> 0.0305751
>>>>>> -0.0220633
>>>>>> 0.0305751
>>>>>> -0.0135158
>>>>>> -0.042185
>>>>>> -0.0220633
>>>>>> -0.042185
>>>>>> -0.058449
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Yours sincerely,
>>>>>> 
>>>>>> TAY wee-beng
>>>>>> 
>>>>>> 
>>>>>> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>>>> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> Do you mean my method is wrong?
>>>>>>> 
>>>>>>> I am following the template of ex22f,
>>>>>>> 
>>>>>>> where the variables are declared as :
>>>>>>> 
>>>>>>> PetscScalar  v(5)
>>>>>>> 
>>>>>>> MatStencil   row(4),col(4,5)
>>>>>>> 
>>>>>>> Hence,
>>>>>>> 
>>>>>>> for the neumann BC
>>>>>>> 
>>>>>>> num = 1
>>>>>>> 
>>>>>>>                if (j/=0) then
>>>>>>> 
>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>> 
>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>> 
>>>>>>>                    col(MatStencil_j,num) = j-1
>>>>>>> 
>>>>>>>                    num = num + 1
>>>>>>> 
>>>>>>>                end if
>>>>>>> 
>>>>>>>                if (i/=0) then
>>>>>>> 
>>>>>>>                    v(num) = -rho*HydHx
>>>>>>> 
>>>>>>>                    col(MatStencil_i,num) = i-1
>>>>>>> 
>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>> 
>>>>>>>                    num = num + 1
>>>>>>> 
>>>>>>>                end if
>>>>>>> 
>>>>>>>                if (i/=mx-1) then
>>>>>>> 
>>>>>>>                    v(num) = -rho*HydHx
>>>>>>> 
>>>>>>>                    col(MatStencil_i,num) = i+1
>>>>>>> 
>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>> 
>>>>>>>                    num = num + 1
>>>>>>> 
>>>>>>>                end if
>>>>>>> 
>>>>>>>                if (j/=my-1) then
>>>>>>> 
>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>> 
>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>> 
>>>>>>>                    col(MatStencil_j,num) = j+1
>>>>>>> 
>>>>>>>                    num = num + 1
>>>>>>> 
>>>>>>>                end if
>>>>>>> 
>>>>>>>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>>>> 
>>>>>>>                print *, v
>>>>>>> 
>>>>>>>                col(MatStencil_i,num) = i
>>>>>>> 
>>>>>>>                col(MatStencil_j,num) = j
>>>>>>> 
>>>>>>>                !num = num + 1
>>>>>>> 
>>>>>>>                call MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>>>> 
>>>>>>> I do not get any more out of range error. However,my ans is still different from that of ex29 in C.
>>>>>>> 
>>>>>>> This is very simple. You have an error in your code. Checking it is very simple: run the code and
>>>>>>> break in MatSetValues(). Make sure ex29 makes calls with exactly the same indices as your ex29f.
>>>>>>> 
>>>>>>>     Matt
>>>>>>> 
>>>>>>> Yours sincerely,
>>>>>>> 
>>>>>>> TAY wee-beng
>>>>>>> 
>>>>>>> -- 
>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>> -- Norbert Wiener
>>>>> 
>>>>> 
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener


From knepley at gmail.com  Fri May  4 14:28:12 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 May 2012 15:28:12 -0400
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
	ksp
In-Reply-To: <4FA42CFC.8090002@gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
	<4FA42CFC.8090002@gmail.com>
Message-ID: <CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>

On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng <zonexo at gmail.com> wrote:

>
> On 4/5/2012 9:16 PM, Barry Smith wrote:
>
>>    Do an hg pull and then run make in src/mat/interface/ftn-custom/
>>
>>     Then it should link.
>>
>>    Barry
>>
>>    There was a E missing from the all caps name of the function.
>>
> After hg pull, I did:
>
> cd src//mat/interface/ftn-custom/
>
> User at User-PC /cygdrive/c/temp/petsc-dev/**src/mat/interface/ftn-custom
> $ make
> make[1]: Warning: File `/cygdrive/c/temp/petsc-dev/**
> petsc-3.2-dev_win32_vs2008/**lib/libpetsc.lib(zmatregf.o)' has
> modification time 787 s in the future
> make[1]: Nothing to be done for `libc'.
> make[1]: warning:  Clock skew detected.  Your build may be incomplete.
>
> But it still can't work.
>

Something is messed up with the clock on this machine.

HOWEVER, development requires certain basic skills in order to debug your
work. We
cannot be the ones debugging your code. Now

  nm $PETSC_ARCH/lib/libpetsc.a | grep -i MatNullSpaceRemove

 will look for the symbol.

    Matt


>
>>
>> On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:
>>
>>  On Fri, May 4, 2012 at 3:01 PM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>
>>> On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>>>
>>>> On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>>
>>>> On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>>>
>>>>> On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Is there anything else I can try to get it working right?
>>>>>
>>>>> The MatGetNullSpaceRemove() is missing.
>>>>>
>>>> Where should I add MatGetNullSpaceRemove and what are its syntax? I
>>>> googled but there's no results.
>>>>
>>>> Fixed in p;etsc-dev:
>>>>
>>>>   http://www.mcs.anl.gov/petsc/**petsc-dev/docs/manualpages/**
>>>> Mat/MatNullSpaceRemove.html<http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html>
>>>>
>>> I just compiled the updated petsc-dev but I got the same error msg when
>>> I use:
>>>
>>> call MatNullSpaceRemove(nullspace,**b,PETSC_NULL,ierr)
>>>
>>> error LNK2019: unresolved external symbol MATNULLSPACEREMOVE referenced
>>> in function COMPUTERHS
>>> 1>c:\obj_tmp\ex29f\Debug\**ex29f.exe : fatal error LNK1120: 1
>>> unresolved externals
>>>
>>> That function is in:
>>>
>>>   src/mat/interface/ftn-custom/**zmatrixf.c
>>>
>>> Did that compile? Can you see the symbol in libpetsc.a?
>>>
>>>     Matt
>>>
>>>       Matt
>>>>
>>>> Thanks.
>>>>
>>>>>    Matt
>>>>>
>>>>> Thanks?
>>>>>
>>>>>
>>>>> On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>>>
>>>>>> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>  wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I did a MatView and VecView on both C and Fortran, right after Mat
>>>>>> and Vec assembly. I have attached the printout below. They are exactly the
>>>>>> same, but yet the result is different in Neumann condition. However, the
>>>>>> dirichlet condition gives the correct ans. Is there anything else that
>>>>>> could be wrong even if the Mat and Vec are the same?
>>>>>>
>>>>>> Did you set the null space for the matrix when you have Neumann
>>>>>> conditions?
>>>>>>
>>>>> Yes, for the matrix, I set as:
>>>>>
>>>>> call MatNullSpaceCreate(PETSC_COMM_**WORLD,PETSC_TRUE,0,PETSC_NULL_**
>>>>> OBJECT,nullspace,ierr)
>>>>>
>>>>>     call MatSetNullSpace(jac,nullspace,**ierr)
>>>>>
>>>>>     call MatNullSpaceDestroy(nullspace,**ierr)
>>>>>
>>>>> for the Vec,
>>>>>
>>>>> call MatNullSpaceCreate(PETSC_COMM_**WORLD,PETSC_TRUE,0,PETSC_NULL_**
>>>>> OBJECT,nullspace,ierr)
>>>>>
>>>>>     !call MatNullSpaceRemove(nullspace,**b,PETSC_NULL,ierr)
>>>>>
>>>>>     call MatNullSpaceDestroy(nullspace,**ierr)
>>>>>
>>>>> MatNullSpaceRemove was comment out because there's error during linking
>>>>>
>>>>>
>>>>>     Matt
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Fortran:
>>>>>>
>>>>>> Matrix Object: 1 MPI processes
>>>>>>   type: seqaij
>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>>>   type: mpi
>>>>>> Process [0]
>>>>>> 0.25
>>>>>> 0.0205213
>>>>>> 1.135e-005
>>>>>> 0.0205213
>>>>>> 0.00168449
>>>>>> 9.31663e-007
>>>>>> 1.135e-005
>>>>>> 9.31663e-007
>>>>>> 5.15289e-010
>>>>>> Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>>>>   type: mpi
>>>>>> Process [0]
>>>>>> 0.14924
>>>>>> 0.0242397
>>>>>> -0.0260347
>>>>>> 0.0242397
>>>>>> -0.0256192
>>>>>> -0.0400102
>>>>>> -0.0260347
>>>>>> -0.0400102
>>>>>> -0.0400102
>>>>>> Press any key to continue . . .
>>>>>>
>>>>>> C:
>>>>>>
>>>>>> Matrix Object: 1 MPI processes
>>>>>>   type: seqaij
>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>>>>   type: mpi
>>>>>> Process [0]
>>>>>> 0.25
>>>>>> 0.0205212
>>>>>> 1.135e-05
>>>>>> 0.0205212
>>>>>> 0.00168449
>>>>>> 9.31663e-07
>>>>>> 1.135e-05
>>>>>> 9.31663e-07
>>>>>> 5.15288e-10
>>>>>> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>>>>   type: mpi
>>>>>> Process [0]
>>>>>> 0.139311
>>>>>> 0.0305751
>>>>>> -0.0220633
>>>>>> 0.0305751
>>>>>> -0.0135158
>>>>>> -0.042185
>>>>>> -0.0220633
>>>>>> -0.042185
>>>>>> -0.058449
>>>>>>
>>>>>>
>>>>>>
>>>>>> Yours sincerely,
>>>>>>
>>>>>> TAY wee-beng
>>>>>>
>>>>>>
>>>>>> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>>>
>>>>>>> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>  wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Do you mean my method is wrong?
>>>>>>>
>>>>>>> I am following the template of ex22f,
>>>>>>>
>>>>>>> where the variables are declared as :
>>>>>>>
>>>>>>> PetscScalar  v(5)
>>>>>>>
>>>>>>> MatStencil   row(4),col(4,5)
>>>>>>>
>>>>>>> Hence,
>>>>>>>
>>>>>>> for the neumann BC
>>>>>>>
>>>>>>> num = 1
>>>>>>>
>>>>>>>                if (j/=0) then
>>>>>>>
>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>>
>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>>
>>>>>>>                    col(MatStencil_j,num) = j-1
>>>>>>>
>>>>>>>                    num = num + 1
>>>>>>>
>>>>>>>                end if
>>>>>>>
>>>>>>>                if (i/=0) then
>>>>>>>
>>>>>>>                    v(num) = -rho*HydHx
>>>>>>>
>>>>>>>                    col(MatStencil_i,num) = i-1
>>>>>>>
>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>>
>>>>>>>                    num = num + 1
>>>>>>>
>>>>>>>                end if
>>>>>>>
>>>>>>>                if (i/=mx-1) then
>>>>>>>
>>>>>>>                    v(num) = -rho*HydHx
>>>>>>>
>>>>>>>                    col(MatStencil_i,num) = i+1
>>>>>>>
>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>>
>>>>>>>                    num = num + 1
>>>>>>>
>>>>>>>                end if
>>>>>>>
>>>>>>>                if (j/=my-1) then
>>>>>>>
>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>>
>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>>
>>>>>>>                    col(MatStencil_j,num) = j+1
>>>>>>>
>>>>>>>                    num = num + 1
>>>>>>>
>>>>>>>                end if
>>>>>>>
>>>>>>>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>>>>
>>>>>>>                print *, v
>>>>>>>
>>>>>>>                col(MatStencil_i,num) = i
>>>>>>>
>>>>>>>                col(MatStencil_j,num) = j
>>>>>>>
>>>>>>>                !num = num + 1
>>>>>>>
>>>>>>>                call MatSetValuesStencil(jac,i1,**
>>>>>>> row,num,col,v,INSERT_VALUES,**ierr)
>>>>>>>
>>>>>>> I do not get any more out of range error. However,my ans is still
>>>>>>> different from that of ex29 in C.
>>>>>>>
>>>>>>> This is very simple. You have an error in your code. Checking it is
>>>>>>> very simple: run the code and
>>>>>>> break in MatSetValues(). Make sure ex29 makes calls with exactly the
>>>>>>> same indices as your ex29f.
>>>>>>>
>>>>>>>     Matt
>>>>>>>
>>>>>>> Yours sincerely,
>>>>>>>
>>>>>>> TAY wee-beng
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/65628039/attachment-0001.htm>

From zonexo at gmail.com  Fri May  4 15:19:26 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Fri, 04 May 2012 22:19:26 +0200
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
 ksp
In-Reply-To: <CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
	<4FA42CFC.8090002@gmail.com>
	<CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>
Message-ID: <4FA439CE.5070102@gmail.com>


On 4/5/2012 9:28 PM, Matthew Knepley wrote:
> On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>
>     On 4/5/2012 9:16 PM, Barry Smith wrote:
>
>            Do an hg pull and then run make in
>         src/mat/interface/ftn-custom/
>
>             Then it should link.
>
>            Barry
>
>            There was a E missing from the all caps name of the function.
>
>     After hg pull, I did:
>
>     cd src//mat/interface/ftn-custom/
>
>     User at User-PC /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom
>     $ make
>     make[1]: Warning: File
>     `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)'
>     has modification time 787 s in the future
>     make[1]: Nothing to be done for `libc'.
>     make[1]: warning:  Clock skew detected.  Your build may be incomplete.
>
>     But it still can't work.
>
>
> Something is messed up with the clock on this machine.
>
> HOWEVER, development requires certain basic skills in order to debug 
> your work. We
> cannot be the ones debugging your code. Now
>
>   nm $PETSC_ARCH/lib/libpetsc.a | grep -i MatNullSpaceRemove
>
>  will look for the symbol.
>
>     Matt

Thanks everyone, it's now working. Sorry for all the trouble.
>
>
>
>         On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:
>
>             On Fri, May 4, 2012 at 3:01 PM, TAY
>             wee-beng<zonexo at gmail.com <mailto:zonexo at gmail.com>>  wrote:
>
>             On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>
>                 On Fri, May 4, 2012 at 11:05 AM, TAY
>                 wee-beng<zonexo at gmail.com <mailto:zonexo at gmail.com>>
>                  wrote:
>
>                 On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>
>                     On Fri, May 4, 2012 at 8:59 AM, TAY
>                     wee-beng<zonexo at gmail.com
>                     <mailto:zonexo at gmail.com>>  wrote:
>
>                     Hi,
>
>                     Is there anything else I can try to get it working
>                     right?
>
>                     The MatGetNullSpaceRemove() is missing.
>
>                 Where should I add MatGetNullSpaceRemove and what are
>                 its syntax? I googled but there's no results.
>
>                 Fixed in p;etsc-dev:
>
>                 http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>
>             I just compiled the updated petsc-dev but I got the same
>             error msg when I use:
>
>             call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>
>             error LNK2019: unresolved external symbol
>             MATNULLSPACEREMOVE referenced in function COMPUTERHS
>             1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120:
>             1 unresolved externals
>
>             That function is in:
>
>               src/mat/interface/ftn-custom/zmatrixf.c
>
>             Did that compile? Can you see the symbol in libpetsc.a?
>
>                 Matt
>
>                      Matt
>
>                 Thanks.
>
>                        Matt
>
>                     Thanks?
>
>
>                     On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>
>                         On Wed, May 2, 2012 at 1:55 PM, TAY
>                         wee-beng<zonexo at gmail.com
>                         <mailto:zonexo at gmail.com>>  wrote:
>                         Hi,
>
>                         I did a MatView and VecView on both C and
>                         Fortran, right after Mat and Vec assembly. I
>                         have attached the printout below. They are
>                         exactly the same, but yet the result is
>                         different in Neumann condition. However, the
>                         dirichlet condition gives the correct ans. Is
>                         there anything else that could be wrong even
>                         if the Mat and Vec are the same?
>
>                         Did you set the null space for the matrix when
>                         you have Neumann conditions?
>
>                     Yes, for the matrix, I set as:
>
>                     call
>                     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>
>                         call MatSetNullSpace(jac,nullspace,ierr)
>
>                         call MatNullSpaceDestroy(nullspace,ierr)
>
>                     for the Vec,
>
>                     call
>                     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>
>                         !call
>                     MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>
>                         call MatNullSpaceDestroy(nullspace,ierr)
>
>                     MatNullSpaceRemove was comment out because there's
>                     error during linking
>
>
>                            Matt
>
>                         Thanks!
>
>                         Fortran:
>
>                         Matrix Object: 1 MPI processes
>                           type: seqaij
>                         row 0: (0, 2)  (1, -1)  (3, -1)
>                         row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>                         row 2: (1, -1)  (2, 2)  (5, -1)
>                         row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>                         row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>                         row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>                         row 6: (3, -1)  (6, 2)  (7, -1)
>                         row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>                         row 8: (5, -1)  (7, -1)  (8, 2)
>                         Vector Object:Vec_0000000084000000_0 1 MPI
>                         processes
>                           type: mpi
>                         Process [0]
>                         0.25
>                         0.0205213
>                         1.135e-005
>                         0.0205213
>                         0.00168449
>                         9.31663e-007
>                         1.135e-005
>                         9.31663e-007
>                         5.15289e-010
>                         Vector Object:Vec_0000000084000000_1 1 MPI
>                         processes
>                           type: mpi
>                         Process [0]
>                         0.14924
>                         0.0242397
>                         -0.0260347
>                         0.0242397
>                         -0.0256192
>                         -0.0400102
>                         -0.0260347
>                         -0.0400102
>                         -0.0400102
>                         Press any key to continue . . .
>
>                         C:
>
>                         Matrix Object: 1 MPI processes
>                           type: seqaij
>                         row 0: (0, 2)  (1, -1)  (3, -1)
>                         row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>                         row 2: (1, -1)  (2, 2)  (5, -1)
>                         row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>                         row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>                         row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>                         row 6: (3, -1)  (6, 2)  (7, -1)
>                         row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>                         row 8: (5, -1)  (7, -1)  (8, 2)
>                         Vector Object:Vec_0x1d3b000_0 1 MPI processes
>                           type: mpi
>                         Process [0]
>                         0.25
>                         0.0205212
>                         1.135e-05
>                         0.0205212
>                         0.00168449
>                         9.31663e-07
>                         1.135e-05
>                         9.31663e-07
>                         5.15288e-10
>                         Vector Object:Vec_0x1d3b000_1 1 MPI processes
>                           type: mpi
>                         Process [0]
>                         0.139311
>                         0.0305751
>                         -0.0220633
>                         0.0305751
>                         -0.0135158
>                         -0.042185
>                         -0.0220633
>                         -0.042185
>                         -0.058449
>
>
>
>                         Yours sincerely,
>
>                         TAY wee-beng
>
>
>                         On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>
>                             On Tue, May 1, 2012 at 5:48 PM, TAY
>                             wee-beng<zonexo at gmail.com
>                             <mailto:zonexo at gmail.com>>  wrote:
>                             Hi,
>
>                             Do you mean my method is wrong?
>
>                             I am following the template of ex22f,
>
>                             where the variables are declared as :
>
>                             PetscScalar  v(5)
>
>                             MatStencil   row(4),col(4,5)
>
>                             Hence,
>
>                             for the neumann BC
>
>                             num = 1
>
>                                            if (j/=0) then
>
>                                                v(num) = -rho*HxdHy
>
>                                                col(MatStencil_i,num) = i
>
>                                                col(MatStencil_j,num) = j-1
>
>                                                num = num + 1
>
>                                            end if
>
>                                            if (i/=0) then
>
>                                                v(num) = -rho*HydHx
>
>                                                col(MatStencil_i,num) = i-1
>
>                                                col(MatStencil_j,num) = j
>
>                                                num = num + 1
>
>                                            end if
>
>                                            if (i/=mx-1) then
>
>                                                v(num) = -rho*HydHx
>
>                                                col(MatStencil_i,num) = i+1
>
>                                                col(MatStencil_j,num) = j
>
>                                                num = num + 1
>
>                                            end if
>
>                                            if (j/=my-1) then
>
>                                                v(num) = -rho*HxdHy
>
>                                                col(MatStencil_i,num) = i
>
>                                                col(MatStencil_j,num) = j+1
>
>                                                num = num + 1
>
>                                            end if
>
>                                            v(num) =
>                             ((num-1)/2.0)*rho*(HxdHy + HydHx)
>
>                                            print *, v
>
>                                            col(MatStencil_i,num) = i
>
>                                            col(MatStencil_j,num) = j
>
>                                            !num = num + 1
>
>                                            call
>                             MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>
>                             I do not get any more out of range error.
>                             However,my ans is still different from
>                             that of ex29 in C.
>
>                             This is very simple. You have an error in
>                             your code. Checking it is very simple: run
>                             the code and
>                             break in MatSetValues(). Make sure ex29
>                             makes calls with exactly the same indices
>                             as your ex29f.
>
>                                 Matt
>
>                             Yours sincerely,
>
>                             TAY wee-beng
>
>                             -- 
>                             What most experimenters take for granted
>                             before they begin their experiments is
>                             infinitely more interesting than any
>                             results to which their experiments lead.
>                             -- Norbert Wiener
>
>
>
>                         -- 
>                         What most experimenters take for granted
>                         before they begin their experiments is
>                         infinitely more interesting than any results
>                         to which their experiments lead.
>                         -- Norbert Wiener
>
>
>
>                     -- 
>                     What most experimenters take for granted before
>                     they begin their experiments is infinitely more
>                     interesting than any results to which their
>                     experiments lead.
>                     -- Norbert Wiener
>
>
>
>                 -- 
>                 What most experimenters take for granted before they
>                 begin their experiments is infinitely more interesting
>                 than any results to which their experiments lead.
>                 -- Norbert Wiener
>
>
>
>             -- 
>             What most experimenters take for granted before they begin
>             their experiments is infinitely more interesting than any
>             results to which their experiments lead.
>             -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/264192ca/attachment.htm>

From zonexo at gmail.com  Fri May  4 16:42:52 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Fri, 04 May 2012 23:42:52 +0200
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
 ksp
In-Reply-To: <CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
	<4FA42CFC.8090002@gmail.com>
	<CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>
Message-ID: <4FA44D5C.6010205@gmail.com>

Hi,

I wonder if you people are interested to include my ex29 fortran version 
in the petsc examples, which can help people who are using fortran.

Thanks.

Yours sincerely,

TAY wee-beng


On 4/5/2012 9:28 PM, Matthew Knepley wrote:
> On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>
>     On 4/5/2012 9:16 PM, Barry Smith wrote:
>
>            Do an hg pull and then run make in
>         src/mat/interface/ftn-custom/
>
>             Then it should link.
>
>            Barry
>
>            There was a E missing from the all caps name of the function.
>
>     After hg pull, I did:
>
>     cd src//mat/interface/ftn-custom/
>
>     User at User-PC /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom
>     $ make
>     make[1]: Warning: File
>     `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)'
>     has modification time 787 s in the future
>     make[1]: Nothing to be done for `libc'.
>     make[1]: warning:  Clock skew detected.  Your build may be incomplete.
>
>     But it still can't work.
>
>
> Something is messed up with the clock on this machine.
>
> HOWEVER, development requires certain basic skills in order to debug 
> your work. We
> cannot be the ones debugging your code. Now
>
>   nm $PETSC_ARCH/lib/libpetsc.a | grep -i MatNullSpaceRemove
>
>  will look for the symbol.
>
>     Matt
>
>
>
>         On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:
>
>             On Fri, May 4, 2012 at 3:01 PM, TAY
>             wee-beng<zonexo at gmail.com <mailto:zonexo at gmail.com>>  wrote:
>
>             On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>
>                 On Fri, May 4, 2012 at 11:05 AM, TAY
>                 wee-beng<zonexo at gmail.com <mailto:zonexo at gmail.com>>
>                  wrote:
>
>                 On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>
>                     On Fri, May 4, 2012 at 8:59 AM, TAY
>                     wee-beng<zonexo at gmail.com
>                     <mailto:zonexo at gmail.com>>  wrote:
>
>                     Hi,
>
>                     Is there anything else I can try to get it working
>                     right?
>
>                     The MatGetNullSpaceRemove() is missing.
>
>                 Where should I add MatGetNullSpaceRemove and what are
>                 its syntax? I googled but there's no results.
>
>                 Fixed in p;etsc-dev:
>
>                 http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>
>             I just compiled the updated petsc-dev but I got the same
>             error msg when I use:
>
>             call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>
>             error LNK2019: unresolved external symbol
>             MATNULLSPACEREMOVE referenced in function COMPUTERHS
>             1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120:
>             1 unresolved externals
>
>             That function is in:
>
>               src/mat/interface/ftn-custom/zmatrixf.c
>
>             Did that compile? Can you see the symbol in libpetsc.a?
>
>                 Matt
>
>                      Matt
>
>                 Thanks.
>
>                        Matt
>
>                     Thanks?
>
>
>                     On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>
>                         On Wed, May 2, 2012 at 1:55 PM, TAY
>                         wee-beng<zonexo at gmail.com
>                         <mailto:zonexo at gmail.com>>  wrote:
>                         Hi,
>
>                         I did a MatView and VecView on both C and
>                         Fortran, right after Mat and Vec assembly. I
>                         have attached the printout below. They are
>                         exactly the same, but yet the result is
>                         different in Neumann condition. However, the
>                         dirichlet condition gives the correct ans. Is
>                         there anything else that could be wrong even
>                         if the Mat and Vec are the same?
>
>                         Did you set the null space for the matrix when
>                         you have Neumann conditions?
>
>                     Yes, for the matrix, I set as:
>
>                     call
>                     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>
>                         call MatSetNullSpace(jac,nullspace,ierr)
>
>                         call MatNullSpaceDestroy(nullspace,ierr)
>
>                     for the Vec,
>
>                     call
>                     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>
>                         !call
>                     MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>
>                         call MatNullSpaceDestroy(nullspace,ierr)
>
>                     MatNullSpaceRemove was comment out because there's
>                     error during linking
>
>
>                            Matt
>
>                         Thanks!
>
>                         Fortran:
>
>                         Matrix Object: 1 MPI processes
>                           type: seqaij
>                         row 0: (0, 2)  (1, -1)  (3, -1)
>                         row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>                         row 2: (1, -1)  (2, 2)  (5, -1)
>                         row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>                         row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>                         row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>                         row 6: (3, -1)  (6, 2)  (7, -1)
>                         row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>                         row 8: (5, -1)  (7, -1)  (8, 2)
>                         Vector Object:Vec_0000000084000000_0 1 MPI
>                         processes
>                           type: mpi
>                         Process [0]
>                         0.25
>                         0.0205213
>                         1.135e-005
>                         0.0205213
>                         0.00168449
>                         9.31663e-007
>                         1.135e-005
>                         9.31663e-007
>                         5.15289e-010
>                         Vector Object:Vec_0000000084000000_1 1 MPI
>                         processes
>                           type: mpi
>                         Process [0]
>                         0.14924
>                         0.0242397
>                         -0.0260347
>                         0.0242397
>                         -0.0256192
>                         -0.0400102
>                         -0.0260347
>                         -0.0400102
>                         -0.0400102
>                         Press any key to continue . . .
>
>                         C:
>
>                         Matrix Object: 1 MPI processes
>                           type: seqaij
>                         row 0: (0, 2)  (1, -1)  (3, -1)
>                         row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>                         row 2: (1, -1)  (2, 2)  (5, -1)
>                         row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>                         row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>                         row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>                         row 6: (3, -1)  (6, 2)  (7, -1)
>                         row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>                         row 8: (5, -1)  (7, -1)  (8, 2)
>                         Vector Object:Vec_0x1d3b000_0 1 MPI processes
>                           type: mpi
>                         Process [0]
>                         0.25
>                         0.0205212
>                         1.135e-05
>                         0.0205212
>                         0.00168449
>                         9.31663e-07
>                         1.135e-05
>                         9.31663e-07
>                         5.15288e-10
>                         Vector Object:Vec_0x1d3b000_1 1 MPI processes
>                           type: mpi
>                         Process [0]
>                         0.139311
>                         0.0305751
>                         -0.0220633
>                         0.0305751
>                         -0.0135158
>                         -0.042185
>                         -0.0220633
>                         -0.042185
>                         -0.058449
>
>
>
>                         Yours sincerely,
>
>                         TAY wee-beng
>
>
>                         On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>
>                             On Tue, May 1, 2012 at 5:48 PM, TAY
>                             wee-beng<zonexo at gmail.com
>                             <mailto:zonexo at gmail.com>>  wrote:
>                             Hi,
>
>                             Do you mean my method is wrong?
>
>                             I am following the template of ex22f,
>
>                             where the variables are declared as :
>
>                             PetscScalar  v(5)
>
>                             MatStencil   row(4),col(4,5)
>
>                             Hence,
>
>                             for the neumann BC
>
>                             num = 1
>
>                                            if (j/=0) then
>
>                                                v(num) = -rho*HxdHy
>
>                                                col(MatStencil_i,num) = i
>
>                                                col(MatStencil_j,num) = j-1
>
>                                                num = num + 1
>
>                                            end if
>
>                                            if (i/=0) then
>
>                                                v(num) = -rho*HydHx
>
>                                                col(MatStencil_i,num) = i-1
>
>                                                col(MatStencil_j,num) = j
>
>                                                num = num + 1
>
>                                            end if
>
>                                            if (i/=mx-1) then
>
>                                                v(num) = -rho*HydHx
>
>                                                col(MatStencil_i,num) = i+1
>
>                                                col(MatStencil_j,num) = j
>
>                                                num = num + 1
>
>                                            end if
>
>                                            if (j/=my-1) then
>
>                                                v(num) = -rho*HxdHy
>
>                                                col(MatStencil_i,num) = i
>
>                                                col(MatStencil_j,num) = j+1
>
>                                                num = num + 1
>
>                                            end if
>
>                                            v(num) =
>                             ((num-1)/2.0)*rho*(HxdHy + HydHx)
>
>                                            print *, v
>
>                                            col(MatStencil_i,num) = i
>
>                                            col(MatStencil_j,num) = j
>
>                                            !num = num + 1
>
>                                            call
>                             MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>
>                             I do not get any more out of range error.
>                             However,my ans is still different from
>                             that of ex29 in C.
>
>                             This is very simple. You have an error in
>                             your code. Checking it is very simple: run
>                             the code and
>                             break in MatSetValues(). Make sure ex29
>                             makes calls with exactly the same indices
>                             as your ex29f.
>
>                                 Matt
>
>                             Yours sincerely,
>
>                             TAY wee-beng
>
>                             -- 
>                             What most experimenters take for granted
>                             before they begin their experiments is
>                             infinitely more interesting than any
>                             results to which their experiments lead.
>                             -- Norbert Wiener
>
>
>
>                         -- 
>                         What most experimenters take for granted
>                         before they begin their experiments is
>                         infinitely more interesting than any results
>                         to which their experiments lead.
>                         -- Norbert Wiener
>
>
>
>                     -- 
>                     What most experimenters take for granted before
>                     they begin their experiments is infinitely more
>                     interesting than any results to which their
>                     experiments lead.
>                     -- Norbert Wiener
>
>
>
>                 -- 
>                 What most experimenters take for granted before they
>                 begin their experiments is infinitely more interesting
>                 than any results to which their experiments lead.
>                 -- Norbert Wiener
>
>
>
>             -- 
>             What most experimenters take for granted before they begin
>             their experiments is infinitely more interesting than any
>             results to which their experiments lead.
>             -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/0c32d5c5/attachment-0001.htm>

From bsmith at mcs.anl.gov  Fri May  4 18:06:28 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 4 May 2012 18:06:28 -0500
Subject: [petsc-users] application of petsc to psr
In-Reply-To: <1336093378.96379.YahooMailRC@web181511.mail.ne1.yahoo.com>
References: <1336013159.64482.YahooMailRC@web181501.mail.ne1.yahoo.com>
	<DF2472D2-33FE-4876-825E-118EC9BB031A@mcs.anl.gov>
	<1336093378.96379.YahooMailRC@web181511.mail.ne1.yahoo.com>
Message-ID: <403E9672-6A0A-429C-B9DF-1F38521135B6@mcs.anl.gov>


  Alan,

    You have a very complicated model (and presumably discretization) which makes it difficult for us to provide coherent advice. What stage are you in your code at using PETSc? Are you thinking about using PETSc?  Have you not started yet? Is it running but not converging with PETSc?  There are many reasons that Newton may not converge: see http://www.mcs.anl.gov/petsc/documentation/faq.html#newton

   Barry



On May 3, 2012, at 8:02 PM, Alan Sayre wrote:

> Barry,
>  
> Thanks but that is not quite what I have in mind. The existing PSR model in my CFD code is not converging for PSR models containing the Method of Moments using the ABF mechanism described as shown in the Chemkin 2006 workshop proceedings. I have tried KINSOL and have still not been able to get it to converge. Any assistance in developing a PETSC based solution algorithm is appreciated.
>  
> thanks,
>  
> Alan
> 
> From: Barry Smith <bsmith at mcs.anl.gov>
> To: PETSc users list <petsc-users at mcs.anl.gov>
> Sent: Wed, May 2, 2012 10:54:21 PM
> Subject: Re: [petsc-users] application of petsc to psr
> 
> 
> http://www.sandia.gov/chemkin/workshop/proceedings/Meeks_CHEMKINWorkshop2006.pdf  ?
> 
> 
> On May 2, 2012, at 9:45 PM, Alan Sayre wrote:
> 
> > I'm new to using PETSc. I need to model perfectly stirred reactors for chemical kinetics (10-1000 species), I welcome recommendations on how to get "up-to-speed". I also need to incorprate solution of the energy (temperature) equation and moments of soot distributions.
> >  
> > Thanks for any suggestions.
> >  
> > Alan
> 


From knepley at gmail.com  Fri May  4 18:43:43 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 4 May 2012 19:43:43 -0400
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
	ksp
In-Reply-To: <4FA44D5C.6010205@gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
	<4FA42CFC.8090002@gmail.com>
	<CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>
	<4FA44D5C.6010205@gmail.com>
Message-ID: <CAMYG4Gm9cJxm1D0UmO9Fepv93uEHYhYDCcU1rKvsTU=D_+v9qQ@mail.gmail.com>

On Fri, May 4, 2012 at 5:42 PM, TAY wee-beng <zonexo at gmail.com> wrote:

>  Hi,
>
> I wonder if you people are interested to include my ex29 fortran version
> in the petsc examples, which can help people who are using fortran.
>

Yes, we will definitely include it. Please send the source, and a
representative output with run options.

  Thanks,

     Matt


> Thanks.
>
> Yours sincerely,
>
> TAY wee-beng
>
>
> On 4/5/2012 9:28 PM, Matthew Knepley wrote:
>
> On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>
>> On 4/5/2012 9:16 PM, Barry Smith wrote:
>>
>>>    Do an hg pull and then run make in src/mat/interface/ftn-custom/
>>>
>>>     Then it should link.
>>>
>>>    Barry
>>>
>>>    There was a E missing from the all caps name of the function.
>>>
>> After hg pull, I did:
>>
>> cd src//mat/interface/ftn-custom/
>>
>> User at User-PC /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom
>> $ make
>> make[1]: Warning: File
>> `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)'
>> has modification time 787 s in the future
>> make[1]: Nothing to be done for `libc'.
>> make[1]: warning:  Clock skew detected.  Your build may be incomplete.
>>
>> But it still can't work.
>>
>
>  Something is messed up with the clock on this machine.
>
>  HOWEVER, development requires certain basic skills in order to debug
> your work. We
> cannot be the ones debugging your code. Now
>
>    nm $PETSC_ARCH/lib/libpetsc.a | grep -i MatNullSpaceRemove
>
>   will look for the symbol.
>
>      Matt
>
>
>>
>>>
>>> On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:
>>>
>>>  On Fri, May 4, 2012 at 3:01 PM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>>
>>>> On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>>>>
>>>>> On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng<zonexo at gmail.com>
>>>>>  wrote:
>>>>>
>>>>> On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>>>>
>>>>>> On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng<zonexo at gmail.com>
>>>>>>  wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Is there anything else I can try to get it working right?
>>>>>>
>>>>>> The MatGetNullSpaceRemove() is missing.
>>>>>>
>>>>> Where should I add MatGetNullSpaceRemove and what are its syntax? I
>>>>> googled but there's no results.
>>>>>
>>>>> Fixed in p;etsc-dev:
>>>>>
>>>>>
>>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>>>>>
>>>> I just compiled the updated petsc-dev but I got the same error msg when
>>>> I use:
>>>>
>>>> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>
>>>> error LNK2019: unresolved external symbol MATNULLSPACEREMOVE referenced
>>>> in function COMPUTERHS
>>>> 1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120: 1 unresolved
>>>> externals
>>>>
>>>> That function is in:
>>>>
>>>>   src/mat/interface/ftn-custom/zmatrixf.c
>>>>
>>>> Did that compile? Can you see the symbol in libpetsc.a?
>>>>
>>>>     Matt
>>>>
>>>>       Matt
>>>>>
>>>>> Thanks.
>>>>>
>>>>>>    Matt
>>>>>>
>>>>>> Thanks?
>>>>>>
>>>>>>
>>>>>> On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>>>>
>>>>>>> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>  wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I did a MatView and VecView on both C and Fortran, right after Mat
>>>>>>> and Vec assembly. I have attached the printout below. They are exactly the
>>>>>>> same, but yet the result is different in Neumann condition. However, the
>>>>>>> dirichlet condition gives the correct ans. Is there anything else that
>>>>>>> could be wrong even if the Mat and Vec are the same?
>>>>>>>
>>>>>>> Did you set the null space for the matrix when you have Neumann
>>>>>>> conditions?
>>>>>>>
>>>>>> Yes, for the matrix, I set as:
>>>>>>
>>>>>> call
>>>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>>>
>>>>>>     call MatSetNullSpace(jac,nullspace,ierr)
>>>>>>
>>>>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>
>>>>>> for the Vec,
>>>>>>
>>>>>> call
>>>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>>>
>>>>>>     !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>>>
>>>>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>
>>>>>> MatNullSpaceRemove was comment out because there's error during
>>>>>> linking
>>>>>>
>>>>>>
>>>>>>     Matt
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Fortran:
>>>>>>>
>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>   type: seqaij
>>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>>>>   type: mpi
>>>>>>> Process [0]
>>>>>>> 0.25
>>>>>>> 0.0205213
>>>>>>> 1.135e-005
>>>>>>> 0.0205213
>>>>>>> 0.00168449
>>>>>>> 9.31663e-007
>>>>>>> 1.135e-005
>>>>>>> 9.31663e-007
>>>>>>> 5.15289e-010
>>>>>>> Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>>>>>   type: mpi
>>>>>>> Process [0]
>>>>>>> 0.14924
>>>>>>> 0.0242397
>>>>>>> -0.0260347
>>>>>>> 0.0242397
>>>>>>> -0.0256192
>>>>>>> -0.0400102
>>>>>>> -0.0260347
>>>>>>> -0.0400102
>>>>>>> -0.0400102
>>>>>>> Press any key to continue . . .
>>>>>>>
>>>>>>> C:
>>>>>>>
>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>   type: seqaij
>>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>>> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>>>>>   type: mpi
>>>>>>> Process [0]
>>>>>>> 0.25
>>>>>>> 0.0205212
>>>>>>> 1.135e-05
>>>>>>> 0.0205212
>>>>>>> 0.00168449
>>>>>>> 9.31663e-07
>>>>>>> 1.135e-05
>>>>>>> 9.31663e-07
>>>>>>> 5.15288e-10
>>>>>>> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>>>>>   type: mpi
>>>>>>> Process [0]
>>>>>>> 0.139311
>>>>>>> 0.0305751
>>>>>>> -0.0220633
>>>>>>> 0.0305751
>>>>>>> -0.0135158
>>>>>>> -0.042185
>>>>>>> -0.0220633
>>>>>>> -0.042185
>>>>>>> -0.058449
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Yours sincerely,
>>>>>>>
>>>>>>> TAY wee-beng
>>>>>>>
>>>>>>>
>>>>>>> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>>>>
>>>>>>>> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>  wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Do you mean my method is wrong?
>>>>>>>>
>>>>>>>> I am following the template of ex22f,
>>>>>>>>
>>>>>>>> where the variables are declared as :
>>>>>>>>
>>>>>>>> PetscScalar  v(5)
>>>>>>>>
>>>>>>>> MatStencil   row(4),col(4,5)
>>>>>>>>
>>>>>>>> Hence,
>>>>>>>>
>>>>>>>> for the neumann BC
>>>>>>>>
>>>>>>>> num = 1
>>>>>>>>
>>>>>>>>                if (j/=0) then
>>>>>>>>
>>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>>>
>>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>>>
>>>>>>>>                    col(MatStencil_j,num) = j-1
>>>>>>>>
>>>>>>>>                    num = num + 1
>>>>>>>>
>>>>>>>>                end if
>>>>>>>>
>>>>>>>>                if (i/=0) then
>>>>>>>>
>>>>>>>>                    v(num) = -rho*HydHx
>>>>>>>>
>>>>>>>>                    col(MatStencil_i,num) = i-1
>>>>>>>>
>>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>>>
>>>>>>>>                    num = num + 1
>>>>>>>>
>>>>>>>>                end if
>>>>>>>>
>>>>>>>>                if (i/=mx-1) then
>>>>>>>>
>>>>>>>>                    v(num) = -rho*HydHx
>>>>>>>>
>>>>>>>>                    col(MatStencil_i,num) = i+1
>>>>>>>>
>>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>>>
>>>>>>>>                    num = num + 1
>>>>>>>>
>>>>>>>>                end if
>>>>>>>>
>>>>>>>>                if (j/=my-1) then
>>>>>>>>
>>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>>>
>>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>>>
>>>>>>>>                    col(MatStencil_j,num) = j+1
>>>>>>>>
>>>>>>>>                    num = num + 1
>>>>>>>>
>>>>>>>>                end if
>>>>>>>>
>>>>>>>>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>>>>>
>>>>>>>>                print *, v
>>>>>>>>
>>>>>>>>                col(MatStencil_i,num) = i
>>>>>>>>
>>>>>>>>                col(MatStencil_j,num) = j
>>>>>>>>
>>>>>>>>                !num = num + 1
>>>>>>>>
>>>>>>>>                call
>>>>>>>> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>>>>>
>>>>>>>> I do not get any more out of range error. However,my ans is still
>>>>>>>> different from that of ex29 in C.
>>>>>>>>
>>>>>>>> This is very simple. You have an error in your code. Checking it is
>>>>>>>> very simple: run the code and
>>>>>>>> break in MatSetValues(). Make sure ex29 makes calls with exactly
>>>>>>>> the same indices as your ex29f.
>>>>>>>>
>>>>>>>>     Matt
>>>>>>>>
>>>>>>>> Yours sincerely,
>>>>>>>>
>>>>>>>> TAY wee-beng
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/0f28dd9e/attachment-0001.htm>

From ansayre at sbcglobal.net  Fri May  4 19:38:24 2012
From: ansayre at sbcglobal.net (Alan Sayre)
Date: Fri, 4 May 2012 17:38:24 -0700 (PDT)
Subject: [petsc-users] application of petsc to psr
In-Reply-To: <403E9672-6A0A-429C-B9DF-1F38521135B6@mcs.anl.gov>
References: <1336013159.64482.YahooMailRC@web181501.mail.ne1.yahoo.com>
	<DF2472D2-33FE-4876-825E-118EC9BB031A@mcs.anl.gov>
	<1336093378.96379.YahooMailRC@web181511.mail.ne1.yahoo.com>
	<403E9672-6A0A-429C-B9DF-1F38521135B6@mcs.anl.gov>
Message-ID: <1336178304.45046.YahooMailRC@web181513.mail.ne1.yahoo.com>

Barry,

Sorry for being imprecise. I haven't started coding anything yet with PETSc. I 
have only tried the native SUNDIALS Kinsol. I have had good success using the 
SUNDIALS Cvode for zero-d ode using the ABF mechanism. My purpose in posting is 
to determine if the PETSc suite of tools is appropriate, there is experience 
using solving similar problems, etc.

Thanks,

Alan




________________________________
From: Barry Smith <bsmith at mcs.anl.gov>
To: PETSc users list <petsc-users at mcs.anl.gov>
Sent: Fri, May 4, 2012 7:06:33 PM
Subject: Re: [petsc-users] application of petsc to psr


? Alan,

? ? You have a very complicated model (and presumably discretization) which 
makes it difficult for us to provide coherent advice. What stage are you in your 
code at using PETSc? Are you thinking about using PETSc?? Have you not started 
yet? Is it running but not converging with PETSc?? There are many reasons that 
Newton may not converge: see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#newton

? Barry



On May 3, 2012, at 8:02 PM, Alan Sayre wrote:

> Barry,
>? 
> Thanks but that is not quite what I have in mind. The existing PSR model in my 
>CFD code is not converging for PSR models containing the Method of Moments using 
>the ABF mechanism described as shown in the Chemkin 2006 workshop proceedings. I 
>have tried KINSOL and have still not been able to get it to converge. Any 
>assistance in developing a PETSC based solution algorithm is appreciated.
>? 
> thanks,
>? 
> Alan
> 
> From: Barry Smith <bsmith at mcs.anl.gov>
> To: PETSc users list <petsc-users at mcs.anl.gov>
> Sent: Wed, May 2, 2012 10:54:21 PM
> Subject: Re: [petsc-users] application of petsc to psr
> 
> 
>http://www.sandia.gov/chemkin/workshop/proceedings/Meeks_CHEMKINWorkshop2006.pdf?
>? ?
> 
> 
> On May 2, 2012, at 9:45 PM, Alan Sayre wrote:
> 
> > I'm new to using PETSc. I need to model perfectly stirred reactors for 
>chemical kinetics (10-1000 species), I welcome recommendations on how to get 
>"up-to-speed". I also need to incorprate solution of the energy (temperature) 
>equation and moments of soot distributions.
> >? 
> > Thanks for any suggestions.
> >? 
> > Alan
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120504/29253efb/attachment.htm>

From five9a2 at gmail.com  Sat May  5 16:18:06 2012
From: five9a2 at gmail.com (Jed Brown)
Date: Sat, 5 May 2012 15:18:06 -0600
Subject: [petsc-users] application of petsc to psr
In-Reply-To: <1336178304.45046.YahooMailRC@web181513.mail.ne1.yahoo.com>
References: <1336013159.64482.YahooMailRC@web181501.mail.ne1.yahoo.com>
	<DF2472D2-33FE-4876-825E-118EC9BB031A@mcs.anl.gov>
	<1336093378.96379.YahooMailRC@web181511.mail.ne1.yahoo.com>
	<403E9672-6A0A-429C-B9DF-1F38521135B6@mcs.anl.gov>
	<1336178304.45046.YahooMailRC@web181513.mail.ne1.yahoo.com>
Message-ID: <CAM9tzSn55=QFtgAxAkOYgh1_2Pw-kY509Mh2LXW-wJDvoh_u8g@mail.gmail.com>

With PETSc, there is a lot more opportunity to change details of the
nonlinear and linear solve. You can use CVode through PETSc, but it may
also be worth trying the ARK-IMEX and Rosenbrock-W methods that we have
implemented, especially if stability near the imaginary axis is useful for
you, or if error control is being finnicky with the BDFs. It is easier to
experiment with different algorithms in PETSc since most choices can be
made at run-time (e.g. command line options).
On May 4, 2012 6:38 PM, "Alan Sayre" <ansayre at sbcglobal.net> wrote:

> Barry,
>
> Sorry for being imprecise. I haven't started coding anything yet with
> PETSc. I have only tried the native SUNDIALS Kinsol. I have had good
> success using the SUNDIALS Cvode for zero-d ode using the ABF mechanism. My
> purpose in posting is to determine if the PETSc suite of tools is
> appropriate, there is experience using solving similar problems, etc.
>
> Thanks,
>
> Alan
>
>  ------------------------------
> *From:* Barry Smith <bsmith at mcs.anl.gov>
> *To:* PETSc users list <petsc-users at mcs.anl.gov>
> *Sent:* Fri, May 4, 2012 7:06:33 PM
> *Subject:* Re: [petsc-users] application of petsc to psr
>
>
>   Alan,
>
>     You have a very complicated model (and presumably discretization)
> which makes it difficult for us to provide coherent advice. What stage are
> you in your code at using PETSc? Are you thinking about using PETSc?  Have
> you not started yet? Is it running but not converging with PETSc?  There
> are many reasons that Newton may not converge: see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#newton
>
>   Barry
>
>
>
> On May 3, 2012, at 8:02 PM, Alan Sayre wrote:
>
> > Barry,
> >
> > Thanks but that is not quite what I have in mind. The existing PSR model
> in my CFD code is not converging for PSR models containing the Method of
> Moments using the ABF mechanism described as shown in the Chemkin 2006
> workshop proceedings. I have tried KINSOL and have still not been able to
> get it to converge. Any assistance in developing a PETSC based solution
> algorithm is appreciated.
> >
> > thanks,
> >
> > Alan
> >
> > From: Barry Smith <bsmith at mcs.anl.gov>
> > To: PETSc users list <petsc-users at mcs.anl.gov>
> > Sent: Wed, May 2, 2012 10:54:21 PM
> > Subject: Re: [petsc-users] application of petsc to psr
> >
> >
> >
> http://www.sandia.gov/chemkin/workshop/proceedings/Meeks_CHEMKINWorkshop2006.pdf
> ?
> >
> >
> > On May 2, 2012, at 9:45 PM, Alan Sayre wrote:
> >
> > > I'm new to using PETSc. I need to model perfectly stirred reactors for
> chemical kinetics (10-1000 species), I welcome recommendations on how to
> get "up-to-speed". I also need to incorprate solution of the energy
> (temperature) equation and moments of soot distributions.
> > >
> > > Thanks for any suggestions.
> > >
> > > Alan
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120505/9cc2078e/attachment.htm>

From jedbrown at mcs.anl.gov  Sat May  5 16:18:59 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 5 May 2012 15:18:59 -0600
Subject: [petsc-users] application of petsc to psr
In-Reply-To: <CAM9tzSn55=QFtgAxAkOYgh1_2Pw-kY509Mh2LXW-wJDvoh_u8g@mail.gmail.com>
References: <1336013159.64482.YahooMailRC@web181501.mail.ne1.yahoo.com>
	<DF2472D2-33FE-4876-825E-118EC9BB031A@mcs.anl.gov>
	<1336093378.96379.YahooMailRC@web181511.mail.ne1.yahoo.com>
	<403E9672-6A0A-429C-B9DF-1F38521135B6@mcs.anl.gov>
	<1336178304.45046.YahooMailRC@web181513.mail.ne1.yahoo.com>
	<CAM9tzSn55=QFtgAxAkOYgh1_2Pw-kY509Mh2LXW-wJDvoh_u8g@mail.gmail.com>
Message-ID: <CAM9tzSmU2mtCQmgOR0E7KG5reKoAXE1aV=txD2J4ASMKtoEzXw@mail.gmail.com>

With PETSc, there is a lot more opportunity to change details of the
nonlinear and linear solve. You can use CVode through PETSc, but it may
also be worth trying the ARK-IMEX and Rosenbrock-W methods that we have
implemented, especially if stability near the imaginary axis is useful for
you, or if error control is being finnicky with the BDFs. It is easier to
experiment with different algorithms in PETSc since most choices can be
made at run-time (e.g. command line options).

> On May 4, 2012 6:38 PM, "Alan Sayre" <ansayre at sbcglobal.net> wrote:
>>
>> Barry,
>>
>> Sorry for being imprecise. I haven't started coding anything yet with
PETSc. I have only tried the native SUNDIALS Kinsol. I have had good
success using the SUNDIALS Cvode for zero-d ode using the ABF mechanism. My
purpose in posting is to determine if the PETSc suite of tools is
appropriate, there is experience using solving similar problems, etc.
>>
>> Thanks,
>>
>> Alan
>>
>> ________________________________
>> From: Barry Smith <bsmith at mcs.anl.gov>
>> To: PETSc users list <petsc-users at mcs.anl.gov>
>> Sent: Fri, May 4, 2012 7:06:33 PM
>> Subject: Re: [petsc-users] application of petsc to psr
>>
>>
>>   Alan,
>>
>>     You have a very complicated model (and presumably discretization)
which makes it difficult for us to provide coherent advice. What stage are
you in your code at using PETSc? Are you thinking about using PETSc?  Have
you not started yet? Is it running but not converging with PETSc?  There
are many reasons that Newton may not converge: see
http://www.mcs.anl.gov/petsc/documentation/faq.html#newton
>>
>>   Barry
>>
>>
>>
>> On May 3, 2012, at 8:02 PM, Alan Sayre wrote:
>>
>> > Barry,
>> >
>> > Thanks but that is not quite what I have in mind. The existing PSR
model in my CFD code is not converging for PSR models containing the Method
of Moments using the ABF mechanism described as shown in the Chemkin 2006
workshop proceedings. I have tried KINSOL and have still not been able to
get it to converge. Any assistance in developing a PETSC based solution
algorithm is appreciated.
>> >
>> > thanks,
>> >
>> > Alan
>> >
>> > From: Barry Smith <bsmith at mcs.anl.gov>
>> > To: PETSc users list <petsc-users at mcs.anl.gov>
>> > Sent: Wed, May 2, 2012 10:54:21 PM
>> > Subject: Re: [petsc-users] application of petsc to psr
>> >
>> >
>> >
http://www.sandia.gov/chemkin/workshop/proceedings/Meeks_CHEMKINWorkshop2006.pdf
?
>> >
>> >
>> > On May 2, 2012, at 9:45 PM, Alan Sayre wrote:
>> >
>> > > I'm new to using PETSc. I need to model perfectly stirred reactors
for chemical kinetics (10-1000 species), I welcome recommendations on how
to get "up-to-speed". I also need to incorprate solution of the energy
(temperature) equation and moments of soot distributions.
>> > >
>> > > Thanks for any suggestions.
>> > >
>> > > Alan
>> >
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120505/b523c5af/attachment.htm>

From coco at dmi.unict.it  Mon May  7 13:15:13 2012
From: coco at dmi.unict.it (coco at dmi.unict.it)
Date: Mon, 07 May 2012 20:15:13 +0200
Subject: [petsc-users] matrix-free methods
Message-ID: <20120507201513.Horde.i9kzPOph4B9PqBEx5hkVmoA@mbox.dmi.unict.it>

Dear all,

I am trying to use a matrix-free method to solve a linear system with  
Petsc. In particular, I noticed that if I solve the same linear system  
with the same KSP solver, the same preconditioner, and the same  
options, I obtain different convergence results if I solve the system  
with the full- or free-matrix method. Is that plausible?

Here is a piece of code:

//Method #1:
ierr=MatCreate(PETSC_COMM_WORLD,&M); CHKERRQ(ierr);
ierr=MatSetType(M, MATMPIAIJ); CHKERRQ(ierr);
ierr=MatSetSizes(M,PETSC_DECIDE,PETSC_DECIDE,N,M); CHKERRQ(ierr);
[...filling the matrix...]
ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);
ierr = KSPSetOperators(ksp,M,M,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
ierr =  
KSPSetTolerances(ksp,1.e-12,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr);
ierr = KSPSolve(ksp,RHS,U);CHKERRQ(ierr);

//Method #2:
extern PetscErrorCode UserMult(Mat,Vec,Vec);
ierr=MatCreateShell(PETSC_COMM_WORLD,N,M,n,M,ctx,&M1); CHKERRQ(ierr);
ierr=MatShellSetOperation(M,MATOP_MULT,(void(*)(void)) UserMult);  
CHKERRQ(ierr);
ierr=MatAssemblyBegin(M,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
ierr=MatAssemblyEnd(M,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);
ierr = KSPCreate(PETSC_COMM_WORLD,&ksp1);CHKERRQ(ierr);
ierr = KSPSetOperators(ksp1,M1,M1,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
ierr =  
KSPSetTolerances(ksp1,1.e-12,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr);
ierr = KSPSolve(ksp1,RHS,U1);CHKERRQ(ierr);

In particular, the method #1 converges in 15 iterations, while the  
method #2 in more than 1000.

Thank you in advance.
Armando


From knepley at gmail.com  Mon May  7 13:19:29 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 7 May 2012 14:19:29 -0400
Subject: [petsc-users] matrix-free methods
In-Reply-To: <20120507201513.Horde.i9kzPOph4B9PqBEx5hkVmoA@mbox.dmi.unict.it>
References: <20120507201513.Horde.i9kzPOph4B9PqBEx5hkVmoA@mbox.dmi.unict.it>
Message-ID: <CAMYG4G=uZrz=QG-hRNE0D3-QcjpBgzC5xW92Z5k7hx1xnhCFTA@mail.gmail.com>

On Mon, May 7, 2012 at 2:15 PM, <coco at dmi.unict.it> wrote:

> Dear all,
>
> I am trying to use a matrix-free method to solve a linear system with
> Petsc. In particular, I noticed that if I solve the same linear system with
> the same KSP solver, the same preconditioner, and the same options, I
> obtain different convergence results if I solve the system with the full-
> or free-matrix method. Is that plausible?
>

Always always always use -ksp_view to see what you are actually doing. The
default for actual matrices is ILU(0), since we can
calculate it. The default for Shell matrices is nothing, since we have no
matrix to calculate from. Thus, the inferior convergence.

   Matt


> Here is a piece of code:
>
> //Method #1:
> ierr=MatCreate(PETSC_COMM_**WORLD,&M); CHKERRQ(ierr);
> ierr=MatSetType(M, MATMPIAIJ); CHKERRQ(ierr);
> ierr=MatSetSizes(M,PETSC_**DECIDE,PETSC_DECIDE,N,M); CHKERRQ(ierr);
> [...filling the matrix...]
> ierr = KSPCreate(PETSC_COMM_WORLD,&**ksp);CHKERRQ(ierr);
> ierr = KSPSetOperators(ksp,M,M,**DIFFERENT_NONZERO_PATTERN);**
> CHKERRQ(ierr);
> ierr = KSPSetTolerances(ksp,1.e-12,**PETSC_DEFAULT,PETSC_DEFAULT,**
> PETSC_DEFAULT);CHKERRQ(ierr);
> ierr = KSPSolve(ksp,RHS,U);CHKERRQ(**ierr);
>
> //Method #2:
> extern PetscErrorCode UserMult(Mat,Vec,Vec);
> ierr=MatCreateShell(PETSC_**COMM_WORLD,N,M,n,M,ctx,&M1); CHKERRQ(ierr);
> ierr=MatShellSetOperation(M,**MATOP_MULT,(void(*)(void)) UserMult);
> CHKERRQ(ierr);
> ierr=MatAssemblyBegin(M,MAT_**FINAL_ASSEMBLY); CHKERRQ(ierr);
> ierr=MatAssemblyEnd(M,MAT_**FINAL_ASSEMBLY); CHKERRQ(ierr);
> ierr = KSPCreate(PETSC_COMM_WORLD,&**ksp1);CHKERRQ(ierr);
> ierr = KSPSetOperators(ksp1,M1,M1,**DIFFERENT_NONZERO_PATTERN);**
> CHKERRQ(ierr);
> ierr = KSPSetTolerances(ksp1,1.e-12,**PETSC_DEFAULT,PETSC_DEFAULT,**
> PETSC_DEFAULT);CHKERRQ(ierr);
> ierr = KSPSolve(ksp1,RHS,U1);CHKERRQ(**ierr);
>
> In particular, the method #1 converges in 15 iterations, while the method
> #2 in more than 1000.
>
> Thank you in advance.
> Armando
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120507/ab4574ac/attachment.htm>

From jedbrown at mcs.anl.gov  Mon May  7 13:31:50 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 7 May 2012 12:31:50 -0600
Subject: [petsc-users] matrix-free methods
In-Reply-To: <20120507201513.Horde.i9kzPOph4B9PqBEx5hkVmoA@mbox.dmi.unict.it>
References: <20120507201513.Horde.i9kzPOph4B9PqBEx5hkVmoA@mbox.dmi.unict.it>
Message-ID: <CAM9tzS=3f15L0rioyn03pcnWZ+kDD-MKocdmdeT__ZumLJtoaA@mail.gmail.com>

On Mon, May 7, 2012 at 12:15 PM, <coco at dmi.unict.it> wrote:

> I am trying to use a matrix-free method to solve a linear system with
> Petsc. In particular, I noticed that if I solve the same linear system with
> the same KSP solver, the same preconditioner, and the same options, I
> obtain different convergence results if I solve the system with the full-
> or free-matrix method. Is that plausible?
>
> Here is a piece of code:
>
> //Method #1:
> ierr=MatCreate(PETSC_COMM_**WORLD,&M); CHKERRQ(ierr);
> ierr=MatSetType(M, MATMPIAIJ); CHKERRQ(ierr);
> ierr=MatSetSizes(M,PETSC_**DECIDE,PETSC_DECIDE,N,M); CHKERRQ(ierr);
> [...filling the matrix...]
> ierr = KSPCreate(PETSC_COMM_WORLD,&**ksp);CHKERRQ(ierr);
> ierr = KSPSetOperators(ksp,M,M,**DIFFERENT_NONZERO_PATTERN);**
> CHKERRQ(ierr);
> ierr = KSPSetTolerances(ksp,1.e-12,**PETSC_DEFAULT,PETSC_DEFAULT,**
> PETSC_DEFAULT);CHKERRQ(ierr);
> ierr = KSPSolve(ksp,RHS,U);CHKERRQ(**ierr);
>
> //Method #2:
> extern PetscErrorCode UserMult(Mat,Vec,Vec);
> ierr=MatCreateShell(PETSC_**COMM_WORLD,N,M,n,M,ctx,&M1); CHKERRQ(ierr);
> ierr=MatShellSetOperation(M,**MATOP_MULT,(void(*)(void)) UserMult);
> CHKERRQ(ierr);
> ierr=MatAssemblyBegin(M,MAT_**FINAL_ASSEMBLY); CHKERRQ(ierr);
> ierr=MatAssemblyEnd(M,MAT_**FINAL_ASSEMBLY); CHKERRQ(ierr);
> ierr = KSPCreate(PETSC_COMM_WORLD,&**ksp1);CHKERRQ(ierr);
> ierr = KSPSetOperators(ksp1,M1,M1,**DIFFERENT_NONZERO_PATTERN);**
> CHKERRQ(ierr);
> ierr = KSPSetTolerances(ksp1,1.e-12,**PETSC_DEFAULT,PETSC_DEFAULT,**
> PETSC_DEFAULT);CHKERRQ(ierr);
> ierr = KSPSolve(ksp1,RHS,U1);CHKERRQ(**ierr);
>
> In particular, the method #1 converges in 15 iterations, while the method
> #2 in more than 1000.
>

What preconditioner?

How did you determine that the matrix-free method applies the same linear
operator?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120507/9f922359/attachment.htm>

From manfred.gratt at uibk.ac.at  Tue May  8 02:32:55 2012
From: manfred.gratt at uibk.ac.at (Manfred Gratt)
Date: Tue, 08 May 2012 09:32:55 +0200
Subject: [petsc-users] petsc gcc intel and compilerflags
Message-ID: <4FA8CC27.5080808@uibk.ac.at>

Hello,

when I compile petsc with --with-debugging=0 with gcc and icc in 
configure there are different optimization flags set depending on 
compiler. So is with gcc -O set wich means -O1 optimization. While with 
icc -O3 optimisation is set. Is there a reason only icc gets the high 
performance flag while gcc not? Are there any problems when some 
optimizations are activated on gcc?

with greetings
Manfred

From dominik at itis.ethz.ch  Tue May  8 06:23:33 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Tue, 8 May 2012 13:23:33 +0200
Subject: [petsc-users] clarification of SNES / FormFunction
Message-ID: <CALm7otrKXiyy0-DZ-3pDGoQQiO_Cs12649M0UC3=SP=5Jif2tg@mail.gmail.com>

Hi, I do set my own KSP to SNES:


	ierr = SNESCreate(PETSC_COMM_WORLD, &snes); CHKERRQ(ierr);
	ierr = SNESSetKSP(snes, ksp); CHKERRQ(ierr);

I set the ksp solver type, tolerances etc. Then I register
FormFunction using SNESSetFunction, and implement it as;


PetscErrorCode FormFunction(SNES snes, Vec x, Vec f, void *ctx)

Now my question: is "x" in the signature above expected to be the same
as "x" obtained after a linear solve in the KSP object used in SNES
(that I pass inside ctx), or will "x" be generally differently-valued
due to e.g. the line search performed?

Thanks for any clarifications.

Dominik

From jedbrown at mcs.anl.gov  Tue May  8 07:50:47 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 8 May 2012 06:50:47 -0600
Subject: [petsc-users] clarification of SNES / FormFunction
In-Reply-To: <CALm7otrKXiyy0-DZ-3pDGoQQiO_Cs12649M0UC3=SP=5Jif2tg@mail.gmail.com>
References: <CALm7otrKXiyy0-DZ-3pDGoQQiO_Cs12649M0UC3=SP=5Jif2tg@mail.gmail.com>
Message-ID: <CAM9tzSn6jsH4BUcVV0Y17J-HTH8XzKT00tWNzqKDfaVySdFDKA@mail.gmail.com>

On Tue, May 8, 2012 at 5:23 AM, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> Hi, I do set my own KSP to SNES:
>
>
>        ierr = SNESCreate(PETSC_COMM_WORLD, &snes); CHKERRQ(ierr);
>        ierr = SNESSetKSP(snes, ksp); CHKERRQ(ierr);
>
> I set the ksp solver type, tolerances etc. Then I register
> FormFunction using SNESSetFunction, and implement it as;
>
>
> PetscErrorCode FormFunction(SNES snes, Vec x, Vec f, void *ctx)
>
> Now my question: is "x" in the signature above expected to be the same
> as "x" obtained after a linear solve in the KSP object used in SNES
> (that I pass inside ctx), or will "x" be generally differently-valued
> due to e.g. the line search performed?
>

Newton-based SNES solves for snes->vec_sol_update, but evaluates at
snes->vec_sol and using another vector inside the line search.

If you are writing code that depends on the Vec being the same, you're
doing it wrong.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/0817ea96/attachment.htm>

From jedbrown at mcs.anl.gov  Tue May  8 07:59:05 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 8 May 2012 06:59:05 -0600
Subject: [petsc-users] petsc gcc intel and compilerflags
In-Reply-To: <4FA8CC27.5080808@uibk.ac.at>
References: <4FA8CC27.5080808@uibk.ac.at>
Message-ID: <CAM9tzSnQ62FVR7WjphZ_DG5J+VcQGLbW=Oes=Toe2sJfOKd-RQ@mail.gmail.com>

On Tue, May 8, 2012 at 1:32 AM, Manfred Gratt <manfred.gratt at uibk.ac.at>wrote:

> when I compile petsc with --with-debugging=0 with gcc and icc in configure
> there are different optimization flags set depending on compiler. So is
> with gcc -O set wich means -O1 optimization. While with icc -O3
> optimisation is set. Is there a reason only icc gets the high performance
> flag while gcc not? Are there any problems when some optimizations are
> activated on gcc?


I don't know why only -O1 is used with gcc, but you can pass COPTFLAGS='-O3
-march=native' or whatever you like to configure to make it use those
instead.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/0c447a6d/attachment.htm>

From dominik at itis.ethz.ch  Tue May  8 08:04:20 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Tue, 8 May 2012 15:04:20 +0200
Subject: [petsc-users] clarification of SNES / FormFunction
In-Reply-To: <CAM9tzSn6jsH4BUcVV0Y17J-HTH8XzKT00tWNzqKDfaVySdFDKA@mail.gmail.com>
References: <CALm7otrKXiyy0-DZ-3pDGoQQiO_Cs12649M0UC3=SP=5Jif2tg@mail.gmail.com>
	<CAM9tzSn6jsH4BUcVV0Y17J-HTH8XzKT00tWNzqKDfaVySdFDKA@mail.gmail.com>
Message-ID: <CALm7otrGc13EFCzpACobfMW6y=Tj33AxiB4XSYXe6wJidJZSRA@mail.gmail.com>

> Newton-based SNES solves for snes->vec_sol_update, but evaluates at
> snes->vec_sol and using another vector inside the line search.
>
> If you are writing code that depends on the Vec being the same, you're doing
> it wrong.

I am currently assuming they are different and am using the "x" from
the signature to set up my A(x) and b(x). I was thinking if they are
the same (i.e. corresponding to x_n in the Newton method), I could
significantly simplify my code. But apparently "x" in the signature is
not simply x_n because of the involved line search, correct me if I am
wrong.

Thanks,
Dominik

From jedbrown at mcs.anl.gov  Tue May  8 08:07:02 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 8 May 2012 07:07:02 -0600
Subject: [petsc-users] clarification of SNES / FormFunction
In-Reply-To: <CALm7otrGc13EFCzpACobfMW6y=Tj33AxiB4XSYXe6wJidJZSRA@mail.gmail.com>
References: <CALm7otrKXiyy0-DZ-3pDGoQQiO_Cs12649M0UC3=SP=5Jif2tg@mail.gmail.com>
	<CAM9tzSn6jsH4BUcVV0Y17J-HTH8XzKT00tWNzqKDfaVySdFDKA@mail.gmail.com>
	<CALm7otrGc13EFCzpACobfMW6y=Tj33AxiB4XSYXe6wJidJZSRA@mail.gmail.com>
Message-ID: <CAM9tzS=G6xCb-J7V3tM=uyg8omjq=jUMBONsYHzF69qFKLi1qw@mail.gmail.com>

On Tue, May 8, 2012 at 7:04 AM, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> I am currently assuming they are different and am using the "x" from
> the signature to set up my A(x) and b(x). I was thinking if they are
> the same (i.e. corresponding to x_n in the Newton method), I could
> significantly simplify my code. But apparently "x" in the signature is
> not simply x_n because of the involved line search, correct me if I am
> wrong.
>

Correct, they are not the same. How could it possibly simplify your code
"significantly"?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/5df52fec/attachment.htm>

From dominik at itis.ethz.ch  Tue May  8 08:16:25 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Tue, 8 May 2012 15:16:25 +0200
Subject: [petsc-users] clarification of SNES / FormFunction
In-Reply-To: <CAM9tzS=G6xCb-J7V3tM=uyg8omjq=jUMBONsYHzF69qFKLi1qw@mail.gmail.com>
References: <CALm7otrKXiyy0-DZ-3pDGoQQiO_Cs12649M0UC3=SP=5Jif2tg@mail.gmail.com>
	<CAM9tzSn6jsH4BUcVV0Y17J-HTH8XzKT00tWNzqKDfaVySdFDKA@mail.gmail.com>
	<CALm7otrGc13EFCzpACobfMW6y=Tj33AxiB4XSYXe6wJidJZSRA@mail.gmail.com>
	<CAM9tzS=G6xCb-J7V3tM=uyg8omjq=jUMBONsYHzF69qFKLi1qw@mail.gmail.com>
Message-ID: <CALm7otrnSWoeuieP3N9rhpDRr4exDpYJDHG9oyYPnATBKm=9KA@mail.gmail.com>

>> I am currently assuming they are different and am using the "x" from
>> the signature to set up my A(x) and b(x). I was thinking if they are
>> the same (i.e. corresponding to x_n in the Newton method), I could
>> significantly simplify my code. But apparently "x" in the signature is
>> not simply x_n because of the involved line search, correct me if I am
>> wrong.
>
>
> Correct, they are not the same. How could it possibly simplify your code
> "significantly"?

Significantly was perhaps too strong a word, but a vector copy and a
couple lines of code would certainly go. Never mind, thanks a lot for
the clarification.

Dominik

From jedbrown at mcs.anl.gov  Tue May  8 08:22:29 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 8 May 2012 07:22:29 -0600
Subject: [petsc-users] clarification of SNES / FormFunction
In-Reply-To: <CALm7otrnSWoeuieP3N9rhpDRr4exDpYJDHG9oyYPnATBKm=9KA@mail.gmail.com>
References: <CALm7otrKXiyy0-DZ-3pDGoQQiO_Cs12649M0UC3=SP=5Jif2tg@mail.gmail.com>
	<CAM9tzSn6jsH4BUcVV0Y17J-HTH8XzKT00tWNzqKDfaVySdFDKA@mail.gmail.com>
	<CALm7otrGc13EFCzpACobfMW6y=Tj33AxiB4XSYXe6wJidJZSRA@mail.gmail.com>
	<CAM9tzS=G6xCb-J7V3tM=uyg8omjq=jUMBONsYHzF69qFKLi1qw@mail.gmail.com>
	<CALm7otrnSWoeuieP3N9rhpDRr4exDpYJDHG9oyYPnATBKm=9KA@mail.gmail.com>
Message-ID: <CAM9tzS=F_L3kBiusyi6+P7zbE-nUg8mwFkX4jjvN1DdtavvLig@mail.gmail.com>

Why a copy?
On May 8, 2012 8:16 AM, "Dominik Szczerba" <dominik at itis.ethz.ch> wrote:

> >> I am currently assuming they are different and am using the "x" from
> >> the signature to set up my A(x) and b(x). I was thinking if they are
> >> the same (i.e. corresponding to x_n in the Newton method), I could
> >> significantly simplify my code. But apparently "x" in the signature is
> >> not simply x_n because of the involved line search, correct me if I am
> >> wrong.
> >
> >
> > Correct, they are not the same. How could it possibly simplify your code
> > "significantly"?
>
> Significantly was perhaps too strong a word, but a vector copy and a
> couple lines of code would certainly go. Never mind, thanks a lot for
> the clarification.
>
> Dominik
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/84399c2a/attachment.htm>

From dominik at itis.ethz.ch  Tue May  8 08:31:12 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Tue, 8 May 2012 15:31:12 +0200
Subject: [petsc-users] clarification of SNES / FormFunction
In-Reply-To: <CAM9tzS=F_L3kBiusyi6+P7zbE-nUg8mwFkX4jjvN1DdtavvLig@mail.gmail.com>
References: <CALm7otrKXiyy0-DZ-3pDGoQQiO_Cs12649M0UC3=SP=5Jif2tg@mail.gmail.com>
	<CAM9tzSn6jsH4BUcVV0Y17J-HTH8XzKT00tWNzqKDfaVySdFDKA@mail.gmail.com>
	<CALm7otrGc13EFCzpACobfMW6y=Tj33AxiB4XSYXe6wJidJZSRA@mail.gmail.com>
	<CAM9tzS=G6xCb-J7V3tM=uyg8omjq=jUMBONsYHzF69qFKLi1qw@mail.gmail.com>
	<CALm7otrnSWoeuieP3N9rhpDRr4exDpYJDHG9oyYPnATBKm=9KA@mail.gmail.com>
	<CAM9tzS=F_L3kBiusyi6+P7zbE-nUg8mwFkX4jjvN1DdtavvLig@mail.gmail.com>
Message-ID: <CALm7otpmAffWKkB-voDEh7ryXO2Y3Hv9yL-K09rOTfi-if8Yng@mail.gmail.com>

Because my (class member) functions to construct A(x) and b(x) rely on
the same (class member) "x" as registered with KSP. I could perhaps
set up my A and b routines to take "x" as an argument, but this is a
bit unnatural for a class member function, and I can not quickly
predict the consequences for the rest of the code either.

Dominik

On Tue, May 8, 2012 at 3:22 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> Why a copy?
>
> On May 8, 2012 8:16 AM, "Dominik Szczerba" <dominik at itis.ethz.ch> wrote:
>>
>> >> I am currently assuming they are different and am using the "x" from
>> >> the signature to set up my A(x) and b(x). I was thinking if they are
>> >> the same (i.e. corresponding to x_n in the Newton method), I could
>> >> significantly simplify my code. But apparently "x" in the signature is
>> >> not simply x_n because of the involved line search, correct me if I am
>> >> wrong.
>> >
>> >
>> > Correct, they are not the same. How could it possibly simplify your code
>> > "significantly"?
>>
>> Significantly was perhaps too strong a word, but a vector copy and a
>> couple lines of code would certainly go. Never mind, thanks a lot for
>> the clarification.
>>
>> Dominik

From christian.staudt at ira.uka.de  Tue May  8 08:50:25 2012
From: christian.staudt at ira.uka.de (Christian Staudt)
Date: Tue, 8 May 2012 15:50:25 +0200
Subject: [petsc-users] petsc4py and numpy
Message-ID: <ED45AEB8-34C5-48F4-8BD0-8D293F2E8BE0@ira.uka.de>

I am currently getting to know PETSc via petsc4py.  Because I am writing a Python offshoot of a project originally in MATLAB, I have to 
come up with equivalents of built-in MATLAB functions. I am discovering that numpy already provides many equivalents (this page documents some numpy equivalents for common MATLAB expressions: [1])

However, the goal is shared-memory parallelism, so numpy is not sufficient. This is where PETSc comes in, and the petsc4py bindings are very convenient. As far as I know, petsc4py is based on numpy. 

My questions to users of petsc4py and numpy:

a) Is there an easy and efficient way to convert a numpy.ndarray to a PETSc.Vec or PETSc.Mat (and vice-versa)?  (A PETSc.Vec.getArray() returns a numpy.ndarray, though I have found no such method for PETSc.Mat yet. )


b) In MATLAB, arrays are used everywhere, also for small collections, where one would use tuples or lists in Python (e.g. multiple return values from a function). When I encounter an array in the original MATLAB code, I have to decide whether a tuple, a list, a numpy.ndarray or a PETSc.Vec/PETSc.Mat is appropriate. One approach would be to use PETSc wherever the array can be larger than a typical tuple. Is there a performance penalty associated with using PETSc Vec/Mat for rather small arrays?


Regards & thank you for your answers,
Chris

[1] http://www.scipy.org/NumPy_for_Matlab_Users

From knepley at gmail.com  Tue May  8 08:59:07 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 May 2012 09:59:07 -0400
Subject: [petsc-users] petsc4py and numpy
In-Reply-To: <ED45AEB8-34C5-48F4-8BD0-8D293F2E8BE0@ira.uka.de>
References: <ED45AEB8-34C5-48F4-8BD0-8D293F2E8BE0@ira.uka.de>
Message-ID: <CAMYG4G=0=aovcNeXrGMTgFci=iBB1EiG6qbONjsx6iHgiBKSZA@mail.gmail.com>

On Tue, May 8, 2012 at 9:50 AM, Christian Staudt <
christian.staudt at ira.uka.de> wrote:

> I am currently getting to know PETSc via petsc4py.  Because I am writing a
> Python offshoot of a project originally in MATLAB, I have to
> come up with equivalents of built-in MATLAB functions. I am discovering
> that numpy already provides many equivalents (this page documents some
> numpy equivalents for common MATLAB expressions: [1])
>
> However, the goal is shared-memory parallelism, so numpy is not
> sufficient. This is where PETSc comes in, and the petsc4py bindings are
> very convenient. As far as I know, petsc4py is based on numpy.
>
> My questions to users of petsc4py and numpy:
>
> a) Is there an easy and efficient way to convert a numpy.ndarray to a
> PETSc.Vec or PETSc.Mat (and vice-versa)?  (A PETSc.Vec.getArray() returns a
> numpy.ndarray, though I have found no such method for PETSc.Mat yet. )
>

If you are using dense matrices (MATDENSE) then MatGetArray() works.


> b) In MATLAB, arrays are used everywhere, also for small collections,
> where one would use tuples or lists in Python (e.g. multiple return values
> from a function). When I encounter an array in the original MATLAB code, I
> have to decide whether a tuple, a list, a numpy.ndarray or a
> PETSc.Vec/PETSc.Mat is appropriate. One approach would be to use PETSc
> wherever the array can be larger than a typical tuple. Is there a
> performance penalty associated with using PETSc Vec/Mat for rather small
> arrays?
>

The opposite. However, it is not as flexible because the type is
constrained.

   Matt


>
> Regards & thank you for your answers,
> Chris
>
> [1] http://www.scipy.org/NumPy_for_Matlab_Users




-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/fb194246/attachment.htm>

From jedbrown at mcs.anl.gov  Tue May  8 09:04:37 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 8 May 2012 08:04:37 -0600
Subject: [petsc-users] clarification of SNES / FormFunction
In-Reply-To: <CALm7otpmAffWKkB-voDEh7ryXO2Y3Hv9yL-K09rOTfi-if8Yng@mail.gmail.com>
References: <CALm7otrKXiyy0-DZ-3pDGoQQiO_Cs12649M0UC3=SP=5Jif2tg@mail.gmail.com>
	<CAM9tzSn6jsH4BUcVV0Y17J-HTH8XzKT00tWNzqKDfaVySdFDKA@mail.gmail.com>
	<CALm7otrGc13EFCzpACobfMW6y=Tj33AxiB4XSYXe6wJidJZSRA@mail.gmail.com>
	<CAM9tzS=G6xCb-J7V3tM=uyg8omjq=jUMBONsYHzF69qFKLi1qw@mail.gmail.com>
	<CALm7otrnSWoeuieP3N9rhpDRr4exDpYJDHG9oyYPnATBKm=9KA@mail.gmail.com>
	<CAM9tzS=F_L3kBiusyi6+P7zbE-nUg8mwFkX4jjvN1DdtavvLig@mail.gmail.com>
	<CALm7otpmAffWKkB-voDEh7ryXO2Y3Hv9yL-K09rOTfi-if8Yng@mail.gmail.com>
Message-ID: <CAM9tzSmC8BSJWYv7f6FByNGnXR5Bif-VxUgSPerjUh__Tt9_sw@mail.gmail.com>

I don't see why it's unnatural for the member functions to take the
location to evaluate as an argument.

In general, it is fragile and bad style to put any more state in a class
member than necessary.
On May 8, 2012 8:31 AM, "Dominik Szczerba" <dominik at itis.ethz.ch> wrote:

> Because my (class member) functions to construct A(x) and b(x) rely on
> the same (class member) "x" as registered with KSP. I could perhaps
> set up my A and b routines to take "x" as an argument, but this is a
> bit unnatural for a class member function, and I can not quickly
> predict the consequences for the rest of the code either.
>
> Dominik
>
> On Tue, May 8, 2012 at 3:22 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > Why a copy?
> >
> > On May 8, 2012 8:16 AM, "Dominik Szczerba" <dominik at itis.ethz.ch> wrote:
> >>
> >> >> I am currently assuming they are different and am using the "x" from
> >> >> the signature to set up my A(x) and b(x). I was thinking if they are
> >> >> the same (i.e. corresponding to x_n in the Newton method), I could
> >> >> significantly simplify my code. But apparently "x" in the signature
> is
> >> >> not simply x_n because of the involved line search, correct me if I
> am
> >> >> wrong.
> >> >
> >> >
> >> > Correct, they are not the same. How could it possibly simplify your
> code
> >> > "significantly"?
> >>
> >> Significantly was perhaps too strong a word, but a vector copy and a
> >> couple lines of code would certainly go. Never mind, thanks a lot for
> >> the clarification.
> >>
> >> Dominik
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/5e05625a/attachment-0001.htm>

From aron.ahmadia at kaust.edu.sa  Tue May  8 09:21:45 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Tue, 8 May 2012 17:21:45 +0300
Subject: [petsc-users] petsc4py and numpy
In-Reply-To: <ED45AEB8-34C5-48F4-8BD0-8D293F2E8BE0@ira.uka.de>
References: <ED45AEB8-34C5-48F4-8BD0-8D293F2E8BE0@ira.uka.de>
Message-ID: <CADVoUZQDMmS=yT8pBZe4-EH_tHTwx96yBACwnHHW4f4fOc78jQ@mail.gmail.com>

>
> b) In MATLAB, arrays are used everywhere, also for small collections,
> where one would use tuples or lists in Python (e.g. multiple return values
> from a function). When I encounter an array in the original MATLAB code, I
> have to decide whether a tuple, a list, a numpy.ndarray or a
> PETSc.Vec/PETSc.Mat is appropriate.


ndarray for numerical data, Python dicts and lists for more flexibility,
PETSc Vecs and Mats if you need the PETSc API.  Conversion between ndarray
and PETSc Vecs is practically free, so I would keep them as numpy arrays
for as long as possible (this is the strategy in pyclaw).


> One approach would be to use PETSc wherever the array can be larger than a
> typical tuple. Is there a performance penalty associated with using PETSc
> Vec/Mat for rather small arrays?
>

I agree with Matt, no performance penalty here.

A
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/9f5932f4/attachment.htm>

From aron.ahmadia at kaust.edu.sa  Tue May  8 09:22:59 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Tue, 8 May 2012 17:22:59 +0300
Subject: [petsc-users] petsc4py and numpy
In-Reply-To: <CADVoUZQDMmS=yT8pBZe4-EH_tHTwx96yBACwnHHW4f4fOc78jQ@mail.gmail.com>
References: <ED45AEB8-34C5-48F4-8BD0-8D293F2E8BE0@ira.uka.de>
	<CADVoUZQDMmS=yT8pBZe4-EH_tHTwx96yBACwnHHW4f4fOc78jQ@mail.gmail.com>
Message-ID: <CADVoUZQTAz-VUQyt+k2tMRTEu+VkSmTsthAW+1AG1Ej410CZng@mail.gmail.com>

Also, I'm excited by the new opportunities for parallel programming and
development offered by the IPython parallel programming extensions (
http://ipython.org/ipython-doc/dev/parallel/index.html), but have not had a
chance to try them out.  If you get a chance to use them for development,
please let us know if you find them to be useful.

A

On Tue, May 8, 2012 at 5:21 PM, Aron Ahmadia <aron.ahmadia at kaust.edu.sa>wrote:

> b) In MATLAB, arrays are used everywhere, also for small collections,
>> where one would use tuples or lists in Python (e.g. multiple return values
>> from a function). When I encounter an array in the original MATLAB code, I
>> have to decide whether a tuple, a list, a numpy.ndarray or a
>> PETSc.Vec/PETSc.Mat is appropriate.
>
>
> ndarray for numerical data, Python dicts and lists for more flexibility,
> PETSc Vecs and Mats if you need the PETSc API.  Conversion between ndarray
> and PETSc Vecs is practically free, so I would keep them as numpy arrays
> for as long as possible (this is the strategy in pyclaw).
>
>
>> One approach would be to use PETSc wherever the array can be larger than
>> a typical tuple. Is there a performance penalty associated with using PETSc
>> Vec/Mat for rather small arrays?
>>
>
> I agree with Matt, no performance penalty here.
>
> A
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/3c9b71fc/attachment.htm>

From jedbrown at mcs.anl.gov  Tue May  8 09:55:06 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 8 May 2012 08:55:06 -0600
Subject: [petsc-users] petsc4py and numpy
In-Reply-To: <CADVoUZQDMmS=yT8pBZe4-EH_tHTwx96yBACwnHHW4f4fOc78jQ@mail.gmail.com>
References: <ED45AEB8-34C5-48F4-8BD0-8D293F2E8BE0@ira.uka.de>
	<CADVoUZQDMmS=yT8pBZe4-EH_tHTwx96yBACwnHHW4f4fOc78jQ@mail.gmail.com>
Message-ID: <CAM9tzSke69UjYe0EpJnh3APO0LhFuiGmF27nBxLC+f6Y6_+0UQ@mail.gmail.com>

On Tue, May 8, 2012 at 8:21 AM, Aron Ahmadia <aron.ahmadia at kaust.edu.sa>wrote:

> Conversion between ndarray and PETSc Vecs is practically free, so I would
> keep them as numpy arrays for as long as possible (this is the strategy in
> pyclaw).


I'm not sure what is meant by "as long as possible", but Vec deals with
parallel consistency. At the local level, parallelism has essentially been
removed by the presence of ghost values, so it's reasonable to pass around
numpy arrays. At the global/analysis level, using numpy arrays is
error-prone because they don't manage parallel consistency.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/a445cb57/attachment.htm>

From christian.staudt at ira.uka.de  Tue May  8 10:00:21 2012
From: christian.staudt at ira.uka.de (Christian Staudt)
Date: Tue, 8 May 2012 17:00:21 +0200
Subject: [petsc-users] petsc4py and numpy
Message-ID: <49B110C8-A0B1-4BE7-A2D9-867983AF6EB4@ira.uka.de>

Matthew Knepley <knepley at gmail.com> wrote:

>> a) Is there an easy and efficient way to convert a numpy.ndarray to a
>> PETSc.Vec or PETSc.Mat (and vice-versa)?  (A PETSc.Vec.getArray() returns a
>> numpy.ndarray, though I have found no such method for PETSc.Mat yet. )
>> 
> 
> If you are using dense matrices (MATDENSE) then MatGetArray() works.

I don't think this is implemented in petsc4py: At least, there's no method PETSc.Mat.getArray()
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/15d46a47/attachment.htm>

From knepley at gmail.com  Tue May  8 10:04:48 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 May 2012 11:04:48 -0400
Subject: [petsc-users] petsc4py and numpy
In-Reply-To: <49B110C8-A0B1-4BE7-A2D9-867983AF6EB4@ira.uka.de>
References: <49B110C8-A0B1-4BE7-A2D9-867983AF6EB4@ira.uka.de>
Message-ID: <CAMYG4Gn6ENysDbi90R6uKwEsOpyWB6uWSpcJsuTsg4fGSZ79Qw@mail.gmail.com>

On Tue, May 8, 2012 at 11:00 AM, Christian Staudt <
christian.staudt at ira.uka.de> wrote:

> Matthew Knepley <knepley at gmail.com> wrote:
>
> a) Is there an easy and efficient way to convert a numpy.ndarray to a
>
> PETSc.Vec or PETSc.Mat (and vice-versa)?  (A PETSc.Vec.getArray() returns a
>
> numpy.ndarray, though I have found no such method for PETSc.Mat yet. )
>
>
>
> If you are using dense matrices (MATDENSE) then MatGetArray() works.
>
>
> I don't think this is implemented in petsc4py: At least, there's no method
> PETSc.Mat.getArray()
>

If you ask Lisandro for it, he will put it in.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/a30e3332/attachment.htm>

From dalcinl at gmail.com  Tue May  8 10:10:52 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 8 May 2012 12:10:52 -0300
Subject: [petsc-users] petsc4py and numpy
In-Reply-To: <ED45AEB8-34C5-48F4-8BD0-8D293F2E8BE0@ira.uka.de>
References: <ED45AEB8-34C5-48F4-8BD0-8D293F2E8BE0@ira.uka.de>
Message-ID: <CAEcYPwAAZWjbeSzRx+JFjAQ-bJVvmyKBrUX4yjP_CHUTOqywKw@mail.gmail.com>

On 8 May 2012 10:50, Christian Staudt <christian.staudt at ira.uka.de> wrote:
> I am currently getting to know PETSc via petsc4py. ?Because I am writing a Python offshoot of a project originally in MATLAB, I have to
> come up with equivalents of built-in MATLAB functions. I am discovering that numpy already provides many equivalents (this page documents some numpy equivalents for common MATLAB expressions: [1])
>
> However, the goal is shared-memory parallelism, so numpy is not sufficient. This is where PETSc comes in, and the petsc4py bindings are very convenient. As far as I know, petsc4py is based on numpy.
>

Well, petsc4py is not "based" on NumPy, but uses NumPy to pass arrays
back and forth between Python land and C land.


> My questions to users of petsc4py and numpy:
>
> a) Is there an easy and efficient way to convert a numpy.ndarray to a PETSc.Vec or PETSc.Mat (and vice-versa)? ?(A PETSc.Vec.getArray() returns a numpy.ndarray, though I have found no such method for PETSc.Mat yet. )
>

To get a NumPy array out of a PETSc.Vec instance, you just call
vec.getArray(), the returned NumPy array shares the memory buffer with
the PETSc Vec, so no copies involved, it is very fast. For the other
way, you can do PETSc.Vec().createWithArray(array), then the Vec will
use the memory buffer of the NumPy array, again no copies involved.

For matrices, it depends on the matrix type. For dense matrices,
currently petsc4py does not support getting the array. For sparse (aka
AIJ) matrices, you can use mat.getValuesCSR(), however this will
involve copies. For the other way, you can use
PETSc.Mat().createAIJ(size=(nrows,ncols), csr=(ai,aj,aa)).

Please take into account that PETSc is basically designed and
optimized to work with SPARSE matrices.

>
> b) In MATLAB, arrays are used everywhere, also for small collections, where one would use tuples or lists in Python (e.g. multiple return values from a function). When I encounter an array in the original MATLAB code, I have to decide whether a tuple, a list, a numpy.ndarray or a PETSc.Vec/PETSc.Mat is appropriate. One approach would be to use PETSc wherever the array can be larger than a typical tuple. Is there a performance penalty associated with using PETSc Vec/Mat for rather small arrays?
>

Not really (other than the usual Python calling overhead when you
compare to C or Fotran code)

>
> Regards & thank you for your answers,
> Chris
>
> [1] http://www.scipy.org/NumPy_for_Matlab_Users



-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From dalcinl at gmail.com  Tue May  8 10:15:19 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 8 May 2012 12:15:19 -0300
Subject: [petsc-users] petsc4py and numpy
In-Reply-To: <CAMYG4Gn6ENysDbi90R6uKwEsOpyWB6uWSpcJsuTsg4fGSZ79Qw@mail.gmail.com>
References: <49B110C8-A0B1-4BE7-A2D9-867983AF6EB4@ira.uka.de>
	<CAMYG4Gn6ENysDbi90R6uKwEsOpyWB6uWSpcJsuTsg4fGSZ79Qw@mail.gmail.com>
Message-ID: <CAEcYPwDC60YhwGfKgnRbYHw3xn3gV-wX5LZArNbM5cP6Xyu7wg@mail.gmail.com>

On 8 May 2012 12:04, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, May 8, 2012 at 11:00 AM, Christian Staudt
> <christian.staudt at ira.uka.de> wrote:
>>
>> Matthew Knepley?<knepley at gmail.com>?wrote:
>>
>> a) Is there an easy and efficient way to convert a numpy.ndarray to a
>>
>> PETSc.Vec or PETSc.Mat (and vice-versa)? ?(A PETSc.Vec.getArray() returns
>> a
>>
>> numpy.ndarray, though I have found no such method for PETSc.Mat yet. )
>>
>>
>>
>> If you are using dense matrices (MATDENSE) then MatGetArray() works.
>>
>>
>> I don't think this is implemented in petsc4py: At least, there's no method
>> PETSc.Mat.getArray()
>
>
> If you ask Lisandro for it, he will put it in.
>

Matt, I've never put that in just because I'm not sure how to make it
general. PETSc have many different matrix formats, and MatGetArray()
call is like a hack, where you can get very different things (the "1D"
values of an AIJ matrix, the "2D" values of a dense matrix in Fortran
order). What about MPIAIJ, what should I return to users? Can you
suggest some approach or API to implement this (I mean, how to get
arrays for the various matrix types)?

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From knepley at gmail.com  Tue May  8 10:25:31 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 May 2012 11:25:31 -0400
Subject: [petsc-users] petsc4py and numpy
In-Reply-To: <CAEcYPwDC60YhwGfKgnRbYHw3xn3gV-wX5LZArNbM5cP6Xyu7wg@mail.gmail.com>
References: <49B110C8-A0B1-4BE7-A2D9-867983AF6EB4@ira.uka.de>
	<CAMYG4Gn6ENysDbi90R6uKwEsOpyWB6uWSpcJsuTsg4fGSZ79Qw@mail.gmail.com>
	<CAEcYPwDC60YhwGfKgnRbYHw3xn3gV-wX5LZArNbM5cP6Xyu7wg@mail.gmail.com>
Message-ID: <CAMYG4GnYv5MVotiBipCSvQREZ1c4KOxfSKpiKxBf=jsTRK4uMQ@mail.gmail.com>

On Tue, May 8, 2012 at 11:15 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> On 8 May 2012 12:04, Matthew Knepley <knepley at gmail.com> wrote:
> > On Tue, May 8, 2012 at 11:00 AM, Christian Staudt
> > <christian.staudt at ira.uka.de> wrote:
> >>
> >> Matthew Knepley <knepley at gmail.com> wrote:
> >>
> >> a) Is there an easy and efficient way to convert a numpy.ndarray to a
> >>
> >> PETSc.Vec or PETSc.Mat (and vice-versa)?  (A PETSc.Vec.getArray()
> returns
> >> a
> >>
> >> numpy.ndarray, though I have found no such method for PETSc.Mat yet. )
> >>
> >>
> >>
> >> If you are using dense matrices (MATDENSE) then MatGetArray() works.
> >>
> >>
> >> I don't think this is implemented in petsc4py: At least, there's no
> method
> >> PETSc.Mat.getArray()
> >
> >
> > If you ask Lisandro for it, he will put it in.
> >
>
> Matt, I've never put that in just because I'm not sure how to make it
> general. PETSc have many different matrix formats, and MatGetArray()
> call is like a hack, where you can get very different things (the "1D"
> values of an AIJ matrix, the "2D" values of a dense matrix in Fortran
> order). What about MPIAIJ, what should I return to users? Can you
> suggest some approach or API to implement this (I mean, how to get
> arrays for the various matrix types)?


I think the whole point is that you should not do it. However, that has
never stopped us from giving users
what they ask for. We are not int he business of reforming computing
practices, which is why we have
MatGetArray(). I would make a wrapper that failed unless the type was
MATDENSE.

  Matt


>
> --
> Lisandro Dalcin
> ---------------
> CIMEC (INTEC/CONICET-UNL)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> 3000 Santa Fe, Argentina
> Tel: +54-342-4511594 (ext 1011)
> Tel/Fax: +54-342-4511169
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/3780cf58/attachment-0001.htm>

From bsmith at mcs.anl.gov  Tue May  8 11:08:56 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 8 May 2012 11:08:56 -0500
Subject: [petsc-users] petsc gcc intel and compilerflags
In-Reply-To: <CAM9tzSnQ62FVR7WjphZ_DG5J+VcQGLbW=Oes=Toe2sJfOKd-RQ@mail.gmail.com>
References: <4FA8CC27.5080808@uibk.ac.at>
	<CAM9tzSnQ62FVR7WjphZ_DG5J+VcQGLbW=Oes=Toe2sJfOKd-RQ@mail.gmail.com>
Message-ID: <8AACE68F-7196-40E9-83F7-2DA630C02362@mcs.anl.gov>


  Manfred,

    ./configure is not smart enough to know what optimization flags are best for all compilers/systems. Thus we have relatively straightforward defaults. As Jed notes, if you know what flags are most appropriate for your build you must pass them in to ./configure

   Barry

On May 8, 2012, at 7:59 AM, Jed Brown wrote:

> On Tue, May 8, 2012 at 1:32 AM, Manfred Gratt <manfred.gratt at uibk.ac.at> wrote:
> when I compile petsc with --with-debugging=0 with gcc and icc in configure there are different optimization flags set depending on compiler. So is with gcc -O set wich means -O1 optimization. While with icc -O3 optimisation is set. Is there a reason only icc gets the high performance flag while gcc not? Are there any problems when some optimizations are activated on gcc?
> 
> I don't know why only -O1 is used with gcc, but you can pass COPTFLAGS='-O3 -march=native' or whatever you like to configure to make it use those instead.


From christian.staudt at ira.uka.de  Tue May  8 12:38:45 2012
From: christian.staudt at ira.uka.de (Christian Staudt)
Date: Tue, 8 May 2012 19:38:45 +0200
Subject: [petsc-users] petsc4py and numpy
In-Reply-To: <mailman.2116.1336490734.4646.petsc-users@mcs.anl.gov>
References: <mailman.2116.1336490734.4646.petsc-users@mcs.anl.gov>
Message-ID: <97F10C5D-0BFB-41BA-B163-BD6CABA0D90F@ira.uka.de>


@ Aron Ahmadia:


> Conversion between ndarray
> and PETSc Vecs is practically free, so I would keep them as numpy arrays
> for as long as possible (this is the strategy in pyclaw).

Thanks for the hint. I was wondering whether I should start with a working prototype based on numpy, then do parallelization with PETSc on demand - or start with PETSc wherever matrices and vectors are needed.


> Also, I'm excited by the new opportunities for parallel programming and
> development offered by the IPython parallel programming extensions (
> http://ipython.org/ipython-doc/dev/parallel/index.html), but have not had a
> chance to try them out.  If you get a chance to use them for development,
> please let us know if you find them to be useful.

You're not the first person to mention iPython, so I'll definitely have a look and let you know in case I have used it successfully.


@ Lisandro Dalcin:

> For matrices, it depends on the matrix type. For dense matrices,
> currently petsc4py does not support getting the array. For sparse (aka
> AIJ) matrices, you can use mat.getValuesCSR(), however this will
> involve copies. For the other way, you can use
> PETSc.Mat().createAIJ(size=(nrows,ncols), csr=(ai,aj,aa)).

That's basically what I wanted to know.


ps. Thanks for providing petsc4py. Being able to develop in Python is currently saving me a lot of time and pain (compared to the alternative, C++)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/5f9008bd/attachment.htm>

From dalcinl at gmail.com  Tue May  8 14:11:22 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 8 May 2012 16:11:22 -0300
Subject: [petsc-users] petsc4py and numpy
In-Reply-To: <CAMYG4GnYv5MVotiBipCSvQREZ1c4KOxfSKpiKxBf=jsTRK4uMQ@mail.gmail.com>
References: <49B110C8-A0B1-4BE7-A2D9-867983AF6EB4@ira.uka.de>
	<CAMYG4Gn6ENysDbi90R6uKwEsOpyWB6uWSpcJsuTsg4fGSZ79Qw@mail.gmail.com>
	<CAEcYPwDC60YhwGfKgnRbYHw3xn3gV-wX5LZArNbM5cP6Xyu7wg@mail.gmail.com>
	<CAMYG4GnYv5MVotiBipCSvQREZ1c4KOxfSKpiKxBf=jsTRK4uMQ@mail.gmail.com>
Message-ID: <CAEcYPwArCJVjaMshbJCCYNpEj2bC9BFarpYRJADe3ZmTyX-2aA@mail.gmail.com>

On 8 May 2012 12:25, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, May 8, 2012 at 11:15 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>
>> On 8 May 2012 12:04, Matthew Knepley <knepley at gmail.com> wrote:
>> > On Tue, May 8, 2012 at 11:00 AM, Christian Staudt
>> > <christian.staudt at ira.uka.de> wrote:
>> >>
>> >> Matthew Knepley?<knepley at gmail.com>?wrote:
>> >>
>> >> a) Is there an easy and efficient way to convert a numpy.ndarray to a
>> >>
>> >> PETSc.Vec or PETSc.Mat (and vice-versa)? ?(A PETSc.Vec.getArray()
>> >> returns
>> >> a
>> >>
>> >> numpy.ndarray, though I have found no such method for PETSc.Mat yet. )
>> >>
>> >>
>> >>
>> >> If you are using dense matrices (MATDENSE) then MatGetArray() works.
>> >>
>> >>
>> >> I don't think this is implemented in petsc4py: At least, there's no
>> >> method
>> >> PETSc.Mat.getArray()
>> >
>> >
>> > If you ask Lisandro for it, he will put it in.
>> >
>>
>> Matt, I've never put that in just because I'm not sure how to make it
>> general. PETSc have many different matrix formats, and MatGetArray()
>> call is like a hack, where you can get very different things (the "1D"
>> values of an AIJ matrix, the "2D" values of a dense matrix in Fortran
>> order). What about MPIAIJ, what should I return to users? Can you
>> suggest some approach or API to implement this (I mean, how to get
>> arrays for the various matrix types)?
>
>
> I think the whole point is that you should not do it. However, that has
> never stopped us from giving users
> what they ask for. We are not int he business of reforming computing
> practices, which is why we have
> MatGetArray(). I would make a wrapper that failed unless the type was
> MATDENSE.
>

Yea, that's really easy.

However, Is there anything better for AIJ matrices? Right now, we have
mat.getValuesCSR()... it works by calling MatGetRow() row by row twice
(first pass to count and allocate, next to get values and copy to
NumPy arrays).


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From zonexo at gmail.com  Tue May  8 14:17:06 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Tue, 08 May 2012 21:17:06 +0200
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
 ksp
In-Reply-To: <CAMYG4Gm9cJxm1D0UmO9Fepv93uEHYhYDCcU1rKvsTU=D_+v9qQ@mail.gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
	<4FA42CFC.8090002@gmail.com>
	<CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>
	<4FA44D5C.6010205@gmail.com>
	<CAMYG4Gm9cJxm1D0UmO9Fepv93uEHYhYDCcU1rKvsTU=D_+v9qQ@mail.gmail.com>
Message-ID: <4FA97132.3080103@gmail.com>

Hi,

I compiled and run my code under visual studio 2008 with intel fortran. 
Everything works ok.

However, when I tried to run the code in linux, I got the error as 
below. The error happens when KSPSetUp(ksp,ierr) is called.

However, I am not able to print VecView or MatView to view if there's 
any errors. Is there any recommendation for debugging? I hope I do not 
need to valgrind if possible.

[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames 
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] DM user function line 0 unknownunknown
[0]PETSC ERROR: [0] DMComputeFunction line 2085 
/home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
[0]PETSC ERROR: [0] KSPSetUp line 182 
/home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: --------------------- Error Message 
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision: 
7ecdd63ec420b1659b960e65d96e822c5ac1a968  HG Date: Mon May 07 21:42:26 
2012 -0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: ./ex29f on a petsc-3.2 named hpc12 by wtay Tue May  8 
20:45:42 2012
[0]PETSC ERROR: Libraries linked from 
/home/wtay/Lib/petsc-3.2-dev_shared_debug/lib
[0]PETSC ERROR: Configure run at Tue May  8 10:47:59 2012
[0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/ 
--with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/ 
--with-debugging=1 --download-hypre=1 
--prefix=/home/wtay/Lib/petsc-3.2-dev_shared_debug --known-mpi-shared=1 
--with-shared-libraries
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

Yours sincerely,

TAY wee-beng


On 5/5/2012 1:43 AM, Matthew Knepley wrote:
> On Fri, May 4, 2012 at 5:42 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I wonder if you people are interested to include my ex29 fortran
>     version in the petsc examples, which can help people who are using
>     fortran.
>
>
> Yes, we will definitely include it. Please send the source, and a 
> representative output with run options.
>
>   Thanks,
>
>      Matt
>
>     Thanks.
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>     On 4/5/2012 9:28 PM, Matthew Knepley wrote:
>>     On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>
>>         On 4/5/2012 9:16 PM, Barry Smith wrote:
>>
>>                Do an hg pull and then run make in
>>             src/mat/interface/ftn-custom/
>>
>>                 Then it should link.
>>
>>                Barry
>>
>>                There was a E missing from the all caps name of the
>>             function.
>>
>>         After hg pull, I did:
>>
>>         cd src//mat/interface/ftn-custom/
>>
>>         User at User-PC
>>         /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom
>>         $ make
>>         make[1]: Warning: File
>>         `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)'
>>         has modification time 787 s in the future
>>         make[1]: Nothing to be done for `libc'.
>>         make[1]: warning:  Clock skew detected.  Your build may be
>>         incomplete.
>>
>>         But it still can't work.
>>
>>
>>     Something is messed up with the clock on this machine.
>>
>>     HOWEVER, development requires certain basic skills in order to
>>     debug your work. We
>>     cannot be the ones debugging your code. Now
>>
>>       nm $PETSC_ARCH/lib/libpetsc.a | grep -i MatNullSpaceRemove
>>
>>      will look for the symbol.
>>
>>         Matt
>>
>>
>>
>>             On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:
>>
>>                 On Fri, May 4, 2012 at 3:01 PM, TAY
>>                 wee-beng<zonexo at gmail.com <mailto:zonexo at gmail.com>>
>>                  wrote:
>>
>>                 On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>>
>>                     On Fri, May 4, 2012 at 11:05 AM, TAY
>>                     wee-beng<zonexo at gmail.com
>>                     <mailto:zonexo at gmail.com>>  wrote:
>>
>>                     On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>
>>                         On Fri, May 4, 2012 at 8:59 AM, TAY
>>                         wee-beng<zonexo at gmail.com
>>                         <mailto:zonexo at gmail.com>>  wrote:
>>
>>                         Hi,
>>
>>                         Is there anything else I can try to get it
>>                         working right?
>>
>>                         The MatGetNullSpaceRemove() is missing.
>>
>>                     Where should I add MatGetNullSpaceRemove and what
>>                     are its syntax? I googled but there's no results.
>>
>>                     Fixed in p;etsc-dev:
>>
>>                     http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>>
>>                 I just compiled the updated petsc-dev but I got the
>>                 same error msg when I use:
>>
>>                 call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>
>>                 error LNK2019: unresolved external symbol
>>                 MATNULLSPACEREMOVE referenced in function COMPUTERHS
>>                 1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error
>>                 LNK1120: 1 unresolved externals
>>
>>                 That function is in:
>>
>>                   src/mat/interface/ftn-custom/zmatrixf.c
>>
>>                 Did that compile? Can you see the symbol in libpetsc.a?
>>
>>                     Matt
>>
>>                          Matt
>>
>>                     Thanks.
>>
>>                            Matt
>>
>>                         Thanks?
>>
>>
>>                         On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>
>>                             On Wed, May 2, 2012 at 1:55 PM, TAY
>>                             wee-beng<zonexo at gmail.com
>>                             <mailto:zonexo at gmail.com>>  wrote:
>>                             Hi,
>>
>>                             I did a MatView and VecView on both C and
>>                             Fortran, right after Mat and Vec
>>                             assembly. I have attached the printout
>>                             below. They are exactly the same, but yet
>>                             the result is different in Neumann
>>                             condition. However, the dirichlet
>>                             condition gives the correct ans. Is there
>>                             anything else that could be wrong even if
>>                             the Mat and Vec are the same?
>>
>>                             Did you set the null space for the matrix
>>                             when you have Neumann conditions?
>>
>>                         Yes, for the matrix, I set as:
>>
>>                         call
>>                         MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>
>>                             call MatSetNullSpace(jac,nullspace,ierr)
>>
>>                             call MatNullSpaceDestroy(nullspace,ierr)
>>
>>                         for the Vec,
>>
>>                         call
>>                         MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>
>>                             !call
>>                         MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>
>>                             call MatNullSpaceDestroy(nullspace,ierr)
>>
>>                         MatNullSpaceRemove was comment out because
>>                         there's error during linking
>>
>>
>>                                Matt
>>
>>                             Thanks!
>>
>>                             Fortran:
>>
>>                             Matrix Object: 1 MPI processes
>>                               type: seqaij
>>                             row 0: (0, 2)  (1, -1)  (3, -1)
>>                             row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>                             row 2: (1, -1)  (2, 2)  (5, -1)
>>                             row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>                             row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)
>>                              (7, -1)
>>                             row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>                             row 6: (3, -1)  (6, 2)  (7, -1)
>>                             row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>                             row 8: (5, -1)  (7, -1)  (8, 2)
>>                             Vector Object:Vec_0000000084000000_0 1
>>                             MPI processes
>>                               type: mpi
>>                             Process [0]
>>                             0.25
>>                             0.0205213
>>                             1.135e-005
>>                             0.0205213
>>                             0.00168449
>>                             9.31663e-007
>>                             1.135e-005
>>                             9.31663e-007
>>                             5.15289e-010
>>                             Vector Object:Vec_0000000084000000_1 1
>>                             MPI processes
>>                               type: mpi
>>                             Process [0]
>>                             0.14924
>>                             0.0242397
>>                             -0.0260347
>>                             0.0242397
>>                             -0.0256192
>>                             -0.0400102
>>                             -0.0260347
>>                             -0.0400102
>>                             -0.0400102
>>                             Press any key to continue . . .
>>
>>                             C:
>>
>>                             Matrix Object: 1 MPI processes
>>                               type: seqaij
>>                             row 0: (0, 2)  (1, -1)  (3, -1)
>>                             row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>                             row 2: (1, -1)  (2, 2)  (5, -1)
>>                             row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>                             row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)
>>                              (7, -1)
>>                             row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>                             row 6: (3, -1)  (6, 2)  (7, -1)
>>                             row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>                             row 8: (5, -1)  (7, -1)  (8, 2)
>>                             Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>                               type: mpi
>>                             Process [0]
>>                             0.25
>>                             0.0205212
>>                             1.135e-05
>>                             0.0205212
>>                             0.00168449
>>                             9.31663e-07
>>                             1.135e-05
>>                             9.31663e-07
>>                             5.15288e-10
>>                             Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>                               type: mpi
>>                             Process [0]
>>                             0.139311
>>                             0.0305751
>>                             -0.0220633
>>                             0.0305751
>>                             -0.0135158
>>                             -0.042185
>>                             -0.0220633
>>                             -0.042185
>>                             -0.058449
>>
>>
>>
>>                             Yours sincerely,
>>
>>                             TAY wee-beng
>>
>>
>>                             On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>
>>                                 On Tue, May 1, 2012 at 5:48 PM, TAY
>>                                 wee-beng<zonexo at gmail.com
>>                                 <mailto:zonexo at gmail.com>>  wrote:
>>                                 Hi,
>>
>>                                 Do you mean my method is wrong?
>>
>>                                 I am following the template of ex22f,
>>
>>                                 where the variables are declared as :
>>
>>                                 PetscScalar  v(5)
>>
>>                                 MatStencil   row(4),col(4,5)
>>
>>                                 Hence,
>>
>>                                 for the neumann BC
>>
>>                                 num = 1
>>
>>                                                if (j/=0) then
>>
>>                                                    v(num) = -rho*HxdHy
>>
>>                                                  
>>                                  col(MatStencil_i,num) = i
>>
>>                                                  
>>                                  col(MatStencil_j,num) = j-1
>>
>>                                                    num = num + 1
>>
>>                                                end if
>>
>>                                                if (i/=0) then
>>
>>                                                    v(num) = -rho*HydHx
>>
>>                                                  
>>                                  col(MatStencil_i,num) = i-1
>>
>>                                                  
>>                                  col(MatStencil_j,num) = j
>>
>>                                                    num = num + 1
>>
>>                                                end if
>>
>>                                                if (i/=mx-1) then
>>
>>                                                    v(num) = -rho*HydHx
>>
>>                                                  
>>                                  col(MatStencil_i,num) = i+1
>>
>>                                                  
>>                                  col(MatStencil_j,num) = j
>>
>>                                                    num = num + 1
>>
>>                                                end if
>>
>>                                                if (j/=my-1) then
>>
>>                                                    v(num) = -rho*HxdHy
>>
>>                                                  
>>                                  col(MatStencil_i,num) = i
>>
>>                                                  
>>                                  col(MatStencil_j,num) = j+1
>>
>>                                                    num = num + 1
>>
>>                                                end if
>>
>>                                                v(num) =
>>                                 ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>
>>                                                print *, v
>>
>>                                                col(MatStencil_i,num) = i
>>
>>                                                col(MatStencil_j,num) = j
>>
>>                                                !num = num + 1
>>
>>                                                call
>>                                 MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>
>>                                 I do not get any more out of range
>>                                 error. However,my ans is still
>>                                 different from that of ex29 in C.
>>
>>                                 This is very simple. You have an
>>                                 error in your code. Checking it is
>>                                 very simple: run the code and
>>                                 break in MatSetValues(). Make sure
>>                                 ex29 makes calls with exactly the
>>                                 same indices as your ex29f.
>>
>>                                     Matt
>>
>>                                 Yours sincerely,
>>
>>                                 TAY wee-beng
>>
>>                                 -- 
>>                                 What most experimenters take for
>>                                 granted before they begin their
>>                                 experiments is infinitely more
>>                                 interesting than any results to which
>>                                 their experiments lead.
>>                                 -- Norbert Wiener
>>
>>
>>
>>                             -- 
>>                             What most experimenters take for granted
>>                             before they begin their experiments is
>>                             infinitely more interesting than any
>>                             results to which their experiments lead.
>>                             -- Norbert Wiener
>>
>>
>>
>>                         -- 
>>                         What most experimenters take for granted
>>                         before they begin their experiments is
>>                         infinitely more interesting than any results
>>                         to which their experiments lead.
>>                         -- Norbert Wiener
>>
>>
>>
>>                     -- 
>>                     What most experimenters take for granted before
>>                     they begin their experiments is infinitely more
>>                     interesting than any results to which their
>>                     experiments lead.
>>                     -- Norbert Wiener
>>
>>
>>
>>                 -- 
>>                 What most experimenters take for granted before they
>>                 begin their experiments is infinitely more
>>                 interesting than any results to which their
>>                 experiments lead.
>>                 -- Norbert Wiener
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/bc27acb1/attachment-0001.htm>

From aron.ahmadia at kaust.edu.sa  Tue May  8 14:18:44 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Tue, 8 May 2012 22:18:44 +0300
Subject: [petsc-users] petsc4py and numpy
In-Reply-To: <97F10C5D-0BFB-41BA-B163-BD6CABA0D90F@ira.uka.de>
References: <mailman.2116.1336490734.4646.petsc-users@mcs.anl.gov>
	<97F10C5D-0BFB-41BA-B163-BD6CABA0D90F@ira.uka.de>
Message-ID: <CADVoUZTTfneMbc-xtBf6riTDdx4EetmBdcZF669mggBrweoRJA@mail.gmail.com>

>
> Thanks for the hint. I was wondering whether I should start with a working
> prototype based on numpy, then do parallelization with PETSc on demand - or
> start with PETSc wherever matrices and vectors are needed.
>

You need to write parallel code in a Single Program Multiple Data fashion
if you want it to scale well to hundreds or thousands of processors.  This
is the best model supported by mpi4py and petsc4py.

A
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/161e8083/attachment.htm>

From john.mousel at gmail.com  Tue May  8 14:41:26 2012
From: john.mousel at gmail.com (John Mousel)
Date: Tue, 8 May 2012 14:41:26 -0500
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
	ksp
In-Reply-To: <4FA97132.3080103@gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
	<4FA42CFC.8090002@gmail.com>
	<CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>
	<4FA44D5C.6010205@gmail.com>
	<CAMYG4Gm9cJxm1D0UmO9Fepv93uEHYhYDCcU1rKvsTU=D_+v9qQ@mail.gmail.com>
	<4FA97132.3080103@gmail.com>
Message-ID: <CA+zegpnTN40NUz+hD=BCqgv2kyz8sywzGDz+jDeVH3kXb+LSBw@mail.gmail.com>

TAY wee-bing,

If you want to be a programmer that writes interesting and reliable code,
you need to be willing to use the tools of the trade. I can't think of a
bigger time-saver than Valgrind. I would suggest that you learn to use it
and use it a lot. I bet it will lead you to the root of your problem pretty
quickly.

John

On Tue, May 8, 2012 at 2:17 PM, TAY wee-beng <zonexo at gmail.com> wrote:

>  Hi,
>
> I compiled and run my code under visual studio 2008 with intel fortran.
> Everything works ok.
>
> However, when I tried to run the code in linux, I got the error as below.
> The error happens when KSPSetUp(ksp,ierr) is called.
>
> However, I am not able to print VecView or MatView to view if there's any
> errors. Is there any recommendation for debugging? I hope I do not need to
> valgrind if possible.
>
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC
> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] DM user function line 0 unknownunknown
> [0]PETSC ERROR: [0] DMComputeFunction line 2085
> /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
> [0]PETSC ERROR: [0] KSPSetUp line 182
> /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development HG revision:
> 7ecdd63ec420b1659b960e65d96e822c5ac1a968  HG Date: Mon May 07 21:42:26 2012
> -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./ex29f on a petsc-3.2 named hpc12 by wtay Tue May  8
> 20:45:42 2012
> [0]PETSC ERROR: Libraries linked from
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/lib
> [0]PETSC ERROR: Configure run at Tue May  8 10:47:59 2012
> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/
> --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/
> --with-debugging=1 --download-hypre=1
> --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_debug --known-mpi-shared=1
> --with-shared-libraries
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> with errorcode 59.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
>
> Yours sincerely,
>
> TAY wee-beng
>
>
> On 5/5/2012 1:43 AM, Matthew Knepley wrote:
>
> On Fri, May 4, 2012 at 5:42 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>  Hi,
>>
>> I wonder if you people are interested to include my ex29 fortran version
>> in the petsc examples, which can help people who are using fortran.
>>
>
>  Yes, we will definitely include it. Please send the source, and a
> representative output with run options.
>
>    Thanks,
>
>       Matt
>
>
>>  Thanks.
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>> On 4/5/2012 9:28 PM, Matthew Knepley wrote:
>>
>> On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>>
>>> On 4/5/2012 9:16 PM, Barry Smith wrote:
>>>
>>>>    Do an hg pull and then run make in src/mat/interface/ftn-custom/
>>>>
>>>>     Then it should link.
>>>>
>>>>    Barry
>>>>
>>>>    There was a E missing from the all caps name of the function.
>>>>
>>> After hg pull, I did:
>>>
>>> cd src//mat/interface/ftn-custom/
>>>
>>> User at User-PC /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom
>>> $ make
>>> make[1]: Warning: File
>>> `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)'
>>> has modification time 787 s in the future
>>> make[1]: Nothing to be done for `libc'.
>>> make[1]: warning:  Clock skew detected.  Your build may be incomplete.
>>>
>>> But it still can't work.
>>>
>>
>>  Something is messed up with the clock on this machine.
>>
>>  HOWEVER, development requires certain basic skills in order to debug
>> your work. We
>> cannot be the ones debugging your code. Now
>>
>>    nm $PETSC_ARCH/lib/libpetsc.a | grep -i MatNullSpaceRemove
>>
>>   will look for the symbol.
>>
>>      Matt
>>
>>
>>>
>>>>
>>>> On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:
>>>>
>>>>  On Fri, May 4, 2012 at 3:01 PM, TAY wee-beng<zonexo at gmail.com>  wrote:
>>>>>
>>>>> On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>>>>>
>>>>>> On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng<zonexo at gmail.com>
>>>>>>  wrote:
>>>>>>
>>>>>> On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>>>>>
>>>>>>> On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>  wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Is there anything else I can try to get it working right?
>>>>>>>
>>>>>>> The MatGetNullSpaceRemove() is missing.
>>>>>>>
>>>>>> Where should I add MatGetNullSpaceRemove and what are its syntax? I
>>>>>> googled but there's no results.
>>>>>>
>>>>>> Fixed in p;etsc-dev:
>>>>>>
>>>>>>
>>>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>>>>>>
>>>>> I just compiled the updated petsc-dev but I got the same error msg
>>>>> when I use:
>>>>>
>>>>> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>>
>>>>> error LNK2019: unresolved external symbol MATNULLSPACEREMOVE
>>>>> referenced in function COMPUTERHS
>>>>> 1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120: 1 unresolved
>>>>> externals
>>>>>
>>>>> That function is in:
>>>>>
>>>>>   src/mat/interface/ftn-custom/zmatrixf.c
>>>>>
>>>>> Did that compile? Can you see the symbol in libpetsc.a?
>>>>>
>>>>>     Matt
>>>>>
>>>>>       Matt
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>>    Matt
>>>>>>>
>>>>>>> Thanks?
>>>>>>>
>>>>>>>
>>>>>>> On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>>>>>
>>>>>>>> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>  wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I did a MatView and VecView on both C and Fortran, right after Mat
>>>>>>>> and Vec assembly. I have attached the printout below. They are exactly the
>>>>>>>> same, but yet the result is different in Neumann condition. However, the
>>>>>>>> dirichlet condition gives the correct ans. Is there anything else that
>>>>>>>> could be wrong even if the Mat and Vec are the same?
>>>>>>>>
>>>>>>>> Did you set the null space for the matrix when you have Neumann
>>>>>>>> conditions?
>>>>>>>>
>>>>>>> Yes, for the matrix, I set as:
>>>>>>>
>>>>>>> call
>>>>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>>>>
>>>>>>>     call MatSetNullSpace(jac,nullspace,ierr)
>>>>>>>
>>>>>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>>
>>>>>>> for the Vec,
>>>>>>>
>>>>>>> call
>>>>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>>>>
>>>>>>>     !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>>>>
>>>>>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>>
>>>>>>> MatNullSpaceRemove was comment out because there's error during
>>>>>>> linking
>>>>>>>
>>>>>>>
>>>>>>>     Matt
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Fortran:
>>>>>>>>
>>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>>   type: seqaij
>>>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>>>>>   type: mpi
>>>>>>>> Process [0]
>>>>>>>> 0.25
>>>>>>>> 0.0205213
>>>>>>>> 1.135e-005
>>>>>>>> 0.0205213
>>>>>>>> 0.00168449
>>>>>>>> 9.31663e-007
>>>>>>>> 1.135e-005
>>>>>>>> 9.31663e-007
>>>>>>>> 5.15289e-010
>>>>>>>> Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>>>>>>   type: mpi
>>>>>>>> Process [0]
>>>>>>>> 0.14924
>>>>>>>> 0.0242397
>>>>>>>> -0.0260347
>>>>>>>> 0.0242397
>>>>>>>> -0.0256192
>>>>>>>> -0.0400102
>>>>>>>> -0.0260347
>>>>>>>> -0.0400102
>>>>>>>> -0.0400102
>>>>>>>> Press any key to continue . . .
>>>>>>>>
>>>>>>>> C:
>>>>>>>>
>>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>>   type: seqaij
>>>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>>>> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>>>>>>   type: mpi
>>>>>>>> Process [0]
>>>>>>>> 0.25
>>>>>>>> 0.0205212
>>>>>>>> 1.135e-05
>>>>>>>> 0.0205212
>>>>>>>> 0.00168449
>>>>>>>> 9.31663e-07
>>>>>>>> 1.135e-05
>>>>>>>> 9.31663e-07
>>>>>>>> 5.15288e-10
>>>>>>>> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>>>>>>   type: mpi
>>>>>>>> Process [0]
>>>>>>>> 0.139311
>>>>>>>> 0.0305751
>>>>>>>> -0.0220633
>>>>>>>> 0.0305751
>>>>>>>> -0.0135158
>>>>>>>> -0.042185
>>>>>>>> -0.0220633
>>>>>>>> -0.042185
>>>>>>>> -0.058449
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Yours sincerely,
>>>>>>>>
>>>>>>>> TAY wee-beng
>>>>>>>>
>>>>>>>>
>>>>>>>> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>>>>>
>>>>>>>>> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>>  wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Do you mean my method is wrong?
>>>>>>>>>
>>>>>>>>> I am following the template of ex22f,
>>>>>>>>>
>>>>>>>>> where the variables are declared as :
>>>>>>>>>
>>>>>>>>> PetscScalar  v(5)
>>>>>>>>>
>>>>>>>>> MatStencil   row(4),col(4,5)
>>>>>>>>>
>>>>>>>>> Hence,
>>>>>>>>>
>>>>>>>>> for the neumann BC
>>>>>>>>>
>>>>>>>>> num = 1
>>>>>>>>>
>>>>>>>>>                if (j/=0) then
>>>>>>>>>
>>>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>>>>
>>>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>>>>
>>>>>>>>>                    col(MatStencil_j,num) = j-1
>>>>>>>>>
>>>>>>>>>                    num = num + 1
>>>>>>>>>
>>>>>>>>>                end if
>>>>>>>>>
>>>>>>>>>                if (i/=0) then
>>>>>>>>>
>>>>>>>>>                    v(num) = -rho*HydHx
>>>>>>>>>
>>>>>>>>>                    col(MatStencil_i,num) = i-1
>>>>>>>>>
>>>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>>>>
>>>>>>>>>                    num = num + 1
>>>>>>>>>
>>>>>>>>>                end if
>>>>>>>>>
>>>>>>>>>                if (i/=mx-1) then
>>>>>>>>>
>>>>>>>>>                    v(num) = -rho*HydHx
>>>>>>>>>
>>>>>>>>>                    col(MatStencil_i,num) = i+1
>>>>>>>>>
>>>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>>>>
>>>>>>>>>                    num = num + 1
>>>>>>>>>
>>>>>>>>>                end if
>>>>>>>>>
>>>>>>>>>                if (j/=my-1) then
>>>>>>>>>
>>>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>>>>
>>>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>>>>
>>>>>>>>>                    col(MatStencil_j,num) = j+1
>>>>>>>>>
>>>>>>>>>                    num = num + 1
>>>>>>>>>
>>>>>>>>>                end if
>>>>>>>>>
>>>>>>>>>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>>>>>>
>>>>>>>>>                print *, v
>>>>>>>>>
>>>>>>>>>                col(MatStencil_i,num) = i
>>>>>>>>>
>>>>>>>>>                col(MatStencil_j,num) = j
>>>>>>>>>
>>>>>>>>>                !num = num + 1
>>>>>>>>>
>>>>>>>>>                call
>>>>>>>>> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>>>>>>
>>>>>>>>> I do not get any more out of range error. However,my ans is still
>>>>>>>>> different from that of ex29 in C.
>>>>>>>>>
>>>>>>>>> This is very simple. You have an error in your code. Checking it
>>>>>>>>> is very simple: run the code and
>>>>>>>>> break in MatSetValues(). Make sure ex29 makes calls with exactly
>>>>>>>>> the same indices as your ex29f.
>>>>>>>>>
>>>>>>>>>     Matt
>>>>>>>>>
>>>>>>>>> Yours sincerely,
>>>>>>>>>
>>>>>>>>> TAY wee-beng
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/5f612909/attachment-0001.htm>

From knepley at gmail.com  Tue May  8 15:43:43 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 8 May 2012 16:43:43 -0400
Subject: [petsc-users] petsc4py and numpy
In-Reply-To: <CAEcYPwArCJVjaMshbJCCYNpEj2bC9BFarpYRJADe3ZmTyX-2aA@mail.gmail.com>
References: <49B110C8-A0B1-4BE7-A2D9-867983AF6EB4@ira.uka.de>
	<CAMYG4Gn6ENysDbi90R6uKwEsOpyWB6uWSpcJsuTsg4fGSZ79Qw@mail.gmail.com>
	<CAEcYPwDC60YhwGfKgnRbYHw3xn3gV-wX5LZArNbM5cP6Xyu7wg@mail.gmail.com>
	<CAMYG4GnYv5MVotiBipCSvQREZ1c4KOxfSKpiKxBf=jsTRK4uMQ@mail.gmail.com>
	<CAEcYPwArCJVjaMshbJCCYNpEj2bC9BFarpYRJADe3ZmTyX-2aA@mail.gmail.com>
Message-ID: <CAMYG4GnRujdqyPZ=2raXN3K0iUzSVgDchw=+EoUtGJG_c6T=fw@mail.gmail.com>

On Tue, May 8, 2012 at 3:11 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> On 8 May 2012 12:25, Matthew Knepley <knepley at gmail.com> wrote:
> > On Tue, May 8, 2012 at 11:15 AM, Lisandro Dalcin <dalcinl at gmail.com>
> wrote:
> >>
> >> On 8 May 2012 12:04, Matthew Knepley <knepley at gmail.com> wrote:
> >> > On Tue, May 8, 2012 at 11:00 AM, Christian Staudt
> >> > <christian.staudt at ira.uka.de> wrote:
> >> >>
> >> >> Matthew Knepley <knepley at gmail.com> wrote:
> >> >>
> >> >> a) Is there an easy and efficient way to convert a numpy.ndarray to a
> >> >>
> >> >> PETSc.Vec or PETSc.Mat (and vice-versa)?  (A PETSc.Vec.getArray()
> >> >> returns
> >> >> a
> >> >>
> >> >> numpy.ndarray, though I have found no such method for PETSc.Mat yet.
> )
> >> >>
> >> >>
> >> >>
> >> >> If you are using dense matrices (MATDENSE) then MatGetArray() works.
> >> >>
> >> >>
> >> >> I don't think this is implemented in petsc4py: At least, there's no
> >> >> method
> >> >> PETSc.Mat.getArray()
> >> >
> >> >
> >> > If you ask Lisandro for it, he will put it in.
> >> >
> >>
> >> Matt, I've never put that in just because I'm not sure how to make it
> >> general. PETSc have many different matrix formats, and MatGetArray()
> >> call is like a hack, where you can get very different things (the "1D"
> >> values of an AIJ matrix, the "2D" values of a dense matrix in Fortran
> >> order). What about MPIAIJ, what should I return to users? Can you
> >> suggest some approach or API to implement this (I mean, how to get
> >> arrays for the various matrix types)?
> >
> >
> > I think the whole point is that you should not do it. However, that has
> > never stopped us from giving users
> > what they ask for. We are not int he business of reforming computing
> > practices, which is why we have
> > MatGetArray(). I would make a wrapper that failed unless the type was
> > MATDENSE.
> >
>
> Yea, that's really easy.
>
> However, Is there anything better for AIJ matrices? Right now, we have
> mat.getValuesCSR()... it works by calling MatGetRow() row by row twice
> (first pass to count and allocate, next to get values and copy to
> NumPy arrays).


For AIJ specifically, you can use MatGetIJ() and MatGetArray().

   Matt


>
> --
> Lisandro Dalcin
> ---------------
> CIMEC (INTEC/CONICET-UNL)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> 3000 Santa Fe, Argentina
> Tel: +54-342-4511594 (ext 1011)
> Tel/Fax: +54-342-4511169
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120508/fd35c686/attachment.htm>

From manfred.gratt at uibk.ac.at  Wed May  9 01:36:53 2012
From: manfred.gratt at uibk.ac.at (Manfred Gratt)
Date: Wed, 09 May 2012 08:36:53 +0200
Subject: [petsc-users] petsc gcc intel and compilerflags
In-Reply-To: <8AACE68F-7196-40E9-83F7-2DA630C02362@mcs.anl.gov>
References: <4FA8CC27.5080808@uibk.ac.at>	<CAM9tzSnQ62FVR7WjphZ_DG5J+VcQGLbW=Oes=Toe2sJfOKd-RQ@mail.gmail.com>
	<8AACE68F-7196-40E9-83F7-2DA630C02362@mcs.anl.gov>
Message-ID: <4FAA1085.9030602@uibk.ac.at>

Barry,
thank you for your reply. I only thought that there was a problem with 
gcc and higher optimizations.  I have for example a problem with gcc 4.6 
as when I compile my program with -O3 the result would then not converge 
any more.  With the optimisation -O2 it works all well also all other 
gcc I had before made no problems. So I thought as configure set gcc 
with only -O and icc with -O3 there was generally a problem with gcc and 
optimizations. But if not it is good.

Manfred

Barry Smith schrieb:
>   Manfred,
>
>     ./configure is not smart enough to know what optimization flags are best for all compilers/systems. Thus we have relatively straightforward defaults. As Jed notes, if you know what flags are most appropriate for your build you must pass them in to ./configure
>
>    Barry
>
> On May 8, 2012, at 7:59 AM, Jed Brown wrote:
>
>   
>> On Tue, May 8, 2012 at 1:32 AM, Manfred Gratt <manfred.gratt at uibk.ac.at> wrote:
>> when I compile petsc with --with-debugging=0 with gcc and icc in configure there are different optimization flags set depending on compiler. So is with gcc -O set wich means -O1 optimization. While with icc -O3 optimisation is set. Is there a reason only icc gets the high performance flag while gcc not? Are there any problems when some optimizations are activated on gcc?
>>
>> I don't know why only -O1 is used with gcc, but you can pass COPTFLAGS='-O3 -march=native' or whatever you like to configure to make it use those instead.
>>     
>
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120509/833f1ab2/attachment.htm>

From martenullberg at gmail.com  Wed May  9 11:14:18 2012
From: martenullberg at gmail.com (=?ISO-8859-1?Q?M=E5rten_Ullberg?=)
Date: Wed, 9 May 2012 18:14:18 +0200
Subject: [petsc-users] Method and PC for indefinite FE-problem
Message-ID: <CAP3JPOkBR4v6OAg=e3eLEf81Q5ByG-WSZL4Z59_jYGSLLPh_rQ@mail.gmail.com>

Dear all,

I am currently working on some large soil-structure interaction
problems and therefor I have a finite element discretization that is
going be solved in the frequency domain. This results in a symmetric,
but indefinite matrix. I have not been able to find a proper method to
use when solving it with PETSc.

I was wondering if you guys have any recommendation on what to try,
both concerning method but especially the PC? Is it recommended to use
a composite PC for indefinite problems? I have tried the methods (such
as MINRES, SYMMLQ) that are suggested for indefinite problems with no
success. For small problems, GMRES+ASM+LU works fine, but for larger
the memory consumption goes off the chart. I've also been able to use
MUMPS but find it both memory and time consuming.

I apologize for my lack of knowledge about iterative methods,

Best regards,
M?rten

From knepley at gmail.com  Wed May  9 11:38:40 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 9 May 2012 12:38:40 -0400
Subject: [petsc-users] Method and PC for indefinite FE-problem
In-Reply-To: <CAP3JPOkBR4v6OAg=e3eLEf81Q5ByG-WSZL4Z59_jYGSLLPh_rQ@mail.gmail.com>
References: <CAP3JPOkBR4v6OAg=e3eLEf81Q5ByG-WSZL4Z59_jYGSLLPh_rQ@mail.gmail.com>
Message-ID: <CAMYG4Gm-VQQMj_V+f4b+0Gk0E4LWOFmUant=LH8j4UHSXbKoSQ@mail.gmail.com>

On Wed, May 9, 2012 at 12:14 PM, M?rten Ullberg <martenullberg at gmail.com>wrote:

> Dear all,
>
> I am currently working on some large soil-structure interaction
> problems and therefor I have a finite element discretization that is
> going be solved in the frequency domain. This results in a symmetric,
> but indefinite matrix. I have not been able to find a proper method to
> use when solving it with PETSc.
>
> I was wondering if you guys have any recommendation on what to try,
> both concerning method but especially the PC? Is it recommended to use
> a composite PC for indefinite problems? I have tried the methods (such
> as MINRES, SYMMLQ) that are suggested for indefinite problems with no
> success. For small problems, GMRES+ASM+LU works fine, but for larger
> the memory consumption goes off the chart. I've also been able to use
> MUMPS but find it both memory and time consuming.
>
> I apologize for my lack of knowledge about iterative methods,
>

It sounds like you are trying to solve the Helmholtz equation. No one knows
how to do this with iterative methods. Sorry the answer is so pessimistic,
but
this is the truth unfortunately. If you can find a method you think will
work, we
can tell you how to do it in PETSc.

    Matt


> Best regards,
> M?rten

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120509/a6861aa5/attachment.htm>

From martenullberg at gmail.com  Wed May  9 15:57:37 2012
From: martenullberg at gmail.com (=?ISO-8859-1?Q?M=E5rten_Ullberg?=)
Date: Wed, 9 May 2012 22:57:37 +0200
Subject: [petsc-users] Method and PC for indefinite FE-problem
In-Reply-To: <CAMYG4Gm-VQQMj_V+f4b+0Gk0E4LWOFmUant=LH8j4UHSXbKoSQ@mail.gmail.com>
References: <CAP3JPOkBR4v6OAg=e3eLEf81Q5ByG-WSZL4Z59_jYGSLLPh_rQ@mail.gmail.com>
	<CAMYG4Gm-VQQMj_V+f4b+0Gk0E4LWOFmUant=LH8j4UHSXbKoSQ@mail.gmail.com>
Message-ID: <CAP3JPOn7jXdks_SagJjPCxqs6B2w8bi+TvKe3nM3QK8ZSKszwQ@mail.gmail.com>

Thanks for the reply,

it is actually a steady-state solution I'm looking for. Didn't think
it would be that hard. I'll let you know if find any appropriate
method to go with my problem.

One other question, is it big difference between the symmetric QMR
method from the two other QMR methods implemented in PETSc? I've found
some references too that method and indefinite problems.

Thanks again,
M?rten

2012/5/9 Matthew Knepley <knepley at gmail.com>:
> On Wed, May 9, 2012 at 12:14 PM, M?rten Ullberg <martenullberg at gmail.com>
> wrote:
>>
>> Dear all,
>>
>> I am currently working on some large soil-structure interaction
>> problems and therefor I have a finite element discretization that is
>> going be solved in the frequency domain. This results in a symmetric,
>> but indefinite matrix. I have not been able to find a proper method to
>> use when solving it with PETSc.
>>
>> I was wondering if you guys have any recommendation on what to try,
>> both concerning method but especially the PC? Is it recommended to use
>> a composite PC for indefinite problems? I have tried the methods (such
>> as MINRES, SYMMLQ) that are suggested for indefinite problems with no
>> success. For small problems, GMRES+ASM+LU works fine, but for larger
>> the memory consumption goes off the chart. I've also been able to use
>> MUMPS but find it both memory and time consuming.
>>
>> I apologize for my lack of knowledge about iterative methods,
>
>
> It sounds like you are trying to solve the Helmholtz equation. No one knows
> how to do this with iterative methods. Sorry the answer is so pessimistic,
> but
> this is the truth unfortunately. If you can find a method you think will
> work, we
> can tell you how to do it in PETSc.
>
> ? ? Matt
>
>>
>> Best regards,
>> M?rten
>
> --
> What most experimenters take for granted before they begin their experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener

From jedbrown at mcs.anl.gov  Wed May  9 16:10:37 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 9 May 2012 16:10:37 -0500
Subject: [petsc-users] Method and PC for indefinite FE-problem
In-Reply-To: <CAP3JPOn7jXdks_SagJjPCxqs6B2w8bi+TvKe3nM3QK8ZSKszwQ@mail.gmail.com>
References: <CAP3JPOkBR4v6OAg=e3eLEf81Q5ByG-WSZL4Z59_jYGSLLPh_rQ@mail.gmail.com>
	<CAMYG4Gm-VQQMj_V+f4b+0Gk0E4LWOFmUant=LH8j4UHSXbKoSQ@mail.gmail.com>
	<CAP3JPOn7jXdks_SagJjPCxqs6B2w8bi+TvKe3nM3QK8ZSKszwQ@mail.gmail.com>
Message-ID: <CAM9tzSm6_zFNS2xen7rVciVTw1bjeC_T8U+QzPrkFJ-31cYDig@mail.gmail.com>

On Wed, May 9, 2012 at 3:57 PM, M?rten Ullberg <martenullberg at gmail.com>wrote:

> it is actually a steady-state solution I'm looking for. Didn't think
> it would be that hard. I'll let you know if find any appropriate
> method to go with my problem.
>

The indefinite Helmholtz problem is notoriously difficult at high
frequency.

http://www.mathe.tu-freiberg.de/~ernst/PubArchive/helmholtzDurham.pdf

For low frequency, many standard methods will work.


>
> One other question, is it big difference between the symmetric QMR
> method from the two other QMR methods implemented in PETSc? I've found
> some references too that method and indefinite problems.
>

Just try it. The QMR algorithms are not equivalent, but they have worked
similarly for some other people solving indefinite problems. If you don't
have an imaginary shift, it would be standard to use MINRES.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120509/600451a0/attachment.htm>

From Daniel.Weflen at Colorado.EDU  Wed May  9 18:01:34 2012
From: Daniel.Weflen at Colorado.EDU (Daniel Christian Weflen)
Date: Wed, 09 May 2012 17:01:34 -0600
Subject: [petsc-users] Getting 0 for both start and end with
	MatGetOwnershipRange()
Message-ID: <4FAAF74E.8000502@colorado.edu>

I have the following code:

     Mat H;
     PetscInt mat_size=x_npts*y_npts;
     ierr=MatCreate(PETSC_COMM_WORLD,&H);CHKERRQ(ierr);
     
ierr=MatSetSizes(H,PETSC_DECIDE,PETSC_DECIDE,mat_size,mat_size);CHKERRQ(ierr);
     ierr=MatSetType(H,MATSEQMAIJ);CHKERRQ(ierr);
     PetscInt* nonzeros;
     PetscInt start_row,end_row,row_index;
     MatGetOwnershipRange(H,&start_row,&end_row);

When I check the values of start_row and end_row right after calling 
MatGetOwnershipRange, both are 0. What am I doing wrong here? Am I 
calling MatCreate(), MatSetSizes() and MatSetType() in the wrong order?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120509/3da7a7ae/attachment.htm>

From dalcinl at gmail.com  Wed May  9 18:35:55 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 9 May 2012 20:35:55 -0300
Subject: [petsc-users] Getting 0 for both start and end with
	MatGetOwnershipRange()
In-Reply-To: <4FAAF74E.8000502@colorado.edu>
References: <4FAAF74E.8000502@colorado.edu>
Message-ID: <CAEcYPwDjV6TDFMJc0AEpfXWxcbwFziXc3fS81w7kTtdhGEm2qg@mail.gmail.com>

On 9 May 2012 20:01, Daniel Christian Weflen <Daniel.Weflen at colorado.edu> wrote:
> I have the following code:
>
> ??? Mat H;
> ??? PetscInt mat_size=x_npts*y_npts;
> ??? ierr=MatCreate(PETSC_COMM_WORLD,&H);CHKERRQ(ierr);
>
> ierr=MatSetSizes(H,PETSC_DECIDE,PETSC_DECIDE,mat_size,mat_size);CHKERRQ(ierr);
> ??? ierr=MatSetType(H,MATSEQMAIJ);CHKERRQ(ierr);
> ??? PetscInt* nonzeros;
> ??? PetscInt start_row,end_row,row_index;
> ??? MatGetOwnershipRange(H,&start_row,&end_row);
>
> When I check the values of start_row and end_row right after calling
> MatGetOwnershipRange, both are 0. What am I doing wrong here? Am I calling
> MatCreate(), MatSetSizes() and MatSetType() in the wrong order?

Call MatSetUp() after MatSetType().


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From Daniel.Weflen at Colorado.EDU  Wed May  9 19:09:21 2012
From: Daniel.Weflen at Colorado.EDU (Daniel Christian Weflen)
Date: Wed, 09 May 2012 18:09:21 -0600
Subject: [petsc-users] Getting 0 for both start and end with
	MatGetOwnershipRange()
In-Reply-To: <CAEcYPwDjV6TDFMJc0AEpfXWxcbwFziXc3fS81w7kTtdhGEm2qg@mail.gmail.com>
References: <4FAAF74E.8000502@colorado.edu>
	<CAEcYPwDjV6TDFMJc0AEpfXWxcbwFziXc3fS81w7kTtdhGEm2qg@mail.gmail.com>
Message-ID: <4FAB0731.4000907@colorado.edu>

I still get 0 for both the start and end rows.

-Dan

On 05/09/2012 05:35 PM, Lisandro Dalcin wrote:
> On 9 May 2012 20:01, Daniel Christian Weflen<Daniel.Weflen at colorado.edu>  wrote:
>> I have the following code:
>>
>>      Mat H;
>>      PetscInt mat_size=x_npts*y_npts;
>>      ierr=MatCreate(PETSC_COMM_WORLD,&H);CHKERRQ(ierr);
>>
>> ierr=MatSetSizes(H,PETSC_DECIDE,PETSC_DECIDE,mat_size,mat_size);CHKERRQ(ierr);
>>      ierr=MatSetType(H,MATSEQMAIJ);CHKERRQ(ierr);
>>      PetscInt* nonzeros;
>>      PetscInt start_row,end_row,row_index;
>>      MatGetOwnershipRange(H,&start_row,&end_row);
>>
>> When I check the values of start_row and end_row right after calling
>> MatGetOwnershipRange, both are 0. What am I doing wrong here? Am I calling
>> MatCreate(), MatSetSizes() and MatSetType() in the wrong order?
> Call MatSetUp() after MatSetType().
>
>

From mike.hui.zhang at hotmail.com  Thu May 10 04:16:44 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Thu, 10 May 2012 11:16:44 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
Message-ID: <BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>

Hi Dmitry,

is there any news about PCGASM? 

thanks,
Hui

On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:

> Okay, thanks.
> I'll take a look.
> 
> Dmitry.
> 
> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> For reference, my results are attached.
> 
> asm1.txt for asm with 1 process,
> asm2.txt for asm with 2 processes,
> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
> gasm2.txt for gasm with 2 processes
> 
> 
> 
> 
> 
> 
> thank you,
> Hui
> 
> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
> 
>> 
>> 
>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> 
>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>> 
>>> 
>>> 
>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> I have a new problem: the results from ASM and GASM are different and it seems
>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>> each subdomain supported only by one subdomain. There are no problems when
>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>> with GASM but no problems with ASM. 
>>> 
>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>> one processor and there seems no problem with GASM. But when I use run my program 
>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>> number is different from the first case and is much larger.  On the other hand
>>> ASM has no such problem.
>>>  
>>> Are the solutions the same?
>>> What problem are you solving?
>> 
>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>> two processors. But GASM did not.  
>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>> I'm solving the Helmholtz equation.  Maybe 
>> I can prepare a simpler example to show this difference.
>> That would be helpful.  
>> Thanks.
>> 
>> Dmitry. 
>> 
>>> 
>>> Dmitry. 
>>> 
>>> 
>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>> 
>>>> You should be able to. 
>>>> This behavior is the same as in PCASM,
>>>> except in GASM the matrices live on subcommunicators.
>>>> I am in transit right now, but I can take a closer look in Friday.
>>>> 
>>>> Dmitry
>>>> 
>>>> 
>>>> 
>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> 
>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>> 
>>>>>> Hi Dmitry,
>>>>>> 
>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>> 
>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>> 
>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>> 
>>>>> As I tested, the row and col are always the same. 
>>>>> 
>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>> in the above func()?
>>>>> 
>>>>> thanks,
>>>>> Hui
>>>>> 
>>>>>> 
>>>>>> thanks,
>>>>>> Hui
>>>>>> 
>>>>>> 
>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>> 
>>>>>>> Yes, that's right.
>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>> 
>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>> 
>>>>>>> Dmitry.
>>>>>>> 
>>>>>>> 
>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>> 
>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Hui
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
>> 
>> 
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120510/dbdb2bb8/attachment-0001.htm>

From knepley at gmail.com  Thu May 10 06:21:39 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 10 May 2012 07:21:39 -0400
Subject: [petsc-users] Getting 0 for both start and end with
	MatGetOwnershipRange()
In-Reply-To: <4FAB0731.4000907@colorado.edu>
References: <4FAAF74E.8000502@colorado.edu>
	<CAEcYPwDjV6TDFMJc0AEpfXWxcbwFziXc3fS81w7kTtdhGEm2qg@mail.gmail.com>
	<4FAB0731.4000907@colorado.edu>
Message-ID: <CAMYG4GmHW-ztFX1cwrjPRVHf3GxhCVNwVR_YysRBcNc9me7_Yw@mail.gmail.com>

On Wed, May 9, 2012 at 8:09 PM, Daniel Christian Weflen <
Daniel.Weflen at colorado.edu> wrote:

> I still get 0 for both the start and end rows.
>

1) Is mat_size 0?

2) If not, put these line in a standalone executable that shows the bug and
send it.

   Thanks,

     Matt


> -Dan
>
>
> On 05/09/2012 05:35 PM, Lisandro Dalcin wrote:
>
>> On 9 May 2012 20:01, Daniel Christian Weflen<Daniel.Weflen at colorado.**edu<Daniel.Weflen at colorado.edu>>
>>  wrote:
>>
>>> I have the following code:
>>>
>>>     Mat H;
>>>     PetscInt mat_size=x_npts*y_npts;
>>>     ierr=MatCreate(PETSC_COMM_**WORLD,&H);CHKERRQ(ierr);
>>>
>>> ierr=MatSetSizes(H,PETSC_**DECIDE,PETSC_DECIDE,mat_size,**
>>> mat_size);CHKERRQ(ierr);
>>>     ierr=MatSetType(H,MATSEQMAIJ);**CHKERRQ(ierr);
>>>     PetscInt* nonzeros;
>>>     PetscInt start_row,end_row,row_index;
>>>     MatGetOwnershipRange(H,&start_**row,&end_row);
>>>
>>> When I check the values of start_row and end_row right after calling
>>> MatGetOwnershipRange, both are 0. What am I doing wrong here? Am I
>>> calling
>>> MatCreate(), MatSetSizes() and MatSetType() in the wrong order?
>>>
>> Call MatSetUp() after MatSetType().
>>
>>
>>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120510/3272c272/attachment.htm>

From karpeev at mcs.anl.gov  Thu May 10 11:37:54 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Thu, 10 May 2012 11:37:54 -0500
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
Message-ID: <CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>

Hui,
There've been several changes to PCGASM ahead of the new release.
Let me go back and see if it affected the convergence problem.
Dmitry.

On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Hi Dmitry,
>
> is there any news about PCGASM?
>
> thanks,
> Hui
>
> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>
> Okay, thanks.
> I'll take a look.
>
> Dmitry.
>
> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>> For reference, my results are attached.
>>
>> asm1.txt for asm with 1 process,
>> asm2.txt for asm with 2 processes,
>> gasm1.txt for gasm with 1 process, (with the iteration numbers different
>> from others)
>> gasm2.txt for gasm with 2 processes
>>
>>
>>
>>
>>
>>
>> thank you,
>> Hui
>>
>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>
>>
>>
>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>
>>>
>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>
>>>
>>>
>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>
>>>> I have a new problem: the results from ASM and GASM are different and
>>>> it seems
>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are
>>>> with
>>>> each subdomain supported only by one subdomain. There are no problems
>>>> when
>>>> I did not modify submatrices.  But when I modify submatrices, there are
>>>> problems
>>>> with GASM but no problems with ASM.
>>>>
>>>> For example, I use two subdomains. In the first case each subdomain is
>>>> supported by
>>>> one processor and there seems no problem with GASM. But when I use run
>>>> my program
>>>> with only one proc. so that it supports both of the two subdomains, the
>>>> iteration
>>>> number is different from the first case and is much larger.  On the
>>>> other hand
>>>> ASM has no such problem.
>>>>
>>>
>>> Are the solutions the same?
>>> What problem are you solving?
>>>
>>>
>>> Yes, the solutions are the same. That's why ASM gives the same results
>>> with one or
>>> two processors. But GASM did not.
>>>
>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the
>> case of two domains per processor?
>>
>>> I'm solving the Helmholtz equation.  Maybe
>>> I can prepare a simpler example to show this difference.
>>>
>> That would be helpful.
>> Thanks.
>>
>> Dmitry.
>>
>>>
>>>
>>> Dmitry.
>>>
>>>>
>>>>
>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>
>>>> You should be able to.
>>>> This behavior is the same as in PCASM,
>>>> except in GASM the matrices live on subcommunicators.
>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>
>>>> Dmitry
>>>>
>>>>
>>>>
>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>
>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>
>>>> Hi Dmitry,
>>>>
>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another
>>>> question
>>>> on PCGASMSetModifySubMatrices(). The user provided function has the
>>>> prototype
>>>>
>>>> func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>
>>>> I think the coloumns from the parameter 'col' are always the same as
>>>> the rows
>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only
>>>> accepts
>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>
>>>>
>>>> As I tested, the row and col are always the same.
>>>>
>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for
>>>> the submat's
>>>> in the above func()?
>>>>
>>>> thanks,
>>>> Hui
>>>>
>>>>
>>>> thanks,
>>>> Hui
>>>>
>>>>
>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>
>>>> Yes, that's right.
>>>> There is no good way to help the user assemble the subdomains at the
>>>> moment beyond the 2D stuff.
>>>> It is expected that they are generated from mesh subdomains.
>>>> Each IS does carry the subdomains subcomm.
>>>>
>>>> There is ISColoringToList() that is supposed to convert a "coloring" of
>>>> indices to an array of ISs,
>>>> each having the indices with the same color and the subcomm that
>>>> supports that color. It is
>>>> largely untested, though.  You could try using it and give us feedback
>>>> on any problems you encounter.
>>>>
>>>> Dmitry.
>>>>
>>>>
>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<mike.hui.zhang at hotmail.com>
>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>
>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain
>>>>> supported by
>>>>> multiple processors, shall I always create the arguments 'is[s]' and
>>>>> 'is_local[s]'
>>>>> in a subcommunicator consisting of processors supporting the subdomain
>>>>> 's'?
>>>>>
>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>
>>>>> Thanks,
>>>>> Hui
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120510/8b4dfb2b/attachment.htm>

From Daniel.Weflen at Colorado.EDU  Thu May 10 12:52:06 2012
From: Daniel.Weflen at Colorado.EDU (Daniel Christian Weflen)
Date: Thu, 10 May 2012 11:52:06 -0600
Subject: [petsc-users] Getting 0 for both start and end with
	MatGetOwnershipRange()
In-Reply-To: <CAMYG4GmHW-ztFX1cwrjPRVHf3GxhCVNwVR_YysRBcNc9me7_Yw@mail.gmail.com>
References: <4FAAF74E.8000502@colorado.edu>
	<CAEcYPwDjV6TDFMJc0AEpfXWxcbwFziXc3fS81w7kTtdhGEm2qg@mail.gmail.com>
	<4FAB0731.4000907@colorado.edu>
	<CAMYG4GmHW-ztFX1cwrjPRVHf3GxhCVNwVR_YysRBcNc9me7_Yw@mail.gmail.com>
Message-ID: <4FAC0046.2010604@colorado.edu>

I changed the type from MATSEQMAIJ to MATSEQAIJ and it works correctly. 
mat_size is not 0, should I still send the standalone executable?

-Dan

On 05/10/2012 05:21 AM, Matthew Knepley wrote:
> On Wed, May 9, 2012 at 8:09 PM, Daniel Christian Weflen 
> <Daniel.Weflen at colorado.edu <mailto:Daniel.Weflen at colorado.edu>> wrote:
>
>     I still get 0 for both the start and end rows.
>
>
> 1) Is mat_size 0?
>
> 2) If not, put these line in a standalone executable that shows the 
> bug and send it.
>
>    Thanks,
>
>      Matt
>
>     -Dan
>
>
>     On 05/09/2012 05:35 PM, Lisandro Dalcin wrote:
>
>         On 9 May 2012 20:01, Daniel Christian
>         Weflen<Daniel.Weflen at colorado.edu
>         <mailto:Daniel.Weflen at colorado.edu>>  wrote:
>
>             I have the following code:
>
>                 Mat H;
>                 PetscInt mat_size=x_npts*y_npts;
>                 ierr=MatCreate(PETSC_COMM_WORLD,&H);CHKERRQ(ierr);
>
>             ierr=MatSetSizes(H,PETSC_DECIDE,PETSC_DECIDE,mat_size,mat_size);CHKERRQ(ierr);
>                 ierr=MatSetType(H,MATSEQMAIJ);CHKERRQ(ierr);
>                 PetscInt* nonzeros;
>                 PetscInt start_row,end_row,row_index;
>                 MatGetOwnershipRange(H,&start_row,&end_row);
>
>             When I check the values of start_row and end_row right
>             after calling
>             MatGetOwnershipRange, both are 0. What am I doing wrong
>             here? Am I calling
>             MatCreate(), MatSetSizes() and MatSetType() in the wrong
>             order?
>
>         Call MatSetUp() after MatSetType().
>
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120510/2dd9289a/attachment.htm>

From mike.hui.zhang at hotmail.com  Fri May 11 06:26:20 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Fri, 11 May 2012 13:26:20 +0200
Subject: [petsc-users] two-level Schwarz: PCMG or PCComposite
Message-ID: <BLU0-SMTP440ACD42EF2D3F61AC7C6F3CB170@phx.gbl>

I want to implement the two-level Schwarz method.  Let us denote P as the one-level
Schwarz projection and P_0 as the coarse projection.  If I want the hybrid form of 
preconditioned system as follows:

     (I-P_0) P (I-P_0) 

which is suitable:  PCMG or PCComposite?  It seems to me that PCComposite is good.
But PCMG is not suitable because it can only implement the following form:

     (I-P) P_0 (I-P)

Any opinions?  Thanks!



From mike.hui.zhang at hotmail.com  Fri May 11 07:00:11 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Fri, 11 May 2012 14:00:11 +0200
Subject: [petsc-users] two-level Schwarz: PCMG or PCComposite
In-Reply-To: <BLU0-SMTP440ACD42EF2D3F61AC7C6F3CB170@phx.gbl>
References: <BLU0-SMTP440ACD42EF2D3F61AC7C6F3CB170@phx.gbl>
Message-ID: <BLU0-SMTP314F3CBBA6561599085F075CB170@phx.gbl>


some corrections:

> I want to implement the two-level Schwarz method.  Let us denote P as the one-level
> Schwarz projection and P_0 as the coarse projection.  If I want the hybrid form of 
> preconditioned system as follows:
> 
>     (I-P_0) P (I-P_0) 

this should be 

      I  -  (I-P_0) (I-P) (I-P_0)

in general case.  Only when P_0 is exact solve, we have the precedent form.
 	 
> which is suitable:  PCMG or PCComposite?  It seems to me that PCComposite is good.
> But PCMG is not suitable because it can only implement the following form:
> 
>     (I-P) P_0 (I-P)

this should be
	
      I - (I-P) (I-P_0) (I-P)

since P is not a projection.  


From bsmith at mcs.anl.gov  Fri May 11 07:55:59 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 11 May 2012 07:55:59 -0500
Subject: [petsc-users] two-level Schwarz: PCMG or PCComposite
In-Reply-To: <BLU0-SMTP440ACD42EF2D3F61AC7C6F3CB170@phx.gbl>
References: <BLU0-SMTP440ACD42EF2D3F61AC7C6F3CB170@phx.gbl>
Message-ID: <45C92E5C-6AA5-470F-95CD-86BF20FDF57C@mcs.anl.gov>


  If this is what you want to do then just use the PCCOMPOSITE. Use PC_COMPOSITE_SYMMETRIC_MULTIPLICATIVE

   Barry

On May 11, 2012, at 6:26 AM, Hui Zhang wrote:

> I want to implement the two-level Schwarz method.  Let us denote P as the one-level
> Schwarz projection and P_0 as the coarse projection.  If I want the hybrid form of 
> preconditioned system as follows:
> 
>     (I-P_0) P (I-P_0) 
> 
> which is suitable:  PCMG or PCComposite?  It seems to me that PCComposite is good.
> But PCMG is not suitable because it can only implement the following form:
> 
>     (I-P) P_0 (I-P)
> 
> Any opinions?  Thanks!
> 
> 


From mike.hui.zhang at hotmail.com  Fri May 11 08:21:24 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Fri, 11 May 2012 15:21:24 +0200
Subject: [petsc-users] two-level Schwarz: PCMG or PCComposite
In-Reply-To: <45C92E5C-6AA5-470F-95CD-86BF20FDF57C@mcs.anl.gov>
References: <BLU0-SMTP440ACD42EF2D3F61AC7C6F3CB170@phx.gbl>
	<45C92E5C-6AA5-470F-95CD-86BF20FDF57C@mcs.anl.gov>
Message-ID: <BLU0-SMTP1760552E9171B2CB17CA103CB170@phx.gbl>


On May 11, 2012, at 2:55 PM, Barry Smith wrote:

> 
>  If this is what you want to do then just use the PCCOMPOSITE. Use PC_COMPOSITE_SYMMETRIC_MULTIPLICATIVE
> 
>   Barry

Aha, I see.  Thank you, Barry!  
Just one more question on PCCompositeAddPC:  is firstly added PC also firstly applied,
i.e. the order of applying PC's is the same as the order of PCCompositeAddPC ?

> 
> On May 11, 2012, at 6:26 AM, Hui Zhang wrote:
> 
>> I want to implement the two-level Schwarz method.  Let us denote P as the one-level
>> Schwarz projection and P_0 as the coarse projection.  If I want the hybrid form of 
>> preconditioned system as follows:
>> 
>>    (I-P_0) P (I-P_0) 
>> 
>> which is suitable:  PCMG or PCComposite?  It seems to me that PCComposite is good.
>> But PCMG is not suitable because it can only implement the following form:
>> 
>>    (I-P) P_0 (I-P)
>> 
>> Any opinions?  Thanks!
>> 
>> 
> 
> 


From zonexo at gmail.com  Fri May 11 08:24:11 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Fri, 11 May 2012 15:24:11 +0200
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
 ksp
In-Reply-To: <CA+zegpnTN40NUz+hD=BCqgv2kyz8sywzGDz+jDeVH3kXb+LSBw@mail.gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
	<4FA42CFC.8090002@gmail.com>
	<CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>
	<4FA44D5C.6010205@gmail.com>
	<CAMYG4Gm9cJxm1D0UmO9Fepv93uEHYhYDCcU1rKvsTU=D_+v9qQ@mail.gmail.com>
	<4FA97132.3080103@gmail.com>
	<CA+zegpnTN40NUz+hD=BCqgv2kyz8sywzGDz+jDeVH3kXb+LSBw@mail.gmail.com>
Message-ID: <4FAD12FB.6080506@gmail.com>

Hi,

I have been using the GUI environment to do debugging so I am a bit 
reluctant to learn Valgrind as its outputs seems a bit daunting.  But I 
guess John is right. I've been spending these few days learning bit by bit.

I realised that the error occurs in computerhs, at:

*call DMDAVecRestoreArrayF90(da,b,array,ierr)*

==27464== Invalid write of size 8
==27464==    at 0x402835: computerhs_ (ex29f.F90:119)
==27464==  Address 0xfffffffffffffef0 is not stack'd, malloc'd or 
(recently) free'd
==27464==
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames 
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] DM user function line 0 unknownunknown
[0]PETSC ERROR: [0] DMComputeFunction line 2085 
/home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
[0]PETSC ERROR: [0] KSPSetUp line 182 
/home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: --------------------- Error Message 
------------------------------------
[0]PETSC ERROR: Signal received!

I have checked that "array" 's values are correct. This statement 
executed without problems in VS2008. If I replace the above with 
something like:

*call VecSet(b,Hy,ierr)*

Everything is fine in Linux.

Is there something wrong with *DMDAVecRestoreArrayF90*?

My code in the area is:

call 
DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)

call DMDAVecGetArrayF90(da,b,array,ierr)

do j = ys,ys+ym-1

     do i = xs,xs+xm-1

         array(i,j) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy

     end do

end do

call DMDAVecRestoreArrayF90(da,b,array,ierr)

call VecAssemblyBegin(b,ierr)

call VecAssemblyEnd(b,ierr)


Yours sincerely,

TAY wee-beng


On 8/5/2012 9:41 PM, John Mousel wrote:
> TAY wee-bing,
>
> If you want to be a programmer that writes interesting and reliable 
> code, you need to be willing to use the tools of the trade. I can't 
> think of a bigger time-saver than Valgrind. I would suggest that you 
> learn to use it and use it a lot. I bet it will lead you to the root 
> of your problem pretty quickly.
>
> John
>
> On Tue, May 8, 2012 at 2:17 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I compiled and run my code under visual studio 2008 with intel
>     fortran. Everything works ok.
>
>     However, when I tried to run the code in linux, I got the error as
>     below. The error happens when KSPSetUp(ksp,ierr) is called.
>
>     However, I am not able to print VecView or MatView to view if
>     there's any errors. Is there any recommendation for debugging? I
>     hope I do not need to valgrind if possible.
>
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>     Violation, probably memory access out of range
>     [0]PETSC ERROR: Try option -start_in_debugger or
>     -on_error_attach_debugger
>     [0]PETSC ERROR: or see
>     http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC
>     ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
>     to find memory corruption errors
>     [0]PETSC ERROR: likely location of problem given in stack below
>     [0]PETSC ERROR: ---------------------  Stack Frames
>     ------------------------------------
>     [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>     available,
>     [0]PETSC ERROR:       INSTEAD the line number of the start of the
>     function
>     [0]PETSC ERROR:       is given.
>     [0]PETSC ERROR: [0] DM user function line 0 unknownunknown
>     [0]PETSC ERROR: [0] DMComputeFunction line 2085
>     /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
>     [0]PETSC ERROR: [0] KSPSetUp line 182
>     /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
>     [0]PETSC ERROR: --------------------- Error Message
>     ------------------------------------
>     [0]PETSC ERROR: Signal received!
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: Petsc Development HG revision:
>     7ecdd63ec420b1659b960e65d96e822c5ac1a968  HG Date: Mon May 07
>     21:42:26 2012 -0500
>     [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>     [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>     [0]PETSC ERROR: See docs/index.html for manual pages.
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: ./ex29f on a petsc-3.2 named hpc12 by wtay Tue
>     May  8 20:45:42 2012
>     [0]PETSC ERROR: Libraries linked from
>     /home/wtay/Lib/petsc-3.2-dev_shared_debug/lib
>     [0]PETSC ERROR: Configure run at Tue May  8 10:47:59 2012
>     [0]PETSC ERROR: Configure options
>     --with-mpi-dir=/opt/openmpi-1.5.3/
>     --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/
>     --with-debugging=1 --download-hypre=1
>     --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_debug
>     --known-mpi-shared=1 --with-shared-libraries
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: User provided function() line 0 in unknown
>     directory unknown file
>     --------------------------------------------------------------------------
>     MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>     with errorcode 59.
>
>     NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>     You may or may not see output from other processes, depending on
>     exactly when Open MPI kills them.
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>     On 5/5/2012 1:43 AM, Matthew Knepley wrote:
>>     On Fri, May 4, 2012 at 5:42 PM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         I wonder if you people are interested to include my ex29
>>         fortran version in the petsc examples, which can help people
>>         who are using fortran.
>>
>>
>>     Yes, we will definitely include it. Please send the source, and a
>>     representative output with run options.
>>
>>       Thanks,
>>
>>          Matt
>>
>>         Thanks.
>>
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>
>>         On 4/5/2012 9:28 PM, Matthew Knepley wrote:
>>>         On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng
>>>         <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>
>>>
>>>             On 4/5/2012 9:16 PM, Barry Smith wrote:
>>>
>>>                    Do an hg pull and then run make in
>>>                 src/mat/interface/ftn-custom/
>>>
>>>                     Then it should link.
>>>
>>>                    Barry
>>>
>>>                    There was a E missing from the all caps name of
>>>                 the function.
>>>
>>>             After hg pull, I did:
>>>
>>>             cd src//mat/interface/ftn-custom/
>>>
>>>             User at User-PC
>>>             /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom
>>>             $ make
>>>             make[1]: Warning: File
>>>             `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)'
>>>             has modification time 787 s in the future
>>>             make[1]: Nothing to be done for `libc'.
>>>             make[1]: warning:  Clock skew detected.  Your build may
>>>             be incomplete.
>>>
>>>             But it still can't work.
>>>
>>>
>>>         Something is messed up with the clock on this machine.
>>>
>>>         HOWEVER, development requires certain basic skills in order
>>>         to debug your work. We
>>>         cannot be the ones debugging your code. Now
>>>
>>>           nm $PETSC_ARCH/lib/libpetsc.a | grep -i MatNullSpaceRemove
>>>
>>>          will look for the symbol.
>>>
>>>             Matt
>>>
>>>
>>>
>>>                 On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:
>>>
>>>                     On Fri, May 4, 2012 at 3:01 PM, TAY
>>>                     wee-beng<zonexo at gmail.com
>>>                     <mailto:zonexo at gmail.com>>  wrote:
>>>
>>>                     On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>>>
>>>                         On Fri, May 4, 2012 at 11:05 AM, TAY
>>>                         wee-beng<zonexo at gmail.com
>>>                         <mailto:zonexo at gmail.com>>  wrote:
>>>
>>>                         On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>>
>>>                             On Fri, May 4, 2012 at 8:59 AM, TAY
>>>                             wee-beng<zonexo at gmail.com
>>>                             <mailto:zonexo at gmail.com>>  wrote:
>>>
>>>                             Hi,
>>>
>>>                             Is there anything else I can try to get
>>>                             it working right?
>>>
>>>                             The MatGetNullSpaceRemove() is missing.
>>>
>>>                         Where should I add MatGetNullSpaceRemove and
>>>                         what are its syntax? I googled but there's
>>>                         no results.
>>>
>>>                         Fixed in p;etsc-dev:
>>>
>>>                         http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>>>
>>>                     I just compiled the updated petsc-dev but I got
>>>                     the same error msg when I use:
>>>
>>>                     call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>
>>>                     error LNK2019: unresolved external symbol
>>>                     MATNULLSPACEREMOVE referenced in function COMPUTERHS
>>>                     1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error
>>>                     LNK1120: 1 unresolved externals
>>>
>>>                     That function is in:
>>>
>>>                       src/mat/interface/ftn-custom/zmatrixf.c
>>>
>>>                     Did that compile? Can you see the symbol in
>>>                     libpetsc.a?
>>>
>>>                         Matt
>>>
>>>                              Matt
>>>
>>>                         Thanks.
>>>
>>>                                Matt
>>>
>>>                             Thanks!
>>>
>>>
>>>                             On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>
>>>                                 On Wed, May 2, 2012 at 1:55 PM, TAY
>>>                                 wee-beng<zonexo at gmail.com
>>>                                 <mailto:zonexo at gmail.com>>  wrote:
>>>                                 Hi,
>>>
>>>                                 I did a MatView and VecView on both
>>>                                 C and Fortran, right after Mat and
>>>                                 Vec assembly. I have attached the
>>>                                 printout below. They are exactly the
>>>                                 same, but yet the result is
>>>                                 different in Neumann condition.
>>>                                 However, the dirichlet condition
>>>                                 gives the correct ans. Is there
>>>                                 anything else that could be wrong
>>>                                 even if the Mat and Vec are the same?
>>>
>>>                                 Did you set the null space for the
>>>                                 matrix when you have Neumann conditions?
>>>
>>>                             Yes, for the matrix, I set as:
>>>
>>>                             call
>>>                             MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>
>>>                                 call MatSetNullSpace(jac,nullspace,ierr)
>>>
>>>                                 call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>>                             for the Vec,
>>>
>>>                             call
>>>                             MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>
>>>                                 !call
>>>                             MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>
>>>                                 call MatNullSpaceDestroy(nullspace,ierr)
>>>
>>>                             MatNullSpaceRemove was comment out
>>>                             because there's error during linking
>>>
>>>
>>>                                    Matt
>>>
>>>                                 Thanks!
>>>
>>>                                 Fortran:
>>>
>>>                                 Matrix Object: 1 MPI processes
>>>                                   type: seqaij
>>>                                 row 0: (0, 2)  (1, -1)  (3, -1)
>>>                                 row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>                                 row 2: (1, -1)  (2, 2)  (5, -1)
>>>                                 row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>                                 row 4: (1, -1)  (3, -1)  (4, 4)  (5,
>>>                                 -1)  (7, -1)
>>>                                 row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>                                 row 6: (3, -1)  (6, 2)  (7, -1)
>>>                                 row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>                                 row 8: (5, -1)  (7, -1)  (8, 2)
>>>                                 Vector Object:Vec_0000000084000000_0
>>>                                 1 MPI processes
>>>                                   type: mpi
>>>                                 Process [0]
>>>                                 0.25
>>>                                 0.0205213
>>>                                 1.135e-005
>>>                                 0.0205213
>>>                                 0.00168449
>>>                                 9.31663e-007
>>>                                 1.135e-005
>>>                                 9.31663e-007
>>>                                 5.15289e-010
>>>                                 Vector Object:Vec_0000000084000000_1
>>>                                 1 MPI processes
>>>                                   type: mpi
>>>                                 Process [0]
>>>                                 0.14924
>>>                                 0.0242397
>>>                                 -0.0260347
>>>                                 0.0242397
>>>                                 -0.0256192
>>>                                 -0.0400102
>>>                                 -0.0260347
>>>                                 -0.0400102
>>>                                 -0.0400102
>>>                                 Press any key to continue . . .
>>>
>>>                                 C:
>>>
>>>                                 Matrix Object: 1 MPI processes
>>>                                   type: seqaij
>>>                                 row 0: (0, 2)  (1, -1)  (3, -1)
>>>                                 row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>                                 row 2: (1, -1)  (2, 2)  (5, -1)
>>>                                 row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>                                 row 4: (1, -1)  (3, -1)  (4, 4)  (5,
>>>                                 -1)  (7, -1)
>>>                                 row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>                                 row 6: (3, -1)  (6, 2)  (7, -1)
>>>                                 row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>                                 row 8: (5, -1)  (7, -1)  (8, 2)
>>>                                 Vector Object:Vec_0x1d3b000_0 1 MPI
>>>                                 processes
>>>                                   type: mpi
>>>                                 Process [0]
>>>                                 0.25
>>>                                 0.0205212
>>>                                 1.135e-05
>>>                                 0.0205212
>>>                                 0.00168449
>>>                                 9.31663e-07
>>>                                 1.135e-05
>>>                                 9.31663e-07
>>>                                 5.15288e-10
>>>                                 Vector Object:Vec_0x1d3b000_1 1 MPI
>>>                                 processes
>>>                                   type: mpi
>>>                                 Process [0]
>>>                                 0.139311
>>>                                 0.0305751
>>>                                 -0.0220633
>>>                                 0.0305751
>>>                                 -0.0135158
>>>                                 -0.042185
>>>                                 -0.0220633
>>>                                 -0.042185
>>>                                 -0.058449
>>>
>>>
>>>
>>>                                 Yours sincerely,
>>>
>>>                                 TAY wee-beng
>>>
>>>
>>>                                 On 1/5/2012 11:54 PM, Matthew
>>>                                 Knepley wrote:
>>>
>>>                                     On Tue, May 1, 2012 at 5:48 PM,
>>>                                     TAY wee-beng<zonexo at gmail.com
>>>                                     <mailto:zonexo at gmail.com>>  wrote:
>>>                                     Hi,
>>>
>>>                                     Do you mean my method is wrong?
>>>
>>>                                     I am following the template of
>>>                                     ex22f,
>>>
>>>                                     where the variables are declared
>>>                                     as :
>>>
>>>                                     PetscScalar  v(5)
>>>
>>>                                     MatStencil   row(4),col(4,5)
>>>
>>>                                     Hence,
>>>
>>>                                     for the neumann BC
>>>
>>>                                     num = 1
>>>
>>>                                                    if (j/=0) then
>>>
>>>                                                        v(num) =
>>>                                     -rho*HxdHy
>>>
>>>                                                      
>>>                                      col(MatStencil_i,num) = i
>>>
>>>                                                      
>>>                                      col(MatStencil_j,num) = j-1
>>>
>>>                                                        num = num + 1
>>>
>>>                                                    end if
>>>
>>>                                                    if (i/=0) then
>>>
>>>                                                        v(num) =
>>>                                     -rho*HydHx
>>>
>>>                                                      
>>>                                      col(MatStencil_i,num) = i-1
>>>
>>>                                                      
>>>                                      col(MatStencil_j,num) = j
>>>
>>>                                                        num = num + 1
>>>
>>>                                                    end if
>>>
>>>                                                    if (i/=mx-1) then
>>>
>>>                                                        v(num) =
>>>                                     -rho*HydHx
>>>
>>>                                                      
>>>                                      col(MatStencil_i,num) = i+1
>>>
>>>                                                      
>>>                                      col(MatStencil_j,num) = j
>>>
>>>                                                        num = num + 1
>>>
>>>                                                    end if
>>>
>>>                                                    if (j/=my-1) then
>>>
>>>                                                        v(num) =
>>>                                     -rho*HxdHy
>>>
>>>                                                      
>>>                                      col(MatStencil_i,num) = i
>>>
>>>                                                      
>>>                                      col(MatStencil_j,num) = j+1
>>>
>>>                                                        num = num + 1
>>>
>>>                                                    end if
>>>
>>>                                                    v(num) =
>>>                                     ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>
>>>                                                    print *, v
>>>
>>>                                                  
>>>                                      col(MatStencil_i,num) = i
>>>
>>>                                                  
>>>                                      col(MatStencil_j,num) = j
>>>
>>>                                                    !num = num + 1
>>>
>>>                                                    call
>>>                                     MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>
>>>                                     I do not get any more out of
>>>                                     range error. However,my ans is
>>>                                     still different from that of
>>>                                     ex29 in C.
>>>
>>>                                     This is very simple. You have an
>>>                                     error in your code. Checking it
>>>                                     is very simple: run the code and
>>>                                     break in MatSetValues(). Make
>>>                                     sure ex29 makes calls with
>>>                                     exactly the same indices as your
>>>                                     ex29f.
>>>
>>>                                         Matt
>>>
>>>                                     Yours sincerely,
>>>
>>>                                     TAY wee-beng
>>>
>>>                                     -- 
>>>                                     What most experimenters take for
>>>                                     granted before they begin their
>>>                                     experiments is infinitely more
>>>                                     interesting than any results to
>>>                                     which their experiments lead.
>>>                                     -- Norbert Wiener
>>>
>>>
>>>
>>>                                 -- 
>>>                                 What most experimenters take for
>>>                                 granted before they begin their
>>>                                 experiments is infinitely more
>>>                                 interesting than any results to
>>>                                 which their experiments lead.
>>>                                 -- Norbert Wiener
>>>
>>>
>>>
>>>                             -- 
>>>                             What most experimenters take for granted
>>>                             before they begin their experiments is
>>>                             infinitely more interesting than any
>>>                             results to which their experiments lead.
>>>                             -- Norbert Wiener
>>>
>>>
>>>
>>>                         -- 
>>>                         What most experimenters take for granted
>>>                         before they begin their experiments is
>>>                         infinitely more interesting than any results
>>>                         to which their experiments lead.
>>>                         -- Norbert Wiener
>>>
>>>
>>>
>>>                     -- 
>>>                     What most experimenters take for granted before
>>>                     they begin their experiments is infinitely more
>>>                     interesting than any results to which their
>>>                     experiments lead.
>>>                     -- Norbert Wiener
>>>
>>>
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120511/dbf50541/attachment-0001.htm>

From knepley at gmail.com  Fri May 11 08:30:23 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 May 2012 09:30:23 -0400
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
	ksp
In-Reply-To: <4FAD12FB.6080506@gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
	<4FA42CFC.8090002@gmail.com>
	<CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>
	<4FA44D5C.6010205@gmail.com>
	<CAMYG4Gm9cJxm1D0UmO9Fepv93uEHYhYDCcU1rKvsTU=D_+v9qQ@mail.gmail.com>
	<4FA97132.3080103@gmail.com>
	<CA+zegpnTN40NUz+hD=BCqgv2kyz8sywzGDz+jDeVH3kXb+LSBw@mail.gmail.com>
	<4FAD12FB.6080506@gmail.com>
Message-ID: <CAMYG4GmW=A9UKr0qQT7t8UTsns_ORr_UB97wUZsJtx87fibOTQ@mail.gmail.com>

On Fri, May 11, 2012 at 9:24 AM, TAY wee-beng <zonexo at gmail.com> wrote:

>  Hi,
>
> I have been using the GUI environment to do debugging so I am a bit
> reluctant to learn Valgrind as its outputs seems a bit daunting.  But I
> guess John is right. I've been spending these few days learning bit by bit.
>
> I realised that the error occurs in computerhs, at:
>

I bet this is a beautiful Fortranism. Do you include the F90 header file
with the interface definition?
If not, Fortran just craps out like this. I can't stress enough how much
time would be saved by
switching languages to something with at least a modicum of error checking.

   Matt


> *call DMDAVecRestoreArrayF90(da,b,array,ierr)*
>
> ==27464== Invalid write of size 8
> ==27464==    at 0x402835: computerhs_ (ex29f.F90:119)
> ==27464==  Address 0xfffffffffffffef0 is not stack'd, malloc'd or
> (recently) free'd
> ==27464==
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC
> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] DM user function line 0 unknownunknown
> [0]PETSC ERROR: [0] DMComputeFunction line 2085
> /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
> [0]PETSC ERROR: [0] KSPSetUp line 182
> /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Signal received!
>
> I have checked that "array" 's values are correct. This statement executed
> without problems in VS2008. If I replace the above with something like:
>
> *call VecSet(b,Hy,ierr)*
>
> Everything is fine in Linux.
>
> Is there something wrong with *DMDAVecRestoreArrayF90*?
>
> My code in the area is:
>
> call
> DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>
> call DMDAVecGetArrayF90(da,b,array,ierr)
>
> do j = ys,ys+ym-1
>
>     do i = xs,xs+xm-1
>
>         array(i,j) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>
>     end do
>
> end do
>
> call DMDAVecRestoreArrayF90(da,b,array,ierr)
>
> call VecAssemblyBegin(b,ierr)
>
> call VecAssemblyEnd(b,ierr)
>
>
> Yours sincerely,
>
> TAY wee-beng
>
>
> On 8/5/2012 9:41 PM, John Mousel wrote:
>
> TAY wee-bing,
>
> If you want to be a programmer that writes interesting and reliable code,
> you need to be willing to use the tools of the trade. I can't think of a
> bigger time-saver than Valgrind. I would suggest that you learn to use it
> and use it a lot. I bet it will lead you to the root of your problem pretty
> quickly.
>
> John
>
> On Tue, May 8, 2012 at 2:17 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>  Hi,
>>
>> I compiled and run my code under visual studio 2008 with intel fortran.
>> Everything works ok.
>>
>> However, when I tried to run the code in linux, I got the error as below.
>> The error happens when KSPSetUp(ksp,ierr) is called.
>>
>> However, I am not able to print VecView or MatView to view if there's any
>> errors. Is there any recommendation for debugging? I hope I do not need to
>> valgrind if possible.
>>
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC
>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
>> find memory corruption errors
>> [0]PETSC ERROR: likely location of problem given in stack below
>> [0]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [0]PETSC ERROR:       is given.
>> [0]PETSC ERROR: [0] DM user function line 0 unknownunknown
>> [0]PETSC ERROR: [0] DMComputeFunction line 2085
>> /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
>> [0]PETSC ERROR: [0] KSPSetUp line 182
>> /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Signal received!
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Development HG revision:
>> 7ecdd63ec420b1659b960e65d96e822c5ac1a968  HG Date: Mon May 07 21:42:26 2012
>> -0500
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: ./ex29f on a petsc-3.2 named hpc12 by wtay Tue May  8
>> 20:45:42 2012
>> [0]PETSC ERROR: Libraries linked from
>> /home/wtay/Lib/petsc-3.2-dev_shared_debug/lib
>> [0]PETSC ERROR: Configure run at Tue May  8 10:47:59 2012
>> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/
>> --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/
>> --with-debugging=1 --download-hypre=1
>> --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_debug --known-mpi-shared=1
>> --with-shared-libraries
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: User provided function() line 0 in unknown directory
>> unknown file
>> --------------------------------------------------------------------------
>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>> with errorcode 59.
>>
>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>> You may or may not see output from other processes, depending on
>> exactly when Open MPI kills them.
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>> On 5/5/2012 1:43 AM, Matthew Knepley wrote:
>>
>> On Fri, May 4, 2012 at 5:42 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>>  Hi,
>>>
>>> I wonder if you people are interested to include my ex29 fortran version
>>> in the petsc examples, which can help people who are using fortran.
>>>
>>
>>  Yes, we will definitely include it. Please send the source, and a
>> representative output with run options.
>>
>>    Thanks,
>>
>>       Matt
>>
>>
>>>  Thanks.
>>>
>>> Yours sincerely,
>>>
>>> TAY wee-beng
>>>
>>>
>>> On 4/5/2012 9:28 PM, Matthew Knepley wrote:
>>>
>>> On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>
>>>>
>>>> On 4/5/2012 9:16 PM, Barry Smith wrote:
>>>>
>>>>>    Do an hg pull and then run make in src/mat/interface/ftn-custom/
>>>>>
>>>>>     Then it should link.
>>>>>
>>>>>    Barry
>>>>>
>>>>>    There was a E missing from the all caps name of the function.
>>>>>
>>>> After hg pull, I did:
>>>>
>>>> cd src//mat/interface/ftn-custom/
>>>>
>>>> User at User-PC /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom
>>>> $ make
>>>> make[1]: Warning: File
>>>> `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)'
>>>> has modification time 787 s in the future
>>>> make[1]: Nothing to be done for `libc'.
>>>> make[1]: warning:  Clock skew detected.  Your build may be incomplete.
>>>>
>>>> But it still can't work.
>>>>
>>>
>>>  Something is messed up with the clock on this machine.
>>>
>>>  HOWEVER, development requires certain basic skills in order to debug
>>> your work. We
>>> cannot be the ones debugging your code. Now
>>>
>>>    nm $PETSC_ARCH/lib/libpetsc.a | grep -i MatNullSpaceRemove
>>>
>>>   will look for the symbol.
>>>
>>>      Matt
>>>
>>>
>>>>
>>>>>
>>>>> On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:
>>>>>
>>>>>  On Fri, May 4, 2012 at 3:01 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>  wrote:
>>>>>>
>>>>>> On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>>>>>>
>>>>>>> On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>  wrote:
>>>>>>>
>>>>>>> On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>>>>>>
>>>>>>>> On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>  wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Is there anything else I can try to get it working right?
>>>>>>>>
>>>>>>>> The MatGetNullSpaceRemove() is missing.
>>>>>>>>
>>>>>>> Where should I add MatGetNullSpaceRemove and what are its syntax? I
>>>>>>> googled but there's no results.
>>>>>>>
>>>>>>> Fixed in p;etsc-dev:
>>>>>>>
>>>>>>>
>>>>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>>>>>>>
>>>>>> I just compiled the updated petsc-dev but I got the same error msg
>>>>>> when I use:
>>>>>>
>>>>>> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>>>
>>>>>> error LNK2019: unresolved external symbol MATNULLSPACEREMOVE
>>>>>> referenced in function COMPUTERHS
>>>>>> 1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120: 1
>>>>>> unresolved externals
>>>>>>
>>>>>> That function is in:
>>>>>>
>>>>>>   src/mat/interface/ftn-custom/zmatrixf.c
>>>>>>
>>>>>> Did that compile? Can you see the symbol in libpetsc.a?
>>>>>>
>>>>>>     Matt
>>>>>>
>>>>>>       Matt
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>>    Matt
>>>>>>>>
>>>>>>>> Thanks?
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>>>>>>
>>>>>>>>> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>>  wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I did a MatView and VecView on both C and Fortran, right after Mat
>>>>>>>>> and Vec assembly. I have attached the printout below. They are exactly the
>>>>>>>>> same, but yet the result is different in Neumann condition. However, the
>>>>>>>>> dirichlet condition gives the correct ans. Is there anything else that
>>>>>>>>> could be wrong even if the Mat and Vec are the same?
>>>>>>>>>
>>>>>>>>> Did you set the null space for the matrix when you have Neumann
>>>>>>>>> conditions?
>>>>>>>>>
>>>>>>>> Yes, for the matrix, I set as:
>>>>>>>>
>>>>>>>> call
>>>>>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>>>>>
>>>>>>>>     call MatSetNullSpace(jac,nullspace,ierr)
>>>>>>>>
>>>>>>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>>>
>>>>>>>> for the Vec,
>>>>>>>>
>>>>>>>> call
>>>>>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>>>>>
>>>>>>>>     !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>>>>>
>>>>>>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>>>
>>>>>>>> MatNullSpaceRemove was comment out because there's error during
>>>>>>>> linking
>>>>>>>>
>>>>>>>>
>>>>>>>>     Matt
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> Fortran:
>>>>>>>>>
>>>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>>>   type: seqaij
>>>>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>>>>>>   type: mpi
>>>>>>>>> Process [0]
>>>>>>>>> 0.25
>>>>>>>>> 0.0205213
>>>>>>>>> 1.135e-005
>>>>>>>>> 0.0205213
>>>>>>>>> 0.00168449
>>>>>>>>> 9.31663e-007
>>>>>>>>> 1.135e-005
>>>>>>>>> 9.31663e-007
>>>>>>>>> 5.15289e-010
>>>>>>>>> Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>>>>>>>   type: mpi
>>>>>>>>> Process [0]
>>>>>>>>> 0.14924
>>>>>>>>> 0.0242397
>>>>>>>>> -0.0260347
>>>>>>>>> 0.0242397
>>>>>>>>> -0.0256192
>>>>>>>>> -0.0400102
>>>>>>>>> -0.0260347
>>>>>>>>> -0.0400102
>>>>>>>>> -0.0400102
>>>>>>>>> Press any key to continue . . .
>>>>>>>>>
>>>>>>>>> C:
>>>>>>>>>
>>>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>>>   type: seqaij
>>>>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>>>>> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>>>>>>>   type: mpi
>>>>>>>>> Process [0]
>>>>>>>>> 0.25
>>>>>>>>> 0.0205212
>>>>>>>>> 1.135e-05
>>>>>>>>> 0.0205212
>>>>>>>>> 0.00168449
>>>>>>>>> 9.31663e-07
>>>>>>>>> 1.135e-05
>>>>>>>>> 9.31663e-07
>>>>>>>>> 5.15288e-10
>>>>>>>>> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>>>>>>>   type: mpi
>>>>>>>>> Process [0]
>>>>>>>>> 0.139311
>>>>>>>>> 0.0305751
>>>>>>>>> -0.0220633
>>>>>>>>> 0.0305751
>>>>>>>>> -0.0135158
>>>>>>>>> -0.042185
>>>>>>>>> -0.0220633
>>>>>>>>> -0.042185
>>>>>>>>> -0.058449
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yours sincerely,
>>>>>>>>>
>>>>>>>>> TAY wee-beng
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>>>>>>
>>>>>>>>>> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>>>  wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Do you mean my method is wrong?
>>>>>>>>>>
>>>>>>>>>> I am following the template of ex22f,
>>>>>>>>>>
>>>>>>>>>> where the variables are declared as :
>>>>>>>>>>
>>>>>>>>>> PetscScalar  v(5)
>>>>>>>>>>
>>>>>>>>>> MatStencil   row(4),col(4,5)
>>>>>>>>>>
>>>>>>>>>> Hence,
>>>>>>>>>>
>>>>>>>>>> for the neumann BC
>>>>>>>>>>
>>>>>>>>>> num = 1
>>>>>>>>>>
>>>>>>>>>>                if (j/=0) then
>>>>>>>>>>
>>>>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>>>>>
>>>>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>>>>>
>>>>>>>>>>                    col(MatStencil_j,num) = j-1
>>>>>>>>>>
>>>>>>>>>>                    num = num + 1
>>>>>>>>>>
>>>>>>>>>>                end if
>>>>>>>>>>
>>>>>>>>>>                if (i/=0) then
>>>>>>>>>>
>>>>>>>>>>                    v(num) = -rho*HydHx
>>>>>>>>>>
>>>>>>>>>>                    col(MatStencil_i,num) = i-1
>>>>>>>>>>
>>>>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>>>>>
>>>>>>>>>>                    num = num + 1
>>>>>>>>>>
>>>>>>>>>>                end if
>>>>>>>>>>
>>>>>>>>>>                if (i/=mx-1) then
>>>>>>>>>>
>>>>>>>>>>                    v(num) = -rho*HydHx
>>>>>>>>>>
>>>>>>>>>>                    col(MatStencil_i,num) = i+1
>>>>>>>>>>
>>>>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>>>>>
>>>>>>>>>>                    num = num + 1
>>>>>>>>>>
>>>>>>>>>>                end if
>>>>>>>>>>
>>>>>>>>>>                if (j/=my-1) then
>>>>>>>>>>
>>>>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>>>>>
>>>>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>>>>>
>>>>>>>>>>                    col(MatStencil_j,num) = j+1
>>>>>>>>>>
>>>>>>>>>>                    num = num + 1
>>>>>>>>>>
>>>>>>>>>>                end if
>>>>>>>>>>
>>>>>>>>>>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>>>>>>>
>>>>>>>>>>                print *, v
>>>>>>>>>>
>>>>>>>>>>                col(MatStencil_i,num) = i
>>>>>>>>>>
>>>>>>>>>>                col(MatStencil_j,num) = j
>>>>>>>>>>
>>>>>>>>>>                !num = num + 1
>>>>>>>>>>
>>>>>>>>>>                call
>>>>>>>>>> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>>>>>>>
>>>>>>>>>> I do not get any more out of range error. However,my ans is still
>>>>>>>>>> different from that of ex29 in C.
>>>>>>>>>>
>>>>>>>>>> This is very simple. You have an error in your code. Checking it
>>>>>>>>>> is very simple: run the code and
>>>>>>>>>> break in MatSetValues(). Make sure ex29 makes calls with exactly
>>>>>>>>>> the same indices as your ex29f.
>>>>>>>>>>
>>>>>>>>>>     Matt
>>>>>>>>>>
>>>>>>>>>> Yours sincerely,
>>>>>>>>>>
>>>>>>>>>> TAY wee-beng
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>
>>>
>>>  --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>
>>
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120511/b021ea4a/attachment-0001.htm>

From zonexo at gmail.com  Fri May 11 08:51:23 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Fri, 11 May 2012 15:51:23 +0200
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
 ksp
In-Reply-To: <CAMYG4GmW=A9UKr0qQT7t8UTsns_ORr_UB97wUZsJtx87fibOTQ@mail.gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
	<4FA42CFC.8090002@gmail.com>
	<CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>
	<4FA44D5C.6010205@gmail.com>
	<CAMYG4Gm9cJxm1D0UmO9Fepv93uEHYhYDCcU1rKvsTU=D_+v9qQ@mail.gmail.com>
	<4FA97132.3080103@gmail.com>
	<CA+zegpnTN40NUz+hD=BCqgv2kyz8sywzGDz+jDeVH3kXb+LSBw@mail.gmail.com>
	<4FAD12FB.6080506@gmail.com>
	<CAMYG4GmW=A9UKr0qQT7t8UTsns_ORr_UB97wUZsJtx87fibOTQ@mail.gmail.com>
Message-ID: <4FAD195B.4030302@gmail.com>



On 11/5/2012 3:30 PM, Matthew Knepley wrote:
> On Fri, May 11, 2012 at 9:24 AM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I have been using the GUI environment to do debugging so I am a
>     bit reluctant to learn Valgrind as its outputs seems a bit
>     daunting.  But I guess John is right. I've been spending these few
>     days learning bit by bit.
>
>     I realised that the error occurs in computerhs, at:
>
>
> I bet this is a beautiful Fortranism. Do you include the F90 header 
> file with the interface definition?
> If not, Fortran just craps out like this. I can't stress enough how 
> much time would be saved by
> switching languages to something with at least a modicum of error 
> checking.

I initially used:

#include "finclude/petsc.h90"

Compilation and linking was fine in Linux and vs2008.

Now I changed it to what ex22.F was using :

#include <finclude/petscsys.h>
#include <finclude/petscvec.h>
#include <finclude/petscmat.h>
#include <finclude/petscpc.h>
#include <finclude/petscksp.h>
#include <finclude/petscdmda.h>

Compiling was ok but linking failed in Linux and VS2008:

undefined reference to `dmdavecgetarrayf90_'

I tried changing #include <finclude/petscdmda.h> to #include 
<finclude/petscdmda.h90> and everything was ok in VS2008 again, giving 
the right answers.

However, in Linux, I got the following error:

[wtay at hpc12:tutorials]$ /opt/openmpi-1.5.3/bin/mpif90 -c  -fPIC -g   
-I/home/wtay/Lib/petsc-3.2-dev_shared_debug/include 
-I/home/wtay/Lib/petsc-3.2-dev_shared_debug/include 
-I/opt/openmpi-1.5.3/include    -o ex29f.o ex29f.F90
/home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): 
error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
         DMDABoundaryType :: pt
-------------------------^
/home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): 
error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
         DMDAStencilType  :: st
-------------------------^
/home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): 
error #6590: This statement is not permitted as a statement within a 
derived-type-def
         DMDABoundaryType :: pt
--------^
/home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): 
error #6590: This statement is not permitted as a statement within a 
derived-type-def
         DMDAStencilType  :: st
--------^
ex29f.F90(68): error #6404: This name does not have a type, and must 
have an explicit type.   [DMDA_BOUNDARY_NONE]
call 
DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
-----------------------------------^
ex29f.F90(68): error #6404: This name does not have a type, and must 
have an explicit type.   [DMDA_STENCIL_STAR]
call 
DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
-------------------------------------------------------------------------^
/home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): 
error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
         DMDABoundaryType :: pt
-------------------------^
/home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): 
error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
         DMDAStencilType  :: st
-------------------------^
/home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): 
error #6590: This statement is not permitted as a statement within a 
derived-type-def
         DMDABoundaryType :: pt
--------^
/home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): 
error #6590: This statement is not permitted as a statement within a 
derived-type-def
         DMDAStencilType  :: st
--------^
/home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): 
error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
         DMDABoundaryType :: pt
-------------------------^
/home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): 
error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
         DMDAStencilType  :: st
-------------------------^
/home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25): 
error #6590: This statement is not permitted as a statement within a 
derived-type-def
         DMDABoundaryType :: pt
--------^
/home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26): 
error #6590: This statement is not permitted as a statement within a 
derived-type-def
         DMDAStencilType  :: st

Is there some errors in petscdmda.h90?


>
>    Matt
>
>     *call DMDAVecRestoreArrayF90(da,b,array,ierr)*
>
>     ==27464== Invalid write of size 8
>     ==27464==    at 0x402835: computerhs_ (ex29f.F90:119)
>     ==27464==  Address 0xfffffffffffffef0 is not stack'd, malloc'd or
>     (recently) free'd
>     ==27464==
>     [0]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>     Violation, probably memory access out of range
>     [0]PETSC ERROR: Try option -start_in_debugger or
>     -on_error_attach_debugger
>     [0]PETSC ERROR: or see
>     http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC
>     ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
>     to find memory corruption errors
>     [0]PETSC ERROR: likely location of problem given in stack below
>     [0]PETSC ERROR: ---------------------  Stack Frames
>     ------------------------------------
>     [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>     available,
>     [0]PETSC ERROR:       INSTEAD the line number of the start of the
>     function
>     [0]PETSC ERROR:       is given.
>     [0]PETSC ERROR: [0] DM user function line 0 unknownunknown
>     [0]PETSC ERROR: [0] DMComputeFunction line 2085
>     /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
>     [0]PETSC ERROR: [0] KSPSetUp line 182
>     /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
>     [0]PETSC ERROR: --------------------- Error Message
>     ------------------------------------
>     [0]PETSC ERROR: Signal received!
>
>     I have checked that "array" 's values are correct. This statement
>     executed without problems in VS2008. If I replace the above with
>     something like:
>
>     *call VecSet(b,Hy,ierr)*
>
>     Everything is fine in Linux.
>
>     Is there something wrong with *DMDAVecRestoreArrayF90*?
>
>     My code in the area is:
>
>     call
>     DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>
>     call DMDAVecGetArrayF90(da,b,array,ierr)
>
>     do j = ys,ys+ym-1
>
>         do i = xs,xs+xm-1
>
>             array(i,j) =
>     exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>
>         end do
>
>     end do
>
>     call DMDAVecRestoreArrayF90(da,b,array,ierr)
>
>     call VecAssemblyBegin(b,ierr)
>
>     call VecAssemblyEnd(b,ierr)
>
>
>     Yours sincerely,
>
>     TAY wee-beng
>
>
>     On 8/5/2012 9:41 PM, John Mousel wrote:
>>     TAY wee-bing,
>>
>>     If you want to be a programmer that writes interesting and
>>     reliable code, you need to be willing to use the tools of the
>>     trade. I can't think of a bigger time-saver than Valgrind. I
>>     would suggest that you learn to use it and use it a lot. I bet it
>>     will lead you to the root of your problem pretty quickly.
>>
>>     John
>>
>>     On Tue, May 8, 2012 at 2:17 PM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         I compiled and run my code under visual studio 2008 with
>>         intel fortran. Everything works ok.
>>
>>         However, when I tried to run the code in linux, I got the
>>         error as below. The error happens when KSPSetUp(ksp,ierr) is
>>         called.
>>
>>         However, I am not able to print VecView or MatView to view if
>>         there's any errors. Is there any recommendation for
>>         debugging? I hope I do not need to valgrind if possible.
>>
>>         [0]PETSC ERROR:
>>         ------------------------------------------------------------------------
>>         [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>         Violation, probably memory access out of range
>>         [0]PETSC ERROR: Try option -start_in_debugger or
>>         -on_error_attach_debugger
>>         [0]PETSC ERROR: or see
>>         http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC
>>         ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>         OS X to find memory corruption errors
>>         [0]PETSC ERROR: likely location of problem given in stack below
>>         [0]PETSC ERROR: ---------------------  Stack Frames
>>         ------------------------------------
>>         [0]PETSC ERROR: Note: The EXACT line numbers in the stack are
>>         not available,
>>         [0]PETSC ERROR:       INSTEAD the line number of the start of
>>         the function
>>         [0]PETSC ERROR:       is given.
>>         [0]PETSC ERROR: [0] DM user function line 0 unknownunknown
>>         [0]PETSC ERROR: [0] DMComputeFunction line 2085
>>         /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
>>         [0]PETSC ERROR: [0] KSPSetUp line 182
>>         /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ------------------------------------
>>         [0]PETSC ERROR: Signal received!
>>         [0]PETSC ERROR:
>>         ------------------------------------------------------------------------
>>         [0]PETSC ERROR: Petsc Development HG revision:
>>         7ecdd63ec420b1659b960e65d96e822c5ac1a968  HG Date: Mon May 07
>>         21:42:26 2012 -0500
>>         [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>         [0]PETSC ERROR: See docs/faq.html for hints about trouble
>>         shooting.
>>         [0]PETSC ERROR: See docs/index.html for manual pages.
>>         [0]PETSC ERROR:
>>         ------------------------------------------------------------------------
>>         [0]PETSC ERROR: ./ex29f on a petsc-3.2 named hpc12 by wtay
>>         Tue May  8 20:45:42 2012
>>         [0]PETSC ERROR: Libraries linked from
>>         /home/wtay/Lib/petsc-3.2-dev_shared_debug/lib
>>         [0]PETSC ERROR: Configure run at Tue May  8 10:47:59 2012
>>         [0]PETSC ERROR: Configure options
>>         --with-mpi-dir=/opt/openmpi-1.5.3/
>>         --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/
>>         --with-debugging=1 --download-hypre=1
>>         --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_debug
>>         --known-mpi-shared=1 --with-shared-libraries
>>         [0]PETSC ERROR:
>>         ------------------------------------------------------------------------
>>         [0]PETSC ERROR: User provided function() line 0 in unknown
>>         directory unknown file
>>         --------------------------------------------------------------------------
>>         MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>>         with errorcode 59.
>>
>>         NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>         processes.
>>         You may or may not see output from other processes, depending on
>>         exactly when Open MPI kills them.
>>
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>
>>         On 5/5/2012 1:43 AM, Matthew Knepley wrote:
>>>         On Fri, May 4, 2012 at 5:42 PM, TAY wee-beng
>>>         <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>
>>>             Hi,
>>>
>>>             I wonder if you people are interested to include my ex29
>>>             fortran version in the petsc examples, which can help
>>>             people who are using fortran.
>>>
>>>
>>>         Yes, we will definitely include it. Please send the source,
>>>         and a representative output with run options.
>>>
>>>           Thanks,
>>>
>>>              Matt
>>>
>>>             Thanks.
>>>
>>>             Yours sincerely,
>>>
>>>             TAY wee-beng
>>>
>>>
>>>             On 4/5/2012 9:28 PM, Matthew Knepley wrote:
>>>>             On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng
>>>>             <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>>
>>>>
>>>>                 On 4/5/2012 9:16 PM, Barry Smith wrote:
>>>>
>>>>                        Do an hg pull and then run make in
>>>>                     src/mat/interface/ftn-custom/
>>>>
>>>>                         Then it should link.
>>>>
>>>>                        Barry
>>>>
>>>>                        There was a E missing from the all caps name
>>>>                     of the function.
>>>>
>>>>                 After hg pull, I did:
>>>>
>>>>                 cd src//mat/interface/ftn-custom/
>>>>
>>>>                 User at User-PC
>>>>                 /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom
>>>>                 $ make
>>>>                 make[1]: Warning: File
>>>>                 `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)'
>>>>                 has modification time 787 s in the future
>>>>                 make[1]: Nothing to be done for `libc'.
>>>>                 make[1]: warning:  Clock skew detected.  Your build
>>>>                 may be incomplete.
>>>>
>>>>                 But it still can't work.
>>>>
>>>>
>>>>             Something is messed up with the clock on this machine.
>>>>
>>>>             HOWEVER, development requires certain basic skills in
>>>>             order to debug your work. We
>>>>             cannot be the ones debugging your code. Now
>>>>
>>>>               nm $PETSC_ARCH/lib/libpetsc.a | grep -i
>>>>             MatNullSpaceRemove
>>>>
>>>>              will look for the symbol.
>>>>
>>>>                 Matt
>>>>
>>>>
>>>>
>>>>                     On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:
>>>>
>>>>                         On Fri, May 4, 2012 at 3:01 PM, TAY
>>>>                         wee-beng<zonexo at gmail.com
>>>>                         <mailto:zonexo at gmail.com>>  wrote:
>>>>
>>>>                         On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>>>>
>>>>                             On Fri, May 4, 2012 at 11:05 AM, TAY
>>>>                             wee-beng<zonexo at gmail.com
>>>>                             <mailto:zonexo at gmail.com>>  wrote:
>>>>
>>>>                             On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>>>
>>>>                                 On Fri, May 4, 2012 at 8:59 AM, TAY
>>>>                                 wee-beng<zonexo at gmail.com
>>>>                                 <mailto:zonexo at gmail.com>>  wrote:
>>>>
>>>>                                 Hi,
>>>>
>>>>                                 Is there anything else I can try to
>>>>                                 get it working right?
>>>>
>>>>                                 The MatGetNullSpaceRemove() is missing.
>>>>
>>>>                             Where should I add
>>>>                             MatGetNullSpaceRemove and what are its
>>>>                             syntax? I googled but there's no results.
>>>>
>>>>                             Fixed in p;etsc-dev:
>>>>
>>>>                             http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>>>>
>>>>                         I just compiled the updated petsc-dev but I
>>>>                         got the same error msg when I use:
>>>>
>>>>                         call
>>>>                         MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>
>>>>                         error LNK2019: unresolved external symbol
>>>>                         MATNULLSPACEREMOVE referenced in function
>>>>                         COMPUTERHS
>>>>                         1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal
>>>>                         error LNK1120: 1 unresolved externals
>>>>
>>>>                         That function is in:
>>>>
>>>>                           src/mat/interface/ftn-custom/zmatrixf.c
>>>>
>>>>                         Did that compile? Can you see the symbol in
>>>>                         libpetsc.a?
>>>>
>>>>                             Matt
>>>>
>>>>                                  Matt
>>>>
>>>>                             Thanks.
>>>>
>>>>                                    Matt
>>>>
>>>>                                 Thanks?
>>>>
>>>>
>>>>                                 On 2/5/2012 10:11 PM, Matthew
>>>>                                 Knepley wrote:
>>>>
>>>>                                     On Wed, May 2, 2012 at 1:55 PM,
>>>>                                     TAY wee-beng<zonexo at gmail.com
>>>>                                     <mailto:zonexo at gmail.com>>  wrote:
>>>>                                     Hi,
>>>>
>>>>                                     I did a MatView and VecView on
>>>>                                     both C and Fortran, right after
>>>>                                     Mat and Vec assembly. I have
>>>>                                     attached the printout below.
>>>>                                     They are exactly the same, but
>>>>                                     yet the result is different in
>>>>                                     Neumann condition. However, the
>>>>                                     dirichlet condition gives the
>>>>                                     correct ans. Is there anything
>>>>                                     else that could be wrong even
>>>>                                     if the Mat and Vec are the same?
>>>>
>>>>                                     Did you set the null space for
>>>>                                     the matrix when you have
>>>>                                     Neumann conditions?
>>>>
>>>>                                 Yes, for the matrix, I set as:
>>>>
>>>>                                 call
>>>>                                 MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>
>>>>                                     call
>>>>                                 MatSetNullSpace(jac,nullspace,ierr)
>>>>
>>>>                                     call
>>>>                                 MatNullSpaceDestroy(nullspace,ierr)
>>>>
>>>>                                 for the Vec,
>>>>
>>>>                                 call
>>>>                                 MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>
>>>>                                     !call
>>>>                                 MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>
>>>>                                     call
>>>>                                 MatNullSpaceDestroy(nullspace,ierr)
>>>>
>>>>                                 MatNullSpaceRemove was comment out
>>>>                                 because there's error during linking
>>>>
>>>>
>>>>                                        Matt
>>>>
>>>>                                     Thanks!
>>>>
>>>>                                     Fortran:
>>>>
>>>>                                     Matrix Object: 1 MPI processes
>>>>                                       type: seqaij
>>>>                                     row 0: (0, 2)  (1, -1)  (3, -1)
>>>>                                     row 1: (0, -1)  (1, 3)  (2, -1)
>>>>                                      (4, -1)
>>>>                                     row 2: (1, -1)  (2, 2)  (5, -1)
>>>>                                     row 3: (0, -1)  (3, 3)  (4, -1)
>>>>                                      (6, -1)
>>>>                                     row 4: (1, -1)  (3, -1)  (4, 4)
>>>>                                      (5, -1)  (7, -1)
>>>>                                     row 5: (2, -1)  (4, -1)  (5, 3)
>>>>                                      (8, -1)
>>>>                                     row 6: (3, -1)  (6, 2)  (7, -1)
>>>>                                     row 7: (4, -1)  (6, -1)  (7, 3)
>>>>                                      (8, -1)
>>>>                                     row 8: (5, -1)  (7, -1)  (8, 2)
>>>>                                     Vector
>>>>                                     Object:Vec_0000000084000000_0 1
>>>>                                     MPI processes
>>>>                                       type: mpi
>>>>                                     Process [0]
>>>>                                     0.25
>>>>                                     0.0205213
>>>>                                     1.135e-005
>>>>                                     0.0205213
>>>>                                     0.00168449
>>>>                                     9.31663e-007
>>>>                                     1.135e-005
>>>>                                     9.31663e-007
>>>>                                     5.15289e-010
>>>>                                     Vector
>>>>                                     Object:Vec_0000000084000000_1 1
>>>>                                     MPI processes
>>>>                                       type: mpi
>>>>                                     Process [0]
>>>>                                     0.14924
>>>>                                     0.0242397
>>>>                                     -0.0260347
>>>>                                     0.0242397
>>>>                                     -0.0256192
>>>>                                     -0.0400102
>>>>                                     -0.0260347
>>>>                                     -0.0400102
>>>>                                     -0.0400102
>>>>                                     Press any key to continue . . .
>>>>
>>>>                                     C:
>>>>
>>>>                                     Matrix Object: 1 MPI processes
>>>>                                       type: seqaij
>>>>                                     row 0: (0, 2)  (1, -1)  (3, -1)
>>>>                                     row 1: (0, -1)  (1, 3)  (2, -1)
>>>>                                      (4, -1)
>>>>                                     row 2: (1, -1)  (2, 2)  (5, -1)
>>>>                                     row 3: (0, -1)  (3, 3)  (4, -1)
>>>>                                      (6, -1)
>>>>                                     row 4: (1, -1)  (3, -1)  (4, 4)
>>>>                                      (5, -1)  (7, -1)
>>>>                                     row 5: (2, -1)  (4, -1)  (5, 3)
>>>>                                      (8, -1)
>>>>                                     row 6: (3, -1)  (6, 2)  (7, -1)
>>>>                                     row 7: (4, -1)  (6, -1)  (7, 3)
>>>>                                      (8, -1)
>>>>                                     row 8: (5, -1)  (7, -1)  (8, 2)
>>>>                                     Vector Object:Vec_0x1d3b000_0 1
>>>>                                     MPI processes
>>>>                                       type: mpi
>>>>                                     Process [0]
>>>>                                     0.25
>>>>                                     0.0205212
>>>>                                     1.135e-05
>>>>                                     0.0205212
>>>>                                     0.00168449
>>>>                                     9.31663e-07
>>>>                                     1.135e-05
>>>>                                     9.31663e-07
>>>>                                     5.15288e-10
>>>>                                     Vector Object:Vec_0x1d3b000_1 1
>>>>                                     MPI processes
>>>>                                       type: mpi
>>>>                                     Process [0]
>>>>                                     0.139311
>>>>                                     0.0305751
>>>>                                     -0.0220633
>>>>                                     0.0305751
>>>>                                     -0.0135158
>>>>                                     -0.042185
>>>>                                     -0.0220633
>>>>                                     -0.042185
>>>>                                     -0.058449
>>>>
>>>>
>>>>
>>>>                                     Yours sincerely,
>>>>
>>>>                                     TAY wee-beng
>>>>
>>>>
>>>>                                     On 1/5/2012 11:54 PM, Matthew
>>>>                                     Knepley wrote:
>>>>
>>>>                                         On Tue, May 1, 2012 at 5:48
>>>>                                         PM, TAY
>>>>                                         wee-beng<zonexo at gmail.com
>>>>                                         <mailto:zonexo at gmail.com>>
>>>>                                          wrote:
>>>>                                         Hi,
>>>>
>>>>                                         Do you mean my method is wrong?
>>>>
>>>>                                         I am following the template
>>>>                                         of ex22f,
>>>>
>>>>                                         where the variables are
>>>>                                         declared as :
>>>>
>>>>                                         PetscScalar  v(5)
>>>>
>>>>                                         MatStencil   row(4),col(4,5)
>>>>
>>>>                                         Hence,
>>>>
>>>>                                         for the neumann BC
>>>>
>>>>                                         num = 1
>>>>
>>>>                                                        if (j/=0) then
>>>>
>>>>                                                            v(num) =
>>>>                                         -rho*HxdHy
>>>>
>>>>                                                          
>>>>                                          col(MatStencil_i,num) = i
>>>>
>>>>                                                          
>>>>                                          col(MatStencil_j,num) = j-1
>>>>
>>>>                                                            num =
>>>>                                         num + 1
>>>>
>>>>                                                        end if
>>>>
>>>>                                                        if (i/=0) then
>>>>
>>>>                                                            v(num) =
>>>>                                         -rho*HydHx
>>>>
>>>>                                                          
>>>>                                          col(MatStencil_i,num) = i-1
>>>>
>>>>                                                          
>>>>                                          col(MatStencil_j,num) = j
>>>>
>>>>                                                            num =
>>>>                                         num + 1
>>>>
>>>>                                                        end if
>>>>
>>>>                                                        if (i/=mx-1)
>>>>                                         then
>>>>
>>>>                                                            v(num) =
>>>>                                         -rho*HydHx
>>>>
>>>>                                                          
>>>>                                          col(MatStencil_i,num) = i+1
>>>>
>>>>                                                          
>>>>                                          col(MatStencil_j,num) = j
>>>>
>>>>                                                            num =
>>>>                                         num + 1
>>>>
>>>>                                                        end if
>>>>
>>>>                                                        if (j/=my-1)
>>>>                                         then
>>>>
>>>>                                                            v(num) =
>>>>                                         -rho*HxdHy
>>>>
>>>>                                                          
>>>>                                          col(MatStencil_i,num) = i
>>>>
>>>>                                                          
>>>>                                          col(MatStencil_j,num) = j+1
>>>>
>>>>                                                            num =
>>>>                                         num + 1
>>>>
>>>>                                                        end if
>>>>
>>>>                                                        v(num) =
>>>>                                         ((num-1)/2.0)*rho*(HxdHy +
>>>>                                         HydHx)
>>>>
>>>>                                                        print *, v
>>>>
>>>>                                                      
>>>>                                          col(MatStencil_i,num) = i
>>>>
>>>>                                                      
>>>>                                          col(MatStencil_j,num) = j
>>>>
>>>>                                                        !num = num + 1
>>>>
>>>>                                                        call
>>>>                                         MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>
>>>>                                         I do not get any more out
>>>>                                         of range error. However,my
>>>>                                         ans is still different from
>>>>                                         that of ex29 in C.
>>>>
>>>>                                         This is very simple. You
>>>>                                         have an error in your code.
>>>>                                         Checking it is very simple:
>>>>                                         run the code and
>>>>                                         break in MatSetValues().
>>>>                                         Make sure ex29 makes calls
>>>>                                         with exactly the same
>>>>                                         indices as your ex29f.
>>>>
>>>>                                             Matt
>>>>
>>>>                                         Yours sincerely,
>>>>
>>>>                                         TAY wee-beng
>>>>
>>>>                                         -- 
>>>>                                         What most experimenters
>>>>                                         take for granted before
>>>>                                         they begin their
>>>>                                         experiments is infinitely
>>>>                                         more interesting than any
>>>>                                         results to which their
>>>>                                         experiments lead.
>>>>                                         -- Norbert Wiener
>>>>
>>>>
>>>>
>>>>                                     -- 
>>>>                                     What most experimenters take
>>>>                                     for granted before they begin
>>>>                                     their experiments is infinitely
>>>>                                     more interesting than any
>>>>                                     results to which their
>>>>                                     experiments lead.
>>>>                                     -- Norbert Wiener
>>>>
>>>>
>>>>
>>>>                                 -- 
>>>>                                 What most experimenters take for
>>>>                                 granted before they begin their
>>>>                                 experiments is infinitely more
>>>>                                 interesting than any results to
>>>>                                 which their experiments lead.
>>>>                                 -- Norbert Wiener
>>>>
>>>>
>>>>
>>>>                             -- 
>>>>                             What most experimenters take for
>>>>                             granted before they begin their
>>>>                             experiments is infinitely more
>>>>                             interesting than any results to which
>>>>                             their experiments lead.
>>>>                             -- Norbert Wiener
>>>>
>>>>
>>>>
>>>>                         -- 
>>>>                         What most experimenters take for granted
>>>>                         before they begin their experiments is
>>>>                         infinitely more interesting than any
>>>>                         results to which their experiments lead.
>>>>                         -- Norbert Wiener
>>>>
>>>>
>>>>
>>>>
>>>>             -- 
>>>>             What most experimenters take for granted before they
>>>>             begin their experiments is infinitely more interesting
>>>>             than any results to which their experiments lead.
>>>>             -- Norbert Wiener
>>>
>>>
>>>
>>>
>>>         -- 
>>>         What most experimenters take for granted before they begin
>>>         their experiments is infinitely more interesting than any
>>>         results to which their experiments lead.
>>>         -- Norbert Wiener
>>
>>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120511/7cf300cb/attachment-0001.htm>

From knepley at gmail.com  Fri May 11 09:02:57 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 May 2012 10:02:57 -0400
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
	ksp
In-Reply-To: <4FAD195B.4030302@gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
	<4FA42CFC.8090002@gmail.com>
	<CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>
	<4FA44D5C.6010205@gmail.com>
	<CAMYG4Gm9cJxm1D0UmO9Fepv93uEHYhYDCcU1rKvsTU=D_+v9qQ@mail.gmail.com>
	<4FA97132.3080103@gmail.com>
	<CA+zegpnTN40NUz+hD=BCqgv2kyz8sywzGDz+jDeVH3kXb+LSBw@mail.gmail.com>
	<4FAD12FB.6080506@gmail.com>
	<CAMYG4GmW=A9UKr0qQT7t8UTsns_ORr_UB97wUZsJtx87fibOTQ@mail.gmail.com>
	<4FAD195B.4030302@gmail.com>
Message-ID: <CAMYG4G=p979GjLSNirbVcD=GeFeQQNLrb-imVMUU3Dm4VKT1JQ@mail.gmail.com>

On Fri, May 11, 2012 at 9:51 AM, TAY wee-beng <zonexo at gmail.com> wrote:

>
>
> On 11/5/2012 3:30 PM, Matthew Knepley wrote:
>
> On Fri, May 11, 2012 at 9:24 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>  Hi,
>>
>> I have been using the GUI environment to do debugging so I am a bit
>> reluctant to learn Valgrind as its outputs seems a bit daunting.  But I
>> guess John is right. I've been spending these few days learning bit by bit.
>>
>> I realised that the error occurs in computerhs, at:
>>
>
>  I bet this is a beautiful Fortranism. Do you include the F90 header file
> with the interface definition?
> If not, Fortran just craps out like this. I can't stress enough how much
> time would be saved by
> switching languages to something with at least a modicum of error checking.
>
>
http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/UsingFortran.html

You need both

#include "finclude/petsc.h"
#include "finclude/petsc.h90"

    Matt


> I initially used:
>
> #include "finclude/petsc.h90"
>
> Compilation and linking was fine in Linux and vs2008.
>
> Now I changed it to what ex22.F was using :
>
> #include <finclude/petscsys.h>
> #include <finclude/petscvec.h>
> #include <finclude/petscmat.h>
> #include <finclude/petscpc.h>
> #include <finclude/petscksp.h>
> #include <finclude/petscdmda.h>
>
> Compiling was ok but linking failed in Linux and VS2008:
>
> undefined reference to `dmdavecgetarrayf90_'
>
> I tried changing #include <finclude/petscdmda.h> to #include
> <finclude/petscdmda.h90> and everything was ok in VS2008 again, giving the
> right answers.
>
> However, in Linux, I got the following error:
>
> [wtay at hpc12:tutorials]$ /opt/openmpi-1.5.3/bin/mpif90 -c  -fPIC -g
> -I/home/wtay/Lib/petsc-3.2-dev_shared_debug/include
> -I/home/wtay/Lib/petsc-3.2-dev_shared_debug/include
> -I/opt/openmpi-1.5.3/include    -o ex29f.o ex29f.F90
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
> error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
>         DMDABoundaryType :: pt
> -------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
> error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
>         DMDAStencilType  :: st
> -------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
> error #6590: This statement is not permitted as a statement within a
> derived-type-def
>         DMDABoundaryType :: pt
> --------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
> error #6590: This statement is not permitted as a statement within a
> derived-type-def
>         DMDAStencilType  :: st
> --------^
> ex29f.F90(68): error #6404: This name does not have a type, and must have
> an explicit type.   [DMDA_BOUNDARY_NONE]
> call
> DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
> -----------------------------------^
> ex29f.F90(68): error #6404: This name does not have a type, and must have
> an explicit type.   [DMDA_STENCIL_STAR]
> call
> DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
> -------------------------------------------------------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
> error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
>         DMDABoundaryType :: pt
> -------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
> error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
>         DMDAStencilType  :: st
> -------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
> error #6590: This statement is not permitted as a statement within a
> derived-type-def
>         DMDABoundaryType :: pt
> --------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
> error #6590: This statement is not permitted as a statement within a
> derived-type-def
>         DMDAStencilType  :: st
> --------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
> error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
>         DMDABoundaryType :: pt
> -------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
> error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
>         DMDAStencilType  :: st
> -------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
> error #6590: This statement is not permitted as a statement within a
> derived-type-def
>         DMDABoundaryType :: pt
> --------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
> error #6590: This statement is not permitted as a statement within a
> derived-type-def
>         DMDAStencilType  :: st
>
> Is there some errors in petscdmda.h90?
>
>
>
>     Matt
>
>
>>  *call DMDAVecRestoreArrayF90(da,b,array,ierr)*
>>
>> ==27464== Invalid write of size 8
>> ==27464==    at 0x402835: computerhs_ (ex29f.F90:119)
>> ==27464==  Address 0xfffffffffffffef0 is not stack'd, malloc'd or
>> (recently) free'd
>> ==27464==
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC
>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
>> find memory corruption errors
>> [0]PETSC ERROR: likely location of problem given in stack below
>> [0]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [0]PETSC ERROR:       is given.
>> [0]PETSC ERROR: [0] DM user function line 0 unknownunknown
>> [0]PETSC ERROR: [0] DMComputeFunction line 2085
>> /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
>> [0]PETSC ERROR: [0] KSPSetUp line 182
>> /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Signal received!
>>
>> I have checked that "array" 's values are correct. This statement
>> executed without problems in VS2008. If I replace the above with something
>> like:
>>
>> *call VecSet(b,Hy,ierr)*
>>
>> Everything is fine in Linux.
>>
>> Is there something wrong with *DMDAVecRestoreArrayF90*?
>>
>> My code in the area is:
>>
>> call
>> DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>
>> call DMDAVecGetArrayF90(da,b,array,ierr)
>>
>> do j = ys,ys+ym-1
>>
>>     do i = xs,xs+xm-1
>>
>>         array(i,j) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>>
>>     end do
>>
>> end do
>>
>> call DMDAVecRestoreArrayF90(da,b,array,ierr)
>>
>> call VecAssemblyBegin(b,ierr)
>>
>> call VecAssemblyEnd(b,ierr)
>>
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>> On 8/5/2012 9:41 PM, John Mousel wrote:
>>
>> TAY wee-bing,
>>
>> If you want to be a programmer that writes interesting and reliable code,
>> you need to be willing to use the tools of the trade. I can't think of a
>> bigger time-saver than Valgrind. I would suggest that you learn to use it
>> and use it a lot. I bet it will lead you to the root of your problem pretty
>> quickly.
>>
>> John
>>
>> On Tue, May 8, 2012 at 2:17 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>>  Hi,
>>>
>>> I compiled and run my code under visual studio 2008 with intel fortran.
>>> Everything works ok.
>>>
>>> However, when I tried to run the code in linux, I got the error as
>>> below. The error happens when KSPSetUp(ksp,ierr) is called.
>>>
>>> However, I am not able to print VecView or MatView to view if there's
>>> any errors. Is there any recommendation for debugging? I hope I do not need
>>> to valgrind if possible.
>>>
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>>> probably memory access out of range
>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> [0]PETSC ERROR: or see
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC
>>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
>>> find memory corruption errors
>>> [0]PETSC ERROR: likely location of problem given in stack below
>>> [0]PETSC ERROR: ---------------------  Stack Frames
>>> ------------------------------------
>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>> available,
>>> [0]PETSC ERROR:       INSTEAD the line number of the start of the
>>> function
>>> [0]PETSC ERROR:       is given.
>>> [0]PETSC ERROR: [0] DM user function line 0 unknownunknown
>>> [0]PETSC ERROR: [0] DMComputeFunction line 2085
>>> /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
>>> [0]PETSC ERROR: [0] KSPSetUp line 182
>>> /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [0]PETSC ERROR: Signal received!
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc Development HG revision:
>>> 7ecdd63ec420b1659b960e65d96e822c5ac1a968  HG Date: Mon May 07 21:42:26 2012
>>> -0500
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: ./ex29f on a petsc-3.2 named hpc12 by wtay Tue May  8
>>> 20:45:42 2012
>>> [0]PETSC ERROR: Libraries linked from
>>> /home/wtay/Lib/petsc-3.2-dev_shared_debug/lib
>>> [0]PETSC ERROR: Configure run at Tue May  8 10:47:59 2012
>>> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/
>>> --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/
>>> --with-debugging=1 --download-hypre=1
>>> --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_debug --known-mpi-shared=1
>>> --with-shared-libraries
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory
>>> unknown file
>>>
>>> --------------------------------------------------------------------------
>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>>> with errorcode 59.
>>>
>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>> You may or may not see output from other processes, depending on
>>> exactly when Open MPI kills them.
>>>
>>> Yours sincerely,
>>>
>>> TAY wee-beng
>>>
>>>
>>> On 5/5/2012 1:43 AM, Matthew Knepley wrote:
>>>
>>> On Fri, May 4, 2012 at 5:42 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>
>>>>  Hi,
>>>>
>>>> I wonder if you people are interested to include my ex29 fortran
>>>> version in the petsc examples, which can help people who are using fortran.
>>>>
>>>
>>>  Yes, we will definitely include it. Please send the source, and a
>>> representative output with run options.
>>>
>>>    Thanks,
>>>
>>>       Matt
>>>
>>>
>>>>  Thanks.
>>>>
>>>> Yours sincerely,
>>>>
>>>> TAY wee-beng
>>>>
>>>>
>>>> On 4/5/2012 9:28 PM, Matthew Knepley wrote:
>>>>
>>>> On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>
>>>>>
>>>>> On 4/5/2012 9:16 PM, Barry Smith wrote:
>>>>>
>>>>>>    Do an hg pull and then run make in src/mat/interface/ftn-custom/
>>>>>>
>>>>>>     Then it should link.
>>>>>>
>>>>>>    Barry
>>>>>>
>>>>>>    There was a E missing from the all caps name of the function.
>>>>>>
>>>>> After hg pull, I did:
>>>>>
>>>>> cd src//mat/interface/ftn-custom/
>>>>>
>>>>> User at User-PC /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom
>>>>> $ make
>>>>> make[1]: Warning: File
>>>>> `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)'
>>>>> has modification time 787 s in the future
>>>>> make[1]: Nothing to be done for `libc'.
>>>>> make[1]: warning:  Clock skew detected.  Your build may be incomplete.
>>>>>
>>>>> But it still can't work.
>>>>>
>>>>
>>>>  Something is messed up with the clock on this machine.
>>>>
>>>>  HOWEVER, development requires certain basic skills in order to debug
>>>> your work. We
>>>> cannot be the ones debugging your code. Now
>>>>
>>>>    nm $PETSC_ARCH/lib/libpetsc.a | grep -i MatNullSpaceRemove
>>>>
>>>>   will look for the symbol.
>>>>
>>>>      Matt
>>>>
>>>>
>>>>>
>>>>>>
>>>>>> On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:
>>>>>>
>>>>>>  On Fri, May 4, 2012 at 3:01 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>  wrote:
>>>>>>>
>>>>>>> On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>>>>>>>
>>>>>>>> On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>  wrote:
>>>>>>>>
>>>>>>>> On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>>>>>>>
>>>>>>>>> On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>>  wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Is there anything else I can try to get it working right?
>>>>>>>>>
>>>>>>>>> The MatGetNullSpaceRemove() is missing.
>>>>>>>>>
>>>>>>>> Where should I add MatGetNullSpaceRemove and what are its syntax? I
>>>>>>>> googled but there's no results.
>>>>>>>>
>>>>>>>> Fixed in p;etsc-dev:
>>>>>>>>
>>>>>>>>
>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>>>>>>>>
>>>>>>> I just compiled the updated petsc-dev but I got the same error msg
>>>>>>> when I use:
>>>>>>>
>>>>>>> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>>>>
>>>>>>> error LNK2019: unresolved external symbol MATNULLSPACEREMOVE
>>>>>>> referenced in function COMPUTERHS
>>>>>>> 1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120: 1
>>>>>>> unresolved externals
>>>>>>>
>>>>>>> That function is in:
>>>>>>>
>>>>>>>   src/mat/interface/ftn-custom/zmatrixf.c
>>>>>>>
>>>>>>> Did that compile? Can you see the symbol in libpetsc.a?
>>>>>>>
>>>>>>>     Matt
>>>>>>>
>>>>>>>       Matt
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>>    Matt
>>>>>>>>>
>>>>>>>>> Thanks?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>>>>>>>
>>>>>>>>>> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>>>  wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I did a MatView and VecView on both C and Fortran, right after
>>>>>>>>>> Mat and Vec assembly. I have attached the printout below. They are exactly
>>>>>>>>>> the same, but yet the result is different in Neumann condition. However,
>>>>>>>>>> the dirichlet condition gives the correct ans. Is there anything else that
>>>>>>>>>> could be wrong even if the Mat and Vec are the same?
>>>>>>>>>>
>>>>>>>>>> Did you set the null space for the matrix when you have Neumann
>>>>>>>>>> conditions?
>>>>>>>>>>
>>>>>>>>> Yes, for the matrix, I set as:
>>>>>>>>>
>>>>>>>>> call
>>>>>>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>>>>>>
>>>>>>>>>     call MatSetNullSpace(jac,nullspace,ierr)
>>>>>>>>>
>>>>>>>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>>>>
>>>>>>>>> for the Vec,
>>>>>>>>>
>>>>>>>>> call
>>>>>>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>>>>>>
>>>>>>>>>     !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>>>>>>
>>>>>>>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>>>>
>>>>>>>>> MatNullSpaceRemove was comment out because there's error during
>>>>>>>>> linking
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>     Matt
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> Fortran:
>>>>>>>>>>
>>>>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>>>>   type: seqaij
>>>>>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>>>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>>>>>>>   type: mpi
>>>>>>>>>> Process [0]
>>>>>>>>>> 0.25
>>>>>>>>>> 0.0205213
>>>>>>>>>> 1.135e-005
>>>>>>>>>> 0.0205213
>>>>>>>>>> 0.00168449
>>>>>>>>>> 9.31663e-007
>>>>>>>>>> 1.135e-005
>>>>>>>>>> 9.31663e-007
>>>>>>>>>> 5.15289e-010
>>>>>>>>>> Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>>>>>>>>   type: mpi
>>>>>>>>>> Process [0]
>>>>>>>>>> 0.14924
>>>>>>>>>> 0.0242397
>>>>>>>>>> -0.0260347
>>>>>>>>>> 0.0242397
>>>>>>>>>> -0.0256192
>>>>>>>>>> -0.0400102
>>>>>>>>>> -0.0260347
>>>>>>>>>> -0.0400102
>>>>>>>>>> -0.0400102
>>>>>>>>>> Press any key to continue . . .
>>>>>>>>>>
>>>>>>>>>> C:
>>>>>>>>>>
>>>>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>>>>   type: seqaij
>>>>>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>>>>>> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>>>>>>>>   type: mpi
>>>>>>>>>> Process [0]
>>>>>>>>>> 0.25
>>>>>>>>>> 0.0205212
>>>>>>>>>> 1.135e-05
>>>>>>>>>> 0.0205212
>>>>>>>>>> 0.00168449
>>>>>>>>>> 9.31663e-07
>>>>>>>>>> 1.135e-05
>>>>>>>>>> 9.31663e-07
>>>>>>>>>> 5.15288e-10
>>>>>>>>>> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>>>>>>>>   type: mpi
>>>>>>>>>> Process [0]
>>>>>>>>>> 0.139311
>>>>>>>>>> 0.0305751
>>>>>>>>>> -0.0220633
>>>>>>>>>> 0.0305751
>>>>>>>>>> -0.0135158
>>>>>>>>>> -0.042185
>>>>>>>>>> -0.0220633
>>>>>>>>>> -0.042185
>>>>>>>>>> -0.058449
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yours sincerely,
>>>>>>>>>>
>>>>>>>>>> TAY wee-beng
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>>>>>>>
>>>>>>>>>>> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>>>>  wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Do you mean my method is wrong?
>>>>>>>>>>>
>>>>>>>>>>> I am following the template of ex22f,
>>>>>>>>>>>
>>>>>>>>>>> where the variables are declared as :
>>>>>>>>>>>
>>>>>>>>>>> PetscScalar  v(5)
>>>>>>>>>>>
>>>>>>>>>>> MatStencil   row(4),col(4,5)
>>>>>>>>>>>
>>>>>>>>>>> Hence,
>>>>>>>>>>>
>>>>>>>>>>> for the neumann BC
>>>>>>>>>>>
>>>>>>>>>>> num = 1
>>>>>>>>>>>
>>>>>>>>>>>                if (j/=0) then
>>>>>>>>>>>
>>>>>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_j,num) = j-1
>>>>>>>>>>>
>>>>>>>>>>>                    num = num + 1
>>>>>>>>>>>
>>>>>>>>>>>                end if
>>>>>>>>>>>
>>>>>>>>>>>                if (i/=0) then
>>>>>>>>>>>
>>>>>>>>>>>                    v(num) = -rho*HydHx
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_i,num) = i-1
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>>>>>>
>>>>>>>>>>>                    num = num + 1
>>>>>>>>>>>
>>>>>>>>>>>                end if
>>>>>>>>>>>
>>>>>>>>>>>                if (i/=mx-1) then
>>>>>>>>>>>
>>>>>>>>>>>                    v(num) = -rho*HydHx
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_i,num) = i+1
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>>>>>>
>>>>>>>>>>>                    num = num + 1
>>>>>>>>>>>
>>>>>>>>>>>                end if
>>>>>>>>>>>
>>>>>>>>>>>                if (j/=my-1) then
>>>>>>>>>>>
>>>>>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_j,num) = j+1
>>>>>>>>>>>
>>>>>>>>>>>                    num = num + 1
>>>>>>>>>>>
>>>>>>>>>>>                end if
>>>>>>>>>>>
>>>>>>>>>>>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>>>>>>>>
>>>>>>>>>>>                print *, v
>>>>>>>>>>>
>>>>>>>>>>>                col(MatStencil_i,num) = i
>>>>>>>>>>>
>>>>>>>>>>>                col(MatStencil_j,num) = j
>>>>>>>>>>>
>>>>>>>>>>>                !num = num + 1
>>>>>>>>>>>
>>>>>>>>>>>                call
>>>>>>>>>>> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>>>>>>>>
>>>>>>>>>>> I do not get any more out of range error. However,my ans is
>>>>>>>>>>> still different from that of ex29 in C.
>>>>>>>>>>>
>>>>>>>>>>> This is very simple. You have an error in your code. Checking it
>>>>>>>>>>> is very simple: run the code and
>>>>>>>>>>> break in MatSetValues(). Make sure ex29 makes calls with exactly
>>>>>>>>>>> the same indices as your ex29f.
>>>>>>>>>>>
>>>>>>>>>>>     Matt
>>>>>>>>>>>
>>>>>>>>>>> Yours sincerely,
>>>>>>>>>>>
>>>>>>>>>>> TAY wee-beng
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>
>>>>
>>>>  --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>>
>>>
>>>
>>>  --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120511/60e852c2/attachment-0001.htm>

From john.mousel at gmail.com  Fri May 11 09:04:08 2012
From: john.mousel at gmail.com (John Mousel)
Date: Fri, 11 May 2012 09:04:08 -0500
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
	ksp
In-Reply-To: <4FAD195B.4030302@gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
	<4FA42CFC.8090002@gmail.com>
	<CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>
	<4FA44D5C.6010205@gmail.com>
	<CAMYG4Gm9cJxm1D0UmO9Fepv93uEHYhYDCcU1rKvsTU=D_+v9qQ@mail.gmail.com>
	<4FA97132.3080103@gmail.com>
	<CA+zegpnTN40NUz+hD=BCqgv2kyz8sywzGDz+jDeVH3kXb+LSBw@mail.gmail.com>
	<4FAD12FB.6080506@gmail.com>
	<CAMYG4GmW=A9UKr0qQT7t8UTsns_ORr_UB97wUZsJtx87fibOTQ@mail.gmail.com>
	<4FAD195B.4030302@gmail.com>
Message-ID: <CA+zegpm5Ca=6rKMBEjQ3vJZM0hCvMyor1BF2Re27Hb_vHwCRRg@mail.gmail.com>

Include both header files.

#include <finclude/petscdmda.h>
#include <finclude/petscdmda.h90>

The h90 header only includes the interface definitions for Fortran.

John

On Fri, May 11, 2012 at 8:51 AM, TAY wee-beng <zonexo at gmail.com> wrote:

>
>
> On 11/5/2012 3:30 PM, Matthew Knepley wrote:
>
> On Fri, May 11, 2012 at 9:24 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>>  Hi,
>>
>> I have been using the GUI environment to do debugging so I am a bit
>> reluctant to learn Valgrind as its outputs seems a bit daunting.  But I
>> guess John is right. I've been spending these few days learning bit by bit.
>>
>> I realised that the error occurs in computerhs, at:
>>
>
>  I bet this is a beautiful Fortranism. Do you include the F90 header file
> with the interface definition?
> If not, Fortran just craps out like this. I can't stress enough how much
> time would be saved by
> switching languages to something with at least a modicum of error checking.
>
>
> I initially used:
>
> #include "finclude/petsc.h90"
>
> Compilation and linking was fine in Linux and vs2008.
>
> Now I changed it to what ex22.F was using :
>
> #include <finclude/petscsys.h>
> #include <finclude/petscvec.h>
> #include <finclude/petscmat.h>
> #include <finclude/petscpc.h>
> #include <finclude/petscksp.h>
> #include <finclude/petscdmda.h>
>
> Compiling was ok but linking failed in Linux and VS2008:
>
> undefined reference to `dmdavecgetarrayf90_'
>
> I tried changing #include <finclude/petscdmda.h> to #include
> <finclude/petscdmda.h90> and everything was ok in VS2008 again, giving the
> right answers.
>
> However, in Linux, I got the following error:
>
> [wtay at hpc12:tutorials]$ /opt/openmpi-1.5.3/bin/mpif90 -c  -fPIC -g
> -I/home/wtay/Lib/petsc-3.2-dev_shared_debug/include
> -I/home/wtay/Lib/petsc-3.2-dev_shared_debug/include
> -I/opt/openmpi-1.5.3/include    -o ex29f.o ex29f.F90
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
> error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
>         DMDABoundaryType :: pt
> -------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
> error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
>         DMDAStencilType  :: st
> -------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
> error #6590: This statement is not permitted as a statement within a
> derived-type-def
>         DMDABoundaryType :: pt
> --------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
> error #6590: This statement is not permitted as a statement within a
> derived-type-def
>         DMDAStencilType  :: st
> --------^
> ex29f.F90(68): error #6404: This name does not have a type, and must have
> an explicit type.   [DMDA_BOUNDARY_NONE]
> call
> DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
> -----------------------------------^
> ex29f.F90(68): error #6404: This name does not have a type, and must have
> an explicit type.   [DMDA_STENCIL_STAR]
> call
> DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
> -------------------------------------------------------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
> error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
>         DMDABoundaryType :: pt
> -------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
> error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
>         DMDAStencilType  :: st
> -------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
> error #6590: This statement is not permitted as a statement within a
> derived-type-def
>         DMDABoundaryType :: pt
> --------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
> error #6590: This statement is not permitted as a statement within a
> derived-type-def
>         DMDAStencilType  :: st
> --------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
> error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
>         DMDABoundaryType :: pt
> -------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
> error #5082: Syntax error, found '::' when expecting one of: ( % : . = =>
>         DMDAStencilType  :: st
> -------------------------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
> error #6590: This statement is not permitted as a statement within a
> derived-type-def
>         DMDABoundaryType :: pt
> --------^
> /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
> error #6590: This statement is not permitted as a statement within a
> derived-type-def
>         DMDAStencilType  :: st
>
> Is there some errors in petscdmda.h90?
>
>
>
>
>     Matt
>
>
>>  *call DMDAVecRestoreArrayF90(da,b,array,ierr)*
>>
>> ==27464== Invalid write of size 8
>> ==27464==    at 0x402835: computerhs_ (ex29f.F90:119)
>> ==27464==  Address 0xfffffffffffffef0 is not stack'd, malloc'd or
>> (recently) free'd
>> ==27464==
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC
>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
>> find memory corruption errors
>> [0]PETSC ERROR: likely location of problem given in stack below
>> [0]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
>> [0]PETSC ERROR:       is given.
>> [0]PETSC ERROR: [0] DM user function line 0 unknownunknown
>> [0]PETSC ERROR: [0] DMComputeFunction line 2085
>> /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
>> [0]PETSC ERROR: [0] KSPSetUp line 182
>> /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Signal received!
>>
>> I have checked that "array" 's values are correct. This statement
>> executed without problems in VS2008. If I replace the above with something
>> like:
>>
>> *call VecSet(b,Hy,ierr)*
>>
>> Everything is fine in Linux.
>>
>> Is there something wrong with *DMDAVecRestoreArrayF90*?
>>
>> My code in the area is:
>>
>> call
>> DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>
>> call DMDAVecGetArrayF90(da,b,array,ierr)
>>
>> do j = ys,ys+ym-1
>>
>>     do i = xs,xs+xm-1
>>
>>         array(i,j) = exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>>
>>     end do
>>
>> end do
>>
>> call DMDAVecRestoreArrayF90(da,b,array,ierr)
>>
>> call VecAssemblyBegin(b,ierr)
>>
>> call VecAssemblyEnd(b,ierr)
>>
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>> On 8/5/2012 9:41 PM, John Mousel wrote:
>>
>> TAY wee-bing,
>>
>> If you want to be a programmer that writes interesting and reliable code,
>> you need to be willing to use the tools of the trade. I can't think of a
>> bigger time-saver than Valgrind. I would suggest that you learn to use it
>> and use it a lot. I bet it will lead you to the root of your problem pretty
>> quickly.
>>
>> John
>>
>> On Tue, May 8, 2012 at 2:17 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>>>  Hi,
>>>
>>> I compiled and run my code under visual studio 2008 with intel fortran.
>>> Everything works ok.
>>>
>>> However, when I tried to run the code in linux, I got the error as
>>> below. The error happens when KSPSetUp(ksp,ierr) is called.
>>>
>>> However, I am not able to print VecView or MatView to view if there's
>>> any errors. Is there any recommendation for debugging? I hope I do not need
>>> to valgrind if possible.
>>>
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>>> probably memory access out of range
>>> [0]PETSC ERROR: Try option -start_in_debugger or
>>> -on_error_attach_debugger
>>> [0]PETSC ERROR: or see
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC
>>> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
>>> find memory corruption errors
>>> [0]PETSC ERROR: likely location of problem given in stack below
>>> [0]PETSC ERROR: ---------------------  Stack Frames
>>> ------------------------------------
>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>>> available,
>>> [0]PETSC ERROR:       INSTEAD the line number of the start of the
>>> function
>>> [0]PETSC ERROR:       is given.
>>> [0]PETSC ERROR: [0] DM user function line 0 unknownunknown
>>> [0]PETSC ERROR: [0] DMComputeFunction line 2085
>>> /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
>>> [0]PETSC ERROR: [0] KSPSetUp line 182
>>> /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>> [0]PETSC ERROR: Signal received!
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc Development HG revision:
>>> 7ecdd63ec420b1659b960e65d96e822c5ac1a968  HG Date: Mon May 07 21:42:26 2012
>>> -0500
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: ./ex29f on a petsc-3.2 named hpc12 by wtay Tue May  8
>>> 20:45:42 2012
>>> [0]PETSC ERROR: Libraries linked from
>>> /home/wtay/Lib/petsc-3.2-dev_shared_debug/lib
>>> [0]PETSC ERROR: Configure run at Tue May  8 10:47:59 2012
>>> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-1.5.3/
>>> --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/
>>> --with-debugging=1 --download-hypre=1
>>> --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_debug --known-mpi-shared=1
>>> --with-shared-libraries
>>> [0]PETSC ERROR:
>>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: User provided function() line 0 in unknown directory
>>> unknown file
>>>
>>> --------------------------------------------------------------------------
>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
>>> with errorcode 59.
>>>
>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>>> You may or may not see output from other processes, depending on
>>> exactly when Open MPI kills them.
>>>
>>> Yours sincerely,
>>>
>>> TAY wee-beng
>>>
>>>
>>> On 5/5/2012 1:43 AM, Matthew Knepley wrote:
>>>
>>> On Fri, May 4, 2012 at 5:42 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>
>>>>  Hi,
>>>>
>>>> I wonder if you people are interested to include my ex29 fortran
>>>> version in the petsc examples, which can help people who are using fortran.
>>>>
>>>
>>>  Yes, we will definitely include it. Please send the source, and a
>>> representative output with run options.
>>>
>>>    Thanks,
>>>
>>>       Matt
>>>
>>>
>>>>  Thanks.
>>>>
>>>> Yours sincerely,
>>>>
>>>> TAY wee-beng
>>>>
>>>>
>>>> On 4/5/2012 9:28 PM, Matthew Knepley wrote:
>>>>
>>>> On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>>>>
>>>>>
>>>>> On 4/5/2012 9:16 PM, Barry Smith wrote:
>>>>>
>>>>>>    Do an hg pull and then run make in src/mat/interface/ftn-custom/
>>>>>>
>>>>>>     Then it should link.
>>>>>>
>>>>>>    Barry
>>>>>>
>>>>>>    There was a E missing from the all caps name of the function.
>>>>>>
>>>>> After hg pull, I did:
>>>>>
>>>>> cd src//mat/interface/ftn-custom/
>>>>>
>>>>> User at User-PC /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom
>>>>> $ make
>>>>> make[1]: Warning: File
>>>>> `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)'
>>>>> has modification time 787 s in the future
>>>>> make[1]: Nothing to be done for `libc'.
>>>>> make[1]: warning:  Clock skew detected.  Your build may be incomplete.
>>>>>
>>>>> But it still can't work.
>>>>>
>>>>
>>>>  Something is messed up with the clock on this machine.
>>>>
>>>>  HOWEVER, development requires certain basic skills in order to debug
>>>> your work. We
>>>> cannot be the ones debugging your code. Now
>>>>
>>>>    nm $PETSC_ARCH/lib/libpetsc.a | grep -i MatNullSpaceRemove
>>>>
>>>>   will look for the symbol.
>>>>
>>>>      Matt
>>>>
>>>>
>>>>>
>>>>>>
>>>>>> On May 4, 2012, at 2:11 PM, Matthew Knepley wrote:
>>>>>>
>>>>>>  On Fri, May 4, 2012 at 3:01 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>  wrote:
>>>>>>>
>>>>>>> On 4/5/2012 5:17 PM, Matthew Knepley wrote:
>>>>>>>
>>>>>>>> On Fri, May 4, 2012 at 11:05 AM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>  wrote:
>>>>>>>>
>>>>>>>> On 4/5/2012 3:05 PM, Matthew Knepley wrote:
>>>>>>>>
>>>>>>>>> On Fri, May 4, 2012 at 8:59 AM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>>  wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Is there anything else I can try to get it working right?
>>>>>>>>>
>>>>>>>>> The MatGetNullSpaceRemove() is missing.
>>>>>>>>>
>>>>>>>> Where should I add MatGetNullSpaceRemove and what are its syntax? I
>>>>>>>> googled but there's no results.
>>>>>>>>
>>>>>>>> Fixed in p;etsc-dev:
>>>>>>>>
>>>>>>>>
>>>>>>>> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>>>>>>>>
>>>>>>> I just compiled the updated petsc-dev but I got the same error msg
>>>>>>> when I use:
>>>>>>>
>>>>>>> call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>>>>
>>>>>>> error LNK2019: unresolved external symbol MATNULLSPACEREMOVE
>>>>>>> referenced in function COMPUTERHS
>>>>>>> 1>c:\obj_tmp\ex29f\Debug\ex29f.exe : fatal error LNK1120: 1
>>>>>>> unresolved externals
>>>>>>>
>>>>>>> That function is in:
>>>>>>>
>>>>>>>   src/mat/interface/ftn-custom/zmatrixf.c
>>>>>>>
>>>>>>> Did that compile? Can you see the symbol in libpetsc.a?
>>>>>>>
>>>>>>>     Matt
>>>>>>>
>>>>>>>       Matt
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>>    Matt
>>>>>>>>>
>>>>>>>>> Thanks?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2/5/2012 10:11 PM, Matthew Knepley wrote:
>>>>>>>>>
>>>>>>>>>> On Wed, May 2, 2012 at 1:55 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>>>  wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I did a MatView and VecView on both C and Fortran, right after
>>>>>>>>>> Mat and Vec assembly. I have attached the printout below. They are exactly
>>>>>>>>>> the same, but yet the result is different in Neumann condition. However,
>>>>>>>>>> the dirichlet condition gives the correct ans. Is there anything else that
>>>>>>>>>> could be wrong even if the Mat and Vec are the same?
>>>>>>>>>>
>>>>>>>>>> Did you set the null space for the matrix when you have Neumann
>>>>>>>>>> conditions?
>>>>>>>>>>
>>>>>>>>> Yes, for the matrix, I set as:
>>>>>>>>>
>>>>>>>>> call
>>>>>>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>>>>>>
>>>>>>>>>     call MatSetNullSpace(jac,nullspace,ierr)
>>>>>>>>>
>>>>>>>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>>>>
>>>>>>>>> for the Vec,
>>>>>>>>>
>>>>>>>>> call
>>>>>>>>> MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>>>>>>
>>>>>>>>>     !call MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>>>>>>
>>>>>>>>>     call MatNullSpaceDestroy(nullspace,ierr)
>>>>>>>>>
>>>>>>>>> MatNullSpaceRemove was comment out because there's error during
>>>>>>>>> linking
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>     Matt
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> Fortran:
>>>>>>>>>>
>>>>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>>>>   type: seqaij
>>>>>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>>>>>> Vector Object:Vec_0000000084000000_0 1 MPI processes
>>>>>>>>>>   type: mpi
>>>>>>>>>> Process [0]
>>>>>>>>>> 0.25
>>>>>>>>>> 0.0205213
>>>>>>>>>> 1.135e-005
>>>>>>>>>> 0.0205213
>>>>>>>>>> 0.00168449
>>>>>>>>>> 9.31663e-007
>>>>>>>>>> 1.135e-005
>>>>>>>>>> 9.31663e-007
>>>>>>>>>> 5.15289e-010
>>>>>>>>>> Vector Object:Vec_0000000084000000_1 1 MPI processes
>>>>>>>>>>   type: mpi
>>>>>>>>>> Process [0]
>>>>>>>>>> 0.14924
>>>>>>>>>> 0.0242397
>>>>>>>>>> -0.0260347
>>>>>>>>>> 0.0242397
>>>>>>>>>> -0.0256192
>>>>>>>>>> -0.0400102
>>>>>>>>>> -0.0260347
>>>>>>>>>> -0.0400102
>>>>>>>>>> -0.0400102
>>>>>>>>>> Press any key to continue . . .
>>>>>>>>>>
>>>>>>>>>> C:
>>>>>>>>>>
>>>>>>>>>> Matrix Object: 1 MPI processes
>>>>>>>>>>   type: seqaij
>>>>>>>>>> row 0: (0, 2)  (1, -1)  (3, -1)
>>>>>>>>>> row 1: (0, -1)  (1, 3)  (2, -1)  (4, -1)
>>>>>>>>>> row 2: (1, -1)  (2, 2)  (5, -1)
>>>>>>>>>> row 3: (0, -1)  (3, 3)  (4, -1)  (6, -1)
>>>>>>>>>> row 4: (1, -1)  (3, -1)  (4, 4)  (5, -1)  (7, -1)
>>>>>>>>>> row 5: (2, -1)  (4, -1)  (5, 3)  (8, -1)
>>>>>>>>>> row 6: (3, -1)  (6, 2)  (7, -1)
>>>>>>>>>> row 7: (4, -1)  (6, -1)  (7, 3)  (8, -1)
>>>>>>>>>> row 8: (5, -1)  (7, -1)  (8, 2)
>>>>>>>>>> Vector Object:Vec_0x1d3b000_0 1 MPI processes
>>>>>>>>>>   type: mpi
>>>>>>>>>> Process [0]
>>>>>>>>>> 0.25
>>>>>>>>>> 0.0205212
>>>>>>>>>> 1.135e-05
>>>>>>>>>> 0.0205212
>>>>>>>>>> 0.00168449
>>>>>>>>>> 9.31663e-07
>>>>>>>>>> 1.135e-05
>>>>>>>>>> 9.31663e-07
>>>>>>>>>> 5.15288e-10
>>>>>>>>>> Vector Object:Vec_0x1d3b000_1 1 MPI processes
>>>>>>>>>>   type: mpi
>>>>>>>>>> Process [0]
>>>>>>>>>> 0.139311
>>>>>>>>>> 0.0305751
>>>>>>>>>> -0.0220633
>>>>>>>>>> 0.0305751
>>>>>>>>>> -0.0135158
>>>>>>>>>> -0.042185
>>>>>>>>>> -0.0220633
>>>>>>>>>> -0.042185
>>>>>>>>>> -0.058449
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yours sincerely,
>>>>>>>>>>
>>>>>>>>>> TAY wee-beng
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 1/5/2012 11:54 PM, Matthew Knepley wrote:
>>>>>>>>>>
>>>>>>>>>>> On Tue, May 1, 2012 at 5:48 PM, TAY wee-beng<zonexo at gmail.com>
>>>>>>>>>>>  wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Do you mean my method is wrong?
>>>>>>>>>>>
>>>>>>>>>>> I am following the template of ex22f,
>>>>>>>>>>>
>>>>>>>>>>> where the variables are declared as :
>>>>>>>>>>>
>>>>>>>>>>> PetscScalar  v(5)
>>>>>>>>>>>
>>>>>>>>>>> MatStencil   row(4),col(4,5)
>>>>>>>>>>>
>>>>>>>>>>> Hence,
>>>>>>>>>>>
>>>>>>>>>>> for the neumann BC
>>>>>>>>>>>
>>>>>>>>>>> num = 1
>>>>>>>>>>>
>>>>>>>>>>>                if (j/=0) then
>>>>>>>>>>>
>>>>>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_j,num) = j-1
>>>>>>>>>>>
>>>>>>>>>>>                    num = num + 1
>>>>>>>>>>>
>>>>>>>>>>>                end if
>>>>>>>>>>>
>>>>>>>>>>>                if (i/=0) then
>>>>>>>>>>>
>>>>>>>>>>>                    v(num) = -rho*HydHx
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_i,num) = i-1
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>>>>>>
>>>>>>>>>>>                    num = num + 1
>>>>>>>>>>>
>>>>>>>>>>>                end if
>>>>>>>>>>>
>>>>>>>>>>>                if (i/=mx-1) then
>>>>>>>>>>>
>>>>>>>>>>>                    v(num) = -rho*HydHx
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_i,num) = i+1
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_j,num) = j
>>>>>>>>>>>
>>>>>>>>>>>                    num = num + 1
>>>>>>>>>>>
>>>>>>>>>>>                end if
>>>>>>>>>>>
>>>>>>>>>>>                if (j/=my-1) then
>>>>>>>>>>>
>>>>>>>>>>>                    v(num) = -rho*HxdHy
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_i,num) = i
>>>>>>>>>>>
>>>>>>>>>>>                    col(MatStencil_j,num) = j+1
>>>>>>>>>>>
>>>>>>>>>>>                    num = num + 1
>>>>>>>>>>>
>>>>>>>>>>>                end if
>>>>>>>>>>>
>>>>>>>>>>>                v(num) = ((num-1)/2.0)*rho*(HxdHy + HydHx)
>>>>>>>>>>>
>>>>>>>>>>>                print *, v
>>>>>>>>>>>
>>>>>>>>>>>                col(MatStencil_i,num) = i
>>>>>>>>>>>
>>>>>>>>>>>                col(MatStencil_j,num) = j
>>>>>>>>>>>
>>>>>>>>>>>                !num = num + 1
>>>>>>>>>>>
>>>>>>>>>>>                call
>>>>>>>>>>> MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>>>>>>>>
>>>>>>>>>>> I do not get any more out of range error. However,my ans is
>>>>>>>>>>> still different from that of ex29 in C.
>>>>>>>>>>>
>>>>>>>>>>> This is very simple. You have an error in your code. Checking it
>>>>>>>>>>> is very simple: run the code and
>>>>>>>>>>> break in MatSetValues(). Make sure ex29 makes calls with exactly
>>>>>>>>>>> the same indices as your ex29f.
>>>>>>>>>>>
>>>>>>>>>>>     Matt
>>>>>>>>>>>
>>>>>>>>>>> Yours sincerely,
>>>>>>>>>>>
>>>>>>>>>>> TAY wee-beng
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>
>>>>
>>>>  --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>>
>>>
>>>
>>>  --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120511/f71f8bfb/attachment-0001.htm>

From bsmith at mcs.anl.gov  Fri May 11 11:24:10 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 11 May 2012 11:24:10 -0500
Subject: [petsc-users] two-level Schwarz: PCMG or PCComposite
In-Reply-To: <BLU0-SMTP1760552E9171B2CB17CA103CB170@phx.gbl>
References: <BLU0-SMTP440ACD42EF2D3F61AC7C6F3CB170@phx.gbl>
	<45C92E5C-6AA5-470F-95CD-86BF20FDF57C@mcs.anl.gov>
	<BLU0-SMTP1760552E9171B2CB17CA103CB170@phx.gbl>
Message-ID: <34D8E416-DEF1-4DF6-8C36-F62FFFAC59A1@mcs.anl.gov>


   Yes,  src/ksp/pc/impls/composite/composite.c

On May 11, 2012, at 8:21 AM, Hui Zhang wrote:

> 
> On May 11, 2012, at 2:55 PM, Barry Smith wrote:
> 
>> 
>> If this is what you want to do then just use the PCCOMPOSITE. Use PC_COMPOSITE_SYMMETRIC_MULTIPLICATIVE
>> 
>>  Barry
> 
> Aha, I see.  Thank you, Barry!  
> Just one more question on PCCompositeAddPC:  is firstly added PC also firstly applied,
> i.e. the order of applying PC's is the same as the order of PCCompositeAddPC ?
> 
>> 
>> On May 11, 2012, at 6:26 AM, Hui Zhang wrote:
>> 
>>> I want to implement the two-level Schwarz method.  Let us denote P as the one-level
>>> Schwarz projection and P_0 as the coarse projection.  If I want the hybrid form of 
>>> preconditioned system as follows:
>>> 
>>>   (I-P_0) P (I-P_0) 
>>> 
>>> which is suitable:  PCMG or PCComposite?  It seems to me that PCComposite is good.
>>> But PCMG is not suitable because it can only implement the following form:
>>> 
>>>   (I-P) P_0 (I-P)
>>> 
>>> Any opinions?  Thanks!
>>> 
>>> 
>> 
>> 
> 


From bsmith at mcs.anl.gov  Fri May 11 11:34:16 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 11 May 2012 11:34:16 -0500
Subject: [petsc-users] Getting 0 for both start and end with
	MatGetOwnershipRange()
In-Reply-To: <4FAC0046.2010604@colorado.edu>
References: <4FAAF74E.8000502@colorado.edu>
	<CAEcYPwDjV6TDFMJc0AEpfXWxcbwFziXc3fS81w7kTtdhGEm2qg@mail.gmail.com>
	<4FAB0731.4000907@colorado.edu>
	<CAMYG4GmHW-ztFX1cwrjPRVHf3GxhCVNwVR_YysRBcNc9me7_Yw@mail.gmail.com>
	<4FAC0046.2010604@colorado.edu>
Message-ID: <D08E930F-A060-4FB3-AEC1-8E44FCAF4CCA@mcs.anl.gov>


  Dan,

     Thanks for this report. MATMAIJ is not intended to be used in the same way as the other MatTypes, I have added a nice error message so you won't see this odd 0 range error that you were getting.

    Barry

On May 10, 2012, at 12:52 PM, Daniel Christian Weflen wrote:

> I changed the type from MATSEQMAIJ to MATSEQAIJ and it works correctly. mat_size is not 0, should I still send the standalone executable?
> 
> -Dan
> 
> On 05/10/2012 05:21 AM, Matthew Knepley wrote:
>> On Wed, May 9, 2012 at 8:09 PM, Daniel Christian Weflen <Daniel.Weflen at colorado.edu> wrote:
>> I still get 0 for both the start and end rows.
>> 
>> 1) Is mat_size 0?
>> 
>> 2) If not, put these line in a standalone executable that shows the bug and send it.
>> 
>>    Thanks,
>> 
>>      Matt
>>  
>> -Dan
>> 
>> 
>> On 05/09/2012 05:35 PM, Lisandro Dalcin wrote:
>> On 9 May 2012 20:01, Daniel Christian Weflen<Daniel.Weflen at colorado.edu>  wrote:
>> I have the following code:
>> 
>>     Mat H;
>>     PetscInt mat_size=x_npts*y_npts;
>>     ierr=MatCreate(PETSC_COMM_WORLD,&H);CHKERRQ(ierr);
>> 
>> ierr=MatSetSizes(H,PETSC_DECIDE,PETSC_DECIDE,mat_size,mat_size);CHKERRQ(ierr);
>>     ierr=MatSetType(H,MATSEQMAIJ);CHKERRQ(ierr);
>>     PetscInt* nonzeros;
>>     PetscInt start_row,end_row,row_index;
>>     MatGetOwnershipRange(H,&start_row,&end_row);
>> 
>> When I check the values of start_row and end_row right after calling
>> MatGetOwnershipRange, both are 0. What am I doing wrong here? Am I calling
>> MatCreate(), MatSetSizes() and MatSetType() in the wrong order?
>> Call MatSetUp() after MatSetType().
>> 
>> 
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener


From mike.hui.zhang at hotmail.com  Fri May 11 11:52:47 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Fri, 11 May 2012 18:52:47 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
Message-ID: <BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>

I just have a question about reuse of PCASM or PCGASM.
Suppose I have seted up the PCASM and related KSP and I solved one time.
Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
different user context for the modifying routine.  

What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
even for PCASM I can re-use it because the partition of subdomains remain the same.

Thanks!


On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:

> Hui,
> There've been several changes to PCGASM ahead of the new release.
> Let me go back and see if it affected the convergence problem.
> Dmitry.
> 
> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> Hi Dmitry,
> 
> is there any news about PCGASM? 
> 
> thanks,
> Hui
> 
> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
> 
>> Okay, thanks.
>> I'll take a look.
>> 
>> Dmitry.
>> 
>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> For reference, my results are attached.
>> 
>> asm1.txt for asm with 1 process,
>> asm2.txt for asm with 2 processes,
>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>> gasm2.txt for gasm with 2 processes
>> 
>> 
>> 
>> 
>> 
>> 
>> thank you,
>> Hui
>> 
>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>> 
>>> 
>>> 
>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> 
>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>> 
>>>> 
>>>> 
>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>> each subdomain supported only by one subdomain. There are no problems when
>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>> with GASM but no problems with ASM. 
>>>> 
>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>> number is different from the first case and is much larger.  On the other hand
>>>> ASM has no such problem.
>>>>  
>>>> Are the solutions the same?
>>>> What problem are you solving?
>>> 
>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>> two processors. But GASM did not.  
>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>> I'm solving the Helmholtz equation.  Maybe 
>>> I can prepare a simpler example to show this difference.
>>> That would be helpful.  
>>> Thanks.
>>> 
>>> Dmitry. 
>>> 
>>>> 
>>>> Dmitry. 
>>>> 
>>>> 
>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>> 
>>>>> You should be able to. 
>>>>> This behavior is the same as in PCASM,
>>>>> except in GASM the matrices live on subcommunicators.
>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>> 
>>>>> Dmitry
>>>>> 
>>>>> 
>>>>> 
>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> 
>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>> 
>>>>>>> Hi Dmitry,
>>>>>>> 
>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>> 
>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>> 
>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>> 
>>>>>> As I tested, the row and col are always the same. 
>>>>>> 
>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>> in the above func()?
>>>>>> 
>>>>>> thanks,
>>>>>> Hui
>>>>>> 
>>>>>>> 
>>>>>>> thanks,
>>>>>>> Hui
>>>>>>> 
>>>>>>> 
>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>> 
>>>>>>>> Yes, that's right.
>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>> 
>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>> 
>>>>>>>> Dmitry.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>> 
>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Hui
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120511/efba433f/attachment.htm>

From karpeev at mcs.anl.gov  Fri May 11 12:17:41 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Fri, 11 May 2012 12:17:41 -0500
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
Message-ID: <CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>

You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and
recreate the matrices (including calling
your modification subroutine), but not the subdomains or the subdomain
solvers.
If you just want to modify the submatrices, you can call
PC(G)ASMGetSubmatrices() and modify the matrices it returns
(in the same order as the subdomains were set). That's a bit of a
hack, since you will essentially be modifying the PC's internal data
structures.  As long as you are careful, you should be okay, since you
already effectively have the same type of access to the submatrices through
the Modify callback.

Dmitry.

On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> I just have a question about reuse of PCASM or PCGASM.
> Suppose I have seted up the PCASM and related KSP and I solved one time.
> Next for the same linear system (matrix and RHS), I just want PCASM modify
> the submatrices (PCSetModifySubmatrices) in a different way, using the same
> routine for modifying but with
> different user context for the modifying routine.
>
> What can I do for this task?  Currently, I destroy the KSP and
> re-construct it. I guess
> even for PCASM I can re-use it because the partition of subdomains remain
> the same.
>
> Thanks!
>
>
> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>
> Hui,
> There've been several changes to PCGASM ahead of the new release.
> Let me go back and see if it affected the convergence problem.
> Dmitry.
>
> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>> Hi Dmitry,
>>
>> is there any news about PCGASM?
>>
>> thanks,
>> Hui
>>
>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>
>> Okay, thanks.
>> I'll take a look.
>>
>> Dmitry.
>>
>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>
>>> For reference, my results are attached.
>>>
>>> asm1.txt for asm with 1 process,
>>> asm2.txt for asm with 2 processes,
>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different
>>> from others)
>>> gasm2.txt for gasm with 2 processes
>>>
>>>
>>>
>>>
>>>
>>>
>>> thank you,
>>> Hui
>>>
>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>
>>>
>>>
>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>
>>>>
>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>
>>>>
>>>>
>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>>
>>>>> I have a new problem: the results from ASM and GASM are different and
>>>>> it seems
>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests
>>>>> are with
>>>>> each subdomain supported only by one subdomain. There are no problems
>>>>> when
>>>>> I did not modify submatrices.  But when I modify submatrices, there
>>>>> are problems
>>>>> with GASM but no problems with ASM.
>>>>>
>>>>> For example, I use two subdomains. In the first case each subdomain is
>>>>> supported by
>>>>> one processor and there seems no problem with GASM. But when I use run
>>>>> my program
>>>>> with only one proc. so that it supports both of the two subdomains,
>>>>> the iteration
>>>>> number is different from the first case and is much larger.  On the
>>>>> other hand
>>>>> ASM has no such problem.
>>>>>
>>>>
>>>> Are the solutions the same?
>>>> What problem are you solving?
>>>>
>>>>
>>>> Yes, the solutions are the same. That's why ASM gives the same results
>>>> with one or
>>>> two processors. But GASM did not.
>>>>
>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the
>>> case of two domains per processor?
>>>
>>>> I'm solving the Helmholtz equation.  Maybe
>>>> I can prepare a simpler example to show this difference.
>>>>
>>> That would be helpful.
>>> Thanks.
>>>
>>> Dmitry.
>>>
>>>>
>>>>
>>>> Dmitry.
>>>>
>>>>>
>>>>>
>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>
>>>>> You should be able to.
>>>>> This behavior is the same as in PCASM,
>>>>> except in GASM the matrices live on subcommunicators.
>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>
>>>>> Dmitry
>>>>>
>>>>>
>>>>>
>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com>
>>>>> wrote:
>>>>>
>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>
>>>>> Hi Dmitry,
>>>>>
>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another
>>>>> question
>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the
>>>>> prototype
>>>>>
>>>>> func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>
>>>>> I think the coloumns from the parameter 'col' are always the same as
>>>>> the rows
>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only
>>>>> accepts
>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>
>>>>>
>>>>> As I tested, the row and col are always the same.
>>>>>
>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for
>>>>> the submat's
>>>>> in the above func()?
>>>>>
>>>>> thanks,
>>>>> Hui
>>>>>
>>>>>
>>>>> thanks,
>>>>> Hui
>>>>>
>>>>>
>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>
>>>>> Yes, that's right.
>>>>> There is no good way to help the user assemble the subdomains at the
>>>>> moment beyond the 2D stuff.
>>>>> It is expected that they are generated from mesh subdomains.
>>>>> Each IS does carry the subdomains subcomm.
>>>>>
>>>>> There is ISColoringToList() that is supposed to convert a "coloring"
>>>>> of indices to an array of ISs,
>>>>> each having the indices with the same color and the subcomm that
>>>>> supports that color. It is
>>>>> largely untested, though.  You could try using it and give us feedback
>>>>> on any problems you encounter.
>>>>>
>>>>> Dmitry.
>>>>>
>>>>>
>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<mike.hui.zhang at hotmail.com>
>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>
>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain
>>>>>> supported by
>>>>>> multiple processors, shall I always create the arguments 'is[s]' and
>>>>>> 'is_local[s]'
>>>>>> in a subcommunicator consisting of processors supporting the
>>>>>> subdomain 's'?
>>>>>>
>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>
>>>>>> Thanks,
>>>>>> Hui
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120511/a85e62c4/attachment.htm>

From mike.hui.zhang at hotmail.com  Fri May 11 12:31:28 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Fri, 11 May 2012 19:31:28 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
Message-ID: <BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>

Hi Dmitry,

thanks for useful hints.  Good day!

Hui

On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:

> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
> your modification subroutine), but not the subdomains or the subdomain solvers. 
> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
> 
> Dmitry.
> 
> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> I just have a question about reuse of PCASM or PCGASM.
> Suppose I have seted up the PCASM and related KSP and I solved one time.
> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
> different user context for the modifying routine.  
> 
> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
> even for PCASM I can re-use it because the partition of subdomains remain the same.
> 
> Thanks!
> 
> 
> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
> 
>> Hui,
>> There've been several changes to PCGASM ahead of the new release.
>> Let me go back and see if it affected the convergence problem.
>> Dmitry.
>> 
>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> Hi Dmitry,
>> 
>> is there any news about PCGASM? 
>> 
>> thanks,
>> Hui
>> 
>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>> 
>>> Okay, thanks.
>>> I'll take a look.
>>> 
>>> Dmitry.
>>> 
>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> For reference, my results are attached.
>>> 
>>> asm1.txt for asm with 1 process,
>>> asm2.txt for asm with 2 processes,
>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>> gasm2.txt for gasm with 2 processes
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> thank you,
>>> Hui
>>> 
>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>> 
>>>> 
>>>> 
>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> 
>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>> 
>>>>> 
>>>>> 
>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>> with GASM but no problems with ASM. 
>>>>> 
>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>> number is different from the first case and is much larger.  On the other hand
>>>>> ASM has no such problem.
>>>>>  
>>>>> Are the solutions the same?
>>>>> What problem are you solving?
>>>> 
>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>> two processors. But GASM did not.  
>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>> I'm solving the Helmholtz equation.  Maybe 
>>>> I can prepare a simpler example to show this difference.
>>>> That would be helpful.  
>>>> Thanks.
>>>> 
>>>> Dmitry. 
>>>> 
>>>>> 
>>>>> Dmitry. 
>>>>> 
>>>>> 
>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>> 
>>>>>> You should be able to. 
>>>>>> This behavior is the same as in PCASM,
>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>> 
>>>>>> Dmitry
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> 
>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>> 
>>>>>>>> Hi Dmitry,
>>>>>>>> 
>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>> 
>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>> 
>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>> 
>>>>>>> As I tested, the row and col are always the same. 
>>>>>>> 
>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>> in the above func()?
>>>>>>> 
>>>>>>> thanks,
>>>>>>> Hui
>>>>>>> 
>>>>>>>> 
>>>>>>>> thanks,
>>>>>>>> Hui
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>> 
>>>>>>>>> Yes, that's right.
>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>> 
>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>> 
>>>>>>>>> Dmitry.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>> 
>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Hui
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120511/9eb46378/attachment-0001.htm>

From wumengda at gmail.com  Fri May 11 13:58:59 2012
From: wumengda at gmail.com (Mengda Wu)
Date: Fri, 11 May 2012 14:58:59 -0400
Subject: [petsc-users] [BUG]: error occurs when calling PetscInitialize and
 PetscFinalize multiple times
Message-ID: <CALmcJSDm1ONHwuaZedr8qDP1XKit-0Evt6_Hkr353GxcDQPsRg@mail.gmail.com>

Hello all,

   I am using Petsc-3.2-p7 complied under cygwin and using Petsc's uni-mpi
implementation. I found I cannot call PetscInitialize and PetscFinalize
multiple times.
Here is the code (between ==) that can reproduce the error. It seems that
the second PetscFinalize() generates an error at line 1221 at pinit.c
which is
 ierr = MPI_Finalize();CHKERRQ(ierr);


==============================================================
#include "petscksp.h"
#include <iostream>

int main(int argc,char *args[])
{
  PetscErrorCode ierr;
  PetscMPIInt    size;

  PetscInitialize(&argc,&args,(char *)0, PETSC_NULL);
  std::cout << "Initialize 1" << std::endl;

  ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr);
  if (size != 1) SETERRQ(PETSC_COMM_WORLD,1,"This is a uniprocessor example
only!");

  ierr = PetscFinalize();
  std::cout << "Finalize 1" << std::endl;

  PetscInitialize(&argc,&args,(char *)0, PETSC_NULL);
  std::cout << "Initialize 2" << std::endl;
  ierr = PetscFinalize();
  std::cout << "Finalize 2" << std::endl;

  PetscInitialize(&argc,&args,(char *)0, PETSC_NULL);
  std::cout << "Initialize 3" << std::endl;
  ierr = PetscFinalize();
  std::cout << "Finalize 3" << std::endl;

  PetscInitialize(&argc,&args,(char *)0, PETSC_NULL);
  std::cout << "Initialize 4" << std::endl;
  ierr = PetscFinalize();
  std::cout << "Finalize 4" << std::endl;

  return 0;
}
==============================================================

I got the following output:
Initialize 1
Finalize 1
Initialize 2
[0]PETSC ERROR: PetscFinalize() line 1221 in
src/sys/objects/D:\Library\PETSc\pe
tsc-3.2\src\sys\objects\pinit.c
Finalize 2
Initialize 3
Options have not been enabled.
You might have forgotten to call PetscInitialize().

This application has requested the Runtime to terminate it in an unusual
way.
Please contact the application's support team for more information.


Thanks,
Mengda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120511/3a09d541/attachment.htm>

From balay at mcs.anl.gov  Fri May 11 14:15:13 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 11 May 2012 14:15:13 -0500 (CDT)
Subject: [petsc-users] [BUG]: error occurs when calling PetscInitialize
 and PetscFinalize multiple times
In-Reply-To: <CALmcJSDm1ONHwuaZedr8qDP1XKit-0Evt6_Hkr353GxcDQPsRg@mail.gmail.com>
References: <CALmcJSDm1ONHwuaZedr8qDP1XKit-0Evt6_Hkr353GxcDQPsRg@mail.gmail.com>
Message-ID: <alpine.LFD.2.02.1205111408210.28809@asterix>

The scemantic for using PetscInitialize/PetscFinalize multiple times
is for the user call MPI_Init()/Finalize() separately - and only once.
[MPI standard imposes the restriction on calling MPI_Init()/Finalize()
only once].

MPIUNI is also preserving this scemantic usage aswell.

On Fri, 11 May 2012, Mengda Wu wrote:

> Hello all,
> 
>    I am using Petsc-3.2-p7 complied under cygwin and using Petsc's uni-mpi
> implementation. I found I cannot call PetscInitialize and PetscFinalize
> multiple times.
> Here is the code (between ==) that can reproduce the error. It seems that
> the second PetscFinalize() generates an error at line 1221 at pinit.c
> which is
>  ierr = MPI_Finalize();CHKERRQ(ierr);
> 
> 
> ==============================================================
> #include "petscksp.h"
> #include <iostream>
> 
> int main(int argc,char *args[])
> {
>   PetscErrorCode ierr;
>   PetscMPIInt    size;

  MPI_Init(&argc,&args);

> 
>   PetscInitialize(&argc,&args,(char *)0, PETSC_NULL);
>   std::cout << "Initialize 1" << std::endl;
> 
>   ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr);
>   if (size != 1) SETERRQ(PETSC_COMM_WORLD,1,"This is a uniprocessor example
> only!");
> 
>   ierr = PetscFinalize();
>   std::cout << "Finalize 1" << std::endl;
> 
>   PetscInitialize(&argc,&args,(char *)0, PETSC_NULL);
>   std::cout << "Initialize 2" << std::endl;
>   ierr = PetscFinalize();
>   std::cout << "Finalize 2" << std::endl;
> 
>   PetscInitialize(&argc,&args,(char *)0, PETSC_NULL);
>   std::cout << "Initialize 3" << std::endl;
>   ierr = PetscFinalize();
>   std::cout << "Finalize 3" << std::endl;
> 
>   PetscInitialize(&argc,&args,(char *)0, PETSC_NULL);
>   std::cout << "Initialize 4" << std::endl;
>   ierr = PetscFinalize();
>   std::cout << "Finalize 4" << std::endl;
> 
  MPI_Finalize();

Satish

>   return 0;
> }
> ==============================================================
> 
> I got the following output:
> Initialize 1
> Finalize 1
> Initialize 2
> [0]PETSC ERROR: PetscFinalize() line 1221 in
> src/sys/objects/D:\Library\PETSc\pe
> tsc-3.2\src\sys\objects\pinit.c
> Finalize 2
> Initialize 3
> Options have not been enabled.
> You might have forgotten to call PetscInitialize().
> 
> This application has requested the Runtime to terminate it in an unusual
> way.
> Please contact the application's support team for more information.
> 
> 
> Thanks,
> Mengda
> 


From bsmith at mcs.anl.gov  Fri May 11 14:23:34 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 11 May 2012 14:23:34 -0500
Subject: [petsc-users] [BUG]: error occurs when calling PetscInitialize
	and PetscFinalize multiple times
In-Reply-To: <CALmcJSDm1ONHwuaZedr8qDP1XKit-0Evt6_Hkr353GxcDQPsRg@mail.gmail.com>
References: <CALmcJSDm1ONHwuaZedr8qDP1XKit-0Evt6_Hkr353GxcDQPsRg@mail.gmail.com>
Message-ID: <18952C0C-39DB-4181-9259-3032BAC78367@mcs.anl.gov>


On May 11, 2012, at 1:58 PM, Mengda Wu wrote:

> Hello all,
> 
>    I am using Petsc-3.2-p7 complied under cygwin and using Petsc's uni-mpi implementation. I found I cannot call PetscInitialize and PetscFinalize multiple times.
> Here is the code (between ==) that can reproduce the error. It seems that the second PetscFinalize() generates an error at line 1221 at pinit.c 
> which is
>  ierr = MPI_Finalize();CHKERRQ(ierr);

   Actually out of the kindness of our own hearts you can call PetscInitialize() PetscFinalize() multiple times.  Can you please try this under petsc-dev (which is now frozen in preparation for our next release) and report any problems ASAP to petsc-maint at mcs.anl.gov

    Barry


Note: Since MPI_Init() can only be called once, if you wish to call PetscInitialize/PetscFinalize() multiple times with a real MPI you must call MPI_Init() ONCE before calling PetscInitialize() 

> 
> 
> ==============================================================
> #include "petscksp.h"
> #include <iostream>
> 
> int main(int argc,char *args[])
> {
>   PetscErrorCode ierr;
>   PetscMPIInt    size;
> 
>   PetscInitialize(&argc,&args,(char *)0, PETSC_NULL);
>   std::cout << "Initialize 1" << std::endl;
>   
>   ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr);
>   if (size != 1) SETERRQ(PETSC_COMM_WORLD,1,"This is a uniprocessor example only!");
> 
>   ierr = PetscFinalize();
>   std::cout << "Finalize 1" << std::endl;
> 
>   PetscInitialize(&argc,&args,(char *)0, PETSC_NULL);
>   std::cout << "Initialize 2" << std::endl;
>   ierr = PetscFinalize();
>   std::cout << "Finalize 2" << std::endl;
> 
>   PetscInitialize(&argc,&args,(char *)0, PETSC_NULL);
>   std::cout << "Initialize 3" << std::endl;
>   ierr = PetscFinalize();
>   std::cout << "Finalize 3" << std::endl;
> 
>   PetscInitialize(&argc,&args,(char *)0, PETSC_NULL);
>   std::cout << "Initialize 4" << std::endl;
>   ierr = PetscFinalize();
>   std::cout << "Finalize 4" << std::endl;
> 
>   return 0;
> }
> ==============================================================
> 
> I got the following output:
> Initialize 1
> Finalize 1
> Initialize 2
> [0]PETSC ERROR: PetscFinalize() line 1221 in src/sys/objects/D:\Library\PETSc\pe
> tsc-3.2\src\sys\objects\pinit.c
> Finalize 2
> Initialize 3
> Options have not been enabled.
> You might have forgotten to call PetscInitialize().
> 
> This application has requested the Runtime to terminate it in an unusual way.
> Please contact the application's support team for more information.
> 
> 
> Thanks,
> Mengda
> 


From dkarthik at stanford.edu  Fri May 11 15:47:08 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Fri, 11 May 2012 13:47:08 -0700 (PDT)
Subject: [petsc-users] AMG using ML : Advanced monitoring
In-Reply-To: <2103094926.2220458.1336768729644.JavaMail.root@zm09.stanford.edu>
Message-ID: <688131969.2229050.1336769228457.JavaMail.root@zm09.stanford.edu>

Dear all,

      I have been able to use AMG (using ML) for some of my problems and am looking to delve deeper into the workings of the algorithm to better understand how to use it for larger and more complex problems. Since my problems are hyperbolic and involve sharp gradients, I think there is a possibility that coarse level operators are not very stable in some problems. I see evidence of this because in many cases, I need to use lu or a very high k ilu(k) on coarser levels to keep the iterative solver from diverging.

    In this regard, I am wondering whether it is possible to obtain the coarse grid operators (or at the very least, the eigenstructure at these levels). Right now, I am doing the monitoring at the baseline level but I am looking for additional information.


Regards,
Karthik. 

-- 

=======================================
Karthik Duraisamy
Assistant Professor (Consulting)
Durand Building Rm 357
Dept of Aeronautics and Astronautics
Stanford University
Stanford CA 94305

Phone: 650-721-2835
Web:     www.stanford.edu/~dkarthik
=======================================

From knepley at gmail.com  Fri May 11 15:50:23 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 May 2012 16:50:23 -0400
Subject: [petsc-users] AMG using ML : Advanced monitoring
In-Reply-To: <688131969.2229050.1336769228457.JavaMail.root@zm09.stanford.edu>
References: <2103094926.2220458.1336768729644.JavaMail.root@zm09.stanford.edu>
	<688131969.2229050.1336769228457.JavaMail.root@zm09.stanford.edu>
Message-ID: <CAMYG4GnTqfiuAdZa8Qt32bDW05K3mm6Jdhp74fH5PmSR_RPeiQ@mail.gmail.com>

On Fri, May 11, 2012 at 4:47 PM, Karthik Duraisamy <dkarthik at stanford.edu>wrote:

> Dear all,
>
>      I have been able to use AMG (using ML) for some of my problems and am
> looking to delve deeper into the workings of the algorithm to better
> understand how to use it for larger and more complex problems. Since my
> problems are hyperbolic and involve sharp gradients, I think there is a
> possibility that coarse level operators are not very stable in some
> problems. I see evidence of this because in many cases, I need to use lu or
> a very high k ilu(k) on coarser levels to keep the iterative solver from
> diverging.
>
>    In this regard, I am wondering whether it is possible to obtain the
> coarse grid operators (or at the very least, the eigenstructure at these
> levels). Right now, I am doing the monitoring at the baseline level but I
> am looking for additional information.
>

You can get the smoother


http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCMGGetSmoother.html

This has the operator, and you can use the Krylov eigen approximation. You
should have complete control.

    Matt


> Regards,
> Karthik.
>
> --
>
> =======================================
> Karthik Duraisamy
> Assistant Professor (Consulting)
> Durand Building Rm 357
> Dept of Aeronautics and Astronautics
> Stanford University
> Stanford CA 94305
>
> Phone: 650-721-2835
> Web:     www.stanford.edu/~dkarthik
> =======================================
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120511/9f8ed363/attachment.htm>

From jedbrown at mcs.anl.gov  Fri May 11 16:00:03 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 11 May 2012 16:00:03 -0500
Subject: [petsc-users] AMG using ML : Advanced monitoring
In-Reply-To: <688131969.2229050.1336769228457.JavaMail.root@zm09.stanford.edu>
References: <2103094926.2220458.1336768729644.JavaMail.root@zm09.stanford.edu>
	<688131969.2229050.1336769228457.JavaMail.root@zm09.stanford.edu>
Message-ID: <CAM9tzSm19uSaY706_6xSrN7Tp6x8kUc2AvEdrwH6E3KTRLteag@mail.gmail.com>

On Fri, May 11, 2012 at 3:47 PM, Karthik Duraisamy <dkarthik at stanford.edu>wrote:

> Dear all,
>
>      I have been able to use AMG (using ML) for some of my problems and am
> looking to delve deeper into the workings of the algorithm to better
> understand how to use it for larger and more complex problems. Since my
> problems are hyperbolic and involve sharp gradients, I think there is a
> possibility that coarse level operators are not very stable in some
> problems. I see evidence of this because in many cases, I need to use lu or
> a very high k ilu(k) on coarser levels to keep the iterative solver from
> diverging.
>
>    In this regard, I am wondering whether it is possible to obtain the
> coarse grid operators (or at the very least, the eigenstructure at these
> levels). Right now, I am doing the monitoring at the baseline level but I
> am looking for additional information.
>

You can estimate eigenvalues with a Krylov method using (e.g. on level 4)

-mg_levels_4_ksp_type gmres -mg_levels_4_ksp_gmres_restart 300
-mg_levels_4_ksp_gmres_max_it 300 -mg_levels_4_ksp_compute_eigenvalues
-mg_levels_4_pc_type none

(use -mg_levels_4_pc_type xxx to look at the condition number of the
preconditioned operator, use -mg_levels_4_ksp_plot_eigenvalues to make a
simple graph in an X11 window). You can also access levels from code
through the PCMG interface.

We recently got news that a proposal was funded which had a part dedicated
to interactive eigenanalysis of multilevel/hierarchical solvers. That
project will enable us to also plot specific eigenvectors and to examine
more targeted contexts (e.g. the spectrum of compatible relaxation and
other "filtered" operators).

What visualization software do you use? I am planning to support plotting
eigenvectors using VisIt and ParaView, but haven't set a relative priority
yet, so it would be useful to know what potential users care most about.

Please let me know if you have ideas for useful spectral analysis
functionality, especially with respect to understanding multigrid
convergence.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120511/9aae8047/attachment.htm>

From zonexo at gmail.com  Fri May 11 16:23:15 2012
From: zonexo at gmail.com (TAY wee-beng)
Date: Fri, 11 May 2012 23:23:15 +0200
Subject: [petsc-users] Fwd: Re: Problem with fortran version of ex29 in
 ksp
In-Reply-To: <CA+zegpm5Ca=6rKMBEjQ3vJZM0hCvMyor1BF2Re27Hb_vHwCRRg@mail.gmail.com>
References: <4FA19C3F.5090604@gmail.com> <4FA3D294.5050202@gmail.com>
	<CAMYG4GmuXKfzo8MMMDU=yPefrF-jeA8STc04+D3E56foCEXiOA@mail.gmail.com>
	<4FA3F056.4090406@gmail.com>
	<CAMYG4G=xhZjQ_5x=VRT4ZEE67SP3=sf8hGzZCyJLQ8+Mig+UOA@mail.gmail.com>
	<4FA4279F.1030204@gmail.com>
	<CAMYG4G=3Vx3RzZCsS6PQ2Dk7yJj2QKN_WZOQgqWY27=iQvMpfA@mail.gmail.com>
	<5A5CB7AE-02E7-461C-BA80-4B61DAC7A3F7@mcs.anl.gov>
	<4FA42CFC.8090002@gmail.com>
	<CAMYG4G=xWRftxiGowV=Nt9SJCEx0+AsH-ZQyMKWs5wUTeSzrBg@mail.gmail.com>
	<4FA44D5C.6010205@gmail.com>
	<CAMYG4Gm9cJxm1D0UmO9Fepv93uEHYhYDCcU1rKvsTU=D_+v9qQ@mail.gmail.com>
	<4FA97132.3080103@gmail.com>
	<CA+zegpnTN40NUz+hD=BCqgv2kyz8sywzGDz+jDeVH3kXb+LSBw@mail.gmail.com>
	<4FAD12FB.6080506@gmail.com>
	<CAMYG4GmW=A9UKr0qQT7t8UTsns_ORr_UB97wUZsJtx87fibOTQ@mail.gmail.com>
	<4FAD195B.4030302@gmail.com>
	<CA+zegpm5Ca=6rKMBEjQ3vJZM0hCvMyor1BF2Re27Hb_vHwCRRg@mail.gmail.com>
Message-ID: <4FAD8343.4090306@gmail.com>


On 11/5/2012 4:04 PM, John Mousel wrote:
> Include both header files.
>
> #include <finclude/petscdmda.h>
> #include <finclude/petscdmda.h90>
>
> The h90 header only includes the interface definitions for Fortran.
>
> John
I now used:

#include <finclude/petscsys.h>
#include <finclude/petscvec.h>
#include <finclude/petscmat.h>
#include <finclude/petscpc.h>
#include <finclude/petscksp.h>
#include <finclude/petscdmda.h>
#include <finclude/petscdmda.h90>

but the same error still occurs for Linux. Works fine with VS2008 though.
>
> On Fri, May 11, 2012 at 8:51 AM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>
>
>     On 11/5/2012 3:30 PM, Matthew Knepley wrote:
>>     On Fri, May 11, 2012 at 9:24 AM, TAY wee-beng <zonexo at gmail.com
>>     <mailto:zonexo at gmail.com>> wrote:
>>
>>         Hi,
>>
>>         I have been using the GUI environment to do debugging so I am
>>         a bit reluctant to learn Valgrind as its outputs seems a bit
>>         daunting.  But I guess John is right. I've been spending
>>         these few days learning bit by bit.
>>
>>         I realised that the error occurs in computerhs, at:
>>
>>
>>     I bet this is a beautiful Fortranism. Do you include the F90
>>     header file with the interface definition?
>>     If not, Fortran just craps out like this. I can't stress enough
>>     how much time would be saved by
>>     switching languages to something with at least a modicum of error
>>     checking.
>
>     I initially used:
>
>     #include "finclude/petsc.h90"
>
>     Compilation and linking was fine in Linux and vs2008.
>
>     Now I changed it to what ex22.F was using :
>
>     #include <finclude/petscsys.h>
>     #include <finclude/petscvec.h>
>     #include <finclude/petscmat.h>
>     #include <finclude/petscpc.h>
>     #include <finclude/petscksp.h>
>     #include <finclude/petscdmda.h>
>
>     Compiling was ok but linking failed in Linux and VS2008:
>
>     undefined reference to `dmdavecgetarrayf90_'
>
>     I tried changing #include <finclude/petscdmda.h> to #include
>     <finclude/petscdmda.h90> and everything was ok in VS2008 again,
>     giving the right answers.
>
>     However, in Linux, I got the following error:
>
>     [wtay at hpc12:tutorials]$ /opt/openmpi-1.5.3/bin/mpif90 -c  -fPIC
>     -g   -I/home/wtay/Lib/petsc-3.2-dev_shared_debug/include
>     -I/home/wtay/Lib/petsc-3.2-dev_shared_debug/include
>     -I/opt/openmpi-1.5.3/include    -o ex29f.o ex29f.F90
>     /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
>     error #5082: Syntax error, found '::' when expecting one of: ( % :
>     . = =>
>             DMDABoundaryType :: pt
>     -------------------------^
>     /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
>     error #5082: Syntax error, found '::' when expecting one of: ( % :
>     . = =>
>             DMDAStencilType  :: st
>     -------------------------^
>     /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
>     error #6590: This statement is not permitted as a statement within
>     a derived-type-def
>             DMDABoundaryType :: pt
>     --------^
>     /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
>     error #6590: This statement is not permitted as a statement within
>     a derived-type-def
>             DMDAStencilType  :: st
>     --------^
>     ex29f.F90(68): error #6404: This name does not have a type, and
>     must have an explicit type.   [DMDA_BOUNDARY_NONE]
>     call
>     DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>     -----------------------------------^
>     ex29f.F90(68): error #6404: This name does not have a type, and
>     must have an explicit type.   [DMDA_STENCIL_STAR]
>     call
>     DMDACreate2d(PETSC_COMM_WORLD,DMDA_BOUNDARY_NONE,DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,i3,i3,PETSC_DECIDE,PETSC_DECIDE,i1,i1,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>     -------------------------------------------------------------------------^
>     /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
>     error #5082: Syntax error, found '::' when expecting one of: ( % :
>     . = =>
>             DMDABoundaryType :: pt
>     -------------------------^
>     /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
>     error #5082: Syntax error, found '::' when expecting one of: ( % :
>     . = =>
>             DMDAStencilType  :: st
>     -------------------------^
>     /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
>     error #6590: This statement is not permitted as a statement within
>     a derived-type-def
>             DMDABoundaryType :: pt
>     --------^
>     /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
>     error #6590: This statement is not permitted as a statement within
>     a derived-type-def
>             DMDAStencilType  :: st
>     --------^
>     /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
>     error #5082: Syntax error, found '::' when expecting one of: ( % :
>     . = =>
>             DMDABoundaryType :: pt
>     -------------------------^
>     /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
>     error #5082: Syntax error, found '::' when expecting one of: ( % :
>     . = =>
>             DMDAStencilType  :: st
>     -------------------------^
>     /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(25):
>     error #6590: This statement is not permitted as a statement within
>     a derived-type-def
>             DMDABoundaryType :: pt
>     --------^
>     /home/wtay/Lib/petsc-3.2-dev_shared_debug/include/finclude/ftn-custom/petscdmda.h90(26):
>     error #6590: This statement is not permitted as a statement within
>     a derived-type-def
>             DMDAStencilType  :: st
>
>     Is there some errors in petscdmda.h90?
>
>
>
>>
>>        Matt
>>
>>         *call DMDAVecRestoreArrayF90(da,b,array,ierr)*
>>
>>         ==27464== Invalid write of size 8
>>         ==27464==    at 0x402835: computerhs_ (ex29f.F90:119)
>>         ==27464==  Address 0xfffffffffffffef0 is not stack'd,
>>         malloc'd or (recently) free'd
>>         ==27464==
>>         [0]PETSC ERROR:
>>         ------------------------------------------------------------------------
>>         [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>>         Violation, probably memory access out of range
>>         [0]PETSC ERROR: Try option -start_in_debugger or
>>         -on_error_attach_debugger
>>         [0]PETSC ERROR: or see
>>         http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC
>>         ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>>         OS X to find memory corruption errors
>>         [0]PETSC ERROR: likely location of problem given in stack below
>>         [0]PETSC ERROR: ---------------------  Stack Frames
>>         ------------------------------------
>>         [0]PETSC ERROR: Note: The EXACT line numbers in the stack are
>>         not available,
>>         [0]PETSC ERROR:       INSTEAD the line number of the start of
>>         the function
>>         [0]PETSC ERROR:       is given.
>>         [0]PETSC ERROR: [0] DM user function line 0 unknownunknown
>>         [0]PETSC ERROR: [0] DMComputeFunction line 2085
>>         /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
>>         [0]PETSC ERROR: [0] KSPSetUp line 182
>>         /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>         [0]PETSC ERROR: --------------------- Error Message
>>         ------------------------------------
>>         [0]PETSC ERROR: Signal received!
>>
>>         I have checked that "array" 's values are correct. This
>>         statement executed without problems in VS2008. If I replace
>>         the above with something like:
>>
>>         *call VecSet(b,Hy,ierr)*
>>
>>         Everything is fine in Linux.
>>
>>         Is there something wrong with *DMDAVecRestoreArrayF90*?
>>
>>         My code in the area is:
>>
>>         call
>>         DMDAGetCorners(da,xs,ys,PETSC_NULL_INTEGER,xm,ym,PETSC_NULL_INTEGER,ierr)
>>
>>         call DMDAVecGetArrayF90(da,b,array,ierr)
>>
>>         do j = ys,ys+ym-1
>>
>>             do i = xs,xs+xm-1
>>
>>                 array(i,j) =
>>         exp(-(i*Hx)*(i*Hx)/nu)*exp(-(j*Hy)*(j*Hy)/nu)*Hx*Hy
>>
>>             end do
>>
>>         end do
>>
>>         call DMDAVecRestoreArrayF90(da,b,array,ierr)
>>
>>         call VecAssemblyBegin(b,ierr)
>>
>>         call VecAssemblyEnd(b,ierr)
>>
>>
>>         Yours sincerely,
>>
>>         TAY wee-beng
>>
>>
>>         On 8/5/2012 9:41 PM, John Mousel wrote:
>>>         TAY wee-bing,
>>>
>>>         If you want to be a programmer that writes interesting and
>>>         reliable code, you need to be willing to use the tools of
>>>         the trade. I can't think of a bigger time-saver than
>>>         Valgrind. I would suggest that you learn to use it and use
>>>         it a lot. I bet it will lead you to the root of your problem
>>>         pretty quickly.
>>>
>>>         John
>>>
>>>         On Tue, May 8, 2012 at 2:17 PM, TAY wee-beng
>>>         <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>
>>>             Hi,
>>>
>>>             I compiled and run my code under visual studio 2008 with
>>>             intel fortran. Everything works ok.
>>>
>>>             However, when I tried to run the code in linux, I got
>>>             the error as below. The error happens when
>>>             KSPSetUp(ksp,ierr) is called.
>>>
>>>             However, I am not able to print VecView or MatView to
>>>             view if there's any errors. Is there any recommendation
>>>             for debugging? I hope I do not need to valgrind if possible.
>>>
>>>             [0]PETSC ERROR:
>>>             ------------------------------------------------------------------------
>>>             [0]PETSC ERROR: Caught signal number 11 SEGV:
>>>             Segmentation Violation, probably memory access out of range
>>>             [0]PETSC ERROR: Try option -start_in_debugger or
>>>             -on_error_attach_debugger
>>>             [0]PETSC ERROR: or see
>>>             http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC
>>>             ERROR: or try http://valgrind.org on GNU/linux and Apple
>>>             Mac OS X to find memory corruption errors
>>>             [0]PETSC ERROR: likely location of problem given in
>>>             stack below
>>>             [0]PETSC ERROR: ---------------------  Stack Frames
>>>             ------------------------------------
>>>             [0]PETSC ERROR: Note: The EXACT line numbers in the
>>>             stack are not available,
>>>             [0]PETSC ERROR:       INSTEAD the line number of the
>>>             start of the function
>>>             [0]PETSC ERROR:       is given.
>>>             [0]PETSC ERROR: [0] DM user function line 0 unknownunknown
>>>             [0]PETSC ERROR: [0] DMComputeFunction line 2085
>>>             /home/wtay/Codes/petsc-dev/src/dm/interface/dm.c
>>>             [0]PETSC ERROR: [0] KSPSetUp line 182
>>>             /home/wtay/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c
>>>             [0]PETSC ERROR: --------------------- Error Message
>>>             ------------------------------------
>>>             [0]PETSC ERROR: Signal received!
>>>             [0]PETSC ERROR:
>>>             ------------------------------------------------------------------------
>>>             [0]PETSC ERROR: Petsc Development HG revision:
>>>             7ecdd63ec420b1659b960e65d96e822c5ac1a968  HG Date: Mon
>>>             May 07 21:42:26 2012 -0500
>>>             [0]PETSC ERROR: See docs/changes/index.html for recent
>>>             updates.
>>>             [0]PETSC ERROR: See docs/faq.html for hints about
>>>             trouble shooting.
>>>             [0]PETSC ERROR: See docs/index.html for manual pages.
>>>             [0]PETSC ERROR:
>>>             ------------------------------------------------------------------------
>>>             [0]PETSC ERROR: ./ex29f on a petsc-3.2 named hpc12 by
>>>             wtay Tue May  8 20:45:42 2012
>>>             [0]PETSC ERROR: Libraries linked from
>>>             /home/wtay/Lib/petsc-3.2-dev_shared_debug/lib
>>>             [0]PETSC ERROR: Configure run at Tue May  8 10:47:59 2012
>>>             [0]PETSC ERROR: Configure options
>>>             --with-mpi-dir=/opt/openmpi-1.5.3/
>>>             --with-blas-lapack-dir=/opt/intelcpro-11.1.059/mkl/lib/em64t/
>>>             --with-debugging=1 --download-hypre=1
>>>             --prefix=/home/wtay/Lib/petsc-3.2-dev_shared_debug
>>>             --known-mpi-shared=1 --with-shared-libraries
>>>             [0]PETSC ERROR:
>>>             ------------------------------------------------------------------------
>>>             [0]PETSC ERROR: User provided function() line 0 in
>>>             unknown directory unknown file
>>>             --------------------------------------------------------------------------
>>>             MPI_ABORT was invoked on rank 0 in communicator
>>>             MPI_COMM_WORLD
>>>             with errorcode 59.
>>>
>>>             NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI
>>>             processes.
>>>             You may or may not see output from other processes,
>>>             depending on
>>>             exactly when Open MPI kills them.
>>>
>>>             Yours sincerely,
>>>
>>>             TAY wee-beng
>>>
>>>
>>>             On 5/5/2012 1:43 AM, Matthew Knepley wrote:
>>>>             On Fri, May 4, 2012 at 5:42 PM, TAY wee-beng
>>>>             <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>>
>>>>                 Hi,
>>>>
>>>>                 I wonder if you people are interested to include my
>>>>                 ex29 fortran version in the petsc examples, which
>>>>                 can help people who are using fortran.
>>>>
>>>>
>>>>             Yes, we will definitely include it. Please send the
>>>>             source, and a representative output with run options.
>>>>
>>>>               Thanks,
>>>>
>>>>                  Matt
>>>>
>>>>                 Thanks.
>>>>
>>>>                 Yours sincerely,
>>>>
>>>>                 TAY wee-beng
>>>>
>>>>
>>>>                 On 4/5/2012 9:28 PM, Matthew Knepley wrote:
>>>>>                 On Fri, May 4, 2012 at 3:24 PM, TAY wee-beng
>>>>>                 <zonexo at gmail.com <mailto:zonexo at gmail.com>> wrote:
>>>>>
>>>>>
>>>>>                     On 4/5/2012 9:16 PM, Barry Smith wrote:
>>>>>
>>>>>                            Do an hg pull and then run make in
>>>>>                         src/mat/interface/ftn-custom/
>>>>>
>>>>>                             Then it should link.
>>>>>
>>>>>                            Barry
>>>>>
>>>>>                            There was a E missing from the all caps
>>>>>                         name of the function.
>>>>>
>>>>>                     After hg pull, I did:
>>>>>
>>>>>                     cd src//mat/interface/ftn-custom/
>>>>>
>>>>>                     User at User-PC
>>>>>                     /cygdrive/c/temp/petsc-dev/src/mat/interface/ftn-custom
>>>>>                     $ make
>>>>>                     make[1]: Warning: File
>>>>>                     `/cygdrive/c/temp/petsc-dev/petsc-3.2-dev_win32_vs2008/lib/libpetsc.lib(zmatregf.o)'
>>>>>                     has modification time 787 s in the future
>>>>>                     make[1]: Nothing to be done for `libc'.
>>>>>                     make[1]: warning:  Clock skew detected.  Your
>>>>>                     build may be incomplete.
>>>>>
>>>>>                     But it still can't work.
>>>>>
>>>>>
>>>>>                 Something is messed up with the clock on this machine.
>>>>>
>>>>>                 HOWEVER, development requires certain basic skills
>>>>>                 in order to debug your work. We
>>>>>                 cannot be the ones debugging your code. Now
>>>>>
>>>>>                   nm $PETSC_ARCH/lib/libpetsc.a | grep -i
>>>>>                 MatNullSpaceRemove
>>>>>
>>>>>                  will look for the symbol.
>>>>>
>>>>>                     Matt
>>>>>
>>>>>
>>>>>
>>>>>                         On May 4, 2012, at 2:11 PM, Matthew
>>>>>                         Knepley wrote:
>>>>>
>>>>>                             On Fri, May 4, 2012 at 3:01 PM, TAY
>>>>>                             wee-beng<zonexo at gmail.com
>>>>>                             <mailto:zonexo at gmail.com>>  wrote:
>>>>>
>>>>>                             On 4/5/2012 5:17 PM, Matthew Knepley
>>>>>                             wrote:
>>>>>
>>>>>                                 On Fri, May 4, 2012 at 11:05 AM,
>>>>>                                 TAY wee-beng<zonexo at gmail.com
>>>>>                                 <mailto:zonexo at gmail.com>>  wrote:
>>>>>
>>>>>                                 On 4/5/2012 3:05 PM, Matthew
>>>>>                                 Knepley wrote:
>>>>>
>>>>>                                     On Fri, May 4, 2012 at 8:59
>>>>>                                     AM, TAY
>>>>>                                     wee-beng<zonexo at gmail.com
>>>>>                                     <mailto:zonexo at gmail.com>>  wrote:
>>>>>
>>>>>                                     Hi,
>>>>>
>>>>>                                     Is there anything else I can
>>>>>                                     try to get it working right?
>>>>>
>>>>>                                     The MatGetNullSpaceRemove() is
>>>>>                                     missing.
>>>>>
>>>>>                                 Where should I add
>>>>>                                 MatGetNullSpaceRemove and what are
>>>>>                                 its syntax? I googled but there's
>>>>>                                 no results.
>>>>>
>>>>>                                 Fixed in p;etsc-dev:
>>>>>
>>>>>                                 http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatNullSpaceRemove.html
>>>>>
>>>>>                             I just compiled the updated petsc-dev
>>>>>                             but I got the same error msg when I use:
>>>>>
>>>>>                             call
>>>>>                             MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>>
>>>>>                             error LNK2019: unresolved external
>>>>>                             symbol MATNULLSPACEREMOVE referenced
>>>>>                             in function COMPUTERHS
>>>>>                             1>c:\obj_tmp\ex29f\Debug\ex29f.exe :
>>>>>                             fatal error LNK1120: 1 unresolved
>>>>>                             externals
>>>>>
>>>>>                             That function is in:
>>>>>
>>>>>                               src/mat/interface/ftn-custom/zmatrixf.c
>>>>>
>>>>>                             Did that compile? Can you see the
>>>>>                             symbol in libpetsc.a?
>>>>>
>>>>>                                 Matt
>>>>>
>>>>>                                      Matt
>>>>>
>>>>>                                 Thanks.
>>>>>
>>>>>                                        Matt
>>>>>
>>>>>                                     Thanks?
>>>>>
>>>>>
>>>>>                                     On 2/5/2012 10:11 PM, Matthew
>>>>>                                     Knepley wrote:
>>>>>
>>>>>                                         On Wed, May 2, 2012 at
>>>>>                                         1:55 PM, TAY
>>>>>                                         wee-beng<zonexo at gmail.com
>>>>>                                         <mailto:zonexo at gmail.com>>
>>>>>                                          wrote:
>>>>>                                         Hi,
>>>>>
>>>>>                                         I did a MatView and
>>>>>                                         VecView on both C and
>>>>>                                         Fortran, right after Mat
>>>>>                                         and Vec assembly. I have
>>>>>                                         attached the printout
>>>>>                                         below. They are exactly
>>>>>                                         the same, but yet the
>>>>>                                         result is different in
>>>>>                                         Neumann condition.
>>>>>                                         However, the dirichlet
>>>>>                                         condition gives the
>>>>>                                         correct ans. Is there
>>>>>                                         anything else that could
>>>>>                                         be wrong even if the Mat
>>>>>                                         and Vec are the same?
>>>>>
>>>>>                                         Did you set the null space
>>>>>                                         for the matrix when you
>>>>>                                         have Neumann conditions?
>>>>>
>>>>>                                     Yes, for the matrix, I set as:
>>>>>
>>>>>                                     call
>>>>>                                     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>>
>>>>>                                         call
>>>>>                                     MatSetNullSpace(jac,nullspace,ierr)
>>>>>
>>>>>                                         call
>>>>>                                     MatNullSpaceDestroy(nullspace,ierr)
>>>>>
>>>>>                                     for the Vec,
>>>>>
>>>>>                                     call
>>>>>                                     MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,PETSC_NULL_OBJECT,nullspace,ierr)
>>>>>
>>>>>                                         !call
>>>>>                                     MatNullSpaceRemove(nullspace,b,PETSC_NULL,ierr)
>>>>>
>>>>>                                         call
>>>>>                                     MatNullSpaceDestroy(nullspace,ierr)
>>>>>
>>>>>                                     MatNullSpaceRemove was comment
>>>>>                                     out because there's error
>>>>>                                     during linking
>>>>>
>>>>>
>>>>>                                            Matt
>>>>>
>>>>>                                         Thanks!
>>>>>
>>>>>                                         Fortran:
>>>>>
>>>>>                                         Matrix Object: 1 MPI processes
>>>>>                                           type: seqaij
>>>>>                                         row 0: (0, 2)  (1, -1)
>>>>>                                          (3, -1)
>>>>>                                         row 1: (0, -1)  (1, 3)
>>>>>                                          (2, -1)  (4, -1)
>>>>>                                         row 2: (1, -1)  (2, 2)
>>>>>                                          (5, -1)
>>>>>                                         row 3: (0, -1)  (3, 3)
>>>>>                                          (4, -1)  (6, -1)
>>>>>                                         row 4: (1, -1)  (3, -1)
>>>>>                                          (4, 4)  (5, -1)  (7, -1)
>>>>>                                         row 5: (2, -1)  (4, -1)
>>>>>                                          (5, 3)  (8, -1)
>>>>>                                         row 6: (3, -1)  (6, 2)
>>>>>                                          (7, -1)
>>>>>                                         row 7: (4, -1)  (6, -1)
>>>>>                                          (7, 3)  (8, -1)
>>>>>                                         row 8: (5, -1)  (7, -1)
>>>>>                                          (8, 2)
>>>>>                                         Vector
>>>>>                                         Object:Vec_0000000084000000_0
>>>>>                                         1 MPI processes
>>>>>                                           type: mpi
>>>>>                                         Process [0]
>>>>>                                         0.25
>>>>>                                         0.0205213
>>>>>                                         1.135e-005
>>>>>                                         0.0205213
>>>>>                                         0.00168449
>>>>>                                         9.31663e-007
>>>>>                                         1.135e-005
>>>>>                                         9.31663e-007
>>>>>                                         5.15289e-010
>>>>>                                         Vector
>>>>>                                         Object:Vec_0000000084000000_1
>>>>>                                         1 MPI processes
>>>>>                                           type: mpi
>>>>>                                         Process [0]
>>>>>                                         0.14924
>>>>>                                         0.0242397
>>>>>                                         -0.0260347
>>>>>                                         0.0242397
>>>>>                                         -0.0256192
>>>>>                                         -0.0400102
>>>>>                                         -0.0260347
>>>>>                                         -0.0400102
>>>>>                                         -0.0400102
>>>>>                                         Press any key to continue
>>>>>                                         . . .
>>>>>
>>>>>                                         C:
>>>>>
>>>>>                                         Matrix Object: 1 MPI processes
>>>>>                                           type: seqaij
>>>>>                                         row 0: (0, 2)  (1, -1)
>>>>>                                          (3, -1)
>>>>>                                         row 1: (0, -1)  (1, 3)
>>>>>                                          (2, -1)  (4, -1)
>>>>>                                         row 2: (1, -1)  (2, 2)
>>>>>                                          (5, -1)
>>>>>                                         row 3: (0, -1)  (3, 3)
>>>>>                                          (4, -1)  (6, -1)
>>>>>                                         row 4: (1, -1)  (3, -1)
>>>>>                                          (4, 4)  (5, -1)  (7, -1)
>>>>>                                         row 5: (2, -1)  (4, -1)
>>>>>                                          (5, 3)  (8, -1)
>>>>>                                         row 6: (3, -1)  (6, 2)
>>>>>                                          (7, -1)
>>>>>                                         row 7: (4, -1)  (6, -1)
>>>>>                                          (7, 3)  (8, -1)
>>>>>                                         row 8: (5, -1)  (7, -1)
>>>>>                                          (8, 2)
>>>>>                                         Vector
>>>>>                                         Object:Vec_0x1d3b000_0 1
>>>>>                                         MPI processes
>>>>>                                           type: mpi
>>>>>                                         Process [0]
>>>>>                                         0.25
>>>>>                                         0.0205212
>>>>>                                         1.135e-05
>>>>>                                         0.0205212
>>>>>                                         0.00168449
>>>>>                                         9.31663e-07
>>>>>                                         1.135e-05
>>>>>                                         9.31663e-07
>>>>>                                         5.15288e-10
>>>>>                                         Vector
>>>>>                                         Object:Vec_0x1d3b000_1 1
>>>>>                                         MPI processes
>>>>>                                           type: mpi
>>>>>                                         Process [0]
>>>>>                                         0.139311
>>>>>                                         0.0305751
>>>>>                                         -0.0220633
>>>>>                                         0.0305751
>>>>>                                         -0.0135158
>>>>>                                         -0.042185
>>>>>                                         -0.0220633
>>>>>                                         -0.042185
>>>>>                                         -0.058449
>>>>>
>>>>>
>>>>>
>>>>>                                         Yours sincerely,
>>>>>
>>>>>                                         TAY wee-beng
>>>>>
>>>>>
>>>>>                                         On 1/5/2012 11:54 PM,
>>>>>                                         Matthew Knepley wrote:
>>>>>
>>>>>                                             On Tue, May 1, 2012 at
>>>>>                                             5:48 PM, TAY
>>>>>                                             wee-beng<zonexo at gmail.com
>>>>>                                             <mailto:zonexo at gmail.com>>
>>>>>                                              wrote:
>>>>>                                             Hi,
>>>>>
>>>>>                                             Do you mean my method
>>>>>                                             is wrong?
>>>>>
>>>>>                                             I am following the
>>>>>                                             template of ex22f,
>>>>>
>>>>>                                             where the variables
>>>>>                                             are declared as :
>>>>>
>>>>>                                             PetscScalar  v(5)
>>>>>
>>>>>                                             MatStencil  
>>>>>                                             row(4),col(4,5)
>>>>>
>>>>>                                             Hence,
>>>>>
>>>>>                                             for the neumann BC
>>>>>
>>>>>                                             num = 1
>>>>>
>>>>>                                                            if
>>>>>                                             (j/=0) then
>>>>>
>>>>>                                                              
>>>>>                                              v(num) = -rho*HxdHy
>>>>>
>>>>>                                                              
>>>>>                                              col(MatStencil_i,num) = i
>>>>>
>>>>>                                                              
>>>>>                                              col(MatStencil_j,num)
>>>>>                                             = j-1
>>>>>
>>>>>                                                                num
>>>>>                                             = num + 1
>>>>>
>>>>>                                                            end if
>>>>>
>>>>>                                                            if
>>>>>                                             (i/=0) then
>>>>>
>>>>>                                                              
>>>>>                                              v(num) = -rho*HydHx
>>>>>
>>>>>                                                              
>>>>>                                              col(MatStencil_i,num)
>>>>>                                             = i-1
>>>>>
>>>>>                                                              
>>>>>                                              col(MatStencil_j,num) = j
>>>>>
>>>>>                                                                num
>>>>>                                             = num + 1
>>>>>
>>>>>                                                            end if
>>>>>
>>>>>                                                            if
>>>>>                                             (i/=mx-1) then
>>>>>
>>>>>                                                              
>>>>>                                              v(num) = -rho*HydHx
>>>>>
>>>>>                                                              
>>>>>                                              col(MatStencil_i,num)
>>>>>                                             = i+1
>>>>>
>>>>>                                                              
>>>>>                                              col(MatStencil_j,num) = j
>>>>>
>>>>>                                                                num
>>>>>                                             = num + 1
>>>>>
>>>>>                                                            end if
>>>>>
>>>>>                                                            if
>>>>>                                             (j/=my-1) then
>>>>>
>>>>>                                                              
>>>>>                                              v(num) = -rho*HxdHy
>>>>>
>>>>>                                                              
>>>>>                                              col(MatStencil_i,num) = i
>>>>>
>>>>>                                                              
>>>>>                                              col(MatStencil_j,num)
>>>>>                                             = j+1
>>>>>
>>>>>                                                                num
>>>>>                                             = num + 1
>>>>>
>>>>>                                                            end if
>>>>>
>>>>>                                                            v(num)
>>>>>                                             =
>>>>>                                             ((num-1)/2.0)*rho*(HxdHy
>>>>>                                             + HydHx)
>>>>>
>>>>>                                                            print *, v
>>>>>
>>>>>                                                          
>>>>>                                              col(MatStencil_i,num) = i
>>>>>
>>>>>                                                          
>>>>>                                              col(MatStencil_j,num) = j
>>>>>
>>>>>                                                            !num =
>>>>>                                             num + 1
>>>>>
>>>>>                                                            call
>>>>>                                             MatSetValuesStencil(jac,i1,row,num,col,v,INSERT_VALUES,ierr)
>>>>>
>>>>>                                             I do not get any more
>>>>>                                             out of range error.
>>>>>                                             However,my ans is
>>>>>                                             still different from
>>>>>                                             that of ex29 in C.
>>>>>
>>>>>                                             This is very simple.
>>>>>                                             You have an error in
>>>>>                                             your code. Checking it
>>>>>                                             is very simple: run
>>>>>                                             the code and
>>>>>                                             break in
>>>>>                                             MatSetValues(). Make
>>>>>                                             sure ex29 makes calls
>>>>>                                             with exactly the same
>>>>>                                             indices as your ex29f.
>>>>>
>>>>>                                                 Matt
>>>>>
>>>>>                                             Yours sincerely,
>>>>>
>>>>>                                             TAY wee-beng
>>>>>
>>>>>                                             -- 
>>>>>                                             What most
>>>>>                                             experimenters take for
>>>>>                                             granted before they
>>>>>                                             begin their
>>>>>                                             experiments is
>>>>>                                             infinitely more
>>>>>                                             interesting than any
>>>>>                                             results to which their
>>>>>                                             experiments lead.
>>>>>                                             -- Norbert Wiener
>>>>>
>>>>>
>>>>>
>>>>>                                         -- 
>>>>>                                         What most experimenters
>>>>>                                         take for granted before
>>>>>                                         they begin their
>>>>>                                         experiments is infinitely
>>>>>                                         more interesting than any
>>>>>                                         results to which their
>>>>>                                         experiments lead.
>>>>>                                         -- Norbert Wiener
>>>>>
>>>>>
>>>>>
>>>>>                                     -- 
>>>>>                                     What most experimenters take
>>>>>                                     for granted before they begin
>>>>>                                     their experiments is
>>>>>                                     infinitely more interesting
>>>>>                                     than any results to which
>>>>>                                     their experiments lead.
>>>>>                                     -- Norbert Wiener
>>>>>
>>>>>
>>>>>
>>>>>                                 -- 
>>>>>                                 What most experimenters take for
>>>>>                                 granted before they begin their
>>>>>                                 experiments is infinitely more
>>>>>                                 interesting than any results to
>>>>>                                 which their experiments lead.
>>>>>                                 -- Norbert Wiener
>>>>>
>>>>>
>>>>>
>>>>>                             -- 
>>>>>                             What most experimenters take for
>>>>>                             granted before they begin their
>>>>>                             experiments is infinitely more
>>>>>                             interesting than any results to which
>>>>>                             their experiments lead.
>>>>>                             -- Norbert Wiener
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                 -- 
>>>>>                 What most experimenters take for granted before
>>>>>                 they begin their experiments is infinitely more
>>>>>                 interesting than any results to which their
>>>>>                 experiments lead.
>>>>>                 -- Norbert Wiener
>>>>
>>>>
>>>>
>>>>
>>>>             -- 
>>>>             What most experimenters take for granted before they
>>>>             begin their experiments is infinitely more interesting
>>>>             than any results to which their experiments lead.
>>>>             -- Norbert Wiener
>>>
>>>
>>
>>
>>
>>     -- 
>>     What most experimenters take for granted before they begin their
>>     experiments is infinitely more interesting than any results to
>>     which their experiments lead.
>>     -- Norbert Wiener
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120511/fa31b535/attachment-0001.htm>

From dkarthik at stanford.edu  Fri May 11 17:30:36 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Fri, 11 May 2012 15:30:36 -0700 (PDT)
Subject: [petsc-users] AMG using ML : Advanced monitoring
In-Reply-To: <CAM9tzSm19uSaY706_6xSrN7Tp6x8kUc2AvEdrwH6E3KTRLteag@mail.gmail.com>
Message-ID: <1409498370.2382856.1336775436071.JavaMail.root@zm09.stanford.edu>

Excellent. This is what I was looking for.

My original aim was to use information from the eigenspectrum of the coarse levels to make some decisions on choosing appropriate smoothers (ilu levels, etc). Alternately, problematic eigenmodes can be isolated and operated on, but I haven't gotten to doing any of this. I will keep you informed about any progress.

I use tecplot for visualization as do many people in my field. 

----- Original Message -----
From: "Jed Brown" <jedbrown at mcs.anl.gov>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Friday, May 11, 2012 2:00:03 PM
Subject: Re: [petsc-users] AMG using ML : Advanced monitoring


On Fri, May 11, 2012 at 3:47 PM, Karthik Duraisamy < dkarthik at stanford.edu > wrote: 


Dear all, 

I have been able to use AMG (using ML) for some of my problems and am looking to delve deeper into the workings of the algorithm to better understand how to use it for larger and more complex problems. Since my problems are hyperbolic and involve sharp gradients, I think there is a possibility that coarse level operators are not very stable in some problems. I see evidence of this because in many cases, I need to use lu or a very high k ilu(k) on coarser levels to keep the iterative solver from diverging. 

In this regard, I am wondering whether it is possible to obtain the coarse grid operators (or at the very least, the eigenstructure at these levels). Right now, I am doing the monitoring at the baseline level but I am looking for additional information. 



You can estimate eigenvalues with a Krylov method using (e.g. on level 4) 


-mg_levels_4_ksp_type gmres -mg_levels_4_ksp_gmres_restart 300 -mg_levels_4_ksp_gmres_max_it 300 -mg_levels_4_ksp_compute_eigenvalues -mg_levels_4_pc_type none 


(use -mg_levels_4_pc_type xxx to look at the condition number of the preconditioned operator, use -mg_levels_4_ksp_plot_eigenvalues to make a simple graph in an X11 window). You can also access levels from code through the PCMG interface. 


We recently got news that a proposal was funded which had a part dedicated to interactive eigenanalysis of multilevel/hierarchical solvers. That project will enable us to also plot specific eigenvectors and to examine more targeted contexts (e.g. the spectrum of compatible relaxation and other "filtered" operators). 


What visualization software do you use? I am planning to support plotting eigenvectors using VisIt and ParaView, but haven't set a relative priority yet, so it would be useful to know what potential users care most about. 


Please let me know if you have ideas for useful spectral analysis functionality, especially with respect to understanding multigrid convergence. 


From dkarthik at stanford.edu  Fri May 11 19:20:29 2012
From: dkarthik at stanford.edu (Karthik Duraisamy)
Date: Fri, 11 May 2012 17:20:29 -0700 (PDT)
Subject: [petsc-users] AMG using ML : Advanced monitoring
In-Reply-To: <CAM9tzSm19uSaY706_6xSrN7Tp6x8kUc2AvEdrwH6E3KTRLteag@mail.gmail.com>
Message-ID: <625445805.2500752.1336782029232.JavaMail.root@zm09.stanford.edu>



-mg_levels_4_ksp_type gmres -mg_levels_4_ksp_gmres_restart 300 -mg_levels_4_ksp_gmres_max_it 300 -mg_levels_4_ksp_compute_eigenvalues -mg_levels_4_pc_type none 


Quick followup question - When solving Ax = b, I have an outer loop that keeps feeding the same A matrix to the KSP solver, but with different initial guesses for x. When doing this, I get slightly different eigenvalue estimates every time (minimum real part off by 10%). I just want to confirm that this is because of the iterative process.

Thanks

From knepley at gmail.com  Fri May 11 19:48:25 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 11 May 2012 20:48:25 -0400
Subject: [petsc-users] AMG using ML : Advanced monitoring
In-Reply-To: <625445805.2500752.1336782029232.JavaMail.root@zm09.stanford.edu>
References: <CAM9tzSm19uSaY706_6xSrN7Tp6x8kUc2AvEdrwH6E3KTRLteag@mail.gmail.com>
	<625445805.2500752.1336782029232.JavaMail.root@zm09.stanford.edu>
Message-ID: <CAMYG4GmaHVTTZ8r_o8JUK9ckjNAiOg7rfy-A=3pLz_0V8PoSkA@mail.gmail.com>

On Fri, May 11, 2012 at 8:20 PM, Karthik Duraisamy <dkarthik at stanford.edu>wrote:

>
>
> -mg_levels_4_ksp_type gmres -mg_levels_4_ksp_gmres_restart 300
> -mg_levels_4_ksp_gmres_max_it 300 -mg_levels_4_ksp_compute_eigenvalues
> -mg_levels_4_pc_type none
>
>
> Quick followup question - When solving Ax = b, I have an outer loop that
> keeps feeding the same A matrix to the KSP solver, but with different
> initial guesses for x. When doing this, I get slightly different eigenvalue
> estimates every time (minimum real part off by 10%). I just want to confirm
> that this is because of the iterative process.
>

Yes. A different initial residual means a different Krylov space

   Matt


> Thanks
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120511/0b528e1b/attachment.htm>

From mirzadeh at gmail.com  Sat May 12 03:33:11 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Sat, 12 May 2012 01:33:11 -0700
Subject: [petsc-users] seg fault with VecGetArray
Message-ID: <CAKsPKw=Psi4Uc0bOxJ94Jwsa2x2kMe4rqnAk_PS5Mcajem=CZw@mail.gmail.com>

Hi guys,

I'm having a really weird issue here! My code seg faults for certain
problem size and after using gdb I have been able to pinpoint the problem
to a VecGetArray call. Here's a series of things I have tried so far

1) -on_error_attach_debugger -----> unsuccessful; does not launch debugger
2) -start_in_debugger -------> unsuccessful; does not start debugger
3) attaching debugger myself -----> code runs in debugger and seg faults
when calling VecGetArray
4) using ierr=VecGetArray;CHKERRQ(ierr) ------> PETSc does not produce
error messages; the code simply seg faults and terminates
5) checking the values of ierr inside the debugger ---------> They are all
0 up untill the code terminates; I think this means petsc does not generate
error?
6) checking for memory leak with valgrind -----------> All I get are leaks
from OpenMPI and PetscInitialize and PetscFinalize; I think these are
just routine and safe?

What else can I try to find the problem? Any recommendation is really
appreciated!

Thanks,
Mohammad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120512/f245629f/attachment.htm>

From mirzadeh at gmail.com  Sat May 12 03:37:04 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Sat, 12 May 2012 01:37:04 -0700
Subject: [petsc-users] seg fault with VecGetArray
In-Reply-To: <CAKsPKw=Psi4Uc0bOxJ94Jwsa2x2kMe4rqnAk_PS5Mcajem=CZw@mail.gmail.com>
References: <CAKsPKw=Psi4Uc0bOxJ94Jwsa2x2kMe4rqnAk_PS5Mcajem=CZw@mail.gmail.com>
Message-ID: <CAKsPKwk_tqurOPvT+xvffukxJPiOD47jynK-rzCcXDuJi+oJmQ@mail.gmail.com>

Also, if it matters, this is in serial. petsc version is petsc-dev and is
compiled with g++ and OpenMPI. Also, OpenMPI is NOT installed by PETSc
configure.

On Sat, May 12, 2012 at 1:33 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Hi guys,
>
> I'm having a really weird issue here! My code seg faults for certain
> problem size and after using gdb I have been able to pinpoint the problem
> to a VecGetArray call. Here's a series of things I have tried so far
>
> 1) -on_error_attach_debugger -----> unsuccessful; does not launch debugger
> 2) -start_in_debugger -------> unsuccessful; does not start debugger
> 3) attaching debugger myself -----> code runs in debugger and seg faults
> when calling VecGetArray
> 4) using ierr=VecGetArray;CHKERRQ(ierr) ------> PETSc does not produce
> error messages; the code simply seg faults and terminates
> 5) checking the values of ierr inside the debugger ---------> They are all
> 0 up untill the code terminates; I think this means petsc does not generate
> error?
> 6) checking for memory leak with valgrind -----------> All I get are leaks
> from OpenMPI and PetscInitialize and PetscFinalize; I think these are
> just routine and safe?
>
> What else can I try to find the problem? Any recommendation is really
> appreciated!
>
> Thanks,
> Mohammad
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120512/a22744fa/attachment.htm>

From knepley at gmail.com  Sat May 12 05:16:04 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 12 May 2012 06:16:04 -0400
Subject: [petsc-users] seg fault with VecGetArray
In-Reply-To: <CAKsPKw=Psi4Uc0bOxJ94Jwsa2x2kMe4rqnAk_PS5Mcajem=CZw@mail.gmail.com>
References: <CAKsPKw=Psi4Uc0bOxJ94Jwsa2x2kMe4rqnAk_PS5Mcajem=CZw@mail.gmail.com>
Message-ID: <CAMYG4GndpeQ2EHYDLjCS_bokkqjYc0MH5vFYRetBUa-ceGmu2g@mail.gmail.com>

On Sat, May 12, 2012 at 4:33 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Hi guys,
>
> I'm having a really weird issue here! My code seg faults for certain
> problem size and after using gdb I have been able to pinpoint the problem
> to a VecGetArray call. Here's a series of things I have tried so far
>
> 1) -on_error_attach_debugger -----> unsuccessful; does not launch debugger
> 2) -start_in_debugger -------> unsuccessful; does not start debugger
>

Run with -log_summary. It will tell you what options the program got. Also,
are there errors relating to X? Send
all output to petsc-maint at mcs.anl.gov


> 3) attaching debugger myself -----> code runs in debugger and seg faults
> when calling VecGetArray
>

Is this a debug build? What dereference is causing the SEGV? Is the Vec a
valid object? It sounds like
it has been corrupted.


> 4) using ierr=VecGetArray;CHKERRQ(ierr) ------> PETSc does not produce
> error messages; the code simply seg faults and terminates
> 5) checking the values of ierr inside the debugger ---------> They are all
> 0 up untill the code terminates; I think this means petsc does not generate
> error?
> 6) checking for memory leak with valgrind -----------> All I get are leaks
> from OpenMPI and PetscInitialize and PetscFinalize; I think these are
> just routine and safe?
>

How can we say anything without the valgrind output?

    Matt


>
> What else can I try to find the problem? Any recommendation is really
> appreciated!
>
> Thanks,
> Mohammad
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120512/c0fbe9c3/attachment.htm>

From mirzadeh at gmail.com  Sun May 13 18:33:57 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Sun, 13 May 2012 16:33:57 -0700
Subject: [petsc-users] seg fault with VecGetArray
In-Reply-To: <CAMYG4GndpeQ2EHYDLjCS_bokkqjYc0MH5vFYRetBUa-ceGmu2g@mail.gmail.com>
References: <CAKsPKw=Psi4Uc0bOxJ94Jwsa2x2kMe4rqnAk_PS5Mcajem=CZw@mail.gmail.com>
	<CAMYG4GndpeQ2EHYDLjCS_bokkqjYc0MH5vFYRetBUa-ceGmu2g@mail.gmail.com>
Message-ID: <CAKsPKwn6QLPYTHEYhz9T52qnEdxj0MdW3VVUz0bSBA2zq=2+kw@mail.gmail.com>

On Sat, May 12, 2012 at 3:16 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, May 12, 2012 at 4:33 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> Hi guys,
>>
>> I'm having a really weird issue here! My code seg faults for certain
>> problem size and after using gdb I have been able to pinpoint the problem
>> to a VecGetArray call. Here's a series of things I have tried so far
>>
>> 1) -on_error_attach_debugger -----> unsuccessful; does not launch
>> debugger
>> 2) -start_in_debugger -------> unsuccessful; does not start debugger
>>
>
> Run with -log_summary. It will tell you what options the program got.
> Also, are there errors relating to X? Send
> all output to petsc-maint at mcs.anl.gov
>
>

Matt, -log_summary also does not generate any output! I was eventually able
to start_in_debugger using xterm. Previously I was trying to start in kdbg.
Even with xterm, -on_error_attach_debugger does not start the debugger. In
either case, starting the debugger in xterm using -start_in_debugger or
attaching the debugger myself manually, I get a segfault at VecGetArray and
then the program terminates without any further output.


> 3) attaching debugger myself -----> code runs in debugger and seg faults
>> when calling VecGetArray
>>
>
> Is this a debug build? What dereference is causing the SEGV? Is the Vec a
> valid object? It sounds like
> it has been corrupted.
>
>

Yes; with the -g option. How can I check if Vec is "valid"?

 4) using ierr=VecGetArray;CHKERRQ(ierr) ------> PETSc does not produce
>> error messages; the code simply seg faults and terminates
>> 5) checking the values of ierr inside the debugger ---------> They are
>> all 0 up untill the code terminates; I think this means petsc does not
>> generate error?
>> 6) checking for memory leak with valgrind -----------> All I get are
>> leaks from OpenMPI and PetscInitialize and PetscFinalize; I think these are
>> just routine and safe?
>>
>
>
Should I attach the whole valgrind output here or send it to petsc-maint? I
just repeast these two a couple of times!:

==4508== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
lost in loss record 2,644 of 2,665
==4508==    at 0x4C2815C: malloc (vg_replace_malloc.c:236)
==4508==    by 0x86417ED: ???
==4508==    by 0x5D4D099: orte_rml_base_comm_start (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==4508==    by 0x8640AD1: ???
==4508==    by 0x5D3AFE6: orte_ess_base_app_setup (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==4508==    by 0x8846E41: ???
==4508==    by 0x5D23A52: orte_init (in
/usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
==4508==    by 0x5A9E806: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.1)
==4508==    by 0x5ABFD7F: PMPI_Init (in
/usr/lib/openmpi/lib/libmpi.so.0.0.1)
==4508==    by 0x530A90: PetscInitialize(int*, char***, char const*, char
const*) (pinit.c:668)
==4508==    by 0x4A4955: PetscSession::PetscSession(int*, char***, char
const*, char const*) (utilities.h:17)
==4508==    by 0x4A1DA5: main (main_Test2.cpp:49)

==4508== 74 bytes in 1 blocks are definitely lost in loss record 2,411 of
2,665
==4508==    at 0x4C2815C: malloc (vg_replace_malloc.c:236)
==4508==    by 0x6F2DDA1: strdup (strdup.c:43)
==4508==    by 0x5F85117: ??? (in /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==4508==    by 0x5F85359: mca_base_param_lookup_string (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==4508==    by 0xB301869: ???
==4508==    by 0xB2F5126: ???
==4508==    by 0x5F82E17: mca_base_components_open (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==4508==    by 0x5ADA6BA: mca_btl_base_open (in
/usr/lib/openmpi/lib/libmpi.so.0.0.1)
==4508==    by 0xA6A9B93: ???
==4508==    by 0x5F82E17: mca_base_components_open (in
/usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
==4508==    by 0x5AE3C88: mca_pml_base_open (in
/usr/lib/openmpi/lib/libmpi.so.0.0.1)
==4508==    by 0x5A9E9E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.1)


but eventually I get:

==4508== LEAK SUMMARY:
==4508==    definitely lost: 5,949 bytes in 55 blocks
==4508==    indirectly lost: 3,562 bytes in 32 blocks
==4508==      possibly lost: 0 bytes in 0 blocks
==4508==    still reachable: 181,516 bytes in 2,660 blocks
==4508==         suppressed: 0 bytes in 0 blocks
==4508== Reachable blocks (those to which a pointer was found) are not
shown.
==4508== To see them, rerun with: --leak-check=full --show-reachable=y

which seems considerable!


>  How can we say anything without the valgrind output?
>
>     Matt
>
>
>>
>> What else can I try to find the problem? Any recommendation is really
>> appreciated!
>>
>> Thanks,
>> Mohammad
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120513/afd9e11a/attachment.htm>

From mirzadeh at gmail.com  Sun May 13 18:39:34 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Sun, 13 May 2012 16:39:34 -0700
Subject: [petsc-users] seg fault with VecGetArray
In-Reply-To: <CAKsPKwn6QLPYTHEYhz9T52qnEdxj0MdW3VVUz0bSBA2zq=2+kw@mail.gmail.com>
References: <CAKsPKw=Psi4Uc0bOxJ94Jwsa2x2kMe4rqnAk_PS5Mcajem=CZw@mail.gmail.com>
	<CAMYG4GndpeQ2EHYDLjCS_bokkqjYc0MH5vFYRetBUa-ceGmu2g@mail.gmail.com>
	<CAKsPKwn6QLPYTHEYhz9T52qnEdxj0MdW3VVUz0bSBA2zq=2+kw@mail.gmail.com>
Message-ID: <CAKsPKw=Lpzj7VEqe4sLba2F55vj1iB7_cUa64eKJqTVb-b-PLw@mail.gmail.com>

on  a somewhat related topic, I recently found I have the CHKERRXX option
which throws exceptions. This seems to be a better option for me since in
some some functions with return values I cannot use CHKERRQ/V. If I use
this, do I need to catch the exception in myself or is it caught by petsc?

On Sun, May 13, 2012 at 4:33 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

>
>
> On Sat, May 12, 2012 at 3:16 AM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Sat, May 12, 2012 at 4:33 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>
>>> Hi guys,
>>>
>>> I'm having a really weird issue here! My code seg faults for certain
>>> problem size and after using gdb I have been able to pinpoint the problem
>>> to a VecGetArray call. Here's a series of things I have tried so far
>>>
>>> 1) -on_error_attach_debugger -----> unsuccessful; does not launch
>>> debugger
>>> 2) -start_in_debugger -------> unsuccessful; does not start debugger
>>>
>>
>> Run with -log_summary. It will tell you what options the program got.
>> Also, are there errors relating to X? Send
>> all output to petsc-maint at mcs.anl.gov
>>
>>
>
> Matt, -log_summary also does not generate any output! I was eventually
> able to start_in_debugger using xterm. Previously I was trying to start in
> kdbg. Even with xterm, -on_error_attach_debugger does not start the
> debugger. In either case, starting the debugger in xterm using
> -start_in_debugger or attaching the debugger myself manually, I get a
> segfault at VecGetArray and then the program terminates without any further
> output.
>
>
>> 3) attaching debugger myself -----> code runs in debugger and seg faults
>>> when calling VecGetArray
>>>
>>
>> Is this a debug build? What dereference is causing the SEGV? Is the Vec a
>> valid object? It sounds like
>> it has been corrupted.
>>
>>
>
> Yes; with the -g option. How can I check if Vec is "valid"?
>
>  4) using ierr=VecGetArray;CHKERRQ(ierr) ------> PETSc does not produce
>>> error messages; the code simply seg faults and terminates
>>> 5) checking the values of ierr inside the debugger ---------> They are
>>> all 0 up untill the code terminates; I think this means petsc does not
>>> generate error?
>>> 6) checking for memory leak with valgrind -----------> All I get are
>>> leaks from OpenMPI and PetscInitialize and PetscFinalize; I think these are
>>> just routine and safe?
>>>
>>
>>
> Should I attach the whole valgrind output here or send it to petsc-maint?
> I just repeast these two a couple of times!:
>
> ==4508== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
> lost in loss record 2,644 of 2,665
> ==4508==    at 0x4C2815C: malloc (vg_replace_malloc.c:236)
> ==4508==    by 0x86417ED: ???
> ==4508==    by 0x5D4D099: orte_rml_base_comm_start (in
> /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
> ==4508==    by 0x8640AD1: ???
> ==4508==    by 0x5D3AFE6: orte_ess_base_app_setup (in
> /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
> ==4508==    by 0x8846E41: ???
> ==4508==    by 0x5D23A52: orte_init (in
> /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
> ==4508==    by 0x5A9E806: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.1)
> ==4508==    by 0x5ABFD7F: PMPI_Init (in
> /usr/lib/openmpi/lib/libmpi.so.0.0.1)
> ==4508==    by 0x530A90: PetscInitialize(int*, char***, char const*, char
> const*) (pinit.c:668)
> ==4508==    by 0x4A4955: PetscSession::PetscSession(int*, char***, char
> const*, char const*) (utilities.h:17)
> ==4508==    by 0x4A1DA5: main (main_Test2.cpp:49)
>
> ==4508== 74 bytes in 1 blocks are definitely lost in loss record 2,411 of
> 2,665
> ==4508==    at 0x4C2815C: malloc (vg_replace_malloc.c:236)
> ==4508==    by 0x6F2DDA1: strdup (strdup.c:43)
> ==4508==    by 0x5F85117: ??? (in
> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
> ==4508==    by 0x5F85359: mca_base_param_lookup_string (in
> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
> ==4508==    by 0xB301869: ???
> ==4508==    by 0xB2F5126: ???
> ==4508==    by 0x5F82E17: mca_base_components_open (in
> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
> ==4508==    by 0x5ADA6BA: mca_btl_base_open (in
> /usr/lib/openmpi/lib/libmpi.so.0.0.1)
> ==4508==    by 0xA6A9B93: ???
> ==4508==    by 0x5F82E17: mca_base_components_open (in
> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
> ==4508==    by 0x5AE3C88: mca_pml_base_open (in
> /usr/lib/openmpi/lib/libmpi.so.0.0.1)
> ==4508==    by 0x5A9E9E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.1)
>
>
> but eventually I get:
>
> ==4508== LEAK SUMMARY:
> ==4508==    definitely lost: 5,949 bytes in 55 blocks
> ==4508==    indirectly lost: 3,562 bytes in 32 blocks
> ==4508==      possibly lost: 0 bytes in 0 blocks
> ==4508==    still reachable: 181,516 bytes in 2,660 blocks
> ==4508==         suppressed: 0 bytes in 0 blocks
> ==4508== Reachable blocks (those to which a pointer was found) are not
> shown.
> ==4508== To see them, rerun with: --leak-check=full --show-reachable=y
>
> which seems considerable!
>
>
>>  How can we say anything without the valgrind output?
>>
>>     Matt
>>
>>
>>>
>>> What else can I try to find the problem? Any recommendation is really
>>> appreciated!
>>>
>>> Thanks,
>>> Mohammad
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120513/0667b8ad/attachment.htm>

From knepley at gmail.com  Sun May 13 19:27:21 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 13 May 2012 20:27:21 -0400
Subject: [petsc-users] seg fault with VecGetArray
In-Reply-To: <CAKsPKw=Lpzj7VEqe4sLba2F55vj1iB7_cUa64eKJqTVb-b-PLw@mail.gmail.com>
References: <CAKsPKw=Psi4Uc0bOxJ94Jwsa2x2kMe4rqnAk_PS5Mcajem=CZw@mail.gmail.com>
	<CAMYG4GndpeQ2EHYDLjCS_bokkqjYc0MH5vFYRetBUa-ceGmu2g@mail.gmail.com>
	<CAKsPKwn6QLPYTHEYhz9T52qnEdxj0MdW3VVUz0bSBA2zq=2+kw@mail.gmail.com>
	<CAKsPKw=Lpzj7VEqe4sLba2F55vj1iB7_cUa64eKJqTVb-b-PLw@mail.gmail.com>
Message-ID: <CAMYG4Gm9zB9kKp_c4BghFf6cWOdyf5zvV4vtfUVc18rngQ9Xhw@mail.gmail.com>

On Sun, May 13, 2012 at 7:39 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> on  a somewhat related topic, I recently found I have the CHKERRXX option
> which throws exceptions. This seems to be a better option for me since in
> some some functions with return values I cannot use CHKERRQ/V. If I use
> this, do I need to catch the exception in myself or is it caught by petsc?
>

You catch the exception.

   Matt


> On Sun, May 13, 2012 at 4:33 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>>
>>
>> On Sat, May 12, 2012 at 3:16 AM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Sat, May 12, 2012 at 4:33 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I'm having a really weird issue here! My code seg faults for certain
>>>> problem size and after using gdb I have been able to pinpoint the problem
>>>> to a VecGetArray call. Here's a series of things I have tried so far
>>>>
>>>> 1) -on_error_attach_debugger -----> unsuccessful; does not launch
>>>> debugger
>>>> 2) -start_in_debugger -------> unsuccessful; does not start debugger
>>>>
>>>
>>> Run with -log_summary. It will tell you what options the program got.
>>> Also, are there errors relating to X? Send
>>> all output to petsc-maint at mcs.anl.gov
>>>
>>>
>>
>> Matt, -log_summary also does not generate any output! I was eventually
>> able to start_in_debugger using xterm. Previously I was trying to start in
>> kdbg. Even with xterm, -on_error_attach_debugger does not start the
>> debugger. In either case, starting the debugger in xterm using
>> -start_in_debugger or attaching the debugger myself manually, I get a
>> segfault at VecGetArray and then the program terminates without any further
>> output.
>>
>>
>>> 3) attaching debugger myself -----> code runs in debugger and seg faults
>>>> when calling VecGetArray
>>>>
>>>
>>> Is this a debug build? What dereference is causing the SEGV? Is the Vec
>>> a valid object? It sounds like
>>> it has been corrupted.
>>>
>>>
>>
>> Yes; with the -g option. How can I check if Vec is "valid"?
>>
>>  4) using ierr=VecGetArray;CHKERRQ(ierr) ------> PETSc does not produce
>>>> error messages; the code simply seg faults and terminates
>>>> 5) checking the values of ierr inside the debugger ---------> They are
>>>> all 0 up untill the code terminates; I think this means petsc does not
>>>> generate error?
>>>> 6) checking for memory leak with valgrind -----------> All I get are
>>>> leaks from OpenMPI and PetscInitialize and PetscFinalize; I think these are
>>>> just routine and safe?
>>>>
>>>
>>>
>> Should I attach the whole valgrind output here or send it to petsc-maint?
>> I just repeast these two a couple of times!:
>>
>> ==4508== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
>> lost in loss record 2,644 of 2,665
>> ==4508==    at 0x4C2815C: malloc (vg_replace_malloc.c:236)
>> ==4508==    by 0x86417ED: ???
>> ==4508==    by 0x5D4D099: orte_rml_base_comm_start (in
>> /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
>> ==4508==    by 0x8640AD1: ???
>> ==4508==    by 0x5D3AFE6: orte_ess_base_app_setup (in
>> /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
>> ==4508==    by 0x8846E41: ???
>> ==4508==    by 0x5D23A52: orte_init (in
>> /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
>> ==4508==    by 0x5A9E806: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.1)
>> ==4508==    by 0x5ABFD7F: PMPI_Init (in
>> /usr/lib/openmpi/lib/libmpi.so.0.0.1)
>> ==4508==    by 0x530A90: PetscInitialize(int*, char***, char const*, char
>> const*) (pinit.c:668)
>> ==4508==    by 0x4A4955: PetscSession::PetscSession(int*, char***, char
>> const*, char const*) (utilities.h:17)
>> ==4508==    by 0x4A1DA5: main (main_Test2.cpp:49)
>>
>> ==4508== 74 bytes in 1 blocks are definitely lost in loss record 2,411 of
>> 2,665
>> ==4508==    at 0x4C2815C: malloc (vg_replace_malloc.c:236)
>> ==4508==    by 0x6F2DDA1: strdup (strdup.c:43)
>> ==4508==    by 0x5F85117: ??? (in
>> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
>> ==4508==    by 0x5F85359: mca_base_param_lookup_string (in
>> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
>> ==4508==    by 0xB301869: ???
>> ==4508==    by 0xB2F5126: ???
>> ==4508==    by 0x5F82E17: mca_base_components_open (in
>> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
>> ==4508==    by 0x5ADA6BA: mca_btl_base_open (in
>> /usr/lib/openmpi/lib/libmpi.so.0.0.1)
>> ==4508==    by 0xA6A9B93: ???
>> ==4508==    by 0x5F82E17: mca_base_components_open (in
>> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
>> ==4508==    by 0x5AE3C88: mca_pml_base_open (in
>> /usr/lib/openmpi/lib/libmpi.so.0.0.1)
>> ==4508==    by 0x5A9E9E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.1)
>>
>>
>> but eventually I get:
>>
>> ==4508== LEAK SUMMARY:
>> ==4508==    definitely lost: 5,949 bytes in 55 blocks
>> ==4508==    indirectly lost: 3,562 bytes in 32 blocks
>> ==4508==      possibly lost: 0 bytes in 0 blocks
>> ==4508==    still reachable: 181,516 bytes in 2,660 blocks
>> ==4508==         suppressed: 0 bytes in 0 blocks
>> ==4508== Reachable blocks (those to which a pointer was found) are not
>> shown.
>> ==4508== To see them, rerun with: --leak-check=full --show-reachable=y
>>
>> which seems considerable!
>>
>>
>>>  How can we say anything without the valgrind output?
>>>
>>>     Matt
>>>
>>>
>>>>
>>>> What else can I try to find the problem? Any recommendation is really
>>>> appreciated!
>>>>
>>>> Thanks,
>>>> Mohammad
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120513/99109f8e/attachment-0001.htm>

From knepley at gmail.com  Sun May 13 19:30:15 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 13 May 2012 20:30:15 -0400
Subject: [petsc-users] seg fault with VecGetArray
In-Reply-To: <CAKsPKwn6QLPYTHEYhz9T52qnEdxj0MdW3VVUz0bSBA2zq=2+kw@mail.gmail.com>
References: <CAKsPKw=Psi4Uc0bOxJ94Jwsa2x2kMe4rqnAk_PS5Mcajem=CZw@mail.gmail.com>
	<CAMYG4GndpeQ2EHYDLjCS_bokkqjYc0MH5vFYRetBUa-ceGmu2g@mail.gmail.com>
	<CAKsPKwn6QLPYTHEYhz9T52qnEdxj0MdW3VVUz0bSBA2zq=2+kw@mail.gmail.com>
Message-ID: <CAMYG4GkQkpviHPR=qZ2LV+cNAKhNjXPCujJa_7vvOanLwL8-hg@mail.gmail.com>

On Sun, May 13, 2012 at 7:33 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

>
>
> On Sat, May 12, 2012 at 3:16 AM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Sat, May 12, 2012 at 4:33 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>
>>> Hi guys,
>>>
>>> I'm having a really weird issue here! My code seg faults for certain
>>> problem size and after using gdb I have been able to pinpoint the problem
>>> to a VecGetArray call. Here's a series of things I have tried so far
>>>
>>> 1) -on_error_attach_debugger -----> unsuccessful; does not launch
>>> debugger
>>> 2) -start_in_debugger -------> unsuccessful; does not start debugger
>>>
>>
>> Run with -log_summary. It will tell you what options the program got.
>> Also, are there errors relating to X? Send
>> all output to petsc-maint at mcs.anl.gov
>>
>>
>
> Matt, -log_summary also does not generate any output! I was eventually
> able to start_in_debugger using xterm. Previously I was trying to start in
> kdbg. Even with xterm, -on_error_attach_debugger does not start the
> debugger. In either case, starting the debugger in xterm using
> -start_in_debugger or attaching the debugger myself manually, I get a
> segfault at VecGetArray and then the program terminates without any further
> output.
>

You SEGV before it can evidently.


> 3) attaching debugger myself -----> code runs in debugger and seg faults
>>> when calling VecGetArray
>>>
>>
>> Is this a debug build? What dereference is causing the SEGV? Is the Vec a
>> valid object? It sounds like
>> it has been corrupted.
>>
>>
>
> Yes; with the -g option. How can I check if Vec is "valid"?
>

PetscValidHeaderSpecific(v,VEC_CLASSID,1);


>  4) using ierr=VecGetArray;CHKERRQ(ierr) ------> PETSc does not produce
>>> error messages; the code simply seg faults and terminates
>>> 5) checking the values of ierr inside the debugger ---------> They are
>>> all 0 up untill the code terminates; I think this means petsc does not
>>> generate error?
>>> 6) checking for memory leak with valgrind -----------> All I get are
>>> leaks from OpenMPI and PetscInitialize and PetscFinalize; I think these are
>>> just routine and safe?
>>>
>>
>>
> Should I attach the whole valgrind output here or send it to petsc-maint?
> I just repeast these two a couple of times!:
>

None of this output about lost memory matters. Send the entire output to
petsc-maint

   Matt


> ==4508== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
> lost in loss record 2,644 of 2,665
> ==4508==    at 0x4C2815C: malloc (vg_replace_malloc.c:236)
> ==4508==    by 0x86417ED: ???
> ==4508==    by 0x5D4D099: orte_rml_base_comm_start (in
> /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
> ==4508==    by 0x8640AD1: ???
> ==4508==    by 0x5D3AFE6: orte_ess_base_app_setup (in
> /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
> ==4508==    by 0x8846E41: ???
> ==4508==    by 0x5D23A52: orte_init (in
> /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
> ==4508==    by 0x5A9E806: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.1)
> ==4508==    by 0x5ABFD7F: PMPI_Init (in
> /usr/lib/openmpi/lib/libmpi.so.0.0.1)
> ==4508==    by 0x530A90: PetscInitialize(int*, char***, char const*, char
> const*) (pinit.c:668)
> ==4508==    by 0x4A4955: PetscSession::PetscSession(int*, char***, char
> const*, char const*) (utilities.h:17)
> ==4508==    by 0x4A1DA5: main (main_Test2.cpp:49)
>
> ==4508== 74 bytes in 1 blocks are definitely lost in loss record 2,411 of
> 2,665
> ==4508==    at 0x4C2815C: malloc (vg_replace_malloc.c:236)
> ==4508==    by 0x6F2DDA1: strdup (strdup.c:43)
> ==4508==    by 0x5F85117: ??? (in
> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
> ==4508==    by 0x5F85359: mca_base_param_lookup_string (in
> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
> ==4508==    by 0xB301869: ???
> ==4508==    by 0xB2F5126: ???
> ==4508==    by 0x5F82E17: mca_base_components_open (in
> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
> ==4508==    by 0x5ADA6BA: mca_btl_base_open (in
> /usr/lib/openmpi/lib/libmpi.so.0.0.1)
> ==4508==    by 0xA6A9B93: ???
> ==4508==    by 0x5F82E17: mca_base_components_open (in
> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
> ==4508==    by 0x5AE3C88: mca_pml_base_open (in
> /usr/lib/openmpi/lib/libmpi.so.0.0.1)
> ==4508==    by 0x5A9E9E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.1)
>
>
> but eventually I get:
>
> ==4508== LEAK SUMMARY:
> ==4508==    definitely lost: 5,949 bytes in 55 blocks
> ==4508==    indirectly lost: 3,562 bytes in 32 blocks
> ==4508==      possibly lost: 0 bytes in 0 blocks
> ==4508==    still reachable: 181,516 bytes in 2,660 blocks
> ==4508==         suppressed: 0 bytes in 0 blocks
> ==4508== Reachable blocks (those to which a pointer was found) are not
> shown.
> ==4508== To see them, rerun with: --leak-check=full --show-reachable=y
>
> which seems considerable!
>
>
>>  How can we say anything without the valgrind output?
>>
>>     Matt
>>
>>
>>>
>>> What else can I try to find the problem? Any recommendation is really
>>> appreciated!
>>>
>>> Thanks,
>>> Mohammad
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120513/689d62b7/attachment.htm>

From mirzadeh at gmail.com  Sun May 13 20:07:11 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Sun, 13 May 2012 18:07:11 -0700
Subject: [petsc-users] seg fault with VecGetArray
In-Reply-To: <CAMYG4GkQkpviHPR=qZ2LV+cNAKhNjXPCujJa_7vvOanLwL8-hg@mail.gmail.com>
References: <CAKsPKw=Psi4Uc0bOxJ94Jwsa2x2kMe4rqnAk_PS5Mcajem=CZw@mail.gmail.com>
	<CAMYG4GndpeQ2EHYDLjCS_bokkqjYc0MH5vFYRetBUa-ceGmu2g@mail.gmail.com>
	<CAKsPKwn6QLPYTHEYhz9T52qnEdxj0MdW3VVUz0bSBA2zq=2+kw@mail.gmail.com>
	<CAMYG4GkQkpviHPR=qZ2LV+cNAKhNjXPCujJa_7vvOanLwL8-hg@mail.gmail.com>
Message-ID: <CAKsPKwnRVRs-4YzzyMJxHzmqCcb3qeZuXf5e8bABXqTGfwApOw@mail.gmail.com>

On Sun, May 13, 2012 at 5:30 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sun, May 13, 2012 at 7:33 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>>
>>
>> On Sat, May 12, 2012 at 3:16 AM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Sat, May 12, 2012 at 4:33 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I'm having a really weird issue here! My code seg faults for certain
>>>> problem size and after using gdb I have been able to pinpoint the problem
>>>> to a VecGetArray call. Here's a series of things I have tried so far
>>>>
>>>> 1) -on_error_attach_debugger -----> unsuccessful; does not launch
>>>> debugger
>>>> 2) -start_in_debugger -------> unsuccessful; does not start debugger
>>>>
>>>
>>> Run with -log_summary. It will tell you what options the program got.
>>> Also, are there errors relating to X? Send
>>> all output to petsc-maint at mcs.anl.gov
>>>
>>>
>>
>> Matt, -log_summary also does not generate any output! I was eventually
>> able to start_in_debugger using xterm. Previously I was trying to start in
>> kdbg. Even with xterm, -on_error_attach_debugger does not start the
>> debugger. In either case, starting the debugger in xterm using
>> -start_in_debugger or attaching the debugger myself manually, I get a
>> segfault at VecGetArray and then the program terminates without any further
>> output.
>>
>
> You SEGV before it can evidently.
>
>

sigh ... Just found it. You are right! I was, and I don't know why!,
allocating a large array on the stack and for certain size I was having a
stack overflow problem. Thanks for the help anyways.


>  3) attaching debugger myself -----> code runs in debugger and seg faults
>>>> when calling VecGetArray
>>>>
>>>
>>> Is this a debug build? What dereference is causing the SEGV? Is the Vec
>>> a valid object? It sounds like
>>> it has been corrupted.
>>>
>>>
>>
>> Yes; with the -g option. How can I check if Vec is "valid"?
>>
>
> PetscValidHeaderSpecific(v,VEC_CLASSID,1);
>
>
>>  4) using ierr=VecGetArray;CHKERRQ(ierr) ------> PETSc does not produce
>>>> error messages; the code simply seg faults and terminates
>>>> 5) checking the values of ierr inside the debugger ---------> They are
>>>> all 0 up untill the code terminates; I think this means petsc does not
>>>> generate error?
>>>> 6) checking for memory leak with valgrind -----------> All I get are
>>>> leaks from OpenMPI and PetscInitialize and PetscFinalize; I think these are
>>>> just routine and safe?
>>>>
>>>
>>>
>> Should I attach the whole valgrind output here or send it to petsc-maint?
>> I just repeast these two a couple of times!:
>>
>
> None of this output about lost memory matters. Send the entire output to
> petsc-maint
>
>    Matt
>
>
>> ==4508== 320 (288 direct, 32 indirect) bytes in 1 blocks are definitely
>> lost in loss record 2,644 of 2,665
>> ==4508==    at 0x4C2815C: malloc (vg_replace_malloc.c:236)
>> ==4508==    by 0x86417ED: ???
>> ==4508==    by 0x5D4D099: orte_rml_base_comm_start (in
>> /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
>> ==4508==    by 0x8640AD1: ???
>> ==4508==    by 0x5D3AFE6: orte_ess_base_app_setup (in
>> /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
>> ==4508==    by 0x8846E41: ???
>> ==4508==    by 0x5D23A52: orte_init (in
>> /usr/lib/openmpi/lib/libopen-rte.so.0.0.0)
>> ==4508==    by 0x5A9E806: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.1)
>> ==4508==    by 0x5ABFD7F: PMPI_Init (in
>> /usr/lib/openmpi/lib/libmpi.so.0.0.1)
>> ==4508==    by 0x530A90: PetscInitialize(int*, char***, char const*, char
>> const*) (pinit.c:668)
>> ==4508==    by 0x4A4955: PetscSession::PetscSession(int*, char***, char
>> const*, char const*) (utilities.h:17)
>> ==4508==    by 0x4A1DA5: main (main_Test2.cpp:49)
>>
>> ==4508== 74 bytes in 1 blocks are definitely lost in loss record 2,411 of
>> 2,665
>> ==4508==    at 0x4C2815C: malloc (vg_replace_malloc.c:236)
>> ==4508==    by 0x6F2DDA1: strdup (strdup.c:43)
>> ==4508==    by 0x5F85117: ??? (in
>> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
>> ==4508==    by 0x5F85359: mca_base_param_lookup_string (in
>> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
>> ==4508==    by 0xB301869: ???
>> ==4508==    by 0xB2F5126: ???
>> ==4508==    by 0x5F82E17: mca_base_components_open (in
>> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
>> ==4508==    by 0x5ADA6BA: mca_btl_base_open (in
>> /usr/lib/openmpi/lib/libmpi.so.0.0.1)
>> ==4508==    by 0xA6A9B93: ???
>> ==4508==    by 0x5F82E17: mca_base_components_open (in
>> /usr/lib/openmpi/lib/libopen-pal.so.0.0.0)
>> ==4508==    by 0x5AE3C88: mca_pml_base_open (in
>> /usr/lib/openmpi/lib/libmpi.so.0.0.1)
>> ==4508==    by 0x5A9E9E0: ??? (in /usr/lib/openmpi/lib/libmpi.so.0.0.1)
>>
>>
>> but eventually I get:
>>
>> ==4508== LEAK SUMMARY:
>> ==4508==    definitely lost: 5,949 bytes in 55 blocks
>> ==4508==    indirectly lost: 3,562 bytes in 32 blocks
>> ==4508==      possibly lost: 0 bytes in 0 blocks
>> ==4508==    still reachable: 181,516 bytes in 2,660 blocks
>> ==4508==         suppressed: 0 bytes in 0 blocks
>> ==4508== Reachable blocks (those to which a pointer was found) are not
>> shown.
>> ==4508== To see them, rerun with: --leak-check=full --show-reachable=y
>>
>> which seems considerable!
>>
>>
>>>  How can we say anything without the valgrind output?
>>>
>>>     Matt
>>>
>>>
>>>>
>>>> What else can I try to find the problem? Any recommendation is really
>>>> appreciated!
>>>>
>>>> Thanks,
>>>> Mohammad
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120513/e88774f3/attachment.htm>

From mike.hui.zhang at hotmail.com  Mon May 14 05:13:27 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Mon, 14 May 2012 12:13:27 +0200
Subject: [petsc-users] Dense multiply Sparse
Message-ID: <BLU0-SMTP213F35083CCB884AA959CC8CB1A0@phx.gbl>

I have two matrices A and B stored in sparse format.  While A is really sparse, B is relatively
dense (say non-zeros entries about 50%).  Now to multiply the two matrices, among 
    
      A*B 
or 
     (B^T * A^T)^T

which is better, or no big difference ?

Thanks!


From bsmith at mcs.anl.gov  Mon May 14 07:29:42 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 14 May 2012 07:29:42 -0500
Subject: [petsc-users] Dense multiply Sparse
In-Reply-To: <BLU0-SMTP213F35083CCB884AA959CC8CB1A0@phx.gbl>
References: <BLU0-SMTP213F35083CCB884AA959CC8CB1A0@phx.gbl>
Message-ID: <7F3B4ABD-CCD2-44B9-BA2E-C4032D3A7F86@mcs.anl.gov>


   If B is 50% dense then store it in a dense format. It will be much faster and the extra memory is minimal.  Even 30% dense.

   Barry

On May 14, 2012, at 5:13 AM, Hui Zhang wrote:

> I have two matrices A and B stored in sparse format.  While A is really sparse, B is relatively
> dense (say non-zeros entries about 50%).  Now to multiply the two matrices, among 
> 
>      A*B 
> or 
>     (B^T * A^T)^T
> 
> which is better, or no big difference ?
> 
> Thanks!
> 


From behzad.baghapour at gmail.com  Mon May 14 10:29:06 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Mon, 14 May 2012 19:59:06 +0430
Subject: [petsc-users] Linesearch algorithm
Message-ID: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>

Dear Developers,

I wish to know when the algorithm of Linesearch used by SNESLineSearchCubic
or SNESLineSearchQuadratic calls the Jacobian calculation routine defined
by user? Is it a way to change the Jacobian before calling the line-search
routines?

Thanks,
BehZad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120514/e2c69edd/attachment.htm>

From zhenglun.wei at gmail.com  Mon May 14 10:43:59 2012
From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei)
Date: Mon, 14 May 2012 10:43:59 -0500
Subject: [petsc-users] 3D Poisson Solver with a Periodic Boundary Condition
	in One Direction?
Message-ID: <4FB1283F.1090808@gmail.com>

Dear All,
     I hope you're having a nice day.
     I wonder if there is any example in PETSc to solve 3D Poisson 
equation with the periodic boundary condition in one direction. 2D is 
fine if there is no 3D example.

thanks,
Alan

From gmulas at oa-cagliari.inaf.it  Mon May 14 11:01:40 2012
From: gmulas at oa-cagliari.inaf.it (Giacomo Mulas)
Date: Mon, 14 May 2012 18:01:40 +0200 (CEST)
Subject: [petsc-users] How to set up -mat_mumps_icntl13 1 in the code?
Message-ID: <alpine.DEB.2.02.1205141744340.14621@capitanata.oa-cagliari.inaf.it>

Hello.

I am relatively new to petsc and slepc. I am trying to use slepc to solve a
sparse eigenvalue problem, in particular getting all eigenvalues in an
interval (spectral slicing). To run in parallel, the documentation states
that I have to specify an external linear solver, e.g. mumps, and explains
that I can use the command line options

-st_ksp_type preonly -st_pc_type cholesky \
-st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_13 1

to set everything up for it. I was indeed able to get my code to work with
these command line options (even if I will need some additional work to make
it run faster in parallel, but that's another story). However, I would
rather like to be able to put these options in my code itself, as defaults.

So far, I was able to do so for all command line options except for
-mat_mumps_icntl_13 1

How should I do that? Here goes a snippet of the relevant code:

(... omit creation/population of H matrix ...)
       ierr = EPSCreate(mixpars->slepc_comm,&eps);CHKERRQ(ierr);
       ierr = EPSSetOperators(eps,H,PETSC_NULL);CHKERRQ(ierr);
       ierr = EPSSetProblemType(eps,EPS_HEP);CHKERRQ(ierr);
       ierr = EPSSetTolerances(eps, tol, PETSC_DECIDE);
       if (statesinlist>500)
 	ierr = EPSSetDimensions(eps, 500, PETSC_IGNORE, PETSC_IGNORE);
       else
 	ierr = EPSSetDimensions(eps, statesinlist, PETSC_IGNORE,
 				PETSC_IGNORE);
       CHKERRQ(ierr);
       if (statesinlist<50) ierr = EPSSetType(eps, EPSLAPACK);
       else {
 	ierr = EPSGetST(eps,&st);CHKERRQ(ierr);
 	ierr = STSetType(st,STSINVERT);CHKERRQ(ierr);
 	ierr = STGetKSP(st,&ksp);CHKERRQ(ierr);
 	ierr = KSPSetType(ksp,KSPPREONLY);
 	ierr = KSPGetPC(ksp, &pc);CHKERRQ(ierr);
 	ierr = PCSetType(pc, PCCHOLESKY);CHKERRQ(ierr);
 	if (mixpars->slepc_size>1) {
 	  ierr = PCFactorSetMatSolverPackage(pc,MATSOLVERMUMPS);CHKERRQ(ierr);
 	  /*
 	  Mat F;
 	  ierr = PCFactorGetMatrix(pc,&F);CHKERRQ(ierr);
 	  PetscInt icntl = 13;
 	  PetscInt ival = 1;
 	  ierr = MatMumpsSetIcntl(F,icntl,ival); CHKERRQ(ierr);
 	  */
 	}
       }
       ierr = EPSSetFromOptions(eps); CHKERRQ(ierr);
       ierr = EPSSetInterval(eps,
 			    (PetscScalar) currsubclass->lowtargetlimit[j],
 			    (PetscScalar) currsubclass->uptargetlimit[j]);
       CHKERRQ(ierr);
       ierr = EPSSetWhichEigenpairs(eps, EPS_ALL); CHKERRQ(ierr);
       ierr = EPSSolve(eps);CHKERRQ(ierr);

The part which is commented out is what I tried to use to try to obtain the
same effect as the -mat_mumps_icntl_13 1 command line option. However, if I
decomment it, I get the following error:

[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: Matrix not yet factored; call after KSPSetUp() or PCSetUp()!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
CDT 2011 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.

The code works fine if I comment out the lines from 
Mat F;
to
ierr = MatMumpsSetIcntl(F,icntl,ival); CHKERRQ(ierr);
and use the command line option 
-mat_mumps_icntl_13 1

Is it indeed possible to set this as a default in the code, and if yes, is
it documented somewhere? Can someone give me a hint?

A side issue: should I explicitly free the PC, KSP and ST I get using the 
EPSGetST(), STGetKSP() and KSPGetPC() calls respectively? I only use them to
set up default options (instead of giving them on the command line). Or can
I trust that EPSDestroy() will do it?

Thanks in advance, bye
Giacomo Mulas

-- 
_________________________________________________________________

Giacomo Mulas <gmulas at oa-cagliari.inaf.it>
_________________________________________________________________

OSSERVATORIO ASTRONOMICO DI CAGLIARI
Str. 54, Loc. Poggio dei Pini * 09012 Capoterra (CA)

Tel. (OAC): +39 070 71180 248     Fax : +39 070 71180 222
Tel. (UNICA): +39 070 675 4916
_________________________________________________________________

"When the storms are raging around you, stay right where you are"
                          (Freddy Mercury)
_________________________________________________________________

From knepley at gmail.com  Mon May 14 11:09:08 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 14 May 2012 12:09:08 -0400
Subject: [petsc-users] Linesearch algorithm
In-Reply-To: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
References: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
Message-ID: <CAMYG4Gk5Kq9gOxyn5MnqXZjezsiO1Gn9cGiQTXODWHWFFgQFGw@mail.gmail.com>

On Mon, May 14, 2012 at 11:29 AM, behzad baghapour <
behzad.baghapour at gmail.com> wrote:

> Dear Developers,
>
> I wish to know when the algorithm of Linesearch used by
> SNESLineSearchCubic or SNESLineSearchQuadratic calls the Jacobian
> calculation routine defined by user? Is it a way to change the Jacobian
> before calling the line-search routines?
>

You can replace these algorithms with whatever you want. It would make no
sense to change the
Jacobian for the algorithms we have in there now.

   Matt


> Thanks,
> BehZad
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120514/fc117a21/attachment.htm>

From knepley at gmail.com  Mon May 14 11:11:04 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 14 May 2012 12:11:04 -0400
Subject: [petsc-users] How to set up -mat_mumps_icntl13 1 in the code?
In-Reply-To: <alpine.DEB.2.02.1205141744340.14621@capitanata.oa-cagliari.inaf.it>
References: <alpine.DEB.2.02.1205141744340.14621@capitanata.oa-cagliari.inaf.it>
Message-ID: <CAMYG4G=CBDdfUw4nAmsPxFsFQON6zWRtY5ikLnTsYxEbxYsgaA@mail.gmail.com>

On Mon, May 14, 2012 at 12:01 PM, Giacomo Mulas
<gmulas at oa-cagliari.inaf.it>wrote:

> Hello.
>
> I am relatively new to petsc and slepc. I am trying to use slepc to solve a
> sparse eigenvalue problem, in particular getting all eigenvalues in an
> interval (spectral slicing). To run in parallel, the documentation states
> that I have to specify an external linear solver, e.g. mumps, and explains
> that I can use the command line options
>
> -st_ksp_type preonly -st_pc_type cholesky \
> -st_pc_factor_mat_solver_**package mumps -mat_mumps_icntl_13 1
>
> to set everything up for it. I was indeed able to get my code to work with
> these command line options (even if I will need some additional work to
> make
> it run faster in parallel, but that's another story). However, I would
> rather like to be able to put these options in my code itself, as defaults.
>

The problems below are exactly why we do not advocate this strategy.
However, you
can get this effect using
http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscOptionsSetValue.html

   Matt


> So far, I was able to do so for all command line options except for
> -mat_mumps_icntl_13 1
>
> How should I do that? Here goes a snippet of the relevant code:
>
> (... omit creation/population of H matrix ...)
>      ierr = EPSCreate(mixpars->slepc_comm,**&eps);CHKERRQ(ierr);
>      ierr = EPSSetOperators(eps,H,PETSC_**NULL);CHKERRQ(ierr);
>      ierr = EPSSetProblemType(eps,EPS_HEP)**;CHKERRQ(ierr);
>      ierr = EPSSetTolerances(eps, tol, PETSC_DECIDE);
>      if (statesinlist>500)
>        ierr = EPSSetDimensions(eps, 500, PETSC_IGNORE, PETSC_IGNORE);
>      else
>        ierr = EPSSetDimensions(eps, statesinlist, PETSC_IGNORE,
>                                PETSC_IGNORE);
>      CHKERRQ(ierr);
>      if (statesinlist<50) ierr = EPSSetType(eps, EPSLAPACK);
>      else {
>        ierr = EPSGetST(eps,&st);CHKERRQ(**ierr);
>        ierr = STSetType(st,STSINVERT);**CHKERRQ(ierr);
>        ierr = STGetKSP(st,&ksp);CHKERRQ(**ierr);
>        ierr = KSPSetType(ksp,KSPPREONLY);
>        ierr = KSPGetPC(ksp, &pc);CHKERRQ(ierr);
>        ierr = PCSetType(pc, PCCHOLESKY);CHKERRQ(ierr);
>        if (mixpars->slepc_size>1) {
>          ierr = PCFactorSetMatSolverPackage(**pc,MATSOLVERMUMPS);CHKERRQ(*
> *ierr);
>          /*
>          Mat F;
>          ierr = PCFactorGetMatrix(pc,&F);**CHKERRQ(ierr);
>          PetscInt icntl = 13;
>          PetscInt ival = 1;
>          ierr = MatMumpsSetIcntl(F,icntl,ival)**; CHKERRQ(ierr);
>          */
>        }
>      }
>      ierr = EPSSetFromOptions(eps); CHKERRQ(ierr);
>      ierr = EPSSetInterval(eps,
>                            (PetscScalar) currsubclass->lowtargetlimit[**
> j],
>                            (PetscScalar) currsubclass->uptargetlimit[j]**
> );
>      CHKERRQ(ierr);
>      ierr = EPSSetWhichEigenpairs(eps, EPS_ALL); CHKERRQ(ierr);
>      ierr = EPSSolve(eps);CHKERRQ(ierr);
>
> The part which is commented out is what I tried to use to try to obtain the
> same effect as the -mat_mumps_icntl_13 1 command line option. However, if I
> decomment it, I get the following error:
>
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [0]PETSC ERROR: Operation done in wrong order!
> [0]PETSC ERROR: Matrix not yet factored; call after KSPSetUp() or
> PCSetUp()!
> [0]PETSC ERROR:
> ------------------------------**------------------------------**
> ------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 5, Sat Oct 29 13:45:54
> CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
>
> The code works fine if I comment out the lines from Mat F;
> to
> ierr = MatMumpsSetIcntl(F,icntl,ival)**; CHKERRQ(ierr);
> and use the command line option -mat_mumps_icntl_13 1
>
> Is it indeed possible to set this as a default in the code, and if yes, is
> it documented somewhere? Can someone give me a hint?
>
> A side issue: should I explicitly free the PC, KSP and ST I get using the
> EPSGetST(), STGetKSP() and KSPGetPC() calls respectively? I only use them to
> set up default options (instead of giving them on the command line). Or can
> I trust that EPSDestroy() will do it?
>
> Thanks in advance, bye
> Giacomo Mulas
>
> --
> ______________________________**______________________________**_____
>
> Giacomo Mulas <gmulas at oa-cagliari.inaf.it>
> ______________________________**______________________________**_____
>
> OSSERVATORIO ASTRONOMICO DI CAGLIARI
> Str. 54, Loc. Poggio dei Pini * 09012 Capoterra (CA)
>
> Tel. (OAC): +39 070 71180 248     Fax : +39 070 71180 222
> Tel. (UNICA): +39 070 675 4916
> ______________________________**______________________________**_____
>
> "When the storms are raging around you, stay right where you are"
>                         (Freddy Mercury)
> ______________________________**______________________________**_____
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120514/d5b10427/attachment.htm>

From bsmith at mcs.anl.gov  Mon May 14 12:00:18 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 14 May 2012 12:00:18 -0500
Subject: [petsc-users] Linesearch algorithm
In-Reply-To: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
References: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
Message-ID: <C7256B25-6176-4431-9BB8-9663C0C71024@mcs.anl.gov>


On May 14, 2012, at 10:29 AM, behzad baghapour wrote:

> Dear Developers,
> 
> I wish to know when the algorithm of Linesearch used by SNESLineSearchCubic or SNESLineSearchQuadratic calls the Jacobian calculation routine defined by user? Is it a way to change the Jacobian before calling the line-search routines?
> 
> Thanks,
> BehZad

   The linesearch routines themselves never call a "compute Jacobian" routine. It is only SNESSolve_LS() that computes the Jacobian and then uses it in the linear solve.   Some of the line search routines reuse the previously computed Jacobian in a matrix vector product to determine the appropriate descent requirements. You can see the code in src/snes/impls/ls/ls.c

   Barry


From gmulas at oa-cagliari.inaf.it  Mon May 14 12:45:53 2012
From: gmulas at oa-cagliari.inaf.it (Giacomo Mulas)
Date: Mon, 14 May 2012 19:45:53 +0200 (CEST)
Subject: [petsc-users] How to set up -mat_mumps_icntl13 1 in the code?
In-Reply-To: <CAMYG4G=CBDdfUw4nAmsPxFsFQON6zWRtY5ikLnTsYxEbxYsgaA@mail.gmail.com>
References: <alpine.DEB.2.02.1205141744340.14621@capitanata.oa-cagliari.inaf.it>
	<CAMYG4G=CBDdfUw4nAmsPxFsFQON6zWRtY5ikLnTsYxEbxYsgaA@mail.gmail.com>
Message-ID: <alpine.DEB.2.02.1205141854490.17844@capitanata.oa-cagliari.inaf.it>

On Mon, 14 May 2012, Matthew Knepley wrote:

> The problems below are exactly why we do not advocate this strategy.
> However, you
> can get this effectusing?http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscOpti
> onsSetValue.html

Thanks Matt!

Hmm, this would be more or less the same as adding the options to argc and
argv before initialising slepc and petsc, I gather. Functional, even if not
quite as clean as I would have liked. My idea was to set some working
default in the code, but still honor command line options if different, 
whereas what you suggest overrides any conflicting command line options. 
Indeed, in my code I call EPSSetFromOptions() _after_ my explicit function
calls, to make sure that command line options override my defaults (at
least, that's my intended logic...).

Maybe I should first check for appropriate command line options, and then
set them in the code _only if_ none were given on the command line?
What about:
 	  PetscInt ival;
 	  PetscBool isset;
 	  ierr = PetscOptionsGetInt(PETSC_NULL, "-mat_mumps_icntl_13",
 	  &ival,
 				    &isset);CHKERRQ(ierr);
 	  if (isset==PETSC_FALSE) {
 	    ierr = PetscOptionsSetValue("-mat_mumps_icntl_13","1");
 	    CHKERRQ(ierr);
 	  }

Does it look sensible?

Also, could you please specify a bit more what strategy you do not advocate,
and for what problems? Do you mean setting up options by direct function
calls, as opposed to setting them via PetscOptionsSetValue() ?
Or do you mean it is deprecated to set options in the code in any way, be it
via direct function calls or via PetscOptionsSetValue() ?

And the problems you are referring to are essentially the runtime error I
get, i.e. calling functions in the wrong order? Or something else?

Finally, if I call EPSGetST(), should I then call STDestroy() on the ST I
obtained, or will it be done by EPSDestroy()? And the same question iterates
for the KSP I obtained via STGetKSP(), and the PC I obtained via KSPGetPC().

Thanks, bye
Giacomo

-- 
_________________________________________________________________

Giacomo Mulas <gmulas at oa-cagliari.inaf.it>
_________________________________________________________________

OSSERVATORIO ASTRONOMICO DI CAGLIARI
Str. 54, Loc. Poggio dei Pini * 09012 Capoterra (CA)

Tel. (OAC): +39 070 71180 248     Fax : +39 070 71180 222
Tel. (UNICA): +39 070 675 4916
_________________________________________________________________

"When the storms are raging around you, stay right where you are"
                          (Freddy Mercury)
_________________________________________________________________

From jroman at dsic.upv.es  Mon May 14 13:26:20 2012
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Mon, 14 May 2012 20:26:20 +0200
Subject: [petsc-users] How to set up -mat_mumps_icntl13 1 in the code?
In-Reply-To: <alpine.DEB.2.02.1205141854490.17844@capitanata.oa-cagliari.inaf.it>
References: <alpine.DEB.2.02.1205141744340.14621@capitanata.oa-cagliari.inaf.it>
	<CAMYG4G=CBDdfUw4nAmsPxFsFQON6zWRtY5ikLnTsYxEbxYsgaA@mail.gmail.com>
	<alpine.DEB.2.02.1205141854490.17844@capitanata.oa-cagliari.inaf.it>
Message-ID: <F301A231-8636-4B36-91CF-0F3EFF12A140@dsic.upv.es>


El 14/05/2012, a las 19:45, Giacomo Mulas escribi?:

> On Mon, 14 May 2012, Matthew Knepley wrote:
> 
>> The problems below are exactly why we do not advocate this strategy.
>> However, you
>> can get this effectusing http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscOpti
>> onsSetValue.html
> 
> Thanks Matt!
> 
> Hmm, this would be more or less the same as adding the options to argc and
> argv before initialising slepc and petsc, I gather. Functional, even if not
> quite as clean as I would have liked. My idea was to set some working
> default in the code, but still honor command line options if different, whereas what you suggest overrides any conflicting command line options. Indeed, in my code I call EPSSetFromOptions() _after_ my explicit function
> calls, to make sure that command line options override my defaults (at
> least, that's my intended logic...).
> 
> Maybe I should first check for appropriate command line options, and then
> set them in the code _only if_ none were given on the command line?
> What about:
> 	  PetscInt ival;
> 	  PetscBool isset;
> 	  ierr = PetscOptionsGetInt(PETSC_NULL, "-mat_mumps_icntl_13",
> 	  &ival,
> 				    &isset);CHKERRQ(ierr);
> 	  if (isset==PETSC_FALSE) {
> 	    ierr = PetscOptionsSetValue("-mat_mumps_icntl_13","1");
> 	    CHKERRQ(ierr);
> 	  }
> 
> Does it look sensible?
> 
> Also, could you please specify a bit more what strategy you do not advocate,
> and for what problems? Do you mean setting up options by direct function
> calls, as opposed to setting them via PetscOptionsSetValue() ?
> Or do you mean it is deprecated to set options in the code in any way, be it
> via direct function calls or via PetscOptionsSetValue() ?
> 
> And the problems you are referring to are essentially the runtime error I
> get, i.e. calling functions in the wrong order? Or something else?

In order to set MUMPS options procedurally with MatMumpsSetIcntl (as in PETSc's ex52) one must first call KSPSetOperators() and PCSetType(). The problem is that in your example KSPSetOperators() will be called in STSetUp(). You could try to call STSetUp() explicitly but you may end up factoring the matrix twice. The design of ST is not prepared for a solution to your problem.

> 
> Finally, if I call EPSGetST(), should I then call STDestroy() on the ST I
> obtained, or will it be done by EPSDestroy()? And the same question iterates
> for the KSP I obtained via STGetKSP(), and the PC I obtained via KSPGetPC().

EPSGetST() does not increase the reference count, so EPSDestroy() will free it for you. When in doubt, run with -malloc_dump to see if there are uncleaned objects.

Jose

> 
> Thanks, bye
> Giacomo


From karpeev at mcs.anl.gov  Mon May 14 13:33:57 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Mon, 14 May 2012 13:33:57 -0500
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
Message-ID: <CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>

Hui,

The convergence issue should be resolved in the latest petsc-dev.
I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API
changes)
that should verify that.

Let me know if it works for you.
Thanks.
Dmitry.
On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Hi Dmitry,
>
> thanks for useful hints.  Good day!
>
> Hui
>
> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>
> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and
> recreate the matrices (including calling
> your modification subroutine), but not the subdomains or the subdomain
> solvers.
> If you just want to modify the submatrices, you can call
> PC(G)ASMGetSubmatrices() and modify the matrices it returns
> (in the same order as the subdomains were set). That's a bit of a
> hack, since you will essentially be modifying the PC's internal data
> structures.  As long as you are careful, you should be okay, since you
> already effectively have the same type of access to the submatrices through
> the Modify callback.
>
> Dmitry.
>
> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>> I just have a question about reuse of PCASM or PCGASM.
>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>> Next for the same linear system (matrix and RHS), I just want PCASM
>> modify the submatrices (PCSetModifySubmatrices) in a different way, using
>> the same routine for modifying but with
>> different user context for the modifying routine.
>>
>> What can I do for this task?  Currently, I destroy the KSP and
>> re-construct it. I guess
>> even for PCASM I can re-use it because the partition of subdomains remain
>> the same.
>>
>> Thanks!
>>
>>
>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>
>> Hui,
>> There've been several changes to PCGASM ahead of the new release.
>> Let me go back and see if it affected the convergence problem.
>> Dmitry.
>>
>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>
>>> Hi Dmitry,
>>>
>>> is there any news about PCGASM?
>>>
>>> thanks,
>>> Hui
>>>
>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>
>>> Okay, thanks.
>>> I'll take a look.
>>>
>>> Dmitry.
>>>
>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>
>>>> For reference, my results are attached.
>>>>
>>>> asm1.txt for asm with 1 process,
>>>> asm2.txt for asm with 2 processes,
>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers
>>>> different from others)
>>>> gasm2.txt for gasm with 2 processes
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> thank you,
>>>> Hui
>>>>
>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>
>>>>
>>>>
>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com
>>>> > wrote:
>>>>
>>>>>
>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com
>>>>> > wrote:
>>>>>
>>>>>> I have a new problem: the results from ASM and GASM are different and
>>>>>> it seems
>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests
>>>>>> are with
>>>>>> each subdomain supported only by one subdomain. There are no problems
>>>>>> when
>>>>>> I did not modify submatrices.  But when I modify submatrices, there
>>>>>> are problems
>>>>>> with GASM but no problems with ASM.
>>>>>>
>>>>>> For example, I use two subdomains. In the first case each subdomain
>>>>>> is supported by
>>>>>> one processor and there seems no problem with GASM. But when I use
>>>>>> run my program
>>>>>> with only one proc. so that it supports both of the two subdomains,
>>>>>> the iteration
>>>>>> number is different from the first case and is much larger.  On the
>>>>>> other hand
>>>>>> ASM has no such problem.
>>>>>>
>>>>>
>>>>> Are the solutions the same?
>>>>> What problem are you solving?
>>>>>
>>>>>
>>>>> Yes, the solutions are the same. That's why ASM gives the same results
>>>>> with one or
>>>>> two processors. But GASM did not.
>>>>>
>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the
>>>> case of two domains per processor?
>>>>
>>>>> I'm solving the Helmholtz equation.  Maybe
>>>>> I can prepare a simpler example to show this difference.
>>>>>
>>>> That would be helpful.
>>>> Thanks.
>>>>
>>>> Dmitry.
>>>>
>>>>>
>>>>>
>>>>> Dmitry.
>>>>>
>>>>>>
>>>>>>
>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>
>>>>>> You should be able to.
>>>>>> This behavior is the same as in PCASM,
>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>
>>>>>> Dmitry
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>
>>>>>> Hi Dmitry,
>>>>>>
>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another
>>>>>> question
>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the
>>>>>> prototype
>>>>>>
>>>>>> func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>
>>>>>> I think the coloumns from the parameter 'col' are always the same as
>>>>>> the rows
>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only
>>>>>> accepts
>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>
>>>>>>
>>>>>> As I tested, the row and col are always the same.
>>>>>>
>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for
>>>>>> the submat's
>>>>>> in the above func()?
>>>>>>
>>>>>> thanks,
>>>>>> Hui
>>>>>>
>>>>>>
>>>>>> thanks,
>>>>>> Hui
>>>>>>
>>>>>>
>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>
>>>>>> Yes, that's right.
>>>>>> There is no good way to help the user assemble the subdomains at the
>>>>>> moment beyond the 2D stuff.
>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>
>>>>>> There is ISColoringToList() that is supposed to convert a "coloring"
>>>>>> of indices to an array of ISs,
>>>>>> each having the indices with the same color and the subcomm that
>>>>>> supports that color. It is
>>>>>> largely untested, though.  You could try using it and give us
>>>>>> feedback on any problems you encounter.
>>>>>>
>>>>>> Dmitry.
>>>>>>
>>>>>>
>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<mike.hui.zhang at hotmail.com>
>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>
>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain
>>>>>>> supported by
>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and
>>>>>>> 'is_local[s]'
>>>>>>> in a subcommunicator consisting of processors supporting the
>>>>>>> subdomain 's'?
>>>>>>>
>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hui
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120514/46f8ed97/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gasm_test.c
Type: text/x-csrc
Size: 13400 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120514/46f8ed97/attachment-0001.c>

From bsmith at mcs.anl.gov  Mon May 14 14:23:57 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 14 May 2012 14:23:57 -0500
Subject: [petsc-users] How to set up -mat_mumps_icntl13 1 in the code?
In-Reply-To: <alpine.DEB.2.02.1205141854490.17844@capitanata.oa-cagliari.inaf.it>
References: <alpine.DEB.2.02.1205141744340.14621@capitanata.oa-cagliari.inaf.it>
	<CAMYG4G=CBDdfUw4nAmsPxFsFQON6zWRtY5ikLnTsYxEbxYsgaA@mail.gmail.com>
	<alpine.DEB.2.02.1205141854490.17844@capitanata.oa-cagliari.inaf.it>
Message-ID: <C4F9EBEC-89FF-4189-ACCD-C8254AB41F91@mcs.anl.gov>


   Giacomo,

    src/snes/examples/tutorials/ex22.c is an example of how one can set options in the program and then have the command line options override them.

char  common_options[]      = "-ksp_type fgmres\
                               -snes_grid_sequence 5 \
                               -pc_type mg\
                               -mg_levels_pc_type none \
                               -mg_coarse_pc_type none \
                               -pc_mg_type full \
                               -mg_coarse_ksp_type gmres \
                               -mg_levels_ksp_type gmres \
                               -mg_coarse_ksp_max_it 6 \
                               -mg_levels_ksp_max_it 3";

  PetscInitialize(&argc,&argv,PETSC_NULL,help);

  /* Hardwire several options; can be changed at command line */
  ierr = PetscOptionsInsertString(common_options);CHKERRQ(ierr);
  ierr = PetscOptionsInsertString(matrix_free_options);CHKERRQ(ierr);
  ierr = PetscOptionsInsert(&argc,&argv,PETSC_NULL);CHKERRQ(ierr); 

You don't need any if checks to see if something has already been set.

   Barry


On May 14, 2012, at 12:45 PM, Giacomo Mulas wrote:

> On Mon, 14 May 2012, Matthew Knepley wrote:
> 
>> The problems below are exactly why we do not advocate this strategy.
>> However, you
>> can get this effectusing http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscOpti
>> onsSetValue.html
> 
> Thanks Matt!
> 
> Hmm, this would be more or less the same as adding the options to argc and
> argv before initialising slepc and petsc, I gather. Functional, even if not
> quite as clean as I would have liked. My idea was to set some working
> default in the code, but still honor command line options if different, whereas what you suggest overrides any conflicting command line options. Indeed, in my code I call EPSSetFromOptions() _after_ my explicit function
> calls, to make sure that command line options override my defaults (at
> least, that's my intended logic...).
> 
> Maybe I should first check for appropriate command line options, and then
> set them in the code _only if_ none were given on the command line?
> What about:
> 	  PetscInt ival;
> 	  PetscBool isset;
> 	  ierr = PetscOptionsGetInt(PETSC_NULL, "-mat_mumps_icntl_13",
> 	  &ival,
> 				    &isset);CHKERRQ(ierr);
> 	  if (isset==PETSC_FALSE) {
> 	    ierr = PetscOptionsSetValue("-mat_mumps_icntl_13","1");
> 	    CHKERRQ(ierr);
> 	  }
> 
> Does it look sensible?
> 
> Also, could you please specify a bit more what strategy you do not advocate,
> and for what problems? Do you mean setting up options by direct function
> calls, as opposed to setting them via PetscOptionsSetValue() ?
> Or do you mean it is deprecated to set options in the code in any way, be it
> via direct function calls or via PetscOptionsSetValue() ?
> 
> And the problems you are referring to are essentially the runtime error I
> get, i.e. calling functions in the wrong order? Or something else?
> 
> Finally, if I call EPSGetST(), should I then call STDestroy() on the ST I
> obtained, or will it be done by EPSDestroy()? And the same question iterates
> for the KSP I obtained via STGetKSP(), and the PC I obtained via KSPGetPC().
> 
> Thanks, bye
> Giacomo
> 
> -- 
> _________________________________________________________________
> 
> Giacomo Mulas <gmulas at oa-cagliari.inaf.it>
> _________________________________________________________________
> 
> OSSERVATORIO ASTRONOMICO DI CAGLIARI
> Str. 54, Loc. Poggio dei Pini * 09012 Capoterra (CA)
> 
> Tel. (OAC): +39 070 71180 248     Fax : +39 070 71180 222
> Tel. (UNICA): +39 070 675 4916
> _________________________________________________________________
> 
> "When the storms are raging around you, stay right where you are"
>                         (Freddy Mercury)
> _________________________________________________________________


From andrew.spott at gmail.com  Mon May 14 22:33:02 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Mon, 14 May 2012 21:33:02 -0600
Subject: [petsc-users] Modifying the structure of a matrix.
Message-ID: <D9FAE42C-ABDA-4670-AEA4-889D2375195A@gmail.com>

What is the best way to selectively add/remove columns/rows of a Mat?

Is there a way to do it in place, or does it require essentially moving the values to a new matrix?

Thanks for the help.

-Andrew

From jedbrown at mcs.anl.gov  Mon May 14 22:38:46 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 14 May 2012 21:38:46 -0600
Subject: [petsc-users] Modifying the structure of a matrix.
In-Reply-To: <D9FAE42C-ABDA-4670-AEA4-889D2375195A@gmail.com>
References: <D9FAE42C-ABDA-4670-AEA4-889D2375195A@gmail.com>
Message-ID: <CAM9tzS=S+xOSpUpPUFzz1zMz=8BY+xai6+PChKzXrxtbTZ46OA@mail.gmail.com>

On Mon, May 14, 2012 at 9:33 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> What is the best way to selectively add/remove columns/rows of a Mat?
>
> Is there a way to do it in place, or does it require essentially moving
> the values to a new matrix?
>

If you only need to add/remove a small number of values, it's best to
allocate for all that might be present and just store explicit zeros.

To add or remove rows, you are also changing the vector size that can be
multiplied against, and the data structure must be rebuilt. This is why
it's frequently preferable to use MatZeroRows (or a variant) instead of
removing them from the data structure.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120514/48d8cd78/attachment.htm>

From andrew.spott at gmail.com  Mon May 14 22:51:07 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Mon, 14 May 2012 21:51:07 -0600
Subject: [petsc-users] Modifying the structure of a matrix.
In-Reply-To: <CAM9tzS=S+xOSpUpPUFzz1zMz=8BY+xai6+PChKzXrxtbTZ46OA@mail.gmail.com>
References: <D9FAE42C-ABDA-4670-AEA4-889D2375195A@gmail.com>
	<CAM9tzS=S+xOSpUpPUFzz1zMz=8BY+xai6+PChKzXrxtbTZ46OA@mail.gmail.com>
Message-ID: <D7D58CA5-7829-404A-81AB-0D3612F64F96@gmail.com>

Is there a computational cost to using a sparse matrix that is much bigger than it should be?

for example, say you have a matrix that is n x n, but you only store anything in the first m rows and columns (m < n).  

Also, would you then have to redo the matrix distribution among nodes by hand, or will PETSC do it for you?

Thanks for the quick reply

-Andrew

On May 14, 2012, at 9:38 PM, Jed Brown wrote:

> On Mon, May 14, 2012 at 9:33 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
> What is the best way to selectively add/remove columns/rows of a Mat?
> 
> Is there a way to do it in place, or does it require essentially moving the values to a new matrix?
> 
> If you only need to add/remove a small number of values, it's best to allocate for all that might be present and just store explicit zeros.
> 
> To add or remove rows, you are also changing the vector size that can be multiplied against, and the data structure must be rebuilt. This is why it's frequently preferable to use MatZeroRows (or a variant) instead of removing them from the data structure.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120514/daff6cbc/attachment.htm>

From abhyshr at mcs.anl.gov  Mon May 14 22:55:46 2012
From: abhyshr at mcs.anl.gov (Shri)
Date: Mon, 14 May 2012 22:55:46 -0500
Subject: [petsc-users] Modifying the structure of a matrix.
In-Reply-To: <D7D58CA5-7829-404A-81AB-0D3612F64F96@gmail.com>
References: <D9FAE42C-ABDA-4670-AEA4-889D2375195A@gmail.com>
	<CAM9tzS=S+xOSpUpPUFzz1zMz=8BY+xai6+PChKzXrxtbTZ46OA@mail.gmail.com>
	<D7D58CA5-7829-404A-81AB-0D3612F64F96@gmail.com>
Message-ID: <32B30BCC-FA15-428F-B4D7-BBDFAA2837B9@mcs.anl.gov>

What are you trying to do by removing the rows/columns of the matrix? Are there any variables that you want to treat as constants and hence eliminating the rows and columns?

Shri

On May 14, 2012, at 10:51 PM, Andrew Spott wrote:

> Is there a computational cost to using a sparse matrix that is much bigger than it should be?
> 
> for example, say you have a matrix that is n x n, but you only store anything in the first m rows and columns (m < n).  
> 
> Also, would you then have to redo the matrix distribution among nodes by hand, or will PETSC do it for you?
> 
> Thanks for the quick reply
> 
> -Andrew
> 
> On May 14, 2012, at 9:38 PM, Jed Brown wrote:
> 
>> On Mon, May 14, 2012 at 9:33 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
>> What is the best way to selectively add/remove columns/rows of a Mat?
>> 
>> Is there a way to do it in place, or does it require essentially moving the values to a new matrix?
>> 
>> If you only need to add/remove a small number of values, it's best to allocate for all that might be present and just store explicit zeros.
>> 
>> To add or remove rows, you are also changing the vector size that can be multiplied against, and the data structure must be rebuilt. This is why it's frequently preferable to use MatZeroRows (or a variant) instead of removing them from the data structure.
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120514/ab63e191/attachment.htm>

From andrew.spott at gmail.com  Mon May 14 23:07:19 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Mon, 14 May 2012 22:07:19 -0600
Subject: [petsc-users] Modifying the structure of a matrix.
In-Reply-To: <32B30BCC-FA15-428F-B4D7-BBDFAA2837B9@mcs.anl.gov>
References: <D9FAE42C-ABDA-4670-AEA4-889D2375195A@gmail.com>
	<CAM9tzS=S+xOSpUpPUFzz1zMz=8BY+xai6+PChKzXrxtbTZ46OA@mail.gmail.com>
	<D7D58CA5-7829-404A-81AB-0D3612F64F96@gmail.com>
	<32B30BCC-FA15-428F-B4D7-BBDFAA2837B9@mcs.anl.gov>
Message-ID: <C3F18BF7-72C1-4605-8EF2-C9D71196F043@gmail.com>

I'm attempting to do a convergence study on my basis, see how large my basis need to be to make my results accurate, does that help?

-Andrew

On May 14, 2012, at 9:55 PM, Shri wrote:

> What are you trying to do by removing the rows/columns of the matrix? Are there any variables that you want to treat as constants and hence eliminating the rows and columns?
> 
> Shri
> 
> On May 14, 2012, at 10:51 PM, Andrew Spott wrote:
> 
>> Is there a computational cost to using a sparse matrix that is much bigger than it should be?
>> 
>> for example, say you have a matrix that is n x n, but you only store anything in the first m rows and columns (m < n).  
>> 
>> Also, would you then have to redo the matrix distribution among nodes by hand, or will PETSC do it for you?
>> 
>> Thanks for the quick reply
>> 
>> -Andrew
>> 
>> On May 14, 2012, at 9:38 PM, Jed Brown wrote:
>> 
>>> On Mon, May 14, 2012 at 9:33 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
>>> What is the best way to selectively add/remove columns/rows of a Mat?
>>> 
>>> Is there a way to do it in place, or does it require essentially moving the values to a new matrix?
>>> 
>>> If you only need to add/remove a small number of values, it's best to allocate for all that might be present and just store explicit zeros.
>>> 
>>> To add or remove rows, you are also changing the vector size that can be multiplied against, and the data structure must be rebuilt. This is why it's frequently preferable to use MatZeroRows (or a variant) instead of removing them from the data structure.
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120514/9d54f416/attachment.htm>

From abhyshr at mcs.anl.gov  Mon May 14 23:30:41 2012
From: abhyshr at mcs.anl.gov (Shri)
Date: Mon, 14 May 2012 23:30:41 -0500
Subject: [petsc-users] Modifying the structure of a matrix.
In-Reply-To: <C3F18BF7-72C1-4605-8EF2-C9D71196F043@gmail.com>
References: <D9FAE42C-ABDA-4670-AEA4-889D2375195A@gmail.com>
	<CAM9tzS=S+xOSpUpPUFzz1zMz=8BY+xai6+PChKzXrxtbTZ46OA@mail.gmail.com>
	<D7D58CA5-7829-404A-81AB-0D3612F64F96@gmail.com>
	<32B30BCC-FA15-428F-B4D7-BBDFAA2837B9@mcs.anl.gov>
	<C3F18BF7-72C1-4605-8EF2-C9D71196F043@gmail.com>
Message-ID: <4589BB71-BD27-4B72-B346-404AFF66141D@mcs.anl.gov>


On May 14, 2012, at 11:07 PM, Andrew Spott wrote:

> I'm attempting to do a convergence study on my basis, see how large my basis need to be to make my results accurate, does that help?

   You could use create a matrix with the set of all basis vectors and then zero out the rows & columns of the matrix using MatZeroRowsColumns. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatZeroRowsColumns.html 
   I think this should be better than each time creating a new matrix when the basis changes.
> 
> -Andrew
> 
> On May 14, 2012, at 9:55 PM, Shri wrote:
> 
>> What are you trying to do by removing the rows/columns of the matrix? Are there any variables that you want to treat as constants and hence eliminating the rows and columns?
>> 
>> Shri
>> 
>> On May 14, 2012, at 10:51 PM, Andrew Spott wrote:
>> 
>>> Is there a computational cost to using a sparse matrix that is much bigger than it should be?
>>> 
>>> for example, say you have a matrix that is n x n, but you only store anything in the first m rows and columns (m < n).  
>>> 
>>> Also, would you then have to redo the matrix distribution among nodes by hand, or will PETSC do it for you?
>>> 
>>> Thanks for the quick reply
>>> 
>>> -Andrew
>>> 
>>> On May 14, 2012, at 9:38 PM, Jed Brown wrote:
>>> 
>>>> On Mon, May 14, 2012 at 9:33 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
>>>> What is the best way to selectively add/remove columns/rows of a Mat?
>>>> 
>>>> Is there a way to do it in place, or does it require essentially moving the values to a new matrix?
>>>> 
>>>> If you only need to add/remove a small number of values, it's best to allocate for all that might be present and just store explicit zeros.
>>>> 
>>>> To add or remove rows, you are also changing the vector size that can be multiplied against, and the data structure must be rebuilt. This is why it's frequently preferable to use MatZeroRows (or a variant) instead of removing them from the data structure.
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120514/b3548b79/attachment-0001.htm>

From andrew.spott at gmail.com  Mon May 14 23:39:29 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Mon, 14 May 2012 22:39:29 -0600
Subject: [petsc-users] Modifying the structure of a matrix.
In-Reply-To: <4589BB71-BD27-4B72-B346-404AFF66141D@mcs.anl.gov>
References: <D9FAE42C-ABDA-4670-AEA4-889D2375195A@gmail.com>
	<CAM9tzS=S+xOSpUpPUFzz1zMz=8BY+xai6+PChKzXrxtbTZ46OA@mail.gmail.com>
	<D7D58CA5-7829-404A-81AB-0D3612F64F96@gmail.com>
	<32B30BCC-FA15-428F-B4D7-BBDFAA2837B9@mcs.anl.gov>
	<C3F18BF7-72C1-4605-8EF2-C9D71196F043@gmail.com>
	<4589BB71-BD27-4B72-B346-404AFF66141D@mcs.anl.gov>
Message-ID: <1E35CF63-4E20-4BF6-A098-E2BBB22600E8@gmail.com>

That is what I figure.

I'm curious though if you need to manually determine the local row distribution after you do that.  (for example, say you completely remove all the values from the local range of one processor? that processor wouldn't be utilized unless you redistribute the matrix)

-Andrew


On May 14, 2012, at 10:30 PM, Shri wrote:

> 
> On May 14, 2012, at 11:07 PM, Andrew Spott wrote:
> 
>> I'm attempting to do a convergence study on my basis, see how large my basis need to be to make my results accurate, does that help?
> 
>    You could use create a matrix with the set of all basis vectors and then zero out the rows & columns of the matrix using MatZeroRowsColumns. http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatZeroRowsColumns.html 
>    I think this should be better than each time creating a new matrix when the basis changes.
>> 
>> -Andrew
>> 
>> On May 14, 2012, at 9:55 PM, Shri wrote:
>> 
>>> What are you trying to do by removing the rows/columns of the matrix? Are there any variables that you want to treat as constants and hence eliminating the rows and columns?
>>> 
>>> Shri
>>> 
>>> On May 14, 2012, at 10:51 PM, Andrew Spott wrote:
>>> 
>>>> Is there a computational cost to using a sparse matrix that is much bigger than it should be?
>>>> 
>>>> for example, say you have a matrix that is n x n, but you only store anything in the first m rows and columns (m < n).  
>>>> 
>>>> Also, would you then have to redo the matrix distribution among nodes by hand, or will PETSC do it for you?
>>>> 
>>>> Thanks for the quick reply
>>>> 
>>>> -Andrew
>>>> 
>>>> On May 14, 2012, at 9:38 PM, Jed Brown wrote:
>>>> 
>>>>> On Mon, May 14, 2012 at 9:33 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
>>>>> What is the best way to selectively add/remove columns/rows of a Mat?
>>>>> 
>>>>> Is there a way to do it in place, or does it require essentially moving the values to a new matrix?
>>>>> 
>>>>> If you only need to add/remove a small number of values, it's best to allocate for all that might be present and just store explicit zeros.
>>>>> 
>>>>> To add or remove rows, you are also changing the vector size that can be multiplied against, and the data structure must be rebuilt. This is why it's frequently preferable to use MatZeroRows (or a variant) instead of removing them from the data structure.
>>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120514/5116a7ab/attachment.htm>

From behzad.baghapour at gmail.com  Tue May 15 01:26:00 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Tue, 15 May 2012 09:56:00 +0330
Subject: [petsc-users] Linesearch algorithm
In-Reply-To: <C7256B25-6176-4431-9BB8-9663C0C71024@mcs.anl.gov>
References: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
	<C7256B25-6176-4431-9BB8-9663C0C71024@mcs.anl.gov>
Message-ID: <CA+jOtD+_4gK-ySHKiCsfQaoYs5+r_jkUcv70cOjdhBoiaS98+w@mail.gmail.com>

Thanks. Actually the jacobian which I obtain has a part of transient as J =
M/dt + dR/dQ. Then I want to eliminate the time effect before calculating
R^T*J*dQ as the descent direction. So, which way is correct in my case?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/34b087a2/attachment.htm>

From thomas.witkowski at tu-dresden.de  Tue May 15 02:25:18 2012
From: thomas.witkowski at tu-dresden.de (Thomas Witkowski)
Date: Tue, 15 May 2012 09:25:18 +0200
Subject: [petsc-users] Using direct solvers in parallel
Message-ID: <4FB204DE.5070206@tu-dresden.de>

I made some comparisons of using umfpack, superlu, superlu_dist and 
mumps to solve systems with sparse matrices arising from finite element 
method. The size of the matrices range from around 50000 to more than 3 
million unknowns. I used 1, 2, 4, 8 and 16 nodes to make the benchmark. 
Now, I wonder that in all cases the sequential umfpack was the fastest 
one. So even with 16 cores, superlu_dist and mumps are slower. Can 
anybody of you confirm this observation? Are there any other parallel 
direct solvers around which are more efficient?

Thomas

From dave.mayhem23 at gmail.com  Tue May 15 02:36:18 2012
From: dave.mayhem23 at gmail.com (Dave May)
Date: Tue, 15 May 2012 09:36:18 +0200
Subject: [petsc-users] Using direct solvers in parallel
In-Reply-To: <4FB204DE.5070206@tu-dresden.de>
References: <4FB204DE.5070206@tu-dresden.de>
Message-ID: <CAJ98EDo6sjx6AXZe98vdhe7Kp0UR_3Oefu5YHG165_QUUuMq5A@mail.gmail.com>

I have seem similar behaviour comparing umfpack and superlu_dist,
however the difference wasn't enormous, possibly umfpack was a factor
of 1.2-1.4 times faster on 1 - 4 cores.
What sort of time differences are you observing? Can you post the
numbers somewhere?

However, umpack will not work on a distributed memory machine.
My personal preference is to use superlu_dist in parallel. In my
experience using it as a coarse grid solver for multigrid, I find it
much more reliable than mumps. However, when mumps works, its is
typically slightly faster than superlu_dist. Again, not by a large
amount - never more than a factor of 2 faster.

The failure rate using mumps is definitely higher (in my experience)
when running on large numbers of cores compared to superlu_dist. I've
never got to the bottom as to why it fails.

Cheers,
  Dave


On 15 May 2012 09:25, Thomas Witkowski <thomas.witkowski at tu-dresden.de> wrote:
> I made some comparisons of using umfpack, superlu, superlu_dist and mumps to
> solve systems with sparse matrices arising from finite element method. The
> size of the matrices range from around 50000 to more than 3 million
> unknowns. I used 1, 2, 4, 8 and 16 nodes to make the benchmark. Now, I
> wonder that in all cases the sequential umfpack was the fastest one. So even
> with 16 cores, superlu_dist and mumps are slower. Can anybody of you confirm
> this observation? Are there any other parallel direct solvers around which
> are more efficient?
>
> Thomas

From thomas.witkowski at tu-dresden.de  Tue May 15 02:54:56 2012
From: thomas.witkowski at tu-dresden.de (Thomas Witkowski)
Date: Tue, 15 May 2012 09:54:56 +0200
Subject: [petsc-users] Using direct solvers in parallel
In-Reply-To: <CAJ98EDo6sjx6AXZe98vdhe7Kp0UR_3Oefu5YHG165_QUUuMq5A@mail.gmail.com>
References: <4FB204DE.5070206@tu-dresden.de>
	<CAJ98EDo6sjx6AXZe98vdhe7Kp0UR_3Oefu5YHG165_QUUuMq5A@mail.gmail.com>
Message-ID: <4FB20BD0.4040809@tu-dresden.de>

Am 15.05.2012 09:36, schrieb Dave May:
> I have seem similar behaviour comparing umfpack and superlu_dist,
> however the difference wasn't enormous, possibly umfpack was a factor
> of 1.2-1.4 times faster on 1 - 4 cores.
> What sort of time differences are you observing? Can you post the
> numbers somewhere?
I attached my data to this mail. For the largest matrix, umfpack failed 
after allocating 4 GB of memory. I have not tried to figure out what's 
the problem there. As you can see, for these matrices the distributed 
solvers are slower by a factor of 2 or 3 compared to umfpack. For all 
solvers, I have used the standard parameters, so I have not played 
around with the permutation strategies and such things. This may be also 
the reason why superlu is much slower than superlu_dist with just one 
core as it makes use of different col and row permutation strategies.
> However, umpack will not work on a distributed memory machine.
> My personal preference is to use superlu_dist in parallel. In my
> experience using it as a coarse grid solver for multigrid, I find it
> much more reliable than mumps. However, when mumps works, its is
> typically slightly faster than superlu_dist. Again, not by a large
> amount - never more than a factor of 2 faster.
In my codes I also make use of the distributed direct solvers for the 
coarse grid problems. I just wanted to make some tests how far away 
these solvers are from the sequential counterparts.

Thomas
>
> The failure rate using mumps is definitely higher (in my experience)
> when running on large numbers of cores compared to superlu_dist. I've
> never got to the bottom as to why it fails.
>
> Cheers,
>    Dave
>
>
> On 15 May 2012 09:25, Thomas Witkowski<thomas.witkowski at tu-dresden.de>  wrote:
>> I made some comparisons of using umfpack, superlu, superlu_dist and mumps to
>> solve systems with sparse matrices arising from finite element method. The
>> size of the matrices range from around 50000 to more than 3 million
>> unknowns. I used 1, 2, 4, 8 and 16 nodes to make the benchmark. Now, I
>> wonder that in all cases the sequential umfpack was the fastest one. So even
>> with 16 cores, superlu_dist and mumps are slower. Can anybody of you confirm
>> this observation? Are there any other parallel direct solvers around which
>> are more efficient?
>>
>> Thomas

-------------- next part --------------
   #rows |  umfpack / 1.core |  superlu / 1.core | superlu_dist / 1.core | mumps / 1.core |
-------------------------------------------------------------------------|----------------|
   49923 |             0.644 |             4.914 |                 2.148 |          1.731 |
  198147 |             3.992 |             41.53 |                 13.05 |          10.04 |
  792507 |             32.33 |             463.5 |                 66.75 |          52.56 |
 3151975 |         4GB limit |                -  |                 394.1 |          303.2 |


   #rows | superlu_dist / 1.core | superlu_dist / 2.core | superlu_dist / 4.core | superlu_dist / 8.core | superlu_dist / 16.core |
---------------------------------|-----------------------|-----------------------|-----------------------|------------------------|
   49923 |                 2.148 |                 1.922 |                 1.742 |                 1.705 |                  1.745 |
  198147 |                 13.05 |		   11.77 |                 10.47 |                 9.832 |                  9.565 |
  792507 |                 66.75 |                 61.07 |                 53.58 |                 49.14 |                  47.06 |
 3151875 |                 394.1 |                failed |                failed |                failed |                 failed |


   #rows | mumps / 1.core | mumps / 2.core | mumps / 4.core | mumps / 8.core | mumps / 16.core |
--------------------------|----------------|----------------|----------------|-----------------|
   49923 |          1.731 |          1.562 |          1.485 |          1.426 |           1.418 |
  198147 |          10.04 |          8.959 |          8.468 |          8.144 |           7.978 |
  792507 |          52.56 |          45.38 |          42.56 |          40.22 |           39.12 |
 3151875 |          303.2 |          255.7 |          232.2 |          216.0 |           210.0 |

From dave.mayhem23 at gmail.com  Tue May 15 03:00:48 2012
From: dave.mayhem23 at gmail.com (Dave May)
Date: Tue, 15 May 2012 10:00:48 +0200
Subject: [petsc-users] Using direct solvers in parallel
In-Reply-To: <4FB20BD0.4040809@tu-dresden.de>
References: <4FB204DE.5070206@tu-dresden.de>
	<CAJ98EDo6sjx6AXZe98vdhe7Kp0UR_3Oefu5YHG165_QUUuMq5A@mail.gmail.com>
	<4FB20BD0.4040809@tu-dresden.de>
Message-ID: <CAJ98EDoemXhUFShQiBsYi8mQa5omzF9Ue_1Oz+1ZM_V1+SboVQ@mail.gmail.com>

Ah okay. Thanks for the timings.

Have you monitored the CPU usage when you using umfpack?
On my machine, it's definitely not running on a single process,
so I wouldn't consider it a sequential solver.




On 15 May 2012 09:54, Thomas Witkowski <thomas.witkowski at tu-dresden.de> wrote:
> Am 15.05.2012 09:36, schrieb Dave May:
>
>> I have seem similar behaviour comparing umfpack and superlu_dist,
>> however the difference wasn't enormous, possibly umfpack was a factor
>> of 1.2-1.4 times faster on 1 - 4 cores.
>> What sort of time differences are you observing? Can you post the
>> numbers somewhere?
>
> I attached my data to this mail. For the largest matrix, umfpack failed
> after allocating 4 GB of memory. I have not tried to figure out what's the
> problem there. As you can see, for these matrices the distributed solvers
> are slower by a factor of 2 or 3 compared to umfpack. For all solvers, I
> have used the standard parameters, so I have not played around with the
> permutation strategies and such things. This may be also the reason why
> superlu is much slower than superlu_dist with just one core as it makes use
> of different col and row permutation strategies.
>
>> However, umpack will not work on a distributed memory machine.
>> My personal preference is to use superlu_dist in parallel. In my
>> experience using it as a coarse grid solver for multigrid, I find it
>> much more reliable than mumps. However, when mumps works, its is
>> typically slightly faster than superlu_dist. Again, not by a large
>> amount - never more than a factor of 2 faster.
>
> In my codes I also make use of the distributed direct solvers for the coarse
> grid problems. I just wanted to make some tests how far away these solvers
> are from the sequential counterparts.
>
> Thomas
>
>>
>> The failure rate using mumps is definitely higher (in my experience)
>> when running on large numbers of cores compared to superlu_dist. I've
>> never got to the bottom as to why it fails.
>>
>> Cheers,
>> ? Dave
>>
>>
>> On 15 May 2012 09:25, Thomas Witkowski<thomas.witkowski at tu-dresden.de>
>> ?wrote:
>>>
>>> I made some comparisons of using umfpack, superlu, superlu_dist and mumps
>>> to
>>> solve systems with sparse matrices arising from finite element method.
>>> The
>>> size of the matrices range from around 50000 to more than 3 million
>>> unknowns. I used 1, 2, 4, 8 and 16 nodes to make the benchmark. Now, I
>>> wonder that in all cases the sequential umfpack was the fastest one. So
>>> even
>>> with 16 cores, superlu_dist and mumps are slower. Can anybody of you
>>> confirm
>>> this observation? Are there any other parallel direct solvers around
>>> which
>>> are more efficient?
>>>
>>> Thomas
>
>

From thomas.witkowski at tu-dresden.de  Tue May 15 03:08:38 2012
From: thomas.witkowski at tu-dresden.de (Thomas Witkowski)
Date: Tue, 15 May 2012 10:08:38 +0200
Subject: [petsc-users] Using direct solvers in parallel
In-Reply-To: <CAJ98EDoemXhUFShQiBsYi8mQa5omzF9Ue_1Oz+1ZM_V1+SboVQ@mail.gmail.com>
References: <4FB204DE.5070206@tu-dresden.de>
	<CAJ98EDo6sjx6AXZe98vdhe7Kp0UR_3Oefu5YHG165_QUUuMq5A@mail.gmail.com>
	<4FB20BD0.4040809@tu-dresden.de>
	<CAJ98EDoemXhUFShQiBsYi8mQa5omzF9Ue_1Oz+1ZM_V1+SboVQ@mail.gmail.com>
Message-ID: <4FB20F06.4090500@tu-dresden.de>

Am 15.05.2012 10:00, schrieb Dave May:
> Ah okay. Thanks for the timings.
>
> Have you monitored the CPU usage when you using umfpack?
> On my machine, it's definitely not running on a single process,
> so I wouldn't consider it a sequential solver.
>
>
>
Yes, the CPU usage is 100% and not more. If you see umfpack using more 
than 1 core its because your BLAS library is compiled for multi-thread 
using. MKL and ACML can do this. In this case just set the environment 
variable OMP_NUM_THREADS to 1. When I use the multi-threaded version of 
MKL, using more than 1 threads is SLOWER than the sequential run.

Thomas

From mike.hui.zhang at hotmail.com  Tue May 15 04:03:50 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Tue, 15 May 2012 11:03:50 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
Message-ID: <BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>

Dmitry,

I got the newest petsc-dev and I run the test by

    mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view

which gives the following output

PC Object:(dd_) 1 MPI processes
  type: asm
    Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1
                                              ^^^
                                         note the above number, it should be 2 

While PCASM has no such problem.

Thanks,
Hui



> Hui,
> 
> The convergence issue should be resolved in the latest petsc-dev.
> I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) 
> that should verify that.
> 
> Let me know if it works for you.
> Thanks.
> Dmitry.
> On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> Hi Dmitry,
> 
> thanks for useful hints.  Good day!
> 
> Hui
> 
> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
> 
>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
>> your modification subroutine), but not the subdomains or the subdomain solvers. 
>> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
>> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
>> 
>> Dmitry.
>> 
>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> I just have a question about reuse of PCASM or PCGASM.
>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
>> different user context for the modifying routine.  
>> 
>> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
>> even for PCASM I can re-use it because the partition of subdomains remain the same.
>> 
>> Thanks!
>> 
>> 
>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>> 
>>> Hui,
>>> There've been several changes to PCGASM ahead of the new release.
>>> Let me go back and see if it affected the convergence problem.
>>> Dmitry.
>>> 
>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> Hi Dmitry,
>>> 
>>> is there any news about PCGASM? 
>>> 
>>> thanks,
>>> Hui
>>> 
>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>> 
>>>> Okay, thanks.
>>>> I'll take a look.
>>>> 
>>>> Dmitry.
>>>> 
>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> For reference, my results are attached.
>>>> 
>>>> asm1.txt for asm with 1 process,
>>>> asm2.txt for asm with 2 processes,
>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>>> gasm2.txt for gasm with 2 processes
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> thank you,
>>>> Hui
>>>> 
>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>> 
>>>>> 
>>>>> 
>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> 
>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>>> with GASM but no problems with ASM. 
>>>>>> 
>>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>>> number is different from the first case and is much larger.  On the other hand
>>>>>> ASM has no such problem.
>>>>>>  
>>>>>> Are the solutions the same?
>>>>>> What problem are you solving?
>>>>> 
>>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>>> two processors. But GASM did not.  
>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>>> I'm solving the Helmholtz equation.  Maybe 
>>>>> I can prepare a simpler example to show this difference.
>>>>> That would be helpful.  
>>>>> Thanks.
>>>>> 
>>>>> Dmitry. 
>>>>> 
>>>>>> 
>>>>>> Dmitry. 
>>>>>> 
>>>>>> 
>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>> 
>>>>>>> You should be able to. 
>>>>>>> This behavior is the same as in PCASM,
>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>> 
>>>>>>> Dmitry
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>> 
>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>> 
>>>>>>>>> Hi Dmitry,
>>>>>>>>> 
>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>>> 
>>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>> 
>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>> 
>>>>>>>> As I tested, the row and col are always the same. 
>>>>>>>> 
>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>>> in the above func()?
>>>>>>>> 
>>>>>>>> thanks,
>>>>>>>> Hui
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Hui
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>> 
>>>>>>>>>> Yes, that's right.
>>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>> 
>>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>>> 
>>>>>>>>>> Dmitry.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>>> 
>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> Hui
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 
> <gasm_test.c>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/b2e19cda/attachment.htm>

From popov at uni-mainz.de  Tue May 15 04:34:59 2012
From: popov at uni-mainz.de (Anton Popov)
Date: Tue, 15 May 2012 11:34:59 +0200
Subject: [petsc-users] Using direct solvers in parallel
In-Reply-To: <4FB204DE.5070206@tu-dresden.de>
References: <4FB204DE.5070206@tu-dresden.de>
Message-ID: <4FB22343.6090201@uni-mainz.de>

On 5/15/12 9:25 AM, Thomas Witkowski wrote:
> I made some comparisons of using umfpack, superlu, superlu_dist and 
> mumps to solve systems with sparse matrices arising from finite 
> element method. The size of the matrices range from around 50000 to 
> more than 3 million unknowns. I used 1, 2, 4, 8 and 16 nodes to make 
> the benchmark. Now, I wonder that in all cases the sequential umfpack 
> was the fastest one. So even with 16 cores, superlu_dist and mumps are 
> slower. Can anybody of you confirm this observation? Are there any 
> other parallel direct solvers around which are more efficient?
>
> Thomas
You should definitely try PARDISO. On shared memory machine is 
unbeatable by any kind of MPI-based solver, and offers functionality for 
both symmetric and non-symmetric matrices. There is a version available 
from MKL library.

If you're fine enough with symmetric matrices, then this thing - HSL 
MA87 - is even better than PARDISO. It's a commercial solver, but it's 
free for academic use.

The only "problem" I see here is that both of them are difficult to use 
together with PETSc, since they're OMP-based. But difficult probably 
doesn't mean impossible.

Cheers,

Anton


From gmulas at oa-cagliari.inaf.it  Tue May 15 05:30:06 2012
From: gmulas at oa-cagliari.inaf.it (Giacomo Mulas)
Date: Tue, 15 May 2012 12:30:06 +0200 (CEST)
Subject: [petsc-users] How to set up -mat_mumps_icntl13 1 in the code?
In-Reply-To: <C4F9EBEC-89FF-4189-ACCD-C8254AB41F91@mcs.anl.gov>
References: <alpine.DEB.2.02.1205141744340.14621@capitanata.oa-cagliari.inaf.it>
	<CAMYG4G=CBDdfUw4nAmsPxFsFQON6zWRtY5ikLnTsYxEbxYsgaA@mail.gmail.com>
	<alpine.DEB.2.02.1205141854490.17844@capitanata.oa-cagliari.inaf.it>
	<C4F9EBEC-89FF-4189-ACCD-C8254AB41F91@mcs.anl.gov>
Message-ID: <alpine.DEB.2.02.1205151222470.15719@capitanata.oa-cagliari.inaf.it>

Hi Barry.

On Mon, 14 May 2012, Barry Smith wrote:

>    src/snes/examples/tutorials/ex22.c is an example of how one can set options in the program and then have the command line options override them.

Thanks! I am looking at that right now.

>  PetscInitialize(&argc,&argv,PETSC_NULL,help);
>
>  /* Hardwire several options; can be changed at command line */
>  ierr = PetscOptionsInsertString(common_options);CHKERRQ(ierr);
>  ierr = PetscOptionsInsert(&argc,&argv,PETSC_NULL);CHKERRQ(ierr);

So, if I understand correctly, I am supposed to call
PetscOptionsInsertString() to add any options I want to have as defaults,
then what is the call PetscOptionsInsert() for? PetscOptionsInsertString()
does not have effect until PetscOptionsInsert() is called?
Another thing which is not clear to me: in that example,
PetscOptionsSetFromOptions() is called before these "hardwired" options are
inserted. So (unless I do not understand the logic) command line options are
applied before hardwired options are inserted. And then
PetscOptionsSetFromOptions() is not called again. Does it mean that
hardwired inserted options override the command line (since they come
after)? Am I supposed to also call again PetscOptionsSetFromOptions() after
inserting new options for the inserted options to come into effect?

Thanks in advance, bye
Giacomo

-- 
_________________________________________________________________

Giacomo Mulas <gmulas at oa-cagliari.inaf.it>
_________________________________________________________________

OSSERVATORIO ASTRONOMICO DI CAGLIARI
Str. 54, Loc. Poggio dei Pini * 09012 Capoterra (CA)

Tel. (OAC): +39 070 71180 248     Fax : +39 070 71180 222
Tel. (UNICA): +39 070 675 4916
_________________________________________________________________

"When the storms are raging around you, stay right where you are"
                          (Freddy Mercury)
_________________________________________________________________

From karpeev at mcs.anl.gov  Tue May 15 07:01:25 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Tue, 15 May 2012 07:01:25 -0500
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
Message-ID: <CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>

Hui,
I'm trying to reproduce this problem, unsuccessfully, so far.
One thing that looks odd is that the output below claims the PC is of type
"asm", even though you are running with -dd_type gasm.  Could you verify
that's the correct output?

Here's the output I get with
${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type asm
-dd_ksp_view

PC Object:(dd_) 1 MPI processes
  type: asm
    Additive Schwarz: total subdomain blocks = 2, user-defined overlap
    Additive Schwarz: restriction/interpolation type - RESTRICT
    Local solve is same for all blocks, in the following KSP and PC objects:
    KSP Object:    (dd_sub_)     1 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test
      <snip>

and with
${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type
gasm -dd_ksp_view

PC Object:(dd_) 1 MPI processes
  type: gasm
  Generalized additive Schwarz:
  Restriction/interpolation type: RESTRICT
  user-defined overlap
  total number of subdomains = 2
  number of local subdomains = 2
  max number of local subdomains = 2
  [0:1] number of locally-supported subdomains = 2
  Subdomain solver info is as follows:
  <snip>

What convergence are you seeing with the two PC types?  It should be the
same with 1 and 2 procs for both PCASM and PCGASM.

Thanks.
Dmitry.

On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Dmitry,
>
> I got the newest petsc-dev and I run the test by
>
>     mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
>
> which gives the following output
>
> PC Object:(dd_) 1 MPI processes
>   type: asm
>     Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1
>                                               ^^^
>                                          note the above number, it should
> be 2
>
> While PCASM has no such problem.
>
> Thanks,
> Hui
>
>
>
> Hui,
>
> The convergence issue should be resolved in the latest petsc-dev.
> I'm attaching a slightly modified gasm_test.c (reflecting some upcoming
> API changes)
> that should verify that.
>
> Let me know if it works for you.
> Thanks.
> Dmitry.
> On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>> Hi Dmitry,
>>
>> thanks for useful hints.  Good day!
>>
>> Hui
>>
>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>
>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and
>> recreate the matrices (including calling
>> your modification subroutine), but not the subdomains or the subdomain
>> solvers.
>> If you just want to modify the submatrices, you can call
>> PC(G)ASMGetSubmatrices() and modify the matrices it returns
>> (in the same order as the subdomains were set). That's a bit of a
>> hack, since you will essentially be modifying the PC's internal data
>> structures.  As long as you are careful, you should be okay, since you
>> already effectively have the same type of access to the submatrices through
>> the Modify callback.
>>
>> Dmitry.
>>
>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>
>>> I just have a question about reuse of PCASM or PCGASM.
>>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>>> Next for the same linear system (matrix and RHS), I just want PCASM
>>> modify the submatrices (PCSetModifySubmatrices) in a different way, using
>>> the same routine for modifying but with
>>> different user context for the modifying routine.
>>>
>>> What can I do for this task?  Currently, I destroy the KSP and
>>> re-construct it. I guess
>>> even for PCASM I can re-use it because the partition of subdomains
>>> remain the same.
>>>
>>> Thanks!
>>>
>>>
>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>
>>> Hui,
>>> There've been several changes to PCGASM ahead of the new release.
>>> Let me go back and see if it affected the convergence problem.
>>> Dmitry.
>>>
>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>
>>>> Hi Dmitry,
>>>>
>>>> is there any news about PCGASM?
>>>>
>>>> thanks,
>>>> Hui
>>>>
>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>
>>>> Okay, thanks.
>>>> I'll take a look.
>>>>
>>>> Dmitry.
>>>>
>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com
>>>> > wrote:
>>>>
>>>>> For reference, my results are attached.
>>>>>
>>>>> asm1.txt for asm with 1 process,
>>>>> asm2.txt for asm with 2 processes,
>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers
>>>>> different from others)
>>>>> gasm2.txt for gasm with 2 processes
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> thank you,
>>>>> Hui
>>>>>
>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <
>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>
>>>>>>
>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <
>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>
>>>>>>> I have a new problem: the results from ASM and GASM are different
>>>>>>> and it seems
>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests
>>>>>>> are with
>>>>>>> each subdomain supported only by one subdomain. There are no
>>>>>>> problems when
>>>>>>> I did not modify submatrices.  But when I modify submatrices, there
>>>>>>> are problems
>>>>>>> with GASM but no problems with ASM.
>>>>>>>
>>>>>>> For example, I use two subdomains. In the first case each subdomain
>>>>>>> is supported by
>>>>>>> one processor and there seems no problem with GASM. But when I use
>>>>>>> run my program
>>>>>>> with only one proc. so that it supports both of the two subdomains,
>>>>>>> the iteration
>>>>>>> number is different from the first case and is much larger.  On the
>>>>>>> other hand
>>>>>>> ASM has no such problem.
>>>>>>>
>>>>>>
>>>>>> Are the solutions the same?
>>>>>> What problem are you solving?
>>>>>>
>>>>>>
>>>>>> Yes, the solutions are the same. That's why ASM gives the same
>>>>>> results with one or
>>>>>> two processors. But GASM did not.
>>>>>>
>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in
>>>>> the case of two domains per processor?
>>>>>
>>>>>> I'm solving the Helmholtz equation.  Maybe
>>>>>> I can prepare a simpler example to show this difference.
>>>>>>
>>>>> That would be helpful.
>>>>> Thanks.
>>>>>
>>>>> Dmitry.
>>>>>
>>>>>>
>>>>>>
>>>>>> Dmitry.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>
>>>>>>> You should be able to.
>>>>>>> This behavior is the same as in PCASM,
>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>
>>>>>>> Dmitry
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>
>>>>>>> Hi Dmitry,
>>>>>>>
>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes
>>>>>>> another question
>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the
>>>>>>> prototype
>>>>>>>
>>>>>>> func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>
>>>>>>> I think the coloumns from the parameter 'col' are always the same as
>>>>>>> the rows
>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only
>>>>>>> accepts
>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>
>>>>>>>
>>>>>>> As I tested, the row and col are always the same.
>>>>>>>
>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for
>>>>>>> the submat's
>>>>>>> in the above func()?
>>>>>>>
>>>>>>> thanks,
>>>>>>> Hui
>>>>>>>
>>>>>>>
>>>>>>> thanks,
>>>>>>> Hui
>>>>>>>
>>>>>>>
>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>
>>>>>>> Yes, that's right.
>>>>>>> There is no good way to help the user assemble the subdomains at the
>>>>>>> moment beyond the 2D stuff.
>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>
>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring"
>>>>>>> of indices to an array of ISs,
>>>>>>> each having the indices with the same color and the subcomm that
>>>>>>> supports that color. It is
>>>>>>> largely untested, though.  You could try using it and give us
>>>>>>> feedback on any problems you encounter.
>>>>>>>
>>>>>>> Dmitry.
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<mike.hui.zhang at hotmail.com>
>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>
>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain
>>>>>>>> supported by
>>>>>>>> multiple processors, shall I always create the arguments 'is[s]'
>>>>>>>> and 'is_local[s]'
>>>>>>>> in a subcommunicator consisting of processors supporting the
>>>>>>>> subdomain 's'?
>>>>>>>>
>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Hui
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
> <gasm_test.c>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/82de5e39/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Tue May 15 07:40:26 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 May 2012 06:40:26 -0600
Subject: [petsc-users] Modifying the structure of a matrix.
In-Reply-To: <1E35CF63-4E20-4BF6-A098-E2BBB22600E8@gmail.com>
References: <D9FAE42C-ABDA-4670-AEA4-889D2375195A@gmail.com>
	<CAM9tzS=S+xOSpUpPUFzz1zMz=8BY+xai6+PChKzXrxtbTZ46OA@mail.gmail.com>
	<D7D58CA5-7829-404A-81AB-0D3612F64F96@gmail.com>
	<32B30BCC-FA15-428F-B4D7-BBDFAA2837B9@mcs.anl.gov>
	<C3F18BF7-72C1-4605-8EF2-C9D71196F043@gmail.com>
	<4589BB71-BD27-4B72-B346-404AFF66141D@mcs.anl.gov>
	<1E35CF63-4E20-4BF6-A098-E2BBB22600E8@gmail.com>
Message-ID: <CAM9tzSmLO9k-QfS1_mBA156-DynRVLtXMSbufQ1rNHqPi1Xefw@mail.gmail.com>

On Mon, May 14, 2012 at 10:39 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> That is what I figure.
>
> I'm curious though if you need to manually determine the local row
> distribution after you do that.  (for example, say you completely remove
> all the values from the local range of one processor? that processor
> wouldn't be utilized unless you redistribute the matrix)
>

What sizes and method are we talking about? Usually additional (compact)
basis functions only make sense to add to one of a small number of
processes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/a1d9acc1/attachment.htm>

From jedbrown at mcs.anl.gov  Tue May 15 08:13:10 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 May 2012 07:13:10 -0600
Subject: [petsc-users] Linesearch algorithm
In-Reply-To: <CA+jOtD+_4gK-ySHKiCsfQaoYs5+r_jkUcv70cOjdhBoiaS98+w@mail.gmail.com>
References: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
	<C7256B25-6176-4431-9BB8-9663C0C71024@mcs.anl.gov>
	<CA+jOtD+_4gK-ySHKiCsfQaoYs5+r_jkUcv70cOjdhBoiaS98+w@mail.gmail.com>
Message-ID: <CAM9tzSm-Qi08oLAv5i5SkUQHZXV0Xmt-k+Hw4QuAeKhJDKxy9w@mail.gmail.com>

On Tue, May 15, 2012 at 12:26 AM, behzad baghapour <
behzad.baghapour at gmail.com> wrote:

> Thanks. Actually the jacobian which I obtain has a part of transient as J
> = M/dt + dR/dQ. Then I want to eliminate the time effect before calculating
> R^T*J*dQ as the descent direction. So, which way is correct in my case?


This sounds like a good way to stagnate. Many transient problems _require_
the "steady-state" residual to increase (climbing a hill) before it can
decrease.

Also, I can't tell from your description if you want the search direction
to come from the transient or steady problem.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/cd451057/attachment.htm>

From bsmith at mcs.anl.gov  Tue May 15 08:20:46 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 15 May 2012 08:20:46 -0500
Subject: [petsc-users] How to set up -mat_mumps_icntl13 1 in the code?
In-Reply-To: <alpine.DEB.2.02.1205151222470.15719@capitanata.oa-cagliari.inaf.it>
References: <alpine.DEB.2.02.1205141744340.14621@capitanata.oa-cagliari.inaf.it>
	<CAMYG4G=CBDdfUw4nAmsPxFsFQON6zWRtY5ikLnTsYxEbxYsgaA@mail.gmail.com>
	<alpine.DEB.2.02.1205141854490.17844@capitanata.oa-cagliari.inaf.it>
	<C4F9EBEC-89FF-4189-ACCD-C8254AB41F91@mcs.anl.gov>
	<alpine.DEB.2.02.1205151222470.15719@capitanata.oa-cagliari.inaf.it>
Message-ID: <7AC25B12-94CF-4A87-9565-760054ED1ED1@mcs.anl.gov>


On May 15, 2012, at 5:30 AM, Giacomo Mulas wrote:

> Hi Barry.
> 
> On Mon, 14 May 2012, Barry Smith wrote:
> 
>>   src/snes/examples/tutorials/ex22.c is an example of how one can set options in the program and then have the command line options override them.
> 
> Thanks! I am looking at that right now.
> 
>> PetscInitialize(&argc,&argv,PETSC_NULL,help);
>> 
>> /* Hardwire several options; can be changed at command line */
>> ierr = PetscOptionsInsertString(common_options);CHKERRQ(ierr);
>> ierr = PetscOptionsInsert(&argc,&argv,PETSC_NULL);CHKERRQ(ierr);
> 
> So, if I understand correctly, I am supposed to call
> PetscOptionsInsertString() to add any options I want to have as defaults,
> then what is the call PetscOptionsInsert() for?

    It "resets" the options database with the command line options. If you did not call this then the InsertString() options would not be overwritten with the command line options.

> PetscOptionsInsertString()
> does not have effect until PetscOptionsInsert() is called?
> Another thing which is not clear to me: in that example,
> PetscOptionsSetFromOptions() is called before these "hardwired" options are
> inserted. So (unless I do not understand the logic) command line options are
> applied before hardwired options are inserted. And then
> PetscOptionsSetFromOptions() is not called again. Does it mean that
> hardwired inserted options override the command line (since they come
> after)? Am I supposed to also call again PetscOptionsSetFromOptions() after
> inserting new options for the inserted options to come into effect?

   Ignore the PetscOptionsSetFromOptions() it has nothing to do with anything there (it is just in that example to test it).  It does not affect the options data base entries.

   Barry







> 
> Thanks in advance, bye
> Giacomo
> 
> -- 
> _________________________________________________________________
> 
> Giacomo Mulas <gmulas at oa-cagliari.inaf.it>
> _________________________________________________________________
> 
> OSSERVATORIO ASTRONOMICO DI CAGLIARI
> Str. 54, Loc. Poggio dei Pini * 09012 Capoterra (CA)
> 
> Tel. (OAC): +39 070 71180 248     Fax : +39 070 71180 222
> Tel. (UNICA): +39 070 675 4916
> _________________________________________________________________
> 
> "When the storms are raging around you, stay right where you are"
>                         (Freddy Mercury)
> _________________________________________________________________


From xdliang at gmail.com  Tue May 15 09:00:35 2012
From: xdliang at gmail.com (Xiangdong Liang)
Date: Tue, 15 May 2012 10:00:35 -0400
Subject: [petsc-users] Using direct solvers in parallel
In-Reply-To: <4FB22343.6090201@uni-mainz.de>
References: <4FB204DE.5070206@tu-dresden.de>
	<4FB22343.6090201@uni-mainz.de>
Message-ID: <CAGzjfYbv499Xy5oOC7mZCraBYhjfkMosWW=Je=OOtyVSQ5-tcQ@mail.gmail.com>

There is another sparse direct solver that PETSc supports: PaStix. You
can try it by --download-pastix.

Xiangdong


On Tue, May 15, 2012 at 5:34 AM, Anton Popov <popov at uni-mainz.de> wrote:
> On 5/15/12 9:25 AM, Thomas Witkowski wrote:
>>
>> I made some comparisons of using umfpack, superlu, superlu_dist and mumps
>> to solve systems with sparse matrices arising from finite element method.
>> The size of the matrices range from around 50000 to more than 3 million
>> unknowns. I used 1, 2, 4, 8 and 16 nodes to make the benchmark. Now, I
>> wonder that in all cases the sequential umfpack was the fastest one. So even
>> with 16 cores, superlu_dist and mumps are slower. Can anybody of you confirm
>> this observation? Are there any other parallel direct solvers around which
>> are more efficient?
>>
>> Thomas
>
> You should definitely try PARDISO. On shared memory machine is unbeatable by
> any kind of MPI-based solver, and offers functionality for both symmetric
> and non-symmetric matrices. There is a version available from MKL library.
>
> If you're fine enough with symmetric matrices, then this thing - HSL MA87 -
> is even better than PARDISO. It's a commercial solver, but it's free for
> academic use.
>
> The only "problem" I see here is that both of them are difficult to use
> together with PETSc, since they're OMP-based. But difficult probably doesn't
> mean impossible.
>
> Cheers,
>
> Anton
>

From hzhang at mcs.anl.gov  Tue May 15 09:40:45 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Tue, 15 May 2012 09:40:45 -0500
Subject: [petsc-users] Using direct solvers in parallel
In-Reply-To: <4FB20BD0.4040809@tu-dresden.de>
References: <4FB204DE.5070206@tu-dresden.de>
	<CAJ98EDo6sjx6AXZe98vdhe7Kp0UR_3Oefu5YHG165_QUUuMq5A@mail.gmail.com>
	<4FB20BD0.4040809@tu-dresden.de>
Message-ID: <CAGCphBuQaFxpmAkjde3c-Cv3Jr5H6-PsWYfoL6Du5BWJCXRnQg@mail.gmail.com>

Thomas :

>
>>  I attached my data to this mail. For the largest matrix, umfpack failed
> after allocating 4 GB of memory. I have not tried to figure out what's the
> problem there. As you can see, for these matrices the distributed solvers
> are


 umfpack is a sequential package. 4GB+ likely exceeds memory capability of
single core in your machine.

slower by a factor of 2 or 3 compared to umfpack. For all solvers, I have
> used the standard parameters, so I have not played around with the
> permutation strategies and such things. This may be also the reason why
> superlu is much slower than superlu_dist with just one core as it makes use
> of different col and row permutation strategies.


The data on superlu_dist and mumps look reasonable to me. The poor parallel
performance
is likely due to the multicore machine being used.
Try these runs on a machine that is desirable for distributed computing. See
http://www.mcs.anl.gov/petsc/documentation/faq.html#computers.

Hong

>
>  However, umpack will not work on a distributed memory machine.
>> My personal preference is to use superlu_dist in parallel. In my
>> experience using it as a coarse grid solver for multigrid, I find it
>> much more reliable than mumps. However, when mumps works, its is
>> typically slightly faster than superlu_dist. Again, not by a large
>> amount - never more than a factor of 2 faster.
>>
> In my codes I also make use of the distributed direct solvers for the
> coarse grid problems. I just wanted to make some tests how far away these
> solvers are from the sequential counterparts.
>
> Thomas
>
>
>> The failure rate using mumps is definitely higher (in my experience)
>> when running on large numbers of cores compared to superlu_dist. I've
>> never got to the bottom as to why it fails.
>>
>> Cheers,
>>   Dave
>>
>>
>> On 15 May 2012 09:25, Thomas Witkowski<thomas.witkowski at tu-**dresden.de<thomas.witkowski at tu-dresden.de>>
>>  wrote:
>>
>>> I made some comparisons of using umfpack, superlu, superlu_dist and
>>> mumps to
>>> solve systems with sparse matrices arising from finite element method.
>>> The
>>> size of the matrices range from around 50000 to more than 3 million
>>> unknowns. I used 1, 2, 4, 8 and 16 nodes to make the benchmark. Now, I
>>> wonder that in all cases the sequential umfpack was the fastest one. So
>>> even
>>> with 16 cores, superlu_dist and mumps are slower. Can anybody of you
>>> confirm
>>> this observation? Are there any other parallel direct solvers around
>>> which
>>> are more efficient?
>>>
>>> Thomas
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/243d1f4a/attachment.htm>

From mike.hui.zhang at hotmail.com  Tue May 15 09:55:22 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Tue, 15 May 2012 16:55:22 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
Message-ID: <BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>

Dmitry,

thanks for remind.  I have a new question about PCASM / PCGASM: 

can I get the restricted extension operators, which maps an overlapping subdomain solution
to the global domain?

Thanks!

On May 15, 2012, at 3:29 PM, Dmitry Karpeev wrote:

> There are some additional minor fixes that mostly have to do with outputting the subdomain information with  -pc_gasm_view_subdomains (in PCView()) and with -pc_gasm_print_subdomains (during PCSetUp()).
> You might want to pull those latest patches, but it won't interfere with your work if you don't use subdomain output.
> 
> Thanks.
> Dmitry.
> 
> On Tue, May 15, 2012 at 7:14 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> Dmitry,
> 
> thanks for reply. I re-download the codes and tried it again and now it works correctly!
> 
> Everything seems ok.
> 
> Thanks,
> Hui
> 
> 
> On May 15, 2012, at 2:01 PM, Dmitry Karpeev wrote:
> 
>> Hui,
>> I'm trying to reproduce this problem, unsuccessfully, so far.
>> One thing that looks odd is that the output below claims the PC is of type "asm", even though you are running with -dd_type gasm.  Could you verify that's the correct output?
>> 
>> Here's the output I get with 
>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type asm -dd_ksp_view
>> 
>> PC Object:(dd_) 1 MPI processes
>>   type: asm
>>     Additive Schwarz: total subdomain blocks = 2, user-defined overlap
>>     Additive Schwarz: restriction/interpolation type - RESTRICT
>>     Local solve is same for all blocks, in the following KSP and PC objects:
>>     KSP Object:    (dd_sub_)     1 MPI processes
>>       type: preonly
>>       maximum iterations=10000, initial guess is zero
>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>       left preconditioning
>>       using NONE norm type for convergence test
>>       <snip>
>> 
>> and with 
>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type gasm -dd_ksp_view
>> 
>> PC Object:(dd_) 1 MPI processes
>>   type: gasm
>>   Generalized additive Schwarz:
>>   Restriction/interpolation type: RESTRICT
>>   user-defined overlap
>>   total number of subdomains = 2
>>   number of local subdomains = 2
>>   max number of local subdomains = 2
>>   [0:1] number of locally-supported subdomains = 2
>>   Subdomain solver info is as follows:
>>   <snip>
>> 
>> What convergence are you seeing with the two PC types?  It should be the same with 1 and 2 procs for both PCASM and PCGASM.  
>> 
>> Thanks.
>> Dmitry.
>> 
>> On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> Dmitry,
>> 
>> I got the newest petsc-dev and I run the test by
>> 
>>     mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
>> 
>> which gives the following output
>> 
>> PC Object:(dd_) 1 MPI processes
>>   type: asm
>>     Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1
>>                                               ^^^
>>                                          note the above number, it should be 2 
>> 
>> While PCASM has no such problem.
>> 
>> Thanks,
>> Hui
>> 
>> 
>> 
>>> Hui,
>>> 
>>> The convergence issue should be resolved in the latest petsc-dev.
>>> I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) 
>>> that should verify that.
>>> 
>>> Let me know if it works for you.
>>> Thanks.
>>> Dmitry.
>>> On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> Hi Dmitry,
>>> 
>>> thanks for useful hints.  Good day!
>>> 
>>> Hui
>>> 
>>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>> 
>>>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
>>>> your modification subroutine), but not the subdomains or the subdomain solvers. 
>>>> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
>>>> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
>>>> 
>>>> Dmitry.
>>>> 
>>>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> I just have a question about reuse of PCASM or PCGASM.
>>>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>>>> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
>>>> different user context for the modifying routine.  
>>>> 
>>>> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
>>>> even for PCASM I can re-use it because the partition of subdomains remain the same.
>>>> 
>>>> Thanks!
>>>> 
>>>> 
>>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>> 
>>>>> Hui,
>>>>> There've been several changes to PCGASM ahead of the new release.
>>>>> Let me go back and see if it affected the convergence problem.
>>>>> Dmitry.
>>>>> 
>>>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> Hi Dmitry,
>>>>> 
>>>>> is there any news about PCGASM? 
>>>>> 
>>>>> thanks,
>>>>> Hui
>>>>> 
>>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>> 
>>>>>> Okay, thanks.
>>>>>> I'll take a look.
>>>>>> 
>>>>>> Dmitry.
>>>>>> 
>>>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> For reference, my results are attached.
>>>>>> 
>>>>>> asm1.txt for asm with 1 process,
>>>>>> asm2.txt for asm with 2 processes,
>>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>>>>> gasm2.txt for gasm with 2 processes
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> thank you,
>>>>>> Hui
>>>>>> 
>>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>> 
>>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>>>>> with GASM but no problems with ASM. 
>>>>>>>> 
>>>>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>>>>> number is different from the first case and is much larger.  On the other hand
>>>>>>>> ASM has no such problem.
>>>>>>>>  
>>>>>>>> Are the solutions the same?
>>>>>>>> What problem are you solving?
>>>>>>> 
>>>>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>>>>> two processors. But GASM did not.  
>>>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>>>>> I'm solving the Helmholtz equation.  Maybe 
>>>>>>> I can prepare a simpler example to show this difference.
>>>>>>> That would be helpful.  
>>>>>>> Thanks.
>>>>>>> 
>>>>>>> Dmitry. 
>>>>>>> 
>>>>>>>> 
>>>>>>>> Dmitry. 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>> 
>>>>>>>>> You should be able to. 
>>>>>>>>> This behavior is the same as in PCASM,
>>>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>>> 
>>>>>>>>> Dmitry
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Dmitry,
>>>>>>>>>>> 
>>>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>>>>> 
>>>>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>>>> 
>>>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>>>> 
>>>>>>>>>> As I tested, the row and col are always the same. 
>>>>>>>>>> 
>>>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>>>>> in the above func()?
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Hui
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> thanks,
>>>>>>>>>>> Hui
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Yes, that's right.
>>>>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>>>> 
>>>>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>>>>> 
>>>>>>>>>>>> Dmitry.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>>>>> 
>>>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Hui
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> <gasm_test.c>
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/96b4b7c2/attachment-0001.htm>

From karpeev at mcs.anl.gov  Tue May 15 10:19:47 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Tue, 15 May 2012 10:19:47 -0500
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
Message-ID: <CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>

On Tue, May 15, 2012 at 9:55 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Dmitry,
>
> thanks for remind.  I have a new question about PCASM / PCGASM:
>
> can I get the restricted extension operators, which maps an overlapping
> subdomain solution
> to the global domain?
>

I'm not exactly sure what you mean.

Are you talking about embedding the subdomain vectors back into the
original vector?
If so, there is substantial difference in how this is handled in ASM and
GASM:
ASM has a bunch of sequential vectors that can be scattered back into the
global vector,
because the subdomains are always local to at most one processor.

In the GASM case this is rather different, since the subdomains can live on
arbitrary subcommunicators
and there is only one scatter, which is applied to the direct sum of all
the subdomain vectors on the original communicator. I'm not sure how useful
that last scatter would be for you, since the details of the structure
of the direct sum vector are internal to GASM.

Dmitry.

>
> Thanks!
>
> On May 15, 2012, at 3:29 PM, Dmitry Karpeev wrote:
>
> There are some additional minor fixes that mostly have to do with
> outputting the subdomain information with  -pc_gasm_view_subdomains (in
> PCView()) and with -pc_gasm_print_subdomains (during PCSetUp()).
> You might want to pull those latest patches, but it won't interfere with
> your work if you don't use subdomain output.
>
> Thanks.
> Dmitry.
>
> On Tue, May 15, 2012 at 7:14 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>> Dmitry,
>>
>> thanks for reply. I re-download the codes and tried it again and now it
>> works correctly!
>>
>> Everything seems ok.
>>
>> Thanks,
>> Hui
>>
>>
>> On May 15, 2012, at 2:01 PM, Dmitry Karpeev wrote:
>>
>> Hui,
>> I'm trying to reproduce this problem, unsuccessfully, so far.
>> One thing that looks odd is that the output below claims the PC is of
>> type "asm", even though you are running with -dd_type gasm.  Could you
>> verify that's the correct output?
>>
>> Here's the output I get with
>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type
>> asm -dd_ksp_view
>>
>> PC Object:(dd_) 1 MPI processes
>>    type: asm
>>     Additive Schwarz: total subdomain blocks = 2, user-defined overlap
>>     Additive Schwarz: restriction/interpolation type - RESTRICT
>>     Local solve is same for all blocks, in the following KSP and PC
>> objects:
>>     KSP Object:    (dd_sub_)     1 MPI processes
>>       type: preonly
>>       maximum iterations=10000, initial guess is zero
>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>       left preconditioning
>>       using NONE norm type for convergence test
>>       <snip>
>>
>> and with
>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type
>> gasm -dd_ksp_view
>>
>> PC Object:(dd_) 1 MPI processes
>>   type: gasm
>>   Generalized additive Schwarz:
>>   Restriction/interpolation type: RESTRICT
>>   user-defined overlap
>>   total number of subdomains = 2
>>   number of local subdomains = 2
>>   max number of local subdomains = 2
>>   [0:1] number of locally-supported subdomains = 2
>>   Subdomain solver info is as follows:
>>   <snip>
>>
>> What convergence are you seeing with the two PC types?  It should be the
>> same with 1 and 2 procs for both PCASM and PCGASM.
>>
>> Thanks.
>> Dmitry.
>>
>> On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>
>>> Dmitry,
>>>
>>> I got the newest petsc-dev and I run the test by
>>>
>>>     mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
>>>
>>> which gives the following output
>>>
>>> PC Object:(dd_) 1 MPI processes
>>>   type: asm
>>>     Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1
>>>                                               ^^^
>>>                                          note the above number, it
>>> should be 2
>>>
>>> While PCASM has no such problem.
>>>
>>> Thanks,
>>> Hui
>>>
>>>
>>>
>>> Hui,
>>>
>>> The convergence issue should be resolved in the latest petsc-dev.
>>> I'm attaching a slightly modified gasm_test.c (reflecting some upcoming
>>> API changes)
>>> that should verify that.
>>>
>>> Let me know if it works for you.
>>> Thanks.
>>> Dmitry.
>>> On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>
>>>> Hi Dmitry,
>>>>
>>>> thanks for useful hints.  Good day!
>>>>
>>>> Hui
>>>>
>>>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>>>
>>>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy
>>>> and recreate the matrices (including calling
>>>> your modification subroutine), but not the subdomains or the subdomain
>>>> solvers.
>>>> If you just want to modify the submatrices, you can call
>>>> PC(G)ASMGetSubmatrices() and modify the matrices it returns
>>>> (in the same order as the subdomains were set). That's a bit of a
>>>> hack, since you will essentially be modifying the PC's internal data
>>>> structures.  As long as you are careful, you should be okay, since you
>>>> already effectively have the same type of access to the submatrices through
>>>> the Modify callback.
>>>>
>>>> Dmitry.
>>>>
>>>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com
>>>> > wrote:
>>>>
>>>>> I just have a question about reuse of PCASM or PCGASM.
>>>>> Suppose I have seted up the PCASM and related KSP and I solved one
>>>>> time.
>>>>> Next for the same linear system (matrix and RHS), I just want PCASM
>>>>> modify the submatrices (PCSetModifySubmatrices) in a different way, using
>>>>> the same routine for modifying but with
>>>>> different user context for the modifying routine.
>>>>>
>>>>> What can I do for this task?  Currently, I destroy the KSP and
>>>>> re-construct it. I guess
>>>>> even for PCASM I can re-use it because the partition of subdomains
>>>>> remain the same.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>>>
>>>>> Hui,
>>>>> There've been several changes to PCGASM ahead of the new release.
>>>>> Let me go back and see if it affected the convergence problem.
>>>>> Dmitry.
>>>>>
>>>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi Dmitry,
>>>>>>
>>>>>> is there any news about PCGASM?
>>>>>>
>>>>>> thanks,
>>>>>> Hui
>>>>>>
>>>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>>>
>>>>>> Okay, thanks.
>>>>>> I'll take a look.
>>>>>>
>>>>>> Dmitry.
>>>>>>
>>>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <
>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>
>>>>>>> For reference, my results are attached.
>>>>>>>
>>>>>>> asm1.txt for asm with 1 process,
>>>>>>> asm2.txt for asm with 2 processes,
>>>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers
>>>>>>> different from others)
>>>>>>> gasm2.txt for gasm with 2 processes
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> thank you,
>>>>>>> Hui
>>>>>>>
>>>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <
>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <
>>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>
>>>>>>>>> I have a new problem: the results from ASM and GASM are different
>>>>>>>>> and it seems
>>>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical
>>>>>>>>> tests are with
>>>>>>>>> each subdomain supported only by one subdomain. There are no
>>>>>>>>> problems when
>>>>>>>>> I did not modify submatrices.  But when I modify submatrices,
>>>>>>>>> there are problems
>>>>>>>>> with GASM but no problems with ASM.
>>>>>>>>>
>>>>>>>>> For example, I use two subdomains. In the first case each
>>>>>>>>> subdomain is supported by
>>>>>>>>> one processor and there seems no problem with GASM. But when I use
>>>>>>>>> run my program
>>>>>>>>> with only one proc. so that it supports both of the two
>>>>>>>>> subdomains, the iteration
>>>>>>>>> number is different from the first case and is much larger.  On
>>>>>>>>> the other hand
>>>>>>>>> ASM has no such problem.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Are the solutions the same?
>>>>>>>> What problem are you solving?
>>>>>>>>
>>>>>>>>
>>>>>>>> Yes, the solutions are the same. That's why ASM gives the same
>>>>>>>> results with one or
>>>>>>>> two processors. But GASM did not.
>>>>>>>>
>>>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in
>>>>>>> the case of two domains per processor?
>>>>>>>
>>>>>>>> I'm solving the Helmholtz equation.  Maybe
>>>>>>>> I can prepare a simpler example to show this difference.
>>>>>>>>
>>>>>>> That would be helpful.
>>>>>>> Thanks.
>>>>>>>
>>>>>>> Dmitry.
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dmitry.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>>>
>>>>>>>>> You should be able to.
>>>>>>>>> This behavior is the same as in PCASM,
>>>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>>>
>>>>>>>>> Dmitry
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>>>
>>>>>>>>> Hi Dmitry,
>>>>>>>>>
>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes
>>>>>>>>> another question
>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has
>>>>>>>>> the prototype
>>>>>>>>>
>>>>>>>>> func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>>
>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as
>>>>>>>>> the rows
>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only
>>>>>>>>> accepts
>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> As I tested, the row and col are always the same.
>>>>>>>>>
>>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping()
>>>>>>>>> for the submat's
>>>>>>>>> in the above func()?
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Hui
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Hui
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>>
>>>>>>>>> Yes, that's right.
>>>>>>>>> There is no good way to help the user assemble the subdomains at
>>>>>>>>> the moment beyond the 2D stuff.
>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>
>>>>>>>>> There is ISColoringToList() that is supposed to convert a
>>>>>>>>> "coloring" of indices to an array of ISs,
>>>>>>>>> each having the indices with the same color and the subcomm that
>>>>>>>>> supports that color. It is
>>>>>>>>> largely untested, though.  You could try using it and give us
>>>>>>>>> feedback on any problems you encounter.
>>>>>>>>>
>>>>>>>>> Dmitry.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<mike.hui.zhang at hotmail.com>
>>>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain
>>>>>>>>>> supported by
>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]'
>>>>>>>>>> and 'is_local[s]'
>>>>>>>>>> in a subcommunicator consisting of processors supporting the
>>>>>>>>>> subdomain 's'?
>>>>>>>>>>
>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Hui
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>> <gasm_test.c>
>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/a59a6a8c/attachment.htm>

From mike.hui.zhang at hotmail.com  Tue May 15 10:31:56 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Tue, 15 May 2012 17:31:56 +0200
Subject: [petsc-users] Fwd:  PCGASMSetLocalSubdomains
References: <75037327-2253-4CF4-9C5E-F9C319870C4A@hotmail.com>
Message-ID: <BLU0-SMTP2042AC81E04D7136CA8C48CCB1B0@phx.gbl>

 to petsc-user


> thanks for reply!
> On May 15, 2012, at 5:19 PM, Dmitry Karpeev wrote:
> 
>> 
>> 
>> On Tue, May 15, 2012 at 9:55 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> Dmitry,
>> 
>> thanks for remind.  I have a new question about PCASM / PCGASM: 
>> 
>> can I get the restricted extension operators, which maps an overlapping subdomain solution
>> to the global domain?
>>  
>> I'm not exactly sure what you mean.
>> 
>> Are you talking about embedding the subdomain vectors back into the original vector?
> 
> Yes, exactly.
> 
>> If so, there is substantial difference in how this is handled in ASM and GASM:
>> ASM has a bunch of sequential vectors that can be scattered back into the global vector,
> 
> Yes. Is there a method to get the scatter?
> 
>> because the subdomains are always local to at most one processor.  
>> 
>> In the GASM case this is rather different, since the subdomains can live on arbitrary subcommunicators 
>> and there is only one scatter, which is applied to the direct sum of all the subdomain vectors on the original communicator. I'm not sure how useful that last scatter would be for you, since the details of the structure 
>> of the direct sum vector are internal to GASM.
> 
> I would prefer to have the scatter for individual subdomain before direct sum. 
> But if I can get the scatter PCGASM has, maybe it is still useful. Please tell me how to get it?
> 
> Thanks!
> 
>> 
>> Dmitry. 
>> 
>> Thanks!
>> 
>> On May 15, 2012, at 3:29 PM, Dmitry Karpeev wrote:
>> 
>>> There are some additional minor fixes that mostly have to do with outputting the subdomain information with  -pc_gasm_view_subdomains (in PCView()) and with -pc_gasm_print_subdomains (during PCSetUp()).
>>> You might want to pull those latest patches, but it won't interfere with your work if you don't use subdomain output.
>>> 
>>> Thanks.
>>> Dmitry.
>>> 
>>> On Tue, May 15, 2012 at 7:14 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> Dmitry,
>>> 
>>> thanks for reply. I re-download the codes and tried it again and now it works correctly!
>>> 
>>> Everything seems ok.
>>> 
>>> Thanks,
>>> Hui
>>> 
>>> 
>>> On May 15, 2012, at 2:01 PM, Dmitry Karpeev wrote:
>>> 
>>>> Hui,
>>>> I'm trying to reproduce this problem, unsuccessfully, so far.
>>>> One thing that looks odd is that the output below claims the PC is of type "asm", even though you are running with -dd_type gasm.  Could you verify that's the correct output?
>>>> 
>>>> Here's the output I get with 
>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type asm -dd_ksp_view
>>>> 
>>>> PC Object:(dd_) 1 MPI processes
>>>>   type: asm
>>>>     Additive Schwarz: total subdomain blocks = 2, user-defined overlap
>>>>     Additive Schwarz: restriction/interpolation type - RESTRICT
>>>>     Local solve is same for all blocks, in the following KSP and PC objects:
>>>>     KSP Object:    (dd_sub_)     1 MPI processes
>>>>       type: preonly
>>>>       maximum iterations=10000, initial guess is zero
>>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>       left preconditioning
>>>>       using NONE norm type for convergence test
>>>>       <snip>
>>>> 
>>>> and with 
>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type gasm -dd_ksp_view
>>>> 
>>>> PC Object:(dd_) 1 MPI processes
>>>>   type: gasm
>>>>   Generalized additive Schwarz:
>>>>   Restriction/interpolation type: RESTRICT
>>>>   user-defined overlap
>>>>   total number of subdomains = 2
>>>>   number of local subdomains = 2
>>>>   max number of local subdomains = 2
>>>>   [0:1] number of locally-supported subdomains = 2
>>>>   Subdomain solver info is as follows:
>>>>   <snip>
>>>> 
>>>> What convergence are you seeing with the two PC types?  It should be the same with 1 and 2 procs for both PCASM and PCGASM.  
>>>> 
>>>> Thanks.
>>>> Dmitry.
>>>> 
>>>> On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> Dmitry,
>>>> 
>>>> I got the newest petsc-dev and I run the test by
>>>> 
>>>>     mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
>>>> 
>>>> which gives the following output
>>>> 
>>>> PC Object:(dd_) 1 MPI processes
>>>>   type: asm
>>>>     Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1
>>>>                                               ^^^
>>>>                                          note the above number, it should be 2 
>>>> 
>>>> While PCASM has no such problem.
>>>> 
>>>> Thanks,
>>>> Hui
>>>> 
>>>> 
>>>> 
>>>>> Hui,
>>>>> 
>>>>> The convergence issue should be resolved in the latest petsc-dev.
>>>>> I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) 
>>>>> that should verify that.
>>>>> 
>>>>> Let me know if it works for you.
>>>>> Thanks.
>>>>> Dmitry.
>>>>> On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> Hi Dmitry,
>>>>> 
>>>>> thanks for useful hints.  Good day!
>>>>> 
>>>>> Hui
>>>>> 
>>>>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>>>> 
>>>>>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
>>>>>> your modification subroutine), but not the subdomains or the subdomain solvers. 
>>>>>> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
>>>>>> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
>>>>>> 
>>>>>> Dmitry.
>>>>>> 
>>>>>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> I just have a question about reuse of PCASM or PCGASM.
>>>>>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>>>>>> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
>>>>>> different user context for the modifying routine.  
>>>>>> 
>>>>>> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
>>>>>> even for PCASM I can re-use it because the partition of subdomains remain the same.
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>> 
>>>>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>>>> 
>>>>>>> Hui,
>>>>>>> There've been several changes to PCGASM ahead of the new release.
>>>>>>> Let me go back and see if it affected the convergence problem.
>>>>>>> Dmitry.
>>>>>>> 
>>>>>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>> Hi Dmitry,
>>>>>>> 
>>>>>>> is there any news about PCGASM? 
>>>>>>> 
>>>>>>> thanks,
>>>>>>> Hui
>>>>>>> 
>>>>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>>>> 
>>>>>>>> Okay, thanks.
>>>>>>>> I'll take a look.
>>>>>>>> 
>>>>>>>> Dmitry.
>>>>>>>> 
>>>>>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>> For reference, my results are attached.
>>>>>>>> 
>>>>>>>> asm1.txt for asm with 1 process,
>>>>>>>> asm2.txt for asm with 2 processes,
>>>>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>>>>>>> gasm2.txt for gasm with 2 processes
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> thank you,
>>>>>>>> Hui
>>>>>>>> 
>>>>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>>>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>>>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>>>>>>> with GASM but no problems with ASM. 
>>>>>>>>>> 
>>>>>>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>>>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>>>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>>>>>>> number is different from the first case and is much larger.  On the other hand
>>>>>>>>>> ASM has no such problem.
>>>>>>>>>>  
>>>>>>>>>> Are the solutions the same?
>>>>>>>>>> What problem are you solving?
>>>>>>>>> 
>>>>>>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>>>>>>> two processors. But GASM did not.  
>>>>>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>>>>>>> I'm solving the Helmholtz equation.  Maybe 
>>>>>>>>> I can prepare a simpler example to show this difference.
>>>>>>>>> That would be helpful.  
>>>>>>>>> Thanks.
>>>>>>>>> 
>>>>>>>>> Dmitry. 
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Dmitry. 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>>>> 
>>>>>>>>>>> You should be able to. 
>>>>>>>>>>> This behavior is the same as in PCASM,
>>>>>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>>>>> 
>>>>>>>>>>> Dmitry
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Dmitry,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>>>>>>> 
>>>>>>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>>>>>> 
>>>>>>>>>>>> As I tested, the row and col are always the same. 
>>>>>>>>>>>> 
>>>>>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>>>>>>> in the above func()?
>>>>>>>>>>>> 
>>>>>>>>>>>> thanks,
>>>>>>>>>>>> Hui
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>> Hui
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Yes, that's right.
>>>>>>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Dmitry.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Hui
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> <gasm_test.c>
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/4b486dcf/attachment-0001.htm>

From mike.hui.zhang at hotmail.com  Tue May 15 11:05:32 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Tue, 15 May 2012 18:05:32 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
Message-ID: <BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>


On May 15, 2012, at 5:42 PM, Dmitry Karpeev wrote:

> 
> 
> On Tue, May 15, 2012 at 10:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> thanks for reply!
> On May 15, 2012, at 5:19 PM, Dmitry Karpeev wrote:
> 
>> 
>> 
>> On Tue, May 15, 2012 at 9:55 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> Dmitry,
>> 
>> thanks for remind.  I have a new question about PCASM / PCGASM: 
>> 
>> can I get the restricted extension operators, which maps an overlapping subdomain solution
>> to the global domain?
>>  
>> I'm not exactly sure what you mean.
>> 
>> Are you talking about embedding the subdomain vectors back into the original vector?
> 
> Yes, exactly.
> 
>> If so, there is substantial difference in how this is handled in ASM and GASM:
>> ASM has a bunch of sequential vectors that can be scattered back into the global vector,
> 
> Yes. Is there a method to get the scatter?
> 
> In the ASM case it's a bunch of scatters -- one for each subdomain.  Currently there is no method to 
> retrieve them.  

this hint is very helpful. Thanks!  

> What requires this functionality?

I am writing some modified ASM method. In construction of energy minimization coarse basis,
I need to solve individual subdomain problems and not to sum them, just to extend them separately.
I wonder whether you guys have ever done this coarse basis.

Thanks,
Hui

> 
> In the ASM case you can construct the scatters yourself easily enough, 
> since you have all of the requisite information -- the array of subdomain ISs and the global vector x.
> The only piece of data you might not have is the set of outer subdomains that have been obtained
> by applying overlap increase to the original inner (nonoverlapping) subdomains.
> 
>> because the subdomains are always local to at most one processor.  
>> 
>> In the GASM case this is rather different, since the subdomains can live on arbitrary subcommunicators 
>> and there is only one scatter, which is applied to the direct sum of all the subdomain vectors on the original communicator. I'm not sure how useful that last scatter would be for you, since the details of the structure 
>> of the direct sum vector are internal to GASM.
> 
> I would prefer to have the scatter for individual subdomain before direct sum. 
> But if I can get the scatter PCGASM has, maybe it is still useful. Please tell me how to get it?
> There are no individual subdomain scatters, but, as in the case of ASM, you can  construct them
> easily enough, except that those would have to operate on subcommunicators.
> In GASM we pack them into a single scatter on the original communicator. Currently there is no method
> to expose this scatter. Why do you need this functionality? 
> 
> Dmitry.
> Thanks!
> 
>> 
>> Dmitry. 
>> 
>> Thanks!
>> 
>> On May 15, 2012, at 3:29 PM, Dmitry Karpeev wrote:
>> 
>>> There are some additional minor fixes that mostly have to do with outputting the subdomain information with  -pc_gasm_view_subdomains (in PCView()) and with -pc_gasm_print_subdomains (during PCSetUp()).
>>> You might want to pull those latest patches, but it won't interfere with your work if you don't use subdomain output.
>>> 
>>> Thanks.
>>> Dmitry.
>>> 
>>> On Tue, May 15, 2012 at 7:14 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> Dmitry,
>>> 
>>> thanks for reply. I re-download the codes and tried it again and now it works correctly!
>>> 
>>> Everything seems ok.
>>> 
>>> Thanks,
>>> Hui
>>> 
>>> 
>>> On May 15, 2012, at 2:01 PM, Dmitry Karpeev wrote:
>>> 
>>>> Hui,
>>>> I'm trying to reproduce this problem, unsuccessfully, so far.
>>>> One thing that looks odd is that the output below claims the PC is of type "asm", even though you are running with -dd_type gasm.  Could you verify that's the correct output?
>>>> 
>>>> Here's the output I get with 
>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type asm -dd_ksp_view
>>>> 
>>>> PC Object:(dd_) 1 MPI processes
>>>>   type: asm
>>>>     Additive Schwarz: total subdomain blocks = 2, user-defined overlap
>>>>     Additive Schwarz: restriction/interpolation type - RESTRICT
>>>>     Local solve is same for all blocks, in the following KSP and PC objects:
>>>>     KSP Object:    (dd_sub_)     1 MPI processes
>>>>       type: preonly
>>>>       maximum iterations=10000, initial guess is zero
>>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>       left preconditioning
>>>>       using NONE norm type for convergence test
>>>>       <snip>
>>>> 
>>>> and with 
>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type gasm -dd_ksp_view
>>>> 
>>>> PC Object:(dd_) 1 MPI processes
>>>>   type: gasm
>>>>   Generalized additive Schwarz:
>>>>   Restriction/interpolation type: RESTRICT
>>>>   user-defined overlap
>>>>   total number of subdomains = 2
>>>>   number of local subdomains = 2
>>>>   max number of local subdomains = 2
>>>>   [0:1] number of locally-supported subdomains = 2
>>>>   Subdomain solver info is as follows:
>>>>   <snip>
>>>> 
>>>> What convergence are you seeing with the two PC types?  It should be the same with 1 and 2 procs for both PCASM and PCGASM.  
>>>> 
>>>> Thanks.
>>>> Dmitry.
>>>> 
>>>> On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> Dmitry,
>>>> 
>>>> I got the newest petsc-dev and I run the test by
>>>> 
>>>>     mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
>>>> 
>>>> which gives the following output
>>>> 
>>>> PC Object:(dd_) 1 MPI processes
>>>>   type: asm
>>>>     Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1
>>>>                                               ^^^
>>>>                                          note the above number, it should be 2 
>>>> 
>>>> While PCASM has no such problem.
>>>> 
>>>> Thanks,
>>>> Hui
>>>> 
>>>> 
>>>> 
>>>>> Hui,
>>>>> 
>>>>> The convergence issue should be resolved in the latest petsc-dev.
>>>>> I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) 
>>>>> that should verify that.
>>>>> 
>>>>> Let me know if it works for you.
>>>>> Thanks.
>>>>> Dmitry.
>>>>> On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> Hi Dmitry,
>>>>> 
>>>>> thanks for useful hints.  Good day!
>>>>> 
>>>>> Hui
>>>>> 
>>>>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>>>> 
>>>>>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
>>>>>> your modification subroutine), but not the subdomains or the subdomain solvers. 
>>>>>> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
>>>>>> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
>>>>>> 
>>>>>> Dmitry.
>>>>>> 
>>>>>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> I just have a question about reuse of PCASM or PCGASM.
>>>>>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>>>>>> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
>>>>>> different user context for the modifying routine.  
>>>>>> 
>>>>>> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
>>>>>> even for PCASM I can re-use it because the partition of subdomains remain the same.
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>> 
>>>>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>>>> 
>>>>>>> Hui,
>>>>>>> There've been several changes to PCGASM ahead of the new release.
>>>>>>> Let me go back and see if it affected the convergence problem.
>>>>>>> Dmitry.
>>>>>>> 
>>>>>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>> Hi Dmitry,
>>>>>>> 
>>>>>>> is there any news about PCGASM? 
>>>>>>> 
>>>>>>> thanks,
>>>>>>> Hui
>>>>>>> 
>>>>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>>>> 
>>>>>>>> Okay, thanks.
>>>>>>>> I'll take a look.
>>>>>>>> 
>>>>>>>> Dmitry.
>>>>>>>> 
>>>>>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>> For reference, my results are attached.
>>>>>>>> 
>>>>>>>> asm1.txt for asm with 1 process,
>>>>>>>> asm2.txt for asm with 2 processes,
>>>>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>>>>>>> gasm2.txt for gasm with 2 processes
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> thank you,
>>>>>>>> Hui
>>>>>>>> 
>>>>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>>>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>>>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>>>>>>> with GASM but no problems with ASM. 
>>>>>>>>>> 
>>>>>>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>>>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>>>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>>>>>>> number is different from the first case and is much larger.  On the other hand
>>>>>>>>>> ASM has no such problem.
>>>>>>>>>>  
>>>>>>>>>> Are the solutions the same?
>>>>>>>>>> What problem are you solving?
>>>>>>>>> 
>>>>>>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>>>>>>> two processors. But GASM did not.  
>>>>>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>>>>>>> I'm solving the Helmholtz equation.  Maybe 
>>>>>>>>> I can prepare a simpler example to show this difference.
>>>>>>>>> That would be helpful.  
>>>>>>>>> Thanks.
>>>>>>>>> 
>>>>>>>>> Dmitry. 
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Dmitry. 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>>>> 
>>>>>>>>>>> You should be able to. 
>>>>>>>>>>> This behavior is the same as in PCASM,
>>>>>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>>>>> 
>>>>>>>>>>> Dmitry
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Dmitry,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>>>>>>> 
>>>>>>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>>>>>> 
>>>>>>>>>>>> As I tested, the row and col are always the same. 
>>>>>>>>>>>> 
>>>>>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>>>>>>> in the above func()?
>>>>>>>>>>>> 
>>>>>>>>>>>> thanks,
>>>>>>>>>>>> Hui
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>> Hui
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Yes, that's right.
>>>>>>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Dmitry.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Hui
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> <gasm_test.c>
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/3cf175e4/attachment-0001.htm>

From karpeev at mcs.anl.gov  Tue May 15 11:06:53 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Tue, 15 May 2012 11:06:53 -0500
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
Message-ID: <CA+Y_9UJ3TsAqmycjqgFBzedo3nkCM8zkNC=SsR1Sco9OqjSxGw@mail.gmail.com>

On Tue, May 15, 2012 at 11:05 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

>
> On May 15, 2012, at 5:42 PM, Dmitry Karpeev wrote:
>
>
>
> On Tue, May 15, 2012 at 10:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>> thanks for reply!
>> On May 15, 2012, at 5:19 PM, Dmitry Karpeev wrote:
>>
>>
>>
>> On Tue, May 15, 2012 at 9:55 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>
>>> Dmitry,
>>>
>>> thanks for remind.  I have a new question about PCASM / PCGASM:
>>>
>>> can I get the restricted extension operators, which maps an overlapping
>>> subdomain solution
>>> to the global domain?
>>>
>>
>> I'm not exactly sure what you mean.
>>
>> Are you talking about embedding the subdomain vectors back into the
>> original vector?
>>
>>
>> Yes, exactly.
>>
>> If so, there is substantial difference in how this is handled in ASM and
>> GASM:
>> ASM has a bunch of sequential vectors that can be scattered back into the
>> global vector,
>>
>>
>> Yes. Is there a method to get the scatter?
>>
>
> In the ASM case it's a bunch of scatters -- one for each subdomain.
>  Currently there is no method to
> retrieve them.
>
>
> this hint is very helpful. Thanks!
>
> What requires this functionality?
>
>
> I am writing some modified ASM method. In construction of energy
> minimization coarse basis,
> I need to solve individual subdomain problems and not to sum them, just to
> extend them separately.
> I wonder whether you guys have ever done this coarse basis.
>
Is there a reference where the basis is described?
Dmtiry.

>
> Thanks,
> Hui
>
>
> In the ASM case you can construct the scatters yourself easily enough,
> since you have all of the requisite information -- the array of subdomain
> ISs and the global vector x.
> The only piece of data you might not have is the set of outer subdomains
> that have been obtained
> by applying overlap increase to the original inner (nonoverlapping)
> subdomains.
>
>>
>> because the subdomains are always local to at most one processor.
>>
>> In the GASM case this is rather different, since the subdomains can live
>> on arbitrary subcommunicators
>> and there is only one scatter, which is applied to the direct sum of all
>> the subdomain vectors on the original communicator. I'm not sure how useful
>> that last scatter would be for you, since the details of the structure
>> of the direct sum vector are internal to GASM.
>>
>>
>> I would prefer to have the scatter for individual subdomain before direct
>> sum.
>> But if I can get the scatter PCGASM has, maybe it is still useful. Please
>> tell me how to get it?
>>
> There are no individual subdomain scatters, but, as in the case of ASM,
> you can  construct them
> easily enough, except that those would have to operate on subcommunicators.
> In GASM we pack them into a single scatter on the original communicator.
> Currently there is no method
> to expose this scatter. Why do you need this functionality?
>
> Dmitry.
>
>> Thanks!
>>
>>
>> Dmitry.
>>
>>>
>>> Thanks!
>>>
>>> On May 15, 2012, at 3:29 PM, Dmitry Karpeev wrote:
>>>
>>> There are some additional minor fixes that mostly have to do with
>>> outputting the subdomain information with  -pc_gasm_view_subdomains (in
>>> PCView()) and with -pc_gasm_print_subdomains (during PCSetUp()).
>>> You might want to pull those latest patches, but it won't interfere with
>>> your work if you don't use subdomain output.
>>>
>>> Thanks.
>>> Dmitry.
>>>
>>> On Tue, May 15, 2012 at 7:14 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>
>>>> Dmitry,
>>>>
>>>> thanks for reply. I re-download the codes and tried it again and now it
>>>> works correctly!
>>>>
>>>> Everything seems ok.
>>>>
>>>> Thanks,
>>>> Hui
>>>>
>>>>
>>>> On May 15, 2012, at 2:01 PM, Dmitry Karpeev wrote:
>>>>
>>>> Hui,
>>>> I'm trying to reproduce this problem, unsuccessfully, so far.
>>>> One thing that looks odd is that the output below claims the PC is of
>>>> type "asm", even though you are running with -dd_type gasm.  Could you
>>>> verify that's the correct output?
>>>>
>>>> Here's the output I get with
>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type
>>>> asm -dd_ksp_view
>>>>
>>>> PC Object:(dd_) 1 MPI processes
>>>>    type: asm
>>>>     Additive Schwarz: total subdomain blocks = 2, user-defined overlap
>>>>     Additive Schwarz: restriction/interpolation type - RESTRICT
>>>>     Local solve is same for all blocks, in the following KSP and PC
>>>> objects:
>>>>     KSP Object:    (dd_sub_)     1 MPI processes
>>>>       type: preonly
>>>>       maximum iterations=10000, initial guess is zero
>>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>       left preconditioning
>>>>       using NONE norm type for convergence test
>>>>       <snip>
>>>>
>>>> and with
>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type
>>>> gasm -dd_ksp_view
>>>>
>>>> PC Object:(dd_) 1 MPI processes
>>>>   type: gasm
>>>>   Generalized additive Schwarz:
>>>>   Restriction/interpolation type: RESTRICT
>>>>   user-defined overlap
>>>>   total number of subdomains = 2
>>>>   number of local subdomains = 2
>>>>   max number of local subdomains = 2
>>>>   [0:1] number of locally-supported subdomains = 2
>>>>   Subdomain solver info is as follows:
>>>>   <snip>
>>>>
>>>> What convergence are you seeing with the two PC types?  It should be
>>>> the same with 1 and 2 procs for both PCASM and PCGASM.
>>>>
>>>> Thanks.
>>>> Dmitry.
>>>>
>>>> On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>>
>>>>> Dmitry,
>>>>>
>>>>> I got the newest petsc-dev and I run the test by
>>>>>
>>>>>     mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
>>>>>
>>>>> which gives the following output
>>>>>
>>>>> PC Object:(dd_) 1 MPI processes
>>>>>   type: asm
>>>>>     Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1
>>>>>                                               ^^^
>>>>>                                          note the above number, it
>>>>> should be 2
>>>>>
>>>>> While PCASM has no such problem.
>>>>>
>>>>> Thanks,
>>>>> Hui
>>>>>
>>>>>
>>>>>
>>>>> Hui,
>>>>>
>>>>> The convergence issue should be resolved in the latest petsc-dev.
>>>>> I'm attaching a slightly modified gasm_test.c (reflecting some
>>>>> upcoming API changes)
>>>>> that should verify that.
>>>>>
>>>>> Let me know if it works for you.
>>>>> Thanks.
>>>>> Dmitry.
>>>>> On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <
>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>
>>>>>> Hi Dmitry,
>>>>>>
>>>>>> thanks for useful hints.  Good day!
>>>>>>
>>>>>> Hui
>>>>>>
>>>>>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>>>>>
>>>>>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy
>>>>>> and recreate the matrices (including calling
>>>>>> your modification subroutine), but not the subdomains or the
>>>>>> subdomain solvers.
>>>>>> If you just want to modify the submatrices, you can call
>>>>>> PC(G)ASMGetSubmatrices() and modify the matrices it returns
>>>>>> (in the same order as the subdomains were set). That's a bit of a
>>>>>> hack, since you will essentially be modifying the PC's internal data
>>>>>> structures.  As long as you are careful, you should be okay, since you
>>>>>> already effectively have the same type of access to the submatrices through
>>>>>> the Modify callback.
>>>>>>
>>>>>> Dmitry.
>>>>>>
>>>>>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <
>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>
>>>>>>> I just have a question about reuse of PCASM or PCGASM.
>>>>>>> Suppose I have seted up the PCASM and related KSP and I solved one
>>>>>>> time.
>>>>>>> Next for the same linear system (matrix and RHS), I just want PCASM
>>>>>>> modify the submatrices (PCSetModifySubmatrices) in a different way, using
>>>>>>> the same routine for modifying but with
>>>>>>> different user context for the modifying routine.
>>>>>>>
>>>>>>> What can I do for this task?  Currently, I destroy the KSP and
>>>>>>> re-construct it. I guess
>>>>>>> even for PCASM I can re-use it because the partition of subdomains
>>>>>>> remain the same.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>>>>>
>>>>>>> Hui,
>>>>>>> There've been several changes to PCGASM ahead of the new release.
>>>>>>> Let me go back and see if it affected the convergence problem.
>>>>>>> Dmitry.
>>>>>>>
>>>>>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <
>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Dmitry,
>>>>>>>>
>>>>>>>> is there any news about PCGASM?
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Hui
>>>>>>>>
>>>>>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>>>>>
>>>>>>>> Okay, thanks.
>>>>>>>> I'll take a look.
>>>>>>>>
>>>>>>>> Dmitry.
>>>>>>>>
>>>>>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <
>>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>
>>>>>>>>> For reference, my results are attached.
>>>>>>>>>
>>>>>>>>> asm1.txt for asm with 1 process,
>>>>>>>>> asm2.txt for asm with 2 processes,
>>>>>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers
>>>>>>>>> different from others)
>>>>>>>>> gasm2.txt for gasm with 2 processes
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> thank you,
>>>>>>>>> Hui
>>>>>>>>>
>>>>>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <
>>>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <
>>>>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I have a new problem: the results from ASM and GASM are
>>>>>>>>>>> different and it seems
>>>>>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical
>>>>>>>>>>> tests are with
>>>>>>>>>>> each subdomain supported only by one subdomain. There are no
>>>>>>>>>>> problems when
>>>>>>>>>>> I did not modify submatrices.  But when I modify submatrices,
>>>>>>>>>>> there are problems
>>>>>>>>>>> with GASM but no problems with ASM.
>>>>>>>>>>>
>>>>>>>>>>> For example, I use two subdomains. In the first case each
>>>>>>>>>>> subdomain is supported by
>>>>>>>>>>> one processor and there seems no problem with GASM. But when I
>>>>>>>>>>> use run my program
>>>>>>>>>>> with only one proc. so that it supports both of the two
>>>>>>>>>>> subdomains, the iteration
>>>>>>>>>>> number is different from the first case and is much larger.  On
>>>>>>>>>>> the other hand
>>>>>>>>>>> ASM has no such problem.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Are the solutions the same?
>>>>>>>>>> What problem are you solving?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yes, the solutions are the same. That's why ASM gives the same
>>>>>>>>>> results with one or
>>>>>>>>>> two processors. But GASM did not.
>>>>>>>>>>
>>>>>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions
>>>>>>>>> in the case of two domains per processor?
>>>>>>>>>
>>>>>>>>>> I'm solving the Helmholtz equation.  Maybe
>>>>>>>>>> I can prepare a simpler example to show this difference.
>>>>>>>>>>
>>>>>>>>> That would be helpful.
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> Dmitry.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Dmitry.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>>>>>
>>>>>>>>>>> You should be able to.
>>>>>>>>>>> This behavior is the same as in PCASM,
>>>>>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>>>>>> I am in transit right now, but I can take a closer look in
>>>>>>>>>>> Friday.
>>>>>>>>>>>
>>>>>>>>>>> Dmitry
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Dmitry,
>>>>>>>>>>>
>>>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes
>>>>>>>>>>> another question
>>>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has
>>>>>>>>>>> the prototype
>>>>>>>>>>>
>>>>>>>>>>> func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void
>>>>>>>>>>> *ctx);
>>>>>>>>>>>
>>>>>>>>>>> I think the coloumns from the parameter 'col' are always the
>>>>>>>>>>> same as the rows
>>>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains()
>>>>>>>>>>> only accepts
>>>>>>>>>>> index sets but not rows and columns. Has I misunderstood
>>>>>>>>>>> something?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> As I tested, the row and col are always the same.
>>>>>>>>>>>
>>>>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping()
>>>>>>>>>>> for the submat's
>>>>>>>>>>> in the above func()?
>>>>>>>>>>>
>>>>>>>>>>> thanks,
>>>>>>>>>>> Hui
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> thanks,
>>>>>>>>>>> Hui
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>>>>
>>>>>>>>>>> Yes, that's right.
>>>>>>>>>>> There is no good way to help the user assemble the subdomains at
>>>>>>>>>>> the moment beyond the 2D stuff.
>>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>>>
>>>>>>>>>>> There is ISColoringToList() that is supposed to convert a
>>>>>>>>>>> "coloring" of indices to an array of ISs,
>>>>>>>>>>> each having the indices with the same color and the subcomm that
>>>>>>>>>>> supports that color. It is
>>>>>>>>>>> largely untested, though.  You could try using it and give us
>>>>>>>>>>> feedback on any problems you encounter.
>>>>>>>>>>>
>>>>>>>>>>> Dmitry.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<mike.hui.zhang at hotmail.com>
>>>>>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain
>>>>>>>>>>>> supported by
>>>>>>>>>>>> multiple processors, shall I always create the arguments
>>>>>>>>>>>> 'is[s]' and 'is_local[s]'
>>>>>>>>>>>> in a subcommunicator consisting of processors supporting the
>>>>>>>>>>>> subdomain 's'?
>>>>>>>>>>>>
>>>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Hui
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>> <gasm_test.c>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/65dae8c7/attachment-0001.htm>

From mike.hui.zhang at hotmail.com  Tue May 15 11:11:46 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Tue, 15 May 2012 18:11:46 +0200
Subject: [petsc-users] Fwd:  PCGASMSetLocalSubdomains
References: <59EB7A9E-CAC0-4508-988B-CC0CEA7C3226@hotmail.com>
Message-ID: <BLU0-SMTP17291C3CF86EF473B2F849CCB1B0@phx.gbl>



Begin forwarded message:

> From: Hui Zhang <mike.hui.zhang at hotmail.com>
> Date: May 15, 2012 6:11:10 PM GMT+02:00
> To: Dmitry Karpeev <karpeev at mcs.anl.gov>
> Subject: Re: [petsc-users] PCGASMSetLocalSubdomains
> 
> 
> On May 15, 2012, at 6:06 PM, Dmitry Karpeev wrote:
> 
>> 
>> 
>> On Tue, May 15, 2012 at 11:05 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> 
>> On May 15, 2012, at 5:42 PM, Dmitry Karpeev wrote:
>> 
>>> 
>>> 
>>> On Tue, May 15, 2012 at 10:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> thanks for reply!
>>> On May 15, 2012, at 5:19 PM, Dmitry Karpeev wrote:
>>> 
>>>> 
>>>> 
>>>> On Tue, May 15, 2012 at 9:55 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> Dmitry,
>>>> 
>>>> thanks for remind.  I have a new question about PCASM / PCGASM: 
>>>> 
>>>> can I get the restricted extension operators, which maps an overlapping subdomain solution
>>>> to the global domain?
>>>>  
>>>> I'm not exactly sure what you mean.
>>>> 
>>>> Are you talking about embedding the subdomain vectors back into the original vector?
>>> 
>>> Yes, exactly.
>>> 
>>>> If so, there is substantial difference in how this is handled in ASM and GASM:
>>>> ASM has a bunch of sequential vectors that can be scattered back into the global vector,
>>> 
>>> Yes. Is there a method to get the scatter?
>>> 
>>> In the ASM case it's a bunch of scatters -- one for each subdomain.  Currently there is no method to 
>>> retrieve them.  
>> 
>> this hint is very helpful. Thanks!  
>> 
>>> What requires this functionality?
>> 
>> I am writing some modified ASM method. In construction of energy minimization coarse basis,
>> I need to solve individual subdomain problems and not to sum them, just to extend them separately.
>> I wonder whether you guys have ever done this coarse basis.
>> Is there a reference where the basis is described?
> 
> Yes, see Mandel & Dorhamann' paper http://ccm.ucdenver.edu/reports/rep194.pdf ,
> but this is for non-overlapping. For overlapping we can see Leong & Widlund 
> http://cs.nyu.edu/web/Research/TechReports/TR2008-914/TR2008-914.pdf
> 
>> Dmtiry. 
>> 
>> Thanks,
>> Hui
>> 
>>> 
>>> In the ASM case you can construct the scatters yourself easily enough, 
>>> since you have all of the requisite information -- the array of subdomain ISs and the global vector x.
>>> The only piece of data you might not have is the set of outer subdomains that have been obtained
>>> by applying overlap increase to the original inner (nonoverlapping) subdomains.
>>> 
>>>> because the subdomains are always local to at most one processor.  
>>>> 
>>>> In the GASM case this is rather different, since the subdomains can live on arbitrary subcommunicators 
>>>> and there is only one scatter, which is applied to the direct sum of all the subdomain vectors on the original communicator. I'm not sure how useful that last scatter would be for you, since the details of the structure 
>>>> of the direct sum vector are internal to GASM.
>>> 
>>> I would prefer to have the scatter for individual subdomain before direct sum. 
>>> But if I can get the scatter PCGASM has, maybe it is still useful. Please tell me how to get it?
>>> There are no individual subdomain scatters, but, as in the case of ASM, you can  construct them
>>> easily enough, except that those would have to operate on subcommunicators.
>>> In GASM we pack them into a single scatter on the original communicator. Currently there is no method
>>> to expose this scatter. Why do you need this functionality? 
>>> 
>>> Dmitry.
>>> Thanks!
>>> 
>>>> 
>>>> Dmitry. 
>>>> 
>>>> Thanks!
>>>> 
>>>> On May 15, 2012, at 3:29 PM, Dmitry Karpeev wrote:
>>>> 
>>>>> There are some additional minor fixes that mostly have to do with outputting the subdomain information with  -pc_gasm_view_subdomains (in PCView()) and with -pc_gasm_print_subdomains (during PCSetUp()).
>>>>> You might want to pull those latest patches, but it won't interfere with your work if you don't use subdomain output.
>>>>> 
>>>>> Thanks.
>>>>> Dmitry.
>>>>> 
>>>>> On Tue, May 15, 2012 at 7:14 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> Dmitry,
>>>>> 
>>>>> thanks for reply. I re-download the codes and tried it again and now it works correctly!
>>>>> 
>>>>> Everything seems ok.
>>>>> 
>>>>> Thanks,
>>>>> Hui
>>>>> 
>>>>> 
>>>>> On May 15, 2012, at 2:01 PM, Dmitry Karpeev wrote:
>>>>> 
>>>>>> Hui,
>>>>>> I'm trying to reproduce this problem, unsuccessfully, so far.
>>>>>> One thing that looks odd is that the output below claims the PC is of type "asm", even though you are running with -dd_type gasm.  Could you verify that's the correct output?
>>>>>> 
>>>>>> Here's the output I get with 
>>>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type asm -dd_ksp_view
>>>>>> 
>>>>>> PC Object:(dd_) 1 MPI processes
>>>>>>   type: asm
>>>>>>     Additive Schwarz: total subdomain blocks = 2, user-defined overlap
>>>>>>     Additive Schwarz: restriction/interpolation type - RESTRICT
>>>>>>     Local solve is same for all blocks, in the following KSP and PC objects:
>>>>>>     KSP Object:    (dd_sub_)     1 MPI processes
>>>>>>       type: preonly
>>>>>>       maximum iterations=10000, initial guess is zero
>>>>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>       left preconditioning
>>>>>>       using NONE norm type for convergence test
>>>>>>       <snip>
>>>>>> 
>>>>>> and with 
>>>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type gasm -dd_ksp_view
>>>>>> 
>>>>>> PC Object:(dd_) 1 MPI processes
>>>>>>   type: gasm
>>>>>>   Generalized additive Schwarz:
>>>>>>   Restriction/interpolation type: RESTRICT
>>>>>>   user-defined overlap
>>>>>>   total number of subdomains = 2
>>>>>>   number of local subdomains = 2
>>>>>>   max number of local subdomains = 2
>>>>>>   [0:1] number of locally-supported subdomains = 2
>>>>>>   Subdomain solver info is as follows:
>>>>>>   <snip>
>>>>>> 
>>>>>> What convergence are you seeing with the two PC types?  It should be the same with 1 and 2 procs for both PCASM and PCGASM.  
>>>>>> 
>>>>>> Thanks.
>>>>>> Dmitry.
>>>>>> 
>>>>>> On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> Dmitry,
>>>>>> 
>>>>>> I got the newest petsc-dev and I run the test by
>>>>>> 
>>>>>>     mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
>>>>>> 
>>>>>> which gives the following output
>>>>>> 
>>>>>> PC Object:(dd_) 1 MPI processes
>>>>>>   type: asm
>>>>>>     Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1
>>>>>>                                               ^^^
>>>>>>                                          note the above number, it should be 2 
>>>>>> 
>>>>>> While PCASM has no such problem.
>>>>>> 
>>>>>> Thanks,
>>>>>> Hui
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> Hui,
>>>>>>> 
>>>>>>> The convergence issue should be resolved in the latest petsc-dev.
>>>>>>> I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) 
>>>>>>> that should verify that.
>>>>>>> 
>>>>>>> Let me know if it works for you.
>>>>>>> Thanks.
>>>>>>> Dmitry.
>>>>>>> On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>> Hi Dmitry,
>>>>>>> 
>>>>>>> thanks for useful hints.  Good day!
>>>>>>> 
>>>>>>> Hui
>>>>>>> 
>>>>>>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>>>>>> 
>>>>>>>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
>>>>>>>> your modification subroutine), but not the subdomains or the subdomain solvers. 
>>>>>>>> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
>>>>>>>> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
>>>>>>>> 
>>>>>>>> Dmitry.
>>>>>>>> 
>>>>>>>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>> I just have a question about reuse of PCASM or PCGASM.
>>>>>>>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>>>>>>>> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
>>>>>>>> different user context for the modifying routine.  
>>>>>>>> 
>>>>>>>> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
>>>>>>>> even for PCASM I can re-use it because the partition of subdomains remain the same.
>>>>>>>> 
>>>>>>>> Thanks!
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>>>>>> 
>>>>>>>>> Hui,
>>>>>>>>> There've been several changes to PCGASM ahead of the new release.
>>>>>>>>> Let me go back and see if it affected the convergence problem.
>>>>>>>>> Dmitry.
>>>>>>>>> 
>>>>>>>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>> Hi Dmitry,
>>>>>>>>> 
>>>>>>>>> is there any news about PCGASM? 
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Hui
>>>>>>>>> 
>>>>>>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>>>>>> 
>>>>>>>>>> Okay, thanks.
>>>>>>>>>> I'll take a look.
>>>>>>>>>> 
>>>>>>>>>> Dmitry.
>>>>>>>>>> 
>>>>>>>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>> For reference, my results are attached.
>>>>>>>>>> 
>>>>>>>>>> asm1.txt for asm with 1 process,
>>>>>>>>>> asm2.txt for asm with 2 processes,
>>>>>>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>>>>>>>>> gasm2.txt for gasm with 2 processes
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> thank you,
>>>>>>>>>> Hui
>>>>>>>>>> 
>>>>>>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>>>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>>>>>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>>>>>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>>>>>>>>> with GASM but no problems with ASM. 
>>>>>>>>>>>> 
>>>>>>>>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>>>>>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>>>>>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>>>>>>>>> number is different from the first case and is much larger.  On the other hand
>>>>>>>>>>>> ASM has no such problem.
>>>>>>>>>>>>  
>>>>>>>>>>>> Are the solutions the same?
>>>>>>>>>>>> What problem are you solving?
>>>>>>>>>>> 
>>>>>>>>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>>>>>>>>> two processors. But GASM did not.  
>>>>>>>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>>>>>>>>> I'm solving the Helmholtz equation.  Maybe 
>>>>>>>>>>> I can prepare a simpler example to show this difference.
>>>>>>>>>>> That would be helpful.  
>>>>>>>>>>> Thanks.
>>>>>>>>>>> 
>>>>>>>>>>> Dmitry. 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Dmitry. 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> You should be able to. 
>>>>>>>>>>>>> This behavior is the same as in PCASM,
>>>>>>>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Dmitry
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Dmitry,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> As I tested, the row and col are always the same. 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>>>>>>>>> in the above func()?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>> Hui
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>> Hui
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Yes, that's right.
>>>>>>>>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Dmitry.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Hui
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> <gasm_test.c>
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/c4671240/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Tue May 15 11:12:25 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 May 2012 10:12:25 -0600
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
Message-ID: <CAM9tzS=fK0sHwJazv+Fvvzg302OpW6d81jXSJFSpiqjeVg9+YA@mail.gmail.com>

On Tue, May 15, 2012 at 10:05 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> I am writing some modified ASM method. In construction of energy
> minimization coarse basis,
> I need to solve individual subdomain problems and not to sum them, just to
> extend them separately.
> I wonder whether you guys have ever done this coarse basis.
>

How are these subdomain problems defined? They are very often different (in
terms of overlap or boundary conditions) than are used in the smoother.

You might want to look at PCEXOTIC (
http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCEXOTIC.html)
which is a basic implementation of a two-level DD method of this type.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/738ee3bd/attachment.htm>

From mike.hui.zhang at hotmail.com  Tue May 15 11:15:32 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Tue, 15 May 2012 18:15:32 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CAM9tzS=fK0sHwJazv+Fvvzg302OpW6d81jXSJFSpiqjeVg9+YA@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<CAM9tzS=fK0sHwJazv+Fvvzg302OpW6d81jXSJFSpiqjeVg9+YA@mail.gmail.com>
Message-ID: <BLU0-SMTP199CDABB5EC8470DB1E39BCCB1B0@phx.gbl>


On May 15, 2012, at 6:12 PM, Jed Brown wrote:

> On Tue, May 15, 2012 at 10:05 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> I am writing some modified ASM method. In construction of energy minimization coarse basis,
> I need to solve individual subdomain problems and not to sum them, just to extend them separately.
> I wonder whether you guys have ever done this coarse basis.
> 
> How are these subdomain problems defined? They are very often different (in terms of overlap or boundary conditions) than are used in the smoother

With overlap and modified boundary conditions.

> 
> You might want to look at PCEXOTIC (http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCEXOTIC.html) which is a basic implementation of a two-level DD method of this type.

Yes, this is interesting :) Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/b312eeea/attachment.htm>

From jedbrown at mcs.anl.gov  Tue May 15 11:20:28 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 May 2012 10:20:28 -0600
Subject: [petsc-users] Fwd: PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP17291C3CF86EF473B2F849CCB1B0@phx.gbl>
References: <59EB7A9E-CAC0-4508-988B-CC0CEA7C3226@hotmail.com>
	<BLU0-SMTP17291C3CF86EF473B2F849CCB1B0@phx.gbl>
Message-ID: <CAM9tzSk9u82DZ1vaL0odcyWMW2ts9BVrEo30ZT05mG-_eOZABA@mail.gmail.com>

On Tue, May 15, 2012 at 10:11 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Yes, see Mandel & Dorhamann' paper
> http://ccm.ucdenver.edu/reports/rep194.pdf ,
>

BDDC needs to solve pinned Neumann problems to build the coarse space.
There is no way to obtain those Neumann problems from an assembled matrix,
and neither PCASM or PCGASM is set up to handle that.

There is a BDDC implementation from Stefano Zampini in petsc-dev, but the
code is still not mature so you have to configure --with-pcbddc to activate
it.


> but this is for non-overlapping. For overlapping we can see Leong &
> Widlund
> http://cs.nyu.edu/web/Research/TechReports/TR2008-914/TR2008-914.pdf
>

This solves non-overlapping Dirichlet problems to construct the coarse
basis, then uses overlapping Dirichlet problems for the smoother. Again,
these problems are different, so it's not useful to extract the subdomain
problems from PC(G)ASM.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/1c42faa8/attachment.htm>

From bsmith at mcs.anl.gov  Tue May 15 11:23:06 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 15 May 2012 11:23:06 -0500
Subject: [petsc-users] 3D Poisson Solver with a Periodic Boundary
	Condition in One Direction?
In-Reply-To: <4FB1283F.1090808@gmail.com>
References: <4FB1283F.1090808@gmail.com>
Message-ID: <3D2F0C2F-471C-4FCE-A9DF-0227AEDF257F@mcs.anl.gov>


  Alan,

   Sorry, we don't have exactly that example. But you can start with src/ksp/ksp/examples/tutorials/ex45.c change the 2nd, 3rd, or 4th argument to DMDA_BOUNDARY_PERIODIC then in ComputeMatrix() remove the appropriate values
from if (i==0 || j==0 || k==0 || i==mx-1 || j==my-1 || k==mz-1){   so, for example, if you make the problem periodic in x then remove the i == 0 and i == mx-1 cases

   Barry



On May 14, 2012, at 10:43 AM, Zhenglun (Alan) Wei wrote:

> Dear All,
>    I hope you're having a nice day.
>    I wonder if there is any example in PETSc to solve 3D Poisson equation with the periodic boundary condition in one direction. 2D is fine if there is no 3D example.
> 
> thanks,
> Alan


From mike.hui.zhang at hotmail.com  Tue May 15 11:25:55 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Tue, 15 May 2012 18:25:55 +0200
Subject: [petsc-users] Fwd: PCGASMSetLocalSubdomains
In-Reply-To: <CAM9tzSk9u82DZ1vaL0odcyWMW2ts9BVrEo30ZT05mG-_eOZABA@mail.gmail.com>
References: <59EB7A9E-CAC0-4508-988B-CC0CEA7C3226@hotmail.com>
	<BLU0-SMTP17291C3CF86EF473B2F849CCB1B0@phx.gbl>
	<CAM9tzSk9u82DZ1vaL0odcyWMW2ts9BVrEo30ZT05mG-_eOZABA@mail.gmail.com>
Message-ID: <BLU0-SMTP2678F8F5D338759135BC095CB1B0@phx.gbl>


On May 15, 2012, at 6:20 PM, Jed Brown wrote:

> On Tue, May 15, 2012 at 10:11 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> Yes, see Mandel & Dorhamann' paper http://ccm.ucdenver.edu/reports/rep194.pdf ,
> 
> BDDC needs to solve pinned Neumann problems to build the coarse space. There is no way to obtain those Neumann problems from an assembled matrix, and neither PCASM or PCGASM is set up to handle that.
> 
> There is a BDDC implementation from Stefano Zampini in petsc-dev, but the code is still not mature so you have to configure --with-pcbddc to activate it.

Yes, thanks!

>  
> but this is for non-overlapping. For overlapping we can see Leong & Widlund 
> http://cs.nyu.edu/web/Research/TechReports/TR2008-914/TR2008-914.pdf
> 
> This solves non-overlapping Dirichlet problems to construct the coarse basis, then uses overlapping Dirichlet problems for the smoother. Again, these problems are different, so it's not useful to extract the subdomain problems from PC(G)ASM.

Yes, but I want to modify it a bit to just use the overlapping subdomains for coarse basis.
I think my choice is more natural because we avoid factorization on the non-overlapping subdomains.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/c585f2b9/attachment.htm>

From jedbrown at mcs.anl.gov  Tue May 15 11:28:10 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 May 2012 10:28:10 -0600
Subject: [petsc-users] Fwd: PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP2678F8F5D338759135BC095CB1B0@phx.gbl>
References: <59EB7A9E-CAC0-4508-988B-CC0CEA7C3226@hotmail.com>
	<BLU0-SMTP17291C3CF86EF473B2F849CCB1B0@phx.gbl>
	<CAM9tzSk9u82DZ1vaL0odcyWMW2ts9BVrEo30ZT05mG-_eOZABA@mail.gmail.com>
	<BLU0-SMTP2678F8F5D338759135BC095CB1B0@phx.gbl>
Message-ID: <CAM9tzS=rWa+pDyfwBq787GKRaz5M4aqcgSuRTZmw6hUQEURyqg@mail.gmail.com>

On Tue, May 15, 2012 at 10:25 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Yes, but I want to modify it a bit to just use the overlapping subdomains
> for coarse basis.
> I think my choice is more natural because we avoid factorization on the
> non-overlapping subdomains.
>

How will you ensure that the coarse basis functions are a partition of
unity?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/e0e347f9/attachment.htm>

From karpeev at mcs.anl.gov  Tue May 15 11:29:44 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Tue, 15 May 2012 11:29:44 -0500
Subject: [petsc-users] Fwd: PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP2678F8F5D338759135BC095CB1B0@phx.gbl>
References: <59EB7A9E-CAC0-4508-988B-CC0CEA7C3226@hotmail.com>
	<BLU0-SMTP17291C3CF86EF473B2F849CCB1B0@phx.gbl>
	<CAM9tzSk9u82DZ1vaL0odcyWMW2ts9BVrEo30ZT05mG-_eOZABA@mail.gmail.com>
	<BLU0-SMTP2678F8F5D338759135BC095CB1B0@phx.gbl>
Message-ID: <CA+Y_9UJ7u-WxafoFym=RTfeDh=N3Mkwi-LDKJu3524ZcJo+1=w@mail.gmail.com>

On Tue, May 15, 2012 at 11:25 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

>
> On May 15, 2012, at 6:20 PM, Jed Brown wrote:
>
> On Tue, May 15, 2012 at 10:11 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>> Yes, see Mandel & Dorhamann' paper
>> http://ccm.ucdenver.edu/reports/rep194.pdf ,
>>
>
> BDDC needs to solve pinned Neumann problems to build the coarse space.
> There is no way to obtain those Neumann problems from an assembled matrix,
> and neither PCASM or PCGASM is set up to handle that.
>
> There is a BDDC implementation from Stefano Zampini in petsc-dev, but the
> code is still not mature so you have to configure --with-pcbddc to activate
> it.
>
>
> Yes, thanks!
>
>
>
>> but this is for non-overlapping. For overlapping we can see Leong &
>> Widlund
>> http://cs.nyu.edu/web/Research/TechReports/TR2008-914/TR2008-914.pdf
>>
>
> This solves non-overlapping Dirichlet problems to construct the coarse
> basis, then uses overlapping Dirichlet problems for the smoother. Again,
> these problems are different, so it's not useful to extract the subdomain
> problems from PC(G)ASM.
>
>
> Yes, but I want to modify it a bit to just use the overlapping subdomains
> for coarse basis.
> I think my choice is more natural because we avoid factorization on the
> non-overlapping subdomains.
>
As far as I can see, you can only reuse the scatters from PCASM, which are
easy enough to construct on your own.
PCGASM doesn't have the individual scatters, so that's of even less help
for you.

Dmitry.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/a4d73fc8/attachment-0001.htm>

From mike.hui.zhang at hotmail.com  Tue May 15 11:33:57 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Tue, 15 May 2012 18:33:57 +0200
Subject: [petsc-users] Fwd: PCGASMSetLocalSubdomains
In-Reply-To: <CAM9tzS=rWa+pDyfwBq787GKRaz5M4aqcgSuRTZmw6hUQEURyqg@mail.gmail.com>
References: <59EB7A9E-CAC0-4508-988B-CC0CEA7C3226@hotmail.com>
	<BLU0-SMTP17291C3CF86EF473B2F849CCB1B0@phx.gbl>
	<CAM9tzSk9u82DZ1vaL0odcyWMW2ts9BVrEo30ZT05mG-_eOZABA@mail.gmail.com>
	<BLU0-SMTP2678F8F5D338759135BC095CB1B0@phx.gbl>
	<CAM9tzS=rWa+pDyfwBq787GKRaz5M4aqcgSuRTZmw6hUQEURyqg@mail.gmail.com>
Message-ID: <BLU0-SMTP383E068444EFE601E8417CDCB1B0@phx.gbl>


On May 15, 2012, at 6:28 PM, Jed Brown wrote:

> On Tue, May 15, 2012 at 10:25 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> Yes, but I want to modify it a bit to just use the overlapping subdomains for coarse basis.
> I think my choice is more natural because we avoid factorization on the non-overlapping subdomains.
> 
> How will you ensure that the coarse basis functions are a partition of unity?

Yes, I do some weighting. 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/f85f13b3/attachment.htm>

From andrew.spott at gmail.com  Tue May 15 13:19:34 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Tue, 15 May 2012 12:19:34 -0600
Subject: [petsc-users] Modifying the structure of a matrix.
In-Reply-To: <CAM9tzSmLO9k-QfS1_mBA156-DynRVLtXMSbufQ1rNHqPi1Xefw@mail.gmail.com>
References: <D9FAE42C-ABDA-4670-AEA4-889D2375195A@gmail.com>
	<CAM9tzS=S+xOSpUpPUFzz1zMz=8BY+xai6+PChKzXrxtbTZ46OA@mail.gmail.com>
	<D7D58CA5-7829-404A-81AB-0D3612F64F96@gmail.com>
	<32B30BCC-FA15-428F-B4D7-BBDFAA2837B9@mcs.anl.gov>
	<C3F18BF7-72C1-4605-8EF2-C9D71196F043@gmail.com>
	<4589BB71-BD27-4B72-B346-404AFF66141D@mcs.anl.gov>
	<1E35CF63-4E20-4BF6-A098-E2BBB22600E8@gmail.com>
	<CAM9tzSmLO9k-QfS1_mBA156-DynRVLtXMSbufQ1rNHqPi1Xefw@mail.gmail.com>
Message-ID: <EE21258E-DD79-401D-B35C-C106EF5CAB8B@gmail.com>

I'm solving the schrodinger equation in a basis state method and looking at adding the azimuthal quantum numbers (the m's) to my basis so I can look at circularly polarized light.

However, when I do this, I'll need to do some sort of convergence study to make sure I add enough of them.  The way my code is structured, it will probably be easier to just remove rows and columns from a bigger matrix, instead of adding them to a smaller matrix.  However, depending on the way I structure the matrix, I could end up removing all the values (or a significant portion of them) from a processor when I do that.

Speaking more on that, is the "PETSC_DECIDE" way of finding the local distribution smart in any way? or does it just assume an even distribution of values?  (I assume that it assumes a even distribution of values before the assembly, but does it redistribute during assembly?)

Thanks,

-Andrew

On May 15, 2012, at 6:40 AM, Jed Brown wrote:

> On Mon, May 14, 2012 at 10:39 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
> That is what I figure.
> 
> I'm curious though if you need to manually determine the local row distribution after you do that.  (for example, say you completely remove all the values from the local range of one processor? that processor wouldn't be utilized unless you redistribute the matrix)
> 
> What sizes and method are we talking about? Usually additional (compact) basis functions only make sense to add to one of a small number of processes.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/aeab303a/attachment.htm>

From abhyshr at mcs.anl.gov  Tue May 15 13:38:16 2012
From: abhyshr at mcs.anl.gov (Shri)
Date: Tue, 15 May 2012 13:38:16 -0500 (CDT)
Subject: [petsc-users] Modifying the structure of a matrix.
In-Reply-To: <EE21258E-DD79-401D-B35C-C106EF5CAB8B@gmail.com>
Message-ID: <1491341588.6187.1337107096488.JavaMail.root@zimbra-mb2.anl.gov>

By default, PETSc uses a row distribution with the first m rows on processor 0, next m rows on processor 1, and so on. If the order is not important, then you could remove rows/columns from each processor instead of removing the last n-m rows on the last processor. Thus, the load balance would still be even. Shri ----- Original Message -----
> I'm solving the schrodinger equation in a basis state method and
> looking at adding the azimuthal quantum numbers (the m's) to my basis
> so I can look at circularly polarized light.
> However, when I do this, I'll need to do some sort of convergence
> study to make sure I add enough of them. The way my code is
> structured, it will probably be easier to just remove rows and columns
> from a bigger matrix, instead of adding them to a smaller matrix.
> However, depending on the way I structure the matrix, I could end up
> removing all the values (or a significant portion of them) from a
> processor when I do that.
> Speaking more on that, is the "PETSC_DECIDE" way of finding the local
> distribution smart in any way? or does it just assume an even
> distribution of values? (I assume that it assumes a even distribution
> of values before the assembly, but does it redistribute during
> assembly?)
> Thanks,
> -Andrew
> On May 15, 2012, at 6:40 AM, Jed Brown wrote:
> > On Mon, May 14, 2012 at 10:39 PM, Andrew Spott <
> > andrew.spott at gmail.com > wrote:
> > > That is what I figure.
> > > I'm curious though if you need to manually determine the local row
> > > distribution after you do that. (for example, say you completely
> > > remove all the values from the local range of one processor? that
> > > processor wouldn't be utilized unless you redistribute the matrix)
> > What sizes and method are we talking about? Usually additional
> > (compact) basis functions only make sense to add to one of a small
> > number of processes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/1d82f08e/attachment.htm>

From andrew.spott at gmail.com  Tue May 15 13:44:27 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Tue, 15 May 2012 12:44:27 -0600
Subject: [petsc-users] Modifying the structure of a matrix.
In-Reply-To: <1491341588.6187.1337107096488.JavaMail.root@zimbra-mb2.anl.gov>
References: <EE21258E-DD79-401D-B35C-C106EF5CAB8B@gmail.com>
	<1491341588.6187.1337107096488.JavaMail.root@zimbra-mb2.anl.gov>
Message-ID: <CAPbJa2iYsXM5K4SgCEVMMp+ef7-yY6RHswvEn5k3_GWS7zYe6Q@mail.gmail.com>

That would be ideal, but may be impractical... is it possible to redo the
distribution after the matrix is assembled?  How would I go about doing
that?
On May 15, 2012 12:38 PM, "Shri" <abhyshr at mcs.anl.gov> wrote:

> By default, PETSc uses a row distribution with the first m rows on
> processor 0, next m rows on processor 1, and so on.
> If the order is not important, then you could remove rows/columns from
> each processor instead of removing the last n-m rows on the last processor.
> Thus, the load balance would still be even.
>
> Shri
>
> ------------------------------
>
> I'm solving the schrodinger equation in a basis state method and looking
> at adding the azimuthal quantum numbers (the m's) to my basis so I can look
> at circularly polarized light.
>
> However, when I do this, I'll need to do some sort of convergence study to
> make sure I add enough of them.  The way my code is structured, it will
> probably be easier to just remove rows and columns from a bigger matrix,
> instead of adding them to a smaller matrix.  However, depending on the way
> I structure the matrix, I could end up removing all the values (or a
> significant portion of them) from a processor when I do that.
>
> Speaking more on that, is the "PETSC_DECIDE" way of finding the local
> distribution smart in any way? or does it just assume an even distribution
> of values?  (I assume that it assumes a even distribution of values before
> the assembly, but does it redistribute during assembly?)
>
> Thanks,
>
> -Andrew
>
> On May 15, 2012, at 6:40 AM, Jed Brown wrote:
>
> On Mon, May 14, 2012 at 10:39 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>
>> That is what I figure.
>>
>> I'm curious though if you need to manually determine the local row
>> distribution after you do that.  (for example, say you completely remove
>> all the values from the local range of one processor? that processor
>> wouldn't be utilized unless you redistribute the matrix)
>>
>
> What sizes and method are we talking about? Usually additional (compact)
> basis functions only make sense to add to one of a small number of
> processes.
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/2455201f/attachment.htm>

From john.mousel at gmail.com  Tue May 15 13:47:37 2012
From: john.mousel at gmail.com (John Mousel)
Date: Tue, 15 May 2012 13:47:37 -0500
Subject: [petsc-users] Modifying the structure of a matrix.
In-Reply-To: <CAPbJa2iYsXM5K4SgCEVMMp+ef7-yY6RHswvEn5k3_GWS7zYe6Q@mail.gmail.com>
References: <EE21258E-DD79-401D-B35C-C106EF5CAB8B@gmail.com>
	<1491341588.6187.1337107096488.JavaMail.root@zimbra-mb2.anl.gov>
	<CAPbJa2iYsXM5K4SgCEVMMp+ef7-yY6RHswvEn5k3_GWS7zYe6Q@mail.gmail.com>
Message-ID: <CA+zegp=8ejJuLKShM9c8ZVBm39WqBvEmSi54sUg-XwzAOsVi1Q@mail.gmail.com>

Maybe look at PCREDISTRIBUTE?

On Tue, May 15, 2012 at 1:44 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> That would be ideal, but may be impractical... is it possible to redo the
> distribution after the matrix is assembled?  How would I go about doing
> that?
> On May 15, 2012 12:38 PM, "Shri" <abhyshr at mcs.anl.gov> wrote:
>
>> By default, PETSc uses a row distribution with the first m rows on
>> processor 0, next m rows on processor 1, and so on.
>> If the order is not important, then you could remove rows/columns from
>> each processor instead of removing the last n-m rows on the last processor.
>> Thus, the load balance would still be even.
>>
>> Shri
>>
>> ------------------------------
>>
>> I'm solving the schrodinger equation in a basis state method and looking
>> at adding the azimuthal quantum numbers (the m's) to my basis so I can look
>> at circularly polarized light.
>>
>> However, when I do this, I'll need to do some sort of convergence study
>> to make sure I add enough of them.  The way my code is structured, it will
>> probably be easier to just remove rows and columns from a bigger matrix,
>> instead of adding them to a smaller matrix.  However, depending on the way
>> I structure the matrix, I could end up removing all the values (or a
>> significant portion of them) from a processor when I do that.
>>
>> Speaking more on that, is the "PETSC_DECIDE" way of finding the local
>> distribution smart in any way? or does it just assume an even distribution
>> of values?  (I assume that it assumes a even distribution of values before
>> the assembly, but does it redistribute during assembly?)
>>
>> Thanks,
>>
>> -Andrew
>>
>> On May 15, 2012, at 6:40 AM, Jed Brown wrote:
>>
>> On Mon, May 14, 2012 at 10:39 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>>
>>> That is what I figure.
>>>
>>> I'm curious though if you need to manually determine the local row
>>> distribution after you do that.  (for example, say you completely remove
>>> all the values from the local range of one processor? that processor
>>> wouldn't be utilized unless you redistribute the matrix)
>>>
>>
>> What sizes and method are we talking about? Usually additional (compact)
>> basis functions only make sense to add to one of a small number of
>> processes.
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/bce546f5/attachment.htm>

From jedbrown at mcs.anl.gov  Tue May 15 13:50:20 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 May 2012 12:50:20 -0600
Subject: [petsc-users] Modifying the structure of a matrix.
In-Reply-To: <CA+zegp=8ejJuLKShM9c8ZVBm39WqBvEmSi54sUg-XwzAOsVi1Q@mail.gmail.com>
References: <EE21258E-DD79-401D-B35C-C106EF5CAB8B@gmail.com>
	<1491341588.6187.1337107096488.JavaMail.root@zimbra-mb2.anl.gov>
	<CAPbJa2iYsXM5K4SgCEVMMp+ef7-yY6RHswvEn5k3_GWS7zYe6Q@mail.gmail.com>
	<CA+zegp=8ejJuLKShM9c8ZVBm39WqBvEmSi54sUg-XwzAOsVi1Q@mail.gmail.com>
Message-ID: <CAM9tzS=FFy0S3WgRioO_JxH=nQxCoJ2zAsAd82+7_yG8L+bfsA@mail.gmail.com>

On Tue, May 15, 2012 at 12:47 PM, John Mousel <john.mousel at gmail.com> wrote:

> Maybe look at PCREDISTRIBUTE?


This does not currently do an entry-based partition, but it would be the
natural place to put such a thing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/9920d6cf/attachment.htm>

From abhyshr at mcs.anl.gov  Tue May 15 14:01:30 2012
From: abhyshr at mcs.anl.gov (Shrirang Abhyankar)
Date: Tue, 15 May 2012 14:01:30 -0500
Subject: [petsc-users] Modifying the structure of a matrix.
In-Reply-To: <CAPbJa2iYsXM5K4SgCEVMMp+ef7-yY6RHswvEn5k3_GWS7zYe6Q@mail.gmail.com>
References: <EE21258E-DD79-401D-B35C-C106EF5CAB8B@gmail.com>
	<1491341588.6187.1337107096488.JavaMail.root@zimbra-mb2.anl.gov>
	<CAPbJa2iYsXM5K4SgCEVMMp+ef7-yY6RHswvEn5k3_GWS7zYe6Q@mail.gmail.com>
Message-ID: <0E5B5F71-7518-493F-AA5F-DBE3DE73C088@mcs.anl.gov>

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrix.html 

On May 15, 2012, at 1:44 PM, Andrew Spott wrote:

> That would be ideal, but may be impractical... is it possible to redo the distribution after the matrix is assembled?  How would I go about doing that?
> 
> On May 15, 2012 12:38 PM, "Shri" <abhyshr at mcs.anl.gov> wrote:
> By default, PETSc uses a row distribution with the first m rows on processor 0, next m rows on processor 1, and so on.
> If the order is not important, then you could remove rows/columns from each processor instead of removing the last n-m rows on the last processor. Thus, the load balance would still be even.
> 
> Shri
> 
> I'm solving the schrodinger equation in a basis state method and looking at adding the azimuthal quantum numbers (the m's) to my basis so I can look at circularly polarized light.
> 
> However, when I do this, I'll need to do some sort of convergence study to make sure I add enough of them.  The way my code is structured, it will probably be easier to just remove rows and columns from a bigger matrix, instead of adding them to a smaller matrix.  However, depending on the way I structure the matrix, I could end up removing all the values (or a significant portion of them) from a processor when I do that.
> 
> Speaking more on that, is the "PETSC_DECIDE" way of finding the local distribution smart in any way? or does it just assume an even distribution of values?  (I assume that it assumes a even distribution of values before the assembly, but does it redistribute during assembly?)
> 
> Thanks,
> 
> -Andrew
> 
> On May 15, 2012, at 6:40 AM, Jed Brown wrote:
> 
> On Mon, May 14, 2012 at 10:39 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
> That is what I figure.
> 
> I'm curious though if you need to manually determine the local row distribution after you do that.  (for example, say you completely remove all the values from the local range of one processor? that processor wouldn't be utilized unless you redistribute the matrix)
> 
> What sizes and method are we talking about? Usually additional (compact) basis functions only make sense to add to one of a small number of processes.
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/ab773184/attachment-0001.htm>

From behzad.baghapour at gmail.com  Tue May 15 15:26:13 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Tue, 15 May 2012 23:56:13 +0330
Subject: [petsc-users] Linesearch algorithm
In-Reply-To: <CAM9tzSm-Qi08oLAv5i5SkUQHZXV0Xmt-k+Hw4QuAeKhJDKxy9w@mail.gmail.com>
References: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
	<C7256B25-6176-4431-9BB8-9663C0C71024@mcs.anl.gov>
	<CA+jOtD+_4gK-ySHKiCsfQaoYs5+r_jkUcv70cOjdhBoiaS98+w@mail.gmail.com>
	<CAM9tzSm-Qi08oLAv5i5SkUQHZXV0Xmt-k+Hw4QuAeKhJDKxy9w@mail.gmail.com>
Message-ID: <CA+jOtD+kV1sKm9GY0F0X0qWZuVbVyQt48df3WMiUA1Vty1eBag@mail.gmail.com>

OK. The (pseudo) transient technique is a way of globalizing the local
convergent Newton methods which is very helpful when dealing with high
nonlinear problems with a pure initial guess. But the classical line-search
method is to minimize the problem 0.5IIR(x)II^2 which basically does not
see the transient phase. Then I have to discard this phase in order to
match with the structure of the method. So I am looking forward to find a
way to tell the SNES to consider just the term: dR/dQ when using it to
modify the search direction with its quadratic or cubic model.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/1ac8efc1/attachment.htm>

From jedbrown at mcs.anl.gov  Tue May 15 15:36:10 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 May 2012 14:36:10 -0600
Subject: [petsc-users] Linesearch algorithm
In-Reply-To: <CA+jOtD+kV1sKm9GY0F0X0qWZuVbVyQt48df3WMiUA1Vty1eBag@mail.gmail.com>
References: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
	<C7256B25-6176-4431-9BB8-9663C0C71024@mcs.anl.gov>
	<CA+jOtD+_4gK-ySHKiCsfQaoYs5+r_jkUcv70cOjdhBoiaS98+w@mail.gmail.com>
	<CAM9tzSm-Qi08oLAv5i5SkUQHZXV0Xmt-k+Hw4QuAeKhJDKxy9w@mail.gmail.com>
	<CA+jOtD+kV1sKm9GY0F0X0qWZuVbVyQt48df3WMiUA1Vty1eBag@mail.gmail.com>
Message-ID: <CAM9tzS=sG0c9rLY97AfP7y+H2mpJ0tFvBPUs1xrLV5mS_NZ-hA@mail.gmail.com>

On Tue, May 15, 2012 at 2:26 PM, behzad baghapour <
behzad.baghapour at gmail.com> wrote:

> OK. The (pseudo) transient technique is a way of globalizing the local
> convergent Newton methods which is very helpful when dealing with high
> nonlinear problems with a pure initial guess. But the classical line-search
> method is to minimize the problem 0.5IIR(x)II^2 which basically does not
> see the transient phase.


1. The transient term may give you a direction that is not a descent
direction for ||R(x)||^2. For such problems, you _must_ allow ||R(x)|| to
_increase_ before you can reach a steady state.

2. ||R(x)||^2 may not be a good objective functional for the line search,
especially for non-convex problems.


> Then I have to discard this phase in order to match with the structure of
> the method. So I am looking forward to find a way to tell the SNES to
> consider just the term: dR/dQ when using it to modify the search direction
> with its quadratic or cubic model.


The line search does not change the search direction, it changes the step
length.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/5b4ca762/attachment.htm>

From behzad.baghapour at gmail.com  Tue May 15 15:41:04 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Wed, 16 May 2012 00:11:04 +0330
Subject: [petsc-users] Linesearch algorithm
In-Reply-To: <CAM9tzS=sG0c9rLY97AfP7y+H2mpJ0tFvBPUs1xrLV5mS_NZ-hA@mail.gmail.com>
References: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
	<C7256B25-6176-4431-9BB8-9663C0C71024@mcs.anl.gov>
	<CA+jOtD+_4gK-ySHKiCsfQaoYs5+r_jkUcv70cOjdhBoiaS98+w@mail.gmail.com>
	<CAM9tzSm-Qi08oLAv5i5SkUQHZXV0Xmt-k+Hw4QuAeKhJDKxy9w@mail.gmail.com>
	<CA+jOtD+kV1sKm9GY0F0X0qWZuVbVyQt48df3WMiUA1Vty1eBag@mail.gmail.com>
	<CAM9tzS=sG0c9rLY97AfP7y+H2mpJ0tFvBPUs1xrLV5mS_NZ-hA@mail.gmail.com>
Message-ID: <CA+jOtDLm4eetEfV39E5dt=a9UTkP1OJgPhBR5ekQu4=M-521fw@mail.gmail.com>

Thanks. Of course, the Line-search just change the step length. But I could
not get What you mean that the residual must be increased before reaching
the steady-state?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/97e879b8/attachment.htm>

From behzad.baghapour at gmail.com  Tue May 15 15:43:56 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Wed, 16 May 2012 00:13:56 +0330
Subject: [petsc-users] Linesearch algorithm
In-Reply-To: <CA+jOtDLm4eetEfV39E5dt=a9UTkP1OJgPhBR5ekQu4=M-521fw@mail.gmail.com>
References: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
	<C7256B25-6176-4431-9BB8-9663C0C71024@mcs.anl.gov>
	<CA+jOtD+_4gK-ySHKiCsfQaoYs5+r_jkUcv70cOjdhBoiaS98+w@mail.gmail.com>
	<CAM9tzSm-Qi08oLAv5i5SkUQHZXV0Xmt-k+Hw4QuAeKhJDKxy9w@mail.gmail.com>
	<CA+jOtD+kV1sKm9GY0F0X0qWZuVbVyQt48df3WMiUA1Vty1eBag@mail.gmail.com>
	<CAM9tzS=sG0c9rLY97AfP7y+H2mpJ0tFvBPUs1xrLV5mS_NZ-hA@mail.gmail.com>
	<CA+jOtDLm4eetEfV39E5dt=a9UTkP1OJgPhBR5ekQu4=M-521fw@mail.gmail.com>
Message-ID: <CA+jOtDLtA20APMvjd-w6jRoD5fAd8f81hO25EsA-8t8RjszU-A@mail.gmail.com>

In addition, Is it possible to define knew objective function for line
search in SNES in order to deal with pseudo-transient continuation?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/73640165/attachment.htm>

From prbrune at gmail.com  Tue May 15 15:52:51 2012
From: prbrune at gmail.com (Peter Brune)
Date: Tue, 15 May 2012 15:52:51 -0500
Subject: [petsc-users] Linesearch algorithm
In-Reply-To: <CA+jOtDLtA20APMvjd-w6jRoD5fAd8f81hO25EsA-8t8RjszU-A@mail.gmail.com>
References: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
	<C7256B25-6176-4431-9BB8-9663C0C71024@mcs.anl.gov>
	<CA+jOtD+_4gK-ySHKiCsfQaoYs5+r_jkUcv70cOjdhBoiaS98+w@mail.gmail.com>
	<CAM9tzSm-Qi08oLAv5i5SkUQHZXV0Xmt-k+Hw4QuAeKhJDKxy9w@mail.gmail.com>
	<CA+jOtD+kV1sKm9GY0F0X0qWZuVbVyQt48df3WMiUA1Vty1eBag@mail.gmail.com>
	<CAM9tzS=sG0c9rLY97AfP7y+H2mpJ0tFvBPUs1xrLV5mS_NZ-hA@mail.gmail.com>
	<CA+jOtDLm4eetEfV39E5dt=a9UTkP1OJgPhBR5ekQu4=M-521fw@mail.gmail.com>
	<CA+jOtDLtA20APMvjd-w6jRoD5fAd8f81hO25EsA-8t8RjszU-A@mail.gmail.com>
Message-ID: <CAKKtykzzHro22ycffLvCyyoDwwDvo=TrpL8zv6JWmzkpFwjgGQ@mail.gmail.com>

On Tue, May 15, 2012 at 3:43 PM, behzad baghapour <
behzad.baghapour at gmail.com> wrote:

> In addition, Is it possible to define knew objective function for line
> search in SNES in order to deal with pseudo-transient continuation?
>

This feature will be added to petsc-dev shortly post-release.

- Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/29368f73/attachment.htm>

From behzad.baghapour at gmail.com  Tue May 15 15:54:46 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Wed, 16 May 2012 00:24:46 +0330
Subject: [petsc-users] Linesearch algorithm
In-Reply-To: <CAKKtykzzHro22ycffLvCyyoDwwDvo=TrpL8zv6JWmzkpFwjgGQ@mail.gmail.com>
References: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
	<C7256B25-6176-4431-9BB8-9663C0C71024@mcs.anl.gov>
	<CA+jOtD+_4gK-ySHKiCsfQaoYs5+r_jkUcv70cOjdhBoiaS98+w@mail.gmail.com>
	<CAM9tzSm-Qi08oLAv5i5SkUQHZXV0Xmt-k+Hw4QuAeKhJDKxy9w@mail.gmail.com>
	<CA+jOtD+kV1sKm9GY0F0X0qWZuVbVyQt48df3WMiUA1Vty1eBag@mail.gmail.com>
	<CAM9tzS=sG0c9rLY97AfP7y+H2mpJ0tFvBPUs1xrLV5mS_NZ-hA@mail.gmail.com>
	<CA+jOtDLm4eetEfV39E5dt=a9UTkP1OJgPhBR5ekQu4=M-521fw@mail.gmail.com>
	<CA+jOtDLtA20APMvjd-w6jRoD5fAd8f81hO25EsA-8t8RjszU-A@mail.gmail.com>
	<CAKKtykzzHro22ycffLvCyyoDwwDvo=TrpL8zv6JWmzkpFwjgGQ@mail.gmail.com>
Message-ID: <CA+jOtDK-NVYXt=d1ih5MpoU4=5ovn0UzQZiab1=C-RF-WNPEiw@mail.gmail.com>

Thanks a lot.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/884147df/attachment.htm>

From avery.bingham at gmail.com  Tue May 15 16:32:21 2012
From: avery.bingham at gmail.com (Avery Bingham)
Date: Tue, 15 May 2012 15:32:21 -0600
Subject: [petsc-users] Divergence when using Line Search
Message-ID: <CAFG1MnY+6ztnqj+PzaJQVGDyO_SAUr1dzDz2V2-b0U94PZ34oQ@mail.gmail.com>

I have also seen this behavior, and I think this might be related to
the scaling of the variables in the nonlinear system.  I am using
PETSc through an application of MOOSE which allows for scaling of the
variables.  This scaling reduces the chance that the default cubic
backtracking line search fails, but it is not reliable on all
problems. Would it be possible to get a scaling-independent Line
Search Method implemented within PETSc?


Thanks,

Avery Bingham






*Dear Developers,

I used SNES with LS cubic and basic. In some cases the cubic model can
stabilised the global convergence BUT in some case it fails while the basic
model has given better convergence !!!

The error is as follows:
.
.
.
46:   CFL = 3894.09, Nonlinear = 6.9086e-05, Linear = (64,4.54181e-10,1e-05)
  Linear solve converged due to CONVERGED_RTOL iterations 66
47:   CFL = 3921.01, Nonlinear = 6.86117e-05, Linear =
(66,3.51905e-10,1e-05)
  Linear solve converged due to CONVERGED_RTOL iterations 64
48:   CFL = 3936.79, Nonlinear = 6.83367e-05, Linear =
(64,4.50959e-10,1e-05)
  Linear solve converged due to CONVERGED_RTOL iterations 61
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Floating point exception!
[0]PETSC ERROR: Infinite or not-a-number generated in norm!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33
CDT 2011
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./code on a linux-gnu named baghapour by baghapour Thu Jan
5 16:44:14 2012
[0]PETSC ERROR: Libraries linked from
/home/baghapour/softs/petsc/linux-gnu-cxx-debug/lib
[0]PETSC ERROR: Configure run at Wed Nov  9 19:16:47 2011
[0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran
--with-cxx=g++ --download-f2cblaslapack=1 --download-mpich=1
--with-clanguage=cxx --with-debugging=no --download-parms --download-hypre
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: VecNorm() line 167 in src/vec/vec/interface/rvector.c
[0]PETSC ERROR: SNESLineSearchCubic() line 581 in src/snes/impls/ls/ls.c
[0]PETSC ERROR: SNESSolve_LS() line 218 in src/snes/impls/ls/ls.c
[0]PETSC ERROR: SNESSolve() line 2676 in src/snes/interface/snes.c
[0]PETSC ERROR: _petsc_NewtonTimeAdvance() line 131 in Newton.cpp

I am wondering while the basic model can solve the problem but cubic model
can not.
Please help me what conditions may leads the above issue in line search.

Regards,
BehZad*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/7690c6e3/attachment.htm>

From jedbrown at mcs.anl.gov  Tue May 15 16:40:24 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 May 2012 15:40:24 -0600
Subject: [petsc-users] Linesearch algorithm
In-Reply-To: <CA+jOtDLm4eetEfV39E5dt=a9UTkP1OJgPhBR5ekQu4=M-521fw@mail.gmail.com>
References: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
	<C7256B25-6176-4431-9BB8-9663C0C71024@mcs.anl.gov>
	<CA+jOtD+_4gK-ySHKiCsfQaoYs5+r_jkUcv70cOjdhBoiaS98+w@mail.gmail.com>
	<CAM9tzSm-Qi08oLAv5i5SkUQHZXV0Xmt-k+Hw4QuAeKhJDKxy9w@mail.gmail.com>
	<CA+jOtD+kV1sKm9GY0F0X0qWZuVbVyQt48df3WMiUA1Vty1eBag@mail.gmail.com>
	<CAM9tzS=sG0c9rLY97AfP7y+H2mpJ0tFvBPUs1xrLV5mS_NZ-hA@mail.gmail.com>
	<CA+jOtDLm4eetEfV39E5dt=a9UTkP1OJgPhBR5ekQu4=M-521fw@mail.gmail.com>
Message-ID: <CAM9tzSn-7kk50Y7ivd8hoPiNAcsLNgFr9w68ASfuup_EauC8EA@mail.gmail.com>

On Tue, May 15, 2012 at 2:41 PM, behzad baghapour <
behzad.baghapour at gmail.com> wrote:

> Thanks. Of course, the Line-search just change the step length. But I
> could not get What you mean that the residual must be increased before
> reaching the steady-state?


This is the key to pseudotransient continuation's global convergence. If
steady state could be reached by simply minimizing the residual, then
steepest descent (and trust region Newton) would be globally convergent. It
would also render optimization research obsolete because nonlinear equation
solvers could naively be applied to the first order optimality conditions
without concern for the objective. Alas, it doesn't work that way: steady
state often cannot be reached without increasing the steady-state residual
and non-convex optimization problems require that the gradient be allowed
to increase.

This is perhaps easier to illustrate for the optimization case. Consider

f(x) = x^2 - exp(-4*(x-2)^2)

which is a nice-looking function with a single minimum very close to x=0.

http://www.wolframalpha.com/input/?i=x%5E2+-+exp%28-4*%28x-2%29%5E2%29

Now the gradient clearly has these local minima, so searching in the norm
of the gradient will get stuck at a point that is not a local minima.

http://www.wolframalpha.com/input/?i=diff%28x%5E2+-+exp%28-4*%28x-2%29%5E2%29%2Cx%29


For transient problems, there are lots of reasons that the residual may
need to grow before steady state can be reached. See, for example, Figure
3.1 and 3.2 of  http://www.cs.odu.edu/~keyes/papers/ptc03.pdf . Note that
the residual is not monotone. Although they do not prove it here, steady
state for these problems cannot be reached without hill climbing:
inherently transient processes like combustion must take place before a
steady state can be reached.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/431b6427/attachment-0001.htm>

From bsmith at mcs.anl.gov  Tue May 15 16:43:49 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 15 May 2012 16:43:49 -0500
Subject: [petsc-users] Divergence when using Line Search
In-Reply-To: <CAFG1MnY+6ztnqj+PzaJQVGDyO_SAUr1dzDz2V2-b0U94PZ34oQ@mail.gmail.com>
References: <CAFG1MnY+6ztnqj+PzaJQVGDyO_SAUr1dzDz2V2-b0U94PZ34oQ@mail.gmail.com>
Message-ID: <566AE2B9-2691-4822-B416-3B96BBEFB327@mcs.anl.gov>


On May 15, 2012, at 4:32 PM, Avery Bingham wrote:

> I have also seen this behavior, and I think this might be related to the scaling of the variables in the nonlinear system.  I am using PETSc through an application of MOOSE which allows for scaling of the variables.  This scaling reduces the chance that the default cubic backtracking line search fails, but it is not reliable on all problems. Would it be possible to get a scaling-independent Line Search Method implemented within PETSc?  

   Citation?

> 
> Thanks,
> Avery Bingham
> 
> 
> 
> 
> 
> 
> 
> 
> Dear Developers,
> 
> I used SNES with LS cubic and basic. In some cases the cubic model can
> stabilised the global convergence BUT in some case it fails while the basic
> model has given better convergence !!!
> 
> The error is as follows:
> .
> .
> .
> 46:   CFL = 3894.09, Nonlinear = 6.9086e-05, Linear = (64,4.54181e-10,1e-05)
>   Linear solve converged due to CONVERGED_RTOL iterations 66
> 47:   CFL = 3921.01, Nonlinear = 6.86117e-05, Linear =
> (66,3.51905e-10,1e-05)
>   Linear solve converged due to CONVERGED_RTOL iterations 64
> 48:   CFL = 3936.79, Nonlinear = 6.83367e-05, Linear =
> (64,4.50959e-10,1e-05)
>   Linear solve converged due to CONVERGED_RTOL iterations 61
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Floating point exception!
> [0]PETSC ERROR: Infinite or not-a-number generated in norm!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33
> CDT 2011
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./code on a linux-gnu named baghapour by baghapour Thu Jan
> 5 16:44:14 2012
> [0]PETSC ERROR: Libraries linked from
> /home/baghapour/softs/petsc/linux-gnu-cxx-debug/lib
> [0]PETSC ERROR: Configure run at Wed Nov  9 19:16:47 2011
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-fc=gfortran
> --with-cxx=g++ --download-f2cblaslapack=1 --download-mpich=1
> --with-clanguage=cxx --with-debugging=no --download-parms --download-hypre
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: VecNorm() line 167 in src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: SNESLineSearchCubic() line 581 in src/snes/impls/ls/ls.c
> [0]PETSC ERROR: SNESSolve_LS() line 218 in src/snes/impls/ls/ls.c
> [0]PETSC ERROR: SNESSolve() line 2676 in src/snes/interface/snes.c
> [0]PETSC ERROR: _petsc_NewtonTimeAdvance() line 131 in Newton.cpp
> 
> I am wondering while the basic model can solve the problem but cubic model
> can not.
> Please help me what conditions may leads the above issue in line search.
> 
> Regards,
> BehZad
> 


From jedbrown at mcs.anl.gov  Tue May 15 16:45:13 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 May 2012 15:45:13 -0600
Subject: [petsc-users] Divergence when using Line Search
In-Reply-To: <CAFG1MnY+6ztnqj+PzaJQVGDyO_SAUr1dzDz2V2-b0U94PZ34oQ@mail.gmail.com>
References: <CAFG1MnY+6ztnqj+PzaJQVGDyO_SAUr1dzDz2V2-b0U94PZ34oQ@mail.gmail.com>
Message-ID: <CAM9tzSmkcGd=dmNMkB2soWgUcF1xFa_ckGcDWaF8Rd6A78BRXQ@mail.gmail.com>

On Tue, May 15, 2012 at 3:32 PM, Avery Bingham <avery.bingham at gmail.com>wrote:

> I have also seen this behavior, and I think this might be related to the scaling of the variables in the nonlinear system.  I am using PETSc through an application of MOOSE which allows for scaling of the variables.  This scaling reduces the chance that the default cubic backtracking line search fails, but it is not reliable on all problems. Would it be possible to get a scaling-independent Line Search Method implemented within PETSc?
>
>
I'm not sure what you mean by scaling, but there is -snes_linesearch_type
cp (critical point) that might be useful.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/e3fbec73/attachment.htm>

From karpeev at mcs.anl.gov  Tue May 15 17:24:15 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Tue, 15 May 2012 17:24:15 -0500
Subject: [petsc-users] Divergence when using Line Search
In-Reply-To: <CAM9tzSmkcGd=dmNMkB2soWgUcF1xFa_ckGcDWaF8Rd6A78BRXQ@mail.gmail.com>
References: <CAFG1MnY+6ztnqj+PzaJQVGDyO_SAUr1dzDz2V2-b0U94PZ34oQ@mail.gmail.com>
	<CAM9tzSmkcGd=dmNMkB2soWgUcF1xFa_ckGcDWaF8Rd6A78BRXQ@mail.gmail.com>
Message-ID: <CA+Y_9UJSQ7WgjKH_fftG5Hihz--HdnFi2TU9BjEYsLA5Dr=2Ew@mail.gmail.com>

In principle, you can implement any type of line search in PETSc by
subclassing SNESLineSearch (although it requires going under the hood some,
as we generally think of LineSearch as a private object).  If you can
elaborate on what you need, we can work with you guys to see how to
accomplish it with what we have or to extend it, if necessary.  That might
require extra hooks into PETSc in libMesh/Moose, but we have plenty of
experience managing that.

Cheers,
Dmitry.

On Tue, May 15, 2012 at 4:45 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Tue, May 15, 2012 at 3:32 PM, Avery Bingham <avery.bingham at gmail.com>wrote:
>
>> I have also seen this behavior, and I think this might be related to the scaling of the variables in the nonlinear system.  I am using PETSc through an application of MOOSE which allows for scaling of the variables.  This scaling reduces the chance that the default cubic backtracking line search fails, but it is not reliable on all problems. Would it be possible to get a scaling-independent Line Search Method implemented within PETSc?
>>
>>
> I'm not sure what you mean by scaling, but there is -snes_linesearch_type
> cp (critical point) that might be useful.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/485cc81e/attachment.htm>

From jedbrown at mcs.anl.gov  Tue May 15 17:55:21 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 15 May 2012 16:55:21 -0600
Subject: [petsc-users] Divergence when using Line Search
In-Reply-To: <CAFG1Mna4ASoxH90JPCcjKkP8-DAKffWwQ7i8xOFPeuREmoruNQ@mail.gmail.com>
References: <CAFG1MnY+6ztnqj+PzaJQVGDyO_SAUr1dzDz2V2-b0U94PZ34oQ@mail.gmail.com>
	<CAM9tzSmkcGd=dmNMkB2soWgUcF1xFa_ckGcDWaF8Rd6A78BRXQ@mail.gmail.com>
	<CAFG1Mna4ASoxH90JPCcjKkP8-DAKffWwQ7i8xOFPeuREmoruNQ@mail.gmail.com>
Message-ID: <CAM9tzS=MpsHu5a_zYkcW-QWoxKotr8S01pLD0CXzrSCsEVHoVw@mail.gmail.com>

On Tue, May 15, 2012 at 4:41 PM, Avery Bingham <avery.bingham at gmail.com>wrote:

> By scaling, I mean to say that a problem is poorly scaled if a change in x
> in a certain direction produces a much larger variation in f(x) than a
> change in another direction.  For a simple example is the scalar function:
> f(x) = 10^10*x1^2+x2^2
>

This is ill-conditioned, it just happens to be diagonal.


> A more practical example would be a physical coupled problem that has
> Pressure in ~10^6 Pascals and temperature in degrees ~100 C.
>

So if you have an accurate linear solve, this kind of ill-conditioning
should be taken care of (provided you are not up against finite precision).
If you can't do an accurate linear solve, or if you define the matrix by
finite differencing, then it's very important to fix the scaling in problem
formulation, otherwise you will constantly struggle with artificially
inaccurate finite differences and tricky convergence tolerances (the
residual is a poor indicator of convergence, etc).


>
>
>
>
> On Tue, May 15, 2012 at 3:45 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> On Tue, May 15, 2012 at 3:32 PM, Avery Bingham <avery.bingham at gmail.com>wrote:
>>
>>> I have also seen this behavior, and I think this might be related to the scaling of the variables in the nonlinear system.  I am using PETSc through an application of MOOSE which allows for scaling of the variables.  This scaling reduces the chance that the default cubic backtracking line search fails, but it is not reliable on all problems. Would it be possible to get a scaling-independent Line Search Method implemented within PETSc?
>>>
>>>
>> I'm not sure what you mean by scaling, but there is -snes_linesearch_type
>> cp (critical point) that might be useful.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120515/d99408ed/attachment.htm>

From behzad.baghapour at gmail.com  Tue May 15 21:38:10 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Wed, 16 May 2012 06:08:10 +0330
Subject: [petsc-users] Linesearch algorithm
In-Reply-To: <CAM9tzSn-7kk50Y7ivd8hoPiNAcsLNgFr9w68ASfuup_EauC8EA@mail.gmail.com>
References: <CA+jOtD+ONx+eZ63W63p1SBqsotx1TeU_EkdaF8VQjJzfM6iRYg@mail.gmail.com>
	<C7256B25-6176-4431-9BB8-9663C0C71024@mcs.anl.gov>
	<CA+jOtD+_4gK-ySHKiCsfQaoYs5+r_jkUcv70cOjdhBoiaS98+w@mail.gmail.com>
	<CAM9tzSm-Qi08oLAv5i5SkUQHZXV0Xmt-k+Hw4QuAeKhJDKxy9w@mail.gmail.com>
	<CA+jOtD+kV1sKm9GY0F0X0qWZuVbVyQt48df3WMiUA1Vty1eBag@mail.gmail.com>
	<CAM9tzS=sG0c9rLY97AfP7y+H2mpJ0tFvBPUs1xrLV5mS_NZ-hA@mail.gmail.com>
	<CA+jOtDLm4eetEfV39E5dt=a9UTkP1OJgPhBR5ekQu4=M-521fw@mail.gmail.com>
	<CAM9tzSn-7kk50Y7ivd8hoPiNAcsLNgFr9w68ASfuup_EauC8EA@mail.gmail.com>
Message-ID: <CA+jOtDLED0wAeu0B=so1h92jh1YvD1z6RgFaf89yj6iyS8nkUw@mail.gmail.com>

Thank you very much for the detailed description.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/2f91e385/attachment.htm>

From andrew.spott at gmail.com  Wed May 16 01:01:58 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Wed, 16 May 2012 00:01:58 -0600
Subject: [petsc-users] How much memory does TS use (in terms of # of
	matrices)
Message-ID: <B461B804-6D7C-4365-AB82-A2B7CB7DAC10@gmail.com>

I'm attempting to run some code with a rather large (>1GB) sparse matrix, and I keep getting kicked out of the que for what I believe is using too much memory.  The unfortunate thing is that as the steps go up, the memory usage seems to as well, which would normally indicate a memory leak.

But I'm not really doing anything that would lead to a memory leak, my jacobian is:

PetscErrorCode HamiltonianJ(TS timeStepContext, PetscReal t, Vec u, Mat *A, Mat *B, MatStructure *flag, void* context)
{
    Mat h;
    PetscErrorCode  err;
    Context         *cntx = (Context*)context;
    PetscScalar     ef = eField(t, cntx->params);
    //if (cntx->rank == 0) std::cout << "ef: " << ef;
    
    
    err = MatDuplicate(cntx->energy_eigenstates,MAT_COPY_VALUES,&h);
    err = MatAXPY(h, ef, cntx->dipole_matrix, DIFFERENT_NONZERO_PATTERN);

	err = MatAssemblyBegin(h, MAT_FINAL_ASSEMBLY);
	err = MatAssemblyEnd(h, MAT_FINAL_ASSEMBLY);
    
	*A = h;
    *flag = DIFFERENT_NONZERO_PATTERN;

    return err;
}

And my Monitor function is:

PetscErrorCode Monitor(TS ts, PetscInt step, PetscReal t, Vec u, void *cntx)
{
    Context         *context = (Context*) cntx;
    PetscErrorCode  ierr = 0;
    PetscScalar     ef = eField(t, context->params);
    
    if (context->rank == 0 && context->progress) std::cout << step << ": " << t << " ef: " << ef << std::endl;
    
	return ierr;
}

I looked at it with valgrind, and got a summary of:

==24941== LEAK SUMMARY:
==24941==    definitely lost: 0 bytes in 0 blocks
==24941==    indirectly lost: 0 bytes in 0 blocks
==24941==      possibly lost: 1,291,626,491 bytes in 4,050 blocks
==24941==    still reachable: 1,795,550 bytes in 88 blocks
==24941==         suppressed: 0 bytes in 0 blocks

with a bunch of things like this:

==24937== 757,113,160 bytes in 10 blocks are possibly lost in loss record 1,834 of 1,834
==24937==    at 0x4A05306: memalign (vg_replace_malloc.c:532)
==24937==    by 0x43083C: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*
, void**) (mal.c:30)
==24937==    by 0x434144: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char co
nst*, void**) (mtr.c:196)
==24937==    by 0x8641BF: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3550)
==24937==    by 0x861DBA: MatSeqAIJSetPreallocation(_p_Mat*, int, int const*) (aij.c:3493)
==24937==    by 0x9201B5: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3209)
==24937==    by 0x922600: MatMPIAIJSetPreallocation(_p_Mat*, int, int const*, int, int const*) (mpiaij.c:3897)
==24937==    by 0x9315BA: MatAXPY_MPIAIJ(_p_Mat*, std::complex<double>, _p_Mat*, MatStructure) (mpiaij.c:2254)
==24937==    by 0x5A9828: MatAXPY(_p_Mat*, std::complex<double>, _p_Mat*, MatStructure) (axpy.c:39)
==24937==    by 0x405EF7: HamiltonianJ(_p_TS*, double, _p_Vec*, _p_Mat**, _p_Mat**, MatStructure*, void*) (in /home/becker/ansp6066/code/EnergyBasisSchrodingerSolver/bin/propagate)
==24937==    by 0x736441: TSComputeRHSJacobian(_p_TS*, double, _p_Vec*, _p_Mat**, _p_Mat**, MatStructure*) (ts.c:186)
==24937==    by 0x736D68: TSComputeRHSFunctionLinear(_p_TS*, double, _p_Vec*, _p_Vec*, void*) (ts.c:2541)

I imagine these are not actual memory errors, but I thought I'm not sure what is leaking, so I thought I would bring it to your attention.

I can attach the full valgrind output, but at 2mb, it is rather large, so I figured I would wait for someone to ask for it.

Any idea what the memory leak could be?  Am I doing something stupid?

Thanks for the help, as always.

-Andrew

From mike.hui.zhang at hotmail.com  Wed May 16 03:46:24 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Wed, 16 May 2012 10:46:24 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CA+Y_9UJ3TsAqmycjqgFBzedo3nkCM8zkNC=SsR1Sco9OqjSxGw@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<CA+Y_9UJ3TsAqmycjqgFBzedo3nkCM8zkNC=SsR1Sco9OqjSxGw@mail.gmail.com>
Message-ID: <BLU0-SMTP4889A2ABCCD0BD3F7D9D43CB180@phx.gbl>


A question about PCASMGetSubKSP(), it says that we must call KSPSetUp() before calling
PCASMGetSubKSP(). But what ksp should be SetUp, the ksp using PCASM or what? 

Thanks!



>> 
>> On Tue, May 15, 2012 at 10:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> thanks for reply!
>> On May 15, 2012, at 5:19 PM, Dmitry Karpeev wrote:
>> 
>>> 
>>> 
>>> On Tue, May 15, 2012 at 9:55 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> Dmitry,
>>> 
>>> thanks for remind.  I have a new question about PCASM / PCGASM: 
>>> 
>>> can I get the restricted extension operators, which maps an overlapping subdomain solution
>>> to the global domain?
>>>  
>>> I'm not exactly sure what you mean.
>>> 
>>> Are you talking about embedding the subdomain vectors back into the original vector?
>> 
>> Yes, exactly.
>> 
>>> If so, there is substantial difference in how this is handled in ASM and GASM:
>>> ASM has a bunch of sequential vectors that can be scattered back into the global vector,
>> 
>> Yes. Is there a method to get the scatter?
>> 
>> In the ASM case it's a bunch of scatters -- one for each subdomain.  Currently there is no method to 
>> retrieve them.  
> 
> this hint is very helpful. Thanks!  
> 
>> What requires this functionality?
> 
> I am writing some modified ASM method. In construction of energy minimization coarse basis,
> I need to solve individual subdomain problems and not to sum them, just to extend them separately.
> I wonder whether you guys have ever done this coarse basis.
> Is there a reference where the basis is described?
> Dmtiry. 
> 
> Thanks,
> Hui
> 
>> 
>> In the ASM case you can construct the scatters yourself easily enough, 
>> since you have all of the requisite information -- the array of subdomain ISs and the global vector x.
>> The only piece of data you might not have is the set of outer subdomains that have been obtained
>> by applying overlap increase to the original inner (nonoverlapping) subdomains.
>> 
>>> because the subdomains are always local to at most one processor.  
>>> 
>>> In the GASM case this is rather different, since the subdomains can live on arbitrary subcommunicators 
>>> and there is only one scatter, which is applied to the direct sum of all the subdomain vectors on the original communicator. I'm not sure how useful that last scatter would be for you, since the details of the structure 
>>> of the direct sum vector are internal to GASM.
>> 
>> I would prefer to have the scatter for individual subdomain before direct sum. 
>> But if I can get the scatter PCGASM has, maybe it is still useful. Please tell me how to get it?
>> There are no individual subdomain scatters, but, as in the case of ASM, you can  construct them
>> easily enough, except that those would have to operate on subcommunicators.
>> In GASM we pack them into a single scatter on the original communicator. Currently there is no method
>> to expose this scatter. Why do you need this functionality? 
>> 
>> Dmitry.
>> Thanks!
>> 
>>> 
>>> Dmitry. 
>>> 
>>> Thanks!
>>> 
>>> On May 15, 2012, at 3:29 PM, Dmitry Karpeev wrote:
>>> 
>>>> There are some additional minor fixes that mostly have to do with outputting the subdomain information with  -pc_gasm_view_subdomains (in PCView()) and with -pc_gasm_print_subdomains (during PCSetUp()).
>>>> You might want to pull those latest patches, but it won't interfere with your work if you don't use subdomain output.
>>>> 
>>>> Thanks.
>>>> Dmitry.
>>>> 
>>>> On Tue, May 15, 2012 at 7:14 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> Dmitry,
>>>> 
>>>> thanks for reply. I re-download the codes and tried it again and now it works correctly!
>>>> 
>>>> Everything seems ok.
>>>> 
>>>> Thanks,
>>>> Hui
>>>> 
>>>> 
>>>> On May 15, 2012, at 2:01 PM, Dmitry Karpeev wrote:
>>>> 
>>>>> Hui,
>>>>> I'm trying to reproduce this problem, unsuccessfully, so far.
>>>>> One thing that looks odd is that the output below claims the PC is of type "asm", even though you are running with -dd_type gasm.  Could you verify that's the correct output?
>>>>> 
>>>>> Here's the output I get with 
>>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type asm -dd_ksp_view
>>>>> 
>>>>> PC Object:(dd_) 1 MPI processes
>>>>>   type: asm
>>>>>     Additive Schwarz: total subdomain blocks = 2, user-defined overlap
>>>>>     Additive Schwarz: restriction/interpolation type - RESTRICT
>>>>>     Local solve is same for all blocks, in the following KSP and PC objects:
>>>>>     KSP Object:    (dd_sub_)     1 MPI processes
>>>>>       type: preonly
>>>>>       maximum iterations=10000, initial guess is zero
>>>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>       left preconditioning
>>>>>       using NONE norm type for convergence test
>>>>>       <snip>
>>>>> 
>>>>> and with 
>>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type gasm -dd_ksp_view
>>>>> 
>>>>> PC Object:(dd_) 1 MPI processes
>>>>>   type: gasm
>>>>>   Generalized additive Schwarz:
>>>>>   Restriction/interpolation type: RESTRICT
>>>>>   user-defined overlap
>>>>>   total number of subdomains = 2
>>>>>   number of local subdomains = 2
>>>>>   max number of local subdomains = 2
>>>>>   [0:1] number of locally-supported subdomains = 2
>>>>>   Subdomain solver info is as follows:
>>>>>   <snip>
>>>>> 
>>>>> What convergence are you seeing with the two PC types?  It should be the same with 1 and 2 procs for both PCASM and PCGASM.  
>>>>> 
>>>>> Thanks.
>>>>> Dmitry.
>>>>> 
>>>>> On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> Dmitry,
>>>>> 
>>>>> I got the newest petsc-dev and I run the test by
>>>>> 
>>>>>     mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
>>>>> 
>>>>> which gives the following output
>>>>> 
>>>>> PC Object:(dd_) 1 MPI processes
>>>>>   type: asm
>>>>>     Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1
>>>>>                                               ^^^
>>>>>                                          note the above number, it should be 2 
>>>>> 
>>>>> While PCASM has no such problem.
>>>>> 
>>>>> Thanks,
>>>>> Hui
>>>>> 
>>>>> 
>>>>> 
>>>>>> Hui,
>>>>>> 
>>>>>> The convergence issue should be resolved in the latest petsc-dev.
>>>>>> I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) 
>>>>>> that should verify that.
>>>>>> 
>>>>>> Let me know if it works for you.
>>>>>> Thanks.
>>>>>> Dmitry.
>>>>>> On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> Hi Dmitry,
>>>>>> 
>>>>>> thanks for useful hints.  Good day!
>>>>>> 
>>>>>> Hui
>>>>>> 
>>>>>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>>>>> 
>>>>>>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
>>>>>>> your modification subroutine), but not the subdomains or the subdomain solvers. 
>>>>>>> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
>>>>>>> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
>>>>>>> 
>>>>>>> Dmitry.
>>>>>>> 
>>>>>>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>> I just have a question about reuse of PCASM or PCGASM.
>>>>>>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>>>>>>> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
>>>>>>> different user context for the modifying routine.  
>>>>>>> 
>>>>>>> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
>>>>>>> even for PCASM I can re-use it because the partition of subdomains remain the same.
>>>>>>> 
>>>>>>> Thanks!
>>>>>>> 
>>>>>>> 
>>>>>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>>>>> 
>>>>>>>> Hui,
>>>>>>>> There've been several changes to PCGASM ahead of the new release.
>>>>>>>> Let me go back and see if it affected the convergence problem.
>>>>>>>> Dmitry.
>>>>>>>> 
>>>>>>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>> Hi Dmitry,
>>>>>>>> 
>>>>>>>> is there any news about PCGASM? 
>>>>>>>> 
>>>>>>>> thanks,
>>>>>>>> Hui
>>>>>>>> 
>>>>>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>>>>> 
>>>>>>>>> Okay, thanks.
>>>>>>>>> I'll take a look.
>>>>>>>>> 
>>>>>>>>> Dmitry.
>>>>>>>>> 
>>>>>>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>> For reference, my results are attached.
>>>>>>>>> 
>>>>>>>>> asm1.txt for asm with 1 process,
>>>>>>>>> asm2.txt for asm with 2 processes,
>>>>>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>>>>>>>> gasm2.txt for gasm with 2 processes
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> thank you,
>>>>>>>>> Hui
>>>>>>>>> 
>>>>>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>>>>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>>>>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>>>>>>>> with GASM but no problems with ASM. 
>>>>>>>>>>> 
>>>>>>>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>>>>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>>>>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>>>>>>>> number is different from the first case and is much larger.  On the other hand
>>>>>>>>>>> ASM has no such problem.
>>>>>>>>>>>  
>>>>>>>>>>> Are the solutions the same?
>>>>>>>>>>> What problem are you solving?
>>>>>>>>>> 
>>>>>>>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>>>>>>>> two processors. But GASM did not.  
>>>>>>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>>>>>>>> I'm solving the Helmholtz equation.  Maybe 
>>>>>>>>>> I can prepare a simpler example to show this difference.
>>>>>>>>>> That would be helpful.  
>>>>>>>>>> Thanks.
>>>>>>>>>> 
>>>>>>>>>> Dmitry. 
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Dmitry. 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> You should be able to. 
>>>>>>>>>>>> This behavior is the same as in PCASM,
>>>>>>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>>>>>> 
>>>>>>>>>>>> Dmitry
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Dmitry,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> As I tested, the row and col are always the same. 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>>>>>>>> in the above func()?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>> Hui
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>> Hui
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Yes, that's right.
>>>>>>>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Dmitry.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Hui
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> <gasm_test.c>
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/397c59ef/attachment-0001.htm>

From aron.ahmadia at kaust.edu.sa  Wed May 16 03:51:03 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Wed, 16 May 2012 11:51:03 +0300
Subject: [petsc-users] How much memory does TS use (in terms of # of
	matrices)
In-Reply-To: <B461B804-6D7C-4365-AB82-A2B7CB7DAC10@gmail.com>
References: <B461B804-6D7C-4365-AB82-A2B7CB7DAC10@gmail.com>
Message-ID: <CADVoUZSiqhjJWE1FOn7AO1TtT4Sm9m5rBSexx2qkMnDVLKzj=A@mail.gmail.com>

Umnn, I hope this isn't too obvious, but you're duplicating your matrix at
every time step without freeing it and expecting that to not be a problem?

A

On Wed, May 16, 2012 at 9:01 AM, Andrew Spott <andrew.spott at gmail.com>wrote:

> I'm attempting to run some code with a rather large (>1GB) sparse matrix,
> and I keep getting kicked out of the que for what I believe is using too
> much memory.  The unfortunate thing is that as the steps go up, the memory
> usage seems to as well, which would normally indicate a memory leak.
>
> But I'm not really doing anything that would lead to a memory leak, my
> jacobian is:
>
> PetscErrorCode HamiltonianJ(TS timeStepContext, PetscReal t, Vec u, Mat
> *A, Mat *B, MatStructure *flag, void* context)
> {
>    Mat h;
>    PetscErrorCode  err;
>    Context         *cntx = (Context*)context;
>    PetscScalar     ef = eField(t, cntx->params);
>    //if (cntx->rank == 0) std::cout << "ef: " << ef;
>
>
>    err = MatDuplicate(cntx->energy_eigenstates,MAT_COPY_VALUES,&h);
>    err = MatAXPY(h, ef, cntx->dipole_matrix, DIFFERENT_NONZERO_PATTERN);
>
>        err = MatAssemblyBegin(h, MAT_FINAL_ASSEMBLY);
>        err = MatAssemblyEnd(h, MAT_FINAL_ASSEMBLY);
>
>        *A = h;
>    *flag = DIFFERENT_NONZERO_PATTERN;
>
>    return err;
> }
>
> And my Monitor function is:
>
> PetscErrorCode Monitor(TS ts, PetscInt step, PetscReal t, Vec u, void
> *cntx)
> {
>    Context         *context = (Context*) cntx;
>    PetscErrorCode  ierr = 0;
>    PetscScalar     ef = eField(t, context->params);
>
>    if (context->rank == 0 && context->progress) std::cout << step << ": "
> << t << " ef: " << ef << std::endl;
>
>        return ierr;
> }
>
> I looked at it with valgrind, and got a summary of:
>
> ==24941== LEAK SUMMARY:
> ==24941==    definitely lost: 0 bytes in 0 blocks
> ==24941==    indirectly lost: 0 bytes in 0 blocks
> ==24941==      possibly lost: 1,291,626,491 bytes in 4,050 blocks
> ==24941==    still reachable: 1,795,550 bytes in 88 blocks
> ==24941==         suppressed: 0 bytes in 0 blocks
>
> with a bunch of things like this:
>
> ==24937== 757,113,160 bytes in 10 blocks are possibly lost in loss record
> 1,834 of 1,834
> ==24937==    at 0x4A05306: memalign (vg_replace_malloc.c:532)
> ==24937==    by 0x43083C: PetscMallocAlign(unsigned long, int, char
> const*, char const*, char const*
> , void**) (mal.c:30)
> ==24937==    by 0x434144: PetscTrMallocDefault(unsigned long, int, char
> const*, char const*, char co
> nst*, void**) (mtr.c:196)
> ==24937==    by 0x8641BF: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3550)
> ==24937==    by 0x861DBA: MatSeqAIJSetPreallocation(_p_Mat*, int, int
> const*) (aij.c:3493)
> ==24937==    by 0x9201B5: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3209)
> ==24937==    by 0x922600: MatMPIAIJSetPreallocation(_p_Mat*, int, int
> const*, int, int const*) (mpiaij.c:3897)
> ==24937==    by 0x9315BA: MatAXPY_MPIAIJ(_p_Mat*, std::complex<double>,
> _p_Mat*, MatStructure) (mpiaij.c:2254)
> ==24937==    by 0x5A9828: MatAXPY(_p_Mat*, std::complex<double>, _p_Mat*,
> MatStructure) (axpy.c:39)
> ==24937==    by 0x405EF7: HamiltonianJ(_p_TS*, double, _p_Vec*, _p_Mat**,
> _p_Mat**, MatStructure*, void*) (in
> /home/becker/ansp6066/code/EnergyBasisSchrodingerSolver/bin/propagate)
> ==24937==    by 0x736441: TSComputeRHSJacobian(_p_TS*, double, _p_Vec*,
> _p_Mat**, _p_Mat**, MatStructure*) (ts.c:186)
> ==24937==    by 0x736D68: TSComputeRHSFunctionLinear(_p_TS*, double,
> _p_Vec*, _p_Vec*, void*) (ts.c:2541)
>
> I imagine these are not actual memory errors, but I thought I'm not sure
> what is leaking, so I thought I would bring it to your attention.
>
> I can attach the full valgrind output, but at 2mb, it is rather large, so
> I figured I would wait for someone to ask for it.
>
> Any idea what the memory leak could be?  Am I doing something stupid?
>
> Thanks for the help, as always.
>
> -Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/f39c9f1e/attachment.htm>

From mike.hui.zhang at hotmail.com  Wed May 16 03:52:34 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Wed, 16 May 2012 10:52:34 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP4889A2ABCCD0BD3F7D9D43CB180@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<CA+Y_9UJ3TsAqmycjqgFBzedo3nkCM8zkNC=SsR1Sco9OqjSxGw@mail.gmail.com>
	<BLU0-SMTP4889A2ABCCD0BD3F7D9D43CB180@phx.gbl>
Message-ID: <BLU0-SMTP36950F87D949651395C79DFCB180@phx.gbl>

It's ok. I found the example program. 

On May 16, 2012, at 10:46 AM, Hui Zhang wrote:

> 
> A question about PCASMGetSubKSP(), it says that we must call KSPSetUp() before calling
> PCASMGetSubKSP(). But what ksp should be SetUp, the ksp using PCASM or what? 
> 
> Thanks!
> 
> 
> 
>>> 
>>> On Tue, May 15, 2012 at 10:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> thanks for reply!
>>> On May 15, 2012, at 5:19 PM, Dmitry Karpeev wrote:
>>> 
>>>> 
>>>> 
>>>> On Tue, May 15, 2012 at 9:55 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> Dmitry,
>>>> 
>>>> thanks for remind.  I have a new question about PCASM / PCGASM: 
>>>> 
>>>> can I get the restricted extension operators, which maps an overlapping subdomain solution
>>>> to the global domain?
>>>>  
>>>> I'm not exactly sure what you mean.
>>>> 
>>>> Are you talking about embedding the subdomain vectors back into the original vector?
>>> 
>>> Yes, exactly.
>>> 
>>>> If so, there is substantial difference in how this is handled in ASM and GASM:
>>>> ASM has a bunch of sequential vectors that can be scattered back into the global vector,
>>> 
>>> Yes. Is there a method to get the scatter?
>>> 
>>> In the ASM case it's a bunch of scatters -- one for each subdomain.  Currently there is no method to 
>>> retrieve them.  
>> 
>> this hint is very helpful. Thanks!  
>> 
>>> What requires this functionality?
>> 
>> I am writing some modified ASM method. In construction of energy minimization coarse basis,
>> I need to solve individual subdomain problems and not to sum them, just to extend them separately.
>> I wonder whether you guys have ever done this coarse basis.
>> Is there a reference where the basis is described?
>> Dmtiry. 
>> 
>> Thanks,
>> Hui
>> 
>>> 
>>> In the ASM case you can construct the scatters yourself easily enough, 
>>> since you have all of the requisite information -- the array of subdomain ISs and the global vector x.
>>> The only piece of data you might not have is the set of outer subdomains that have been obtained
>>> by applying overlap increase to the original inner (nonoverlapping) subdomains.
>>> 
>>>> because the subdomains are always local to at most one processor.  
>>>> 
>>>> In the GASM case this is rather different, since the subdomains can live on arbitrary subcommunicators 
>>>> and there is only one scatter, which is applied to the direct sum of all the subdomain vectors on the original communicator. I'm not sure how useful that last scatter would be for you, since the details of the structure 
>>>> of the direct sum vector are internal to GASM.
>>> 
>>> I would prefer to have the scatter for individual subdomain before direct sum. 
>>> But if I can get the scatter PCGASM has, maybe it is still useful. Please tell me how to get it?
>>> There are no individual subdomain scatters, but, as in the case of ASM, you can  construct them
>>> easily enough, except that those would have to operate on subcommunicators.
>>> In GASM we pack them into a single scatter on the original communicator. Currently there is no method
>>> to expose this scatter. Why do you need this functionality? 
>>> 
>>> Dmitry.
>>> Thanks!
>>> 
>>>> 
>>>> Dmitry. 
>>>> 
>>>> Thanks!
>>>> 
>>>> On May 15, 2012, at 3:29 PM, Dmitry Karpeev wrote:
>>>> 
>>>>> There are some additional minor fixes that mostly have to do with outputting the subdomain information with  -pc_gasm_view_subdomains (in PCView()) and with -pc_gasm_print_subdomains (during PCSetUp()).
>>>>> You might want to pull those latest patches, but it won't interfere with your work if you don't use subdomain output.
>>>>> 
>>>>> Thanks.
>>>>> Dmitry.
>>>>> 
>>>>> On Tue, May 15, 2012 at 7:14 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> Dmitry,
>>>>> 
>>>>> thanks for reply. I re-download the codes and tried it again and now it works correctly!
>>>>> 
>>>>> Everything seems ok.
>>>>> 
>>>>> Thanks,
>>>>> Hui
>>>>> 
>>>>> 
>>>>> On May 15, 2012, at 2:01 PM, Dmitry Karpeev wrote:
>>>>> 
>>>>>> Hui,
>>>>>> I'm trying to reproduce this problem, unsuccessfully, so far.
>>>>>> One thing that looks odd is that the output below claims the PC is of type "asm", even though you are running with -dd_type gasm.  Could you verify that's the correct output?
>>>>>> 
>>>>>> Here's the output I get with 
>>>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type asm -dd_ksp_view
>>>>>> 
>>>>>> PC Object:(dd_) 1 MPI processes
>>>>>>   type: asm
>>>>>>     Additive Schwarz: total subdomain blocks = 2, user-defined overlap
>>>>>>     Additive Schwarz: restriction/interpolation type - RESTRICT
>>>>>>     Local solve is same for all blocks, in the following KSP and PC objects:
>>>>>>     KSP Object:    (dd_sub_)     1 MPI processes
>>>>>>       type: preonly
>>>>>>       maximum iterations=10000, initial guess is zero
>>>>>>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>       left preconditioning
>>>>>>       using NONE norm type for convergence test
>>>>>>       <snip>
>>>>>> 
>>>>>> and with 
>>>>>> ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type gasm -dd_ksp_view
>>>>>> 
>>>>>> PC Object:(dd_) 1 MPI processes
>>>>>>   type: gasm
>>>>>>   Generalized additive Schwarz:
>>>>>>   Restriction/interpolation type: RESTRICT
>>>>>>   user-defined overlap
>>>>>>   total number of subdomains = 2
>>>>>>   number of local subdomains = 2
>>>>>>   max number of local subdomains = 2
>>>>>>   [0:1] number of locally-supported subdomains = 2
>>>>>>   Subdomain solver info is as follows:
>>>>>>   <snip>
>>>>>> 
>>>>>> What convergence are you seeing with the two PC types?  It should be the same with 1 and 2 procs for both PCASM and PCGASM.  
>>>>>> 
>>>>>> Thanks.
>>>>>> Dmitry.
>>>>>> 
>>>>>> On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> Dmitry,
>>>>>> 
>>>>>> I got the newest petsc-dev and I run the test by
>>>>>> 
>>>>>>     mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
>>>>>> 
>>>>>> which gives the following output
>>>>>> 
>>>>>> PC Object:(dd_) 1 MPI processes
>>>>>>   type: asm
>>>>>>     Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1
>>>>>>                                               ^^^
>>>>>>                                          note the above number, it should be 2 
>>>>>> 
>>>>>> While PCASM has no such problem.
>>>>>> 
>>>>>> Thanks,
>>>>>> Hui
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> Hui,
>>>>>>> 
>>>>>>> The convergence issue should be resolved in the latest petsc-dev.
>>>>>>> I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) 
>>>>>>> that should verify that.
>>>>>>> 
>>>>>>> Let me know if it works for you.
>>>>>>> Thanks.
>>>>>>> Dmitry.
>>>>>>> On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>> Hi Dmitry,
>>>>>>> 
>>>>>>> thanks for useful hints.  Good day!
>>>>>>> 
>>>>>>> Hui
>>>>>>> 
>>>>>>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>>>>>> 
>>>>>>>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
>>>>>>>> your modification subroutine), but not the subdomains or the subdomain solvers. 
>>>>>>>> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
>>>>>>>> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
>>>>>>>> 
>>>>>>>> Dmitry.
>>>>>>>> 
>>>>>>>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>> I just have a question about reuse of PCASM or PCGASM.
>>>>>>>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>>>>>>>> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
>>>>>>>> different user context for the modifying routine.  
>>>>>>>> 
>>>>>>>> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
>>>>>>>> even for PCASM I can re-use it because the partition of subdomains remain the same.
>>>>>>>> 
>>>>>>>> Thanks!
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>>>>>> 
>>>>>>>>> Hui,
>>>>>>>>> There've been several changes to PCGASM ahead of the new release.
>>>>>>>>> Let me go back and see if it affected the convergence problem.
>>>>>>>>> Dmitry.
>>>>>>>>> 
>>>>>>>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>> Hi Dmitry,
>>>>>>>>> 
>>>>>>>>> is there any news about PCGASM? 
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Hui
>>>>>>>>> 
>>>>>>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>>>>>> 
>>>>>>>>>> Okay, thanks.
>>>>>>>>>> I'll take a look.
>>>>>>>>>> 
>>>>>>>>>> Dmitry.
>>>>>>>>>> 
>>>>>>>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>> For reference, my results are attached.
>>>>>>>>>> 
>>>>>>>>>> asm1.txt for asm with 1 process,
>>>>>>>>>> asm2.txt for asm with 2 processes,
>>>>>>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>>>>>>>>> gasm2.txt for gasm with 2 processes
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> thank you,
>>>>>>>>>> Hui
>>>>>>>>>> 
>>>>>>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>>>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>>>>>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>>>>>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>>>>>>>>> with GASM but no problems with ASM. 
>>>>>>>>>>>> 
>>>>>>>>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>>>>>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>>>>>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>>>>>>>>> number is different from the first case and is much larger.  On the other hand
>>>>>>>>>>>> ASM has no such problem.
>>>>>>>>>>>>  
>>>>>>>>>>>> Are the solutions the same?
>>>>>>>>>>>> What problem are you solving?
>>>>>>>>>>> 
>>>>>>>>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>>>>>>>>> two processors. But GASM did not.  
>>>>>>>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>>>>>>>>> I'm solving the Helmholtz equation.  Maybe 
>>>>>>>>>>> I can prepare a simpler example to show this difference.
>>>>>>>>>>> That would be helpful.  
>>>>>>>>>>> Thanks.
>>>>>>>>>>> 
>>>>>>>>>>> Dmitry. 
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Dmitry. 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> You should be able to. 
>>>>>>>>>>>>> This behavior is the same as in PCASM,
>>>>>>>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Dmitry
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Dmitry,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> As I tested, the row and col are always the same. 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>>>>>>>>> in the above func()?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>> Hui
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>> Hui
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Yes, that's right.
>>>>>>>>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Dmitry.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Hui
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> <gasm_test.c>
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/6d8a6573/attachment-0001.htm>

From mike.hui.zhang at hotmail.com  Wed May 16 05:15:04 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Wed, 16 May 2012 12:15:04 +0200
Subject: [petsc-users] shortening digits from MatView
Message-ID: <BLU0-SMTP2271E84ADABB55ED7DD6C29CB180@phx.gbl>

Is there any way to shorten the digits of outputs from MatView?
The numbers are just too long to have a good look.

Thanks!

From gmulas at oa-cagliari.inaf.it  Wed May 16 05:51:01 2012
From: gmulas at oa-cagliari.inaf.it (Giacomo Mulas)
Date: Wed, 16 May 2012 12:51:01 +0200 (CEST)
Subject: [petsc-users] How to set up -mat_mumps_icntl13 1 in the code?
In-Reply-To: <7AC25B12-94CF-4A87-9565-760054ED1ED1@mcs.anl.gov>
References: <alpine.DEB.2.02.1205141744340.14621@capitanata.oa-cagliari.inaf.it>
	<CAMYG4G=CBDdfUw4nAmsPxFsFQON6zWRtY5ikLnTsYxEbxYsgaA@mail.gmail.com>
	<alpine.DEB.2.02.1205141854490.17844@capitanata.oa-cagliari.inaf.it>
	<C4F9EBEC-89FF-4189-ACCD-C8254AB41F91@mcs.anl.gov>
	<alpine.DEB.2.02.1205151222470.15719@capitanata.oa-cagliari.inaf.it>
	<7AC25B12-94CF-4A87-9565-760054ED1ED1@mcs.anl.gov>
Message-ID: <alpine.DEB.2.02.1205161239250.17776@capitanata.oa-cagliari.inaf.it>

On Tue, 15 May 2012, Barry Smith wrote:

>> So, if I understand correctly, I am supposed to call
>> PetscOptionsInsertString() to add any options I want to have as defaults,
>> then what is the call PetscOptionsInsert() for?
>
>    It "resets" the options database with the command line options. If you
> did not call this then the InsertString() options would not be overwritten
> with the command line options.

Ah ok! It makes sense now that you explain. I could not quite understand
this from the man pages :(

>   Ignore the PetscOptionsSetFromOptions() it has nothing to do with
> anything there (it is just in that example to test it).  It does not
> affect the options data base entries.

So I can omit that. Good. However, I suspect this is not the case for the
similar slepc function EPSSetFromOptions(): I had to put that _after_ the
PetscOptionsInsertString() call, or inserted options would be ignored by
slepc. So I suppose the correct sequence should be:

PetscOptionsInsertString() # this adds my defaults
PetscOptionsInsert()       # this has command line options replace defaults
EPSSetFromOptions()        # this applies resulting options to my eps object

Is that right?

Thanks, bye
Giacomo

-- 
_________________________________________________________________

Giacomo Mulas <gmulas at oa-cagliari.inaf.it>
_________________________________________________________________

OSSERVATORIO ASTRONOMICO DI CAGLIARI
Str. 54, Loc. Poggio dei Pini * 09012 Capoterra (CA)

Tel. (OAC): +39 070 71180 248     Fax : +39 070 71180 222
Tel. (UNICA): +39 070 675 4916
_________________________________________________________________

"When the storms are raging around you, stay right where you are"
                          (Freddy Mercury)
_________________________________________________________________

From gmulas at oa-cagliari.inaf.it  Wed May 16 06:12:28 2012
From: gmulas at oa-cagliari.inaf.it (Giacomo Mulas)
Date: Wed, 16 May 2012 13:12:28 +0200 (CEST)
Subject: [petsc-users] request suggestions for most appropriate eigenvalue
	solver
Message-ID: <alpine.DEB.2.02.1205161251170.17776@capitanata.oa-cagliari.inaf.it>

Hello. This is a slepc issue.

I have developed a code to compute anharmonic corrections to a harmonic
vibrational analysis of a molecule, including an explicit treatment of
accidental resonances.
This results in setting up a number of eigenvalue problems "around" pure 
harmonic states, which are my basis set. These eigenvalue problems are
sparse, and I only need a relatively small subset of the solutions. However,
what is unusual is _which_ eigenpairs I want: I want the eigenpairs whose
eigenvectors span a subspace which covers (within a predetermined accuracy)
the few pure harmonic states I am interested in. That is, I want enough
eigenpairs that the projection of my pure harmonic state on these
eigenvectors is "close enough" to it.

So far, I am relying on spectral slicing to obtain the eigenpairs in a
neighbourhood of the pure harmonic states I start from, increasing
neighbourhood radius until I cover the starting state adequately. However,
this results in a lot of waste: many eigenstates are accidentally close to
my target harmonic states, with little or no projection on them. I end up
computing 1-2 orders of magnitude more states than the needed ones (checked
a posteriori).

The best, for my needs, would be to be able to specify, in slepc, that my
target solutions are the ones with the highest projection on some vector
(or, better, subspace spanned by some vectors), instead of using a selection
criterion based on eigenvalues closest to a target or in an interval.  Is
there some (not too complex) way to "convince" slepc to work like this?  I
can think of providing my target vectors (one by one, or a linear
combination) as a starting point to generate the Krylov subspace, but then
how do I select eigenvectors to really be the ones I want?

Thanks in advance
Giacomo

-- 
_________________________________________________________________

Giacomo Mulas <gmulas at oa-cagliari.inaf.it>
_________________________________________________________________

OSSERVATORIO ASTRONOMICO DI CAGLIARI
Str. 54, Loc. Poggio dei Pini * 09012 Capoterra (CA)

Tel. (OAC): +39 070 71180 248     Fax : +39 070 71180 222
Tel. (UNICA): +39 070 675 4916
_________________________________________________________________

"When the storms are raging around you, stay right where you are"
                          (Freddy Mercury)
_________________________________________________________________

From jroman at dsic.upv.es  Wed May 16 06:32:22 2012
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 16 May 2012 13:32:22 +0200
Subject: [petsc-users] request suggestions for most appropriate
	eigenvalue solver
In-Reply-To: <alpine.DEB.2.02.1205161251170.17776@capitanata.oa-cagliari.inaf.it>
References: <alpine.DEB.2.02.1205161251170.17776@capitanata.oa-cagliari.inaf.it>
Message-ID: <62F26FC3-EF61-4A93-8749-19913849D026@dsic.upv.es>


El 16/05/2012, a las 13:12, Giacomo Mulas escribi?:

> Hello. This is a slepc issue.
> 
> I have developed a code to compute anharmonic corrections to a harmonic
> vibrational analysis of a molecule, including an explicit treatment of
> accidental resonances.
> This results in setting up a number of eigenvalue problems "around" pure harmonic states, which are my basis set. These eigenvalue problems are
> sparse, and I only need a relatively small subset of the solutions. However,
> what is unusual is _which_ eigenpairs I want: I want the eigenpairs whose
> eigenvectors span a subspace which covers (within a predetermined accuracy)
> the few pure harmonic states I am interested in. That is, I want enough
> eigenpairs that the projection of my pure harmonic state on these
> eigenvectors is "close enough" to it.
> 
> So far, I am relying on spectral slicing to obtain the eigenpairs in a
> neighbourhood of the pure harmonic states I start from, increasing
> neighbourhood radius until I cover the starting state adequately. However,
> this results in a lot of waste: many eigenstates are accidentally close to
> my target harmonic states, with little or no projection on them. I end up
> computing 1-2 orders of magnitude more states than the needed ones (checked
> a posteriori).
> 
> The best, for my needs, would be to be able to specify, in slepc, that my
> target solutions are the ones with the highest projection on some vector
> (or, better, subspace spanned by some vectors), instead of using a selection
> criterion based on eigenvalues closest to a target or in an interval.  Is
> there some (not too complex) way to "convince" slepc to work like this?  I
> can think of providing my target vectors (one by one, or a linear
> combination) as a starting point to generate the Krylov subspace, but then
> how do I select eigenvectors to really be the ones I want?
> 
> Thanks in advance
> Giacomo

Currently there is no way to do this. But we have had a couple of similar requests before. We are now reorganizing parts of code within SLEPc, so I will think if it is viable to provide a solution for this. I will get back to you.

Jose


From gmulas at oa-cagliari.inaf.it  Wed May 16 06:44:37 2012
From: gmulas at oa-cagliari.inaf.it (Giacomo Mulas)
Date: Wed, 16 May 2012 13:44:37 +0200 (CEST)
Subject: [petsc-users] request suggestions for most appropriate
 eigenvalue solver
In-Reply-To: <62F26FC3-EF61-4A93-8749-19913849D026@dsic.upv.es>
References: <alpine.DEB.2.02.1205161251170.17776@capitanata.oa-cagliari.inaf.it>
	<62F26FC3-EF61-4A93-8749-19913849D026@dsic.upv.es>
Message-ID: <alpine.DEB.2.02.1205161338140.17776@capitanata.oa-cagliari.inaf.it>

On Wed, 16 May 2012, Jose E. Roman wrote:

> Currently there is no way to do this. But we have had a couple of similar
> requests before.  We are now reorganizing parts of code within SLEPc, so I
> will think if it is viable to provide a solution for this.  I will get
> back to you.

Great, thanks! I am more than willing to do testing and/or some coding, as
far as I can help. I am a physicist, not a mathematician superexpert in
theoretical numerical analysis nor a computer scientist. However, I am good
enough at C programming, and extremely (un)lucky in finding ways to trigger
obscure, difficult to hit bugs in code paths which not many other people 
use...

Bye, let me know.
Giacomo

-- 
_________________________________________________________________

Giacomo Mulas <gmulas at oa-cagliari.inaf.it>
_________________________________________________________________

OSSERVATORIO ASTRONOMICO DI CAGLIARI
Str. 54, Loc. Poggio dei Pini * 09012 Capoterra (CA)

Tel. (OAC): +39 070 71180 248     Fax : +39 070 71180 222
Tel. (UNICA): +39 070 675 4916
_________________________________________________________________

"When the storms are raging around you, stay right where you are"
                          (Freddy Mercury)
_________________________________________________________________

From andrew.spott at gmail.com  Wed May 16 08:08:03 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Wed, 16 May 2012 07:08:03 -0600
Subject: [petsc-users] How much memory does TS use (in terms of # of
	matrices)
In-Reply-To: <CADVoUZSiqhjJWE1FOn7AO1TtT4Sm9m5rBSexx2qkMnDVLKzj=A@mail.gmail.com>
References: <B461B804-6D7C-4365-AB82-A2B7CB7DAC10@gmail.com>
	<CADVoUZSiqhjJWE1FOn7AO1TtT4Sm9m5rBSexx2qkMnDVLKzj=A@mail.gmail.com>
Message-ID: <0AC142DA-66B2-4D18-915A-71BA1EE79222@gmail.com>

Not until you pointed it out? :).  Though it probably should have been.  It was late.

Though I'm not sure how to fix that? should I do a MatDestroy(A) at the beginning of that function every time?.

Is *A a pointer to the last used matrix, should I just do the mat duplicate into A?

Thanks for the help,

-Andrew


On May 16, 2012, at 2:51 AM, Aron Ahmadia wrote:

> Umnn, I hope this isn't too obvious, but you're duplicating your matrix at every time step without freeing it and expecting that to not be a problem?
> 
> A
> 
> On Wed, May 16, 2012 at 9:01 AM, Andrew Spott <andrew.spott at gmail.com> wrote:
> I'm attempting to run some code with a rather large (>1GB) sparse matrix, and I keep getting kicked out of the que for what I believe is using too much memory.  The unfortunate thing is that as the steps go up, the memory usage seems to as well, which would normally indicate a memory leak.
> 
> But I'm not really doing anything that would lead to a memory leak, my jacobian is:
> 
> PetscErrorCode HamiltonianJ(TS timeStepContext, PetscReal t, Vec u, Mat *A, Mat *B, MatStructure *flag, void* context)
> {
>    Mat h;
>    PetscErrorCode  err;
>    Context         *cntx = (Context*)context;
>    PetscScalar     ef = eField(t, cntx->params);
>    //if (cntx->rank == 0) std::cout << "ef: " << ef;
> 
> 
>    err = MatDuplicate(cntx->energy_eigenstates,MAT_COPY_VALUES,&h);
>    err = MatAXPY(h, ef, cntx->dipole_matrix, DIFFERENT_NONZERO_PATTERN);
> 
>        err = MatAssemblyBegin(h, MAT_FINAL_ASSEMBLY);
>        err = MatAssemblyEnd(h, MAT_FINAL_ASSEMBLY);
> 
>        *A = h;
>    *flag = DIFFERENT_NONZERO_PATTERN;
> 
>    return err;
> }
> 
> And my Monitor function is:
> 
> PetscErrorCode Monitor(TS ts, PetscInt step, PetscReal t, Vec u, void *cntx)
> {
>    Context         *context = (Context*) cntx;
>    PetscErrorCode  ierr = 0;
>    PetscScalar     ef = eField(t, context->params);
> 
>    if (context->rank == 0 && context->progress) std::cout << step << ": " << t << " ef: " << ef << std::endl;
> 
>        return ierr;
> }
> 
> I looked at it with valgrind, and got a summary of:
> 
> ==24941== LEAK SUMMARY:
> ==24941==    definitely lost: 0 bytes in 0 blocks
> ==24941==    indirectly lost: 0 bytes in 0 blocks
> ==24941==      possibly lost: 1,291,626,491 bytes in 4,050 blocks
> ==24941==    still reachable: 1,795,550 bytes in 88 blocks
> ==24941==         suppressed: 0 bytes in 0 blocks
> 
> with a bunch of things like this:
> 
> ==24937== 757,113,160 bytes in 10 blocks are possibly lost in loss record 1,834 of 1,834
> ==24937==    at 0x4A05306: memalign (vg_replace_malloc.c:532)
> ==24937==    by 0x43083C: PetscMallocAlign(unsigned long, int, char const*, char const*, char const*
> , void**) (mal.c:30)
> ==24937==    by 0x434144: PetscTrMallocDefault(unsigned long, int, char const*, char const*, char co
> nst*, void**) (mtr.c:196)
> ==24937==    by 0x8641BF: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3550)
> ==24937==    by 0x861DBA: MatSeqAIJSetPreallocation(_p_Mat*, int, int const*) (aij.c:3493)
> ==24937==    by 0x9201B5: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3209)
> ==24937==    by 0x922600: MatMPIAIJSetPreallocation(_p_Mat*, int, int const*, int, int const*) (mpiaij.c:3897)
> ==24937==    by 0x9315BA: MatAXPY_MPIAIJ(_p_Mat*, std::complex<double>, _p_Mat*, MatStructure) (mpiaij.c:2254)
> ==24937==    by 0x5A9828: MatAXPY(_p_Mat*, std::complex<double>, _p_Mat*, MatStructure) (axpy.c:39)
> ==24937==    by 0x405EF7: HamiltonianJ(_p_TS*, double, _p_Vec*, _p_Mat**, _p_Mat**, MatStructure*, void*) (in /home/becker/ansp6066/code/EnergyBasisSchrodingerSolver/bin/propagate)
> ==24937==    by 0x736441: TSComputeRHSJacobian(_p_TS*, double, _p_Vec*, _p_Mat**, _p_Mat**, MatStructure*) (ts.c:186)
> ==24937==    by 0x736D68: TSComputeRHSFunctionLinear(_p_TS*, double, _p_Vec*, _p_Vec*, void*) (ts.c:2541)
> 
> I imagine these are not actual memory errors, but I thought I'm not sure what is leaking, so I thought I would bring it to your attention.
> 
> I can attach the full valgrind output, but at 2mb, it is rather large, so I figured I would wait for someone to ask for it.
> 
> Any idea what the memory leak could be?  Am I doing something stupid?
> 
> Thanks for the help, as always.
> 
> -Andrew
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/2dc2f2ac/attachment.htm>

From jedbrown at mcs.anl.gov  Wed May 16 08:17:32 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 16 May 2012 07:17:32 -0600
Subject: [petsc-users] How much memory does TS use (in terms of # of
	matrices)
In-Reply-To: <0AC142DA-66B2-4D18-915A-71BA1EE79222@gmail.com>
References: <B461B804-6D7C-4365-AB82-A2B7CB7DAC10@gmail.com>
	<CADVoUZSiqhjJWE1FOn7AO1TtT4Sm9m5rBSexx2qkMnDVLKzj=A@mail.gmail.com>
	<0AC142DA-66B2-4D18-915A-71BA1EE79222@gmail.com>
Message-ID: <CAM9tzS=T6+GNcHcg5b5Zzi7mF+H=_oTxmYmXoPAksf3cG0Z59A@mail.gmail.com>

On Wed, May 16, 2012 at 7:08 AM, Andrew Spott <andrew.spott at gmail.com>wrote:

> Not until you pointed it out? :).  Though it probably should have been.
>  It was late.
>
> Though I'm not sure how to fix that? should I do a MatDestroy(A) at the
> beginning of that function every time?.
>
> Is *A a pointer to the last used matrix, should I just do the mat
> duplicate into A?
>

Do you really mean to only change A (the operator), but not B (the
preconditioning matrix)?

Why do you want to create a new matrix instead of reusing those?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/5b2ea259/attachment-0001.htm>

From andrew.spott at gmail.com  Wed May 16 08:38:55 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Wed, 16 May 2012 07:38:55 -0600
Subject: [petsc-users] How much memory does TS use (in terms of # of
	matrices)
In-Reply-To: <CAM9tzS=T6+GNcHcg5b5Zzi7mF+H=_oTxmYmXoPAksf3cG0Z59A@mail.gmail.com>
References: <B461B804-6D7C-4365-AB82-A2B7CB7DAC10@gmail.com>
	<CADVoUZSiqhjJWE1FOn7AO1TtT4Sm9m5rBSexx2qkMnDVLKzj=A@mail.gmail.com>
	<0AC142DA-66B2-4D18-915A-71BA1EE79222@gmail.com>
	<CAM9tzS=T6+GNcHcg5b5Zzi7mF+H=_oTxmYmXoPAksf3cG0Z59A@mail.gmail.com>
Message-ID: <7619C4A4-7466-470B-9024-24919DF5CF69@gmail.com>

I'm honestly not sure why I did that, however what I thought would be the obvious fix ( 'Mat h = *A' ) isn't, I'm still getting the error.

Is there a reason I can't do that?

-Andrew


On May 16, 2012, at 7:17 AM, Jed Brown wrote:

> On Wed, May 16, 2012 at 7:08 AM, Andrew Spott <andrew.spott at gmail.com> wrote:
> Not until you pointed it out? :).  Though it probably should have been.  It was late.
> 
> Though I'm not sure how to fix that? should I do a MatDestroy(A) at the beginning of that function every time?.
> 
> Is *A a pointer to the last used matrix, should I just do the mat duplicate into A?
> 
> Do you really mean to only change A (the operator), but not B (the preconditioning matrix)?
> 
> Why do you want to create a new matrix instead of reusing those?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/b52b1c4c/attachment.htm>

From andrew.spott at gmail.com  Wed May 16 09:17:46 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Wed, 16 May 2012 08:17:46 -0600
Subject: [petsc-users] How much memory does TS use (in terms of # of
	matrices)
In-Reply-To: <CAM9tzS=T6+GNcHcg5b5Zzi7mF+H=_oTxmYmXoPAksf3cG0Z59A@mail.gmail.com>
References: <B461B804-6D7C-4365-AB82-A2B7CB7DAC10@gmail.com>
	<CADVoUZSiqhjJWE1FOn7AO1TtT4Sm9m5rBSexx2qkMnDVLKzj=A@mail.gmail.com>
	<0AC142DA-66B2-4D18-915A-71BA1EE79222@gmail.com>
	<CAM9tzS=T6+GNcHcg5b5Zzi7mF+H=_oTxmYmXoPAksf3cG0Z59A@mail.gmail.com>
Message-ID: <1D6DA7E5-522C-43AF-BFF6-8906F059947D@gmail.com>

Ok, so now I'm leaking memory without even creating the extra matrix:

PetscErrorCode HamiltonianJ(TS timeStepContext, PetscReal t, Vec u, Mat *A, Mat *B, MatStructure *flag, void* context)
{
    PetscErrorCode  err;
    Context         *cntx = (Context*)context;
    PetscScalar     ef = eField(t, cntx->params);
    //if (cntx->rank == 0) std::cout << "ef: " << ef;
    
    err = MatDuplicate(cntx->energy_eigenstates,MAT_COPY_VALUES,A);
    err = MatAXPY(*A, ef, cntx->dipole_matrix, DIFFERENT_NONZERO_PATTERN);

	err = MatAssemblyBegin(*A, MAT_FINAL_ASSEMBLY);
	err = MatAssemblyEnd(*A, MAT_FINAL_ASSEMBLY);
    
    *flag = DIFFERENT_NONZERO_PATTERN;

    // absorb!

    return err;
}

I'm honestly stumped now?

-Andrew


On May 16, 2012, at 7:17 AM, Jed Brown wrote:

> On Wed, May 16, 2012 at 7:08 AM, Andrew Spott <andrew.spott at gmail.com> wrote:
> Not until you pointed it out? :).  Though it probably should have been.  It was late.
> 
> Though I'm not sure how to fix that? should I do a MatDestroy(A) at the beginning of that function every time?.
> 
> Is *A a pointer to the last used matrix, should I just do the mat duplicate into A?
> 
> Do you really mean to only change A (the operator), but not B (the preconditioning matrix)?
> 
> Why do you want to create a new matrix instead of reusing those?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/4e40476a/attachment.htm>

From jedbrown at mcs.anl.gov  Wed May 16 09:20:44 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 16 May 2012 08:20:44 -0600
Subject: [petsc-users] How much memory does TS use (in terms of # of
	matrices)
In-Reply-To: <1D6DA7E5-522C-43AF-BFF6-8906F059947D@gmail.com>
References: <B461B804-6D7C-4365-AB82-A2B7CB7DAC10@gmail.com>
	<CADVoUZSiqhjJWE1FOn7AO1TtT4Sm9m5rBSexx2qkMnDVLKzj=A@mail.gmail.com>
	<0AC142DA-66B2-4D18-915A-71BA1EE79222@gmail.com>
	<CAM9tzS=T6+GNcHcg5b5Zzi7mF+H=_oTxmYmXoPAksf3cG0Z59A@mail.gmail.com>
	<1D6DA7E5-522C-43AF-BFF6-8906F059947D@gmail.com>
Message-ID: <CAM9tzS=c90=Fd2t=cSoQUnbDuKnzhBQN5Fqv2h+Uw-jfd8P4fQ@mail.gmail.com>

On Wed, May 16, 2012 at 8:17 AM, Andrew Spott <andrew.spott at gmail.com>wrote:

> Ok, so now I'm leaking memory without even creating the extra matrix:
>

Send -malloc_dump output


>
> PetscErrorCode HamiltonianJ(TS timeStepContext, PetscReal t, Vec u, Mat
> *A, Mat *B, MatStructure *flag, void* context)
> {
>     PetscErrorCode  err;
>     Context         *cntx = (Context*)context;
>     PetscScalar     ef = eField(t, cntx->params);
>     //if (cntx->rank == 0) std::cout << "ef: " << ef;
>
>     err = MatDuplicate(cntx->energy_eigenstates,MAT_COPY_VALUES,A);
>     err = MatAXPY(*A, ef, cntx->dipole_matrix, DIFFERENT_NONZERO_PATTERN);
>
> err = MatAssemblyBegin(*A, MAT_FINAL_ASSEMBLY);
> err = MatAssemblyEnd(*A, MAT_FINAL_ASSEMBLY);
>
>     *flag = DIFFERENT_NONZERO_PATTERN;
>
>     // absorb!
>
>     return err;
> }
>
> I'm honestly stumped now?
>
> -Andrew
>
>
> On May 16, 2012, at 7:17 AM, Jed Brown wrote:
>
> On Wed, May 16, 2012 at 7:08 AM, Andrew Spott <andrew.spott at gmail.com>wrote:
>
>> Not until you pointed it out? :).  Though it probably should have been.
>>  It was late.
>>
>> Though I'm not sure how to fix that? should I do a MatDestroy(A) at the
>> beginning of that function every time?.
>>
>> Is *A a pointer to the last used matrix, should I just do the mat
>> duplicate into A?
>>
>
> Do you really mean to only change A (the operator), but not B (the
> preconditioning matrix)?
>
> Why do you want to create a new matrix instead of reusing those?
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/901e425c/attachment-0001.htm>

From aron.ahmadia at kaust.edu.sa  Wed May 16 09:22:13 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Wed, 16 May 2012 17:22:13 +0300
Subject: [petsc-users] How much memory does TS use (in terms of # of
	matrices)
In-Reply-To: <1D6DA7E5-522C-43AF-BFF6-8906F059947D@gmail.com>
References: <B461B804-6D7C-4365-AB82-A2B7CB7DAC10@gmail.com>
	<CADVoUZSiqhjJWE1FOn7AO1TtT4Sm9m5rBSexx2qkMnDVLKzj=A@mail.gmail.com>
	<0AC142DA-66B2-4D18-915A-71BA1EE79222@gmail.com>
	<CAM9tzS=T6+GNcHcg5b5Zzi7mF+H=_oTxmYmXoPAksf3cG0Z59A@mail.gmail.com>
	<1D6DA7E5-522C-43AF-BFF6-8906F059947D@gmail.com>
Message-ID: <CADVoUZTki+7snG8yR1K0b45vROcct5O5pOD3+z9aRPPW=YUHAA@mail.gmail.com>

You do realize that MatDuplicate is making a copy, right?

A

On Wed, May 16, 2012 at 5:17 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> Ok, so now I'm leaking memory without even creating the extra matrix:
>
> PetscErrorCode HamiltonianJ(TS timeStepContext, PetscReal t, Vec u, Mat
> *A, Mat *B, MatStructure *flag, void* context)
> {
>     PetscErrorCode  err;
>     Context         *cntx = (Context*)context;
>     PetscScalar     ef = eField(t, cntx->params);
>     //if (cntx->rank == 0) std::cout << "ef: " << ef;
>
>     err = MatDuplicate(cntx->energy_eigenstates,MAT_COPY_VALUES,A);
>     err = MatAXPY(*A, ef, cntx->dipole_matrix, DIFFERENT_NONZERO_PATTERN);
>
> err = MatAssemblyBegin(*A, MAT_FINAL_ASSEMBLY);
> err = MatAssemblyEnd(*A, MAT_FINAL_ASSEMBLY);
>
>     *flag = DIFFERENT_NONZERO_PATTERN;
>
>     // absorb!
>
>     return err;
> }
>
> I'm honestly stumped now?
>
> -Andrew
>
>
> On May 16, 2012, at 7:17 AM, Jed Brown wrote:
>
> On Wed, May 16, 2012 at 7:08 AM, Andrew Spott <andrew.spott at gmail.com>wrote:
>
>> Not until you pointed it out? :).  Though it probably should have been.
>>  It was late.
>>
>> Though I'm not sure how to fix that? should I do a MatDestroy(A) at the
>> beginning of that function every time?.
>>
>> Is *A a pointer to the last used matrix, should I just do the mat
>> duplicate into A?
>>
>
> Do you really mean to only change A (the operator), but not B (the
> preconditioning matrix)?
>
> Why do you want to create a new matrix instead of reusing those?
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/fe67eab7/attachment-0001.htm>

From knepley at gmail.com  Wed May 16 09:28:40 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 16 May 2012 10:28:40 -0400
Subject: [petsc-users] How much memory does TS use (in terms of # of
	matrices)
In-Reply-To: <1D6DA7E5-522C-43AF-BFF6-8906F059947D@gmail.com>
References: <B461B804-6D7C-4365-AB82-A2B7CB7DAC10@gmail.com>
	<CADVoUZSiqhjJWE1FOn7AO1TtT4Sm9m5rBSexx2qkMnDVLKzj=A@mail.gmail.com>
	<0AC142DA-66B2-4D18-915A-71BA1EE79222@gmail.com>
	<CAM9tzS=T6+GNcHcg5b5Zzi7mF+H=_oTxmYmXoPAksf3cG0Z59A@mail.gmail.com>
	<1D6DA7E5-522C-43AF-BFF6-8906F059947D@gmail.com>
Message-ID: <CAMYG4GmSWYrBFNUgOrUr+7ZbSsuNNz4BRqg7tF2eHida_qM41A@mail.gmail.com>

On Wed, May 16, 2012 at 10:17 AM, Andrew Spott <andrew.spott at gmail.com>wrote:

> Ok, so now I'm leaking memory without even creating the extra matrix:
>
> PetscErrorCode HamiltonianJ(TS timeStepContext, PetscReal t, Vec u, Mat
> *A, Mat *B, MatStructure *flag, void* context)
> {
>     PetscErrorCode  err;
>     Context         *cntx = (Context*)context;
>     PetscScalar     ef = eField(t, cntx->params);
>     //if (cntx->rank == 0) std::cout << "ef: " << ef;
>
>     err = MatDuplicate(cntx->energy_eigenstates,MAT_COPY_VALUES,A);
>

I think you do not understand what this call does. It allocates a new
matrix. That is what Duplicate means.

   Matt


>     err = MatAXPY(*A, ef, cntx->dipole_matrix, DIFFERENT_NONZERO_PATTERN);
>
> err = MatAssemblyBegin(*A, MAT_FINAL_ASSEMBLY);
> err = MatAssemblyEnd(*A, MAT_FINAL_ASSEMBLY);
>
>     *flag = DIFFERENT_NONZERO_PATTERN;
>
>     // absorb!
>
>     return err;
> }
>
> I'm honestly stumped now?
>
> -Andrew
>
>
> On May 16, 2012, at 7:17 AM, Jed Brown wrote:
>
> On Wed, May 16, 2012 at 7:08 AM, Andrew Spott <andrew.spott at gmail.com>wrote:
>
>> Not until you pointed it out? :).  Though it probably should have been.
>>  It was late.
>>
>> Though I'm not sure how to fix that? should I do a MatDestroy(A) at the
>> beginning of that function every time?.
>>
>> Is *A a pointer to the last used matrix, should I just do the mat
>> duplicate into A?
>>
>
> Do you really mean to only change A (the operator), but not B (the
> preconditioning matrix)?
>
> Why do you want to create a new matrix instead of reusing those?
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/abc8d65f/attachment-0001.htm>

From jedbrown at mcs.anl.gov  Wed May 16 09:33:02 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 16 May 2012 08:33:02 -0600
Subject: [petsc-users] How much memory does TS use (in terms of # of
	matrices)
In-Reply-To: <CADVoUZTki+7snG8yR1K0b45vROcct5O5pOD3+z9aRPPW=YUHAA@mail.gmail.com>
References: <B461B804-6D7C-4365-AB82-A2B7CB7DAC10@gmail.com>
	<CADVoUZSiqhjJWE1FOn7AO1TtT4Sm9m5rBSexx2qkMnDVLKzj=A@mail.gmail.com>
	<0AC142DA-66B2-4D18-915A-71BA1EE79222@gmail.com>
	<CAM9tzS=T6+GNcHcg5b5Zzi7mF+H=_oTxmYmXoPAksf3cG0Z59A@mail.gmail.com>
	<1D6DA7E5-522C-43AF-BFF6-8906F059947D@gmail.com>
	<CADVoUZTki+7snG8yR1K0b45vROcct5O5pOD3+z9aRPPW=YUHAA@mail.gmail.com>
Message-ID: <CAM9tzSnZPeX6d+Zefx-Ban65RPh=YHFSRvM4ZfL_krz_mh7X1Q@mail.gmail.com>

On Wed, May 16, 2012 at 8:22 AM, Aron Ahmadia <aron.ahmadia at kaust.edu.sa>wrote:

> You do realize that MatDuplicate is making a copy, right?


Maybe you were intending to use MatCopy()?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/8ceddd3b/attachment.htm>

From andrew.spott at gmail.com  Wed May 16 09:38:50 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Wed, 16 May 2012 08:38:50 -0600
Subject: [petsc-users] How much memory does TS use (in terms of # of
	matrices)
In-Reply-To: <CAM9tzSnZPeX6d+Zefx-Ban65RPh=YHFSRvM4ZfL_krz_mh7X1Q@mail.gmail.com>
References: <B461B804-6D7C-4365-AB82-A2B7CB7DAC10@gmail.com>
	<CADVoUZSiqhjJWE1FOn7AO1TtT4Sm9m5rBSexx2qkMnDVLKzj=A@mail.gmail.com>
	<0AC142DA-66B2-4D18-915A-71BA1EE79222@gmail.com>
	<CAM9tzS=T6+GNcHcg5b5Zzi7mF+H=_oTxmYmXoPAksf3cG0Z59A@mail.gmail.com>
	<1D6DA7E5-522C-43AF-BFF6-8906F059947D@gmail.com>
	<CADVoUZTki+7snG8yR1K0b45vROcct5O5pOD3+z9aRPPW=YUHAA@mail.gmail.com>
	<CAM9tzSnZPeX6d+Zefx-Ban65RPh=YHFSRvM4ZfL_krz_mh7X1Q@mail.gmail.com>
Message-ID: <9CA55644-4F7B-418A-8D0C-70BDD16129BF@gmail.com>

Yea, I think I was.  Replacing it fixed the problem

Thanks all, sorry for the trouble.

-Andrew

On May 16, 2012, at 8:33 AM, Jed Brown wrote:

> On Wed, May 16, 2012 at 8:22 AM, Aron Ahmadia <aron.ahmadia at kaust.edu.sa> wrote:
> You do realize that MatDuplicate is making a copy, right?
> 
> Maybe you were intending to use MatCopy()?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/c6fcb809/attachment.htm>

From bsmith at mcs.anl.gov  Wed May 16 13:02:40 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 16 May 2012 13:02:40 -0500
Subject: [petsc-users] How to set up -mat_mumps_icntl13 1 in the code?
In-Reply-To: <alpine.DEB.2.02.1205161239250.17776@capitanata.oa-cagliari.inaf.it>
References: <alpine.DEB.2.02.1205141744340.14621@capitanata.oa-cagliari.inaf.it>
	<CAMYG4G=CBDdfUw4nAmsPxFsFQON6zWRtY5ikLnTsYxEbxYsgaA@mail.gmail.com>
	<alpine.DEB.2.02.1205141854490.17844@capitanata.oa-cagliari.inaf.it>
	<C4F9EBEC-89FF-4189-ACCD-C8254AB41F91@mcs.anl.gov>
	<alpine.DEB.2.02.1205151222470.15719@capitanata.oa-cagliari.inaf.it>
	<7AC25B12-94CF-4A87-9565-760054ED1ED1@mcs.anl.gov>
	<alpine.DEB.2.02.1205161239250.17776@capitanata.oa-cagliari.inaf.it>
Message-ID: <E32C7790-74A5-447B-8F73-C96AA1037C0B@mcs.anl.gov>


On May 16, 2012, at 5:51 AM, Giacomo Mulas wrote:

> On Tue, 15 May 2012, Barry Smith wrote:
> 
>>> So, if I understand correctly, I am supposed to call
>>> PetscOptionsInsertString() to add any options I want to have as defaults,
>>> then what is the call PetscOptionsInsert() for?
>> 
>>   It "resets" the options database with the command line options. If you
>> did not call this then the InsertString() options would not be overwritten
>> with the command line options.
> 
> Ah ok! It makes sense now that you explain. I could not quite understand
> this from the man pages :(
> 
>>  Ignore the PetscOptionsSetFromOptions() it has nothing to do with
>> anything there (it is just in that example to test it).  It does not
>> affect the options data base entries.
> 
> So I can omit that. Good. However, I suspect this is not the case for the
> similar slepc function EPSSetFromOptions(): I had to put that _after_ the
> PetscOptionsInsertString() call, or inserted options would be ignored by
> slepc. So I suppose the correct sequence should be:
> 
> PetscOptionsInsertString() # this adds my defaults
> PetscOptionsInsert()       # this has command line options replace defaults
> EPSSetFromOptions()        # this applies resulting options to my eps object
> 
> Is that right?

  Yes. Generally I would put the PetscOptionsInsertString() and PetscOptionsInsert() right at the beginning of the program after PetscInitialize() then you don't have to monkey with it later.  Meanwhile EPSSetFromOptions() comes later in the code whenever it comes.

   Barry

> 
> Thanks, bye
> Giacomo
> 
> -- 
> _________________________________________________________________
> 
> Giacomo Mulas <gmulas at oa-cagliari.inaf.it>
> _________________________________________________________________
> 
> OSSERVATORIO ASTRONOMICO DI CAGLIARI
> Str. 54, Loc. Poggio dei Pini * 09012 Capoterra (CA)
> 
> Tel. (OAC): +39 070 71180 248     Fax : +39 070 71180 222
> Tel. (UNICA): +39 070 675 4916
> _________________________________________________________________
> 
> "When the storms are raging around you, stay right where you are"
>                         (Freddy Mercury)
> _________________________________________________________________


From andrew.spott at gmail.com  Wed May 16 15:54:40 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Wed, 16 May 2012 14:54:40 -0600
Subject: [petsc-users] Petsc Bag example redundancy
Message-ID: <96CC964F-D185-4737-9310-15F99C245B80@gmail.com>

Isn't this redundant?

104:   PetscBagRegisterReal(bag, &user->alpha, 1.0, "alpha", "Linear coefficient");
105:   PetscBagRegisterReal(bag, &user->lambda, 6.0, "lambda", "Nonlinear coefficient");
106:   PetscBagSetFromOptions(bag);
107:   PetscOptionsGetReal(PETSC_NULL,"-alpha",&user->alpha,PETSC_NULL);
108:   PetscOptionsGetReal(PETSC_NULL,"-lambda",&user->lambda,PETSC_NULL);
(from http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex8.c.html )

Doesn't the PetscBagSetFromOptions allow you to load values into the structure using "-name", which means that the capabilities in lines 107 and 108 are implicit in the "PetscBagSetFromOptions" call.

Or do I not understand what PetscBagSetFromOptions does?

-Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120516/0d334e8c/attachment.htm>

From bsmith at mcs.anl.gov  Wed May 16 17:06:46 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 16 May 2012 17:06:46 -0500
Subject: [petsc-users] Petsc Bag example redundancy
In-Reply-To: <96CC964F-D185-4737-9310-15F99C245B80@gmail.com>
References: <96CC964F-D185-4737-9310-15F99C245B80@gmail.com>
Message-ID: <D83FC850-BF54-4BD9-BEA0-2289BCCD08D4@mcs.anl.gov>


On May 16, 2012, at 3:54 PM, Andrew Spott wrote:

> Isn't this redundant?
> 
> 104:   PetscBagRegisterReal(bag, &user->alpha, 1.0, "alpha", "Linear coefficient"
> );
> 
> 105:   PetscBagRegisterReal(bag, &user->lambda, 6.0, "lambda", "Nonlinear coefficient"
> );
> 
> 106:   PetscBagSetFromOptions
> (bag);
> 
> 107:   PetscOptionsGetReal(PETSC_NULL,"-alpha",&user->alpha,PETSC_NULL
> );
> 
> 108:   PetscOptionsGetReal(PETSC_NULL,"-lambda",&user->lambda,PETSC_NULL);
> (from http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex8.c.html )
> 
> Doesn't the PetscBagSetFromOptions allow you to load values into the structure using "-name", which means that the capabilities in lines 107 and 108 are implicit in the "PetscBagSetFromOptions" call.

   Yes they are redundant.

> 
> Or do I not understand what PetscBagSetFromOptions does?
> 
> -Andrew


From christian.staudt at ira.uka.de  Thu May 17 11:06:35 2012
From: christian.staudt at ira.uka.de (Christian Staudt)
Date: Thu, 17 May 2012 18:06:35 +0200
Subject: [petsc-users] petsc4py and high performance python
Message-ID: <CBF6E062-87F8-459C-96A6-94093BAECA86@ira.uka.de>

Hello petsc4py users,

I need some advice on petsc4py and how it can coexist with other approaches to get high performance with Python. Here's the situation:

- I am working on a reimplementation of a graph laplacian solver that uses a multigrid approach. [1]
- The original sequential implementation is in MATLAB, so Python was a natural choice because it makes it easy to translate concepts from MATLAB. The project is not small and translating everything to C++ would have been difficult and tedious. Numpy/Scipy also provide many equivalents for built-in MATLAB functions.
- Unlike the original implementation, the new version should be parallelized. Here, PETSc (via petsc4py) should provide the parallel data structures and some operations.
- There are probably functions for which pure Python does not deliver the necessary speed (and PETSc probably does not provide the operations needed). I am researching how to rewrite such performance-critical parts in C/C++, and embed them in the Python code - using methods like scipy.weave or ctypes. [2]
- My question: Is it possible to share PETSc objects between the Python code and the embedded C/C++ code? For example, pass a PETSc.Mat via Python, operate on it with C/C++ and return the matrix to Python?

In general: Do you think my approach makes sense? I am new to PETSc and have not done high performance Python before, so I hope to learn from your answers, hints and criticism. 

Chris


[1] https://code.google.com/p/lamg/
[2] http://www.scipy.org/PerformancePython#head-cafc55bbf8fd74071b2c2ebcfb6f24ed1989d540

From aron.ahmadia at kaust.edu.sa  Thu May 17 11:14:25 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Thu, 17 May 2012 19:14:25 +0300
Subject: [petsc-users] petsc4py and high performance python
In-Reply-To: <CBF6E062-87F8-459C-96A6-94093BAECA86@ira.uka.de>
References: <CBF6E062-87F8-459C-96A6-94093BAECA86@ira.uka.de>
Message-ID: <CADVoUZQBFwZiqUHsBH7eerJYvyr-xRQ3UQCMzKJrgHVOApZqsA@mail.gmail.com>

>
> - There are probably functions for which pure Python does not deliver the
> necessary speed (and PETSc probably does not provide the operations
> needed). I am researching how to rewrite such performance-critical parts in
> C/C++, and embed them in the Python code - using methods like scipy.weave
> or ctypes. [2]
>

There are a bunch of examples of doing this in the petsc4py repository in
demo/.  Start with Cython as your wrapper.


> - My question: Is it possible to share PETSc objects between the Python
> code and the embedded C/C++ code? For example, pass a PETSc.Mat via Python,
> operate on it with C/C++ and return the matrix to Python?
>

Yes.


> In general: Do you think my approach makes sense? I am new to PETSc and
> have not done high performance Python before, so I hope to learn from your
> answers, hints and criticism.
>

It depends on the project, but this is a reasonable path to consider.

A
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120517/6757f27e/attachment.htm>

From knepley at gmail.com  Thu May 17 11:14:34 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 May 2012 12:14:34 -0400
Subject: [petsc-users] petsc4py and high performance python
In-Reply-To: <CBF6E062-87F8-459C-96A6-94093BAECA86@ira.uka.de>
References: <CBF6E062-87F8-459C-96A6-94093BAECA86@ira.uka.de>
Message-ID: <CAMYG4GnqXd0UOHdZt4_789aDw8X9L6hHQmCvmP8-y7o2GtYt6w@mail.gmail.com>

On Thu, May 17, 2012 at 12:06 PM, Christian Staudt <
christian.staudt at ira.uka.de> wrote:

> Hello petsc4py users,
>
> I need some advice on petsc4py and how it can coexist with other
> approaches to get high performance with Python. Here's the situation:
>
> - I am working on a reimplementation of a graph laplacian solver that uses
> a multigrid approach. [1]
> - The original sequential implementation is in MATLAB, so Python was a
> natural choice because it makes it easy to translate concepts from MATLAB.
> The project is not small and translating everything to C++ would have been
> difficult and tedious. Numpy/Scipy also provide many equivalents for
> built-in MATLAB functions.
> - Unlike the original implementation, the new version should be
> parallelized. Here, PETSc (via petsc4py) should provide the parallel data
> structures and some operations.
> - There are probably functions for which pure Python does not deliver the
> necessary speed (and PETSc probably does not provide the operations
> needed). I am researching how to rewrite such performance-critical parts in
> C/C++, and embed them in the Python code - using methods like scipy.weave
> or ctypes. [2]
> - My question: Is it possible to share PETSc objects between the Python
> code and the embedded C/C++ code? For example, pass a PETSc.Mat via Python,
> operate on it with C/C++ and return the matrix to Python?
>

Yes, PETSc is just C code, so this works the same way.

   Matt


> In general: Do you think my approach makes sense? I am new to PETSc and
> have not done high performance Python before, so I hope to learn from your
> answers, hints and criticism.
>
> Chris
>
>
> [1] https://code.google.com/p/lamg/
> [2]
> http://www.scipy.org/PerformancePython#head-cafc55bbf8fd74071b2c2ebcfb6f24ed1989d540




-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120517/f1ef94bf/attachment.htm>

From bmhautz at sandia.gov  Thu May 17 11:37:19 2012
From: bmhautz at sandia.gov (Hautzenroeder, Brenna M)
Date: Thu, 17 May 2012 16:37:19 +0000
Subject: [petsc-users] Problem creating an array of KSP objects
Message-ID: <AC73979A4C206F458238893BE877E7020E0870A3@EXMB01.srn.sandia.gov>

I'm running into time and memory problems creating an array to store multiple KSP objects.

We are using a Fourier method for the temporal portion of our operator; currently, this is anywhere between 1024-2048 time samples equating to 1024-2048 frequencies to advance our solution. Each frequency has its own matrix, meaning there are 1024-2048 linear equations of the form Ax = b. So, while the matrices are different for each frequency, when we propagate the solution one step, the 1024-2048 matrices don't change since the frequencies haven't changed between steps.

We would like to store each KSP solution (all 1024-2048 of them) in an array so that we can reuse them for the next step since this saves us time factoring the matrix. The problem I am seeing is that storing these KSP objects in any kind of array increases the runtime and storage space exponentially - I am seeing times on the order of 30 mins to an hour and memory consumed hover around 20-30GB! Is there something inherent about the KSP object that I am missing and/or a better way to store these objects? I don't have any problems if I use one KSP object and set the operators each time to the different matrices, except that it takes longer than it normally would after the first step since it has to recalculate the preconditioner matrix.

Note: We are using distributed matrices and vectors for our linear system.

I have included some pseudocode to illustrate our linear system, which works:

KSP k;
for steps = 1 to 100 do:
for time = 1 to 1024 do:
                create distributed matrix A and fill with local values (A[step=1,time=1] = A[step=2,time=1], etc)
                create distributed vector b and fill with local values for given time
                create distributed vector x for result
                create KSP k and set matrices
                KSPSolve(k, b, x) => takes a little extra time each step

However, the following creates time/storage problems:

KSP* kArray;
create kArray => takes inordinate amout of time and storage
for steps = 1 to 100 do:
        for time=1 to 1024 do:
                create distributed matrix A and fill with local values (A[step=1,time=1] = A[step=2,time=1], etc)
                create distributed vector b and fill with local values for given time
                create distributed vector x for result
                if time == 1 do:
            set matrices on kArray[time] => also takes an inordinate amount of time and storage
                KSPSolve(kArray[time], b, x)

Any insight would be greatly appreciated!
Brenna

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120517/9516df63/attachment-0001.htm>

From knepley at gmail.com  Thu May 17 11:59:04 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 May 2012 12:59:04 -0400
Subject: [petsc-users] Problem creating an array of KSP objects
In-Reply-To: <AC73979A4C206F458238893BE877E7020E0870A3@EXMB01.srn.sandia.gov>
References: <AC73979A4C206F458238893BE877E7020E0870A3@EXMB01.srn.sandia.gov>
Message-ID: <CAMYG4GnOJ+-yXR0eDqL=+AVGhubLKKbiSK-Kp0PnNMV8JjZhgw@mail.gmail.com>

On Thu, May 17, 2012 at 12:37 PM, Hautzenroeder, Brenna M <
bmhautz at sandia.gov> wrote:

>  I?m running into time and memory problems creating an array to store
> multiple KSP objects.
>
> We are using a Fourier method for the temporal portion of our operator;
> currently, this is anywhere between 1024-2048 time samples equating to
> 1024-2048 frequencies to advance our solution. Each frequency has its own
> matrix, meaning there are 1024-2048 linear equations of the form Ax = b.
> So, while the matrices are different for each frequency, when we propagate
> the solution one step, the 1024-2048 matrices don?t change since the
> frequencies haven?t changed between steps.
>
> We would like to store each KSP solution (all 1024-2048 of them) in an
> array so that we can reuse them for the next step since this saves us time
> factoring the matrix. The problem I am seeing is that storing these KSP
> objects in any kind of array increases the runtime and storage space
> exponentially ? I am seeing times on the order of 30 mins to an hour and
> memory consumed hover around 20-30GB! Is there something inherent
>

I don't think you really mean "exponential", but just big. Did you
calculate how much space you expect? Storing a factor can take
N^3 space, and then you have 1000 of them, so it will take 1000x the space
of 1 KSP. Did you compare the storage for both cases?
There is really not enough information in your question, however

   Yes, storing KSPs is the right thing here

   No, they do not take more space than necessary

       Matt


> about the KSP object that I am missing and/or a better way to store these
> objects? I don?t have any problems if I use one KSP object and set the
> operators each time to the different matrices, except that it takes longer
> than it normally would after the first step since it has to recalculate the
> preconditioner matrix.
>
> Note: We are using distributed matrices and vectors for our linear system.
>
> I have included some pseudocode to illustrate our linear system, which
> works:
>
> KSP k;
> for steps = 1 to 100 do:
> for time = 1 to 1024 do:
>                 create distributed matrix A and fill with local values
> (A[step=1,time=1] = A[step=2,time=1], etc)
>                 create distributed vector b and fill with local values for
> given time
>                 create distributed vector x for result
>                 create KSP k and set matrices
>                 KSPSolve(k, b, x) => takes a little extra time each step
>
> However, the following creates time/storage problems:
>
> KSP* kArray;
> create kArray => takes inordinate amout of time and storage
> for steps = 1 to 100 do:
>         for time=1 to 1024 do:
>                 create distributed matrix A and fill with local values
> (A[step=1,time=1] = A[step=2,time=1], etc)
>                 create distributed vector b and fill with local values for
> given time
>                 create distributed vector x for result
>                 if time == 1 do:
> set matrices on kArray[time] => also takes an inordinate amount of time
> and storage
>                 KSPSolve(kArray[time], b, x)
>
> Any insight would be greatly appreciated!
> Brenna
>
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120517/1a5cb57c/attachment.htm>

From zhenglun.wei at gmail.com  Thu May 17 12:47:29 2012
From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei)
Date: Thu, 17 May 2012 12:47:29 -0500
Subject: [petsc-users] Solve the Poisson equation twice with
	/src/ksp/ksp/examples/tutorials/ex45.c
Message-ID: <4FB539B1.3050101@gmail.com>

Dear All,
     I hope you're having a nice day.
     I was trying to solve the Poisson equation with 
/src/ksp/ksp/exmaples/tutorial/ex45.c; however, it always give me the 
same result. I recalled the PetscLogStages and resolved this problem. 
However, the process of reconstructing DM costs a lot of computational 
time. I wonder do I have to use PetscLogStages to solve this problem, 
since, I thought, this function is only needed for using DMMG. Or as 
long as DM twice in one program, then I need to call PetscLogStages.

thanks,
Alan

From knepley at gmail.com  Thu May 17 13:11:36 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 May 2012 14:11:36 -0400
Subject: [petsc-users] Solve the Poisson equation twice with
	/src/ksp/ksp/examples/tutorials/ex45.c
In-Reply-To: <4FB539B1.3050101@gmail.com>
References: <4FB539B1.3050101@gmail.com>
Message-ID: <CAMYG4Gm8bUCMJorLo8k-G_mfw6wvR5YO1NwZ78twxUY-xDbn=Q@mail.gmail.com>

On Thu, May 17, 2012 at 1:47 PM, Zhenglun (Alan) Wei <zhenglun.wei at gmail.com
> wrote:

> Dear All,
>    I hope you're having a nice day.
>    I was trying to solve the Poisson equation with /src/ksp/ksp/exmaples/*
> *tutorial/ex45.c; however, it always give me the same result. I recalled
> the PetscLogStages and resolved this problem. However, the process of
> reconstructing DM costs a lot of computational time. I wonder do I have to
> use PetscLogStages to solve this problem, since, I thought, this function
> is only needed for using DMMG. Or as long as DM twice in one program, then
> I need to call PetscLogStages.
>

I cannot understand what you mean. PetscLog functions only affect logging.

   Matt


> thanks,
> Alan
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120517/bbb077fc/attachment.htm>

From zhenglun.wei at gmail.com  Thu May 17 13:43:52 2012
From: zhenglun.wei at gmail.com (Zhenglun (Alan) Wei)
Date: Thu, 17 May 2012 13:43:52 -0500
Subject: [petsc-users] Solve the Poisson equation twice with
	/src/ksp/ksp/examples/tutorials/ex45.c
In-Reply-To: <CAMYG4Gm8bUCMJorLo8k-G_mfw6wvR5YO1NwZ78twxUY-xDbn=Q@mail.gmail.com>
References: <4FB539B1.3050101@gmail.com>
	<CAMYG4Gm8bUCMJorLo8k-G_mfw6wvR5YO1NwZ78twxUY-xDbn=Q@mail.gmail.com>
Message-ID: <4FB546E8.10400@gmail.com>

On 5/17/2012 1:11 PM, Matthew Knepley wrote:
> On Thu, May 17, 2012 at 1:47 PM, Zhenglun (Alan) Wei 
> <zhenglun.wei at gmail.com <mailto:zhenglun.wei at gmail.com>> wrote:
>
>     Dear All,
>        I hope you're having a nice day.
>        I was trying to solve the Poisson equation with
>     /src/ksp/ksp/exmaples/tutorial/ex45.c; however, it always give me
>     the same result. I recalled the PetscLogStages and resolved this
>     problem. However, the process of reconstructing DM costs a lot of
>     computational time. I wonder do I have to use PetscLogStages to
>     solve this problem, since, I thought, this function is only needed
>     for using DMMG. Or as long as DM twice in one program, then I need
>     to call PetscLogStages.
>
>
> I cannot understand what you mean. PetscLog functions only affect logging.
>
>    Matt
>
>     thanks,
>     Alan
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
Dear Dr. Knepley,
     I'm sorry for confusing you. Here I attached a old version of 
/src/ksp/ksp/examples/tutorials/ex50.c. By adopting its approach of 
PetscLogStage, I can solve Poisson equation twice with two different RHS 
in one program. However, as you may noticed, this method, in ex50.c, is 
used for DMMG. What I understand is that the DMMG has a severe 
limitation so that the PetscLogStage is needed in order to solve the 
equation twice with two different RHS. I wonder if I have to use this 
approach for KSP? This method requires a lot of time to KSPCreate and 
DMMGCreate3d/2d in order to construct the DM. That's why I want to find 
an alternative way.

thanks,
Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120517/1ac78088/attachment.htm>
-------------- next part --------------
/*   Concepts: DMMG/KSP solving a system of linear equations.
     Poisson equation in 2D: 

     div(grad p) = f,  0 < x,y < 1
     with
       forcing function f = -cos(m*pi*x)*cos(n*pi*y),
       Neuman boundary conditions
        dp/dx = 0 for x = 0, x = 1.
        dp/dy = 0 for y = 0, y = 1.

     Contributed by Michael Boghosian <boghmic at iit.edu>, 2008,
         based on petsc/src/ksp/ksp/examples/tutorials/ex29.c and ex32.c

     Example of Usage: 
          ./ex50 -mglevels 3 -ksp_monitor -M 3 -N 3 -ksp_view -da_view_draw -draw_pause -1 
          ./ex50 -M 100 -N 100 -mglevels 1 -mg_levels_0_pc_factor_levels <ilu_levels> -ksp_monitor -cmp_solu
          ./ex50 -M 100 -N 100 -mglevels 1 -mg_levels_0_pc_type lu -mg_levels_0_pc_factor_shift_type NONZERO -ksp_monitor -cmp_solu
          mpiexec -n 4 ./ex50 -M 3 -N 3 -ksp_monitor -ksp_view -mglevels 10 -log_summary
*/

static char help[] = "Solves 2D Poisson equation using multigrid.\n\n";

#include <petscdmda.h>
#include <petscksp.h>
#include <petscpcmg.h>
#include <petscdmmg.h>
#include <petscsys.h>
#include <petscvec.h>

extern PetscErrorCode ComputeJacobian(DMMG,Mat,Mat);
extern PetscErrorCode ComputeRHS(DMMG,Vec);
extern PetscErrorCode ComputeTrueSolution(DMMG *, Vec);
extern PetscErrorCode VecView_VTK(Vec, const char [], const char []);

typedef enum {DIRICHLET, NEUMANN} BCType;

typedef struct { 
  PetscScalar  uu, tt;
  BCType       bcType;
} UserContext;

#undef __FUNCT__
#define __FUNCT__ "main"
int main(int argc,char **argv)
{
  DMMG           *dmmg;
  DM             da;
  UserContext    user;
  PetscInt       l, bc, mglevels, M, N, stages[3];
  PetscReal      norm;
  PetscErrorCode ierr;
  PetscMPIInt    rank,nproc;
  PetscBool      flg;
      
  PetscInitialize(&argc,&argv,(char *)0,help);
  ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr);
  ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&nproc);CHKERRQ(ierr);
  ierr = PetscLogStageRegister("DMMG Setup",&stages[0]);  CHKERRQ(ierr);
  ierr = PetscLogStageRegister("DMMG Solve",&stages[1]);  CHKERRQ(ierr);
      
  ierr = PetscLogStagePush(stages[0]);CHKERRQ(ierr);   /* Start DMMG Setup */
  
  /* SET VARIABLES: */
  mglevels = 1;  
  ierr = PetscOptionsGetInt(PETSC_NULL,"-mglevels",&mglevels,PETSC_NULL);  CHKERRQ(ierr);
  M = 11;        /* number of grid points in x dir. on coarse grid */
  N = 11;        /* number of grid points in y dir. on coarse grid */
  ierr = PetscOptionsGetInt(PETSC_NULL,"-M",&M,PETSC_NULL);  CHKERRQ(ierr);
  ierr = PetscOptionsGetInt(PETSC_NULL,"-N",&N,PETSC_NULL);  CHKERRQ(ierr);
  
  ierr = DMMGCreate(PETSC_COMM_WORLD,mglevels,PETSC_NULL,&dmmg); CHKERRQ(ierr);
  ierr = DMDACreate2d(PETSC_COMM_WORLD, DMDA_BOUNDARY_NONE, DMDA_BOUNDARY_NONE,DMDA_STENCIL_STAR,M,N,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,&da); CHKERRQ(ierr);  
  ierr = DMMGSetDM(dmmg,(DM)da);
  ierr = DMDestroy(&da); CHKERRQ(ierr);
  
  /* Set user contex */
  user.uu = 1.0;
  user.tt = 1.0;
  bc   = (PetscInt)NEUMANN; // Use Neumann Boundary Conditions
  user.bcType = (BCType)bc;
  for (l = 0; l < DMMGGetLevels(dmmg); l++) {
    ierr = DMMGSetUser(dmmg,l,&user); CHKERRQ(ierr);
  }  
  
  ierr = DMMGSetKSP(dmmg,ComputeRHS,ComputeJacobian);  CHKERRQ(ierr);
  if (user.bcType == NEUMANN){
     ierr = DMMGSetNullSpace(dmmg,PETSC_TRUE,0,PETSC_NULL);  CHKERRQ(ierr);
  }
  ierr = PetscLogStagePop();  CHKERRQ(ierr); /* Finish DMMG Setup */

  /* DMMG SOLVE: */
  ierr = PetscLogStagePush(stages[1]);  CHKERRQ(ierr); /* Start DMMG Solve */
  ierr = DMMGSolve(dmmg);  CHKERRQ(ierr);
  ierr = PetscLogStagePop();  CHKERRQ(ierr); /* Finish DMMG Solve */

  /* Compare solution with the true solution p */
  ierr = PetscOptionsHasName(PETSC_NULL, "-cmp_solu", &flg);CHKERRQ(ierr);
  if (flg){
    Vec   x,p;
    if (mglevels != 1) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_SUP,"mglevels must equls 1 for comparison");
    if (nproc > 1) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_SUP,"num of proc must equls 1 for comparison");
    x = DMMGGetx(dmmg);
    ierr = VecDuplicate(x,&p);CHKERRQ(ierr);
    ierr = ComputeTrueSolution(dmmg,p);CHKERRQ(ierr);

    ierr = VecAXPY(p,-1.0,x);CHKERRQ(ierr);  /* p <- (p-x) */
    ierr = VecNorm(p, NORM_2, &norm);CHKERRQ(ierr);
    if (!rank){printf("| solu_compt - solu_true | = %g\n",norm);}
    ierr = VecDestroy(&p);CHKERRQ(ierr);
  }
   
  ierr = DMMGDestroy(dmmg);CHKERRQ(ierr);
  ierr = PetscFinalize();CHKERRQ(ierr);
  return 0;
}

// COMPUTE RHS:--------------------------------------------------------------
#undef __FUNCT__
#define __FUNCT__ "ComputeRHS" 
PetscErrorCode ComputeRHS(DMMG dmmg, Vec b)
{
  DM             da = dmmg->dm;
  UserContext    *user = (UserContext *) dmmg->user;
  PetscErrorCode ierr;
  PetscInt       i, j, M, N, xm ,ym ,xs, ys;
  PetscScalar    Hx, Hy, pi, uu, tt;
  PetscScalar    **array;

  PetscFunctionBegin;
  ierr = DMDAGetInfo(da, 0, &M, &N, 0,0,0,0,0,0,0,0,0,0); CHKERRQ(ierr);
  uu = user->uu; tt = user->tt;
  pi = 4*atan(1.0); 
  Hx   = 1.0/(PetscReal)(M);
  Hy   = 1.0/(PetscReal)(N);

  ierr = DMDAGetCorners(da,&xs,&ys,0,&xm,&ym,0); CHKERRQ(ierr); // Fine grid
  //printf(" M N: %d %d; xm ym: %d %d; xs ys: %d %d\n",M,N,xm,ym,xs,ys);
  ierr = DMDAVecGetArray(da, b, &array); CHKERRQ(ierr);
  for (j=ys; j<ys+ym; j++){
    for(i=xs; i<xs+xm; i++){
      array[j][i] = -PetscCosScalar(uu*pi*((PetscReal)i+0.5)*Hx)*cos(tt*pi*((PetscReal)j+0.5)*Hy)*Hx*Hy;
    }
  }
  ierr = DMDAVecRestoreArray(da, b, &array); CHKERRQ(ierr);
  ierr = VecAssemblyBegin(b); CHKERRQ(ierr);
  ierr = VecAssemblyEnd(b); CHKERRQ(ierr);

  /* force right hand side to be consistent for singular matrix */
  /* note this is really a hack, normally the model would provide you with a consistent right handside */
  if (user->bcType == NEUMANN) {
    MatNullSpace nullspace;
    ierr = KSPGetNullSpace(dmmg->ksp,&nullspace); CHKERRQ(ierr);
    ierr = MatNullSpaceRemove(nullspace,b,PETSC_NULL); CHKERRQ(ierr);
  }
  PetscFunctionReturn(0);
}

// COMPUTE JACOBIAN:--------------------------------------------------------------
#undef __FUNCT__
#define __FUNCT__ "ComputeJacobian" 
PetscErrorCode ComputeJacobian(DMMG dmmg, Mat J, Mat jac)
{
  DM             da =  dmmg->dm;
  UserContext    *user = (UserContext *) dmmg->user;
  PetscErrorCode ierr;
  PetscInt       i, j, M, N, xm, ym, xs, ys, num, numi, numj;
  PetscScalar    v[5], Hx, Hy, HydHx, HxdHy;
  MatStencil     row, col[5];

  PetscFunctionBegin;
  ierr = DMDAGetInfo(da,0,&M,&N,0,0,0,0,0,0,0,0,0,0); CHKERRQ(ierr);  
  Hx    = 1.0 / (PetscReal)(M);
  Hy    = 1.0 / (PetscReal)(N);
  HxdHy = Hx/Hy;
  HydHx = Hy/Hx;
  ierr = DMDAGetCorners(da,&xs,&ys,0,&xm,&ym,0); CHKERRQ(ierr);
  for (j=ys; j<ys+ym; j++){
    for(i=xs; i<xs+xm; i++){
      row.i = i; row.j = j;
      
      if (i==0 || j==0 || i==M-1 || j==N-1) {
        if (user->bcType == DIRICHLET){
          SETERRQ(PETSC_COMM_SELF,PETSC_ERR_SUP,"Dirichlet boundary conditions not supported !\n");	  
        } else if (user->bcType == NEUMANN){
          num=0; numi=0; numj=0;
          if (j!=0) {
            v[num] = -HxdHy;              col[num].i = i;   col[num].j = j-1;
            num++; numj++;
          }
          if (i!=0) {
            v[num] = -HydHx;              col[num].i = i-1; col[num].j = j;
            num++; numi++;
          }
          if (i!=M-1) {
            v[num] = -HydHx;              col[num].i = i+1; col[num].j = j;
            num++; numi++;
          }
          if (j!=N-1) {
            v[num] = -HxdHy;              col[num].i = i;   col[num].j = j+1;
            num++; numj++;
          }
          v[num] = ( (PetscReal)(numj)*HxdHy + (PetscReal)(numi)*HydHx ); col[num].i = i;   col[num].j = j;
          num++;
          ierr = MatSetValuesStencil(jac,1,&row,num,col,v,INSERT_VALUES); CHKERRQ(ierr);
        }
      } else {
        v[0] = -HxdHy;              col[0].i = i;   col[0].j = j-1;
        v[1] = -HydHx;              col[1].i = i-1; col[1].j = j;
        v[2] = 2.0*(HxdHy + HydHx); col[2].i = i;   col[2].j = j;
        v[3] = -HydHx;              col[3].i = i+1; col[3].j = j;
        v[4] = -HxdHy;              col[4].i = i;   col[4].j = j+1;
        ierr = MatSetValuesStencil(jac,1,&row,5,col,v,INSERT_VALUES); CHKERRQ(ierr);
      }
    }
  }
  ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  PetscFunctionReturn(0);
}

// COMPUTE TrueSolution:--------------------------------------------------------------
#undef __FUNCT__
#define __FUNCT__ "ComputeTrueSolution" 
PetscErrorCode ComputeTrueSolution(DMMG *dmmg, Vec b)
{
  DM             da = (*dmmg)->dm;
  UserContext    *user = (UserContext *) (*dmmg)->user;
  PetscErrorCode ierr;
  PetscInt       i, j, M, N, xm ,ym ,xs, ys;
  PetscScalar    Hx, Hy, pi, uu, tt, cc;
  PetscScalar    *array;

  PetscFunctionBegin;
  ierr = DMDAGetInfo(da, 0, &M, &N, 0,0,0,0,0,0,0,0,0,0); CHKERRQ(ierr); /* level_0 ! */
  //printf("ComputeTrueSolution - M N: %d %d;\n",M,N);

  uu = user->uu; tt = user->tt;
  pi = 4*atan(1.0); 
  cc = -1.0/( (uu*pi)*(uu*pi) + (tt*pi)*(tt*pi) );
  Hx   = 1.0/(PetscReal)(M);
  Hy   = 1.0/(PetscReal)(N);

  ierr = DMDAGetCorners(da,&xs,&ys,0,&xm,&ym,0); CHKERRQ(ierr);
  ierr = VecGetArray(b, &array); CHKERRQ(ierr);
  for (j=ys; j<ys+ym; j++){
    for(i=xs; i<xs+xm; i++){
      array[j*xm+i] = cos(uu*pi*((PetscReal)i+0.5)*Hx)*cos(tt*pi*((PetscReal)j+0.5)*Hy)*cc;
    }
  }
  ierr = VecRestoreArray(b, &array); CHKERRQ(ierr);
  PetscFunctionReturn(0);
}

// VECVIEW_VTK:--------------------------------------------------------------
#undef __FUNCT__
#define __FUNCT__ "VecView_VTK" 
PetscErrorCode VecView_VTK(Vec x, const char filename[], const char bcName[])
{
  MPI_Comm           comm;
  DM                 da;
  Vec                coords;
  PetscViewer        viewer;
  PetscScalar        *array, *values;
  PetscInt           nn, NN, maxn, M, N;
  PetscInt           i, p, dof = 1;
  MPI_Status         status;
  PetscMPIInt        rank, size, tag;
  PetscErrorCode     ierr;
  PetscBool          flg;

  PetscFunctionBegin;
  ierr = PetscObjectGetComm((PetscObject) x, &comm); CHKERRQ(ierr);
  ierr = PetscViewerASCIIOpen(comm, filename, &viewer); CHKERRQ(ierr);

  ierr = VecGetSize(x, &NN); CHKERRQ(ierr);
  ierr = VecGetLocalSize(x, &nn); CHKERRQ(ierr);
  ierr = PetscObjectQuery((PetscObject) x, "DM", (PetscObject *) &da); CHKERRQ(ierr);
  if (!da) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_WRONG,"Vector not generated from a DMDA");
  ierr = PetscTypeCompare((PetscObject)da,DMDA,&flg);CHKERRQ(ierr);
  if (!flg) SETERRQ(PETSC_COMM_WORLD,PETSC_ERR_ARG_WRONG,"Vector not generated from a DMDA");

  ierr = DMDAGetInfo(da, 0, &M, &N, 0,0,0,0,&dof,0,0,0,0,0); CHKERRQ(ierr);

  ierr = PetscViewerASCIIPrintf(viewer, "# vtk DataFile Version 2.0\n"); CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "Inhomogeneous Poisson Equation with %s boundary conditions\n", bcName); CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "ASCII\n"); CHKERRQ(ierr);
  // get coordinates of nodes 
  ierr = DMDAGetCoordinates(da, &coords); CHKERRQ(ierr);
  if (!coords) {
    ierr = DMDASetUniformCoordinates(da, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0); CHKERRQ(ierr);
    ierr = DMDAGetCoordinates(da, &coords); CHKERRQ(ierr);
  }
  ierr = PetscViewerASCIIPrintf(viewer, "DATASET RECTILINEAR_GRID\n"); CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "DIMENSIONS %d %d %d\n", M, N, 1); CHKERRQ(ierr);
  ierr = VecGetArray(coords, &array);CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "X_COORDINATES %d double\n", M); CHKERRQ(ierr);
  for(i = 0; i < M; i++) {
    ierr = PetscViewerASCIIPrintf(viewer, "%G ", PetscRealPart(array[i*2])); CHKERRQ(ierr);
  }
  ierr = PetscViewerASCIIPrintf(viewer, "\n"); CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "Y_COORDINATES %d double\n", N); CHKERRQ(ierr);
  for(i = 0; i < N; i++) {
    ierr = PetscViewerASCIIPrintf(viewer, "%G ", PetscRealPart(array[i*M*2+1])); CHKERRQ(ierr);
  }
  ierr = PetscViewerASCIIPrintf(viewer, "\n"); CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "Z_COORDINATES %d double\n", 1); CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "%G\n", 0.0); CHKERRQ(ierr);
  ierr = VecRestoreArray(coords, &array); CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "POINT_DATA %d\n", N); CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "SCALARS scalars double %d\n", 1); CHKERRQ(ierr);
  ierr = PetscViewerASCIIPrintf(viewer, "LOOKUP_TABLE default\n"); CHKERRQ(ierr);
  ierr = VecGetArray(x, &array); CHKERRQ(ierr);

  // Determine maximum message to arrive:
  ierr = MPI_Comm_rank(comm, &rank); CHKERRQ(ierr);
  ierr = MPI_Comm_size(comm, &size) ;CHKERRQ(ierr);
  ierr = MPI_Reduce(&nn, &maxn, 1, MPIU_INT, MPI_MAX, 0, comm); CHKERRQ(ierr);
  tag  = ((PetscObject) viewer)->tag;
  if (!rank) {
    ierr = PetscMalloc((maxn+1) * sizeof(PetscScalar), &values); CHKERRQ(ierr);
    for(i = 0; i < nn; i++) {
      ierr = PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i])); CHKERRQ(ierr);
    }
    for(p = 1; p < size; p++) {
      ierr = MPI_Recv(values, (PetscMPIInt) nn, MPIU_SCALAR, p, tag, comm, &status); CHKERRQ(ierr);
      ierr = MPI_Get_count(&status, MPIU_SCALAR, &nn); CHKERRQ(ierr);        
      for(i = 0; i < nn; i++) {
        ierr = PetscViewerASCIIPrintf(viewer, "%G\n", PetscRealPart(array[i])); CHKERRQ(ierr);
      }
    }
    ierr = PetscFree(values); CHKERRQ(ierr);
  } else {
    ierr = MPI_Send(array, nn, MPIU_SCALAR, 0, tag, comm); CHKERRQ(ierr);
  }
  ierr = VecRestoreArray(x, &array); CHKERRQ(ierr);
  ierr = PetscViewerFlush(viewer); CHKERRQ(ierr);
  ierr = PetscViewerDestroy(&viewer); CHKERRQ(ierr);
      
  PetscFunctionReturn(0);
  }





From cjm2176 at columbia.edu  Thu May 17 14:23:35 2012
From: cjm2176 at columbia.edu (Colin McAuliffe)
Date: Thu, 17 May 2012 15:23:35 -0400
Subject: [petsc-users] KSPSetDiagonalScale
Message-ID: <20120517152335.td44o7w54wkgok8g@cubmail.cc.columbia.edu>

Hello!

I am solving a coupled multiphysics problem which has several  
different units, which leads to a poorly conditioned jacobian. I am  
using the following ksp options:

-ksp_type preonly -pc_type lu -pc_factor_mat_solver_package umfpack

The results I am getting are not right and I suspect that the  
conditioning of the jacobian may be playing a part in that. I would  
like to improve the conditioning by using -ksp_diagonal_scale, but  
adding it does not seem to help. The documentation for this option  
says 'this routine is only used if the matrix and preconditioner  
matrix are the same thing'.

Does that mean the command can only be used with no preconditioner?

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetDiagonalScale.html

All the best,

-- 
Colin McAuliffe
PhD Candidate
Columbia University
Department of Civil Engineering and Engineering Mechanics

From five9a2 at gmail.com  Thu May 17 14:32:49 2012
From: five9a2 at gmail.com (Jed Brown)
Date: Thu, 17 May 2012 14:32:49 -0500
Subject: [petsc-users] KSPSetDiagonalScale
In-Reply-To: <20120517152335.td44o7w54wkgok8g@cubmail.cc.columbia.edu>
References: <20120517152335.td44o7w54wkgok8g@cubmail.cc.columbia.edu>
Message-ID: <CAM9tzSkY25jf_+5-NRScDi48KedStgrZZWjvbd6yyT1VWYE1kQ@mail.gmail.com>

It just means that both matrix arguments A and B are the same.

What happens if you use this direct solver as a preconditioner?
On May 17, 2012 2:23 PM, "Colin McAuliffe" <cjm2176 at columbia.edu> wrote:

> Hello!
>
> I am solving a coupled multiphysics problem which has several different
> units, which leads to a poorly conditioned jacobian. I am using the
> following ksp options:
>
> -ksp_type preonly -pc_type lu -pc_factor_mat_solver_package umfpack
>
> The results I am getting are not right and I suspect that the conditioning
> of the jacobian may be playing a part in that. I would like to improve the
> conditioning by using -ksp_diagonal_scale, but adding it does not seem to
> help. The documentation for this option says 'this routine is only used if
> the matrix and preconditioner matrix are the same thing'.
>
> Does that mean the command can only be used with no preconditioner?
>
> http://www.mcs.anl.gov/petsc/**petsc-current/docs/**manualpages/KSP/**
> KSPSetDiagonalScale.html<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetDiagonalScale.html>
>
> All the best,
>
> --
> Colin McAuliffe
> PhD Candidate
> Columbia University
> Department of Civil Engineering and Engineering Mechanics
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120517/44d014a6/attachment-0001.htm>

From cjm2176 at columbia.edu  Thu May 17 14:50:27 2012
From: cjm2176 at columbia.edu (Colin McAuliffe)
Date: Thu, 17 May 2012 15:50:27 -0400
Subject: [petsc-users] KSPSetDiagonalScale
In-Reply-To: <CAM9tzSkY25jf_+5-NRScDi48KedStgrZZWjvbd6yyT1VWYE1kQ@mail.gmail.com>
References: <20120517152335.td44o7w54wkgok8g@cubmail.cc.columbia.edu>
	<CAM9tzSkY25jf_+5-NRScDi48KedStgrZZWjvbd6yyT1VWYE1kQ@mail.gmail.com>
Message-ID: <20120517155027.dzwy513g0okoksk4@cubmail.cc.columbia.edu>

Using the direct solver as a preconditioner I can't seem to find any  
difference between the results with -ksp_diagonal_scale vs the results  
without this option.  I also cant tell if the diagonal scaling is  
actually being applied in this case.


Quoting Jed Brown <five9a2 at gmail.com>:

> It just means that both matrix arguments A and B are the same.
>
> What happens if you use this direct solver as a preconditioner?
> On May 17, 2012 2:23 PM, "Colin McAuliffe" <cjm2176 at columbia.edu> wrote:
>
>> Hello!
>>
>> I am solving a coupled multiphysics problem which has several different
>> units, which leads to a poorly conditioned jacobian. I am using the
>> following ksp options:
>>
>> -ksp_type preonly -pc_type lu -pc_factor_mat_solver_package umfpack
>>
>> The results I am getting are not right and I suspect that the conditioning
>> of the jacobian may be playing a part in that. I would like to improve the
>> conditioning by using -ksp_diagonal_scale, but adding it does not seem to
>> help. The documentation for this option says 'this routine is only used if
>> the matrix and preconditioner matrix are the same thing'.
>>
>> Does that mean the command can only be used with no preconditioner?
>>
>> http://www.mcs.anl.gov/petsc/**petsc-current/docs/**manualpages/KSP/**
>> KSPSetDiagonalScale.html<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetDiagonalScale.html>
>>
>> All the best,
>>
>> --
>> Colin McAuliffe
>> PhD Candidate
>> Columbia University
>> Department of Civil Engineering and Engineering Mechanics
>>
>



-- 
Colin McAuliffe
PhD Candidate
Columbia University
Department of Civil Engineering and Engineering Mechanics

From knepley at gmail.com  Thu May 17 15:11:42 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 May 2012 16:11:42 -0400
Subject: [petsc-users] Solve the Poisson equation twice with
	/src/ksp/ksp/examples/tutorials/ex45.c
In-Reply-To: <4FB546E8.10400@gmail.com>
References: <4FB539B1.3050101@gmail.com>
	<CAMYG4Gm8bUCMJorLo8k-G_mfw6wvR5YO1NwZ78twxUY-xDbn=Q@mail.gmail.com>
	<4FB546E8.10400@gmail.com>
Message-ID: <CAMYG4G=xqeYD-_45QHJ1q_MdtE42Kd23_t6uhMmLochq2rKLmg@mail.gmail.com>

On Thu, May 17, 2012 at 2:43 PM, Zhenglun (Alan) Wei <zhenglun.wei at gmail.com
> wrote:

>  On 5/17/2012 1:11 PM, Matthew Knepley wrote:
>
> On Thu, May 17, 2012 at 1:47 PM, Zhenglun (Alan) Wei <
> zhenglun.wei at gmail.com> wrote:
>
>> Dear All,
>>    I hope you're having a nice day.
>>    I was trying to solve the Poisson equation with
>> /src/ksp/ksp/exmaples/tutorial/ex45.c; however, it always give me the same
>> result. I recalled the PetscLogStages and resolved this problem. However,
>> the process of reconstructing DM costs a lot of computational time. I
>> wonder do I have to use PetscLogStages to solve this problem, since, I
>> thought, this function is only needed for using DMMG. Or as long as DM
>> twice in one program, then I need to call PetscLogStages.
>>
>
>  I cannot understand what you mean. PetscLog functions only affect
> logging.
>
>     Matt
>
>
>> thanks,
>> Alan
>>
>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> Dear Dr. Knepley,
>     I'm sorry for confusing you. Here I attached a old version of
> /src/ksp/ksp/examples/tutorials/ex50.c. By adopting its approach of
> PetscLogStage, I can solve Poisson equation twice with two different RHS in
> one program. However, as you may noticed, this method, in ex50.c, is used
> for DMMG. What I understand is that the DMMG has a severe limitation so
> that the PetscLogStage is needed in order to solve the equation twice with
> two different RHS. I wonder if I have to use this approach for KSP? This
> method requires a lot of time to KSPCreate and DMMGCreate3d/2d in order to
> construct the DM. That's why I want to find an alternative way.
>

No, PetscLogStage has nothing to do with solving with different RHS. It is
only there so you can separately time the solves.

   Matt


> thanks,
> Alan
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120517/da4b7175/attachment.htm>

From andrew.spott at gmail.com  Thu May 17 17:36:00 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Thu, 17 May 2012 16:36:00 -0600
Subject: [petsc-users] explicit symmetric matrices
Message-ID: <89B9DF8C-1E80-4E0D-A20B-39D1D1067F67@gmail.com>

Is there a way to explicitly say that a sparse matrix is symmetric or hermitian and only save/load half of it?

-Andrew

From knepley at gmail.com  Thu May 17 17:44:04 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 17 May 2012 18:44:04 -0400
Subject: [petsc-users] explicit symmetric matrices
In-Reply-To: <89B9DF8C-1E80-4E0D-A20B-39D1D1067F67@gmail.com>
References: <89B9DF8C-1E80-4E0D-A20B-39D1D1067F67@gmail.com>
Message-ID: <CAMYG4Gkjt0P6duy9KZN9AXKgMpzOaXydo_9wQvK0AKve4_LdqQ@mail.gmail.com>

On Thu, May 17, 2012 at 6:36 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> Is there a way to explicitly say that a sparse matrix is symmetric or
> hermitian and only save/load half of it?


PETSc distinguishes two kinds of information. Information about the
operator (I am symmetric) can be set with
options. This can be queried by other code in order to optimize algorithms.
It can also be checked by functions
like MatIsSymmetric().

Information about an implementation is in the type, like MATSBAIJ which
says I only store half my values. There
is no need for a symmetric matrix to be stored this way, but you can choose
this if you want.

   Matt


>
> -Andrew

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120517/d8aa013c/attachment.htm>

From dominik at itis.ethz.ch  Fri May 18 07:48:17 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Fri, 18 May 2012 14:48:17 +0200
Subject: [petsc-users] vfork problems compiling on Windows
Message-ID: <CALm7oto6__NabvTzFkbnHt-GFAGM0J=y+M-+f1Gyc1y=yutNoA@mail.gmail.com>

Hi,

I already reported this problem before. I get vfork problem on Windows
and one or more subfolders break during compilation resulting in an
incomplete target library.
I was advised before to go to the failed subfolder and run make in
there. That works, but now the question is how do I re-make the final
library and only the library. If I go back to the petssc root folder
and issue "make" the whole thing gets rebuilt from scratch, ignoring
what has been so far built. This not only takes ages on Windows, but
also occasionally breaks due to the vfork problem in another
subfolder. How can I trigger continuing the build in this case?

Thanks
Dominik

From balay at mcs.anl.gov  Fri May 18 08:06:39 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 18 May 2012 08:06:39 -0500 (CDT)
Subject: [petsc-users] vfork problems compiling on Windows
In-Reply-To: <CALm7oto6__NabvTzFkbnHt-GFAGM0J=y+M-+f1Gyc1y=yutNoA@mail.gmail.com>
References: <CALm7oto6__NabvTzFkbnHt-GFAGM0J=y+M-+f1Gyc1y=yutNoA@mail.gmail.com>
Message-ID: <alpine.LFD.2.02.1205180802380.1999@asterix>

On Fri, 18 May 2012, Dominik Szczerba wrote:

> Hi,
> 
> I already reported this problem before. I get vfork problem on Windows
> and one or more subfolders break during compilation resulting in an
> incomplete target library.
> I was advised before to go to the failed subfolder and run make in
> there. That works, but now the question is how do I re-make the final
> library and only the library.

Running 'make' in the subfolder' where you got 'vfork' errors should
update the library with the missing pieces - so you shouldn't have to
do any more 're-make final library' stuff.

> If I go back to the petssc root folder
> and issue "make" the whole thing gets rebuilt from scratch, ignoring
> what has been so far built. This not only takes ages on Windows, but
> also occasionally breaks due to the vfork problem in another
> subfolder. How can I trigger continuing the build in this case?

Alternative way of achieving incremental build is the following
command in the top-level dir. [But this can also take a while to
complete]

make ACTION=lib tree

Satish

From balay at mcs.anl.gov  Fri May 18 10:38:49 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 18 May 2012 10:38:49 -0500 (CDT)
Subject: [petsc-users] vfork problems compiling on Windows
In-Reply-To: <CALm7oto6__NabvTzFkbnHt-GFAGM0J=y+M-+f1Gyc1y=yutNoA@mail.gmail.com>
References: <CALm7oto6__NabvTzFkbnHt-GFAGM0J=y+M-+f1Gyc1y=yutNoA@mail.gmail.com>
Message-ID: <alpine.LFD.2.02.1205181036250.1999@asterix>

On Fri, 18 May 2012, Dominik Szczerba wrote:

> I already reported this problem before. I get vfork problem on Windows

BTW: do you have an antivirus or some type of scanner software thats
running? Would disabling such a thing make the "vfork: Resource
temporarily unavailable" problem go away?

Satish

From bmhautz at sandia.gov  Fri May 18 10:41:32 2012
From: bmhautz at sandia.gov (Hautzenroeder, Brenna M)
Date: Fri, 18 May 2012 15:41:32 +0000
Subject: [petsc-users] [EXTERNAL] Re: Problem creating an array of KSP
 objects
In-Reply-To: <CAMYG4GnOJ+-yXR0eDqL=+AVGhubLKKbiSK-Kp0PnNMV8JjZhgw@mail.gmail.com>
References: <AC73979A4C206F458238893BE877E7020E0870A3@EXMB01.srn.sandia.gov>
	<CAMYG4GnOJ+-yXR0eDqL=+AVGhubLKKbiSK-Kp0PnNMV8JjZhgw@mail.gmail.com>
Message-ID: <AC73979A4C206F458238893BE877E7020E0873BB@EXMB01.srn.sandia.gov>

Matt,

Thanks for replying back to me. Actually, we are only seeing this problem on debug, non-fortran compilations of PETSc. When we turn on fortran and turn off debugging in the configuration, the behavior is normal, the timing is fast (as to be expected), and the storage space is more in lines with our calculations. Maybe this is expected behavior for all debug/non-fortran builds? Or maybe this is a test case for finding an edge-case?

Anyways, thanks for your help!

-Brenna

From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Matthew Knepley
Sent: Thursday, May 17, 2012 10:59 AM
To: PETSc users list
Subject: [EXTERNAL] Re: [petsc-users] Problem creating an array of KSP objects

On Thu, May 17, 2012 at 12:37 PM, Hautzenroeder, Brenna M <bmhautz at sandia.gov<mailto:bmhautz at sandia.gov>> wrote:
I'm running into time and memory problems creating an array to store multiple KSP objects.

We are using a Fourier method for the temporal portion of our operator; currently, this is anywhere between 1024-2048 time samples equating to 1024-2048 frequencies to advance our solution. Each frequency has its own matrix, meaning there are 1024-2048 linear equations of the form Ax = b. So, while the matrices are different for each frequency, when we propagate the solution one step, the 1024-2048 matrices don't change since the frequencies haven't changed between steps.

We would like to store each KSP solution (all 1024-2048 of them) in an array so that we can reuse them for the next step since this saves us time factoring the matrix. The problem I am seeing is that storing these KSP objects in any kind of array increases the runtime and storage space exponentially - I am seeing times on the order of 30 mins to an hour and memory consumed hover around 20-30GB! Is there something inherent

I don't think you really mean "exponential", but just big. Did you calculate how much space you expect? Storing a factor can take
N^3 space, and then you have 1000 of them, so it will take 1000x the space of 1 KSP. Did you compare the storage for both cases?
There is really not enough information in your question, however

   Yes, storing KSPs is the right thing here

   No, they do not take more space than necessary

       Matt

about the KSP object that I am missing and/or a better way to store these objects? I don't have any problems if I use one KSP object and set the operators each time to the different matrices, except that it takes longer than it normally would after the first step since it has to recalculate the preconditioner matrix.

Note: We are using distributed matrices and vectors for our linear system.

I have included some pseudocode to illustrate our linear system, which works:

KSP k;
for steps = 1 to 100 do:
for time = 1 to 1024 do:
                create distributed matrix A and fill with local values (A[step=1,time=1] = A[step=2,time=1], etc)
                create distributed vector b and fill with local values for given time
                create distributed vector x for result
                create KSP k and set matrices
                KSPSolve(k, b, x) => takes a little extra time each step

However, the following creates time/storage problems:

KSP* kArray;
create kArray => takes inordinate amout of time and storage
for steps = 1 to 100 do:
        for time=1 to 1024 do:
                create distributed matrix A and fill with local values (A[step=1,time=1] = A[step=2,time=1], etc)
                create distributed vector b and fill with local values for given time
                create distributed vector x for result
                if time == 1 do:
set matrices on kArray[time] => also takes an inordinate amount of time and storage
                KSPSolve(kArray[time], b, x)

Any insight would be greatly appreciated!
Brenna




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120518/b1111844/attachment-0001.htm>

From knepley at gmail.com  Fri May 18 10:52:46 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 18 May 2012 11:52:46 -0400
Subject: [petsc-users] [EXTERNAL] Re: Problem creating an array of KSP
	objects
In-Reply-To: <AC73979A4C206F458238893BE877E7020E0873BB@EXMB01.srn.sandia.gov>
References: <AC73979A4C206F458238893BE877E7020E0870A3@EXMB01.srn.sandia.gov>
	<CAMYG4GnOJ+-yXR0eDqL=+AVGhubLKKbiSK-Kp0PnNMV8JjZhgw@mail.gmail.com>
	<AC73979A4C206F458238893BE877E7020E0873BB@EXMB01.srn.sandia.gov>
Message-ID: <CAMYG4GkOhK3TCFgFzpDRr=NkL0k7HRMvhZF0ukEAFii-WRb4rg@mail.gmail.com>

On Fri, May 18, 2012 at 11:41 AM, Hautzenroeder, Brenna M <
bmhautz at sandia.gov> wrote:

>  Matt,****
>
> ** **
>
> Thanks for replying back to me. Actually, we are only seeing this problem
> on debug, non-fortran compilations of PETSc. When we turn on fortran and
> turn off debugging in the configuration, the behavior is normal, the timing
> is fast (as to be expected), and the storage space is more in lines with
> our calculations. Maybe this is expected behavior for all debug/non-fortran
> builds? Or maybe this is a test case for finding an edge-case?
>

No. KSP is most used part of PETSc, and has been there for > 15 years. We
run nightly valgrind tests to check for leaks.
I am confident that this is misconfiguration. Send a sample code to
petsc-maint at mcs.anl.gov if -malloc_dump does
not find the problem.

    Matt


>
>
> Anyways, thanks for your help!****
>
> ** **
>
> -Brenna****
>
> ** **
>
> *From:* petsc-users-bounces at mcs.anl.gov [mailto:
> petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley
> *Sent:* Thursday, May 17, 2012 10:59 AM
> *To:* PETSc users list
> *Subject:* [EXTERNAL] Re: [petsc-users] Problem creating an array of KSP
> objects****
>
> ** **
>
> On Thu, May 17, 2012 at 12:37 PM, Hautzenroeder, Brenna M <
> bmhautz at sandia.gov> wrote:****
>
>  I?m running into time and memory problems creating an array to store
> multiple KSP objects.****
>
>  ****
>
> We are using a Fourier method for the temporal portion of our operator;
> currently, this is anywhere between 1024-2048 time samples equating to
> 1024-2048 frequencies to advance our solution. Each frequency has its own
> matrix, meaning there are 1024-2048 linear equations of the form Ax = b.
> So, while the matrices are different for each frequency, when we propagate
> the solution one step, the 1024-2048 matrices don?t change since the
> frequencies haven?t changed between steps.****
>
>  ****
>
> We would like to store each KSP solution (all 1024-2048 of them) in an
> array so that we can reuse them for the next step since this saves us time
> factoring the matrix. The problem I am seeing is that storing these KSP
> objects in any kind of array increases the runtime and storage space
> exponentially ? I am seeing times on the order of 30 mins to an hour and
> memory consumed hover around 20-30GB! Is there something inherent ****
>
>  ** **
>
> I don't think you really mean "exponential", but just big. Did you
> calculate how much space you expect? Storing a factor can take****
>
> N^3 space, and then you have 1000 of them, so it will take 1000x the space
> of 1 KSP. Did you compare the storage for both cases?****
>
> There is really not enough information in your question, however****
>
> ** **
>
>    Yes, storing KSPs is the right thing here****
>
> ** **
>
>    No, they do not take more space than necessary****
>
> ** **
>
>        Matt****
>
>  ****
>
>  about the KSP object that I am missing and/or a better way to store
> these objects? I don?t have any problems if I use one KSP object and set
> the operators each time to the different matrices, except that it takes
> longer than it normally would after the first step since it has to
> recalculate the preconditioner matrix.****
>
>  ****
>
> Note: We are using distributed matrices and vectors for our linear system.
> ****
>
>  ****
>
> I have included some pseudocode to illustrate our linear system, which
> works:****
>
>  ****
>
> KSP k;****
>
> for steps = 1 to 100 do:****
>
> for time = 1 to 1024 do:****
>
>                 create distributed matrix A and fill with local values
> (A[step=1,time=1] = A[step=2,time=1], etc)****
>
>                 create distributed vector b and fill with local values for
> given time****
>
>                 create distributed vector x for result****
>
>                 create KSP k and set matrices****
>
>                 KSPSolve(k, b, x) => takes a little extra time each step**
> **
>
>  ****
>
> However, the following creates time/storage problems:****
>
>  ****
>
> KSP* kArray;****
>
> create kArray => takes inordinate amout of time and storage****
>
> for steps = 1 to 100 do:****
>
>         for time=1 to 1024 do:****
>
>                 create distributed matrix A and fill with local values
> (A[step=1,time=1] = A[step=2,time=1], etc)****
>
>                 create distributed vector b and fill with local values for
> given time****
>
>                 create distributed vector x for result****
>
>                 if time == 1 do:****
>
> set matrices on kArray[time] => also takes an inordinate amount of time
> and storage****
>
>                 KSPSolve(kArray[time], b, x)****
>
>  ****
>
> Any insight would be greatly appreciated!****
>
> Brenna****
>
>  ****
>
>
>
> ****
>
> ** **
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener****
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120518/fa375425/attachment.htm>

From andrew.spott at gmail.com  Fri May 18 15:26:37 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Fri, 18 May 2012 14:26:37 -0600
Subject: [petsc-users] nonzero structure setting and remembering
Message-ID: <8AB4A662-FCC1-455F-962A-8ABD168D42A8@gmail.com>

I have a couple of questions about how to deal with the nonzero structure.

0)  Is the "nonzero structure" the same as "number of non zeros per row"? Or are they different?

1)  How much of a speedup do you get for copying/creating/loading matrices when you set the number of non zeros per row beforehand? (if the nonzero structure is different from the number of non zeros per row, what about the nonzero structure)

2)  If I have a matrix, how do I add to the nonzero structure, without actually adding nonzero values, do I just add "0.0"? (the matrix will have values there, but doesn't at the moment)

As usual, thanks for all the help

-Andrew

From jedbrown at mcs.anl.gov  Fri May 18 15:31:45 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 18 May 2012 15:31:45 -0500
Subject: [petsc-users] nonzero structure setting and remembering
In-Reply-To: <8AB4A662-FCC1-455F-962A-8ABD168D42A8@gmail.com>
References: <8AB4A662-FCC1-455F-962A-8ABD168D42A8@gmail.com>
Message-ID: <CAM9tzSmu+h4sHxrrZNrdUuU79cA6-By0TmJxm9gNsUBNcQ2N4Q@mail.gmail.com>

On Fri, May 18, 2012 at 3:26 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> I have a couple of questions about how to deal with the nonzero structure.
>
> 0)  Is the "nonzero structure" the same as "number of non zeros per row"?
> Or are they different?
>

Yes, for parallel *AIJ matrices, it's the number of nonzeros in the
diagonal part and in the off-diagonal part. Preallocation can optionally
also set the column indices, but it's not critical.


>
> 1)  How much of a speedup do you get for copying/creating/loading matrices
> when you set the number of non zeros per row beforehand? (if the nonzero
> structure is different from the number of non zeros per row, what about the
> nonzero structure)
>

a million


>
> 2)  If I have a matrix, how do I add to the nonzero structure, without
> actually adding nonzero values, do I just add "0.0"? (the matrix will have
> values there, but doesn't at the moment)
>

Mat*AIJSetPreallocation(), then reassemble.


>
> As usual, thanks for all the help
>
> -Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120518/cfbfa87d/attachment.htm>

From dominik at itis.ethz.ch  Fri May 18 16:49:47 2012
From: dominik at itis.ethz.ch (Dominik Szczerba)
Date: Fri, 18 May 2012 23:49:47 +0200
Subject: [petsc-users] vfork problems compiling on Windows
In-Reply-To: <alpine.LFD.2.02.1205180802380.1999@asterix>
References: <CALm7oto6__NabvTzFkbnHt-GFAGM0J=y+M-+f1Gyc1y=yutNoA@mail.gmail.com>
	<alpine.LFD.2.02.1205180802380.1999@asterix>
Message-ID: <CALm7otpY50K_wFLyJco6vibpj=zcUrMQ53CYOs4RXhuAj1FPYQ@mail.gmail.com>

> Running 'make' in the subfolder' where you got 'vfork' errors should
> update the library with the missing pieces - so you shouldn't have to
> do any more 're-make final library' stuff.

After "make" in the concerned subfolder I came back to the root folder
and issued "make test". But errors were reported during the tests so I
thought the library had not been finalized.

> Alternative way of achieving incremental build is the following
> command in the top-level dir. [But this can also take a while to
> complete]
>
> make ACTION=lib tree

> BTW: do you have an antivirus or some type of scanner software thats
> running? Would disabling such a thing make the "vfork: Resource
> temporarily unavailable" problem go away?

Thanks for the hints. Yes I have a virus scanner and I believe I once
disabled it for a test and it did not help, but am not absolutely
certain any more. Will try next time the problem occurs.

Regards,
Dominik

From mirzadeh at gmail.com  Fri May 18 18:06:29 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Fri, 18 May 2012 16:06:29 -0700
Subject: [petsc-users] petsc example with known scalability
Message-ID: <CAKsPKwmT4S0HrrB+OtA4Lt2zyEPxq0q98JZBisQCB0bg7hF=Dw@mail.gmail.com>

Hi guys,

I'm trying to generate scalability plots for my code and do profiling and
fine tuning. In doing so I have noticed that some of the factors affecting
my results are sort of subtle. For example, I figured, the other day, that
using all of the cores on a single node is somewhat (50-60%) slower when
compared to using only half of the cores which I suspect is due to memory
bandwidth and/or other hardware-related issues.

So I thought to ask and see if there is any example in petsc that has been
tested for scalability and has been documented? Basically I want to use
this test example as a benchmark to compare my results with. My own test
code is currently a linear Poisson solver on an adaptive quadtree grid and
involves non-trivial geometry (well basically a circle for the boundary but
still not a simple box).

Thanks,
Mohammad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120518/1abcfb44/attachment.htm>

From knepley at gmail.com  Fri May 18 18:18:44 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 18 May 2012 19:18:44 -0400
Subject: [petsc-users] petsc example with known scalability
In-Reply-To: <CAKsPKwmT4S0HrrB+OtA4Lt2zyEPxq0q98JZBisQCB0bg7hF=Dw@mail.gmail.com>
References: <CAKsPKwmT4S0HrrB+OtA4Lt2zyEPxq0q98JZBisQCB0bg7hF=Dw@mail.gmail.com>
Message-ID: <CAMYG4Gm5ih-gGyijh_rdCKXUufL67NUms4PXAtY+ELonNCrfnQ@mail.gmail.com>

On Fri, May 18, 2012 at 7:06 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Hi guys,
>
> I'm trying to generate scalability plots for my code and do profiling and
> fine tuning. In doing so I have noticed that some of the factors affecting
> my results are sort of subtle. For example, I figured, the other day, that
> using all of the cores on a single node is somewhat (50-60%) slower when
> compared to using only half of the cores which I suspect is due to memory
> bandwidth and/or other hardware-related issues.
>
> So I thought to ask and see if there is any example in petsc that has been
> tested for scalability and has been documented? Basically I want to use
> this test example as a benchmark to compare my results with. My own test
> code is currently a linear Poisson solver on an adaptive quadtree grid and
> involves non-trivial geometry (well basically a circle for the boundary but
> still not a simple box).
>

Unfortunately, I do not even know what that means. We can't guarantee a
certain level of performance because it not
only depends on the hardware, but how you use it (as evident in your case).
In a perfect world, we would have an abstract
model of the computation (available for MatMult) and your machine (not
available anywhere) and we would automatically
work out the consequences and tell you what to expect. Instead today, we
tell you to look at a few key indicators like the
MatMult event, to see what is going on. When MatMult stops scaling, you
have run out of bandwidth.

    Matt


> Thanks,
> Mohammad
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120518/45fe417d/attachment.htm>

From mirzadeh at gmail.com  Fri May 18 18:43:23 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Fri, 18 May 2012 16:43:23 -0700
Subject: [petsc-users] petsc example with known scalability
In-Reply-To: <CAMYG4Gm5ih-gGyijh_rdCKXUufL67NUms4PXAtY+ELonNCrfnQ@mail.gmail.com>
References: <CAKsPKwmT4S0HrrB+OtA4Lt2zyEPxq0q98JZBisQCB0bg7hF=Dw@mail.gmail.com>
	<CAMYG4Gm5ih-gGyijh_rdCKXUufL67NUms4PXAtY+ELonNCrfnQ@mail.gmail.com>
Message-ID: <CAKsPKwntWYUUh0cE3PFkMyMKbASiPPwXKm-pvKcKJ=CqTUnqaQ@mail.gmail.com>

I see; well that's a fair point. So i have my timing results obtained via
-log_summary; what should I be looking into for MatMult? Should I be
looking at wall timings? Or do I need to look into MFlops/s? I'm sorry but
I'm not sure what measure I should be looking into to determine scalability.

Also, is there any general meaningful advice one could give? in terms of
using the resources, compiler flags (beyond -O3), etc?

Thanks,
Mohammad

On Fri, May 18, 2012 at 4:18 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, May 18, 2012 at 7:06 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> Hi guys,
>>
>> I'm trying to generate scalability plots for my code and do profiling and
>> fine tuning. In doing so I have noticed that some of the factors affecting
>> my results are sort of subtle. For example, I figured, the other day, that
>> using all of the cores on a single node is somewhat (50-60%) slower when
>> compared to using only half of the cores which I suspect is due to memory
>> bandwidth and/or other hardware-related issues.
>>
>> So I thought to ask and see if there is any example in petsc that has
>> been tested for scalability and has been documented? Basically I want to
>> use this test example as a benchmark to compare my results with. My own
>> test code is currently a linear Poisson solver on an adaptive quadtree grid
>> and involves non-trivial geometry (well basically a circle for the boundary
>> but still not a simple box).
>>
>
> Unfortunately, I do not even know what that means. We can't guarantee a
> certain level of performance because it not
> only depends on the hardware, but how you use it (as evident in your
> case). In a perfect world, we would have an abstract
> model of the computation (available for MatMult) and your machine (not
> available anywhere) and we would automatically
> work out the consequences and tell you what to expect. Instead today, we
> tell you to look at a few key indicators like the
> MatMult event, to see what is going on. When MatMult stops scaling, you
> have run out of bandwidth.
>
>     Matt
>
>
>> Thanks,
>> Mohammad
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120518/b1697b93/attachment.htm>

From knepley at gmail.com  Fri May 18 18:47:17 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 18 May 2012 19:47:17 -0400
Subject: [petsc-users] petsc example with known scalability
In-Reply-To: <CAKsPKwntWYUUh0cE3PFkMyMKbASiPPwXKm-pvKcKJ=CqTUnqaQ@mail.gmail.com>
References: <CAKsPKwmT4S0HrrB+OtA4Lt2zyEPxq0q98JZBisQCB0bg7hF=Dw@mail.gmail.com>
	<CAMYG4Gm5ih-gGyijh_rdCKXUufL67NUms4PXAtY+ELonNCrfnQ@mail.gmail.com>
	<CAKsPKwntWYUUh0cE3PFkMyMKbASiPPwXKm-pvKcKJ=CqTUnqaQ@mail.gmail.com>
Message-ID: <CAMYG4GnvCosrEDVu0YaPGPaWKL-eGhpwBdVjZv0ifGuLkNDkcQ@mail.gmail.com>

On Fri, May 18, 2012 at 7:43 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> I see; well that's a fair point. So i have my timing results obtained via
> -log_summary; what should I be looking into for MatMult? Should I be
> looking at wall timings? Or do I need to look into MFlops/s? I'm sorry but
> I'm not sure what measure I should be looking into to determine scalability.
>

Time is only meaningful in isolation if I know how big your matrix is, but
you obviously take the ratio to look how it is scaling. I am
assuming you are looking at weak scalability so it should remain constant.
MF/s will let you know how the routine is performing
independent of size, and thus is an easy way to see what is happening. It
should scale like P, and when that drops off you have
insufficient bandwidth. VecMDot is a good way to look at the latency of
reductions (assuming you use GMRES). There is indeed no
good guide to this. Barry should write one.

    Matt


> Also, is there any general meaningful advice one could give? in terms of
> using the resources, compiler flags (beyond -O3), etc?
>
> Thanks,
> Mohammad
>
> On Fri, May 18, 2012 at 4:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Fri, May 18, 2012 at 7:06 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>
>>> Hi guys,
>>>
>>> I'm trying to generate scalability plots for my code and do profiling
>>> and fine tuning. In doing so I have noticed that some of the factors
>>> affecting my results are sort of subtle. For example, I figured, the other
>>> day, that using all of the cores on a single node is somewhat (50-60%)
>>> slower when compared to using only half of the cores which I suspect is due
>>> to memory bandwidth and/or other hardware-related issues.
>>>
>>> So I thought to ask and see if there is any example in petsc that has
>>> been tested for scalability and has been documented? Basically I want to
>>> use this test example as a benchmark to compare my results with. My own
>>> test code is currently a linear Poisson solver on an adaptive quadtree grid
>>> and involves non-trivial geometry (well basically a circle for the boundary
>>> but still not a simple box).
>>>
>>
>> Unfortunately, I do not even know what that means. We can't guarantee a
>> certain level of performance because it not
>> only depends on the hardware, but how you use it (as evident in your
>> case). In a perfect world, we would have an abstract
>> model of the computation (available for MatMult) and your machine (not
>> available anywhere) and we would automatically
>> work out the consequences and tell you what to expect. Instead today, we
>> tell you to look at a few key indicators like the
>> MatMult event, to see what is going on. When MatMult stops scaling, you
>> have run out of bandwidth.
>>
>>     Matt
>>
>>
>>> Thanks,
>>> Mohammad
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120518/0948b133/attachment-0001.htm>

From mirzadeh at gmail.com  Fri May 18 19:02:14 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Fri, 18 May 2012 17:02:14 -0700
Subject: [petsc-users] petsc example with known scalability
In-Reply-To: <CAMYG4GnvCosrEDVu0YaPGPaWKL-eGhpwBdVjZv0ifGuLkNDkcQ@mail.gmail.com>
References: <CAKsPKwmT4S0HrrB+OtA4Lt2zyEPxq0q98JZBisQCB0bg7hF=Dw@mail.gmail.com>
	<CAMYG4Gm5ih-gGyijh_rdCKXUufL67NUms4PXAtY+ELonNCrfnQ@mail.gmail.com>
	<CAKsPKwntWYUUh0cE3PFkMyMKbASiPPwXKm-pvKcKJ=CqTUnqaQ@mail.gmail.com>
	<CAMYG4GnvCosrEDVu0YaPGPaWKL-eGhpwBdVjZv0ifGuLkNDkcQ@mail.gmail.com>
Message-ID: <CAKsPKw=ZXUbLusGvwjU3=Zeo340ThwdG8+Yjcyf=Xe3xhj4Q8g@mail.gmail.com>

Yes, I'm looking at weak scalability right now. I'm using BiCGSTAB with
BoomerAMG (all default options except for rtol = 1e-12). I've not looked
into MF/s yet but I'll surely do to see if I'm having any problem there. So
far, just timing the KSPSolve, I get [0.231, 0.238, 0.296, 0.451, 0.599]
seconds/KSP iteration for p=[1, 4, 16, 64, 256] with almost 93K nodes
(matrix-row) per proc. Which is not bad I guess but still increased by a
factor of 3 for 256 proc. Problem is, I don't know how good/bad this is. In
fact I'm not even sure that is a valid question to ask since it may be very
problem dependent.

Something I just though about, how crucial is the matrix structure for KSP
solvers? The nodes have bad numbering and I do partitioning to get a better
one here.



On Fri, May 18, 2012 at 4:47 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, May 18, 2012 at 7:43 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> I see; well that's a fair point. So i have my timing results obtained via
>> -log_summary; what should I be looking into for MatMult? Should I be
>> looking at wall timings? Or do I need to look into MFlops/s? I'm sorry but
>> I'm not sure what measure I should be looking into to determine scalability.
>>
>
> Time is only meaningful in isolation if I know how big your matrix is, but
> you obviously take the ratio to look how it is scaling. I am
> assuming you are looking at weak scalability so it should remain constant.
> MF/s will let you know how the routine is performing
> independent of size, and thus is an easy way to see what is happening. It
> should scale like P, and when that drops off you have
> insufficient bandwidth. VecMDot is a good way to look at the latency of
> reductions (assuming you use GMRES). There is indeed no
> good guide to this. Barry should write one.
>
>     Matt
>
>
>>  Also, is there any general meaningful advice one could give? in terms of
>> using the resources, compiler flags (beyond -O3), etc?
>>
>> Thanks,
>> Mohammad
>>
>> On Fri, May 18, 2012 at 4:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Fri, May 18, 2012 at 7:06 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I'm trying to generate scalability plots for my code and do profiling
>>>> and fine tuning. In doing so I have noticed that some of the factors
>>>> affecting my results are sort of subtle. For example, I figured, the other
>>>> day, that using all of the cores on a single node is somewhat (50-60%)
>>>> slower when compared to using only half of the cores which I suspect is due
>>>> to memory bandwidth and/or other hardware-related issues.
>>>>
>>>> So I thought to ask and see if there is any example in petsc that has
>>>> been tested for scalability and has been documented? Basically I want to
>>>> use this test example as a benchmark to compare my results with. My own
>>>> test code is currently a linear Poisson solver on an adaptive quadtree grid
>>>> and involves non-trivial geometry (well basically a circle for the boundary
>>>> but still not a simple box).
>>>>
>>>
>>> Unfortunately, I do not even know what that means. We can't guarantee a
>>> certain level of performance because it not
>>> only depends on the hardware, but how you use it (as evident in your
>>> case). In a perfect world, we would have an abstract
>>> model of the computation (available for MatMult) and your machine (not
>>> available anywhere) and we would automatically
>>> work out the consequences and tell you what to expect. Instead today, we
>>> tell you to look at a few key indicators like the
>>> MatMult event, to see what is going on. When MatMult stops scaling, you
>>> have run out of bandwidth.
>>>
>>>     Matt
>>>
>>>
>>>> Thanks,
>>>> Mohammad
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120518/228cab52/attachment.htm>

From knepley at gmail.com  Fri May 18 19:06:33 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 18 May 2012 20:06:33 -0400
Subject: [petsc-users] petsc example with known scalability
In-Reply-To: <CAKsPKw=ZXUbLusGvwjU3=Zeo340ThwdG8+Yjcyf=Xe3xhj4Q8g@mail.gmail.com>
References: <CAKsPKwmT4S0HrrB+OtA4Lt2zyEPxq0q98JZBisQCB0bg7hF=Dw@mail.gmail.com>
	<CAMYG4Gm5ih-gGyijh_rdCKXUufL67NUms4PXAtY+ELonNCrfnQ@mail.gmail.com>
	<CAKsPKwntWYUUh0cE3PFkMyMKbASiPPwXKm-pvKcKJ=CqTUnqaQ@mail.gmail.com>
	<CAMYG4GnvCosrEDVu0YaPGPaWKL-eGhpwBdVjZv0ifGuLkNDkcQ@mail.gmail.com>
	<CAKsPKw=ZXUbLusGvwjU3=Zeo340ThwdG8+Yjcyf=Xe3xhj4Q8g@mail.gmail.com>
Message-ID: <CAMYG4GkKTHRnj3tS_WPNsuaa5JENSo2qbDOdu_DgZFn7WAh-hg@mail.gmail.com>

On Fri, May 18, 2012 at 8:02 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Yes, I'm looking at weak scalability right now. I'm using BiCGSTAB with
> BoomerAMG (all default options except for rtol = 1e-12). I've not looked
> into MF/s yet but I'll surely do to see if I'm having any problem there. So
> far, just timing the KSPSolve, I get [0.231, 0.238, 0.296, 0.451, 0.599]
> seconds/KSP iteration for p=[1, 4, 16, 64, 256] with almost 93K nodes
> (matrix-row) per proc. Which is not bad I guess but still increased by a
> factor of 3 for 256 proc. Problem is, I don't know how good/bad this is. In
> fact I'm not even sure that is a valid question to ask since it may be very
> problem dependent.
>
> Something I just though about, how crucial is the matrix structure for KSP
> solvers? The nodes have bad numbering and I do partitioning to get a better
> one here.
>

Did you look at the number of iterates?

   Matt


>
> On Fri, May 18, 2012 at 4:47 PM, Matthew Knepley <knepley at gmail.com>wrote:
>
>> On Fri, May 18, 2012 at 7:43 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>
>>> I see; well that's a fair point. So i have my timing results obtained
>>> via -log_summary; what should I be looking into for MatMult? Should I be
>>> looking at wall timings? Or do I need to look into MFlops/s? I'm sorry but
>>> I'm not sure what measure I should be looking into to determine scalability.
>>>
>>
>> Time is only meaningful in isolation if I know how big your matrix is,
>> but you obviously take the ratio to look how it is scaling. I am
>> assuming you are looking at weak scalability so it should remain
>> constant. MF/s will let you know how the routine is performing
>> independent of size, and thus is an easy way to see what is happening. It
>> should scale like P, and when that drops off you have
>> insufficient bandwidth. VecMDot is a good way to look at the latency of
>> reductions (assuming you use GMRES). There is indeed no
>> good guide to this. Barry should write one.
>>
>>     Matt
>>
>>
>>>  Also, is there any general meaningful advice one could give? in terms
>>> of using the resources, compiler flags (beyond -O3), etc?
>>>
>>> Thanks,
>>> Mohammad
>>>
>>> On Fri, May 18, 2012 at 4:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>
>>>> On Fri, May 18, 2012 at 7:06 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>>
>>>>> Hi guys,
>>>>>
>>>>> I'm trying to generate scalability plots for my code and do profiling
>>>>> and fine tuning. In doing so I have noticed that some of the factors
>>>>> affecting my results are sort of subtle. For example, I figured, the other
>>>>> day, that using all of the cores on a single node is somewhat (50-60%)
>>>>> slower when compared to using only half of the cores which I suspect is due
>>>>> to memory bandwidth and/or other hardware-related issues.
>>>>>
>>>>> So I thought to ask and see if there is any example in petsc that has
>>>>> been tested for scalability and has been documented? Basically I want to
>>>>> use this test example as a benchmark to compare my results with. My own
>>>>> test code is currently a linear Poisson solver on an adaptive quadtree grid
>>>>> and involves non-trivial geometry (well basically a circle for the boundary
>>>>> but still not a simple box).
>>>>>
>>>>
>>>> Unfortunately, I do not even know what that means. We can't guarantee a
>>>> certain level of performance because it not
>>>> only depends on the hardware, but how you use it (as evident in your
>>>> case). In a perfect world, we would have an abstract
>>>> model of the computation (available for MatMult) and your machine (not
>>>> available anywhere) and we would automatically
>>>> work out the consequences and tell you what to expect. Instead today,
>>>> we tell you to look at a few key indicators like the
>>>> MatMult event, to see what is going on. When MatMult stops scaling, you
>>>> have run out of bandwidth.
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> Thanks,
>>>>> Mohammad
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120518/64764d49/attachment.htm>

From mirzadeh at gmail.com  Fri May 18 19:08:09 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Fri, 18 May 2012 17:08:09 -0700
Subject: [petsc-users] petsc example with known scalability
In-Reply-To: <CAMYG4GkKTHRnj3tS_WPNsuaa5JENSo2qbDOdu_DgZFn7WAh-hg@mail.gmail.com>
References: <CAKsPKwmT4S0HrrB+OtA4Lt2zyEPxq0q98JZBisQCB0bg7hF=Dw@mail.gmail.com>
	<CAMYG4Gm5ih-gGyijh_rdCKXUufL67NUms4PXAtY+ELonNCrfnQ@mail.gmail.com>
	<CAKsPKwntWYUUh0cE3PFkMyMKbASiPPwXKm-pvKcKJ=CqTUnqaQ@mail.gmail.com>
	<CAMYG4GnvCosrEDVu0YaPGPaWKL-eGhpwBdVjZv0ifGuLkNDkcQ@mail.gmail.com>
	<CAKsPKw=ZXUbLusGvwjU3=Zeo340ThwdG8+Yjcyf=Xe3xhj4Q8g@mail.gmail.com>
	<CAMYG4GkKTHRnj3tS_WPNsuaa5JENSo2qbDOdu_DgZFn7WAh-hg@mail.gmail.com>
Message-ID: <CAKsPKwn7aU8SFuJDb7g5FMBNOh=v7OcuZTsXt4QyS9tThNW6Mg@mail.gmail.com>

Yes. Its fairly flat which I think is due to BoomerAMG. [8, 8, 9, 11, 12]

On Fri, May 18, 2012 at 5:06 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, May 18, 2012 at 8:02 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>
>> Yes, I'm looking at weak scalability right now. I'm using BiCGSTAB with
>> BoomerAMG (all default options except for rtol = 1e-12). I've not looked
>> into MF/s yet but I'll surely do to see if I'm having any problem there. So
>> far, just timing the KSPSolve, I get [0.231, 0.238, 0.296, 0.451, 0.599]
>> seconds/KSP iteration for p=[1, 4, 16, 64, 256] with almost 93K nodes
>> (matrix-row) per proc. Which is not bad I guess but still increased by a
>> factor of 3 for 256 proc. Problem is, I don't know how good/bad this is. In
>> fact I'm not even sure that is a valid question to ask since it may be very
>> problem dependent.
>>
>> Something I just though about, how crucial is the matrix structure for
>> KSP solvers? The nodes have bad numbering and I do partitioning to get a
>> better one here.
>>
>
>  Did you look at the number of iterates?
>
>    Matt
>
>
>>
>> On Fri, May 18, 2012 at 4:47 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Fri, May 18, 2012 at 7:43 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:
>>>
>>>> I see; well that's a fair point. So i have my timing results obtained
>>>> via -log_summary; what should I be looking into for MatMult? Should I be
>>>> looking at wall timings? Or do I need to look into MFlops/s? I'm sorry but
>>>> I'm not sure what measure I should be looking into to determine scalability.
>>>>
>>>
>>> Time is only meaningful in isolation if I know how big your matrix is,
>>> but you obviously take the ratio to look how it is scaling. I am
>>> assuming you are looking at weak scalability so it should remain
>>> constant. MF/s will let you know how the routine is performing
>>> independent of size, and thus is an easy way to see what is happening.
>>> It should scale like P, and when that drops off you have
>>> insufficient bandwidth. VecMDot is a good way to look at the latency of
>>> reductions (assuming you use GMRES). There is indeed no
>>> good guide to this. Barry should write one.
>>>
>>>     Matt
>>>
>>>
>>>>  Also, is there any general meaningful advice one could give? in terms
>>>> of using the resources, compiler flags (beyond -O3), etc?
>>>>
>>>> Thanks,
>>>> Mohammad
>>>>
>>>> On Fri, May 18, 2012 at 4:18 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>>>
>>>>> On Fri, May 18, 2012 at 7:06 PM, Mohammad Mirzadeh <mirzadeh at gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> I'm trying to generate scalability plots for my code and do profiling
>>>>>> and fine tuning. In doing so I have noticed that some of the factors
>>>>>> affecting my results are sort of subtle. For example, I figured, the other
>>>>>> day, that using all of the cores on a single node is somewhat (50-60%)
>>>>>> slower when compared to using only half of the cores which I suspect is due
>>>>>> to memory bandwidth and/or other hardware-related issues.
>>>>>>
>>>>>> So I thought to ask and see if there is any example in petsc that has
>>>>>> been tested for scalability and has been documented? Basically I want to
>>>>>> use this test example as a benchmark to compare my results with. My own
>>>>>> test code is currently a linear Poisson solver on an adaptive quadtree grid
>>>>>> and involves non-trivial geometry (well basically a circle for the boundary
>>>>>> but still not a simple box).
>>>>>>
>>>>>
>>>>> Unfortunately, I do not even know what that means. We can't guarantee
>>>>> a certain level of performance because it not
>>>>> only depends on the hardware, but how you use it (as evident in your
>>>>> case). In a perfect world, we would have an abstract
>>>>> model of the computation (available for MatMult) and your machine (not
>>>>> available anywhere) and we would automatically
>>>>> work out the consequences and tell you what to expect. Instead today,
>>>>> we tell you to look at a few key indicators like the
>>>>> MatMult event, to see what is going on. When MatMult stops scaling,
>>>>> you have run out of bandwidth.
>>>>>
>>>>>     Matt
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>> Mohammad
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120518/d4223442/attachment-0001.htm>

From mike.hui.zhang at hotmail.com  Sat May 19 04:10:58 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Sat, 19 May 2012 11:10:58 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
Message-ID: <BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>

If PCASMSetSortIndices is used to set false, i.e. DO NOT sort indices,
is that true rows of submatrices are in the same order as the IS we passed to 
PCASMSetLocalSubdomains?  

Thanks!

From five9a2 at gmail.com  Sat May 19 17:51:37 2012
From: five9a2 at gmail.com (Jed Brown)
Date: Sat, 19 May 2012 17:51:37 -0500
Subject: [petsc-users] KSPSetDiagonalScale
In-Reply-To: <20120517155027.dzwy513g0okoksk4@cubmail.cc.columbia.edu>
References: <20120517152335.td44o7w54wkgok8g@cubmail.cc.columbia.edu>
	<CAM9tzSkY25jf_+5-NRScDi48KedStgrZZWjvbd6yyT1VWYE1kQ@mail.gmail.com>
	<20120517155027.dzwy513g0okoksk4@cubmail.cc.columbia.edu>
Message-ID: <CAM9tzSnZ3bSUY_bni5h90KXiVr4sruGNu40b5gAA3mQWHUGcvA@mail.gmail.com>

On Thu, May 17, 2012 at 2:50 PM, Colin McAuliffe <cjm2176 at columbia.edu>wrote:

> Using the direct solver as a preconditioner I can't seem to find any
> difference between the results with -ksp_diagonal_scale vs the results
> without this option.


I wouldn't expect it to be any different.

 I also cant tell if the diagonal scaling is actually being applied in this
> case.


Run with -ksp_view and look for a line that says "diagonally scaled system".
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120519/289026dc/attachment.html>

From mike.hui.zhang at hotmail.com  Sun May 20 05:06:29 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Sun, 20 May 2012 12:06:29 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
Message-ID: <BLU0-SMTP350328939FD9E40C247F5CCCB1C0@phx.gbl>

I have the following MatView output: 

row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517)  (5, -0.508793) 

Note that there are two values for the column 1, is this normal?

The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).

Thanks!



On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:

> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
> your modification subroutine), but not the subdomains or the subdomain solvers. 
> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
> 
> Dmitry.
> 
> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> I just have a question about reuse of PCASM or PCGASM.
> Suppose I have seted up the PCASM and related KSP and I solved one time.
> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
> different user context for the modifying routine.  
> 
> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
> even for PCASM I can re-use it because the partition of subdomains remain the same.
> 
> Thanks!
> 
> 
> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
> 
>> Hui,
>> There've been several changes to PCGASM ahead of the new release.
>> Let me go back and see if it affected the convergence problem.
>> Dmitry.
>> 
>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> Hi Dmitry,
>> 
>> is there any news about PCGASM? 
>> 
>> thanks,
>> Hui
>> 
>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>> 
>>> Okay, thanks.
>>> I'll take a look.
>>> 
>>> Dmitry.
>>> 
>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> For reference, my results are attached.
>>> 
>>> asm1.txt for asm with 1 process,
>>> asm2.txt for asm with 2 processes,
>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>> gasm2.txt for gasm with 2 processes
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> thank you,
>>> Hui
>>> 
>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>> 
>>>> 
>>>> 
>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> 
>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>> 
>>>>> 
>>>>> 
>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>> with GASM but no problems with ASM. 
>>>>> 
>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>> number is different from the first case and is much larger.  On the other hand
>>>>> ASM has no such problem.
>>>>>  
>>>>> Are the solutions the same?
>>>>> What problem are you solving?
>>>> 
>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>> two processors. But GASM did not.  
>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>> I'm solving the Helmholtz equation.  Maybe 
>>>> I can prepare a simpler example to show this difference.
>>>> That would be helpful.  
>>>> Thanks.
>>>> 
>>>> Dmitry. 
>>>> 
>>>>> 
>>>>> Dmitry. 
>>>>> 
>>>>> 
>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>> 
>>>>>> You should be able to. 
>>>>>> This behavior is the same as in PCASM,
>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>> 
>>>>>> Dmitry
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> 
>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>> 
>>>>>>>> Hi Dmitry,
>>>>>>>> 
>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>> 
>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>> 
>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>> 
>>>>>>> As I tested, the row and col are always the same. 
>>>>>>> 
>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>> in the above func()?
>>>>>>> 
>>>>>>> thanks,
>>>>>>> Hui
>>>>>>> 
>>>>>>>> 
>>>>>>>> thanks,
>>>>>>>> Hui
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>> 
>>>>>>>>> Yes, that's right.
>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>> 
>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>> 
>>>>>>>>> Dmitry.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>> 
>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Hui
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120520/9b755d30/attachment.html>

From karpeev at mcs.anl.gov  Sun May 20 09:34:06 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Sun, 20 May 2012 09:34:06 -0500
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
Message-ID: <CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>

On Sat, May 19, 2012 at 4:10 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> If PCASMSetSortIndices is used to set false, i.e. DO NOT sort indices,
> is that true rows of submatrices are in the same order as the IS we passed
> to
> PCASMSetLocalSubdomains?
>
Yes.
Dmitry.

>
> Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120520/435924af/attachment.html>

From karpeev at mcs.anl.gov  Sun May 20 09:36:33 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Sun, 20 May 2012 09:36:33 -0500
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP350328939FD9E40C247F5CCCB1C0@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP350328939FD9E40C247F5CCCB1C0@phx.gbl>
Message-ID: <CA+Y_9ULQfg=qet8mA8CeKAuknSH2jR3+yMwgARFue0QyZa+99w@mail.gmail.com>

On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> I have the following MatView output:
>
> row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4,
> -0.517586 - 0.617823 i) (1, -1.03517)  (5, -0.508793)
>
> Note that there are two values for the column 1, is this normal?
>
> The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and
> KSPGetOperators(subksp[0], ...).
>

This is odd. Can you explain a bit more what leads up to this, so we can
try and reproduce the problem?
Thanks.
Dmitry.

>
> Thanks!
>
>
>
> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>
> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and
> recreate the matrices (including calling
> your modification subroutine), but not the subdomains or the subdomain
> solvers.
> If you just want to modify the submatrices, you can call
> PC(G)ASMGetSubmatrices() and modify the matrices it returns
> (in the same order as the subdomains were set). That's a bit of a
> hack, since you will essentially be modifying the PC's internal data
> structures.  As long as you are careful, you should be okay, since you
> already effectively have the same type of access to the submatrices through
> the Modify callback.
>
> Dmitry.
>
> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>> I just have a question about reuse of PCASM or PCGASM.
>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>> Next for the same linear system (matrix and RHS), I just want PCASM
>> modify the submatrices (PCSetModifySubmatrices) in a different way, using
>> the same routine for modifying but with
>> different user context for the modifying routine.
>>
>> What can I do for this task?  Currently, I destroy the KSP and
>> re-construct it. I guess
>> even for PCASM I can re-use it because the partition of subdomains remain
>> the same.
>>
>> Thanks!
>>
>>
>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>
>> Hui,
>> There've been several changes to PCGASM ahead of the new release.
>> Let me go back and see if it affected the convergence problem.
>> Dmitry.
>>
>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>
>>> Hi Dmitry,
>>>
>>> is there any news about PCGASM?
>>>
>>> thanks,
>>> Hui
>>>
>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>
>>> Okay, thanks.
>>> I'll take a look.
>>>
>>> Dmitry.
>>>
>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>
>>>> For reference, my results are attached.
>>>>
>>>> asm1.txt for asm with 1 process,
>>>> asm2.txt for asm with 2 processes,
>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers
>>>> different from others)
>>>> gasm2.txt for gasm with 2 processes
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> thank you,
>>>> Hui
>>>>
>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>
>>>>
>>>>
>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com
>>>> > wrote:
>>>>
>>>>>
>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com
>>>>> > wrote:
>>>>>
>>>>>> I have a new problem: the results from ASM and GASM are different and
>>>>>> it seems
>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests
>>>>>> are with
>>>>>> each subdomain supported only by one subdomain. There are no problems
>>>>>> when
>>>>>> I did not modify submatrices.  But when I modify submatrices, there
>>>>>> are problems
>>>>>> with GASM but no problems with ASM.
>>>>>>
>>>>>> For example, I use two subdomains. In the first case each subdomain
>>>>>> is supported by
>>>>>> one processor and there seems no problem with GASM. But when I use
>>>>>> run my program
>>>>>> with only one proc. so that it supports both of the two subdomains,
>>>>>> the iteration
>>>>>> number is different from the first case and is much larger.  On the
>>>>>> other hand
>>>>>> ASM has no such problem.
>>>>>>
>>>>>
>>>>> Are the solutions the same?
>>>>> What problem are you solving?
>>>>>
>>>>>
>>>>> Yes, the solutions are the same. That's why ASM gives the same results
>>>>> with one or
>>>>> two processors. But GASM did not.
>>>>>
>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the
>>>> case of two domains per processor?
>>>>
>>>>> I'm solving the Helmholtz equation.  Maybe
>>>>> I can prepare a simpler example to show this difference.
>>>>>
>>>> That would be helpful.
>>>> Thanks.
>>>>
>>>> Dmitry.
>>>>
>>>>>
>>>>>
>>>>> Dmitry.
>>>>>
>>>>>>
>>>>>>
>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>
>>>>>> You should be able to.
>>>>>> This behavior is the same as in PCASM,
>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>
>>>>>> Dmitry
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>
>>>>>> Hi Dmitry,
>>>>>>
>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another
>>>>>> question
>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the
>>>>>> prototype
>>>>>>
>>>>>> func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>
>>>>>> I think the coloumns from the parameter 'col' are always the same as
>>>>>> the rows
>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only
>>>>>> accepts
>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>
>>>>>>
>>>>>> As I tested, the row and col are always the same.
>>>>>>
>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for
>>>>>> the submat's
>>>>>> in the above func()?
>>>>>>
>>>>>> thanks,
>>>>>> Hui
>>>>>>
>>>>>>
>>>>>> thanks,
>>>>>> Hui
>>>>>>
>>>>>>
>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>
>>>>>> Yes, that's right.
>>>>>> There is no good way to help the user assemble the subdomains at the
>>>>>> moment beyond the 2D stuff.
>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>
>>>>>> There is ISColoringToList() that is supposed to convert a "coloring"
>>>>>> of indices to an array of ISs,
>>>>>> each having the indices with the same color and the subcomm that
>>>>>> supports that color. It is
>>>>>> largely untested, though.  You could try using it and give us
>>>>>> feedback on any problems you encounter.
>>>>>>
>>>>>> Dmitry.
>>>>>>
>>>>>>
>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<mike.hui.zhang at hotmail.com>
>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>
>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain
>>>>>>> supported by
>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and
>>>>>>> 'is_local[s]'
>>>>>>> in a subcommunicator consisting of processors supporting the
>>>>>>> subdomain 's'?
>>>>>>>
>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hui
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120520/9b83aaf9/attachment-0001.html>

From mike.hui.zhang at hotmail.com  Sun May 20 10:00:10 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Sun, 20 May 2012 17:00:10 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CA+Y_9ULQfg=qet8mA8CeKAuknSH2jR3+yMwgARFue0QyZa+99w@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP350328939FD9E40C247F5CCCB1C0@phx.gbl>
	<CA+Y_9ULQfg=qet8mA8CeKAuknSH2jR3+yMwgARFue0QyZa+99w@mail.gmail.com>
Message-ID: <BLU0-SMTP21997A9FD1E6B1EA4FA5273CB1C0@phx.gbl>


On May 20, 2012, at 4:36 PM, Dmitry Karpeev wrote:

> 
> 
> On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> I have the following MatView output: 
> 
> row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517)  (5, -0.508793) 
> 
> Note that there are two values for the column 1, is this normal?
> 
> The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).
> 
> This is odd. Can you explain a bit more what leads up to this, so we can try and reproduce the problem?

Yes, I will try to make a simple example program.  




> Thanks.
> Dmitry.
> 
> Thanks!
> 
> 
> 
> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
> 
>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
>> your modification subroutine), but not the subdomains or the subdomain solvers. 
>> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
>> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
>> 
>> Dmitry.
>> 
>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> I just have a question about reuse of PCASM or PCGASM.
>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
>> different user context for the modifying routine.  
>> 
>> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
>> even for PCASM I can re-use it because the partition of subdomains remain the same.
>> 
>> Thanks!
>> 
>> 
>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>> 
>>> Hui,
>>> There've been several changes to PCGASM ahead of the new release.
>>> Let me go back and see if it affected the convergence problem.
>>> Dmitry.
>>> 
>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> Hi Dmitry,
>>> 
>>> is there any news about PCGASM? 
>>> 
>>> thanks,
>>> Hui
>>> 
>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>> 
>>>> Okay, thanks.
>>>> I'll take a look.
>>>> 
>>>> Dmitry.
>>>> 
>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> For reference, my results are attached.
>>>> 
>>>> asm1.txt for asm with 1 process,
>>>> asm2.txt for asm with 2 processes,
>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>>> gasm2.txt for gasm with 2 processes
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> thank you,
>>>> Hui
>>>> 
>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>> 
>>>>> 
>>>>> 
>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> 
>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>>> with GASM but no problems with ASM. 
>>>>>> 
>>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>>> number is different from the first case and is much larger.  On the other hand
>>>>>> ASM has no such problem.
>>>>>>  
>>>>>> Are the solutions the same?
>>>>>> What problem are you solving?
>>>>> 
>>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>>> two processors. But GASM did not.  
>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>>> I'm solving the Helmholtz equation.  Maybe 
>>>>> I can prepare a simpler example to show this difference.
>>>>> That would be helpful.  
>>>>> Thanks.
>>>>> 
>>>>> Dmitry. 
>>>>> 
>>>>>> 
>>>>>> Dmitry. 
>>>>>> 
>>>>>> 
>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>> 
>>>>>>> You should be able to. 
>>>>>>> This behavior is the same as in PCASM,
>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>> 
>>>>>>> Dmitry
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>> 
>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>> 
>>>>>>>>> Hi Dmitry,
>>>>>>>>> 
>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>>> 
>>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>> 
>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>> 
>>>>>>>> As I tested, the row and col are always the same. 
>>>>>>>> 
>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>>> in the above func()?
>>>>>>>> 
>>>>>>>> thanks,
>>>>>>>> Hui
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Hui
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>> 
>>>>>>>>>> Yes, that's right.
>>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>> 
>>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>>> 
>>>>>>>>>> Dmitry.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>>> 
>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> Hui
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120520/bb6c2dd8/attachment.html>

From mike.hui.zhang at hotmail.com  Mon May 21 13:59:52 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Mon, 21 May 2012 20:59:52 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CA+Y_9ULQfg=qet8mA8CeKAuknSH2jR3+yMwgARFue0QyZa+99w@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP350328939FD9E40C247F5CCCB1C0@phx.gbl>
	<CA+Y_9ULQfg=qet8mA8CeKAuknSH2jR3+yMwgARFue0QyZa+99w@mail.gmail.com>
Message-ID: <BLU0-SMTP169C9E4B3677CEFF78F426ECB1D0@phx.gbl>

Dmitry,

finally, I am able to make a simple program and reproduce the error. Since I had moved to
petsc-dev (before I used petsc-3.2), the error is a little different.

Download the program attached and run the run.sh.  You will see the output from PCASM.
There is a submatrix (submat[1] on rank 0) which has a row like

   row 1: (0, -1)  (1, 0 + 0.4 i) (4, -1)  (2, -1) 

Note that the column 2 is placed in the last and the column numbers are not in the normal order.
Following that, we can also see the error message like

   [0]PETSC ERROR: New nonzero at (1,2) caused a malloc!

You can also try with GASM, use

    mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1 

which partitions the domain to 1 * 2 and mapped to the two processors, and for each processor
we further partition its local domain to 2 * 1 subdomains. 

Then, you will see that the row 1 of submat[1] on rank 1 becomes

   row 1: (0, -1)  (1, 0 + 0.4 i) (3, -1)  (6, -1) 

which is totally wrong because from the IS for this overlapping subdomain and the big matrix A,
row 1 should have non-zeros like the output from PCASM.

I guess the problem is due to that we set PCASMSetSortIndices to FALSE and something goes wrong
in asm.c and gasm.c.

Thanks!
Hui








> On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> I have the following MatView output: 
> 
> row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517)  (5, -0.508793) 
> 
> Note that there are two values for the column 1, is this normal?
> 
> The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).
> 
> This is odd. Can you explain a bit more what leads up to this, so we can try and reproduce the problem?
> Thanks.
> Dmitry.
> 
> Thanks!
> 
> 
> 
> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
> 
>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
>> your modification subroutine), but not the subdomains or the subdomain solvers. 
>> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
>> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
>> 
>> Dmitry.
>> 
>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> I just have a question about reuse of PCASM or PCGASM.
>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
>> different user context for the modifying routine.  
>> 
>> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
>> even for PCASM I can re-use it because the partition of subdomains remain the same.
>> 
>> Thanks!
>> 
>> 
>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>> 
>>> Hui,
>>> There've been several changes to PCGASM ahead of the new release.
>>> Let me go back and see if it affected the convergence problem.
>>> Dmitry.
>>> 
>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> Hi Dmitry,
>>> 
>>> is there any news about PCGASM? 
>>> 
>>> thanks,
>>> Hui
>>> 
>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>> 
>>>> Okay, thanks.
>>>> I'll take a look.
>>>> 
>>>> Dmitry.
>>>> 
>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> For reference, my results are attached.
>>>> 
>>>> asm1.txt for asm with 1 process,
>>>> asm2.txt for asm with 2 processes,
>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>>> gasm2.txt for gasm with 2 processes
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> thank you,
>>>> Hui
>>>> 
>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>> 
>>>>> 
>>>>> 
>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> 
>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>>> with GASM but no problems with ASM. 
>>>>>> 
>>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>>> number is different from the first case and is much larger.  On the other hand
>>>>>> ASM has no such problem.
>>>>>>  
>>>>>> Are the solutions the same?
>>>>>> What problem are you solving?
>>>>> 
>>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>>> two processors. But GASM did not.  
>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>>> I'm solving the Helmholtz equation.  Maybe 
>>>>> I can prepare a simpler example to show this difference.
>>>>> That would be helpful.  
>>>>> Thanks.
>>>>> 
>>>>> Dmitry. 
>>>>> 
>>>>>> 
>>>>>> Dmitry. 
>>>>>> 
>>>>>> 
>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>> 
>>>>>>> You should be able to. 
>>>>>>> This behavior is the same as in PCASM,
>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>> 
>>>>>>> Dmitry
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>> 
>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>> 
>>>>>>>>> Hi Dmitry,
>>>>>>>>> 
>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>>> 
>>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>> 
>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>> 
>>>>>>>> As I tested, the row and col are always the same. 
>>>>>>>> 
>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>>> in the above func()?
>>>>>>>> 
>>>>>>>> thanks,
>>>>>>>> Hui
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Hui
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>> 
>>>>>>>>>> Yes, that's right.
>>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>> 
>>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>>> 
>>>>>>>>>> Dmitry.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>>> 
>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> Hui
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/ea19bf1d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GASM_test.zip
Type: application/zip
Size: 7088 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/ea19bf1d/attachment.zip>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/ea19bf1d/attachment-0001.html>

From mike.hui.zhang at hotmail.com  Mon May 21 14:11:54 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Mon, 21 May 2012 21:11:54 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP169C9E4B3677CEFF78F426ECB1D0@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP350328939FD9E40C247F5CCCB1C0@phx.gbl>
	<CA+Y_9ULQfg=qet8mA8CeKAuknSH2jR3+yMwgARFue0QyZa+99w@mail.gmail.com>
	<BLU0-SMTP169C9E4B3677CEFF78F426ECB1D0@phx.gbl>
Message-ID: <BLU0-SMTP191BA30F4023240B6508FBBCB1D0@phx.gbl>

Sorry, I uses PCASMSetSortIndices for GASM. Now, I correct it and the output becomes the
same as PCASM. See attached.




On May 21, 2012, at 8:59 PM, Hui Zhang wrote:

> Dmitry,
> 
> finally, I am able to make a simple program and reproduce the error. Since I had moved to
> petsc-dev (before I used petsc-3.2), the error is a little different.
> 
> Download the program attached and run the run.sh.  You will see the output from PCASM.
> There is a submatrix (submat[1] on rank 0) which has a row like
> 
>    row 1: (0, -1)  (1, 0 + 0.4 i) (4, -1)  (2, -1) 
> 
> Note that the column 2 is placed in the last and the column numbers are not in the normal order.
> Following that, we can also see the error message like
> 
>    [0]PETSC ERROR: New nonzero at (1,2) caused a malloc!
> 
> You can also try with GASM, use
> 
>     mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1 
> 
> which partitions the domain to 1 * 2 and mapped to the two processors, and for each processor
> we further partition its local domain to 2 * 1 subdomains. 
> 
> Then, you will see that the row 1 of submat[1] on rank 1 becomes
> 
>    row 1: (0, -1)  (1, 0 + 0.4 i) (3, -1)  (6, -1) 
> 
> which is totally wrong because from the IS for this overlapping subdomain and the big matrix A,
> row 1 should have non-zeros like the output from PCASM.
> 
> I guess the problem is due to that we set PCASMSetSortIndices to FALSE and something goes wrong
> in asm.c and gasm.c.
> 
> Thanks!
> Hui
> 
> 
> <GASM_test.zip>
> 
> 
> 
> 
> 
>> On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> I have the following MatView output: 
>> 
>> row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517)  (5, -0.508793) 
>> 
>> Note that there are two values for the column 1, is this normal?
>> 
>> The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).
>> 
>> This is odd. Can you explain a bit more what leads up to this, so we can try and reproduce the problem?
>> Thanks.
>> Dmitry.
>> 
>> Thanks!
>> 
>> 
>> 
>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>> 
>>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
>>> your modification subroutine), but not the subdomains or the subdomain solvers. 
>>> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
>>> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
>>> 
>>> Dmitry.
>>> 
>>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> I just have a question about reuse of PCASM or PCGASM.
>>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>>> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
>>> different user context for the modifying routine.  
>>> 
>>> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
>>> even for PCASM I can re-use it because the partition of subdomains remain the same.
>>> 
>>> Thanks!
>>> 
>>> 
>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>> 
>>>> Hui,
>>>> There've been several changes to PCGASM ahead of the new release.
>>>> Let me go back and see if it affected the convergence problem.
>>>> Dmitry.
>>>> 
>>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> Hi Dmitry,
>>>> 
>>>> is there any news about PCGASM? 
>>>> 
>>>> thanks,
>>>> Hui
>>>> 
>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>> 
>>>>> Okay, thanks.
>>>>> I'll take a look.
>>>>> 
>>>>> Dmitry.
>>>>> 
>>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> For reference, my results are attached.
>>>>> 
>>>>> asm1.txt for asm with 1 process,
>>>>> asm2.txt for asm with 2 processes,
>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>>>> gasm2.txt for gasm with 2 processes
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> thank you,
>>>>> Hui
>>>>> 
>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> 
>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>>>> with GASM but no problems with ASM. 
>>>>>>> 
>>>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>>>> number is different from the first case and is much larger.  On the other hand
>>>>>>> ASM has no such problem.
>>>>>>>  
>>>>>>> Are the solutions the same?
>>>>>>> What problem are you solving?
>>>>>> 
>>>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>>>> two processors. But GASM did not.  
>>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>>>> I'm solving the Helmholtz equation.  Maybe 
>>>>>> I can prepare a simpler example to show this difference.
>>>>>> That would be helpful.  
>>>>>> Thanks.
>>>>>> 
>>>>>> Dmitry. 
>>>>>> 
>>>>>>> 
>>>>>>> Dmitry. 
>>>>>>> 
>>>>>>> 
>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>> 
>>>>>>>> You should be able to. 
>>>>>>>> This behavior is the same as in PCASM,
>>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>> 
>>>>>>>> Dmitry
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>> 
>>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>>> 
>>>>>>>>>> Hi Dmitry,
>>>>>>>>>> 
>>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>>>> 
>>>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>>> 
>>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>>> 
>>>>>>>>> As I tested, the row and col are always the same. 
>>>>>>>>> 
>>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>>>> in the above func()?
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> Hui
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Hui
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>>> 
>>>>>>>>>>> Yes, that's right.
>>>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>>> 
>>>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>>>> 
>>>>>>>>>>> Dmitry.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>>>> 
>>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Hui
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/b78f83b4/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GASM_test.zip
Type: application/zip
Size: 7094 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/b78f83b4/attachment-0001.zip>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/b78f83b4/attachment-0003.html>

From andrew.spott at gmail.com  Mon May 21 14:20:01 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Mon, 21 May 2012 13:20:01 -0600
Subject: [petsc-users] nonzero structure setting and remembering
In-Reply-To: <CAM9tzSmu+h4sHxrrZNrdUuU79cA6-By0TmJxm9gNsUBNcQ2N4Q@mail.gmail.com>
References: <8AB4A662-FCC1-455F-962A-8ABD168D42A8@gmail.com>
	<CAM9tzSmu+h4sHxrrZNrdUuU79cA6-By0TmJxm9gNsUBNcQ2N4Q@mail.gmail.com>
Message-ID: <0ECE363B-DA8B-4B2E-A776-AC1A92DD75D7@gmail.com>

How do I read the nonzero structure (from an assembled matrix?)

-Andrew

On May 18, 2012, at 2:31 PM, Jed Brown wrote:

> On Fri, May 18, 2012 at 3:26 PM, Andrew Spott <andrew.spott at gmail.com> wrote:
> I have a couple of questions about how to deal with the nonzero structure.
> 
> 0)  Is the "nonzero structure" the same as "number of non zeros per row"? Or are they different?
> 
> Yes, for parallel *AIJ matrices, it's the number of nonzeros in the diagonal part and in the off-diagonal part. Preallocation can optionally also set the column indices, but it's not critical.
>  
> 
> 1)  How much of a speedup do you get for copying/creating/loading matrices when you set the number of non zeros per row beforehand? (if the nonzero structure is different from the number of non zeros per row, what about the nonzero structure)
> 
> a million
>  
> 
> 2)  If I have a matrix, how do I add to the nonzero structure, without actually adding nonzero values, do I just add "0.0"? (the matrix will have values there, but doesn't at the moment)
> 
> Mat*AIJSetPreallocation(), then reassemble.
>  
> 
> As usual, thanks for all the help
> 
> -Andrew
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/80c4d042/attachment.html>

From jedbrown at mcs.anl.gov  Mon May 21 14:29:05 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 21 May 2012 14:29:05 -0500
Subject: [petsc-users] nonzero structure setting and remembering
In-Reply-To: <0ECE363B-DA8B-4B2E-A776-AC1A92DD75D7@gmail.com>
References: <8AB4A662-FCC1-455F-962A-8ABD168D42A8@gmail.com>
	<CAM9tzSmu+h4sHxrrZNrdUuU79cA6-By0TmJxm9gNsUBNcQ2N4Q@mail.gmail.com>
	<0ECE363B-DA8B-4B2E-A776-AC1A92DD75D7@gmail.com>
Message-ID: <CAM9tzSmBwR6JB_aakNQSMYRskH=eUkHys9moYKBzpio9xUyEcQ@mail.gmail.com>

On Mon, May 21, 2012 at 2:20 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> How do I read the nonzero structure (from an assembled matrix?)
>

MatGetRow()


>
> -Andrew
>
> On May 18, 2012, at 2:31 PM, Jed Brown wrote:
>
> On Fri, May 18, 2012 at 3:26 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>
>> I have a couple of questions about how to deal with the nonzero structure.
>>
>> 0)  Is the "nonzero structure" the same as "number of non zeros per row"?
>> Or are they different?
>>
>
> Yes, for parallel *AIJ matrices, it's the number of nonzeros in the
> diagonal part and in the off-diagonal part. Preallocation can optionally
> also set the column indices, but it's not critical.
>
>
>>
>> 1)  How much of a speedup do you get for copying/creating/loading
>> matrices when you set the number of non zeros per row beforehand? (if the
>> nonzero structure is different from the number of non zeros per row, what
>> about the nonzero structure)
>>
>
> a million
>
>
>>
>> 2)  If I have a matrix, how do I add to the nonzero structure, without
>> actually adding nonzero values, do I just add "0.0"? (the matrix will have
>> values there, but doesn't at the moment)
>>
>
> Mat*AIJSetPreallocation(), then reassemble.
>
>
>>
>> As usual, thanks for all the help
>>
>> -Andrew
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/9229353e/attachment.html>

From karpeev at mcs.anl.gov  Mon May 21 14:32:25 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Mon, 21 May 2012 14:32:25 -0500
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP191BA30F4023240B6508FBBCB1D0@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP350328939FD9E40C247F5CCCB1C0@phx.gbl>
	<CA+Y_9ULQfg=qet8mA8CeKAuknSH2jR3+yMwgARFue0QyZa+99w@mail.gmail.com>
	<BLU0-SMTP169C9E4B3677CEFF78F426ECB1D0@phx.gbl>
	<BLU0-SMTP191BA30F4023240B6508FBBCB1D0@phx.gbl>
Message-ID: <CA+Y_9UK428a74+QOW5Vn-XW6wwrbB6ei0J20YE6xO6Jssotf+A@mail.gmail.com>

Okay, thanks.
I'll look into it.
Dmitry.

On Mon, May 21, 2012 at 2:11 PM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Sorry, I uses PCASMSetSortIndices for GASM. Now, I correct it and the
> output becomes the
> same as PCASM. See attached.
>
>
>
>
> On May 21, 2012, at 8:59 PM, Hui Zhang wrote:
>
> Dmitry,
>
> finally, I am able to make a simple program and reproduce the error. Since
> I had moved to
> petsc-dev (before I used petsc-3.2), the error is a little different.
>
> Download the program attached and run the run.sh.  You will see the output
> from PCASM.
> There is a submatrix (submat[1] on rank 0) which has a row like
>
>    row 1: (0, -1)  (1, 0 + 0.4 i) (4, -1)  (2, -1)
>
> Note that the column 2 is placed in the last and the column numbers are
> not in the normal order.
> Following that, we can also see the error message like
>
>    [0]PETSC ERROR: New nonzero at (1,2) caused a malloc!
>
> You can also try with GASM, use
>
>     mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1
>
> which partitions the domain to 1 * 2 and mapped to the two processors, and
> for each processor
> we further partition its local domain to 2 * 1 subdomains.
>
> Then, you will see that the row 1 of submat[1] on rank 1 becomes
>
>    row 1: (0, -1)  (1, 0 + 0.4 i) (3, -1)  (6, -1)
>
> which is totally wrong because from the IS for this overlapping subdomain
> and the big matrix A,
> row 1 should have non-zeros like the output from PCASM.
>
> I guess the problem is due to that we set PCASMSetSortIndices to FALSE and
> something goes wrong
> in asm.c and gasm.c.
>
> Thanks!
> Hui
>
>
> <GASM_test.zip>
>
>
>
>
>
> On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>> I have the following MatView output:
>>
>> row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4,
>> -0.517586 - 0.617823 i) (1, -1.03517)  (5, -0.508793)
>>
>> Note that there are two values for the column 1, is this normal?
>>
>> The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and
>> KSPGetOperators(subksp[0], ...).
>>
>
> This is odd. Can you explain a bit more what leads up to this, so we can
> try and reproduce the problem?
> Thanks.
> Dmitry.
>
>>
>> Thanks!
>>
>>
>>
>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>
>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and
>> recreate the matrices (including calling
>> your modification subroutine), but not the subdomains or the subdomain
>> solvers.
>> If you just want to modify the submatrices, you can call
>> PC(G)ASMGetSubmatrices() and modify the matrices it returns
>> (in the same order as the subdomains were set). That's a bit of a
>> hack, since you will essentially be modifying the PC's internal data
>> structures.  As long as you are careful, you should be okay, since you
>> already effectively have the same type of access to the submatrices through
>> the Modify callback.
>>
>> Dmitry.
>>
>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>
>>> I just have a question about reuse of PCASM or PCGASM.
>>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>>> Next for the same linear system (matrix and RHS), I just want PCASM
>>> modify the submatrices (PCSetModifySubmatrices) in a different way, using
>>> the same routine for modifying but with
>>> different user context for the modifying routine.
>>>
>>> What can I do for this task?  Currently, I destroy the KSP and
>>> re-construct it. I guess
>>> even for PCASM I can re-use it because the partition of subdomains
>>> remain the same.
>>>
>>> Thanks!
>>>
>>>
>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>
>>> Hui,
>>> There've been several changes to PCGASM ahead of the new release.
>>> Let me go back and see if it affected the convergence problem.
>>> Dmitry.
>>>
>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>
>>>> Hi Dmitry,
>>>>
>>>> is there any news about PCGASM?
>>>>
>>>> thanks,
>>>> Hui
>>>>
>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>
>>>> Okay, thanks.
>>>> I'll take a look.
>>>>
>>>> Dmitry.
>>>>
>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com
>>>> > wrote:
>>>>
>>>>> For reference, my results are attached.
>>>>>
>>>>> asm1.txt for asm with 1 process,
>>>>> asm2.txt for asm with 2 processes,
>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers
>>>>> different from others)
>>>>> gasm2.txt for gasm with 2 processes
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> thank you,
>>>>> Hui
>>>>>
>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <
>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>
>>>>>>
>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <
>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>
>>>>>>> I have a new problem: the results from ASM and GASM are different
>>>>>>> and it seems
>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests
>>>>>>> are with
>>>>>>> each subdomain supported only by one subdomain. There are no
>>>>>>> problems when
>>>>>>> I did not modify submatrices.  But when I modify submatrices, there
>>>>>>> are problems
>>>>>>> with GASM but no problems with ASM.
>>>>>>>
>>>>>>> For example, I use two subdomains. In the first case each subdomain
>>>>>>> is supported by
>>>>>>> one processor and there seems no problem with GASM. But when I use
>>>>>>> run my program
>>>>>>> with only one proc. so that it supports both of the two subdomains,
>>>>>>> the iteration
>>>>>>> number is different from the first case and is much larger.  On the
>>>>>>> other hand
>>>>>>> ASM has no such problem.
>>>>>>>
>>>>>>
>>>>>> Are the solutions the same?
>>>>>> What problem are you solving?
>>>>>>
>>>>>>
>>>>>> Yes, the solutions are the same. That's why ASM gives the same
>>>>>> results with one or
>>>>>> two processors. But GASM did not.
>>>>>>
>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in
>>>>> the case of two domains per processor?
>>>>>
>>>>>> I'm solving the Helmholtz equation.  Maybe
>>>>>> I can prepare a simpler example to show this difference.
>>>>>>
>>>>> That would be helpful.
>>>>> Thanks.
>>>>>
>>>>> Dmitry.
>>>>>
>>>>>>
>>>>>>
>>>>>> Dmitry.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>
>>>>>>> You should be able to.
>>>>>>> This behavior is the same as in PCASM,
>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>
>>>>>>> Dmitry
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>
>>>>>>> Hi Dmitry,
>>>>>>>
>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes
>>>>>>> another question
>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the
>>>>>>> prototype
>>>>>>>
>>>>>>> func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>
>>>>>>> I think the coloumns from the parameter 'col' are always the same as
>>>>>>> the rows
>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only
>>>>>>> accepts
>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>
>>>>>>>
>>>>>>> As I tested, the row and col are always the same.
>>>>>>>
>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for
>>>>>>> the submat's
>>>>>>> in the above func()?
>>>>>>>
>>>>>>> thanks,
>>>>>>> Hui
>>>>>>>
>>>>>>>
>>>>>>> thanks,
>>>>>>> Hui
>>>>>>>
>>>>>>>
>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>
>>>>>>> Yes, that's right.
>>>>>>> There is no good way to help the user assemble the subdomains at the
>>>>>>> moment beyond the 2D stuff.
>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>
>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring"
>>>>>>> of indices to an array of ISs,
>>>>>>> each having the indices with the same color and the subcomm that
>>>>>>> supports that color. It is
>>>>>>> largely untested, though.  You could try using it and give us
>>>>>>> feedback on any problems you encounter.
>>>>>>>
>>>>>>> Dmitry.
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<mike.hui.zhang at hotmail.com>
>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>
>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain
>>>>>>>> supported by
>>>>>>>> multiple processors, shall I always create the arguments 'is[s]'
>>>>>>>> and 'is_local[s]'
>>>>>>>> in a subcommunicator consisting of processors supporting the
>>>>>>>> subdomain 's'?
>>>>>>>>
>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Hui
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/7b569045/attachment-0001.html>

From mike.hui.zhang at hotmail.com  Mon May 21 14:33:41 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Mon, 21 May 2012 21:33:41 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP191BA30F4023240B6508FBBCB1D0@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP350328939FD9E40C247F5CCCB1C0@phx.gbl>
	<CA+Y_9ULQfg=qet8mA8CeKAuknSH2jR3+yMwgARFue0QyZa+99w@mail.gmail.com>
	<BLU0-SMTP169C9E4B3677CEFF78F426ECB1D0@phx.gbl>
	<BLU0-SMTP191BA30F4023240B6508FBBCB1D0@phx.gbl>
Message-ID: <BLU0-SMTP50102871DC41171EC903A9CB1D0@phx.gbl>

Hi, 

It might be unclear.  The error here is that the submat[1] does have a (1,2) entry,
but MatSetValues at (1,2) gives an error message.  GASM and ASM give the same error.

Thanks!

On May 21, 2012, at 9:11 PM, Hui Zhang wrote:

> Sorry, I uses PCASMSetSortIndices for GASM. Now, I correct it and the output becomes the
> same as PCASM. See attached.
> 
> <GASM_test.zip>
> 
> 
> On May 21, 2012, at 8:59 PM, Hui Zhang wrote:
> 
>> Dmitry,
>> 
>> finally, I am able to make a simple program and reproduce the error. Since I had moved to
>> petsc-dev (before I used petsc-3.2), the error is a little different.
>> 
>> Download the program attached and run the run.sh.  You will see the output from PCASM.
>> There is a submatrix (submat[1] on rank 0) which has a row like
>> 
>>    row 1: (0, -1)  (1, 0 + 0.4 i) (4, -1)  (2, -1) 
>> 
>> Note that the column 2 is placed in the last and the column numbers are not in the normal order.
>> Following that, we can also see the error message like
>> 
>>    [0]PETSC ERROR: New nonzero at (1,2) caused a malloc!
>> 
>> You can also try with GASM, use
>> 
>>     mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1 
>> 
>> which partitions the domain to 1 * 2 and mapped to the two processors, and for each processor
>> we further partition its local domain to 2 * 1 subdomains. 
>> 
>> Then, you will see that the row 1 of submat[1] on rank 1 becomes
>> 
>>    row 1: (0, -1)  (1, 0 + 0.4 i) (3, -1)  (6, -1) 
>> 
>> which is totally wrong because from the IS for this overlapping subdomain and the big matrix A,
>> row 1 should have non-zeros like the output from PCASM.
>> 
>> I guess the problem is due to that we set PCASMSetSortIndices to FALSE and something goes wrong
>> in asm.c and gasm.c.
>> 
>> Thanks!
>> Hui
>> 
>> 
>> <GASM_test.zip>
>> 
>> 
>> 
>> 
>> 
>>> On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> I have the following MatView output: 
>>> 
>>> row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517)  (5, -0.508793) 
>>> 
>>> Note that there are two values for the column 1, is this normal?
>>> 
>>> The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).
>>> 
>>> This is odd. Can you explain a bit more what leads up to this, so we can try and reproduce the problem?
>>> Thanks.
>>> Dmitry.
>>> 
>>> Thanks!
>>> 
>>> 
>>> 
>>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>> 
>>>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
>>>> your modification subroutine), but not the subdomains or the subdomain solvers. 
>>>> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
>>>> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
>>>> 
>>>> Dmitry.
>>>> 
>>>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> I just have a question about reuse of PCASM or PCGASM.
>>>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>>>> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
>>>> different user context for the modifying routine.  
>>>> 
>>>> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
>>>> even for PCASM I can re-use it because the partition of subdomains remain the same.
>>>> 
>>>> Thanks!
>>>> 
>>>> 
>>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>> 
>>>>> Hui,
>>>>> There've been several changes to PCGASM ahead of the new release.
>>>>> Let me go back and see if it affected the convergence problem.
>>>>> Dmitry.
>>>>> 
>>>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> Hi Dmitry,
>>>>> 
>>>>> is there any news about PCGASM? 
>>>>> 
>>>>> thanks,
>>>>> Hui
>>>>> 
>>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>> 
>>>>>> Okay, thanks.
>>>>>> I'll take a look.
>>>>>> 
>>>>>> Dmitry.
>>>>>> 
>>>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> For reference, my results are attached.
>>>>>> 
>>>>>> asm1.txt for asm with 1 process,
>>>>>> asm2.txt for asm with 2 processes,
>>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>>>>> gasm2.txt for gasm with 2 processes
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> thank you,
>>>>>> Hui
>>>>>> 
>>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>> 
>>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>>>>> with GASM but no problems with ASM. 
>>>>>>>> 
>>>>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>>>>> number is different from the first case and is much larger.  On the other hand
>>>>>>>> ASM has no such problem.
>>>>>>>>  
>>>>>>>> Are the solutions the same?
>>>>>>>> What problem are you solving?
>>>>>>> 
>>>>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>>>>> two processors. But GASM did not.  
>>>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>>>>> I'm solving the Helmholtz equation.  Maybe 
>>>>>>> I can prepare a simpler example to show this difference.
>>>>>>> That would be helpful.  
>>>>>>> Thanks.
>>>>>>> 
>>>>>>> Dmitry. 
>>>>>>> 
>>>>>>>> 
>>>>>>>> Dmitry. 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>> 
>>>>>>>>> You should be able to. 
>>>>>>>>> This behavior is the same as in PCASM,
>>>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>>> 
>>>>>>>>> Dmitry
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Dmitry,
>>>>>>>>>>> 
>>>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>>>>> 
>>>>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>>>> 
>>>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>>>> 
>>>>>>>>>> As I tested, the row and col are always the same. 
>>>>>>>>>> 
>>>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>>>>> in the above func()?
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> Hui
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> thanks,
>>>>>>>>>>> Hui
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Yes, that's right.
>>>>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>>>> 
>>>>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>>>>> 
>>>>>>>>>>>> Dmitry.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>>>>> 
>>>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Hui
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/eb173c57/attachment.html>

From w_ang_temp at 163.com  Tue May 22 04:17:39 2012
From: w_ang_temp at 163.com (w_ang_temp)
Date: Tue, 22 May 2012 17:17:39 +0800 (CST)
Subject: [petsc-users] A qustion of PETSc
Message-ID: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com>

Hello
     I am a beginner of PETSc.I find the petsc-users lists is very useful, however I do not know how to ask a question in it. So can you tell me? And below is what I want to ask.
    I am solving Ax=b with PETSc. And I only use it in a fortran subroutine to get x ,then the main function of my program use x to do other things. The question is that: when the process is 1(mpiexec -n 1),the result of my program is ok, but when the process is more than 1 the result is wrong.
    So where could be wrong? Can you give me some hints on how I can debug? Thanks.
                                  Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/4a64ccce/attachment.html>

From mike.hui.zhang at hotmail.com  Tue May 22 05:23:26 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Tue, 22 May 2012 12:23:26 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
	<CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
Message-ID: <BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>

Hi,

I have got a new question.  I'm now constructing a PCComposite from PCASM and another PCKSP (coarse
problem). And construction of PCKSP needs to use the subksp's of PCASM. So I need to PCSetUp on the
PCASM, right?  But the KSP using PCComposite would setup PCASM again.  How can I avoid twice setup
of PCASM ?

Thanks,
Hui



From w_ang_temp at 163.com  Tue May 22 06:04:06 2012
From: w_ang_temp at 163.com (w_ang_temp)
Date: Tue, 22 May 2012 19:04:06 +0800 (CST)
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
	<CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
	<BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
Message-ID: <436e553b.1b98b.1377437d99d.Coremail.w_ang_temp@163.com>

PCASM?????sorry





At 2012-05-22 18:23:26,"Hui Zhang" <mike.hui.zhang at hotmail.com> wrote:
>Hi,
>
>I have got a new question.  I'm now constructing a PCComposite from PCASM and another PCKSP (coarse
>problem). And construction of PCKSP needs to use the subksp's of PCASM. So I need to PCSetUp on the
>PCASM, right?  But the KSP using PCComposite would setup PCASM again.  How can I avoid twice setup
>of PCASM ?
>
>Thanks,
>Hui
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/39a3a889/attachment.html>

From C.Klaij at marin.nl  Tue May 22 06:10:51 2012
From: C.Klaij at marin.nl (Klaij, Christiaan)
Date: Tue, 22 May 2012 11:10:51 +0000
Subject: [petsc-users] KSPBuildResidual in ex42.c
Message-ID: <A9AE64C525438E45B1D60F4D09AEAA9E06201F@MAR160N1.marin.local>

In src/ksp/ksp/examples/tutorials/ex42.c.html the Stokes system
is solved and KSPBuildResidual is used to compute the residuals
for the velocity components and pressure individually. Are these
the preconditioned residuals or the true residuals? And how to
get both?


dr. ir. Christiaan Klaij
CFD Researcher
Research & Development
E mailto:C.Klaij at marin.nl
T +31 317 49 33 44

MARIN
2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands
T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl


From knepley at gmail.com  Tue May 22 06:33:10 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 22 May 2012 07:33:10 -0400
Subject: [petsc-users] A qustion of PETSc
In-Reply-To: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com>
References: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com>
Message-ID: <CAMYG4G=TX7URdbveZA6bTs6LOi7kiDV3kbf5F=JW81JAPPYeSg@mail.gmail.com>

On Tue, May 22, 2012 at 5:17 AM, w_ang_temp <w_ang_temp at 163.com> wrote:

> Hello
>      I am a beginner of PETSc.I find the petsc-users lists is very useful,
> however I do not know how to ask a question in it. So can you tell me? And
> below is what I want to ask.
>     I am solving Ax=b with PETSc. And I only use it in a fortran
> subroutine to get x ,then the main function of my program use x to do other
> things. The question is that: when the process is 1(mpiexec -n 1),the
> result of my program is ok, but when the process is more than 1 the result
> is wrong.
>

"Wrong" is not an acceptable description. I suspect that you do not
understand that only part of a Vec
is stored on each MPI process. First, get the book Using MPI, and then read
the PETSc manual
section on vectors.

  Thanks,

     Matt


>     So where could be wrong? Can you give me some hints on how I can
> debug? Thanks.
>                                   Jim
>
-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/0b620372/attachment.html>

From jedbrown at mcs.anl.gov  Tue May 22 06:44:39 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 22 May 2012 06:44:39 -0500
Subject: [petsc-users] KSPBuildResidual in ex42.c
In-Reply-To: <A9AE64C525438E45B1D60F4D09AEAA9E06201F@MAR160N1.marin.local>
References: <A9AE64C525438E45B1D60F4D09AEAA9E06201F@MAR160N1.marin.local>
Message-ID: <CAM9tzSkt3a9HsrfMFPPtsXW=E09zgmPwoSuGUFzAb8NDZdmaWA@mail.gmail.com>

On Tue, May 22, 2012 at 6:10 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:

> In src/ksp/ksp/examples/tutorials/ex42.c.html the Stokes system
> is solved and KSPBuildResidual is used to compute the residuals
> for the velocity components and pressure individually. Are these
> the preconditioned residuals or the true residuals?


The true residual.


> And how to
> get both?
>

Precondition it.

Note that if you only want norms, GMRES will compute one (preconditioned by
default).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/d639206b/attachment.html>

From w_ang_temp at 163.com  Tue May 22 09:19:05 2012
From: w_ang_temp at 163.com (w_ang_temp)
Date: Tue, 22 May 2012 22:19:05 +0800 (CST)
Subject: [petsc-users] A qustion of PETSc
In-Reply-To: <CAMYG4G=TX7URdbveZA6bTs6LOi7kiDV3kbf5F=JW81JAPPYeSg@mail.gmail.com>
References: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com>
	<CAMYG4G=TX7URdbveZA6bTs6LOi7kiDV3kbf5F=JW81JAPPYeSg@mail.gmail.com>
Message-ID: <501bd00d.2b726.13774ea5c7a.Coremail.w_ang_temp@163.com>

Hello Matt
    What you said that "I suspect that you do not understand that only part of a Vec is stored on each MPI process" may be the reason.And I have spent lots of time testing this,especially get results from Vec x (the result of Ax=b)to the main program variable.  And I will have a try again.
    Besides, I have a puzzle about the process (mpiexec -n 2).As you know,I only use PETSc in the subroutine(a function named PETSCSOLVE,may be taken as a tool function) and the main program call it for many times. So,where should I call "PetscInitialize" and "PetscFinalize",in the main program or in the subroutine PETSCSOLVE?
    Thanks.                                                  Jim




? 2012-05-22 19:33:10?"Matthew Knepley" <knepley at gmail.com> ???
On Tue, May 22, 2012 at 5:17 AM, w_ang_temp <w_ang_temp at 163.com> wrote:

Hello
     I am a beginner of PETSc.I find the petsc-users lists is very useful, however I do not know how to ask a question in it. So can you tell me? And below is what I want to ask.
    I am solving Ax=b with PETSc. And I only use it in a fortran subroutine to get x ,then the main function of my program use x to do other things. The question is that: when the process is 1(mpiexec -n 1),the result of my program is ok, but when the process is more than 1 the result is wrong.



"Wrong" is not an acceptable description. I suspect that you do not understand that only part of a Vec
is stored on each MPI process. First, get the book Using MPI, and then read the PETSc manual
section on vectors.


  Thanks,


     Matt
 
    So where could be wrong? Can you give me some hints on how I can debug? Thanks.
                                  Jim
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/9c9c78de/attachment.html>

From aron.ahmadia at kaust.edu.sa  Tue May 22 09:29:09 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Tue, 22 May 2012 17:29:09 +0300
Subject: [petsc-users] A qustion of PETSc
In-Reply-To: <501bd00d.2b726.13774ea5c7a.Coremail.w_ang_temp@163.com>
References: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com>
	<CAMYG4G=TX7URdbveZA6bTs6LOi7kiDV3kbf5F=JW81JAPPYeSg@mail.gmail.com>
	<501bd00d.2b726.13774ea5c7a.Coremail.w_ang_temp@163.com>
Message-ID: <CADVoUZQrWjrb0tHFroPd=m1g1C9_koa2avO4498Udt2yLGUQyA@mail.gmail.com>

>
>     Besides, I have a puzzle about the process (mpiexec -n 2).As you
> know,I only use PETSc in the subroutine(a function named PETSCSOLVE,may be
> taken as a tool function) and the main program call it for many times.
> So,where should I call "PetscInitialize" and "PetscFinalize",in the main
> program or in the subroutine PETSCSOLVE?
>     Thanks.                                                  Jim
>

You should call PetscInitialize as soon as possible and PetscFinalize as
late as possible.  Please read the PETSc user's guide and consult it before
asking questions that are clearly answered in the manual.

-Aron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/a9b6d039/attachment.html>

From w_ang_temp at 163.com  Tue May 22 09:34:17 2012
From: w_ang_temp at 163.com (w_ang_temp)
Date: Tue, 22 May 2012 22:34:17 +0800 (CST)
Subject: [petsc-users] A qustion of PETSc
In-Reply-To: <CADVoUZQrWjrb0tHFroPd=m1g1C9_koa2avO4498Udt2yLGUQyA@mail.gmail.com>
References: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com>
	<CAMYG4G=TX7URdbveZA6bTs6LOi7kiDV3kbf5F=JW81JAPPYeSg@mail.gmail.com>
	<501bd00d.2b726.13774ea5c7a.Coremail.w_ang_temp@163.com>
	<CADVoUZQrWjrb0tHFroPd=m1g1C9_koa2avO4498Udt2yLGUQyA@mail.gmail.com>
Message-ID: <5e2c748c.2f72f.13774f84988.Coremail.w_ang_temp@163.com>


Really sorry!



? 2012-05-22 22:29:09?"Aron Ahmadia" <aron.ahmadia at kaust.edu.sa> ???

> >    Besides, I have a puzzle about the process (mpiexec -n 2).As you know,I only use PETSc in the subroutine(a function named >>  PETSCSOLVE,may be taken as a tool function) and the main program call it for many times. So,where should I call "PetscInitialize" >>  and "PetscFinalize",in the main program or in the subroutine PETSCSOLVE?
>>      Thanks.                                                  Jim



> You should call PetscInitialize as soon as possible and PetscFinalize as late as possible.  Please read the PETSc user's guide and consult it > before asking questions that are clearly answered in the manual.


> -Aron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/3eec69f0/attachment-0001.html>

From C.Klaij at marin.nl  Tue May 22 09:54:40 2012
From: C.Klaij at marin.nl (Klaij, Christiaan)
Date: Tue, 22 May 2012 14:54:40 +0000
Subject: [petsc-users] KSPBuildResidual in ex42.c
Message-ID: <A9AE64C525438E45B1D60F4D09AEAA9E062051@MAR160N1.marin.local>

> > In src/ksp/ksp/examples/tutorials/ex42.c.html the Stokes system
> > is solved and KSPBuildResidual is used to compute the residuals
> > for the velocity components and pressure individually. Are these
> > the preconditioned residuals or the true residuals?
>
>
> The true residual.
>
>
> > And how to
> > get both?
> >
>
> Precondition it.
>
> Note that if you only want norms, GMRES will compute one (preconditioned by
> default).

Yes, only the norms. As I understand now, GMRES will compute the
norm of the total system residual [U,V,W,P], but I would like the
norms of [U], [V], [W] and [P] residuals separated. Based on
ex42, I got the impression that I have to use KSPBuildResidual.


dr. ir. Christiaan Klaij
CFD Researcher
Research & Development
E mailto:C.Klaij at marin.nl
T +31 317 49 33 44

MARIN
2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands
T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl


From jedbrown at mcs.anl.gov  Tue May 22 10:04:30 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 22 May 2012 10:04:30 -0500
Subject: [petsc-users] KSPBuildResidual in ex42.c
In-Reply-To: <A9AE64C525438E45B1D60F4D09AEAA9E062051@MAR160N1.marin.local>
References: <A9AE64C525438E45B1D60F4D09AEAA9E062051@MAR160N1.marin.local>
Message-ID: <CAM9tzSmbgq6uO=qGnPC6YOJqsfOsAjmMx8Ko=8dQQ0Af7M-Wqg@mail.gmail.com>

On Tue, May 22, 2012 at 9:54 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:

> Yes, only the norms. As I understand now, GMRES will compute the
> norm of the total system residual [U,V,W,P], but I would like the
> norms of [U], [V], [W] and [P] residuals separated. Based on
> ex42, I got the impression that I have to use KSPBuildResidual.
>

Yes, the cost of building the residual (check -log_summary to see how big
it is) is a reason why people often avoid GMRES if they need the full
residual. It's directly available with GCR, for example (which is otherwise
similar in work to FGMRES) and with most of the non-residual-minimizing
methods.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/97410adc/attachment.html>

From karpeev at mcs.anl.gov  Tue May 22 13:04:45 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Tue, 22 May 2012 13:04:45 -0500
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP50102871DC41171EC903A9CB1D0@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP350328939FD9E40C247F5CCCB1C0@phx.gbl>
	<CA+Y_9ULQfg=qet8mA8CeKAuknSH2jR3+yMwgARFue0QyZa+99w@mail.gmail.com>
	<BLU0-SMTP169C9E4B3677CEFF78F426ECB1D0@phx.gbl>
	<BLU0-SMTP191BA30F4023240B6508FBBCB1D0@phx.gbl>
	<BLU0-SMTP50102871DC41171EC903A9CB1D0@phx.gbl>
Message-ID: <CA+Y_9UL6GKa4LZZe8UR7w6O8mw5=-0jjtAmKAhZyqAbfhO9Geg@mail.gmail.com>

Hui,
There appears to be a bug in MatGetSubMatrices() which inserts submatrix
column indices in the same order
as they appeared in the original matrix.  If a column IS has an inversion
--  IS[2] = 8 and IS[6] = 4,
as in your case --  then the submatrix column index 2 will appear after the
submatrix column index 4, since that's
how they were ordered in the original matrix (4 < 8, and the column indices
were properly sorted). This violates
the ordering of column indices, which prevents binary search from finding
column index 2.  That results in an
apparent new nonzero insertion, which is in turn caught, since by default
no new nonzeros are allowed.

I'm looking into this, and I'm not sure that there is an easy workaround
for you at this time: if you sort the IS indices,
it will violate the implicit local-to-global submatrix mapping you are
using in your ModifySubMatrices().

Thanks.
Dmitry.

On Mon, May 21, 2012 at 2:33 PM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Hi,
>
> It might be unclear.  The error here is that the submat[1] does have a
> (1,2) entry,
> but MatSetValues at (1,2) gives an error message.  GASM and ASM give the
> same error.
>
> Thanks!
>
> On May 21, 2012, at 9:11 PM, Hui Zhang wrote:
>
> Sorry, I uses PCASMSetSortIndices for GASM. Now, I correct it and the
> output becomes the
> same as PCASM. See attached.
>
> <GASM_test.zip>
>
>
> On May 21, 2012, at 8:59 PM, Hui Zhang wrote:
>
> Dmitry,
>
> finally, I am able to make a simple program and reproduce the error. Since
> I had moved to
> petsc-dev (before I used petsc-3.2), the error is a little different.
>
> Download the program attached and run the run.sh.  You will see the output
> from PCASM.
> There is a submatrix (submat[1] on rank 0) which has a row like
>
>    row 1: (0, -1)  (1, 0 + 0.4 i) (4, -1)  (2, -1)
>
> Note that the column 2 is placed in the last and the column numbers are
> not in the normal order.
> Following that, we can also see the error message like
>
>    [0]PETSC ERROR: New nonzero at (1,2) caused a malloc!
>
> You can also try with GASM, use
>
>     mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1
>
> which partitions the domain to 1 * 2 and mapped to the two processors, and
> for each processor
> we further partition its local domain to 2 * 1 subdomains.
>
> Then, you will see that the row 1 of submat[1] on rank 1 becomes
>
>    row 1: (0, -1)  (1, 0 + 0.4 i) (3, -1)  (6, -1)
>
> which is totally wrong because from the IS for this overlapping subdomain
> and the big matrix A,
> row 1 should have non-zeros like the output from PCASM.
>
> I guess the problem is due to that we set PCASMSetSortIndices to FALSE and
> something goes wrong
> in asm.c and gasm.c.
>
> Thanks!
> Hui
>
>
> <GASM_test.zip>
>
>
>
>
>
> On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>> I have the following MatView output:
>>
>> row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4,
>> -0.517586 - 0.617823 i) (1, -1.03517)  (5, -0.508793)
>>
>> Note that there are two values for the column 1, is this normal?
>>
>> The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and
>> KSPGetOperators(subksp[0], ...).
>>
>
> This is odd. Can you explain a bit more what leads up to this, so we can
> try and reproduce the problem?
> Thanks.
> Dmitry.
>
>>
>> Thanks!
>>
>>
>>
>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>
>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and
>> recreate the matrices (including calling
>> your modification subroutine), but not the subdomains or the subdomain
>> solvers.
>> If you just want to modify the submatrices, you can call
>> PC(G)ASMGetSubmatrices() and modify the matrices it returns
>> (in the same order as the subdomains were set). That's a bit of a
>> hack, since you will essentially be modifying the PC's internal data
>> structures.  As long as you are careful, you should be okay, since you
>> already effectively have the same type of access to the submatrices through
>> the Modify callback.
>>
>> Dmitry.
>>
>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>
>>> I just have a question about reuse of PCASM or PCGASM.
>>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>>> Next for the same linear system (matrix and RHS), I just want PCASM
>>> modify the submatrices (PCSetModifySubmatrices) in a different way, using
>>> the same routine for modifying but with
>>> different user context for the modifying routine.
>>>
>>> What can I do for this task?  Currently, I destroy the KSP and
>>> re-construct it. I guess
>>> even for PCASM I can re-use it because the partition of subdomains
>>> remain the same.
>>>
>>> Thanks!
>>>
>>>
>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>
>>> Hui,
>>> There've been several changes to PCGASM ahead of the new release.
>>> Let me go back and see if it affected the convergence problem.
>>> Dmitry.
>>>
>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>>
>>>> Hi Dmitry,
>>>>
>>>> is there any news about PCGASM?
>>>>
>>>> thanks,
>>>> Hui
>>>>
>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>
>>>> Okay, thanks.
>>>> I'll take a look.
>>>>
>>>> Dmitry.
>>>>
>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com
>>>> > wrote:
>>>>
>>>>> For reference, my results are attached.
>>>>>
>>>>> asm1.txt for asm with 1 process,
>>>>> asm2.txt for asm with 2 processes,
>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers
>>>>> different from others)
>>>>> gasm2.txt for gasm with 2 processes
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> thank you,
>>>>> Hui
>>>>>
>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <
>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>
>>>>>>
>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <
>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>
>>>>>>> I have a new problem: the results from ASM and GASM are different
>>>>>>> and it seems
>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests
>>>>>>> are with
>>>>>>> each subdomain supported only by one subdomain. There are no
>>>>>>> problems when
>>>>>>> I did not modify submatrices.  But when I modify submatrices, there
>>>>>>> are problems
>>>>>>> with GASM but no problems with ASM.
>>>>>>>
>>>>>>> For example, I use two subdomains. In the first case each subdomain
>>>>>>> is supported by
>>>>>>> one processor and there seems no problem with GASM. But when I use
>>>>>>> run my program
>>>>>>> with only one proc. so that it supports both of the two subdomains,
>>>>>>> the iteration
>>>>>>> number is different from the first case and is much larger.  On the
>>>>>>> other hand
>>>>>>> ASM has no such problem.
>>>>>>>
>>>>>>
>>>>>> Are the solutions the same?
>>>>>> What problem are you solving?
>>>>>>
>>>>>>
>>>>>> Yes, the solutions are the same. That's why ASM gives the same
>>>>>> results with one or
>>>>>> two processors. But GASM did not.
>>>>>>
>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in
>>>>> the case of two domains per processor?
>>>>>
>>>>>> I'm solving the Helmholtz equation.  Maybe
>>>>>> I can prepare a simpler example to show this difference.
>>>>>>
>>>>> That would be helpful.
>>>>> Thanks.
>>>>>
>>>>> Dmitry.
>>>>>
>>>>>>
>>>>>>
>>>>>> Dmitry.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>
>>>>>>> You should be able to.
>>>>>>> This behavior is the same as in PCASM,
>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>
>>>>>>> Dmitry
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>
>>>>>>> Hi Dmitry,
>>>>>>>
>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes
>>>>>>> another question
>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the
>>>>>>> prototype
>>>>>>>
>>>>>>> func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>
>>>>>>> I think the coloumns from the parameter 'col' are always the same as
>>>>>>> the rows
>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only
>>>>>>> accepts
>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>
>>>>>>>
>>>>>>> As I tested, the row and col are always the same.
>>>>>>>
>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for
>>>>>>> the submat's
>>>>>>> in the above func()?
>>>>>>>
>>>>>>> thanks,
>>>>>>> Hui
>>>>>>>
>>>>>>>
>>>>>>> thanks,
>>>>>>> Hui
>>>>>>>
>>>>>>>
>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>
>>>>>>> Yes, that's right.
>>>>>>> There is no good way to help the user assemble the subdomains at the
>>>>>>> moment beyond the 2D stuff.
>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>
>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring"
>>>>>>> of indices to an array of ISs,
>>>>>>> each having the indices with the same color and the subcomm that
>>>>>>> supports that color. It is
>>>>>>> largely untested, though.  You could try using it and give us
>>>>>>> feedback on any problems you encounter.
>>>>>>>
>>>>>>> Dmitry.
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<mike.hui.zhang at hotmail.com>
>>>>>>> mike.hui.zhang at hotmail.com> wrote:
>>>>>>>
>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain
>>>>>>>> supported by
>>>>>>>> multiple processors, shall I always create the arguments 'is[s]'
>>>>>>>> and 'is_local[s]'
>>>>>>>> in a subcommunicator consisting of processors supporting the
>>>>>>>> subdomain 's'?
>>>>>>>>
>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Hui
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/e7977b50/attachment.html>

From karpeev at mcs.anl.gov  Tue May 22 13:14:15 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Tue, 22 May 2012 13:14:15 -0500
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
	<CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
	<BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
Message-ID: <CA+Y_9UJFNrdAwPcdoqvTEutqQPr7C-RrcsYLbcR-fJ-dgnDpxQ@mail.gmail.com>

It sounds like you need to write a new PC implementation, possibly reusing
pieces of PCASM.
Then you have control of what is being set up and when. Otherwise you are
essentially hacking
PCASM and using its internal subsolvers, which are only available after
PCSetUp().  PCCOMPOSITE
doesn't look like the right framework for this.

Dmitry.

On Tue, May 22, 2012 at 5:23 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Hi,
>
> I have got a new question.  I'm now constructing a PCComposite from PCASM
> and another PCKSP (coarse
> problem). And construction of PCKSP needs to use the subksp's of PCASM. So
> I need to PCSetUp on the
> PCASM, right?  But the KSP using PCComposite would setup PCASM again.  How
> can I avoid twice setup
> of PCASM ?
>
> Thanks,
> Hui
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/6f0cb7be/attachment.html>

From christian.staudt at ira.uka.de  Tue May 22 13:27:59 2012
From: christian.staudt at ira.uka.de (Christian Staudt)
Date: Tue, 22 May 2012 20:27:59 +0200
Subject: [petsc-users] =?iso-8859-1?q?=5Bpetsc4py=5D=A0Cannot_cast_array_d?=
 =?iso-8859-1?q?ata_from_dtype=28=27int64=27=29_to_dtype=28=27int32=27=29_?=
 =?iso-8859-1?q?according_to_the_rule_=27safe=27?=
Message-ID: <3210AB27-AF79-493E-8D11-6698BF3630CD@ira.uka.de>

Hi petsc4py users,

I am running into the following error:

>   File "/Users/cls/workspace/LAMG-Python/src/lamg/amg/setup.py", line 945, in buildInternal
>     Acc = A[c, c]
>   File "Mat.pyx", line 185, in petsc4py.PETSc.Mat.__getitem__ (src/petsc4py.PETSc.c:71407)
>   File "petscmat.pxi", line 862, in petsc4py.PETSc.mat_getitem (src/petsc4py.PETSc.c:22442)
>   File "petscmat.pxi", line 773, in petsc4py.PETSc.matgetvalues (src/petsc4py.PETSc.c:21226)
>   File "arraynpy.pxi", line 123, in petsc4py.PETSc.iarray_i (src/petsc4py.PETSc.c:5284)
>   File "arraynpy.pxi", line 117, in petsc4py.PETSc.iarray (src/petsc4py.PETSc.c:5192)
> TypeError: Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'

I am indexing the PETSc.Mat A with an array c ( numpy.ndarray with dtype('int64')).  I assume there is a simple solution (allowing the cast, making everything  32 or 64 bit...). Can you point me in the right direction? Thanks.

Chris


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/dad02e66/attachment-0001.html>

From mike.hui.zhang at hotmail.com  Tue May 22 14:39:16 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Tue, 22 May 2012 21:39:16 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CA+Y_9UL6GKa4LZZe8UR7w6O8mw5=-0jjtAmKAhZyqAbfhO9Geg@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP350328939FD9E40C247F5CCCB1C0@phx.gbl>
	<CA+Y_9ULQfg=qet8mA8CeKAuknSH2jR3+yMwgARFue0QyZa+99w@mail.gmail.com>
	<BLU0-SMTP169C9E4B3677CEFF78F426ECB1D0@phx.gbl>
	<BLU0-SMTP191BA30F4023240B6508FBBCB1D0@phx.gbl>
	<BLU0-SMTP50102871DC41171EC903A9CB1D0@phx.gbl>
	<CA+Y_9UL6GKa4LZZe8UR7w6O8mw5=-0jjtAmKAhZyqAbfhO9Geg@mail.gmail.com>
Message-ID: <BLU0-SMTP167BBB475C8E2E66FC12BE6CB020@phx.gbl>

Dmitry,

thanks! For the moment, I can use the sorted IS and I just need to pass through user context
some AO or ISLocalToGlobalMapping.

Hope you will get the unsorted IS case work soon!

Thanks!


On May 22, 2012, at 8:04 PM, Dmitry Karpeev wrote:

> Hui,
> There appears to be a bug in MatGetSubMatrices() which inserts submatrix column indices in the same order
> as they appeared in the original matrix.  If a column IS has an inversion --  IS[2] = 8 and IS[6] = 4, 
> as in your case --  then the submatrix column index 2 will appear after the submatrix column index 4, since that's 
> how they were ordered in the original matrix (4 < 8, and the column indices were properly sorted). This violates 
> the ordering of column indices, which prevents binary search from finding column index 2.  That results in an 
> apparent new nonzero insertion, which is in turn caught, since by default no new nonzeros are allowed.
> 
> I'm looking into this, and I'm not sure that there is an easy workaround for you at this time: if you sort the IS indices, 
> it will violate the implicit local-to-global submatrix mapping you are using in your ModifySubMatrices().
> 
> Thanks.
> Dmitry.
> 
> On Mon, May 21, 2012 at 2:33 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> Hi, 
> 
> It might be unclear.  The error here is that the submat[1] does have a (1,2) entry,
> but MatSetValues at (1,2) gives an error message.  GASM and ASM give the same error.
> 
> Thanks!
> 
> On May 21, 2012, at 9:11 PM, Hui Zhang wrote:
> 
>> Sorry, I uses PCASMSetSortIndices for GASM. Now, I correct it and the output becomes the
>> same as PCASM. See attached.
>> 
>> <GASM_test.zip>
>> 
>> 
>> On May 21, 2012, at 8:59 PM, Hui Zhang wrote:
>> 
>>> Dmitry,
>>> 
>>> finally, I am able to make a simple program and reproduce the error. Since I had moved to
>>> petsc-dev (before I used petsc-3.2), the error is a little different.
>>> 
>>> Download the program attached and run the run.sh.  You will see the output from PCASM.
>>> There is a submatrix (submat[1] on rank 0) which has a row like
>>> 
>>>    row 1: (0, -1)  (1, 0 + 0.4 i) (4, -1)  (2, -1) 
>>> 
>>> Note that the column 2 is placed in the last and the column numbers are not in the normal order.
>>> Following that, we can also see the error message like
>>> 
>>>    [0]PETSC ERROR: New nonzero at (1,2) caused a malloc!
>>> 
>>> You can also try with GASM, use
>>> 
>>>     mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1 
>>> 
>>> which partitions the domain to 1 * 2 and mapped to the two processors, and for each processor
>>> we further partition its local domain to 2 * 1 subdomains. 
>>> 
>>> Then, you will see that the row 1 of submat[1] on rank 1 becomes
>>> 
>>>    row 1: (0, -1)  (1, 0 + 0.4 i) (3, -1)  (6, -1) 
>>> 
>>> which is totally wrong because from the IS for this overlapping subdomain and the big matrix A,
>>> row 1 should have non-zeros like the output from PCASM.
>>> 
>>> I guess the problem is due to that we set PCASMSetSortIndices to FALSE and something goes wrong
>>> in asm.c and gasm.c.
>>> 
>>> Thanks!
>>> Hui
>>> 
>>> 
>>> <GASM_test.zip>
>>> 
>>> 
>>> 
>>> 
>>> 
>>>> On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>> I have the following MatView output: 
>>>> 
>>>> row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517)  (5, -0.508793) 
>>>> 
>>>> Note that there are two values for the column 1, is this normal?
>>>> 
>>>> The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).
>>>> 
>>>> This is odd. Can you explain a bit more what leads up to this, so we can try and reproduce the problem?
>>>> Thanks.
>>>> Dmitry.
>>>> 
>>>> Thanks!
>>>> 
>>>> 
>>>> 
>>>> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
>>>> 
>>>>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling 
>>>>> your modification subroutine), but not the subdomains or the subdomain solvers. 
>>>>> If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns
>>>>> (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures.  As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
>>>>> 
>>>>> Dmitry.
>>>>> 
>>>>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>> I just have a question about reuse of PCASM or PCGASM.
>>>>> Suppose I have seted up the PCASM and related KSP and I solved one time.
>>>>> Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with
>>>>> different user context for the modifying routine.  
>>>>> 
>>>>> What can I do for this task?  Currently, I destroy the KSP and re-construct it. I guess
>>>>> even for PCASM I can re-use it because the partition of subdomains remain the same.
>>>>> 
>>>>> Thanks!
>>>>> 
>>>>> 
>>>>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
>>>>> 
>>>>>> Hui,
>>>>>> There've been several changes to PCGASM ahead of the new release.
>>>>>> Let me go back and see if it affected the convergence problem.
>>>>>> Dmitry.
>>>>>> 
>>>>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>> Hi Dmitry,
>>>>>> 
>>>>>> is there any news about PCGASM? 
>>>>>> 
>>>>>> thanks,
>>>>>> Hui
>>>>>> 
>>>>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
>>>>>> 
>>>>>>> Okay, thanks.
>>>>>>> I'll take a look.
>>>>>>> 
>>>>>>> Dmitry.
>>>>>>> 
>>>>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>> For reference, my results are attached.
>>>>>>> 
>>>>>>> asm1.txt for asm with 1 process,
>>>>>>> asm2.txt for asm with 2 processes,
>>>>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different from others)
>>>>>>> gasm2.txt for gasm with 2 processes
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> thank you,
>>>>>>> Hui
>>>>>>> 
>>>>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>> 
>>>>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>> I have a new problem: the results from ASM and GASM are different and it seems
>>>>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are with 
>>>>>>>>> each subdomain supported only by one subdomain. There are no problems when
>>>>>>>>> I did not modify submatrices.  But when I modify submatrices, there are problems
>>>>>>>>> with GASM but no problems with ASM. 
>>>>>>>>> 
>>>>>>>>> For example, I use two subdomains. In the first case each subdomain is supported by
>>>>>>>>> one processor and there seems no problem with GASM. But when I use run my program 
>>>>>>>>> with only one proc. so that it supports both of the two subdomains, the iteration 
>>>>>>>>> number is different from the first case and is much larger.  On the other hand
>>>>>>>>> ASM has no such problem.
>>>>>>>>>  
>>>>>>>>> Are the solutions the same?
>>>>>>>>> What problem are you solving?
>>>>>>>> 
>>>>>>>> Yes, the solutions are the same. That's why ASM gives the same results with one or
>>>>>>>> two processors. But GASM did not.  
>>>>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
>>>>>>>> I'm solving the Helmholtz equation.  Maybe 
>>>>>>>> I can prepare a simpler example to show this difference.
>>>>>>>> That would be helpful.  
>>>>>>>> Thanks.
>>>>>>>> 
>>>>>>>> Dmitry. 
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Dmitry. 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
>>>>>>>>> 
>>>>>>>>>> You should be able to. 
>>>>>>>>>> This behavior is the same as in PCASM,
>>>>>>>>>> except in GASM the matrices live on subcommunicators.
>>>>>>>>>> I am in transit right now, but I can take a closer look in Friday.
>>>>>>>>>> 
>>>>>>>>>> Dmitry
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Dmitry,
>>>>>>>>>>>> 
>>>>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes another question
>>>>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the prototype
>>>>>>>>>>>> 
>>>>>>>>>>>>     func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
>>>>>>>>>>>> 
>>>>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as the rows 
>>>>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts 
>>>>>>>>>>>> index sets but not rows and columns. Has I misunderstood something?
>>>>>>>>>>> 
>>>>>>>>>>> As I tested, the row and col are always the same. 
>>>>>>>>>>> 
>>>>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's
>>>>>>>>>>> in the above func()?
>>>>>>>>>>> 
>>>>>>>>>>> thanks,
>>>>>>>>>>> Hui
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> thanks,
>>>>>>>>>>>> Hui
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Yes, that's right.
>>>>>>>>>>>>> There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff.
>>>>>>>>>>>>> It is expected that they are generated from mesh subdomains.
>>>>>>>>>>>>> Each IS does carry the subdomains subcomm.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs,
>>>>>>>>>>>>> each having the indices with the same color and the subcomm that supports that color. It is
>>>>>>>>>>>>> largely untested, though.  You could try using it and give us feedback on any problems you encounter.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Dmitry.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>>>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by
>>>>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]'
>>>>>>>>>>>>> in a subcommunicator consisting of processors supporting the subdomain 's'?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Hui
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/5fa8fe3c/attachment.html>

From mike.hui.zhang at hotmail.com  Tue May 22 14:43:11 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Tue, 22 May 2012 21:43:11 +0200
Subject: [petsc-users] PCGASMSetLocalSubdomains
In-Reply-To: <CA+Y_9UJFNrdAwPcdoqvTEutqQPr7C-RrcsYLbcR-fJ-dgnDpxQ@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
	<CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
	<BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
	<CA+Y_9UJFNrdAwPcdoqvTEutqQPr7C-RrcsYLbcR-fJ-dgnDpxQ@mail.gmail.com>
Message-ID: <BLU0-SMTP44DB6D2CF2B152D643F927CB020@phx.gbl>


On May 22, 2012, at 8:14 PM, Dmitry Karpeev wrote:

> It sounds like you need to write a new PC implementation, possibly reusing pieces of PCASM.

Thanks! But then it will be too much work, I think, and what I tried to avoid.

> Then you have control of what is being set up and when. Otherwise you are essentially hacking
> PCASM and using its internal subsolvers, which are only available after PCSetUp().

Yes. Maybe I should follow your advice. Thanks again!

>  PCCOMPOSITE doesn't look like the right framework for this.
> 
> Dmitry.
> 
> On Tue, May 22, 2012 at 5:23 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> Hi,
> 
> I have got a new question.  I'm now constructing a PCComposite from PCASM and another PCKSP (coarse
> problem). And construction of PCKSP needs to use the subksp's of PCASM. So I need to PCSetUp on the
> PCASM, right?  But the KSP using PCComposite would setup PCASM again.  How can I avoid twice setup
> of PCASM ?
> 
> Thanks,
> Hui
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/569d86f9/attachment-0001.html>

From andrew.spott at gmail.com  Tue May 22 20:45:55 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Tue, 22 May 2012 19:45:55 -0600
Subject: [petsc-users] How do you write/read to the .info file
Message-ID: <5A10F2D0-2ABF-4094-B79A-53AE3307CFD9@gmail.com>

When writing out a Mat in binary form, the .info file is created, but it usually empty, how do you read/write to it?

Thanks

-Andrew

From jedbrown at mcs.anl.gov  Tue May 22 21:01:19 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 22 May 2012 21:01:19 -0500
Subject: [petsc-users] How do you write/read to the .info file
In-Reply-To: <5A10F2D0-2ABF-4094-B79A-53AE3307CFD9@gmail.com>
References: <5A10F2D0-2ABF-4094-B79A-53AE3307CFD9@gmail.com>
Message-ID: <CAM9tzSkZ-yN2ZRfxKSdC2ckFo_1HGSL-sU+P841_=6yrOJ9KWA@mail.gmail.com>

On Tue, May 22, 2012 at 8:45 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> When writing out a Mat in binary form, the .info file is created, but it
> usually empty, how do you read/write to it?
>

They are written and read automatically by the PETSc binary viewers. You
shouldn't need to look at them.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/98c913f7/attachment.html>

From abhyshr at mcs.anl.gov  Tue May 22 22:31:52 2012
From: abhyshr at mcs.anl.gov (Shri)
Date: Tue, 22 May 2012 22:31:52 -0500 (CDT)
Subject: [petsc-users] How do you write/read to the .info file
In-Reply-To: <CAM9tzSkZ-yN2ZRfxKSdC2ckFo_1HGSL-sU+P841_=6yrOJ9KWA@mail.gmail.com>
Message-ID: <1862596027.36968.1337743912223.JavaMail.root@zimbra-mb2.anl.gov>

You can also use the option -viewer_binary_skip_info to have PETSc not create the .info file. ----- Original Message -----
> On Tue, May 22, 2012 at 8:45 PM, Andrew Spott < andrew.spott at gmail.com
> > wrote:
> > When writing out a Mat in binary form, the .info file is created,
> > but
> > it usually empty, how do you read/write to it?
> They are written and read automatically by the PETSc binary viewers.
> You shouldn't need to look at them.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/117f0c97/attachment.html>

From andrew.spott at gmail.com  Tue May 22 22:32:53 2012
From: andrew.spott at gmail.com (Andrew Spott)
Date: Tue, 22 May 2012 21:32:53 -0600
Subject: [petsc-users] How do you write/read to the .info file
In-Reply-To: <1862596027.36968.1337743912223.JavaMail.root@zimbra-mb2.anl.gov>
References: <CAM9tzSkZ-yN2ZRfxKSdC2ckFo_1HGSL-sU+P841_=6yrOJ9KWA@mail.gmail.com>
	<1862596027.36968.1337743912223.JavaMail.root@zimbra-mb2.anl.gov>
Message-ID: <CAPbJa2gLpfKJXRB4xd_=bj77q1CV4JJRa8u-2OJjzO=wquS2pQ@mail.gmail.com>

What purpose does the info file have?
On May 22, 2012 9:31 PM, "Shri" <abhyshr at mcs.anl.gov> wrote:

> You can also use the option -viewer_binary_skip_info to have PETSc not
> create the .info file.
>
> ------------------------------
>
> On Tue, May 22, 2012 at 8:45 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>
>> When writing out a Mat in binary form, the .info file is created, but it
>> usually empty, how do you read/write to it?
>>
>
> They are written and read automatically by the PETSc binary viewers. You
> shouldn't need to look at them.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/3ebf61e7/attachment.html>

From knepley at gmail.com  Tue May 22 22:33:31 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 22 May 2012 23:33:31 -0400
Subject: [petsc-users] How do you write/read to the .info file
In-Reply-To: <CAPbJa2gLpfKJXRB4xd_=bj77q1CV4JJRa8u-2OJjzO=wquS2pQ@mail.gmail.com>
References: <CAM9tzSkZ-yN2ZRfxKSdC2ckFo_1HGSL-sU+P841_=6yrOJ9KWA@mail.gmail.com>
	<1862596027.36968.1337743912223.JavaMail.root@zimbra-mb2.anl.gov>
	<CAPbJa2gLpfKJXRB4xd_=bj77q1CV4JJRa8u-2OJjzO=wquS2pQ@mail.gmail.com>
Message-ID: <CAMYG4G=B6gx+xAap5VBhB5RS950w6v4vnkKzOJC2_tRcch25jg@mail.gmail.com>

On Tue, May 22, 2012 at 11:32 PM, Andrew Spott <andrew.spott at gmail.com>wrote:

> What purpose does the info file have?
>
Caching metadata that might optimize the load.

   Matt


> On May 22, 2012 9:31 PM, "Shri" <abhyshr at mcs.anl.gov> wrote:
>
>> You can also use the option -viewer_binary_skip_info to have PETSc not
>> create the .info file.
>>
>> ------------------------------
>>
>> On Tue, May 22, 2012 at 8:45 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
>>
>>> When writing out a Mat in binary form, the .info file is created, but it
>>> usually empty, how do you read/write to it?
>>>
>>
>> They are written and read automatically by the PETSc binary viewers. You
>> shouldn't need to look at them.
>>
>>
>>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120522/f1a323ed/attachment.html>

From renzhengyong at gmail.com  Wed May 23 04:01:17 2012
From: renzhengyong at gmail.com (RenZhengYong)
Date: Wed, 23 May 2012 11:01:17 +0200
Subject: [petsc-users] using metis in petsc
Message-ID: <CA+OnyrAw2UD+jYu8bJtAH-WxtP1NKHNj-FNF3npaA+yyPCfoPg@mail.gmail.com>

Dear PETscs,

I want to use the metis in my code and calling petsc to solve the system of
linear equation.
I find that there is a libmetis.a included in petsc. So, I want to use the
metis routins via petsc.

However, when I included:

include ${PETSC_DIR}/conf/variables
include ${PETSC_DIR}/conf/rules

in my makefile.

And the compiler icpc told me that:
catastrophic error: cannot open source file "metis.h".

My question is where is the "metis.h" in petsc?

I actually installed the latest version of metis 5.0 in my disk.  And I
find the interfaces of this 5.0 version metis and the older version metis
in petsc-3.2-p7 are quite different. So, I want to use the metis in the
petsc-3.2-p7.

Best wishes
Zhengyong
-- 
Zhengyong Ren
AUG Group, Institute of Geophysics
Department of Geosciences, ETH Zurich
NO H 47 Sonneggstrasse 5
CH-8092, Z?rich, Switzerland
Tel: +41 44 633 37561
e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch
Gmail: renzhengyong at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120523/a4b90dce/attachment.html>

From C.Klaij at marin.nl  Wed May 23 06:08:44 2012
From: C.Klaij at marin.nl (Klaij, Christiaan)
Date: Wed, 23 May 2012 11:08:44 +0000
Subject: [petsc-users] KSPBuildResidual in ex42.c
Message-ID: <A9AE64C525438E45B1D60F4D09AEAA9E0620D9@MAR160N1.marin.local>

> > Yes, only the norms. As I understand now, GMRES will compute the
> > norm of the total system residual [U,V,W,P], but I would like the
> > norms of [U], [V], [W] and [P] residuals separated. Based on
> > ex42, I got the impression that I have to use KSPBuildResidual.
> >
>
> Yes, the cost of building the residual (check -log_summary to see how big
> it is) is a reason why people often avoid GMRES if they need the full
> residual. It's directly available with GCR, for example (which is otherwise
> similar in work to FGMRES) and with most of the non-residual-minimizing
> methods.

I find GCR to be less robust than FGMRES, but ok, suppose I switch
to GCR, how would I get the residual vector?


dr. ir. Christiaan Klaij
CFD Researcher
Research & Development
E mailto:C.Klaij at marin.nl
T +31 317 49 33 44

MARIN
2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands
T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl


From knepley at gmail.com  Wed May 23 06:42:25 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 23 May 2012 07:42:25 -0400
Subject: [petsc-users] using metis in petsc
In-Reply-To: <CA+OnyrAw2UD+jYu8bJtAH-WxtP1NKHNj-FNF3npaA+yyPCfoPg@mail.gmail.com>
References: <CA+OnyrAw2UD+jYu8bJtAH-WxtP1NKHNj-FNF3npaA+yyPCfoPg@mail.gmail.com>
Message-ID: <CAMYG4G=VkGoN5fyTLYL6jWtU-fEoK-_-jtC8nCyzQn9tbN9agg@mail.gmail.com>

On Wed, May 23, 2012 at 5:01 AM, RenZhengYong <renzhengyong at gmail.com>wrote:

> Dear PETscs,
>
> I want to use the metis in my code and calling petsc to solve the system
> of linear equation.
> I find that there is a libmetis.a included in petsc. So, I want to use the
> metis routins via petsc.
>
> However, when I included:
>
> include ${PETSC_DIR}/conf/variables
> include ${PETSC_DIR}/conf/rules
>
> in my makefile.
>
> And the compiler icpc told me that:
> catastrophic error: cannot open source file "metis.h".
>
> My question is where is the "metis.h" in petsc?
>
> I actually installed the latest version of metis 5.0 in my disk.  And I
> find the interfaces of this 5.0 version metis and the older version metis
> in petsc-3.2-p7 are quite different. So, I want to use the metis in the
> petsc-3.2-p7.
>

If you want to use Metis 5.0, upgrade to petsc-dev. We are frozen for
release, so its the same as 3.3 now anyway.

   Matt


> Best wishes
> Zhengyong
> --
> Zhengyong Ren
> AUG Group, Institute of Geophysics
> Department of Geosciences, ETH Zurich
> NO H 47 Sonneggstrasse 5
> CH-8092, Z?rich, Switzerland
> Tel: +41 44 633 37561
> e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch
> Gmail: renzhengyong at gmail.com
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120523/67f8dc88/attachment-0001.html>

From jedbrown at mcs.anl.gov  Wed May 23 07:06:50 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 23 May 2012 07:06:50 -0500
Subject: [petsc-users] KSPBuildResidual in ex42.c
In-Reply-To: <A9AE64C525438E45B1D60F4D09AEAA9E0620D9@MAR160N1.marin.local>
References: <A9AE64C525438E45B1D60F4D09AEAA9E0620D9@MAR160N1.marin.local>
Message-ID: <CAM9tzSmFQ-JqWM6Z387Xc=E92RuMVG6JC1dezaXMfrLVOUiSdA@mail.gmail.com>

On Wed, May 23, 2012 at 6:08 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:

> > > Yes, only the norms. As I understand now, GMRES will compute the
> > > norm of the total system residual [U,V,W,P], but I would like the
> > > norms of [U], [V], [W] and [P] residuals separated. Based on
> > > ex42, I got the impression that I have to use KSPBuildResidual.
> > >
> >
> > Yes, the cost of building the residual (check -log_summary to see how big
> > it is) is a reason why people often avoid GMRES if they need the full
> > residual. It's directly available with GCR, for example (which is
> otherwise
> > similar in work to FGMRES) and with most of the non-residual-minimizing
> > methods.
>
> I find GCR to be less robust than FGMRES,


Right, it can break down in some cases that FGMRES does not.


> but ok, suppose I switch
> to GCR, how would I get the residual vector?


KSPBuildResidual(), but it's trivially fast with GCR.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120523/8075eb10/attachment.html>

From renzhengyong at gmail.com  Wed May 23 07:14:39 2012
From: renzhengyong at gmail.com (RenZhengYong)
Date: Wed, 23 May 2012 14:14:39 +0200
Subject: [petsc-users] using metis in petsc
In-Reply-To: <CAMYG4G=VkGoN5fyTLYL6jWtU-fEoK-_-jtC8nCyzQn9tbN9agg@mail.gmail.com>
References: <CA+OnyrAw2UD+jYu8bJtAH-WxtP1NKHNj-FNF3npaA+yyPCfoPg@mail.gmail.com>
	<CAMYG4G=VkGoN5fyTLYL6jWtU-fEoK-_-jtC8nCyzQn9tbN9agg@mail.gmail.com>
Message-ID: <CA+OnyrDG6Gov3BT6G9+jSSFESi6BoUOGwa3iFSxoP3_sV_=BPQ@mail.gmail.com>

Hi, Matt.
Cool.
Thanks a lot for your great helps as usual.
Best wishes
Zhengyong


On Wed, May 23, 2012 at 1:42 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, May 23, 2012 at 5:01 AM, RenZhengYong <renzhengyong at gmail.com>wrote:
>
>> Dear PETscs,
>>
>> I want to use the metis in my code and calling petsc to solve the system
>> of linear equation.
>> I find that there is a libmetis.a included in petsc. So, I want to use
>> the metis routins via petsc.
>>
>> However, when I included:
>>
>> include ${PETSC_DIR}/conf/variables
>> include ${PETSC_DIR}/conf/rules
>>
>> in my makefile.
>>
>> And the compiler icpc told me that:
>> catastrophic error: cannot open source file "metis.h".
>>
>> My question is where is the "metis.h" in petsc?
>>
>> I actually installed the latest version of metis 5.0 in my disk.  And I
>> find the interfaces of this 5.0 version metis and the older version metis
>> in petsc-3.2-p7 are quite different. So, I want to use the metis in the
>> petsc-3.2-p7.
>>
>
> If you want to use Metis 5.0, upgrade to petsc-dev. We are frozen for
> release, so its the same as 3.3 now anyway.
>
>    Matt
>
>
>> Best wishes
>> Zhengyong
>> --
>> Zhengyong Ren
>> AUG Group, Institute of Geophysics
>> Department of Geosciences, ETH Zurich
>> NO H 47 Sonneggstrasse 5
>> CH-8092, Z?rich, Switzerland
>> Tel: +41 44 633 37561
>> e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch
>> Gmail: renzhengyong at gmail.com
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
Zhengyong Ren
AUG Group, Institute of Geophysics
Department of Geosciences, ETH Zurich
NO H 47 Sonneggstrasse 5
CH-8092, Z?rich, Switzerland
Tel: +41 44 633 37561
e-mail: zhengyong.ren at aug.ig.erdw.ethz.ch
Gmail: renzhengyong at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120523/d35cbff3/attachment.html>

From balay at mcs.anl.gov  Wed May 23 09:32:11 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 23 May 2012 09:32:11 -0500 (CDT)
Subject: [petsc-users] How do you write/read to the .info file
In-Reply-To: <CAMYG4G=B6gx+xAap5VBhB5RS950w6v4vnkKzOJC2_tRcch25jg@mail.gmail.com>
References: <CAM9tzSkZ-yN2ZRfxKSdC2ckFo_1HGSL-sU+P841_=6yrOJ9KWA@mail.gmail.com>
	<1862596027.36968.1337743912223.JavaMail.root@zimbra-mb2.anl.gov>
	<CAPbJa2gLpfKJXRB4xd_=bj77q1CV4JJRa8u-2OJjzO=wquS2pQ@mail.gmail.com>
	<CAMYG4G=B6gx+xAap5VBhB5RS950w6v4vnkKzOJC2_tRcch25jg@mail.gmail.com>
Message-ID: <alpine.LFD.2.02.1205230931560.15733@asterix>

On Tue, 22 May 2012, Matthew Knepley wrote:

> On Tue, May 22, 2012 at 11:32 PM, Andrew Spott <andrew.spott at gmail.com>wrote:
> 
> > What purpose does the info file have?
> >
> Caching metadata that might optimize the load.

for eg:

[petsc:~/datafiles/matrices] petsc> cat arco6.info
-matload_block_size 3
[petsc:~/datafiles/matrices] petsc> 


Satish

From ildar.kk at gmail.com  Wed May 23 16:15:06 2012
From: ildar.kk at gmail.com (Ildar Rakhmanov)
Date: Wed, 23 May 2012 17:15:06 -0400
Subject: [petsc-users] lsqr usage
Message-ID: <CAO8ENUR2r8caDnaKEz0uBFRNWjNKOWKBwobF2a6dSh=_53aJbg@mail.gmail.com>

Hi,
I seems I am missing something. I want to use KSPLSQR.
I set up lsqr.c (see attachment)
it uses 3x2 matrix.

The results is

ildar at ildar:~/lsqr> ./lsqr
Matrix A after construction
row 0: (0, 1)  (1, 0)
row 1: (0, 0)  (1, 1)
row 2: (0, 1)  (1, 1)
u
1
1
b
0
0
0
b
1
1
2
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Must be square matrix, rows 3 columns 2!
[0]PETSC ERROR:
------------------------------------------------------------------------



Could any one point out how to correctly use LSQR solver?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120523/a8bd9a93/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lsqr.c
Type: text/x-csrc
Size: 5622 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120523/a8bd9a93/attachment.c>

From bsmith at mcs.anl.gov  Wed May 23 16:29:33 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 23 May 2012 16:29:33 -0500
Subject: [petsc-users] lsqr usage
In-Reply-To: <CAO8ENUR2r8caDnaKEz0uBFRNWjNKOWKBwobF2a6dSh=_53aJbg@mail.gmail.com>
References: <CAO8ENUR2r8caDnaKEz0uBFRNWjNKOWKBwobF2a6dSh=_53aJbg@mail.gmail.com>
Message-ID: <32BE3EE3-1D37-4152-9E06-F5B24B9011B3@mcs.anl.gov>


  You cut and threw away the most important part of the error message. Where is the stack trace? Are we to guess where the error message was triggered? Please send the entire error message.

   Barry

On May 23, 2012, at 4:15 PM, Ildar Rakhmanov wrote:

> Hi, 
> I seems I am missing something. I want to use KSPLSQR.
> I set up lsqr.c (see attachment)
> it uses 3x2 matrix.
> 
> The results is
> 
> ildar at ildar:~/lsqr> ./lsqr 
> Matrix A after construction
> row 0: (0, 1)  (1, 0) 
> row 1: (0, 0)  (1, 1) 
> row 2: (0, 1)  (1, 1) 
> u 
> 1
> 1
> b 
> 0
> 0
> 0
> b 
> 1
> 1
> 2
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Must be square matrix, rows 3 columns 2!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> 
> 
> 
> Could any one point out how to correctly use LSQR solver?
> 
> <lsqr.c>


From ildar.kk at gmail.com  Wed May 23 16:58:03 2012
From: ildar.kk at gmail.com (Ildar Rakhmanov)
Date: Wed, 23 May 2012 17:58:03 -0400
Subject: [petsc-users] lsqr usage
In-Reply-To: <32BE3EE3-1D37-4152-9E06-F5B24B9011B3@mcs.anl.gov>
References: <CAO8ENUR2r8caDnaKEz0uBFRNWjNKOWKBwobF2a6dSh=_53aJbg@mail.gmail.com>
	<32BE3EE3-1D37-4152-9E06-F5B24B9011B3@mcs.anl.gov>
Message-ID: <CAO8ENURJ0oYiPRZwRsA8grtwhd645ybyDnH2LtSFF6rMvL6QMQ@mail.gmail.com>

Here is full message:

Matrix A after construction
row 0: (0, 1)  (1, 0)
row 1: (0, 0)  (1, 1)
row 2: (0, 1)  (1, 1)
u
1
1
b
0
0
0
b
1
1
2
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Must be square matrix, rows 3 columns 2!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48
CDT 2011
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./lsqr on a linux-gnu named ildar by ildar Wed May 23
17:13:44 2012
[0]PETSC ERROR: Libraries linked from /home/ildar/pbsm/lib
[0]PETSC ERROR: Configure run at Sat Mar  3 12:01:22 2012
[0]PETSC ERROR: Configure options --prefix=/home/ildar/pbsm
--download-hypre=1 --download-spooles=1 --download-plapack=1
--download-spai=1 --download-blacs=1 --download-triangle=1
--download-f-blas-lapack=1 --download-umfpack=1 --download-sowing=1
--download-c2html=1 --download-superlu_dist=1 --download-parmetis=1
--download-scalapack=1 --download-superlu=1 --download-mumps=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: MatGetOrdering() line 223 in src/mat/order/sorder.c
[0]PETSC ERROR: PCSetUp_ILU() line 194 in src/ksp/pc/impls/factor/ilu/ilu.c
[0]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: main() line 109 in "unknowndirectory/"lsqr.c
application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0




On Wed, May 23, 2012 at 5:29 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>  You cut and threw away the most important part of the error message.
> Where is the stack trace? Are we to guess where the error message was
> triggered? Please send the entire error message.
>
>   Barry
>
> On May 23, 2012, at 4:15 PM, Ildar Rakhmanov wrote:
>
> > Hi,
> > I seems I am missing something. I want to use KSPLSQR.
> > I set up lsqr.c (see attachment)
> > it uses 3x2 matrix.
> >
> > The results is
> >
> > ildar at ildar:~/lsqr> ./lsqr
> > Matrix A after construction
> > row 0: (0, 1)  (1, 0)
> > row 1: (0, 0)  (1, 1)
> > row 2: (0, 1)  (1, 1)
> > u
> > 1
> > 1
> > b
> > 0
> > 0
> > 0
> > b
> > 1
> > 1
> > 2
> > [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [0]PETSC ERROR: Invalid argument!
> > [0]PETSC ERROR: Must be square matrix, rows 3 columns 2!
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> >
> >
> >
> > Could any one point out how to correctly use LSQR solver?
> >
> > <lsqr.c>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120523/c26b5342/attachment.html>

From bsmith at mcs.anl.gov  Wed May 23 17:29:39 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 23 May 2012 17:29:39 -0500
Subject: [petsc-users] lsqr usage
In-Reply-To: <CAO8ENURJ0oYiPRZwRsA8grtwhd645ybyDnH2LtSFF6rMvL6QMQ@mail.gmail.com>
References: <CAO8ENUR2r8caDnaKEz0uBFRNWjNKOWKBwobF2a6dSh=_53aJbg@mail.gmail.com>
	<32BE3EE3-1D37-4152-9E06-F5B24B9011B3@mcs.anl.gov>
	<CAO8ENURJ0oYiPRZwRsA8grtwhd645ybyDnH2LtSFF6rMvL6QMQ@mail.gmail.com>
Message-ID: <25E96D64-6588-4001-B797-1C04763D1F9A@mcs.anl.gov>


  You are trying the default preconditioner ILU(0) which requires a square matrix.  Run with -pc_type none    or see the manual page

KSPLSQR - This implements LSQR

   Options Database Keys:
+   -ksp_lsqr_set_standard_error  - Set Standard Error Estimates of Solution see KSPLSQRSetStandardErrorVec()
.   -ksp_lsqr_monitor - Monitor residual norm and norm of residual of normal equations
-   see KSPSolve()

   Level: beginner

   Notes:  
     This varient, when applied with no preconditioning is identical to the original algorithm in exact arithematic; however, in practice, with no preconditioning
     due to inexact arithematic, it can converge differently. Hence when no preconditioner is used (PCType PCNONE) it automatically reverts to the original algorithm.

     With the PETSc built-in preconditioners, such as ICC, one should call KSPSetOperators(ksp,A,A'*A,...) since the preconditioner needs to work 
     for the normal equations A'*A.

     Supports only left preconditioning.

   References:The original unpreconditioned algorithm can be found in Paige and Saunders, ACM Transactions on Mathematical Software, Vol 8, pp 43-71, 1982. 
     In exact arithmetic the LSQR method (with no preconditioning) is identical to the KSPCG algorithm applied to the normal equations.
     The preconditioned varient was implemented by Bas van't Hof and is essentially a left preconditioning for the Normal Equations. It appears the implementation with preconditioner
     track the true norm of the residual and uses that in the convergence test.

   Developer Notes: How is this related to the KSPCGNE implementation? One difference is that KSPCGNE applies
            the preconditioner transpose times the preconditioner,  so one does not need to pass A'*A as the third argument to KSPSetOperators().
            

   For least squares problems without a zero to A*x = b, there are additional convergence tests for the residual of the normal equations, A'*(b - Ax), see KSPLSQRDefaultConverged()

.seealso:  KSPCreate(), KSPSetType(), KSPType (for list of available types), KSP, KSPLSQRDefaultConverged()



   Barry

On May 23, 2012, at 4:58 PM, Ildar Rakhmanov wrote:

> Here is full message:
> 
> Matrix A after construction
> row 0: (0, 1)  (1, 0) 
> row 1: (0, 0)  (1, 1) 
> row 2: (0, 1)  (1, 1) 
> u 
> 1
> 1
> b 
> 0
> 0
> 0
> b 
> 1
> 1
> 2
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Invalid argument!
> [0]PETSC ERROR: Must be square matrix, rows 3 columns 2!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./lsqr on a linux-gnu named ildar by ildar Wed May 23 17:13:44 2012
> [0]PETSC ERROR: Libraries linked from /home/ildar/pbsm/lib
> [0]PETSC ERROR: Configure run at Sat Mar  3 12:01:22 2012
> [0]PETSC ERROR: Configure options --prefix=/home/ildar/pbsm --download-hypre=1 --download-spooles=1 --download-plapack=1 --download-spai=1 --download-blacs=1 --download-triangle=1 --download-f-blas-lapack=1 --download-umfpack=1 --download-sowing=1 --download-c2html=1 --download-superlu_dist=1 --download-parmetis=1 --download-scalapack=1 --download-superlu=1 --download-mumps=1
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: MatGetOrdering() line 223 in src/mat/order/sorder.c
> [0]PETSC ERROR: PCSetUp_ILU() line 194 in src/ksp/pc/impls/factor/ilu/ilu.c
> [0]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: main() line 109 in "unknowndirectory/"lsqr.c
> application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0
> 
> 
> 
> 
> On Wed, May 23, 2012 at 5:29 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>  You cut and threw away the most important part of the error message. Where is the stack trace? Are we to guess where the error message was triggered? Please send the entire error message.
> 
>   Barry
> 
> On May 23, 2012, at 4:15 PM, Ildar Rakhmanov wrote:
> 
> > Hi,
> > I seems I am missing something. I want to use KSPLSQR.
> > I set up lsqr.c (see attachment)
> > it uses 3x2 matrix.
> >
> > The results is
> >
> > ildar at ildar:~/lsqr> ./lsqr
> > Matrix A after construction
> > row 0: (0, 1)  (1, 0)
> > row 1: (0, 0)  (1, 1)
> > row 2: (0, 1)  (1, 1)
> > u
> > 1
> > 1
> > b
> > 0
> > 0
> > 0
> > b
> > 1
> > 1
> > 2
> > [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> > [0]PETSC ERROR: Invalid argument!
> > [0]PETSC ERROR: Must be square matrix, rows 3 columns 2!
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> >
> >
> >
> > Could any one point out how to correctly use LSQR solver?
> >
> > <lsqr.c>
> 
> 


From ildar.kk at gmail.com  Wed May 23 17:46:14 2012
From: ildar.kk at gmail.com (Ildar Rakhmanov)
Date: Wed, 23 May 2012 18:46:14 -0400
Subject: [petsc-users] lsqr usage
In-Reply-To: <25E96D64-6588-4001-B797-1C04763D1F9A@mcs.anl.gov>
References: <CAO8ENUR2r8caDnaKEz0uBFRNWjNKOWKBwobF2a6dSh=_53aJbg@mail.gmail.com>
	<32BE3EE3-1D37-4152-9E06-F5B24B9011B3@mcs.anl.gov>
	<CAO8ENURJ0oYiPRZwRsA8grtwhd645ybyDnH2LtSFF6rMvL6QMQ@mail.gmail.com>
	<25E96D64-6588-4001-B797-1C04763D1F9A@mcs.anl.gov>
Message-ID: <CAO8ENUT-9AD3ip9Cf84RkXgskWTgF9_jPTK=QkCTf8tAp1-2Ww@mail.gmail.com>

Great, thank you.

Adding this

ierr = KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);
  ierr = KSPSetType(ksp,KSPLSQR);//CHKERRQ(ierr);
  ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr);
  ierr = PCSetType(pc,PCNONE);CHKERRQ(ierr);

solved the problem

On Wed, May 23, 2012 at 6:29 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>  You are trying the default preconditioner ILU(0) which requires a square
> matrix.  Run with -pc_type none    or see the manual page
>
> KSPLSQR - This implements LSQR
>
>   Options Database Keys:
> +   -ksp_lsqr_set_standard_error  - Set Standard Error Estimates of
> Solution see KSPLSQRSetStandardErrorVec()
> .   -ksp_lsqr_monitor - Monitor residual norm and norm of residual of
> normal equations
> -   see KSPSolve()
>
>   Level: beginner
>
>   Notes:
>     This varient, when applied with no preconditioning is identical to the
> original algorithm in exact arithematic; however, in practice, with no
> preconditioning
>     due to inexact arithematic, it can converge differently. Hence when no
> preconditioner is used (PCType PCNONE) it automatically reverts to the
> original algorithm.
>
>     With the PETSc built-in preconditioners, such as ICC, one should call
> KSPSetOperators(ksp,A,A'*A,...) since the preconditioner needs to work
>     for the normal equations A'*A.
>
>     Supports only left preconditioning.
>
>   References:The original unpreconditioned algorithm can be found in Paige
> and Saunders, ACM Transactions on Mathematical Software, Vol 8, pp 43-71,
> 1982.
>     In exact arithmetic the LSQR method (with no preconditioning) is
> identical to the KSPCG algorithm applied to the normal equations.
>     The preconditioned varient was implemented by Bas van't Hof and is
> essentially a left preconditioning for the Normal Equations. It appears the
> implementation with preconditioner
>     track the true norm of the residual and uses that in the convergence
> test.
>
>   Developer Notes: How is this related to the KSPCGNE implementation? One
> difference is that KSPCGNE applies
>            the preconditioner transpose times the preconditioner,  so one
> does not need to pass A'*A as the third argument to KSPSetOperators().
>
>
>   For least squares problems without a zero to A*x = b, there are
> additional convergence tests for the residual of the normal equations,
> A'*(b - Ax), see KSPLSQRDefaultConverged()
>
> .seealso:  KSPCreate(), KSPSetType(), KSPType (for list of available
> types), KSP, KSPLSQRDefaultConverged()
>
>
>
>   Barry
>
> On May 23, 2012, at 4:58 PM, Ildar Rakhmanov wrote:
>
> > Here is full message:
> >
> > Matrix A after construction
> > row 0: (0, 1)  (1, 0)
> > row 1: (0, 0)  (1, 1)
> > row 2: (0, 1)  (1, 1)
> > u
> > 1
> > 1
> > b
> > 0
> > 0
> > 0
> > b
> > 1
> > 1
> > 2
> > [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [0]PETSC ERROR: Invalid argument!
> > [0]PETSC ERROR: Must be square matrix, rows 3 columns 2!
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17
> 13:37:48 CDT 2011
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: ./lsqr on a linux-gnu named ildar by ildar Wed May 23
> 17:13:44 2012
> > [0]PETSC ERROR: Libraries linked from /home/ildar/pbsm/lib
> > [0]PETSC ERROR: Configure run at Sat Mar  3 12:01:22 2012
> > [0]PETSC ERROR: Configure options --prefix=/home/ildar/pbsm
> --download-hypre=1 --download-spooles=1 --download-plapack=1
> --download-spai=1 --download-blacs=1 --download-triangle=1
> --download-f-blas-lapack=1 --download-umfpack=1 --download-sowing=1
> --download-c2html=1 --download-superlu_dist=1 --download-parmetis=1
> --download-scalapack=1 --download-superlu=1 --download-mumps=1
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: MatGetOrdering() line 223 in src/mat/order/sorder.c
> > [0]PETSC ERROR: PCSetUp_ILU() line 194 in
> src/ksp/pc/impls/factor/ilu/ilu.c
> > [0]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c
> > [0]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: KSPSolve() line 353 in src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: main() line 109 in "unknowndirectory/"lsqr.c
> > application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0
> >
> >
> >
> >
> > On Wed, May 23, 2012 at 5:29 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >  You cut and threw away the most important part of the error message.
> Where is the stack trace? Are we to guess where the error message was
> triggered? Please send the entire error message.
> >
> >   Barry
> >
> > On May 23, 2012, at 4:15 PM, Ildar Rakhmanov wrote:
> >
> > > Hi,
> > > I seems I am missing something. I want to use KSPLSQR.
> > > I set up lsqr.c (see attachment)
> > > it uses 3x2 matrix.
> > >
> > > The results is
> > >
> > > ildar at ildar:~/lsqr> ./lsqr
> > > Matrix A after construction
> > > row 0: (0, 1)  (1, 0)
> > > row 1: (0, 0)  (1, 1)
> > > row 2: (0, 1)  (1, 1)
> > > u
> > > 1
> > > 1
> > > b
> > > 0
> > > 0
> > > 0
> > > b
> > > 1
> > > 1
> > > 2
> > > [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > > [0]PETSC ERROR: Invalid argument!
> > > [0]PETSC ERROR: Must be square matrix, rows 3 columns 2!
> > > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > >
> > >
> > >
> > > Could any one point out how to correctly use LSQR solver?
> > >
> > > <lsqr.c>
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120523/2be144b4/attachment-0001.html>

From coco at dmi.unict.it  Wed May 23 18:04:56 2012
From: coco at dmi.unict.it (coco at dmi.unict.it)
Date: Thu, 24 May 2012 01:04:56 +0200
Subject: [petsc-users] number of pre- and post- smoothing in multigrid as a
	preconditioner
Message-ID: <20120524010456.Horde.I1kaUuph4B9PvW0YaN4gN_A@mbox.dmi.unict.it>

Dear all,

I am using multigrid as a preconditioner and I noticed that if I try  
to perform 2 or more pre (or post) smoothing I obtain the same results  
as I perform just 1 smoothing. The ksp_view output is the following:

[...]

Down solver (pre-smoother) on level 1 -------------------------------
     KSP Object:    (mg_levels_1_)     1 MPI processes
       type: richardson
         Richardson: damping factor=1
       maximum iterations=2, initial guess is zero
       tolerances:  relative=0, absolute=0, divergence=inf
       left preconditioning
       using PRECONDITIONED norm type for convergence test
     PC Object:    (mg_levels_1_)     1 MPI processes
       type: sor
         SOR: type = local_symmetric, iterations = 1, local iterations  
= 1, omega = 1
       linear system matrix = precond matrix:
       Matrix Object:       1 MPI processes
         type: shell
         rows=18522, cols=18522

[...]

Even if maximum iterations=2 and the tolerances are zeros, it performs  
just one iteration. Anyway, the converged reason for the down smoother  
is 4, then it seems it converges with the maximum iterations, but the  
results I obtain are the same as in the case I set 1 pre smoothing.
I implemented the relaxation in a routine UserSOR and I registered it  
as MATOP_SOR for the matrix of the kspSmootherDown context (together  
with a MATOP_MULT).

Finally, the output of the KSPMonitorTrueResidualNorm() for the first  
iteration is:
1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm  
5.094557710929e+02 ||r(i)||/||b|| 2.001351267432e+00

Thank you.
Armando


From bsmith at mcs.anl.gov  Wed May 23 18:15:31 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 23 May 2012 18:15:31 -0500
Subject: [petsc-users] number of pre- and post- smoothing in multigrid
	as a preconditioner
In-Reply-To: <20120524010456.Horde.I1kaUuph4B9PvW0YaN4gN_A@mbox.dmi.unict.it>
References: <20120524010456.Horde.I1kaUuph4B9PvW0YaN4gN_A@mbox.dmi.unict.it>
Message-ID: <0D966127-FEB5-47E8-9290-F6D89DAD907C@mcs.anl.gov>


On May 23, 2012, at 6:04 PM, coco at dmi.unict.it wrote:

> Dear all,
> 
> I am using multigrid as a preconditioner and I noticed that if I try to perform 2 or more pre (or post) smoothing I obtain the same results as I perform just 1 smoothing. The ksp_view output is the following:
> 
> [...]
> 
> Down solver (pre-smoother) on level 1 -------------------------------
>    KSP Object:    (mg_levels_1_)     1 MPI processes
>      type: richardson
>        Richardson: damping factor=1
>      maximum iterations=2, initial guess is zero
>      tolerances:  relative=0, absolute=0, divergence=inf
>      left preconditioning
>      using PRECONDITIONED norm type for convergence test
>    PC Object:    (mg_levels_1_)     1 MPI processes
>      type: sor
>        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
>      linear system matrix = precond matrix:
>      Matrix Object:       1 MPI processes
>        type: shell
>        rows=18522, cols=18522
> 
> [...]
> 
> Even if maximum iterations=2 and the tolerances are zeros, it performs just one iteration. Anyway, the converged reason for the down smoother is 4, then it seems it converges with the maximum iterations, but the results I obtain are the same as in the case I set 1 pre smoothing.
> I implemented the relaxation in a routine UserSOR and I registered it as MATOP_SOR for the matrix of the kspSmootherDown context (together with a MATOP_MULT).
> 
> Finally, the output of the KSPMonitorTrueResidualNorm() for the first iteration is:
> 1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm 5.094557710929e+02 ||r(i)||/||b|| 2.001351267432e+00

   With that 0 residual norm it has to stop. Something is wrong that the preconditioned residual is 0 while unpreconditioned is nonzero. That cannot be true to since the smoother is suppose to be a linear operator. Investigate why the it thinks the preconditioned residual is 0 but the nonpreconditioned is not.

   Barry

> 
> Thank you.
> Armando
> 


From hanangul12 at yahoo.co.uk  Thu May 24 14:14:58 2012
From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh)
Date: Thu, 24 May 2012 20:14:58 +0100 (BST)
Subject: [petsc-users] Multigrid Vcycle.
Message-ID: <1337886898.95768.YahooMailNeo@web28706.mail.ir2.yahoo.com>

Dear all, 

I am quit beginner in PETSc. 

I wrote a code for multigrid Vcycle (as preconditioer) by following the instructions on manual page. 

I want to see how much memory a Vcycle allocates by default in terms of number of matrices and vectors. 

I made a try to see by comparing with jacobi preconditioner, which surely allocates 1 vector ONLY. 

It turns out that a VCycle for FIVE (5) levels including coarsest level allocates memory for 53 vectors and 

subsequently it destroys too.

Is it normal ?? 



Thanks in advance, 

Abdul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120524/14c23e10/attachment.html>

From jedbrown at mcs.anl.gov  Thu May 24 14:22:17 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 24 May 2012 14:22:17 -0500
Subject: [petsc-users] Multigrid Vcycle.
In-Reply-To: <1337886898.95768.YahooMailNeo@web28706.mail.ir2.yahoo.com>
References: <1337886898.95768.YahooMailNeo@web28706.mail.ir2.yahoo.com>
Message-ID: <CAM9tzSmCdiWVKE=yv2fEaR=RDsk3Ot48zv8JWV8QRP7vbjevPA@mail.gmail.com>

On Thu, May 24, 2012 at 2:14 PM, Abdul Hanan Sheikh
<hanangul12 at yahoo.co.uk>wrote:

> Dear all,
> I am quit beginner in PETSc.
> I wrote a code for multigrid Vcycle (as preconditioer) by following the
> instructions on manual page.
> I want to see how much memory a Vcycle allocates by default in terms of
> number of matrices and vectors.
> I made a try to see by comparing with jacobi preconditioner, which surely
> allocates 1 vector ONLY.
> It turns out that a VCycle for FIVE (5) levels including coarsest level
> allocates memory for 53 vectors and
> subsequently it destroys too.
>

Some of those vectors are very small due to being on the coarse level.
There are also the vectors used by the outer Krylov method.


>
> Is it normal ??
>
>
> Thanks in advance,
> Abdul
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120524/00c486ad/attachment.html>

From hanangul12 at yahoo.co.uk  Thu May 24 14:30:05 2012
From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh)
Date: Thu, 24 May 2012 20:30:05 +0100 (BST)
Subject: [petsc-users] Fw:  Multigrid Vcycle.
In-Reply-To: <1337887779.66946.YahooMailNeo@web28702.mail.ir2.yahoo.com>
References: <1337886898.95768.YahooMailNeo@web28706.mail.ir2.yahoo.com>
	<CAM9tzSmCdiWVKE=yv2fEaR=RDsk3Ot48zv8JWV8QRP7vbjevPA@mail.gmail.com>
	<1337887779.66946.YahooMailNeo@web28702.mail.ir2.yahoo.com>
Message-ID: <1337887805.95128.YahooMailNeo@web28706.mail.ir2.yahoo.com>



Thanks for early response Jed!, 
>
>I fixed outer Krylov as PREONLY so it does not use any vectors (I guess).?
>Standard Vcycle includes 1 pre and 1 post smoothing. 
>
>KSP-Richardson + PC-Jacobi are ingredients for pre-smoothing and? post smoothing.?
>To my understanding, 2 Vectors are used by KSP Richardson + 1 Vector is used by PC-Jacobi?
>at pre and post smoothing. This makes 6 vectors at each level and for 5 levels, they should be 30! 
>
>
>
>Do I think right ? 
>
>Further hint to make PCMG more memory-efficient is appreciated. 
>with warm regard, Abdul 
>
>
>
>
>
>
>
>
>>________________________________
>> From: Jed Brown <jedbrown at mcs.anl.gov>
>>To: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>; PETSc users list <petsc-users at mcs.anl.gov> 
>>Sent: Thursday, 24 May 2012, 21:22
>>Subject: Re: [petsc-users] Multigrid Vcycle.
>> 
>>
>>On Thu, May 24, 2012 at 2:14 PM, Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk> wrote:
>>
>>Dear all, 
>>>
>>>I am quit beginner in PETSc. 
>>>
>>>I wrote a code for multigrid Vcycle (as preconditioer) by following the instructions on manual page. 
>>>
>>>I want to see how much memory a Vcycle allocates by default in terms of number of matrices and vectors. 
>>>
>>>I made a try to see by comparing with jacobi preconditioner, which surely allocates 1 vector ONLY. 
>>>
>>>It turns out that a VCycle for FIVE (5) levels including coarsest level allocates memory for 53 vectors and 
>>>
>>>subsequently it destroys too.
>>
>>
>>Some of those vectors are very small due to being on the coarse level. There are also the vectors used by the outer Krylov method.
>>?
>>
>>>
>>>Is it normal ?? 
>>>
>>>
>>>
>>>
>>>
>>>Thanks in advance, 
>>>
>>>Abdul
>>>
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120524/d508e26a/attachment.html>

From jedbrown at mcs.anl.gov  Thu May 24 15:06:58 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 24 May 2012 15:06:58 -0500
Subject: [petsc-users] Multigrid Vcycle.
In-Reply-To: <1337887779.66946.YahooMailNeo@web28702.mail.ir2.yahoo.com>
References: <1337886898.95768.YahooMailNeo@web28706.mail.ir2.yahoo.com>
	<CAM9tzSmCdiWVKE=yv2fEaR=RDsk3Ot48zv8JWV8QRP7vbjevPA@mail.gmail.com>
	<1337887779.66946.YahooMailNeo@web28702.mail.ir2.yahoo.com>
Message-ID: <CAM9tzSmP06g9SL7VAEcGwbRkQub7qxgU7NEb3ZyAzUQwjdUVZQ@mail.gmail.com>

On Thu, May 24, 2012 at 2:29 PM, Abdul Hanan Sheikh
<hanangul12 at yahoo.co.uk>wrote:

> Thanks for early response Jed!,
> I fixed outer Krylov as PREONLY so it does not use any vectors (I guess).
> Standard Vcycle includes 1 pre and 1 post smoothing.
> KSP-Richardson + PC-Jacobi are ingredients for pre-smoothing and  post
> smoothing.
> To my understanding, 2 Vectors are used by KSP Richardson + 1 Vector is
> used by PC-Jacobi
> at pre and post smoothing. This makes 6 vectors at each level and for 5
> levels, they should be 30!
>

DMSetUp_DA_nD creates and destroys two vectors as part of its set up
process. There is also a scaling vector associated with restriction (so
that restriction and prolongation can use the same matrix, which they
usually do), including a momentary work vector used to compute the scaling.
So many of those vectors are not alive at once.

Are you seeing the memory used by those vectors being a significant part of
the overall memory usage of the program?

Barry,


>
> Do I think right ?
> Further hint to make PCMG more memory-efficient is appreciated.
> with warm regard, Abdul
>
>
>   ------------------------------
> *From:* Jed Brown <jedbrown at mcs.anl.gov>
> *To:* Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>; PETSc users list <
> petsc-users at mcs.anl.gov>
> *Sent:* Thursday, 24 May 2012, 21:22
> *Subject:* Re: [petsc-users] Multigrid Vcycle.
>
> On Thu, May 24, 2012 at 2:14 PM, Abdul Hanan Sheikh <
> hanangul12 at yahoo.co.uk> wrote:
>
> Dear all,
> I am quit beginner in PETSc.
> I wrote a code for multigrid Vcycle (as preconditioer) by following the
> instructions on manual page.
> I want to see how much memory a Vcycle allocates by default in terms of
> number of matrices and vectors.
> I made a try to see by comparing with jacobi preconditioner, which surely
> allocates 1 vector ONLY.
> It turns out that a VCycle for FIVE (5) levels including coarsest level
> allocates memory for 53 vectors and
> subsequently it destroys too.
>
>
> Some of those vectors are very small due to being on the coarse level.
> There are also the vectors used by the outer Krylov method.
>
>
>
> Is it normal ??
>
>
> Thanks in advance,
> Abdul
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120524/f7e22795/attachment.html>

From sunnyleekr at gmail.com  Fri May 25 01:16:42 2012
From: sunnyleekr at gmail.com (Sunhee Lee)
Date: Fri, 25 May 2012 15:16:42 +0900
Subject: [petsc-users] petsc compile error using Intel compiler 10.1
Message-ID: <CANOnnSSSwqxZkko4ur-R+wTPHavykYbJwmOFfot764fB4SeGSQ@mail.gmail.com>

I have trouble in compiling some of the petsc files (in complex mode) using
intel compiler.

The relevant files are: baijfact2/5/7.c sbaijfact9~12.c

It says


Fatal compilation error: Out of memory asking for 2097160.
compilation aborted for baijfact5.c (code 1)


I am currently using icc -O3 and that's pretty much it.
Does anyone know the cause and following solution to this issue?

Thanks
Sunhee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120525/73644f3d/attachment.html>

From balay at mcs.anl.gov  Fri May 25 01:25:29 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 25 May 2012 01:25:29 -0500 (CDT)
Subject: [petsc-users] petsc compile error using Intel compiler 10.1
In-Reply-To: <CANOnnSSSwqxZkko4ur-R+wTPHavykYbJwmOFfot764fB4SeGSQ@mail.gmail.com>
References: <CANOnnSSSwqxZkko4ur-R+wTPHavykYbJwmOFfot764fB4SeGSQ@mail.gmail.com>
Message-ID: <alpine.LFD.2.02.1205250117560.15733@asterix>

On Fri, 25 May 2012, Sunhee Lee wrote:

> I have trouble in compiling some of the petsc files (in complex mode) using
> intel compiler.
> 
> The relevant files are: baijfact2/5/7.c sbaijfact9~12.c
> 
> It says
> 
> 
> Fatal compilation error: Out of memory asking for 2097160.
> compilation aborted for baijfact5.c (code 1)
> 
> 
> I am currently using icc -O3 and that's pretty much it.
> Does anyone know the cause and following solution to this issue?

Perhaps you have low process memory limit? What do you have for 'ulimit -a'?

you can try 'ulimit -m unlimited' and see if that helps.

Alternatively - you can try manually compiling the failed sources with
a lower optimiation. Assuming you are using legacy build - you can
edit PETSC_ARCH/conf/petscvariables - and change the -O3 flag - and
then invoke 'make' in the src dirs where compile failed.

Satish


From coco at dmi.unict.it  Fri May 25 07:14:23 2012
From: coco at dmi.unict.it (coco at dmi.unict.it)
Date: Fri, 25 May 2012 14:14:23 +0200
Subject: [petsc-users] number of pre- and post- smoothing in multigrid as a
	preconditioner
In-Reply-To: <mailman.75.1337878813.26963.petsc-users@mcs.anl.gov>
References: <mailman.75.1337878813.26963.petsc-users@mcs.anl.gov>
Message-ID: <20120525141423.Horde.AfM1Q_ph4B9Pv3efFG00SWA@mbox.dmi.unict.it>

Thanks Barry.
I think have found the trouble. The user-provided routine UserSOR  
takes in input the parameter its, which is the number of relaxations  
(its=2 in my case). I didn't understand this fact, then I wrote  
UserSOR as a single relaxation. Now I added a code line:  
"while(its--)" at the beginning of UserSOR and everything is fine. It  
remains unclear why the preconditioned residual is zero, but now the  
code works.

Best regards,
Armando


petsc-users-request at mcs.anl.gov ha scritto:

> Send petsc-users mailing list submissions to
> 	petsc-users at mcs.anl.gov
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
> or, via email, send a message with subject or body 'help' to
> 	petsc-users-request at mcs.anl.gov
>
> You can reach the person managing the list at
> 	petsc-users-owner at mcs.anl.gov
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of petsc-users digest..."
>
>
> Today's Topics:
>
>    1.  number of pre- and post- smoothing in multigrid as a
>       preconditioner (coco at dmi.unict.it)
>    2. Re:  number of pre- and post- smoothing in multigrid	as a
>       preconditioner (Barry Smith)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 24 May 2012 01:04:56 +0200
> From: coco at dmi.unict.it
> To: petsc-users at mcs.anl.gov
> Subject: [petsc-users] number of pre- and post- smoothing in multigrid
> 	as a	preconditioner
> Message-ID:
> 	<20120524010456.Horde.I1kaUuph4B9PvW0YaN4gN_A at mbox.dmi.unict.it>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed; DelSp=Yes
>
> Dear all,
>
> I am using multigrid as a preconditioner and I noticed that if I try
> to perform 2 or more pre (or post) smoothing I obtain the same results
> as I perform just 1 smoothing. The ksp_view output is the following:
>
> [...]
>
> Down solver (pre-smoother) on level 1 -------------------------------
>      KSP Object:    (mg_levels_1_)     1 MPI processes
>        type: richardson
>          Richardson: damping factor=1
>        maximum iterations=2, initial guess is zero
>        tolerances:  relative=0, absolute=0, divergence=inf
>        left preconditioning
>        using PRECONDITIONED norm type for convergence test
>      PC Object:    (mg_levels_1_)     1 MPI processes
>        type: sor
>          SOR: type = local_symmetric, iterations = 1, local iterations
> = 1, omega = 1
>        linear system matrix = precond matrix:
>        Matrix Object:       1 MPI processes
>          type: shell
>          rows=18522, cols=18522
>
> [...]
>
> Even if maximum iterations=2 and the tolerances are zeros, it performs
> just one iteration. Anyway, the converged reason for the down smoother
> is 4, then it seems it converges with the maximum iterations, but the
> results I obtain are the same as in the case I set 1 pre smoothing.
> I implemented the relaxation in a routine UserSOR and I registered it
> as MATOP_SOR for the matrix of the kspSmootherDown context (together
> with a MATOP_MULT).
>
> Finally, the output of the KSPMonitorTrueResidualNorm() for the first
> iteration is:
> 1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm
> 5.094557710929e+02 ||r(i)||/||b|| 2.001351267432e+00
>
> Thank you.
> Armando
>
>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 23 May 2012 18:15:31 -0500
> From: Barry Smith <bsmith at mcs.anl.gov>
> To: PETSc users list <petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] number of pre- and post- smoothing in
> 	multigrid	as a preconditioner
> Message-ID: <0D966127-FEB5-47E8-9290-F6D89DAD907C at mcs.anl.gov>
> Content-Type: text/plain; charset=us-ascii
>
>
> On May 23, 2012, at 6:04 PM, coco at dmi.unict.it wrote:
>
>> Dear all,
>>
>> I am using multigrid as a preconditioner and I noticed that if I  
>> try to perform 2 or more pre (or post) smoothing I obtain the same  
>> results as I perform just 1 smoothing. The ksp_view output is the  
>> following:
>>
>> [...]
>>
>> Down solver (pre-smoother) on level 1 -------------------------------
>>    KSP Object:    (mg_levels_1_)     1 MPI processes
>>      type: richardson
>>        Richardson: damping factor=1
>>      maximum iterations=2, initial guess is zero
>>      tolerances:  relative=0, absolute=0, divergence=inf
>>      left preconditioning
>>      using PRECONDITIONED norm type for convergence test
>>    PC Object:    (mg_levels_1_)     1 MPI processes
>>      type: sor
>>        SOR: type = local_symmetric, iterations = 1, local  
>> iterations = 1, omega = 1
>>      linear system matrix = precond matrix:
>>      Matrix Object:       1 MPI processes
>>        type: shell
>>        rows=18522, cols=18522
>>
>> [...]
>>
>> Even if maximum iterations=2 and the tolerances are zeros, it  
>> performs just one iteration. Anyway, the converged reason for the  
>> down smoother is 4, then it seems it converges with the maximum  
>> iterations, but the results I obtain are the same as in the case I  
>> set 1 pre smoothing.
>> I implemented the relaxation in a routine UserSOR and I registered  
>> it as MATOP_SOR for the matrix of the kspSmootherDown context  
>> (together with a MATOP_MULT).
>>
>> Finally, the output of the KSPMonitorTrueResidualNorm() for the  
>> first iteration is:
>> 1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm  
>> 5.094557710929e+02 ||r(i)||/||b|| 2.001351267432e+00
>
>    With that 0 residual norm it has to stop. Something is wrong that  
> the preconditioned residual is 0 while unpreconditioned is nonzero.  
> That cannot be true to since the smoother is suppose to be a linear  
> operator. Investigate why the it thinks the preconditioned residual  
> is 0 but the nonpreconditioned is not.
>
>    Barry
>
>>
>> Thank you.
>> Armando
>>
>
>
>
> ------------------------------
>
> _______________________________________________
> petsc-users mailing list
> petsc-users at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/petsc-users
>
>
> End of petsc-users Digest, Vol 41, Issue 90
> *******************************************




From mike.hui.zhang at hotmail.com  Fri May 25 08:44:05 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Fri, 25 May 2012 15:44:05 +0200
Subject: [petsc-users] Dense multiply Sparse
In-Reply-To: <7F3B4ABD-CCD2-44B9-BA2E-C4032D3A7F86@mcs.anl.gov>
References: <BLU0-SMTP213F35083CCB884AA959CC8CB1A0@phx.gbl>
	<7F3B4ABD-CCD2-44B9-BA2E-C4032D3A7F86@mcs.anl.gov>
Message-ID: <BLU0-SMTP442BECEFB51891DF9F7D82BCB010@phx.gbl>

A new question: can I use MatSetValues in the following way?

MatGetArray(A,&a);
a[0]= 1;
MatSetValues(A,...);    /* in particular, is this allowed inside GetArray/RestoreArray? */
MatRestoreArray(A,&a);

Thanks!
Hui

On May 14, 2012, at 2:29 PM, Barry Smith wrote:

> 
>   If B is 50% dense then store it in a dense format. It will be much faster and the extra memory is minimal.  Even 30% dense.
> 
>   Barry
> 
> On May 14, 2012, at 5:13 AM, Hui Zhang wrote:
> 
>> I have two matrices A and B stored in sparse format.  While A is really sparse, B is relatively
>> dense (say non-zeros entries about 50%).  Now to multiply the two matrices, among 
>> 
>>     A*B 
>> or 
>>    (B^T * A^T)^T
>> 
>> which is better, or no big difference ?
>> 
>> Thanks!
>> 
> 
> 


From jedbrown at mcs.anl.gov  Fri May 25 08:49:05 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 25 May 2012 08:49:05 -0500
Subject: [petsc-users] Dense multiply Sparse
In-Reply-To: <BLU0-SMTP442BECEFB51891DF9F7D82BCB010@phx.gbl>
References: <BLU0-SMTP213F35083CCB884AA959CC8CB1A0@phx.gbl>
	<7F3B4ABD-CCD2-44B9-BA2E-C4032D3A7F86@mcs.anl.gov>
	<BLU0-SMTP442BECEFB51891DF9F7D82BCB010@phx.gbl>
Message-ID: <CAM9tzSkyybAXmGpwiKLeocN-eWsP-PdMEHcK5xR2_+qRSzu_HA@mail.gmail.com>

No, its not okay.

I think MatGetArray needs to be removed. Pretty much any use of it is
wrong. It is not even the right thing for dense matrices because they
should be distributed differently than PETSc does now.
On May 25, 2012 8:44 AM, "Hui Zhang" <mike.hui.zhang at hotmail.com> wrote:

> A new question: can I use MatSetValues in the following way?
>
> MatGetArray(A,&a);
> a[0]= 1;
> MatSetValues(A,...);    /* in particular, is this allowed inside
> GetArray/RestoreArray? */
> MatRestoreArray(A,&a);
>
> Thanks!
> Hui
>
> On May 14, 2012, at 2:29 PM, Barry Smith wrote:
>
> >
> >   If B is 50% dense then store it in a dense format. It will be much
> faster and the extra memory is minimal.  Even 30% dense.
> >
> >   Barry
> >
> > On May 14, 2012, at 5:13 AM, Hui Zhang wrote:
> >
> >> I have two matrices A and B stored in sparse format.  While A is really
> sparse, B is relatively
> >> dense (say non-zeros entries about 50%).  Now to multiply the two
> matrices, among
> >>
> >>     A*B
> >> or
> >>    (B^T * A^T)^T
> >>
> >> which is better, or no big difference ?
> >>
> >> Thanks!
> >>
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120525/ec261eaa/attachment.html>

From knepley at gmail.com  Fri May 25 09:49:31 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 25 May 2012 10:49:31 -0400
Subject: [petsc-users] Dense multiply Sparse
In-Reply-To: <CAM9tzSkyybAXmGpwiKLeocN-eWsP-PdMEHcK5xR2_+qRSzu_HA@mail.gmail.com>
References: <BLU0-SMTP213F35083CCB884AA959CC8CB1A0@phx.gbl>
	<7F3B4ABD-CCD2-44B9-BA2E-C4032D3A7F86@mcs.anl.gov>
	<BLU0-SMTP442BECEFB51891DF9F7D82BCB010@phx.gbl>
	<CAM9tzSkyybAXmGpwiKLeocN-eWsP-PdMEHcK5xR2_+qRSzu_HA@mail.gmail.com>
Message-ID: <CAMYG4G=BZt6G42LfLTiLvcOdZJ20wUGvygwHXiYsmpAfD_8jLw@mail.gmail.com>

On Fri, May 25, 2012 at 9:49 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> No, its not okay.
>
> I think MatGetArray needs to be removed. Pretty much any use of it is
> wrong. It is not even the right thing for dense matrices because they
> should be distributed differently than PETSc does now.
>
Without it, people cannot get directly to the AIJ data structure. No matter
how wrong it is,
it is a constant request. I would keep it in just to satisfy that large
number of petsc-maints.

   Matt

> On May 25, 2012 8:44 AM, "Hui Zhang" <mike.hui.zhang at hotmail.com> wrote:
>
>> A new question: can I use MatSetValues in the following way?
>>
>> MatGetArray(A,&a);
>> a[0]= 1;
>> MatSetValues(A,...);    /* in particular, is this allowed inside
>> GetArray/RestoreArray? */
>> MatRestoreArray(A,&a);
>>
>> Thanks!
>> Hui
>>
>> On May 14, 2012, at 2:29 PM, Barry Smith wrote:
>>
>> >
>> >   If B is 50% dense then store it in a dense format. It will be much
>> faster and the extra memory is minimal.  Even 30% dense.
>> >
>> >   Barry
>> >
>> > On May 14, 2012, at 5:13 AM, Hui Zhang wrote:
>> >
>> >> I have two matrices A and B stored in sparse format.  While A is
>> really sparse, B is relatively
>> >> dense (say non-zeros entries about 50%).  Now to multiply the two
>> matrices, among
>> >>
>> >>     A*B
>> >> or
>> >>    (B^T * A^T)^T
>> >>
>> >> which is better, or no big difference ?
>> >>
>> >> Thanks!
>> >>
>> >
>> >
>>
>>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120525/f3b2f68a/attachment.html>

From jedbrown at mcs.anl.gov  Fri May 25 09:57:07 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Fri, 25 May 2012 09:57:07 -0500
Subject: [petsc-users] Dense multiply Sparse
In-Reply-To: <CAMYG4G=BZt6G42LfLTiLvcOdZJ20wUGvygwHXiYsmpAfD_8jLw@mail.gmail.com>
References: <BLU0-SMTP213F35083CCB884AA959CC8CB1A0@phx.gbl>
	<7F3B4ABD-CCD2-44B9-BA2E-C4032D3A7F86@mcs.anl.gov>
	<BLU0-SMTP442BECEFB51891DF9F7D82BCB010@phx.gbl>
	<CAM9tzSkyybAXmGpwiKLeocN-eWsP-PdMEHcK5xR2_+qRSzu_HA@mail.gmail.com>
	<CAMYG4G=BZt6G42LfLTiLvcOdZJ20wUGvygwHXiYsmpAfD_8jLw@mail.gmail.com>
Message-ID: <CAM9tzSnaUzsW88uJc7p0db03jbkEpFqgXd=E2V=WhoJrpha8JA@mail.gmail.com>

On Fri, May 25, 2012 at 9:49 AM, Matthew Knepley <knepley at gmail.com> wrote:

> I think MatGetArray needs to be removed. Pretty much any use of it is
>> wrong. It is not even the right thing for dense matrices because they
>> should be distributed differently than PETSc does now.
>>
> Without it, people cannot get directly to the AIJ data structure. No
> matter how wrong it is,
> it is a constant request. I would keep it in just to satisfy that large
> number of petsc-maints.
>

There can be a MatSeqAIJGetArray() and others.

These arrays are all different things so the user should call a different
function to make it clear what they are getting.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120525/a91770c9/attachment.html>

From hanangul12 at yahoo.co.uk  Fri May 25 12:12:31 2012
From: hanangul12 at yahoo.co.uk (Abdul Hanan Sheikh)
Date: Fri, 25 May 2012 18:12:31 +0100 (BST)
Subject: [petsc-users] Multigrid Vcycle.
In-Reply-To: <CAM9tzSmP06g9SL7VAEcGwbRkQub7qxgU7NEb3ZyAzUQwjdUVZQ@mail.gmail.com>
References: <1337886898.95768.YahooMailNeo@web28706.mail.ir2.yahoo.com>
	<CAM9tzSmCdiWVKE=yv2fEaR=RDsk3Ot48zv8JWV8QRP7vbjevPA@mail.gmail.com>
	<1337887779.66946.YahooMailNeo@web28702.mail.ir2.yahoo.com>
	<CAM9tzSmP06g9SL7VAEcGwbRkQub7qxgU7NEb3ZyAzUQwjdUVZQ@mail.gmail.com>
Message-ID: <1337965951.82693.YahooMailNeo@web28701.mail.ir2.yahoo.com>

Dear all,?
Are you seeing the memory used by those vectors being a significant part of the overall memory usage of the program??

To this question, I think yes.?
Is it also normal, number of vectors is dependent upon how many levels used ? ?
I experienced this. When I increase levels in PCMG, number of vectors used (created and destroyed) increases.?

Thanks,?
Abdul .


>________________________________
> From: Jed Brown <jedbrown at mcs.anl.gov>
>To: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>; PETSc users list <petsc-users at mcs.anl.gov> 
>Sent: Thursday, 24 May 2012, 22:06
>Subject: Re: [petsc-users] Multigrid Vcycle.
> 
>
>On Thu, May 24, 2012 at 2:29 PM, Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk> wrote:
>
>Thanks for early response Jed!, 
>>
>>I fixed outer Krylov as PREONLY so it does not use any vectors (I guess).?
>>Standard Vcycle includes 1 pre and 1 post smoothing. 
>>
>>KSP-Richardson + PC-Jacobi are ingredients for pre-smoothing and? post smoothing.?
>>To my understanding, 2 Vectors are used by KSP Richardson + 1 Vector is used by PC-Jacobi?
>>at pre and post smoothing. This makes 6 vectors at each level and for 5 levels, they should be 30! 
>>
>
>
>DMSetUp_DA_nD creates and destroys two vectors as part of its set up process. There is also a scaling vector associated with restriction (so that restriction and prolongation can use the same matrix, which they usually do), including a momentary work vector used to compute the scaling. So many of those vectors are not alive at once.
>
>
>Are you seeing the memory used by those vectors being a significant part of the overall memory usage of the program?
>
>
>Barry,?
>?
>
>>
>>Do I think right ? 
>>
>>Further hint to make PCMG more memory-efficient is appreciated. 
>>with warm regard, Abdul 
>>
>>
>>
>>
>>
>>>________________________________
>>> From: Jed Brown <jedbrown at mcs.anl.gov>
>>>To: Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>; PETSc users list <petsc-users at mcs.anl.gov> 
>>>Sent: Thursday, 24 May 2012, 21:22
>>>Subject: Re: [petsc-users] Multigrid Vcycle.
>>> 
>>>
>>>
>>>On Thu, May 24, 2012 at 2:14 PM, Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk> wrote:
>>>
>>>Dear all, 
>>>>
>>>>I am quit beginner in PETSc. 
>>>>
>>>>I wrote a code for multigrid Vcycle (as preconditioer) by following the instructions on manual page. 
>>>>
>>>>I want to see how much memory a Vcycle allocates by default in terms of number of matrices and vectors. 
>>>>
>>>>I made a try to see by comparing with jacobi preconditioner, which surely allocates 1 vector ONLY. 
>>>>
>>>>It turns out that a VCycle for FIVE (5) levels including coarsest level allocates memory for 53 vectors and 
>>>>
>>>>subsequently it destroys too.
>>>
>>>
>>>Some of those vectors are very small due to being on the coarse level. There are also the vectors used by the outer Krylov method.
>>>?
>>>
>>>>
>>>>Is it normal ?? 
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>Thanks in advance, 
>>>>
>>>>Abdul
>>>>
>>>
>>>
>>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120525/99026ad2/attachment.html>

From knepley at gmail.com  Fri May 25 12:14:13 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 25 May 2012 13:14:13 -0400
Subject: [petsc-users] Multigrid Vcycle.
In-Reply-To: <1337965951.82693.YahooMailNeo@web28701.mail.ir2.yahoo.com>
References: <1337886898.95768.YahooMailNeo@web28706.mail.ir2.yahoo.com>
	<CAM9tzSmCdiWVKE=yv2fEaR=RDsk3Ot48zv8JWV8QRP7vbjevPA@mail.gmail.com>
	<1337887779.66946.YahooMailNeo@web28702.mail.ir2.yahoo.com>
	<CAM9tzSmP06g9SL7VAEcGwbRkQub7qxgU7NEb3ZyAzUQwjdUVZQ@mail.gmail.com>
	<1337965951.82693.YahooMailNeo@web28701.mail.ir2.yahoo.com>
Message-ID: <CAMYG4G=oZ_b_L=giH+XTB=sJb+tC5gwxM5Lj3PKhWyDEno9O9A@mail.gmail.com>

On Fri, May 25, 2012 at 1:12 PM, Abdul Hanan Sheikh
<hanangul12 at yahoo.co.uk>wrote:

> Dear all,
> Are you seeing the memory used by those vectors being a significant part
> of the overall memory usage of the program?
> To this question, I think yes.
>

Measurement?


> Is it also normal, number of vectors is dependent upon how many levels
> used ?
>

How else would you propose to solve problems on each level?


> I experienced this. When I increase levels in PCMG, number of vectors used
> (created and destroyed) increases.
>

Yes.

    Matt


> Thanks,
> Abdul .
>
>   ------------------------------
> *From:* Jed Brown <jedbrown at mcs.anl.gov>
> *To:* Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>; PETSc users list <
> petsc-users at mcs.anl.gov>
> *Sent:* Thursday, 24 May 2012, 22:06
> *Subject:* Re: [petsc-users] Multigrid Vcycle.
>
> On Thu, May 24, 2012 at 2:29 PM, Abdul Hanan Sheikh <
> hanangul12 at yahoo.co.uk> wrote:
>
> Thanks for early response Jed!,
> I fixed outer Krylov as PREONLY so it does not use any vectors (I guess).
> Standard Vcycle includes 1 pre and 1 post smoothing.
> KSP-Richardson + PC-Jacobi are ingredients for pre-smoothing and  post
> smoothing.
> To my understanding, 2 Vectors are used by KSP Richardson + 1 Vector is
> used by PC-Jacobi
> at pre and post smoothing. This makes 6 vectors at each level and for 5
> levels, they should be 30!
>
>
> DMSetUp_DA_nD creates and destroys two vectors as part of its set up
> process. There is also a scaling vector associated with restriction (so
> that restriction and prolongation can use the same matrix, which they
> usually do), including a momentary work vector used to compute the scaling.
> So many of those vectors are not alive at once.
>
> Are you seeing the memory used by those vectors being a significant part
> of the overall memory usage of the program?
>
> Barry,
>
>
>
> Do I think right ?
> Further hint to make PCMG more memory-efficient is appreciated.
> with warm regard, Abdul
>
>
>   ------------------------------
> *From:* Jed Brown <jedbrown at mcs.anl.gov>
> *To:* Abdul Hanan Sheikh <hanangul12 at yahoo.co.uk>; PETSc users list <
> petsc-users at mcs.anl.gov>
> *Sent:* Thursday, 24 May 2012, 21:22
> *Subject:* Re: [petsc-users] Multigrid Vcycle.
>
> On Thu, May 24, 2012 at 2:14 PM, Abdul Hanan Sheikh <
> hanangul12 at yahoo.co.uk> wrote:
>
> Dear all,
> I am quit beginner in PETSc.
> I wrote a code for multigrid Vcycle (as preconditioer) by following the
> instructions on manual page.
> I want to see how much memory a Vcycle allocates by default in terms of
> number of matrices and vectors.
> I made a try to see by comparing with jacobi preconditioner, which surely
> allocates 1 vector ONLY.
> It turns out that a VCycle for FIVE (5) levels including coarsest level
> allocates memory for 53 vectors and
> subsequently it destroys too.
>
>
> Some of those vectors are very small due to being on the coarse level.
> There are also the vectors used by the outer Krylov method.
>
>
>
> Is it normal ??
>
>
> Thanks in advance,
> Abdul
>
>
>
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120525/925f4425/attachment.html>

From w_ang_temp at 163.com  Sat May 26 02:01:28 2012
From: w_ang_temp at 163.com (w_ang_temp)
Date: Sat, 26 May 2012 15:01:28 +0800 (CST)
Subject: [petsc-users] A qustion of PETSc
In-Reply-To: <CAMYG4G=TX7URdbveZA6bTs6LOi7kiDV3kbf5F=JW81JAPPYeSg@mail.gmail.com>
References: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com>
	<CAMYG4G=TX7URdbveZA6bTs6LOi7kiDV3kbf5F=JW81JAPPYeSg@mail.gmail.com>
Message-ID: <ad33a04.1a57d.13787f32753.Coremail.w_ang_temp@163.com>

Hello, Matt
   First,thanks for your help, and I have a new recognition of MPI. As you said(only part of a Vec is stored on each MPI process),I use MPI_Send,MPI_Recv and MPI_Bcast to deal with it and get some right results.
   However,I have not yet fully solved this problem.The number of loops of the subroutine named PETSCSOLVE to solve Ax=b with PETSc is 20 times.I print the results of each loop and find that the front 9 times are right. In the 10 time,there is a strange error: plug the result x into the equation I find that Ax!=b.
   The code and the A,x,b are shown below.So can you give me some hints? Thank you very much!
                                                        Jim
   (1)code
      call MatView(A,PETSC_VIEWER_STDOUT_WORLD,ierr)
      call VecView(b,PETSC_VIEWER_STDOUT_WORLD,ierr)
      call KSPSolve(ksp,b,x,ierr)
      call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
   (2)Ax=b
  A:
  type: mpiaij
 row 0: (0, 1)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7, 0)  (8, 0)
 row 1: (0, 0)  (1, 1)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7, 0)  (8, 0)
 row 2: (0, 0)  (1, 0)  (2, -2e-06)  (3, -50)  (4, 0)  (5, 0)  (6, 0)  (7, -50)  (8, 0)
 row 3: (0, 0)  (1, 0)  (2, -50)  (3, 1.36265e+07)  (4, 0)  (5, 0)  (6, 0)  (7, 1.23877e+06)  (8, 0)
 row 4: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 1)  (5, 0)  (6, 0)  (7, 0)  (8, 0)
 row 5: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 1)  (6, 0)  (7, 0)  (8, 0)
 row 6: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 1)  (7, 0)  (8, 0)
 row 7: (0, 0)  (1, 0)  (2, -50)  (3, 1.23877e+06)  (4, 0)  (5, 0)  (6, 0)  (7, 1.36265e+07)  (8, 0)
 row 8: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7, 0)  (8, 1)
  b:
  type: mpi
 Process [0]
 0
 0
 -0.75
 96949.5
 0
 Process [1]
 0
 -0.015
 96949.5
 -0.015
  x:
  type: mpi
 Process [0]
 0
 0
 1167.84
 0.00819383
 0
 Process [1]
 0
 -0.00824055
 0.00390865
 -0.00824055





>? 2012-05-22 19:33:10?"Matthew Knepley" <knepley at gmail.com> ???
>On Tue, May 22, 2012 at 5:17 AM, w_ang_temp <w_ang_temp at 163.com> wrote:

>Hello
 >    I am a beginner of PETSc.I find the petsc-users lists is very useful, however I do not know how to ask a question in it. So can you tell me? >And below is what I want to ask.
 >   I am solving Ax=b with PETSc. And I only use it in a fortran subroutine to get x ,then the main function of my program use x to do other >things. The question is that: when the process is 1(mpiexec -n 1),the result of my program is ok, but when the process is more than 1 the > result is wrong.



> "Wrong" is not an acceptable description. I suspect that you do not understand that only part of a Vec
> is stored on each MPI process. First, get the book Using MPI, and then read the PETSc manual
> section on vectors.


  > Thanks,


     > Matt
 
>     So where could be wrong? Can you give me some hints on how I can debug? Thanks.
>                                   Jim
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their > experiments lead.
> -- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120526/3339c187/attachment-0001.html>

From w_ang_temp at 163.com  Sat May 26 03:04:07 2012
From: w_ang_temp at 163.com (w_ang_temp)
Date: Sat, 26 May 2012 16:04:07 +0800 (CST)
Subject: [petsc-users] A qustion of PETSc
In-Reply-To: <ad33a04.1a57d.13787f32753.Coremail.w_ang_temp@163.com>
References: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com>
	<CAMYG4G=TX7URdbveZA6bTs6LOi7kiDV3kbf5F=JW81JAPPYeSg@mail.gmail.com>
	<ad33a04.1a57d.13787f32753.Coremail.w_ang_temp@163.com>
Message-ID: <5eedef2f.1e190.137882c844b.Coremail.w_ang_temp@163.com>

Besides,I test several situations.
When process=1(mpiexec -n 1),the results are right,compared with the true results.
When process=2 or 5,both of the results are same.And from the tenth loop, it begins the strange error Ax!=b.
When process=3 or 4,both of the results are same.And from the first loop,it begins the error.
Thanks.




>At 2012-05-26 15:01:28,w_ang_temp <w_ang_temp at 163.com> wrote:

>Hello, Matt
>   First,thanks for your help, and I have a new recognition of MPI. As you said(only part of a Vec is stored on each MPI process),I use >MPI_Send,MPI_Recv and MPI_Bcast to deal with it and get some right results.
>   However,I have not yet fully solved this problem.The number of loops of the subroutine named PETSCSOLVE to solve Ax=b with PETSc is 20 >times.I print the results of each loop and find that the front 9 times are right. In the 10 time,there is a strange error: plug the result x into the >equation I find that Ax!=b.
>   The code and the A,x,b are shown below.So can you give me some hints? Thank you very much!
>                                                        Jim
>   (1)code
>      call MatView(A,PETSC_VIEWER_STDOUT_WORLD,ierr)
>      call VecView(b,PETSC_VIEWER_STDOUT_WORLD,ierr)
>      call KSPSolve(ksp,b,x,ierr)
>      call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
>   (2)Ax=b
>  A:
>  type: mpiaij
> row 0: (0, 1)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7, 0)  (8, 0)
> row 1: (0, 0)  (1, 1)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7, 0)  (8, 0)
> row 2: (0, 0)  (1, 0)  (2, -2e-06)  (3, -50)  (4, 0)  (5, 0)  (6, 0)  (7, -50)  (8, 0)
> row 3: (0, 0)  (1, 0)  (2, -50)  (3, 1.36265e+07)  (4, 0)  (5, 0)  (6, 0)  (7, 1.23877e+06)  (8, 0)
> row 4: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 1)  (5, 0)  (6, 0)  (7, 0)  (8, 0)
> row 5: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 1)  (6, 0)  (7, 0)  (8, 0)
> row 6: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 1)  (7, 0)  (8, 0)
> row 7: (0, 0)  (1, 0)  (2, -50)  (3, 1.23877e+06)  (4, 0)  (5, 0)  (6, 0)  (7, 1.36265e+07)  (8, 0)
> row 8: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7, 0)  (8, 1)
>  b:
>  type: mpi
> Process [0]
> 0
> 0
> -0.75
> 96949.5
> 0
> Process [1]
> 0
> -0.015
> 96949.5
> -0.015
 > x:
  >type: mpi
 >Process [0]
> 0
> 0
> 1167.84
> 0.00819383
 >0
 >Process [1]
 >0
> -0.00824055
> 0.00390865
> -0.00824055





>? 2012-05-22 19:33:10?"Matthew Knepley" <knepley at gmail.com> ???
>On Tue, May 22, 2012 at 5:17 AM, w_ang_temp <w_ang_temp at 163.com> wrote:

>Hello
 >    I am a beginner of PETSc.I find the petsc-users lists is very useful, however I do not know how to ask a question in it. So can you tell me? >And below is what I want to ask.
 >   I am solving Ax=b with PETSc. And I only use it in a fortran subroutine to get x ,then the main function of my program use x to do other >things. The question is that: when the process is 1(mpiexec -n 1),the result of my program is ok, but when the process is more than 1 the > result is wrong.



> "Wrong" is not an acceptable description. I suspect that you do not understand that only part of a Vec
> is stored on each MPI process. First, get the book Using MPI, and then read the PETSc manual
> section on vectors.


  > Thanks,


     > Matt
 
>     So where could be wrong? Can you give me some hints on how I can debug? Thanks.
>                                   Jim
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their > experiments lead.
> -- Norbert Wiener


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120526/5d9ce6e5/attachment.html>

From yelpoo at gmail.com  Sat May 26 03:47:37 2012
From: yelpoo at gmail.com (Ye Jianbo Bob)
Date: Sat, 26 May 2012 16:47:37 +0800
Subject: [petsc-users] SLEPc with offline preconditioner
Message-ID: <CAHgJxPurnEBBLhCtpSHZ1xxVu0r2wDnQtatgWAO1C1X96WhioQ@mail.gmail.com>

Hi, I have some problem when using SLEPc to compute eigenvalue
problems. I do not know how to implement my idea properly. Let me
explain in detail.

I am interested in solving a set of eigenvalue problems (to find the
smallest magnitude eigenvalues)

A x = \lambda B_i x
for i=1,2,3,...

where A is the Laplacian of some regular grid, B_i is diagonal. It is
known that A is semi-positive with a null vector [1,1,...,1].

I found SLEPc provides EPSSetDeflationSpace to set deflation space
when applies iterative eigen solver. But I am not sure what default
precondition is used when I defined my problem by setting
eps_smallest_magnitude through EPSSetWhichEigenpairs.

It is possible to use shift-and-invert to explicitly address my
problem. If the shift value is 0, zero pivot will be reported during
LU precondition stage. But since my set of eigen problems has the
exact the same A, I hope I could somehow do the precondition (direct
solver) offline only associate with A and apply the iterative eigen
solver online with set of different B_i.

The very initiative is the direct solver applied in precondition stage
is O(N^2) while the matrix-free eigen solver is O(N), thus I think
this would improve the efficiency of my situation. Is there any way to
realize it?

Here is some rough idea:
To prevent the zero pivot during LU, I would dampshift A with a small
quantity A-\sigma I, and then compute its LU offline and store it. In
online stage, I could build an ST shell that read LU computed in
offline stage and solve. This is only an approximated approach,
hopefully not degrading the performance.

Thank you!

From jroman at dsic.upv.es  Sat May 26 04:47:29 2012
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Sat, 26 May 2012 11:47:29 +0200
Subject: [petsc-users] SLEPc with offline preconditioner
In-Reply-To: <CAHgJxPurnEBBLhCtpSHZ1xxVu0r2wDnQtatgWAO1C1X96WhioQ@mail.gmail.com>
References: <CAHgJxPurnEBBLhCtpSHZ1xxVu0r2wDnQtatgWAO1C1X96WhioQ@mail.gmail.com>
Message-ID: <7C6C4F35-88F2-4894-BF0E-C94598E559D9@dsic.upv.es>


El 26/05/2012, a las 10:47, Ye Jianbo Bob escribi?:

> Hi, I have some problem when using SLEPc to compute eigenvalue
> problems. I do not know how to implement my idea properly. Let me
> explain in detail.
> 
> I am interested in solving a set of eigenvalue problems (to find the
> smallest magnitude eigenvalues)
> 
> A x = \lambda B_i x
> for i=1,2,3,...
> 
> where A is the Laplacian of some regular grid, B_i is diagonal. It is
> known that A is semi-positive with a null vector [1,1,...,1].
> 
> I found SLEPc provides EPSSetDeflationSpace to set deflation space
> when applies iterative eigen solver. But I am not sure what default
> precondition is used when I defined my problem by setting
> eps_smallest_magnitude through EPSSetWhichEigenpairs.
> 
> It is possible to use shift-and-invert to explicitly address my
> problem. If the shift value is 0, zero pivot will be reported during
> LU precondition stage. But since my set of eigen problems has the
> exact the same A, I hope I could somehow do the precondition (direct
> solver) offline only associate with A and apply the iterative eigen
> solver online with set of different B_i.
> 
> The very initiative is the direct solver applied in precondition stage
> is O(N^2) while the matrix-free eigen solver is O(N), thus I think
> this would improve the efficiency of my situation. Is there any way to
> realize it?
> 
> Here is some rough idea:
> To prevent the zero pivot during LU, I would dampshift A with a small
> quantity A-\sigma I, and then compute its LU offline and store it. In
> online stage, I could build an ST shell that read LU computed in
> offline stage and solve. This is only an approximated approach,
> hopefully not degrading the performance.
> 
> Thank you!

You can use STSHELL to perform any customized spectral transformation. But in your case I don't think it is necessary. [1,1,...,1] is also an eigenvector of the matrix pair (A,B), so you can pass it through EPSSetDeflationSpace, then use shift-and-invert with zero target - in that case the internal KSP will have A as the coefficient matrix and KSPSetNullSpace will be invoked with the [1,1,...,1] vector. The only thing is that you have to use a KSP solver that supports nullspaces. I am not sure if PETSc's direct solvers (or external solvers) support it, so you may have to use an iterative solver (e.g., -st_ksp_type bcgs -st_pc_type bjacobi -st_ksp_rtol 1e-9).

Jose


From knepley at gmail.com  Sat May 26 07:17:42 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 26 May 2012 12:17:42 +0000
Subject: [petsc-users] A qustion of PETSc
In-Reply-To: <5eedef2f.1e190.137882c844b.Coremail.w_ang_temp@163.com>
References: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com>
	<CAMYG4G=TX7URdbveZA6bTs6LOi7kiDV3kbf5F=JW81JAPPYeSg@mail.gmail.com>
	<ad33a04.1a57d.13787f32753.Coremail.w_ang_temp@163.com>
	<5eedef2f.1e190.137882c844b.Coremail.w_ang_temp@163.com>
Message-ID: <CAMYG4GkfTVM6gh+dysdsoiRZjSq9J84yzKug-=Vd=mK4EFXzWA@mail.gmail.com>

Use -ksp_type preonly -pc_type lu.

   Matt

On Sat, May 26, 2012 at 8:04 AM, w_ang_temp <w_ang_temp at 163.com> wrote:

> Besides,I test several situations.
> When process=1(mpiexec -n 1),the results are right,compared with the true
> results.
> When process=2 or 5,both of the results are same.And from the tenth loop,
> it begins the strange error Ax!=b.
> When process=3 or 4,both of the results are same.And from the first
> loop,it begins the error.
> Thanks.
>
>
> >At 2012-05-26 15:01:28,w_ang_temp <w_ang_temp at 163.com> wrote:
>
>  >Hello, Matt
> >   First,thanks for your help, and I have a new recognition of MPI. As
> you said(only part of a Vec is stored on each MPI process),I use
> >MPI_Send,MPI_Recv and MPI_Bcast to deal with it and get some right results.
> >   However,I have not yet fully solved this problem.The number of loops
> of the subroutine named PETSCSOLVE to solve Ax=b with PETSc is 20 >times.I
> print the results of each loop and find that the front 9 times are right.
> In the 10 time,there is a strange error: plug the result x into the
> >equation I find that Ax!=b.
> >   The code and the A,x,b are shown below.So can you give me some hints?
> Thank you very much!
> >                                                        Jim
> >   (1)code
> >      call MatView(A,PETSC_VIEWER_STDOUT_WORLD,ierr)
> >      call VecView(b,PETSC_VIEWER_STDOUT_WORLD,ierr)
> >      call KSPSolve(ksp,b,x,ierr)
> >      call VecView(x,PETSC_VIEWER_STDOUT_WORLD,ierr)
> >   (2)Ax=b
> >  A:
> >  type: mpiaij
> > row 0: (0, 1)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7, 0)
> (8, 0)
> > row 1: (0, 0)  (1, 1)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7, 0)
> (8, 0)
> > row 2: (0, 0)  (1, 0)  (2, -2e-06)  (3, -50)  (4, 0)  (5, 0)  (6, 0)
> (7, -50)  (8, 0)
> > row 3: (0, 0)  (1, 0)  (2, -50)  (3, 1.36265e+07)  (4, 0)  (5, 0)  (6,
> 0)  (7, 1.23877e+06)  (8, 0)
> > row 4: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 1)  (5, 0)  (6, 0)  (7, 0)
> (8, 0)
> > row 5: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 1)  (6, 0)  (7, 0)
> (8, 0)
> > row 6: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 1)  (7, 0)
> (8, 0)
> > row 7: (0, 0)  (1, 0)  (2, -50)  (3, 1.23877e+06)  (4, 0)  (5, 0)  (6,
> 0)  (7, 1.36265e+07)  (8, 0)
> > row 8: (0, 0)  (1, 0)  (2, 0)  (3, 0)  (4, 0)  (5, 0)  (6, 0)  (7, 0)
> (8, 1)
> >  b:
> >  type: mpi
> > Process [0]
> > 0
> > 0
> > -0.75
> > 96949.5
> > 0
> > Process [1]
> > 0
> > -0.015
> > 96949.5
> > -0.015
>  > x:
>   >type: mpi
>  >Process [0]
> > 0
> > 0
> > 1167.84
> > 0.00819383
>  >0
>  >Process [1]
>  >0
> > -0.00824055
> > 0.00390865
> > -0.00824055
>
>
>
> >? 2012-05-22 19:33:10?"Matthew Knepley" <knepley at gmail.com> ???
>
> >On Tue, May 22, 2012 at 5:17 AM, w_ang_temp <w_ang_temp at 163.com> wrote:
>
>>  >Hello
>>  >    I am a beginner of PETSc.I find the petsc-users lists is very
>> useful, however I do not know how to ask a question in it. So can you tell
>> me? >And below is what I want to ask.
>>  >   I am solving Ax=b with PETSc. And I only use it in a fortran
>> subroutine to get x ,then the main function of my program use x to do other
>> >things. The question is that: when the process is 1(mpiexec -n 1),the
>> result of my program is ok, but when the process is more than 1 the
>> > result is wrong.
>>
>
> > "Wrong" is not an acceptable description. I suspect that you do not
> understand that only part of a Vec
> > is stored on each MPI process. First, get the book Using MPI, and then
> read the PETSc manual
> > section on vectors.
>
>   > Thanks,
>
>      > Matt
>
>
>>  >     So where could be wrong? Can you give me some hints on how I can
>> debug? Thanks.
>> >                                   Jim
>>
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> > experiments lead.
> > -- Norbert Wiener
>
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120526/42a4403e/attachment-0001.html>

From aron.ahmadia at kaust.edu.sa  Sat May 26 08:17:06 2012
From: aron.ahmadia at kaust.edu.sa (Aron Ahmadia)
Date: Sat, 26 May 2012 16:17:06 +0300
Subject: [petsc-users] [petsc4py] Cannot cast array data from
 dtype('int64') to dtype('int32') according to the rule 'safe'
Message-ID: <CADVoUZTkBYV=g752trOrMG0Fc4y8PGVRcs+Z8sk2ZQf_PycKhg@mail.gmail.com>

Dear Chris,

Sorry, Lisandro and I missed this.  I answered your question over on
scicomp: http://scicomp.stackexchange.com/a/2356/9

Thanks for the persistence!

Regards,
Aron

On Tue, May 22, 2012 at 9:27 PM, Christian Staudt
<christian.staudt at ira.uka.de> wrote:
> Hi petsc4py users,
>
> I am running into the following error:
>
> ??File "/Users/cls/workspace/LAMG-Python/src/lamg/amg/setup.py", line 945,
> in buildInternal
> ? ? Acc = A[c, c]
> ? File "Mat.pyx", line 185, in petsc4py.PETSc.Mat.__getitem__
> (src/petsc4py.PETSc.c:71407)
> ? File "petscmat.pxi", line 862, in petsc4py.PETSc.mat_getitem
> (src/petsc4py.PETSc.c:22442)
> ? File "petscmat.pxi", line 773, in petsc4py.PETSc.matgetvalues
> (src/petsc4py.PETSc.c:21226)
> ? File "arraynpy.pxi", line 123, in petsc4py.PETSc.iarray_i
> (src/petsc4py.PETSc.c:5284)
> ? File "arraynpy.pxi", line 117, in petsc4py.PETSc.iarray
> (src/petsc4py.PETSc.c:5192)
> TypeError: Cannot cast array data from dtype('int64') to dtype('int32')
> according to the rule 'safe'
>
>
> I am indexing the PETSc.Mat A with an array c ( numpy.ndarray with
> dtype('int64')). ?I assume there is a simple solution (allowing the cast,
> making everything ?32 or 64 bit...). Can you point me in the right
> direction? Thanks.
>
> Chris
>
>

From yelpoo at gmail.com  Sat May 26 08:37:20 2012
From: yelpoo at gmail.com (Ye Jianbo Bob)
Date: Sat, 26 May 2012 21:37:20 +0800
Subject: [petsc-users] SLEPc with offline preconditioner
In-Reply-To: <7C6C4F35-88F2-4894-BF0E-C94598E559D9@dsic.upv.es>
References: <CAHgJxPurnEBBLhCtpSHZ1xxVu0r2wDnQtatgWAO1C1X96WhioQ@mail.gmail.com>
	<7C6C4F35-88F2-4894-BF0E-C94598E559D9@dsic.upv.es>
Message-ID: <CAHgJxPtpx6tKXZ_b50PPsMgYXwx-2xAccJ3Yg-Ckt58D2S8kjw@mail.gmail.com>

Hi Jose, I found shift-and-invert does not work with an iterative pc
solver. Say the examples given in source code

/home/bobye/pub/slepc/slepc-3.2-p3/src/eps/examples/tutorials/ex11.c
------------------------------------------------------------------
if I use the default setting, in fact I am not sure which pc it is
used, even which is coefficient matrix. But it works well. However, in
my cases, I am interested in how to reuse manipulation of A for
different eigenvalues problem.
bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5

Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)

 Number of iterations of the method: 20
 Solution method: krylovschur

 Number of requested eigenvalues: 5
 Stopping condition: tol=1e-08, maxit=100
 Number of converged approximate eigenpairs: 7

           k          ||Ax-kx||/||kx||
   ----------------- ------------------
       0.010956        1.60437e-09
       0.021912        1.97096e-09
       0.043705         5.3322e-10
       0.054661        4.01148e-09
       0.087410         8.7469e-10
       0.097887         2.0693e-09
       0.108843        3.46988e-09
------------------------------------------------------------------
if I use direct pc
bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5 -st_type sinvert -st_pc_type lu
It will report Detected zero pivot in LU factorization
------------------------------------------------------------------
if I use direct pc with shift value
bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5 -st_type sinvert -st_shift
1e-9 -st_pc_type lu

Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)

 Number of iterations of the method: 1
 Solution method: krylovschur

 Number of requested eigenvalues: 5
 Stopping condition: tol=1e-08, maxit=100
 Number of converged approximate eigenpairs: 5

           k          ||Ax-kx||/||kx||
   ----------------- ------------------
       0.010956        5.37411e-08
       0.010956         5.3741e-08
       0.021912        5.13947e-14
       0.043705        1.95903e-11
       0.054661        2.62529e-09

------------------------------------------------------------------
if I use iterative pc with shift-and-invert, the result seems to be wrong.
bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5 -st_type sinvert -st_pc_type
jacobi -st_ksp_rtol 1e-12

Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)

 Number of iterations of the method: 1
 Solution method: krylovschur

 Number of requested eigenvalues: 5
 Stopping condition: tol=1e-08, maxit=100
 Number of converged approximate eigenpairs: 12

           k          ||Ax-kx||/||kx||
   ----------------- ------------------
       2.000000           0.707107
       2.000000           0.707107
       2.000000           0.707107
       2.004191           0.704215
       3.000000           0.579634
       3.000000           0.578826
       3.000000           0.580728
       3.000000            0.57262
       3.000000           0.597176
       3.000000           0.562004
       3.000000           0.587957
       3.096958           0.710225


-jianbo

On Sat, May 26, 2012 at 5:47 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
>
> El 26/05/2012, a las 10:47, Ye Jianbo Bob escribi?:
>
>> Hi, I have some problem when using SLEPc to compute eigenvalue
>> problems. I do not know how to implement my idea properly. Let me
>> explain in detail.
>>
>> I am interested in solving a set of eigenvalue problems (to find the
>> smallest magnitude eigenvalues)
>>
>> A x = \lambda B_i x
>> for i=1,2,3,...
>>
>> where A is the Laplacian of some regular grid, B_i is diagonal. It is
>> known that A is semi-positive with a null vector [1,1,...,1].
>>
>> I found SLEPc provides EPSSetDeflationSpace to set deflation space
>> when applies iterative eigen solver. But I am not sure what default
>> precondition is used when I defined my problem by setting
>> eps_smallest_magnitude through EPSSetWhichEigenpairs.
>>
>> It is possible to use shift-and-invert to explicitly address my
>> problem. If the shift value is 0, zero pivot will be reported during
>> LU precondition stage. But since my set of eigen problems has the
>> exact the same A, I hope I could somehow do the precondition (direct
>> solver) offline only associate with A and apply the iterative eigen
>> solver online with set of different B_i.
>>
>> The very initiative is the direct solver applied in precondition stage
>> is O(N^2) while the matrix-free eigen solver is O(N), thus I think
>> this would improve the efficiency of my situation. Is there any way to
>> realize it?
>>
>> Here is some rough idea:
>> To prevent the zero pivot during LU, I would dampshift A with a small
>> quantity A-\sigma I, and then compute its LU offline and store it. In
>> online stage, I could build an ST shell that read LU computed in
>> offline stage and solve. This is only an approximated approach,
>> hopefully not degrading the performance.
>>
>> Thank you!
>
> You can use STSHELL to perform any customized spectral transformation. But in your case I don't think it is necessary. [1,1,...,1] is also an eigenvector of the matrix pair (A,B), so you can pass it through EPSSetDeflationSpace, then use shift-and-invert with zero target - in that case the internal KSP will have A as the coefficient matrix and KSPSetNullSpace will be invoked with the [1,1,...,1] vector. The only thing is that you have to use a KSP solver that supports nullspaces. I am not sure if PETSc's direct solvers (or external solvers) support it, so you may have to use an iterative solver (e.g., -st_ksp_type bcgs -st_pc_type bjacobi -st_ksp_rtol 1e-9).
>
> Jose
>

From yelpoo at gmail.com  Sat May 26 08:41:21 2012
From: yelpoo at gmail.com (Ye Jianbo Bob)
Date: Sat, 26 May 2012 21:41:21 +0800
Subject: [petsc-users] SLEPc with offline preconditioner
In-Reply-To: <CAHgJxPtpx6tKXZ_b50PPsMgYXwx-2xAccJ3Yg-Ckt58D2S8kjw@mail.gmail.com>
References: <CAHgJxPurnEBBLhCtpSHZ1xxVu0r2wDnQtatgWAO1C1X96WhioQ@mail.gmail.com>
	<7C6C4F35-88F2-4894-BF0E-C94598E559D9@dsic.upv.es>
	<CAHgJxPtpx6tKXZ_b50PPsMgYXwx-2xAccJ3Yg-Ckt58D2S8kjw@mail.gmail.com>
Message-ID: <CAHgJxPuVT=N-Q27-qfkhh50=9epYs_S-8Lkbj0ysYBZx9P0Gdw@mail.gmail.com>

Some additional test with different option
------------------------------------------------------------------
./ex11 -n 30 -eps_nev 5 -eps_target 0 -st_type sinvert -st_pc_type bjacobi
Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)

 Number of iterations of the method: 5
 Solution method: krylovschur

 Number of requested eigenvalues: 5
 Stopping condition: tol=1e-08, maxit=100
 Number of converged approximate eigenpairs: 5

           k          ||Ax-kx||/||kx||
   ----------------- ------------------
       0.457508           0.886784
       0.482412           0.930914
       0.492672           0.900046
       0.513074           0.880722
       0.526509           0.898048
------------------------------------------------------------------
bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5 -st_type sinvert -st_shift
1e-9 -st_pc_type ilu
Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)

 Number of iterations of the method: 5
 Solution method: krylovschur

 Number of requested eigenvalues: 5
 Stopping condition: tol=1e-08, maxit=100
 Number of converged approximate eigenpairs: 5

           k          ||Ax-kx||/||kx||
   ----------------- ------------------
       0.457508           0.886784
       0.482412           0.930914
       0.492672           0.900046
       0.513074           0.880722
       0.526509           0.898048


On Sat, May 26, 2012 at 9:37 PM, Ye Jianbo Bob <yelpoo at gmail.com> wrote:
> Hi Jose, I found shift-and-invert does not work with an iterative pc
> solver. Say the examples given in source code
>
> /home/bobye/pub/slepc/slepc-3.2-p3/src/eps/examples/tutorials/ex11.c
> ------------------------------------------------------------------
> if I use the default setting, in fact I am not sure which pc it is
> used, even which is coefficient matrix. But it works well. However, in
> my cases, I am interested in how to reuse manipulation of A for
> different eigenvalues problem.
> bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5
>
> Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)
>
> ?Number of iterations of the method: 20
> ?Solution method: krylovschur
>
> ?Number of requested eigenvalues: 5
> ?Stopping condition: tol=1e-08, maxit=100
> ?Number of converged approximate eigenpairs: 7
>
> ? ? ? ? ? k ? ? ? ? ?||Ax-kx||/||kx||
> ? ----------------- ------------------
> ? ? ? 0.010956 ? ? ? ?1.60437e-09
> ? ? ? 0.021912 ? ? ? ?1.97096e-09
> ? ? ? 0.043705 ? ? ? ? 5.3322e-10
> ? ? ? 0.054661 ? ? ? ?4.01148e-09
> ? ? ? 0.087410 ? ? ? ? 8.7469e-10
> ? ? ? 0.097887 ? ? ? ? 2.0693e-09
> ? ? ? 0.108843 ? ? ? ?3.46988e-09
> ------------------------------------------------------------------
> if I use direct pc
> bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5 -st_type sinvert -st_pc_type lu
> It will report Detected zero pivot in LU factorization
> ------------------------------------------------------------------
> if I use direct pc with shift value
> bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5 -st_type sinvert -st_shift
> 1e-9 -st_pc_type lu
>
> Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)
>
> ?Number of iterations of the method: 1
> ?Solution method: krylovschur
>
> ?Number of requested eigenvalues: 5
> ?Stopping condition: tol=1e-08, maxit=100
> ?Number of converged approximate eigenpairs: 5
>
> ? ? ? ? ? k ? ? ? ? ?||Ax-kx||/||kx||
> ? ----------------- ------------------
> ? ? ? 0.010956 ? ? ? ?5.37411e-08
> ? ? ? 0.010956 ? ? ? ? 5.3741e-08
> ? ? ? 0.021912 ? ? ? ?5.13947e-14
> ? ? ? 0.043705 ? ? ? ?1.95903e-11
> ? ? ? 0.054661 ? ? ? ?2.62529e-09
>
> ------------------------------------------------------------------
> if I use iterative pc with shift-and-invert, the result seems to be wrong.
> bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5 -st_type sinvert -st_pc_type
> jacobi -st_ksp_rtol 1e-12
>
> Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)
>
> ?Number of iterations of the method: 1
> ?Solution method: krylovschur
>
> ?Number of requested eigenvalues: 5
> ?Stopping condition: tol=1e-08, maxit=100
> ?Number of converged approximate eigenpairs: 12
>
> ? ? ? ? ? k ? ? ? ? ?||Ax-kx||/||kx||
> ? ----------------- ------------------
> ? ? ? 2.000000 ? ? ? ? ? 0.707107
> ? ? ? 2.000000 ? ? ? ? ? 0.707107
> ? ? ? 2.000000 ? ? ? ? ? 0.707107
> ? ? ? 2.004191 ? ? ? ? ? 0.704215
> ? ? ? 3.000000 ? ? ? ? ? 0.579634
> ? ? ? 3.000000 ? ? ? ? ? 0.578826
> ? ? ? 3.000000 ? ? ? ? ? 0.580728
> ? ? ? 3.000000 ? ? ? ? ? ?0.57262
> ? ? ? 3.000000 ? ? ? ? ? 0.597176
> ? ? ? 3.000000 ? ? ? ? ? 0.562004
> ? ? ? 3.000000 ? ? ? ? ? 0.587957
> ? ? ? 3.096958 ? ? ? ? ? 0.710225
>
>
> -jianbo
>
> On Sat, May 26, 2012 at 5:47 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
>>
>> El 26/05/2012, a las 10:47, Ye Jianbo Bob escribi?:
>>
>>> Hi, I have some problem when using SLEPc to compute eigenvalue
>>> problems. I do not know how to implement my idea properly. Let me
>>> explain in detail.
>>>
>>> I am interested in solving a set of eigenvalue problems (to find the
>>> smallest magnitude eigenvalues)
>>>
>>> A x = \lambda B_i x
>>> for i=1,2,3,...
>>>
>>> where A is the Laplacian of some regular grid, B_i is diagonal. It is
>>> known that A is semi-positive with a null vector [1,1,...,1].
>>>
>>> I found SLEPc provides EPSSetDeflationSpace to set deflation space
>>> when applies iterative eigen solver. But I am not sure what default
>>> precondition is used when I defined my problem by setting
>>> eps_smallest_magnitude through EPSSetWhichEigenpairs.
>>>
>>> It is possible to use shift-and-invert to explicitly address my
>>> problem. If the shift value is 0, zero pivot will be reported during
>>> LU precondition stage. But since my set of eigen problems has the
>>> exact the same A, I hope I could somehow do the precondition (direct
>>> solver) offline only associate with A and apply the iterative eigen
>>> solver online with set of different B_i.
>>>
>>> The very initiative is the direct solver applied in precondition stage
>>> is O(N^2) while the matrix-free eigen solver is O(N), thus I think
>>> this would improve the efficiency of my situation. Is there any way to
>>> realize it?
>>>
>>> Here is some rough idea:
>>> To prevent the zero pivot during LU, I would dampshift A with a small
>>> quantity A-\sigma I, and then compute its LU offline and store it. In
>>> online stage, I could build an ST shell that read LU computed in
>>> offline stage and solve. This is only an approximated approach,
>>> hopefully not degrading the performance.
>>>
>>> Thank you!
>>
>> You can use STSHELL to perform any customized spectral transformation. But in your case I don't think it is necessary. [1,1,...,1] is also an eigenvector of the matrix pair (A,B), so you can pass it through EPSSetDeflationSpace, then use shift-and-invert with zero target - in that case the internal KSP will have A as the coefficient matrix and KSPSetNullSpace will be invoked with the [1,1,...,1] vector. The only thing is that you have to use a KSP solver that supports nullspaces. I am not sure if PETSc's direct solvers (or external solvers) support it, so you may have to use an iterative solver (e.g., -st_ksp_type bcgs -st_pc_type bjacobi -st_ksp_rtol 1e-9).
>>
>> Jose
>>

From w_ang_temp at 163.com  Sat May 26 09:22:38 2012
From: w_ang_temp at 163.com (w_ang_temp)
Date: Sat, 26 May 2012 22:22:38 +0800 (CST)
Subject: [petsc-users] A qustion of PETSc
In-Reply-To: <CAMYG4GkfTVM6gh+dysdsoiRZjSq9J84yzKug-=Vd=mK4EFXzWA@mail.gmail.com>
References: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com>
	<CAMYG4G=TX7URdbveZA6bTs6LOi7kiDV3kbf5F=JW81JAPPYeSg@mail.gmail.com>
	<ad33a04.1a57d.13787f32753.Coremail.w_ang_temp@163.com>
	<5eedef2f.1e190.137882c844b.Coremail.w_ang_temp@163.com>
	<CAMYG4GkfTVM6gh+dysdsoiRZjSq9J84yzKug-=Vd=mK4EFXzWA@mail.gmail.com>
Message-ID: <74a86660.1c5ae.13789870c4a.Coremail.w_ang_temp@163.com>

Hello,Matt
    Thanks again.
    What you said("Use -ksp_type preonly -pc_type lu") means that I should use a direct solver.And I am puzzled.
    (1)I think this problem (single processor is ok and more than one are problematic)is typical,but I cannot
find useful infomation both in the manual and the petsc-users.So can you give me some explanation for this
phenomenon.
    (2)When I use -pc_type lu,the error message is "Matrix format mpiaij does not have a built-in PETSc LU!".
    (3)The manual says that "This approach prevents the user from the unexpected surprise of having a
corrupted matrix after a linear solve"(P72).So do you mean the problem of mine is here?
    (4)Generally I want to use the iterative approach other than the direct method.
    As a beginner,I have to learn more about the manual.And also I think an experienced senior will be a more
 useful help for me because I have gotten stuck on this problem for more than a week.
     Thanks.                                Jim




>At 2012-05-26 20:17:42,"Matthew Knepley" <knepley at gmail.com> wrote:
>Use -ksp_type preonly -pc_type lu.


 >  Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120526/1f36db6e/attachment.html>

From knepley at gmail.com  Sat May 26 09:26:20 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 26 May 2012 14:26:20 +0000
Subject: [petsc-users] A qustion of PETSc
In-Reply-To: <74a86660.1c5ae.13789870c4a.Coremail.w_ang_temp@163.com>
References: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com>
	<CAMYG4G=TX7URdbveZA6bTs6LOi7kiDV3kbf5F=JW81JAPPYeSg@mail.gmail.com>
	<ad33a04.1a57d.13787f32753.Coremail.w_ang_temp@163.com>
	<5eedef2f.1e190.137882c844b.Coremail.w_ang_temp@163.com>
	<CAMYG4GkfTVM6gh+dysdsoiRZjSq9J84yzKug-=Vd=mK4EFXzWA@mail.gmail.com>
	<74a86660.1c5ae.13789870c4a.Coremail.w_ang_temp@163.com>
Message-ID: <CAMYG4Gk_ZM4fdXQu_=PtabuizpNeX9QiGBe3idPC0pHLM_F_eA@mail.gmail.com>

On Sat, May 26, 2012 at 2:22 PM, w_ang_temp <w_ang_temp at 163.com> wrote:

> Hello,Matt
>     Thanks again.
>     What you said("Use -ksp_type preonly -pc_type lu") means that I should
> use a direct solver.And I am puzzled.
>

Okay, you want to remove variables from the investigation. You say there is
a difference between
the parallel and serial solve. We do this all the time, so it is not a bug.
It is an issue of understanding.
First, take away the tolerance issues:

  -ksp_rtol 1.0e-15 -ksp_converged_reason -ksp_monitor_true_residual

and send output of the serial and parallel run.

   Matt


>     (1)I think this problem (single processor is ok and more than one are
> problematic)is typical,but I cannot
> find useful infomation both in the manual and the petsc-users.So can you
> give me some explanation for this
> phenomenon.
>     (2)When I use -pc_type lu,the error message is "Matrix format mpiaij
> does not have a built-in PETSc LU!".
>     (3)The manual says that "This approach prevents the user from the
> unexpected surprise of having a
> corrupted matrix after a linear solve"(P72).So do you mean the problem of
> mine is here?
>     (4)Generally I want to use the iterative approach other than the
> direct method.
>     As a beginne r,I have to learn more about the manual.And also I think
> an experienced senior will be a more
>  useful help for me because I have gotten stuck on this problem for more
> than a week.
>      Thanks.                                Jim
>
>
> >At 2012-05-26 20:17:42,"Matthew Knepley" <knepley at gmail.com> wrote:
>
> >Use -ksp_type preonly -pc_type lu.
>
>  >  Matt
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120526/4cc8acfd/attachment-0001.html>

From w_ang_temp at 163.com  Sat May 26 10:05:26 2012
From: w_ang_temp at 163.com (w_ang_temp)
Date: Sat, 26 May 2012 23:05:26 +0800 (CST)
Subject: [petsc-users] A qustion of PETSc
In-Reply-To: <CAMYG4Gk_ZM4fdXQu_=PtabuizpNeX9QiGBe3idPC0pHLM_F_eA@mail.gmail.com>
References: <503130c5.1a664.13773d66431.Coremail.w_ang_temp@163.com>
	<CAMYG4G=TX7URdbveZA6bTs6LOi7kiDV3kbf5F=JW81JAPPYeSg@mail.gmail.com>
	<ad33a04.1a57d.13787f32753.Coremail.w_ang_temp@163.com>
	<5eedef2f.1e190.137882c844b.Coremail.w_ang_temp@163.com>
	<CAMYG4GkfTVM6gh+dysdsoiRZjSq9J84yzKug-=Vd=mK4EFXzWA@mail.gmail.com>
	<74a86660.1c5ae.13789870c4a.Coremail.w_ang_temp@163.com>
	<CAMYG4Gk_ZM4fdXQu_=PtabuizpNeX9QiGBe3idPC0pHLM_F_eA@mail.gmail.com>
Message-ID: <2d3e7f56.223bf.13789ae3c60.Coremail.w_ang_temp@163.com>

Hello, Matt
     The results of the parallel and the serial  are identical by the "-ksp_rtol 1.0e-15 -ksp_converged_reason -ksp_monitor_true_residual".
     I will make a carefulanalysis of these  parameters next day because it is late night here now.
     Thanks again.                                                       Jim

>? 2012-05-26 22:26:20?"Matthew Knepley" <knepley at gmail.com> ???
>On Sat, May 26, 2012 at 2:22 PM, w_ang_temp <w_ang_temp at 163.com> wrote:

>Hello,Matt
>    Thanks again.
>    What you said("Use -ksp_type preonly -pc_type lu") means that I should use a direct solver.And I am puzzled.



>Okay, you want to remove variables from the investigation. You say there is a difference between
>the parallel and serial solve. We do this all the time, so it is not a bug. It is an issue of understanding.
>First, take away the tolerance issues:


>  -ksp_rtol 1.0e-15 -ksp_converged_reason -ksp_monitor_true_residual


>and send output of the serial and parallel run.


>   Matt
 
>    (1)I think this problem (single processor is ok and more than one are problematic)is typical,but I cannot
>find useful infomation both in the manual and the petsc-users.So can you give me some explanation for this
>phenomenon.
>    (2)When I use -pc_type lu,the error message is "Matrix format mpiaij does not have a built-in PETSc LU!".
>    (3)The manual says that "This approach prevents the user from the unexpected surprise of having a
>corrupted matrix after a linear solve"(P72).So do you mean the problem of mine is here?
>    (4)Generally I want to use the iterative approach other than the direct method.
>    As a beginne r,I have to learn more about the manual.And also I think an experienced senior will be a more
> useful help for me because I have gotten stuck on this problem for more than a week.
>     Thanks.                                Jim




>At 2012-05-26 20:17:42,"Matthew Knepley" <knepley at gmail.com> wrote:
>Use -ksp_type preonly -pc_type lu.


 >  Matt








--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120526/8483637d/attachment.html>

From jroman at dsic.upv.es  Sat May 26 13:30:41 2012
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Sat, 26 May 2012 20:30:41 +0200
Subject: [petsc-users] SLEPc with offline preconditioner
In-Reply-To: <CAHgJxPuVT=N-Q27-qfkhh50=9epYs_S-8Lkbj0ysYBZx9P0Gdw@mail.gmail.com>
References: <CAHgJxPurnEBBLhCtpSHZ1xxVu0r2wDnQtatgWAO1C1X96WhioQ@mail.gmail.com>
	<7C6C4F35-88F2-4894-BF0E-C94598E559D9@dsic.upv.es>
	<CAHgJxPtpx6tKXZ_b50PPsMgYXwx-2xAccJ3Yg-Ckt58D2S8kjw@mail.gmail.com>
	<CAHgJxPuVT=N-Q27-qfkhh50=9epYs_S-8Lkbj0ysYBZx9P0Gdw@mail.gmail.com>
Message-ID: <43461D8B-6223-4814-ABB7-5A3F8F886229@dsic.upv.es>

You are putting options randomly, but always missing a relevant one.
First, have a look at the source code to see what options are set there, then change the required options and use -eps_view to see if they have been changed the way you want.

Jose

$ ./ex11 -n 30 -eps_nev 5 -eps_target 0 -st_type sinvert -st_ksp_type bcgs -st_pc_type bjacobi -st_ksp_rtol 1e-10

Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)

 Number of iterations of the method: 2
 Solution method: krylovschur

 Number of requested eigenvalues: 5
 Stopping condition: tol=1e-08, maxit=100
 Number of converged approximate eigenpairs: 5

           k          ||Ax-kx||/||kx||
   ----------------- ------------------
       0.010956        9.93776e-09
       0.010956        1.00716e-08
       0.021912        6.37777e-11
       0.043705        9.35519e-10
       0.043705        3.18199e-07





El 26/05/2012, a las 15:41, Ye Jianbo Bob escribi?:

> Some additional test with different option
> ------------------------------------------------------------------
> ./ex11 -n 30 -eps_nev 5 -eps_target 0 -st_type sinvert -st_pc_type bjacobi
> Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)
> 
> Number of iterations of the method: 5
> Solution method: krylovschur
> 
> Number of requested eigenvalues: 5
> Stopping condition: tol=1e-08, maxit=100
> Number of converged approximate eigenpairs: 5
> 
>           k          ||Ax-kx||/||kx||
>   ----------------- ------------------
>       0.457508           0.886784
>       0.482412           0.930914
>       0.492672           0.900046
>       0.513074           0.880722
>       0.526509           0.898048
> ------------------------------------------------------------------
> bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5 -st_type sinvert -st_shift
> 1e-9 -st_pc_type ilu
> Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)
> 
> Number of iterations of the method: 5
> Solution method: krylovschur
> 
> Number of requested eigenvalues: 5
> Stopping condition: tol=1e-08, maxit=100
> Number of converged approximate eigenpairs: 5
> 
>           k          ||Ax-kx||/||kx||
>   ----------------- ------------------
>       0.457508           0.886784
>       0.482412           0.930914
>       0.492672           0.900046
>       0.513074           0.880722
>       0.526509           0.898048
> 
> 
> On Sat, May 26, 2012 at 9:37 PM, Ye Jianbo Bob <yelpoo at gmail.com> wrote:
>> Hi Jose, I found shift-and-invert does not work with an iterative pc
>> solver. Say the examples given in source code
>> 
>> /home/bobye/pub/slepc/slepc-3.2-p3/src/eps/examples/tutorials/ex11.c
>> ------------------------------------------------------------------
>> if I use the default setting, in fact I am not sure which pc it is
>> used, even which is coefficient matrix. But it works well. However, in
>> my cases, I am interested in how to reuse manipulation of A for
>> different eigenvalues problem.
>> bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5
>> 
>> Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)
>> 
>>  Number of iterations of the method: 20
>>  Solution method: krylovschur
>> 
>>  Number of requested eigenvalues: 5
>>  Stopping condition: tol=1e-08, maxit=100
>>  Number of converged approximate eigenpairs: 7
>> 
>>           k          ||Ax-kx||/||kx||
>>   ----------------- ------------------
>>       0.010956        1.60437e-09
>>       0.021912        1.97096e-09
>>       0.043705         5.3322e-10
>>       0.054661        4.01148e-09
>>       0.087410         8.7469e-10
>>       0.097887         2.0693e-09
>>       0.108843        3.46988e-09
>> ------------------------------------------------------------------
>> if I use direct pc
>> bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5 -st_type sinvert -st_pc_type lu
>> It will report Detected zero pivot in LU factorization
>> ------------------------------------------------------------------
>> if I use direct pc with shift value
>> bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5 -st_type sinvert -st_shift
>> 1e-9 -st_pc_type lu
>> 
>> Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)
>> 
>>  Number of iterations of the method: 1
>>  Solution method: krylovschur
>> 
>>  Number of requested eigenvalues: 5
>>  Stopping condition: tol=1e-08, maxit=100
>>  Number of converged approximate eigenpairs: 5
>> 
>>           k          ||Ax-kx||/||kx||
>>   ----------------- ------------------
>>       0.010956        5.37411e-08
>>       0.010956         5.3741e-08
>>       0.021912        5.13947e-14
>>       0.043705        1.95903e-11
>>       0.054661        2.62529e-09
>> 
>> ------------------------------------------------------------------
>> if I use iterative pc with shift-and-invert, the result seems to be wrong.
>> bobye@:tutorials$ ./ex11 -n 30 -eps_nev 5 -st_type sinvert -st_pc_type
>> jacobi -st_ksp_rtol 1e-12
>> 
>> Fiedler vector of a 2-D regular mesh, N=900 (30x30 grid)
>> 
>>  Number of iterations of the method: 1
>>  Solution method: krylovschur
>> 
>>  Number of requested eigenvalues: 5
>>  Stopping condition: tol=1e-08, maxit=100
>>  Number of converged approximate eigenpairs: 12
>> 
>>           k          ||Ax-kx||/||kx||
>>   ----------------- ------------------
>>       2.000000           0.707107
>>       2.000000           0.707107
>>       2.000000           0.707107
>>       2.004191           0.704215
>>       3.000000           0.579634
>>       3.000000           0.578826
>>       3.000000           0.580728
>>       3.000000            0.57262
>>       3.000000           0.597176
>>       3.000000           0.562004
>>       3.000000           0.587957
>>       3.096958           0.710225
>> 
>> 
>> -jianbo
>> 
>> On Sat, May 26, 2012 at 5:47 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
>>> 
>>> El 26/05/2012, a las 10:47, Ye Jianbo Bob escribi?:
>>> 
>>>> Hi, I have some problem when using SLEPc to compute eigenvalue
>>>> problems. I do not know how to implement my idea properly. Let me
>>>> explain in detail.
>>>> 
>>>> I am interested in solving a set of eigenvalue problems (to find the
>>>> smallest magnitude eigenvalues)
>>>> 
>>>> A x = \lambda B_i x
>>>> for i=1,2,3,...
>>>> 
>>>> where A is the Laplacian of some regular grid, B_i is diagonal. It is
>>>> known that A is semi-positive with a null vector [1,1,...,1].
>>>> 
>>>> I found SLEPc provides EPSSetDeflationSpace to set deflation space
>>>> when applies iterative eigen solver. But I am not sure what default
>>>> precondition is used when I defined my problem by setting
>>>> eps_smallest_magnitude through EPSSetWhichEigenpairs.
>>>> 
>>>> It is possible to use shift-and-invert to explicitly address my
>>>> problem. If the shift value is 0, zero pivot will be reported during
>>>> LU precondition stage. But since my set of eigen problems has the
>>>> exact the same A, I hope I could somehow do the precondition (direct
>>>> solver) offline only associate with A and apply the iterative eigen
>>>> solver online with set of different B_i.
>>>> 
>>>> The very initiative is the direct solver applied in precondition stage
>>>> is O(N^2) while the matrix-free eigen solver is O(N), thus I think
>>>> this would improve the efficiency of my situation. Is there any way to
>>>> realize it?
>>>> 
>>>> Here is some rough idea:
>>>> To prevent the zero pivot during LU, I would dampshift A with a small
>>>> quantity A-\sigma I, and then compute its LU offline and store it. In
>>>> online stage, I could build an ST shell that read LU computed in
>>>> offline stage and solve. This is only an approximated approach,
>>>> hopefully not degrading the performance.
>>>> 
>>>> Thank you!
>>> 
>>> You can use STSHELL to perform any customized spectral transformation. But in your case I don't think it is necessary. [1,1,...,1] is also an eigenvector of the matrix pair (A,B), so you can pass it through EPSSetDeflationSpace, then use shift-and-invert with zero target - in that case the internal KSP will have A as the coefficient matrix and KSPSetNullSpace will be invoked with the [1,1,...,1] vector. The only thing is that you have to use a KSP solver that supports nullspaces. I am not sure if PETSc's direct solvers (or external solvers) support it, so you may have to use an iterative solver (e.g., -st_ksp_type bcgs -st_pc_type bjacobi -st_ksp_rtol 1e-9).
>>> 
>>> Jose
>>> 


From roland at wb.tu-darmstadt.de  Sat May 26 17:13:33 2012
From: roland at wb.tu-darmstadt.de (Aron Roland)
Date: Sun, 27 May 2012 00:13:33 +0200
Subject: [petsc-users] scaling problem
Message-ID: <4FC1558D.6040400@wb.tu-darmstadt.de>

Dear All,

I have some question on some recent implementation of PETSc for solving 
a large linear system from a 4d problem on hybrid unstructured meshes.

The point is that we have implemented all the mappings and the solution 
is fine, the number of iterations too. The results are robust with 
respect to the amount of CPU used but we have a scaling issue.

The system is an intel cluster of the latest generation on Infiniband.

We have attached the summary ... with hooefully a lot of informations.

Any comments, suggestions, ideas are very welcome.

We have been reading the threads with that are dealing with multi-core 
and the bus-limitation stuff, so we are aware of this.

I am thinking now on an open/mpi hybrid stuff but I am not quite happy 
with the bus-limitation explanation, most of the systems are multicore.

I hope the limitation are not the sparse matrix mapping that we are 
using ...

Thanks in advance ...

Cheers

Aron



*
*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120527/571f656d/attachment.html>
-------------- next part --------------
PuertoRico Ymir
Nodes:  124815

25.5.2012

Solver: KSPLGMRES
Preconditioner: PCMG

Matrix Size: 71,893,440 x 71,893,440
Total matrix NNZ: 482,500,000

MSC = MCD = 24
dt = 60sec
Simulationszeit 20min
20 Zeitschritte (20 calls to KSPSolve)

                    FLUCT        Solver    Solver       Solver    Eff.    App          App        Eff.
Nodes    Threads    time[sec]    Iter      Time[sec]    Speedup   [%]     Time[sec]    Speedup    [%]
  4         8         33-38        7-9        18.8        3.2     40         998          4.5      56
  5        25         10-11        8-10        6.85       8.8     35         358         12.6      50
  4        32         10-12        8-11        6.45       9.3     29         317         14.2      44
  8        32         12-14        8-11        8.75       6.9     22         392         11.5      36
 10        40         11-13        9-10        7.10       8.5     21         355         12.7      32
 10        50          7-8         8-11        5.10      11.8     24         252         17.9      36

---------------------------------------------- PETSc Performance Summary: ----------------------------------------------

/Utilisateurs/aroland/bin/selfewwm_thomas on a linux-int named r1i3n7 with 40 processors, by aroland Fri May 25 19:17:22 2012
Using Petsc Release Version 3.2.0, Patch 6, Wed Jan 11 09:28:45 CST 2012

                         Max       Max/Min        Avg      Total
Time (sec):           3.359e+02      1.00000   3.359e+02
Objects:              1.400e+02      1.00000   1.400e+02
Flops:                3.776e+10      1.14746   3.558e+10  1.423e+12
Flops/sec:            1.124e+08      1.14746   1.059e+08  4.237e+09
MPI Messages:         5.280e+03      6.00000   2.442e+03  9.768e+04
MPI Message Lengths:  5.991e+08      3.80128   1.469e+05  1.435e+10
MPI Reductions:       1.406e+03      1.00000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flops -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total
 0:      Main Stage: 3.3594e+02 100.0%  1.4233e+12 100.0%  9.768e+04 100.0%  1.469e+05      100.0%  1.405e+03  99.9%


------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

MatMult              438 1.0 4.4104e+01 1.9 1.06e+10 1.1 9.7e+04 1.5e+05 0.0e+00 11 28100100  0  11 28100100  0  9131
MatSolve             458 1.0 3.6361e+01 2.8 1.09e+10 1.1 0.0e+00 0.0e+00 0.0e+00 10 29  0  0  0  10 29  0  0  0 11380
MatLUFactorNum        20 1.0 6.9722e+00 2.7 4.73e+0810.5 0.0e+00 0.0e+00 0.0e+00  1  1  0  0  0   1  1  0  0  0  1534
MatILUFactorSym        1 1.0 2.2795e-01 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyBegin      45 1.0 8.7023e-05 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd        45 1.0 2.9763e+00 1.7 0.00e+00 0.0 4.4e+02 3.7e+04 8.0e+00  1  0  0  0  1   1  0  0  0  1     0
MatGetRowIJ            1 1.0 6.1989e-06 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetSubMatrice      20 1.0 5.9878e+00 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  2  0  0  0  0   2  0  0  0  0     0
MatGetOrdering         1 1.0 1.8968e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatZeroEntries        19 1.0 9.1826e-01 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecMax                20 1.0 1.0160e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+01  0  0  0  0  1   0  0  0  0  1     0
VecMin                20 1.0 1.0970e+0012.8 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+01  0  0  0  0  1   0  0  0  0  1     0
VecMDot              418 1.0 3.4962e+01 4.9 4.69e+09 1.2 0.0e+00 0.0e+00 4.2e+02  4 12  0  0 30   4 12  0  0 30  5013
VecNorm              687 1.0 4.1451e+01 5.3 2.64e+09 1.2 0.0e+00 0.0e+00 6.9e+02  5  7  0  0 49   5  7  0  0 49  2383
VecScale             667 1.0 5.9862e+00 3.2 1.28e+09 1.2 0.0e+00 0.0e+00 0.0e+00  2  3  0  0  0   2  3  0  0  0  8011
VecCopy              269 1.0 2.2040e+00 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
VecSet              1225 1.0 8.5033e+00 2.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  2  0  0  0  0   2  0  0  0  0     0
VecAXPY              269 1.0 3.6360e+00 2.7 1.03e+09 1.2 0.0e+00 0.0e+00 0.0e+00  1  3  0  0  0   1  3  0  0  0 10638
VecMAXPY             667 1.0 1.5898e+01 2.9 6.30e+09 1.2 0.0e+00 0.0e+00 0.0e+00  4 17  0  0  0   4 17  0  0  0 14806
VecAssemblyBegin      40 1.0 2.4674e+00297.8 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+02  1  0  0  0  9   1  0  0  0  9     0
VecAssemblyEnd        40 1.0 1.2517e-04 4.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin      438 1.0 1.1889e+00 9.1 0.00e+00 0.0 9.7e+04 1.5e+05 0.0e+00  0  0100100  0   0  0100100  0     0
VecScatterEnd        438 1.0 2.2600e+0189.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
VecNormalize         458 1.0 3.5072e+01 3.8 2.64e+09 1.2 0.0e+00 0.0e+00 4.6e+02  5  7  0  0 33   5  7  0  0 33  2817
KSPGMRESOrthog       418 1.0 3.9814e+01 2.2 9.38e+09 1.2 0.0e+00 0.0e+00 4.2e+02  7 25  0  0 30   7 25  0  0 30  8805
KSPSetup              60 1.0 1.7510e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  1   0  0  0  0  1     0
KSPSolve              20 1.0 1.4208e+02 1.0 3.78e+10 1.1 9.7e+04 1.5e+05 1.2e+03 42100100100 82  42100100100 82 10017
PCSetUp               40 1.0 1.3285e+01 2.1 4.73e+0810.5 0.0e+00 0.0e+00 1.2e+01  3  1  0  0  1   3  1  0  0  1   805
PCSetUpOnBlocks      687 1.0 4.4281e+01 2.5 1.13e+10 1.1 0.0e+00 0.0e+00 3.0e+00 12 30  0  0  0  12 30  0  0  0  9586
PCApply              229 1.0 1.0103e+02 1.1 2.30e+10 1.1 5.1e+04 1.5e+05 6.9e+02 29 61 52 52 49  29 61 52 52 49  8567

------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

           Index Set    10             10      7700472     0
   IS L to G Mapping     1              1          564     0
   Application Order     1              1       999160     0
              Matrix     5              5    386024408     0
              Vector   114            114   1062454512     0
      Vector Scatter     2              2         2072     0
       Krylov Solver     3              3        37800     0
      Preconditioner     3              3         2800     0
              Viewer     1              0            0     0

========================================================================================================================
Average time to get PetscTime(): 0
Average time for MPI_Barrier(): 3.93867e-05
Average time for zero size MPI_Send(): 4.75049e-06
#PETSc Option Table entries:
-log_summary
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8
Configure run at: Thu Apr 26 22:05:34 2012
Configure options: --prefix=/Utilisateurs/aroland/thomas/opt/petsc_3.2-p6 --download-f-blas-lapack=1 --with-mpi-dir=/Utilisateurs/aroland/thomas/opt/mpich2/ --with-superlu_dist=true --download-superlu_dist=yes --with-parmetis-lib="[/Utilisateurs/aroland/opt/ParMetis-3.1-64bit/libparmetis.a,/Utilisateurs/aroland/opt/ParMetis-3.1-64bit/libmetis.a]" --with-parmetis-include=/Utilisateurs/aroland/opt/ParMetis-3.1-64bit/ --with-debugging=0 COPTFLAGS=-O3 FOPTFLAGS=-O3
-----------------------------------------
Libraries compiled on Thu Apr 26 22:05:34 2012 on service1
Machine characteristics: Linux-2.6.32.12-0.7-default-x86_64-with-SuSE-11-x86_64
Using PETSc directory: /Utilisateurs/aroland/thomas/opt/downloads/petsc-3.2-p6
Using PETSc arch: linux-intel-performance
-----------------------------------------

Using C compiler: /Utilisateurs/aroland/thomas/opt/mpich2/bin/mpicc  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O3  ${COPTFLAGS} ${CFLAGS}
Using Fortran compiler: /Utilisateurs/aroland/thomas/opt/mpich2/bin/mpif90  -O3   ${FOPTFLAGS} ${FFLAGS}
-----------------------------------------

Using include paths: -I/Utilisateurs/aroland/thomas/opt/downloads/petsc-3.2-p6/linux-intel-performance/include -I/Utilisateurs/aroland/thomas/opt/downloads/petsc-3.2-p6/include -I/Utilisateurs/aroland/thomas/opt/downloads/petsc-3.2-p6/include -I/Utilisateurs/aroland/thomas/opt/downloads/petsc-3.2-p6/linux-intel-performance/include -I/Utilisateurs/aroland/opt/ParMetis-3.1-64bit -I/Utilisateurs/aroland/thomas/opt/mpich2/include -I/Utilisateurs/aroland/thomas/opt/mpich2-1.4.1p1/include
-----------------------------------------

Using C linker: /Utilisateurs/aroland/thomas/opt/mpich2/bin/mpicc
Using Fortran linker: /Utilisateurs/aroland/thomas/opt/mpich2/bin/mpif90
Using libraries: -Wl,-rpath,/Utilisateurs/aroland/thomas/opt/downloads/petsc-3.2-p6/linux-intel-performance/lib -L/Utilisateurs/aroland/thomas/opt/downloads/petsc-3.2-p6/linux-intel-performance/lib -lpetsc -lX11 -lpthread -Wl,-rpath,/Utilisateurs/aroland/thomas/opt/downloads/petsc-3.2-p6/linux-intel-performance/lib -L/Utilisateurs/aroland/thomas/opt/downloads/petsc-3.2-p6/linux-intel-performance/lib -lsuperlu_dist_2.5 -Wl,-rpath,/Utilisateurs/aroland/opt/ParMetis-3.1-64bit -L/Utilisateurs/aroland/opt/ParMetis-3.1-64bit -lparmetis -lmetis -lflapack -lfblas -lm -L/Utilisateurs/aroland/thomas/opt/mpich2-1.4.1p1/lib -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/Calcul/Apps2/intel/composer_xe_2011_sp1.6.233/compiler/lib/intel64 -L/Calcul/Apps2/intel/composer_xe_2011_sp1.6.233/ipp/lib/intel64 -L/Calcul/Apps2/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64 -L/Calcul/Apps2/intel/composer_xe_2011_sp1.6.233/tbb/lib/intel64/cc4.1.0_libc2.4_kernel2.6.16.21 -L/Calcul/Apps/intel/impi/4.0.3.008/lib64 -L/usr/x86_64-suse-linux/lib -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -lmpichf90 -lifport -lifcore -limf -lsvml -lm -lipgo -lirc -lirc_s -lm -ldl -lmpich -lopa -lmpl -lrt -lpthread -lgcc_s -ldl
-----------------------------------------

From mark.adams at columbia.edu  Sat May 26 17:28:33 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Sat, 26 May 2012 18:28:33 -0400
Subject: [petsc-users] scaling problem
In-Reply-To: <4FC1558D.6040400@wb.tu-darmstadt.de>
References: <4FC1558D.6040400@wb.tu-darmstadt.de>
Message-ID: <FC70406D-BB25-470B-82BC-7B1DA73EAE11@columbia.edu>

Just a few comments on the perf data.  I see a lot (~5x) load imbalance on reduction stuff like VecMDot and VecNorm (Time Ratio, max/min) but even a lot of imbalance on simple non-collective vector ops like VecAXPY (~3x).  So if the work is load balanced well then I would suspect its a NUMA issue.

Mark

On May 26, 2012, at 6:13 PM, Aron Roland wrote:

> Dear All,
> 
> I have some question on some recent implementation of PETSc for solving a large linear system from a 4d problem on hybrid unstructured meshes. 
> 
> The point is that we have implemented all the mappings and the solution is fine, the number of iterations too. The results are robust with respect to the amount of CPU used but we have a scaling issue. 
> 
> The system is an intel cluster of the latest generation on Infiniband.
> 
> We have attached the summary ... with hooefully a lot of informations. 
> 
> Any comments, suggestions, ideas are very welcome. 
> 
> We have been reading the threads with that are dealing with multi-core and the bus-limitation stuff, so we are aware of this. 
> 
> I am thinking now on an open/mpi hybrid stuff but I am not quite happy with the bus-limitation explanation, most of the systems are multicore. 
> 
> I hope the limitation are not the sparse matrix mapping that we are using ... 
> 
> Thanks in advance ...
> 
> Cheers
> 
> Aron 
> 
> 
> 
> 
> 
> <benchmark.txt>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120526/3a82de0e/attachment.html>

From jedbrown at mcs.anl.gov  Sat May 26 17:32:37 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sat, 26 May 2012 17:32:37 -0500
Subject: [petsc-users] scaling problem
In-Reply-To: <FC70406D-BB25-470B-82BC-7B1DA73EAE11@columbia.edu>
References: <4FC1558D.6040400@wb.tu-darmstadt.de>
	<FC70406D-BB25-470B-82BC-7B1DA73EAE11@columbia.edu>
Message-ID: <CAM9tzSncsrn_-UnT1k4VrWkkDeRR6t3yEyPebvT1cQNTXwrdfQ@mail.gmail.com>

That load imbalance often comes from whatever came *before* the reduction.
On May 26, 2012 5:25 PM, "Mark F. Adams" <mark.adams at columbia.edu> wrote:

> Just a few comments on the perf data.  I see a lot (~5x) load imbalance on
> reduction stuff like VecMDot and VecNorm (Time Ratio, max/min) but even a
> lot of imbalance on simple non-collective vector ops like VecAXPY (~3x).
>  So if the work is load balanced well then I would suspect its a NUMA issue.
>
> Mark
>
> On May 26, 2012, at 6:13 PM, Aron Roland wrote:
>
>  Dear All,
>
> I have some question on some recent implementation of PETSc for solving a
> large linear system from a 4d problem on hybrid unstructured meshes.
>
> The point is that we have implemented all the mappings and the solution is
> fine, the number of iterations too. The results are robust with respect to
> the amount of CPU used but we have a scaling issue.
>
> The system is an intel cluster of the latest generation on Infiniband.
>
> We have attached the summary ... with hooefully a lot of informations.
>
> Any comments, suggestions, ideas are very welcome.
>
> We have been reading the threads with that are dealing with multi-core and
> the bus-limitation stuff, so we are aware of this.
>
> I am thinking now on an open/mpi hybrid stuff but I am not quite happy
> with the bus-limitation explanation, most of the systems are multicore.
>
> I hope the limitation are not the sparse matrix mapping that we are using
> ...
>
> Thanks in advance ...
>
> Cheers
>
> Aron
>
>
>
> *
> *
>  <benchmark.txt>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120526/c039586a/attachment-0001.html>

From mark.adams at columbia.edu  Sat May 26 18:34:19 2012
From: mark.adams at columbia.edu (Mark F. Adams)
Date: Sat, 26 May 2012 19:34:19 -0400
Subject: [petsc-users] scaling problem
In-Reply-To: <CAM9tzSncsrn_-UnT1k4VrWkkDeRR6t3yEyPebvT1cQNTXwrdfQ@mail.gmail.com>
References: <4FC1558D.6040400@wb.tu-darmstadt.de>
	<FC70406D-BB25-470B-82BC-7B1DA73EAE11@columbia.edu>
	<CAM9tzSncsrn_-UnT1k4VrWkkDeRR6t3yEyPebvT1cQNTXwrdfQ@mail.gmail.com>
Message-ID: <DEC980B1-071A-4251-915F-3E5AD597C07D@columbia.edu>


On May 26, 2012, at 6:32 PM, Jed Brown wrote:

> That load imbalance often comes from whatever came *before* the reduction.
> 
> 
Yes I assumed that was understood.

> On May 26, 2012 5:25 PM, "Mark F. Adams" <mark.adams at columbia.edu> wrote:
> Just a few comments on the perf data.  I see a lot (~5x) load imbalance on reduction stuff like VecMDot and VecNorm (Time Ratio, max/min) but even a lot of imbalance on simple non-collective vector ops like VecAXPY (~3x).  So if the work is load balanced well then I would suspect its a NUMA issue.
> 
> Mark
> 
> On May 26, 2012, at 6:13 PM, Aron Roland wrote:
> 
>> Dear All,
>> 
>> I have some question on some recent implementation of PETSc for solving a large linear system from a 4d problem on hybrid unstructured meshes. 
>> 
>> The point is that we have implemented all the mappings and the solution is fine, the number of iterations too. The results are robust with respect to the amount of CPU used but we have a scaling issue. 
>> 
>> The system is an intel cluster of the latest generation on Infiniband.
>> 
>> We have attached the summary ... with hooefully a lot of informations. 
>> 
>> Any comments, suggestions, ideas are very welcome. 
>> 
>> We have been reading the threads with that are dealing with multi-core and the bus-limitation stuff, so we are aware of this. 
>> 
>> I am thinking now on an open/mpi hybrid stuff but I am not quite happy with the bus-limitation explanation, most of the systems are multicore. 
>> 
>> I hope the limitation are not the sparse matrix mapping that we are using ... 
>> 
>> Thanks in advance ...
>> 
>> Cheers
>> 
>> Aron 
>> 
>> 
>> 
>> 
>> 
>> <benchmark.txt>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120526/1f4192f0/attachment.html>

From bsmith at mcs.anl.gov  Sat May 26 21:48:01 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 26 May 2012 21:48:01 -0500
Subject: [petsc-users] scaling problem
In-Reply-To: <4FC1558D.6040400@wb.tu-darmstadt.de>
References: <4FC1558D.6040400@wb.tu-darmstadt.de>
Message-ID: <87632305-D3A3-4220-9B1C-E64D1AE1151E@mcs.anl.gov>


   Aron,

     Please send to petsc-maint at mcs.anl.gov (to save petsc-user bandwidth) the -log_summary for 1 node 1 core, 1 node 2 core, 1 node 4 core, 1 node 8 core, 2 node, 1 core per node, 4 node 1 core per node, 8 core 1 core per node, 10 node, 1 core per node.

   Barry

On May 26, 2012, at 5:13 PM, Aron Roland wrote:

> Dear All,
> 
> I have some question on some recent implementation of PETSc for solving a large linear system from a 4d problem on hybrid unstructured meshes. 
> 
> The point is that we have implemented all the mappings and the solution is fine, the number of iterations too. The results are robust with respect to the amount of CPU used but we have a scaling issue. 
> 
> The system is an intel cluster of the latest generation on Infiniband.
> 
> We have attached the summary ... with hooefully a lot of informations. 
> 
> Any comments, suggestions, ideas are very welcome. 
> 
> We have been reading the threads with that are dealing with multi-core and the bus-limitation stuff, so we are aware of this. 
> 
> I am thinking now on an open/mpi hybrid stuff but I am not quite happy with the bus-limitation explanation, most of the systems are multicore. 
> 
> I hope the limitation are not the sparse matrix mapping that we are using ... 
> 
> Thanks in advance ...
> 
> Cheers
> 
> Aron 
> 
> 
> 
> 
> 
> <benchmark.txt>


From mike.hui.zhang at hotmail.com  Sun May 27 03:54:39 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Sun, 27 May 2012 10:54:39 +0200
Subject: [petsc-users] MatSetValues: reorder the values
Message-ID: <BLU0-SMTP42ED906F03580EBB5E5D59CB070@phx.gbl>

Current MatSetValues is
MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv)
I want something like
MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[], const PetscInt idxv[], InsertMode addv)
         -----------------------
         add a new parameter

such that the values v[0..m*n] are first reordered to v[idxv[0..m*n]] before set to the mat.
We may first do permutation on v before passing to current MatSetValues.  But won't the 
new function be easier to use? 

In the case that idxm[], idxn[] are obtained from a certain application ordering
but not the ordering of v[], I think I need the new function.
Of course, this new function sounds unnecessary because reordering v is equivalent to reordering 
idxm[] and idxn[].  But the second method seems not an easy task to me, is it?

Thanks!  



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120527/d1c05462/attachment.html>

From mike.hui.zhang at hotmail.com  Sun May 27 04:01:07 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Sun, 27 May 2012 11:01:07 +0200
Subject: [petsc-users] MatSetValues: reorder the values
In-Reply-To: <BLU0-SMTP42ED906F03580EBB5E5D59CB070@phx.gbl>
References: <BLU0-SMTP42ED906F03580EBB5E5D59CB070@phx.gbl>
Message-ID: <BLU0-SMTP1796F29ED41E091A5F7078DCB070@phx.gbl>

Also, in the case that only a *subset* of v[] is used, the new function sounds easy to me.


> Current MatSetValues is
> MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv)
> I want something like
> MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[], const PetscInt idxv[], InsertMode addv)
>          -----------------------
>          add a new parameter
> 
> such that the values v[0..m*n] are first reordered to v[idxv[0..m*n]] before set to the mat.
> We may first do permutation on v before passing to current MatSetValues.  But won't the 
> new function be easier to use? 
> 
> In the case that idxm[], idxn[] are obtained from a certain application ordering
> but not the ordering of v[], I think I need the new function.
> Of course, this new function sounds unnecessary because reordering v is equivalent to reordering 
> idxm[] and idxn[].  But the second method seems not an easy task to me, is it?
> 
> Thanks!  
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120527/a11ddef8/attachment.html>

From knepley at gmail.com  Sun May 27 06:55:45 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 27 May 2012 11:55:45 +0000
Subject: [petsc-users] MatSetValues: reorder the values
In-Reply-To: <BLU0-SMTP1796F29ED41E091A5F7078DCB070@phx.gbl>
References: <BLU0-SMTP42ED906F03580EBB5E5D59CB070@phx.gbl>
	<BLU0-SMTP1796F29ED41E091A5F7078DCB070@phx.gbl>
Message-ID: <CAMYG4GmLzbmm6QzdKsWBUzzSOJ_fA32-OYpB1b8r3tGkVVWbTg@mail.gmail.com>

On Sun, May 27, 2012 at 9:01 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Also, in the case that only a *subset* of v[] is used, the new function
> sounds easy to me.
>

The problem here is that this is one of the most used functions in PETSc.
Thus we want to
keep it is simple and fast as possible. Another level of indirection slows
it down, and having
a parameter most people do not use is confusing. I suggest you write this
wapper for your
own code and see how it works out.

   Matt


> Current MatSetValues is
>
> MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv)
>
> I want something like
>
> MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[], const PetscInt idxv[], InsertMode addv)
>
> -----------------------
> add a new parameter
>
> such that the values v[0..m*n] are first reordered to v[idxv[0..m*n]]
> before set to the mat.
> We may first do permutation on v before passing to current MatSetValues.
> But won't the
> new function be easier to use?
>
> In the case that idxm[], idxn[] are obtained from a certain application ordering
> but not the ordering of v[], I think I need the new function.
>
> Of course, this new function sounds unnecessary because reordering v is
> equivalent to reordering
> idxm[] and idxn[]. But the second method seems not an easy task to me, is
> it?
>
> Thanks!
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120527/2ee00d08/attachment.html>

From karpeev at mcs.anl.gov  Sun May 27 07:12:34 2012
From: karpeev at mcs.anl.gov (Dmitry Karpeev)
Date: Sun, 27 May 2012 07:12:34 -0500
Subject: [petsc-users] MatSetValues: reorder the values
In-Reply-To: <BLU0-SMTP1796F29ED41E091A5F7078DCB070@phx.gbl>
References: <BLU0-SMTP42ED906F03580EBB5E5D59CB070@phx.gbl>
	<BLU0-SMTP1796F29ED41E091A5F7078DCB070@phx.gbl>
Message-ID: <CA+Y_9UK5hdC6jwr_5ryzXv2TW9cNTFY=OUz6CaWUj27yOMrMNA@mail.gmail.com>

Have you looked at MatSetValuesLocal()
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetValuesLocal.html?

That routine might be close enough to what you need.
Dmitry.

On Sun, May 27, 2012 at 4:01 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Also, in the case that only a *subset* of v[] is used, the new function
> sounds easy to me.
>
>
>
>
> Current MatSetValues is
>
> MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv)
>
> I want something like
>
>
> MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[], const PetscInt idxv[], InsertMode addv)
>
> -----------------------
> add a new parameter
>
> such that the values v[0..m*n] are first reordered to v[idxv[0..m*n]]
> before set to the mat.
> We may first do permutation on v before passing to current MatSetValues.
> But won't the
> new function be easier to use?
>
> In the case that idxm[], idxn[] are obtained from a certain application ordering
> but not the ordering of v[], I think I need the new function.
>
> Of course, this new function sounds unnecessary because reordering v is
> equivalent to reordering
> idxm[] and idxn[]. But the second method seems not an easy task to me, is
> it?
>
> Thanks!
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120527/3c6fe56f/attachment-0001.html>

From mike.hui.zhang at hotmail.com  Sun May 27 07:32:46 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Sun, 27 May 2012 14:32:46 +0200
Subject: [petsc-users] MatSetValues: reorder the values
In-Reply-To: <CA+Y_9UK5hdC6jwr_5ryzXv2TW9cNTFY=OUz6CaWUj27yOMrMNA@mail.gmail.com>
References: <BLU0-SMTP42ED906F03580EBB5E5D59CB070@phx.gbl>
	<BLU0-SMTP1796F29ED41E091A5F7078DCB070@phx.gbl>
	<CA+Y_9UK5hdC6jwr_5ryzXv2TW9cNTFY=OUz6CaWUj27yOMrMNA@mail.gmail.com>
Message-ID: <BLU0-SMTP227B6C71C41CE92346DB6C9CB070@phx.gbl>

Thanks for the hint! I should your advice.

On May 27, 2012, at 2:12 PM, Dmitry Karpeev wrote:

> Have you looked at MatSetValuesLocal() 
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetValuesLocal.html?  
> That routine might be close enough to what you need.
> Dmitry.
> 
> On Sun, May 27, 2012 at 4:01 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> Also, in the case that only a *subset* of v[] is used, the new function sounds easy to me.
> 
> 
> 
>> 
>> Current MatSetValues is
>> MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv)
>> I want something like
>> 
>> 
>> MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[], const PetscInt idxv[], InsertMode addv)
>>          -----------------------
>>          add a new parameter
>> 
>> 
>> 
>> such that the values v[0..m*n] are first reordered to v[idxv[0..m*n]] before set to the mat.
>> We may first do permutation on v before passing to current MatSetValues.  But won't the 
>> 
>> 
>> new function be easier to use? 
>> 
>> 
>> In the case that idxm[], idxn[] are obtained from a certain application ordering
>> but not the ordering of v[], I think I need the new function.
>> Of course, this new function sounds unnecessary because reordering v is equivalent to reordering 
>> 
>> 
>> idxm[] and idxn[].  But the second method seems not an easy task to me, is it?
>> 
>> Thanks!  
>> 
>> 
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120527/640cfbfa/attachment.html>

From behzad.baghapour at gmail.com  Sun May 27 09:30:15 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Sun, 27 May 2012 19:00:15 +0430
Subject: [petsc-users] Adding a processing on the solution
Message-ID: <CA+jOtDLGcXriaScvBrUYWBzTsXQEnDrsT=L-2y=OZqh-mwz2Xg@mail.gmail.com>

Dear Developers,

I want to process each solution iteration of SNES by limiter function
(preserving TVD condition for fluid flow). How should I deal with this?

Thanks,
BehZad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120527/a418e86f/attachment.html>

From mike.hui.zhang at hotmail.com  Sun May 27 10:06:37 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Sun, 27 May 2012 17:06:37 +0200
Subject: [petsc-users] MatSetValues: reorder the values
In-Reply-To: <CA+Y_9UK5hdC6jwr_5ryzXv2TW9cNTFY=OUz6CaWUj27yOMrMNA@mail.gmail.com>
References: <BLU0-SMTP42ED906F03580EBB5E5D59CB070@phx.gbl>
	<BLU0-SMTP1796F29ED41E091A5F7078DCB070@phx.gbl>
	<CA+Y_9UK5hdC6jwr_5ryzXv2TW9cNTFY=OUz6CaWUj27yOMrMNA@mail.gmail.com>
Message-ID: <BLU0-SMTP415597D858E3DA3B3FBA2BFCB070@phx.gbl>

Since the values I want to take from v[] are not contiguous in v[], I found even with
MatSetValuesLocal I can not avoid to make a copy of the values(that I want to take) in
a new array vnew[]. 

On May 27, 2012, at 2:12 PM, Dmitry Karpeev wrote:

> Have you looked at MatSetValuesLocal() 
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetValuesLocal.html?  
> That routine might be close enough to what you need.
> Dmitry.
> 
> On Sun, May 27, 2012 at 4:01 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> Also, in the case that only a *subset* of v[] is used, the new function sounds easy to me.
> 
> 
> 
>> 
>> Current MatSetValues is
>> MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv)
>> I want something like
>> 
>> 
>> MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[], const PetscInt idxv[], InsertMode addv)
>>          -----------------------
>>          add a new parameter
>> 
>> 
>> 
>> such that the values v[0..m*n] are first reordered to v[idxv[0..m*n]] before set to the mat.
>> We may first do permutation on v before passing to current MatSetValues.  But won't the 
>> 
>> 
>> new function be easier to use? 
>> 
>> 
>> In the case that idxm[], idxn[] are obtained from a certain application ordering
>> but not the ordering of v[], I think I need the new function.
>> Of course, this new function sounds unnecessary because reordering v is equivalent to reordering 
>> 
>> 
>> idxm[] and idxn[].  But the second method seems not an easy task to me, is it?
>> 
>> Thanks!  
>> 
>> 
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120527/3e11f32d/attachment.html>

From jedbrown at mcs.anl.gov  Sun May 27 10:09:25 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Sun, 27 May 2012 10:09:25 -0500
Subject: [petsc-users] Adding a processing on the solution
In-Reply-To: <CA+jOtDLGcXriaScvBrUYWBzTsXQEnDrsT=L-2y=OZqh-mwz2Xg@mail.gmail.com>
References: <CA+jOtDLGcXriaScvBrUYWBzTsXQEnDrsT=L-2y=OZqh-mwz2Xg@mail.gmail.com>
Message-ID: <CAM9tzSnBU+DUM=Xrt_u3dVcUHawmMekJJLkeE_DtYo6U8+Cq0A@mail.gmail.com>

TVD methods need to modify the flux function. If you just do an unlimited
flux function and then project after each step, you lose local
conservation, which if you look in the dictionary, is the definition of
throwing the baby out with the bath water.

So put the limiter in the residual evaluation (as slope reconstruction or
flux limiting; as you prefer), but try to use a limiter that has a
continuous first derivative, otherwise Newton may converge slowly or even
stagnate before reaching a tight tolerance.

On Sun, May 27, 2012 at 9:30 AM, behzad baghapour <
behzad.baghapour at gmail.com> wrote:

> Dear Developers,
>
> I want to process each solution iteration of SNES by limiter function
> (preserving TVD condition for fluid flow). How should I deal with this?
>
> Thanks,
> BehZad
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120527/0cf40923/attachment.html>

From bsmith at mcs.anl.gov  Sun May 27 10:40:23 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 27 May 2012 10:40:23 -0500
Subject: [petsc-users] MatSetValues: reorder the values
In-Reply-To: <BLU0-SMTP415597D858E3DA3B3FBA2BFCB070@phx.gbl>
References: <BLU0-SMTP42ED906F03580EBB5E5D59CB070@phx.gbl>
	<BLU0-SMTP1796F29ED41E091A5F7078DCB070@phx.gbl>
	<CA+Y_9UK5hdC6jwr_5ryzXv2TW9cNTFY=OUz6CaWUj27yOMrMNA@mail.gmail.com>
	<BLU0-SMTP415597D858E3DA3B3FBA2BFCB070@phx.gbl>
Message-ID: <7518A336-1181-493C-A17F-60F709E69487@mcs.anl.gov>


On May 27, 2012, at 10:06 AM, Hui Zhang wrote:

> Since the values I want to take from v[] are not contiguous in v[], I found even with
> MatSetValuesLocal I can not avoid to make a copy of the values(that I want to take) in
> a new array vnew[]. 

   That extra copy is inexpensive compared to all the other overheads of MatSetValues() it will be barely noticeable.

   Barry


> 
> On May 27, 2012, at 2:12 PM, Dmitry Karpeev wrote:
> 
>> Have you looked at MatSetValuesLocal() 
>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetValuesLocal.html?  
>> That routine might be close enough to what you need.
>> Dmitry.
>> 
>> On Sun, May 27, 2012 at 4:01 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> Also, in the case that only a *subset* of v[] is used, the new function sounds easy to me.
>> 
>> 
>> 
>>> Current MatSetValues is
>>> MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv)
>>> I want something like
>>> MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[], const PetscInt idxv[], InsertMode addv)
>>>          -----------------------
>>>          add a new parameter
>>> 
>>> 
>>> 
>>> 
>>> such that the values v[0..m*n] are first reordered to v[idxv[0..m*n]] before set to the mat.
>>> We may first do permutation on v before passing to current MatSetValues.  But won't the 
>>> 
>>> 
>>> 
>>> new function be easier to use? 
>>> 
>>> In the case that idxm[], idxn[] are obtained from a certain application ordering
>>> but not the ordering of v[], I think I need the new function.
>>> Of course, this new function sounds unnecessary because reordering v is equivalent to reordering 
>>> 
>>> 
>>> 
>>> idxm[] and idxn[].  But the second method seems not an easy task to me, is it?
>>> 
>>> Thanks!  
>>> 
>>> 
>>> 
>> 
>> 
> 


From mirzadeh at gmail.com  Sun May 27 13:58:11 2012
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Sun, 27 May 2012 11:58:11 -0700
Subject: [petsc-users] MatSetValues: reorder the values
In-Reply-To: <7518A336-1181-493C-A17F-60F709E69487@mcs.anl.gov>
References: <BLU0-SMTP42ED906F03580EBB5E5D59CB070@phx.gbl>
	<BLU0-SMTP1796F29ED41E091A5F7078DCB070@phx.gbl>
	<CA+Y_9UK5hdC6jwr_5ryzXv2TW9cNTFY=OUz6CaWUj27yOMrMNA@mail.gmail.com>
	<BLU0-SMTP415597D858E3DA3B3FBA2BFCB070@phx.gbl>
	<7518A336-1181-493C-A17F-60F709E69487@mcs.anl.gov>
Message-ID: <CAKsPKwn4zTy71Du8Hu_iGQ1KMCmuz2AfQVzvZUTsh5gvsjNmgg@mail.gmail.com>

> Of course, this new function sounds unnecessary because reordering v is equivalent to reordering
> idxm[] and idxn[].  But the second method seems not an easy task to me, is it?


Why not use a single AO object to hold the mapping between PETSc mapping
and your maping? Seems to me that the indecies used in v[]
are completely arbitrary; they should always go in [0,1,...,m*n], don't
they?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120527/3c6e429a/attachment.html>

From mike.hui.zhang at hotmail.com  Sun May 27 14:17:51 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Sun, 27 May 2012 21:17:51 +0200
Subject: [petsc-users] MatSetValues: reorder the values
In-Reply-To: <CAKsPKwn4zTy71Du8Hu_iGQ1KMCmuz2AfQVzvZUTsh5gvsjNmgg@mail.gmail.com>
References: <BLU0-SMTP42ED906F03580EBB5E5D59CB070@phx.gbl>
	<BLU0-SMTP1796F29ED41E091A5F7078DCB070@phx.gbl>
	<CA+Y_9UK5hdC6jwr_5ryzXv2TW9cNTFY=OUz6CaWUj27yOMrMNA@mail.gmail.com>
	<BLU0-SMTP415597D858E3DA3B3FBA2BFCB070@phx.gbl>
	<7518A336-1181-493C-A17F-60F709E69487@mcs.anl.gov>
	<CAKsPKwn4zTy71Du8Hu_iGQ1KMCmuz2AfQVzvZUTsh5gvsjNmgg@mail.gmail.com>
Message-ID: <BLU0-SMTP564813F8EF214403E42745CB070@phx.gbl>


On May 27, 2012, at 8:58 PM, Mohammad Mirzadeh wrote:

> > Of course, this new function sounds unnecessary because reordering v is equivalent to reordering 
> > idxm[] and idxn[].  But the second method seems not an easy task to me, is it?
> 
> 
> 
> Why not use a single AO object to hold the mapping between PETSc mapping and your maping?

Yes, thanks! I'm doing so.

> Seems to me that the indecies used in v[] are completely arbitrary; they should always go in [0,1,...,m*n], don't they?

No, I only take some non-contiguous entries of v[] to the mat. So, I need to first take out
those values to a new array, what I worried about the cost. But as Barry Smith told us, 
the cost is small.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120527/b02b52ff/attachment.html>

From kenway at utias.utoronto.ca  Sun May 27 15:45:51 2012
From: kenway at utias.utoronto.ca (Gaetan Kenway)
Date: Sun, 27 May 2012 16:45:51 -0400
Subject: [petsc-users] Alternative to VecGetArrayF90
Message-ID: <CAA3=JrNrE03j8LxFqKxsLTYZzjdSmm4x_GxBSk=7jyV22Znksg@mail.gmail.com>

Hello

I was wondering are there any effective alternatives to VecGetArrayF90. In
the code I'm working on, I need to reorder variables from a field based
order in the application to a packed variable ordering that I use with
PETSc. The code I'm running is as follows which converts the blocked based
petsc ordering to the application field based order. There is an equilivent
routine that converts the field based residual back to petsc block ordering.

 call VecGetArrayF90(wVec,wvec_pointer,ierr)
  call EChk(ierr,__FILE__,__LINE__)

  ii = 1
  do nn=1,nDom
     do sps=1,nTimeIntervalsSpectral
        call setPointersAdj(nn,1_intType,sps)

        do k=2,kl
           do j=2,jl
              do i=2,il
                 do l=1,nw
                    w(i,j,k,l) = wvec_pointer(ii)
                    ii = ii + 1
                 end do
              end do
           end do
        end do
     end do
  end do

  call VecRestoreArrayF90(wVec,wvec_pointer,ierr)
  call EChk(ierr,__FILE__,__LINE__)


 The issue is that the VecGetArrayF90 command it highly finicky and often
prevents the code from compiling on various architectures.

My understanding is that the preferred way to of doing this is:

use petscvec
Vec wVe
PetscScalar, pointer :: wvec_pointer

call VecGetArrayF90(wVec,wvec_pointer,ierr)


However, many times the code will refuse to compile as there
are incompatibilities between petscvec module that was compiled with petsc
and the current compilation.

Another approach I tried is to eliminate the petscvec module and just use
an integer fro the petsc vector:

PetscInt :: wVec
PetscScalar, pointer :: wvec_pointer

call VecGetArrayF90(wVec,wvec_pointer,ierr)


This will compile, but it throws an error code 68, invalid pointer
argument. Again, this is only on some architectures with certain compilers.
(Ubuntu 12.04, gfortran 64bit will give this error, Ubuntu 10.10 ifort
32bit will work with either form)

My question is if you use the F77 form:  VecGetArray(x,x_array,i_x,ierr) is
there a performance penalty? Is it an actual pointer or is the data copied?
Do you think it would be more robust than using the F90 pointer form?

Also, would this be a suitable place to use the application ordering in
petsc?

Thanks,

Gaetan Kenway
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120527/3307b5d5/attachment.html>

From jedbrown at mcs.anl.gov  Mon May 28 08:09:08 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 28 May 2012 08:09:08 -0500
Subject: [petsc-users] Adding a processing on the solution
In-Reply-To: <CA+jOtDKDb+pUMakxjhv0ijiN6MA9wjf5mQp9_RKb4KHnzOBw3Q@mail.gmail.com>
References: <CA+jOtDLGcXriaScvBrUYWBzTsXQEnDrsT=L-2y=OZqh-mwz2Xg@mail.gmail.com>
	<CAM9tzSnBU+DUM=Xrt_u3dVcUHawmMekJJLkeE_DtYo6U8+Cq0A@mail.gmail.com>
	<CA+jOtDLCH9qoZ0zNYtyVQPCGZ0h-+p+m2z4ZOZc6XA36mRKgWQ@mail.gmail.com>
	<CA+jOtDKDb+pUMakxjhv0ijiN6MA9wjf5mQp9_RKb4KHnzOBw3Q@mail.gmail.com>
Message-ID: <CAM9tzSkXAu8sFkXfw2GdmHPFS7pUrfzCPi9Rm25ELL=XTMV1mg@mail.gmail.com>

On Mon, May 28, 2012 at 1:48 AM, behzad baghapour <
behzad.baghapour at gmail.com> wrote:

> Besides, Is the SNES context call residual evaluation then Jacobian or
> vise versa? I just concern that the limiting procedure affects both
> residual and Jacobian in the same iteration.


The Jacobian is, by definition, the derivative of the residual. If you are
doing Newton's method, then you are solving with the Jacobian, therefore
the Jacobian also "sees" the limiter.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120528/99b89922/attachment.html>

From jedbrown at mcs.anl.gov  Mon May 28 08:16:02 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 28 May 2012 08:16:02 -0500
Subject: [petsc-users] two-level Schwarz: PCMG or PCComposite
In-Reply-To: <BLU0-SMTP4666E6EB11C9C685F9C7F3CCB040@phx.gbl>
References: <BLU0-SMTP440ACD42EF2D3F61AC7C6F3CB170@phx.gbl>
	<45C92E5C-6AA5-470F-95CD-86BF20FDF57C@mcs.anl.gov>
	<BLU0-SMTP4666E6EB11C9C685F9C7F3CCB040@phx.gbl>
Message-ID: <CAM9tzSm+2qX+jotXHjuEewKsCRqeiA+t9jLnSxOc0st-7U4rvA@mail.gmail.com>

Fixed in petsc-dev, thanks.

http://petsc.cs.iit.edu/petsc/petsc-dev/rev/2abe2ed8a15d

The type was only recorded when set via -pc_composite_type.
PCCompositeSetType() was using the type, but not recording it.

On Mon, May 28, 2012 at 4:11 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> I use -ksp_view to check the PCComposite used by the ksp, but whatever
> type I set for pccomposite,
> using PCCompositeSetType, the output from -ksp_view always says it was
> ADDITIVE
>
> PC Object: 1 MPI processes
>  type: composite
>  Composite PC type - ADDITIVE
>  PCs on composite preconditioner follow
>
> On the other hand, I saw from -ksp_monitor that GMRES iterations are
> different.
>
> Thanks!
> Hui
>
>
> On May 11, 2012, at 2:55 PM, Barry Smith wrote:
>
> >
> >  If this is what you want to do then just use the PCCOMPOSITE. Use
> PC_COMPOSITE_SYMMETRIC_MULTIPLICATIVE
> >
> >   Barry
> >
> > On May 11, 2012, at 6:26 AM, Hui Zhang wrote:
> >
> >> I want to implement the two-level Schwarz method.  Let us denote P as
> the one-level
> >> Schwarz projection and P_0 as the coarse projection.  If I want the
> hybrid form of
> >> preconditioned system as follows:
> >>
> >>    (I-P_0) P (I-P_0)
> >>
> >> which is suitable:  PCMG or PCComposite?  It seems to me that
> PCComposite is good.
> >> But PCMG is not suitable because it can only implement the following
> form:
> >>
> >>    (I-P) P_0 (I-P)
> >>
> >> Any opinions?  Thanks!
> >>
> >>
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120528/d65edc20/attachment.html>

From jedbrown at mcs.anl.gov  Mon May 28 08:18:26 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 28 May 2012 08:18:26 -0500
Subject: [petsc-users] PCGALERKIN error message
In-Reply-To: <BLU0-SMTP255C217211A5A962387346CB040@phx.gbl>
References: <BLU0-SMTP255C217211A5A962387346CB040@phx.gbl>
Message-ID: <CAM9tzS=WRsBpbtCLCjPi9P--ZSsQqYg0=42grR6uZgR5tQEoMg@mail.gmail.com>

Thanks, fixed.

http://petsc.cs.iit.edu/petsc/petsc-dev/rev/e41d5a76fc08

On Mon, May 28, 2012 at 4:14 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> I got the following error message when using PCGALERKIN and first set
> restriction then immediately
> set interpolation.
>
> Thanks!
> Hui
>
>
> petsc-dev/src/ksp/pc/impls/galerkin/galerkin.c:207:
> __FUNCT__="PCGalerkinSetRestriction" does not agree with
> __func__="PCGalerkinSetInterpolation
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120528/aca47ec2/attachment.html>

From jedbrown at mcs.anl.gov  Mon May 28 08:20:02 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 28 May 2012 08:20:02 -0500
Subject: [petsc-users] PCComposite
In-Reply-To: <BLU0-SMTP172F312724561CAD5A642D0CB040@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
	<CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
	<BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
	<BLU0-SMTP172F312724561CAD5A642D0CB040@phx.gbl>
Message-ID: <CAM9tzSk7XRHDbmq1D547-5ZsPRFGbnUiH29zh5AtXL7RniVg7A@mail.gmail.com>

On Mon, May 28, 2012 at 5:19 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Sorry for repost the following question, but after reading composite.c and
> asm.c,
> I can not understand.  I observed that my subroutine for modifying
> submatrices of PCASM was
> called twice.  In composite.c it seems there is no calling of PCSetUp on
> its componential PC's.
> In asm.c, the PCSetUp_ASM first determines whether pc->setupcalled is true.
>
> I think this is a general question for PCComposite say pc. If it consists
> of two pc's  pc0, pc1
> and I have PCSetUp(pc0), I would not expect pc0 to be SetUp again.  I can
> not understand
> in my case why PCASM seems SetUp twice.
>

Does the routine do anything or just return immediately? If nothing has
been changed since the last time a PC was set up, it should return
immediately. If you have changed some configuration, it may need to be set
up again.



>
> Thanks!
> Hui
>
>
> > I have got a new question.  I'm now constructing a PCComposite from
> PCASM and another PCKSP (coarse
> > problem). And construction of PCKSP needs to use the subksp's of PCASM.
> So I need to PCSetUp on the
> > PCASM, right?  But the KSP using PCComposite would setup PCASM again.
>  How can I avoid twice setup
> > of PCASM ?
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120528/23f5dc22/attachment-0001.html>

From mike.hui.zhang at hotmail.com  Mon May 28 08:40:58 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Mon, 28 May 2012 15:40:58 +0200
Subject: [petsc-users] PCComposite
In-Reply-To: <CAM9tzSk7XRHDbmq1D547-5ZsPRFGbnUiH29zh5AtXL7RniVg7A@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
	<CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
	<BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
	<BLU0-SMTP172F312724561CAD5A642D0CB040@phx.gbl>
	<CAM9tzSk7XRHDbmq1D547-5ZsPRFGbnUiH29zh5AtXL7RniVg7A@mail.gmail.com>
Message-ID: <BLU0-SMTP20270D0A530B99D0BC60FA6CB040@phx.gbl>


On May 28, 2012, at 3:20 PM, Jed Brown wrote:

> On Mon, May 28, 2012 at 5:19 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> Sorry for repost the following question, but after reading composite.c and asm.c,
> I can not understand.  I observed that my subroutine for modifying submatrices of PCASM was
> called twice.  In composite.c it seems there is no calling of PCSetUp on its componential PC's.
> In asm.c, the PCSetUp_ASM first determines whether pc->setupcalled is true.
> 
> I think this is a general question for PCComposite say pc. If it consists of two pc's  pc0, pc1
> and I have PCSetUp(pc0), I would not expect pc0 to be SetUp again.  I can not understand
> in my case why PCASM seems SetUp twice.
> 
> Does the routine do anything or just return immediately? If nothing has been changed since the last time a PC was set up, it should return immediately. If you have changed some configuration, it may need to be set up again.

Thanks. But I said something wrong.  The source code asm.c does call the ModifySubMatrices
whenever PCSetUp is called on PCASM, no matter whether it was called before.  So this 
explains my observation (two calls of ModifySubMatrices).  My new question is:  can you
add a user control to determine whether MatGetSubMatrices and PCModifySubMatrices 
should be called inside PCSetUp_ASM ?  But maybe this is too invasive and not
very useful to most people (although useful to me).


> 
>  
> 
> Thanks!
> Hui
> 
> 
> > I have got a new question.  I'm now constructing a PCComposite from PCASM and another PCKSP (coarse
> > problem). And construction of PCKSP needs to use the subksp's of PCASM. So I need to PCSetUp on the
> > PCASM, right?  But the KSP using PCComposite would setup PCASM again.  How can I avoid twice setup
> > of PCASM ?
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120528/3fed6c2b/attachment.html>

From jedbrown at mcs.anl.gov  Mon May 28 08:42:34 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 28 May 2012 08:42:34 -0500
Subject: [petsc-users] PCComposite
In-Reply-To: <BLU0-SMTP20270D0A530B99D0BC60FA6CB040@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
	<CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
	<BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
	<BLU0-SMTP172F312724561CAD5A642D0CB040@phx.gbl>
	<CAM9tzSk7XRHDbmq1D547-5ZsPRFGbnUiH29zh5AtXL7RniVg7A@mail.gmail.com>
	<BLU0-SMTP20270D0A530B99D0BC60FA6CB040@phx.gbl>
Message-ID: <CAM9tzSmEVaE7bWDkY+2avSPVb+q2fZ2p3_Wh6iBrMkZKiksj0A@mail.gmail.com>

On Mon, May 28, 2012 at 8:40 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

>
> On May 28, 2012, at 3:20 PM, Jed Brown wrote:
>
> On Mon, May 28, 2012 at 5:19 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>> Sorry for repost the following question, but after reading composite.c
>> and asm.c,
>> I can not understand.  I observed that my subroutine for modifying
>> submatrices of PCASM was
>> called twice.  In composite.c it seems there is no calling of PCSetUp on
>> its componential PC's.
>> In asm.c, the PCSetUp_ASM first determines whether pc->setupcalled is
>> true.
>>
>> I think this is a general question for PCComposite say pc. If it consists
>> of two pc's  pc0, pc1
>> and I have PCSetUp(pc0), I would not expect pc0 to be SetUp again.  I can
>> not understand
>> in my case why PCASM seems SetUp twice.
>>
>
> Does the routine do anything or just return immediately? If nothing has
> been changed since the last time a PC was set up, it should return
> immediately. If you have changed some configuration, it may need to be set
> up again.
>
>
> Thanks. But I said something wrong.  The source code asm.c does call the
> ModifySubMatrices
> whenever PCSetUp is called on PCASM, no matter whether it was called
> before.  So this
> explains my observation (two calls of ModifySubMatrices).  My new question
> is:  can you
> add a user control to determine whether MatGetSubMatrices and
> PCModifySubMatrices
> should be called inside PCSetUp_ASM ?  But maybe this is too invasive and
> not
> very useful to most people (although useful to me).
>

It does not reach PCSetUp_ASM unless something has changed. Run with -info
and/or in a debugger to see why.

PetscErrorCode  PCSetUp(PC pc)
{
  PetscErrorCode ierr;
  const char     *def;

  PetscFunctionBegin;
  PetscValidHeaderSpecific(pc,PC_CLASSID,1);
  if (!pc->mat)
SETERRQ(((PetscObject)pc)->comm,PETSC_ERR_ARG_WRONGSTATE,"Matrix must be
set first");

  if (pc->setupcalled > 1) {
    ierr = PetscInfo(pc,"Setting PC with identical
preconditioner\n");CHKERRQ(ierr);
    PetscFunctionReturn(0);



>
>
>
>
>
>>
>> Thanks!
>> Hui
>>
>>
>> > I have got a new question.  I'm now constructing a PCComposite from
>> PCASM and another PCKSP (coarse
>> > problem). And construction of PCKSP needs to use the subksp's of PCASM.
>> So I need to PCSetUp on the
>> > PCASM, right?  But the KSP using PCComposite would setup PCASM again.
>>  How can I avoid twice setup
>> > of PCASM ?
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120528/fb0af8b3/attachment.html>

From mike.hui.zhang at hotmail.com  Mon May 28 09:40:57 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Mon, 28 May 2012 16:40:57 +0200
Subject: [petsc-users] PCComposite
In-Reply-To: <CAM9tzSmEVaE7bWDkY+2avSPVb+q2fZ2p3_Wh6iBrMkZKiksj0A@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
	<CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
	<BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
	<BLU0-SMTP172F312724561CAD5A642D0CB040@phx.gbl>
	<CAM9tzSk7XRHDbmq1D547-5ZsPRFGbnUiH29zh5AtXL7RniVg7A@mail.gmail.com>
	<BLU0-SMTP20270D0A530B99D0BC60FA6CB040@phx.gbl>
	<CAM9tzSmEVaE7bWDkY+2avSPVb+q2fZ2p3_Wh6iBrMkZKiksj0A@mail.gmail.com>
Message-ID: <BLU0-SMTP96F263747361CCB12B01D3CB040@phx.gbl>


On May 28, 2012, at 3:42 PM, Jed Brown wrote:

> On Mon, May 28, 2012 at 8:40 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> 
> On May 28, 2012, at 3:20 PM, Jed Brown wrote:
> 
>> On Mon, May 28, 2012 at 5:19 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> Sorry for repost the following question, but after reading composite.c and asm.c,
>> I can not understand.  I observed that my subroutine for modifying submatrices of PCASM was
>> called twice.  In composite.c it seems there is no calling of PCSetUp on its componential PC's.
>> In asm.c, the PCSetUp_ASM first determines whether pc->setupcalled is true.
>> 
>> I think this is a general question for PCComposite say pc. If it consists of two pc's  pc0, pc1
>> and I have PCSetUp(pc0), I would not expect pc0 to be SetUp again.  I can not understand
>> in my case why PCASM seems SetUp twice.
>> 
>> Does the routine do anything or just return immediately? If nothing has been changed since the last time a PC was set up, it should return immediately. If you have changed some configuration, it may need to be set up again.
> 
> Thanks. But I said something wrong.  The source code asm.c does call the ModifySubMatrices
> whenever PCSetUp is called on PCASM, no matter whether it was called before.  So this 
> explains my observation (two calls of ModifySubMatrices).  My new question is:  can you
> add a user control to determine whether MatGetSubMatrices and PCModifySubMatrices 
> should be called inside PCSetUp_ASM ?  But maybe this is too invasive and not
> very useful to most people (although useful to me).
> 
> It does not reach PCSetUp_ASM unless something has changed. Run with -info and/or in a debugger to see why.
> 
> PetscErrorCode  PCSetUp(PC pc)
> {
>   PetscErrorCode ierr;
>   const char     *def;
> 
>   PetscFunctionBegin;
>   PetscValidHeaderSpecific(pc,PC_CLASSID,1);
>   if (!pc->mat) SETERRQ(((PetscObject)pc)->comm,PETSC_ERR_ARG_WRONGSTATE,"Matrix must be set first");
> 
>   if (pc->setupcalled > 1) {
>     ierr = PetscInfo(pc,"Setting PC with identical preconditioner\n");CHKERRQ(ierr);
>     PetscFunctionReturn(0);
> 


Thanks for the hints.  I found why (I hope).  We have KSP named ksp and its PC is pc,
and pc is PCComposite with pc0, pc1. The program runs like KSPSetUp on ksp, which
calls PCSetUp on pc, i.e. PCSetUp_Composite on pc.  The source codes read

static PetscErrorCode PCSetUp_Composite(PC pc)
168: {
169:   PetscErrorCode   ierr;
170:   PC_Composite     *jac = (PC_Composite*)pc->data;
171:   PC_CompositeLink next = jac->head;

174:   if (!jac->work1) {
175:    MatGetVecs(pc->pmat,&jac->work1,0);
176:   }
177:   while (next) {
178:     PCSetOperators(next->pc,pc->mat,pc->pmat,pc->flag);
179:     next = next->next;
180:   }
181:   return(0);
182: }
Note the 178 line which calls PCSetOperators on pc0 and pc1.  The source codes of PCSetOperators
read

1050:  if (pc->setupcalled == 2 && flag != SAME_PRECONDITIONER) {
1051:     pc->setupcalled = 1;
1052:   }
Note the 1051 line reset pc->setupcalled to 1, here the pc from PCSetUp_Composite is pc0 and pc1.
So although if pc0  has been set up, this will cause pc0 be set up again. 

Is my reading correct?

Thanks a lot!
Hui


>  
> 
> 
>> 
>>  
>> 
>> Thanks!
>> Hui
>> 
>> 
>> > I have got a new question.  I'm now constructing a PCComposite from PCASM and another PCKSP (coarse
>> > problem). And construction of PCKSP needs to use the subksp's of PCASM. So I need to PCSetUp on the
>> > PCASM, right?  But the KSP using PCComposite would setup PCASM again.  How can I avoid twice setup
>> > of PCASM ?
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120528/32594aad/attachment.html>

From mike.hui.zhang at hotmail.com  Mon May 28 09:49:48 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Mon, 28 May 2012 16:49:48 +0200
Subject: [petsc-users] PCComposite
In-Reply-To: <BLU0-SMTP96F263747361CCB12B01D3CB040@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
	<CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
	<BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
	<BLU0-SMTP172F312724561CAD5A642D0CB040@phx.gbl>
	<CAM9tzSk7XRHDbmq1D547-5ZsPRFGbnUiH29zh5AtXL7RniVg7A@mail.gmail.com>
	<BLU0-SMTP20270D0A530B99D0BC60FA6CB040@phx.gbl>
	<CAM9tzSmEVaE7bWDkY+2avSPVb+q2fZ2p3_Wh6iBrMkZKiksj0A@mail.gmail.com>
	<BLU0-SMTP96F263747361CCB12B01D3CB040@phx.gbl>
Message-ID: <BLU0-SMTP17100F8700ACA89E31834C1CB040@phx.gbl>


On May 28, 2012, at 4:40 PM, Hui Zhang wrote:

> 
> On May 28, 2012, at 3:42 PM, Jed Brown wrote:
> 
>> On Mon, May 28, 2012 at 8:40 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> 
>> On May 28, 2012, at 3:20 PM, Jed Brown wrote:
>> 
>>> On Mon, May 28, 2012 at 5:19 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> Sorry for repost the following question, but after reading composite.c and asm.c,
>>> I can not understand.  I observed that my subroutine for modifying submatrices of PCASM was
>>> called twice.  In composite.c it seems there is no calling of PCSetUp on its componential PC's.
>>> In asm.c, the PCSetUp_ASM first determines whether pc->setupcalled is true.
>>> 
>>> I think this is a general question for PCComposite say pc. If it consists of two pc's  pc0, pc1
>>> and I have PCSetUp(pc0), I would not expect pc0 to be SetUp again.  I can not understand
>>> in my case why PCASM seems SetUp twice.
>>> 
>>> Does the routine do anything or just return immediately? If nothing has been changed since the last time a PC was set up, it should return immediately. If you have changed some configuration, it may need to be set up again.
>> 
>> Thanks. But I said something wrong.  The source code asm.c does call the ModifySubMatrices
>> whenever PCSetUp is called on PCASM, no matter whether it was called before.  So this 
>> explains my observation (two calls of ModifySubMatrices).  My new question is:  can you
>> add a user control to determine whether MatGetSubMatrices and PCModifySubMatrices 
>> should be called inside PCSetUp_ASM ?  But maybe this is too invasive and not
>> very useful to most people (although useful to me).
>> 
>> It does not reach PCSetUp_ASM unless something has changed. Run with -info and/or in a debugger to see why.
>> 
>> PetscErrorCode  PCSetUp(PC pc)
>> {
>>   PetscErrorCode ierr;
>>   const char     *def;
>> 
>>   PetscFunctionBegin;
>>   PetscValidHeaderSpecific(pc,PC_CLASSID,1);
>>   if (!pc->mat) SETERRQ(((PetscObject)pc)->comm,PETSC_ERR_ARG_WRONGSTATE,"Matrix must be set first");
>> 
>>   if (pc->setupcalled > 1) {
>>     ierr = PetscInfo(pc,"Setting PC with identical preconditioner\n");CHKERRQ(ierr);
>>     PetscFunctionReturn(0);
>> 
> 
> 
> Thanks for the hints.  I found why (I hope).  We have KSP named ksp and its PC is pc,
> and pc is PCComposite with pc0, pc1. The program runs like KSPSetUp on ksp, which
> calls PCSetUp on pc, i.e. PCSetUp_Composite on pc.  The source codes read
> 
> static PetscErrorCode PCSetUp_Composite(PC pc)
> 168: {
> 169:   PetscErrorCode   ierr;
> 170:   PC_Composite     *jac = (PC_Composite*)pc->data;
> 171:   PC_CompositeLink next = jac->head;
> 
> 174:   if (!jac->work1) {
> 175:    MatGetVecs(pc->pmat,&jac->work1,0);
> 176:   }
> 177:   while (next) {
> 178:     PCSetOperators(next->pc,pc->mat,pc->pmat,pc->flag);
> 179:     next = next->next;
> 180:   }
> 181:   return(0);
> 182: }
> Note the 178 line which calls PCSetOperators on pc0 and pc1.  The source codes of PCSetOperators
> read
> 
> 1050:  if (pc->setupcalled == 2 && flag != SAME_PRECONDITIONER) {
> 1051:     pc->setupcalled = 1;
> 1052:   }
> Note the 1051 line reset pc->setupcalled to 1, here the pc from PCSetUp_Composite is pc0 and pc1.
> So although if pc0  has been set up, this will cause pc0 be set up again. 
> 
> Is my reading correct?

I forgot to say: I first PCSetUp on pc0 before KSPSetUp on ksp.  Because construction of my pc1
needs to use PCSetUp'ed pc0. It seems current petsc assumes PCComposite is first SetUp before
its components pc0 and pc1 are SetUp'ed.



> 
> Thanks a lot!
> Hui
> 
> 
>>  
>> 
>> 
>>> 
>>>  
>>> 
>>> Thanks!
>>> Hui
>>> 
>>> 
>>> > I have got a new question.  I'm now constructing a PCComposite from PCASM and another PCKSP (coarse
>>> > problem). And construction of PCKSP needs to use the subksp's of PCASM. So I need to PCSetUp on the
>>> > PCASM, right?  But the KSP using PCComposite would setup PCASM again.  How can I avoid twice setup
>>> > of PCASM ?
>>> 
>>> 
>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120528/8eb51aad/attachment-0001.html>

From behzad.baghapour at gmail.com  Mon May 28 10:00:25 2012
From: behzad.baghapour at gmail.com (behzad baghapour)
Date: Mon, 28 May 2012 19:30:25 +0430
Subject: [petsc-users] Adding a processing on the solution
In-Reply-To: <CAM9tzSkXAu8sFkXfw2GdmHPFS7pUrfzCPi9Rm25ELL=XTMV1mg@mail.gmail.com>
References: <CA+jOtDLGcXriaScvBrUYWBzTsXQEnDrsT=L-2y=OZqh-mwz2Xg@mail.gmail.com>
	<CAM9tzSnBU+DUM=Xrt_u3dVcUHawmMekJJLkeE_DtYo6U8+Cq0A@mail.gmail.com>
	<CA+jOtDLCH9qoZ0zNYtyVQPCGZ0h-+p+m2z4ZOZc6XA36mRKgWQ@mail.gmail.com>
	<CA+jOtDKDb+pUMakxjhv0ijiN6MA9wjf5mQp9_RKb4KHnzOBw3Q@mail.gmail.com>
	<CAM9tzSkXAu8sFkXfw2GdmHPFS7pUrfzCPi9Rm25ELL=XTMV1mg@mail.gmail.com>
Message-ID: <CA+jOtDL0j1HEa0q7Q3ux+hcmqLkkr+5KKjRVVab01_iXBWKXaA@mail.gmail.com>

OK.Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120528/daf9f15f/attachment.html>

From jedbrown at mcs.anl.gov  Mon May 28 10:16:09 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 28 May 2012 10:16:09 -0500
Subject: [petsc-users] PCComposite
In-Reply-To: <BLU0-SMTP96F263747361CCB12B01D3CB040@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
	<CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
	<BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
	<BLU0-SMTP172F312724561CAD5A642D0CB040@phx.gbl>
	<CAM9tzSk7XRHDbmq1D547-5ZsPRFGbnUiH29zh5AtXL7RniVg7A@mail.gmail.com>
	<BLU0-SMTP20270D0A530B99D0BC60FA6CB040@phx.gbl>
	<CAM9tzSmEVaE7bWDkY+2avSPVb+q2fZ2p3_Wh6iBrMkZKiksj0A@mail.gmail.com>
	<BLU0-SMTP96F263747361CCB12B01D3CB040@phx.gbl>
Message-ID: <CAM9tzSn8J9FDO21_G_GAOW9uJ3DPvebiVK7yseDb0BH6w1qwDA@mail.gmail.com>

On Mon, May 28, 2012 at 9:40 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

>
> On May 28, 2012, at 3:42 PM, Jed Brown wrote:
>
> On Mon, May 28, 2012 at 8:40 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>
>>
>> On May 28, 2012, at 3:20 PM, Jed Brown wrote:
>>
>> On Mon, May 28, 2012 at 5:19 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:
>>
>>> Sorry for repost the following question, but after reading composite.c
>>> and asm.c,
>>> I can not understand.  I observed that my subroutine for modifying
>>> submatrices of PCASM was
>>> called twice.  In composite.c it seems there is no calling of PCSetUp on
>>> its componential PC's.
>>> In asm.c, the PCSetUp_ASM first determines whether pc->setupcalled is
>>> true.
>>>
>>> I think this is a general question for PCComposite say pc. If it
>>> consists of two pc's  pc0, pc1
>>> and I have PCSetUp(pc0), I would not expect pc0 to be SetUp again.  I
>>> can not understand
>>> in my case why PCASM seems SetUp twice.
>>>
>>
>> Does the routine do anything or just return immediately? If nothing has
>> been changed since the last time a PC was set up, it should return
>> immediately. If you have changed some configuration, it may need to be set
>> up again.
>>
>>
>> Thanks. But I said something wrong.  The source code asm.c does call the
>> ModifySubMatrices
>> whenever PCSetUp is called on PCASM, no matter whether it was called
>> before.  So this
>> explains my observation (two calls of ModifySubMatrices).  My new
>> question is:  can you
>> add a user control to determine whether MatGetSubMatrices and
>> PCModifySubMatrices
>> should be called inside PCSetUp_ASM ?  But maybe this is too invasive and
>> not
>> very useful to most people (although useful to me).
>>
>
> It does not reach PCSetUp_ASM unless something has changed. Run with -info
> and/or in a debugger to see why.
>
> PetscErrorCode  PCSetUp(PC pc)
> {
>   PetscErrorCode ierr;
>   const char     *def;
>
>   PetscFunctionBegin;
>   PetscValidHeaderSpecific(pc,PC_CLASSID,1);
>   if (!pc->mat)
> SETERRQ(((PetscObject)pc)->comm,PETSC_ERR_ARG_WRONGSTATE,"Matrix must be
> set first");
>
>   if (pc->setupcalled > 1) {
>     ierr = PetscInfo(pc,"Setting PC with identical
> preconditioner\n");CHKERRQ(ierr);
>     PetscFunctionReturn(0);
>
>
>
> Thanks for the hints.  I found why (I hope).  We have KSP named ksp and
> its PC is pc,
> and pc is PCComposite with pc0, pc1. The program runs like KSPSetUp on
> ksp, which
> calls PCSetUp on pc, i.e. PCSetUp_Composite on pc.  The source codes read
>
> *static PetscErrorCode <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscErrorCode.html#PetscErrorCode> PCSetUp_Composite(PC <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PC.html#PC> pc)*168: {169:   PetscErrorCode <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscErrorCode.html#PetscErrorCode>   ierr;170:   PC_Composite     *jac = (PC_Composite*)pc->data;171:   PC_CompositeLink next = jac->head;
> 174:   if (!jac->work1) {175:    MatGetVecs <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetVecs.html#MatGetVecs>(pc->pmat,&jac->work1,0);176:   }177:   while (next) {178:     PCSetOperators <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCSetOperators.html#PCSetOperators>(next->pc,pc->mat,pc->pmat,pc->flag);179:     next = next->next;180:   }181:   return(0);182: }
>
> Note the 178 line which calls PCSetOperators on pc0 and pc1.  The source
> codes of PCSetOperators
> read
>
> 1050:  if (pc->setupcalled == 2 && flag != SAME_PRECONDITIONER) {1051:     pc->setupcalled = 1;1052:   }
>
> Note the 1051 line reset pc->setupcalled to 1, here the pc from
> PCSetUp_Composite is pc0 and pc1.
> So although if pc0  has been set up, this will cause pc0 be set up again.
>
> Is my reading correct?
>

When you change the operators, the preconditioner has to be set up again
(perhaps not a "complete" setup, but some setup is required).


>
> Thanks a lot!
> Hui
>
>
>
>
>>
>>
>>
>>
>>
>>>
>>> Thanks!
>>> Hui
>>>
>>>
>>> > I have got a new question.  I'm now constructing a PCComposite from
>>> PCASM and another PCKSP (coarse
>>> > problem). And construction of PCKSP needs to use the subksp's of
>>> PCASM. So I need to PCSetUp on the
>>> > PCASM, right?  But the KSP using PCComposite would setup PCASM again.
>>>  How can I avoid twice setup
>>> > of PCASM ?
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120528/c808b14f/attachment.html>

From mike.hui.zhang at hotmail.com  Mon May 28 10:25:49 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Mon, 28 May 2012 17:25:49 +0200
Subject: [petsc-users] PCComposite
In-Reply-To: <CAM9tzSn8J9FDO21_G_GAOW9uJ3DPvebiVK7yseDb0BH6w1qwDA@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
	<CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
	<BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
	<BLU0-SMTP172F312724561CAD5A642D0CB040@phx.gbl>
	<CAM9tzSk7XRHDbmq1D547-5ZsPRFGbnUiH29zh5AtXL7RniVg7A@mail.gmail.com>
	<BLU0-SMTP20270D0A530B99D0BC60FA6CB040@phx.gbl>
	<CAM9tzSmEVaE7bWDkY+2avSPVb+q2fZ2p3_Wh6iBrMkZKiksj0A@mail.gmail.com>
	<BLU0-SMTP96F263747361CCB12B01D3CB040@phx.gbl>
	<CAM9tzSn8J9FDO21_G_GAOW9uJ3DPvebiVK7yseDb0BH6w1qwDA@mail.gmail.com>
Message-ID: <BLU0-SMTP2292F24AF381066BBA12699CB040@phx.gbl>


On May 28, 2012, at 5:16 PM, Jed Brown wrote:

> On Mon, May 28, 2012 at 9:40 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> 
> On May 28, 2012, at 3:42 PM, Jed Brown wrote:
> 
>> On Mon, May 28, 2012 at 8:40 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> 
>> On May 28, 2012, at 3:20 PM, Jed Brown wrote:
>> 
>>> On Mon, May 28, 2012 at 5:19 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>>> Sorry for repost the following question, but after reading composite.c and asm.c,
>>> I can not understand.  I observed that my subroutine for modifying submatrices of PCASM was
>>> called twice.  In composite.c it seems there is no calling of PCSetUp on its componential PC's.
>>> In asm.c, the PCSetUp_ASM first determines whether pc->setupcalled is true.
>>> 
>>> I think this is a general question for PCComposite say pc. If it consists of two pc's  pc0, pc1
>>> and I have PCSetUp(pc0), I would not expect pc0 to be SetUp again.  I can not understand
>>> in my case why PCASM seems SetUp twice.
>>> 
>>> Does the routine do anything or just return immediately? If nothing has been changed since the last time a PC was set up, it should return immediately. If you have changed some configuration, it may need to be set up again.
>> 
>> Thanks. But I said something wrong.  The source code asm.c does call the ModifySubMatrices
>> whenever PCSetUp is called on PCASM, no matter whether it was called before.  So this 
>> explains my observation (two calls of ModifySubMatrices).  My new question is:  can you
>> add a user control to determine whether MatGetSubMatrices and PCModifySubMatrices 
>> should be called inside PCSetUp_ASM ?  But maybe this is too invasive and not
>> very useful to most people (although useful to me).
>> 
>> It does not reach PCSetUp_ASM unless something has changed. Run with -info and/or in a debugger to see why.
>> 
>> PetscErrorCode  PCSetUp(PC pc)
>> {
>>   PetscErrorCode ierr;
>>   const char     *def;
>> 
>>   PetscFunctionBegin;
>>   PetscValidHeaderSpecific(pc,PC_CLASSID,1);
>>   if (!pc->mat) SETERRQ(((PetscObject)pc)->comm,PETSC_ERR_ARG_WRONGSTATE,"Matrix must be set first");
>> 
>>   if (pc->setupcalled > 1) {
>>     ierr = PetscInfo(pc,"Setting PC with identical preconditioner\n");CHKERRQ(ierr);
>>     PetscFunctionReturn(0);
>> 
> 
> 
> Thanks for the hints.  I found why (I hope).  We have KSP named ksp and its PC is pc,
> and pc is PCComposite with pc0, pc1. The program runs like KSPSetUp on ksp, which
> calls PCSetUp on pc, i.e. PCSetUp_Composite on pc.  The source codes read
> 
> static PetscErrorCode PCSetUp_Composite(PC pc)
> 168: {
> 169:   PetscErrorCode   ierr;
> 170:   PC_Composite     *jac = (PC_Composite*)pc->data;
> 171:   PC_CompositeLink next = jac->head;
> 
> 174:   if (!jac->work1) {
> 175:    MatGetVecs(pc->pmat,&jac->work1,0);
> 176:   }
> 177:   while (next) {
> 178:     PCSetOperators(next->pc,pc->mat,pc->pmat,pc->flag);
> 179:     next = next->next;
> 180:   }
> 181:   return(0);
> 182: }
> Note the 178 line which calls PCSetOperators on pc0 and pc1.  The source codes of PCSetOperators
> read
> 
> 1050:  if (pc->setupcalled == 2 && flag != SAME_PRECONDITIONER) {
> 1051:     pc->setupcalled = 1;
> 1052:   }
> Note the 1051 line reset pc->setupcalled to 1, here the pc from PCSetUp_Composite is pc0 and pc1.
> So although if pc0  has been set up, this will cause pc0 be set up again. 
> 
> Is my reading correct?
> 
> When you change the operators, the preconditioner has to be set up again (perhaps not a "complete" setup, but some setup is required).

But I'm not *changing* the operator. It is PCSetUp_Composite that call PCSetOperators on sub pc
a second time-- he does not know that the sub pc has been set up before.


>  
> 
> Thanks a lot!
> Hui
> 
> 
>>  
>> 
>> 
>>> 
>>>  
>>> 
>>> Thanks!
>>> Hui
>>> 
>>> 
>>> > I have got a new question.  I'm now constructing a PCComposite from PCASM and another PCKSP (coarse
>>> > problem). And construction of PCKSP needs to use the subksp's of PCASM. So I need to PCSetUp on the
>>> > PCASM, right?  But the KSP using PCComposite would setup PCASM again.  How can I avoid twice setup
>>> > of PCASM ?
>>> 
>>> 
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120528/ea9cbb03/attachment-0001.html>

From jedbrown at mcs.anl.gov  Mon May 28 10:29:39 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Mon, 28 May 2012 10:29:39 -0500
Subject: [petsc-users] PCComposite
In-Reply-To: <BLU0-SMTP2292F24AF381066BBA12699CB040@phx.gbl>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
	<CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
	<BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
	<BLU0-SMTP172F312724561CAD5A642D0CB040@phx.gbl>
	<CAM9tzSk7XRHDbmq1D547-5ZsPRFGbnUiH29zh5AtXL7RniVg7A@mail.gmail.com>
	<BLU0-SMTP20270D0A530B99D0BC60FA6CB040@phx.gbl>
	<CAM9tzSmEVaE7bWDkY+2avSPVb+q2fZ2p3_Wh6iBrMkZKiksj0A@mail.gmail.com>
	<BLU0-SMTP96F263747361CCB12B01D3CB040@phx.gbl>
	<CAM9tzSn8J9FDO21_G_GAOW9uJ3DPvebiVK7yseDb0BH6w1qwDA@mail.gmail.com>
	<BLU0-SMTP2292F24AF381066BBA12699CB040@phx.gbl>
Message-ID: <CAM9tzSn4ckAQNtWuiwgceoXxcmU+M3Q_k9dEjg+pD2KNgscR8w@mail.gmail.com>

On Mon, May 28, 2012 at 10:25 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> But I'm not *changing* the operator. It is PCSetUp_Composite that call
> PCSetOperators on sub pc
> a second time-- he does not know that the sub pc has been set up before.
>

Why do you call PCSetUp the first time?

We don't currently have a mechanism to automatically know whether the
operator has changed since the last setup, that's what the flag argument is
for.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120528/068f6b0a/attachment.html>

From mike.hui.zhang at hotmail.com  Mon May 28 10:37:00 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Mon, 28 May 2012 17:37:00 +0200
Subject: [petsc-users] PCComposite
In-Reply-To: <CAM9tzSn4ckAQNtWuiwgceoXxcmU+M3Q_k9dEjg+pD2KNgscR8w@mail.gmail.com>
References: <BLU0-SMTP3901BE21C3CCDB74DB29183CB790@phx.gbl>
	<CA+Y_9UJcH7wxLUc4HxA5XSWVryKfkqF4D7ZcRE0Wh0b8kMHhLw@mail.gmail.com>
	<BLU0-SMTP206305D4FB2FB736378ADDBCB7D0@phx.gbl>
	<BLU0-SMTP4408DEE137E2670B260B7FDCB7D0@phx.gbl>
	<7E09CD1C-EE42-41B3-9107-20B3C887CA8A@gmail.com>
	<BLU0-SMTP207177A24612F505AD8A373CB610@phx.gbl>
	<CA+Y_9UL1gpwh=L70ui1eR4q=LV8WXeWiO3xKt+J3a33_ja+AXw@mail.gmail.com>
	<BLU0-SMTP102E8BBB003186C1B263D64CB660@phx.gbl>
	<CA+Y_9U+J4Tw+Q-5+5BoTg9MRS=3AgUR-SQ8RWq_8DAVZO0Vo1Q@mail.gmail.com>
	<BLU0-SMTP1648594A36A05659C18C92FCB660@phx.gbl>
	<CA+Y_9UKqKfuPaEXQjnruaU1Y7yTNg2n5LOuJgCxyAwXGXS+WYQ@mail.gmail.com>
	<BLU0-SMTP2607BEEC77232423C997D33CB160@phx.gbl>
	<CA+Y_9UJj74okW5TMX-ou_OA6nW3BRDjfeWuhOBb16cL2Eru5Ug@mail.gmail.com>
	<BLU0-SMTP491B948094B7B354780258CB170@phx.gbl>
	<CA+Y_9U+1n50PVtCPZz601eVEaBjx3JpZ3h9J-V+1RTDWTW9njw@mail.gmail.com>
	<BLU0-SMTP2573CA7B7CDC1F7A57AB01BCB170@phx.gbl>
	<CA+Y_9UJz10kUXB3vEhHYa245ZGGJr2LhxWALtq6G-VQVY6nEbg@mail.gmail.com>
	<BLU0-SMTP460961B60019926DC9A6FE5CB1B0@phx.gbl>
	<CA+Y_9UKV0q4=95Xtg5KSvn8QeWyX2hG7QZV2V06LB87cvOiQSA@mail.gmail.com>
	<BLU0-SMTP2951099681C0611CB44060CB1B0@phx.gbl>
	<CA+Y_9UKXaiP5zmMtAJ3=y-TkxQy0E6dGaDpz7x-tnzPwDmOMSg@mail.gmail.com>
	<BLU0-SMTP42369BD89352E0A40A996D0CB1B0@phx.gbl>
	<CA+Y_9ULjk6+aDJV8Lgo7Ruir56y5DQR3SQvcxqtyEa_p5Y4Hqw@mail.gmail.com>
	<BLU0-SMTP95B8A643939B2071FC92F7CB1B0@phx.gbl>
	<CA+Y_9ULBgCHDygVN_Ha+XEmVEV3zB7nmGNfm7O_wxK7A4UNyQw@mail.gmail.com>
	<BLU0-SMTP181C93E20E74A7C640FFF90CB1B0@phx.gbl>
	<BLU0-SMTP1419BD08302FB9F86553AA2CB1F0@phx.gbl>
	<CA+Y_9UJb-ngkATi1zWk6tG31z9U=jLazSDHXS0D_DYbvOjzjPw@mail.gmail.com>
	<BLU0-SMTP332CA9F2D9CCB1DDC401BCFCB020@phx.gbl>
	<BLU0-SMTP172F312724561CAD5A642D0CB040@phx.gbl>
	<CAM9tzSk7XRHDbmq1D547-5ZsPRFGbnUiH29zh5AtXL7RniVg7A@mail.gmail.com>
	<BLU0-SMTP20270D0A530B99D0BC60FA6CB040@phx.gbl>
	<CAM9tzSmEVaE7bWDkY+2avSPVb+q2fZ2p3_Wh6iBrMkZKiksj0A@mail.gmail.com>
	<BLU0-SMTP96F263747361CCB12B01D3CB040@phx.gbl>
	<CAM9tzSn8J9FDO21_G_GAOW9uJ3DPvebiVK7yseDb0BH6w1qwDA@mail.gmail.com>
	<BLU0-SMTP2292F24AF381066BBA12699CB040@phx.gbl>
	<CAM9tzSn4ckAQNtWuiwgceoXxcmU+M3Q_k9dEjg+pD2KNgscR8w@mail.gmail.com>
Message-ID: <BLU0-SMTP80547913250B9565CEE3ECB040@phx.gbl>


On May 28, 2012, at 5:29 PM, Jed Brown wrote:

> On Mon, May 28, 2012 at 10:25 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
> But I'm not *changing* the operator. It is PCSetUp_Composite that call PCSetOperators on sub pc
> a second time-- he does not know that the sub pc has been set up before.
> 
> Why do you call PCSetUp the first time?

Because the PCComposite has two subpc's and one of the subpc is needed to be set up and used for 
construction of the other subpc.

> 
> We don't currently have a mechanism to automatically know whether the operator has changed since the last setup, that's what the flag argument is for.

Yes, I found this is a feasible way.  I just set SAME_PRECONDITIONER for the PCComposite. 
It is fine to me.  Thanks!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120528/b1c90dfd/attachment.html>

From mike.hui.zhang at hotmail.com  Tue May 29 07:22:11 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Tue, 29 May 2012 14:22:11 +0200
Subject: [petsc-users] Warning in PCCompositeAddPC
Message-ID: <BLU0-SMTP342B82633CF759EB62E770CCB050@phx.gbl>

Hi, 
I also have these warnings although I'm using latest petsc-dev. 

> You can ignore these warnings about 'const' for char *. I believe this is
> fixed in petsc-dev.
> 
>    Matt
> 
> > Hi PETSc experts,
> >
> > When I used the following functions:
> >
> > ierr = PCCompositeAddPC(finepc,PCSHELL);CHKERRQ(ierr);
> > ierr = PCCompositeAddPC(finepc,PCASM);CHKERRQ(ierr);
> > I got the following warning:
> >
> > joab.c:340: warning: passing argument 2 of ?PCCompositeAddPC? discards
> > qualifiers from pointer target type
> > joab.c:341: warning: passing argument 2 of ?PCCompositeAddPC? discards
> > qualifiers from pointer target type
> >
> > Can you tell me how to fix it?  Thank you.
> >
> > Best,
> >
> > Rongliang
> >
> > UCB
> >
> >


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120529/21a5ab5d/attachment.html>

From jedbrown at mcs.anl.gov  Tue May 29 07:41:06 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 29 May 2012 07:41:06 -0500
Subject: [petsc-users] Warning in PCCompositeAddPC
In-Reply-To: <BLU0-SMTP342B82633CF759EB62E770CCB050@phx.gbl>
References: <BLU0-SMTP342B82633CF759EB62E770CCB050@phx.gbl>
Message-ID: <CAM9tzSm=hBwZpEebf1hgq8As1CxgdBYRyb4sRN79H5ZYU1=GOg@mail.gmail.com>

On Tue, May 29, 2012 at 7:22 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> Hi,
>
> I also have these warnings although I'm using latest petsc-dev.
>
>
Thanks, fixed here.

http://petsc.cs.iit.edu/petsc/petsc-dev/rev/df7df246f8d3


>
> You can ignore these warnings about 'const' for char *. I believe this is
> fixed in petsc-dev.
>
>    Matt
>
>
> >* Hi PETSc experts,*>**>* When I used the following functions:*>**>* ierr = PCCompositeAddPC(finepc,PCSHELL);CHKERRQ(ierr);*>* ierr = PCCompositeAddPC(finepc,PCASM);CHKERRQ(ierr);*
>
> >* I got the following warning: *>* *>* joab.c:340: warning: passing
> argument 2 of ?PCCompositeAddPC? discards *>* qualifiers from pointer
> target type *>* joab.c:341: warning: passing argument 2 of
> ?PCCompositeAddPC? discards *>* qualifiers from pointer target type *>* *>
> * Can you tell me how to fix it? Thank you. *>* *>* Best, *>* *>*Rongliang
> *>* *>* UCB *>* *>* *
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120529/ae93cc93/attachment.html>

From B.Sanderse at cwi.nl  Tue May 29 10:52:15 2012
From: B.Sanderse at cwi.nl (Benjamin Sanderse)
Date: Tue, 29 May 2012 17:52:15 +0200 (CEST)
Subject: [petsc-users] MatMult
In-Reply-To: <c1d5f8a4-d8da-4d18-9556-7f5aba51edef@zembox02.zaas.igi.nl>
Message-ID: <657f7efd-db40-4c1c-a35b-d1e13e1ffa56@zembox02.zaas.igi.nl>

Hello all,

I have a simple question about using MatMult (or MatMultAdd) in parallel.

I am performing the matrix-vector multiplication

z = A*x + y

in my code by using

call MatMultAdd(A,x,y,z,ierr); CHKERRQ(ierr)

A is a sparse matrix, type MPIAIJ, and x, y, and z have been obtained using 

call MatGetVecs(A,x,y,ierr); CHKERRQ(ierr)    
call MatGetVecs(A,PETSC_NULL_OBJECT,z,ierr); CHKERRQ(ierr)    

x, y, and z are vecs of type mpi.

The problem is that in the sequential case the MatMultAdd is MUCH faster than in the parallel case (at least a factor 100 difference).

As an example, here is the output with some properties of A when using -mat_view_info and -info:

2 processors:
[1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374781
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780
[1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374782
[1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374782
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780
[0] MatStashScatterBegin_Private(): No of messages: 0 
[1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
[1] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 450; storage space: 100 unneeded,900 used
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 450; storage space: 100 unneeded,900 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
[0] Mat_CheckInode(): Found 500 nodes out of 500 rows. Not using Inode routines
[1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
[1] Mat_CheckInode(): Found 500 nodes out of 500 rows. Not using Inode routines
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780
[1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374782
[0] MatSetUpMultiply_MPIAIJ(): Using block index set to define scatter
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780
[1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374782
[0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter
[0] VecScatterCreate(): General case: MPI to Seq
[1] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 0; storage space: 0 unneeded,0 used
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 0; storage space: 0 unneeded,0 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
Matrix Object: 2 MPI processes
  type: mpiaij
  rows=1000, cols=900
  total: nonzeros=1800, allocated nonzeros=2000
  total number of mallocs used during MatSetValues calls =0
    not using I-node (on process 0) routines

1 processor:
[0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688 -2080374783
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 1000 X 900; storage space: 200 unneeded,1800 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
[0] Mat_CheckInode(): Found 1000 nodes out of 1000 rows. Not using Inode routines
Matrix Object: 1 MPI processes
  type: seqaij
  rows=1000, cols=900
  total: nonzeros=1800, allocated nonzeros=2000
  total number of mallocs used during MatSetValues calls =0
    not using I-node routines

When I look at the partitioning of the vectors, I have the following for the parallel case:
x:
          0         450
         450         900
y:
           0         500
         500        1000
z:
           0         500
         500        1000

This seems OK to me. 

Certainly I am missing something in performing this matrix-vector multiplication efficiently. Any ideas?

Best regards,

Benjamin

From knepley at gmail.com  Tue May 29 10:56:09 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 29 May 2012 15:56:09 +0000
Subject: [petsc-users] MatMult
In-Reply-To: <657f7efd-db40-4c1c-a35b-d1e13e1ffa56@zembox02.zaas.igi.nl>
References: <c1d5f8a4-d8da-4d18-9556-7f5aba51edef@zembox02.zaas.igi.nl>
	<657f7efd-db40-4c1c-a35b-d1e13e1ffa56@zembox02.zaas.igi.nl>
Message-ID: <CAMYG4G=61p+JXO8OW-kZt_DfO2kYTLnPJHrhEuZOsEyuP0hW+w@mail.gmail.com>

On Tue, May 29, 2012 at 3:52 PM, Benjamin Sanderse <B.Sanderse at cwi.nl>wrote:

> Hello all,
>
> I have a simple question about using MatMult (or MatMultAdd) in parallel.
>
> I am performing the matrix-vector multiplication
>
> z = A*x + y
>
> in my code by using
>
> call MatMultAdd(A,x,y,z,ierr); CHKERRQ(ierr)
>
> A is a sparse matrix, type MPIAIJ, and x, y, and z have been obtained using
>
> call MatGetVecs(A,x,y,ierr); CHKERRQ(ierr)
> call MatGetVecs(A,PETSC_NULL_OBJECT,z,ierr); CHKERRQ(ierr)
>
> x, y, and z are vecs of type mpi.
>
> The problem is that in the sequential case the MatMultAdd is MUCH faster
> than in the parallel case (at least a factor 100 difference).
>

With any performance question, always always always send the output of
-log_summary to petsc-maint at mcs.anl.gov.

   Matt


> As an example, here is the output with some properties of A when using
> -mat_view_info and -info:
>
> 2 processors:
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> -2080374781
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374782
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374782
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [0] MatStashScatterBegin_Private(): No of messages: 0
> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 450; storage space: 100
> unneeded,900 used
> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 450; storage space: 100
> unneeded,900 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> [0] Mat_CheckInode(): Found 500 nodes out of 500 rows. Not using Inode
> routines
> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> [1] Mat_CheckInode(): Found 500 nodes out of 500 rows. Not using Inode
> routines
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374782
> [0] MatSetUpMultiply_MPIAIJ(): Using block index set to define scatter
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374782
> [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter
> [0] VecScatterCreate(): General case: MPI to Seq
> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 0; storage space: 0
> unneeded,0 used
> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 0; storage space: 0
> unneeded,0 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
> Matrix Object: 2 MPI processes
>  type: mpiaij
>  rows=1000, cols=900
>  total: nonzeros=1800, allocated nonzeros=2000
>  total number of mallocs used during MatSetValues calls =0
>    not using I-node (on process 0) routines
>
> 1 processor:
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> -2080374783
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 1000 X 900; storage space: 200
> unneeded,1800 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> [0] Mat_CheckInode(): Found 1000 nodes out of 1000 rows. Not using Inode
> routines
> Matrix Object: 1 MPI processes
>  type: seqaij
>  rows=1000, cols=900
>  total: nonzeros=1800, allocated nonzeros=2000
>  total number of mallocs used during MatSetValues calls =0
>    not using I-node routines
>
> When I look at the partitioning of the vectors, I have the following for
> the parallel case:
> x:
>          0         450
>         450         900
> y:
>           0         500
>         500        1000
> z:
>           0         500
>         500        1000
>
> This seems OK to me.
>
> Certainly I am missing something in performing this matrix-vector
> multiplication efficiently. Any ideas?
>
> Best regards,
>
> Benjamin
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120529/2796f43c/attachment.html>

From jedbrown at mcs.anl.gov  Tue May 29 10:56:51 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Tue, 29 May 2012 10:56:51 -0500
Subject: [petsc-users] MatMult
In-Reply-To: <657f7efd-db40-4c1c-a35b-d1e13e1ffa56@zembox02.zaas.igi.nl>
References: <c1d5f8a4-d8da-4d18-9556-7f5aba51edef@zembox02.zaas.igi.nl>
	<657f7efd-db40-4c1c-a35b-d1e13e1ffa56@zembox02.zaas.igi.nl>
Message-ID: <CAM9tzSkon4iAy05K2ApyJBCG3jx96jRb+2bkWrCr38Ecqi7WHg@mail.gmail.com>

On Tue, May 29, 2012 at 10:52 AM, Benjamin Sanderse <B.Sanderse at cwi.nl>wrote:

> Hello all,
>
> I have a simple question about using MatMult (or MatMultAdd) in parallel.
>
> I am performing the matrix-vector multiplication
>
> z = A*x + y
>
> in my code by using
>
> call MatMultAdd(A,x,y,z,ierr); CHKERRQ(ierr)
>
> A is a sparse matrix, type MPIAIJ, and x, y, and z have been obtained using
>
> call MatGetVecs(A,x,y,ierr); CHKERRQ(ierr)
> call MatGetVecs(A,PETSC_NULL_OBJECT,z,ierr); CHKERRQ(ierr)
>
> x, y, and z are vecs of type mpi.
>
> The problem is that in the sequential case the MatMultAdd is MUCH faster
> than in the parallel case (at least a factor 100 difference).
>

1. Send output of -log_summary

2. This matrix is tiny (1000x1000) and very sparse (at most 2 nonzeros per
row) so you should not expect speedup from running in parallel.


>
> As an example, here is the output with some properties of A when using
> -mat_view_info and -info:
>
> 2 processors:
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> -2080374781
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374782
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374782
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [0] MatStashScatterBegin_Private(): No of messages: 0
> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 450; storage space: 100
> unneeded,900 used
> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 450; storage space: 100
> unneeded,900 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> [0] Mat_CheckInode(): Found 500 nodes out of 500 rows. Not using Inode
> routines
> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> [1] Mat_CheckInode(): Found 500 nodes out of 500 rows. Not using Inode
> routines
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374782
> [0] MatSetUpMultiply_MPIAIJ(): Using block index set to define scatter
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374782
> [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter
> [0] VecScatterCreate(): General case: MPI to Seq
> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 0; storage space: 0
> unneeded,0 used
> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 0; storage space: 0
> unneeded,0 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
> Matrix Object: 2 MPI processes
>  type: mpiaij
>  rows=1000, cols=900
>  total: nonzeros=1800, allocated nonzeros=2000
>  total number of mallocs used during MatSetValues calls =0
>    not using I-node (on process 0) routines
>
> 1 processor:
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> -2080374783
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 1000 X 900; storage space: 200
> unneeded,1800 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> [0] Mat_CheckInode(): Found 1000 nodes out of 1000 rows. Not using Inode
> routines
> Matrix Object: 1 MPI processes
>  type: seqaij
>  rows=1000, cols=900
>  total: nonzeros=1800, allocated nonzeros=2000
>  total number of mallocs used during MatSetValues calls =0
>    not using I-node routines
>
> When I look at the partitioning of the vectors, I have the following for
> the parallel case:
> x:
>          0         450
>         450         900
> y:
>           0         500
>         500        1000
> z:
>           0         500
>         500        1000
>
> This seems OK to me.
>
> Certainly I am missing something in performing this matrix-vector
> multiplication efficiently. Any ideas?
>
> Best regards,
>
> Benjamin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120529/5a4adccd/attachment-0001.html>

From B.Sanderse at cwi.nl  Wed May 30 02:23:43 2012
From: B.Sanderse at cwi.nl (Benjamin Sanderse)
Date: Wed, 30 May 2012 09:23:43 +0200 (CEST)
Subject: [petsc-users] MatMult
In-Reply-To: <CAM9tzSkon4iAy05K2ApyJBCG3jx96jRb+2bkWrCr38Ecqi7WHg@mail.gmail.com>
Message-ID: <8eff2f56-fbe5-4840-86f6-bece772c9e92@zembox02.zaas.igi.nl>

Sorry for forgetting -log_summary. Attached are log_summary for 1 and 2 processors, for both a problem with about 1000 unknowns and one with 125000 unknowns. The summary is for a run of the entire code, which involves many MatMults. I hope this still provides insight on what is going on.
As you can see there is an extraordinary use of MatGetRow - I am working to change this - but they should not influence speed of the MatMults. Any thoughts?

Benjamin

----- Original Message -----
From: "Jed Brown" <jedbrown at mcs.anl.gov>
To: "PETSc users list" <petsc-users at mcs.anl.gov>
Sent: Tuesday, May 29, 2012 5:56:51 PM
Subject: Re: [petsc-users] MatMult

On Tue, May 29, 2012 at 10:52 AM, Benjamin Sanderse <B.Sanderse at cwi.nl>wrote:

> Hello all,
>
> I have a simple question about using MatMult (or MatMultAdd) in parallel.
>
> I am performing the matrix-vector multiplication
>
> z = A*x + y
>
> in my code by using
>
> call MatMultAdd(A,x,y,z,ierr); CHKERRQ(ierr)
>
> A is a sparse matrix, type MPIAIJ, and x, y, and z have been obtained using
>
> call MatGetVecs(A,x,y,ierr); CHKERRQ(ierr)
> call MatGetVecs(A,PETSC_NULL_OBJECT,z,ierr); CHKERRQ(ierr)
>
> x, y, and z are vecs of type mpi.
>
> The problem is that in the sequential case the MatMultAdd is MUCH faster
> than in the parallel case (at least a factor 100 difference).
>

1. Send output of -log_summary

2. This matrix is tiny (1000x1000) and very sparse (at most 2 nonzeros per
row) so you should not expect speedup from running in parallel.


>
> As an example, here is the output with some properties of A when using
> -mat_view_info and -info:
>
> 2 processors:
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> -2080374781
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374782
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374782
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [0] MatStashScatterBegin_Private(): No of messages: 0
> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 450; storage space: 100
> unneeded,900 used
> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 450; storage space: 100
> unneeded,900 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> [0] Mat_CheckInode(): Found 500 nodes out of 500 rows. Not using Inode
> routines
> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> [1] Mat_CheckInode(): Found 500 nodes out of 500 rows. Not using Inode
> routines
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374782
> [0] MatSetUpMultiply_MPIAIJ(): Using block index set to define scatter
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374780
> [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374782
> [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter
> [0] VecScatterCreate(): General case: MPI to Seq
> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 0; storage space: 0
> unneeded,0 used
> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 0; storage space: 0
> unneeded,0 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
> Matrix Object: 2 MPI processes
>  type: mpiaij
>  rows=1000, cols=900
>  total: nonzeros=1800, allocated nonzeros=2000
>  total number of mallocs used during MatSetValues calls =0
>    not using I-node (on process 0) routines
>
> 1 processor:
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> -2080374783
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 1000 X 900; storage space: 200
> unneeded,1800 used
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> [0] Mat_CheckInode(): Found 1000 nodes out of 1000 rows. Not using Inode
> routines
> Matrix Object: 1 MPI processes
>  type: seqaij
>  rows=1000, cols=900
>  total: nonzeros=1800, allocated nonzeros=2000
>  total number of mallocs used during MatSetValues calls =0
>    not using I-node routines
>
> When I look at the partitioning of the vectors, I have the following for
> the parallel case:
> x:
>          0         450
>         450         900
> y:
>           0         500
>         500        1000
> z:
>           0         500
>         500        1000
>
> This seems OK to me.
>
> Certainly I am missing something in performing this matrix-vector
> multiplication efficiently. Any ideas?
>
> Best regards,
>
> Benjamin
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: out_large_n1
Type: application/octet-stream
Size: 11031 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/d3a76f7f/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: out_large_n2
Type: application/octet-stream
Size: 11908 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/d3a76f7f/attachment-0005.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: out_small_n1
Type: application/octet-stream
Size: 11030 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/d3a76f7f/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: out_small_n2
Type: application/octet-stream
Size: 11908 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/d3a76f7f/attachment-0007.obj>

From jedbrown at mcs.anl.gov  Wed May 30 06:43:26 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 May 2012 06:43:26 -0500
Subject: [petsc-users] MatMult
In-Reply-To: <8eff2f56-fbe5-4840-86f6-bece772c9e92@zembox02.zaas.igi.nl>
References: <CAM9tzSkon4iAy05K2ApyJBCG3jx96jRb+2bkWrCr38Ecqi7WHg@mail.gmail.com>
	<8eff2f56-fbe5-4840-86f6-bece772c9e92@zembox02.zaas.igi.nl>
Message-ID: <CAM9tzS=BHw277r-5e-0AOQV4OYGOU5LigbxAzEUr6WH-PkobPA@mail.gmail.com>

On Wed, May 30, 2012 at 2:23 AM, Benjamin Sanderse <B.Sanderse at cwi.nl>wrote:

> Sorry for forgetting -log_summary. Attached are log_summary for 1 and 2
> processors, for both a problem with about 1000 unknowns and one with 125000
> unknowns. The summary is for a run of the entire code, which involves many
> MatMults. I hope this still provides insight on what is going on.
> As you can see there is an extraordinary use of MatGetRow - I am working
> to change this - but they should not influence speed of the MatMults. Any
> thoughts?
>

1. What computer is this running on? Specifically, how is its memory
hierarchy laid out?
http://www.mcs.anl.gov/petsc/documentation/faq.html#computers  Can you run
the benchmarks in src/benchmarks/streams/?

2. It's worth heeding this message, the performance will look significantly
different. If the parallel version is still much slower, please send that
-log_summary.

      ##########################################################
      #                                                        #
      #                          WARNING!!!                    #
      #                                                        #
      #   This code was compiled with a debugging option,      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################



>
> Benjamin
>
> ----- Original Message -----
> From: "Jed Brown" <jedbrown at mcs.anl.gov>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Tuesday, May 29, 2012 5:56:51 PM
> Subject: Re: [petsc-users] MatMult
>
> On Tue, May 29, 2012 at 10:52 AM, Benjamin Sanderse <B.Sanderse at cwi.nl
> >wrote:
>
> > Hello all,
> >
> > I have a simple question about using MatMult (or MatMultAdd) in parallel.
> >
> > I am performing the matrix-vector multiplication
> >
> > z = A*x + y
> >
> > in my code by using
> >
> > call MatMultAdd(A,x,y,z,ierr); CHKERRQ(ierr)
> >
> > A is a sparse matrix, type MPIAIJ, and x, y, and z have been obtained
> using
> >
> > call MatGetVecs(A,x,y,ierr); CHKERRQ(ierr)
> > call MatGetVecs(A,PETSC_NULL_OBJECT,z,ierr); CHKERRQ(ierr)
> >
> > x, y, and z are vecs of type mpi.
> >
> > The problem is that in the sequential case the MatMultAdd is MUCH faster
> > than in the parallel case (at least a factor 100 difference).
> >
>
> 1. Send output of -log_summary
>
> 2. This matrix is tiny (1000x1000) and very sparse (at most 2 nonzeros per
> row) so you should not expect speedup from running in parallel.
>
>
> >
> > As an example, here is the output with some properties of A when using
> > -mat_view_info and -info:
> >
> > 2 processors:
> > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> > -2080374781
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374780
> > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374782
> > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374782
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374780
> > [0] MatStashScatterBegin_Private(): No of messages: 0
> > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 450; storage space: 100
> > unneeded,900 used
> > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 450; storage space: 100
> > unneeded,900 used
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> > [0] Mat_CheckInode(): Found 500 nodes out of 500 rows. Not using Inode
> > routines
> > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> > [1] Mat_CheckInode(): Found 500 nodes out of 500 rows. Not using Inode
> > routines
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374780
> > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374782
> > [0] MatSetUpMultiply_MPIAIJ(): Using block index set to define scatter
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374780
> > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374782
> > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter
> > [0] VecScatterCreate(): General case: MPI to Seq
> > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 0; storage space: 0
> > unneeded,0 used
> > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 0; storage space: 0
> > unneeded,0 used
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
> > Matrix Object: 2 MPI processes
> >  type: mpiaij
> >  rows=1000, cols=900
> >  total: nonzeros=1800, allocated nonzeros=2000
> >  total number of mallocs used during MatSetValues calls =0
> >    not using I-node (on process 0) routines
> >
> > 1 processor:
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> > -2080374783
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 1000 X 900; storage space: 200
> > unneeded,1800 used
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> > [0] Mat_CheckInode(): Found 1000 nodes out of 1000 rows. Not using Inode
> > routines
> > Matrix Object: 1 MPI processes
> >  type: seqaij
> >  rows=1000, cols=900
> >  total: nonzeros=1800, allocated nonzeros=2000
> >  total number of mallocs used during MatSetValues calls =0
> >    not using I-node routines
> >
> > When I look at the partitioning of the vectors, I have the following for
> > the parallel case:
> > x:
> >          0         450
> >         450         900
> > y:
> >           0         500
> >         500        1000
> > z:
> >           0         500
> >         500        1000
> >
> > This seems OK to me.
> >
> > Certainly I am missing something in performing this matrix-vector
> > multiplication efficiently. Any ideas?
> >
> > Best regards,
> >
> > Benjamin
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/16534f13/attachment.html>

From w_ang_temp at 163.com  Wed May 30 08:20:29 2012
From: w_ang_temp at 163.com (w_ang_temp)
Date: Wed, 30 May 2012 21:20:29 +0800 (CST)
Subject: [petsc-users] PETSc installed without X windows on this machine
Message-ID: <5526cfd4.12183.1379de7983b.Coremail.w_ang_temp@163.com>

Hello
    When I try to view a matrix with the option -mat_view_draw, I get :
    [0]PETSC ERROR: PETSc installed without X windows on this machine proceeding without graphics.
    But calling MatView in side the code is ok.
    I install the libX11-dev as Matt said in one message in the petsc-users in 2008,  but the problem doesn't solved.
    The system is ubuntu 10.04.
    So what might be the reason?
    Thanks.
                                          Jim

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/5c5c5449/attachment.html>

From knepley at gmail.com  Wed May 30 08:27:40 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 May 2012 09:27:40 -0400
Subject: [petsc-users] PETSc installed without X windows on this machine
In-Reply-To: <5526cfd4.12183.1379de7983b.Coremail.w_ang_temp@163.com>
References: <5526cfd4.12183.1379de7983b.Coremail.w_ang_temp@163.com>
Message-ID: <CAMYG4GnLDywr8qk9R1FuTsNbTTf1i2GbzwTx08tHw_K4Hq=Fkw@mail.gmail.com>

On Wed, May 30, 2012 at 9:20 AM, w_ang_temp <w_ang_temp at 163.com> wrote:

> Hello
>     When I try to view a matrix with the option -mat_view_draw, I get :
>     [0]PETSC ERROR: PETSc installed without X windows on this machine
> proceeding without graphics.
>     But calling MatView in side the code is ok.
>

Are you saying that from within the code, MatView() draws a picture in an
X-window? I do not see
how that can possibly be the case. PETSc did not compile support for that,
as it says in the error
message.

   Matt


>     I install the libX11-dev as Matt said in one message in the
> petsc-users in 2008,  but the problem doesn't solved.
>     The system is ubuntu 10.04.
>     So what might be the reason?
>     Thanks.
>                                           Jim
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/4c263907/attachment.html>

From w_ang_temp at 163.com  Wed May 30 08:46:29 2012
From: w_ang_temp at 163.com (w_ang_temp)
Date: Wed, 30 May 2012 21:46:29 +0800 (CST)
Subject: [petsc-users] PETSc installed without X windows on this machine
In-Reply-To: <CAMYG4GnLDywr8qk9R1FuTsNbTTf1i2GbzwTx08tHw_K4Hq=Fkw@mail.gmail.com>
References: <5526cfd4.12183.1379de7983b.Coremail.w_ang_temp@163.com>
	<CAMYG4GnLDywr8qk9R1FuTsNbTTf1i2GbzwTx08tHw_K4Hq=Fkw@mail.gmail.com>
Message-ID: <62361f46.125ac.1379dff6439.Coremail.w_ang_temp@163.com>

Hello Matt
     I am sorry for the vague statements.
     When use MatView() in the code, it can print the matrix by the CSR form in the terminal.
     When I try to view the matrix with the runtime option -mat_view_draw, I get the error message. I just want to draw the matrix nonzero structure.
      Thanks.
                                                 Jim




>? 2012-05-30 21:27:40?"Matthew Knepley" <knepley at gmail.com> ???
>On Wed, May 30, 2012 at 9:20 AM, w_ang_temp <w_ang_temp at 163.com> wrote:

>Hello
>    When I try to view a matrix with the option -mat_view_draw, I get :
>    [0]PETSC ERROR: PETSc installed without X windows on this machine proceeding without graphics.
>    But calling MatView in side the code is ok.


>Are you saying that from within the code, MatView() draws a picture in an X-window? I do not see
>how that can possibly be the case. PETSc did not compile support for that, as it says in the error
>message.


>   Matt
 
>    I install the libX11-dev as Matt said in one message in the petsc-users in 2008,  but the problem doesn't solved.
>    The system is ubuntu 10.04.
>    So what might be the reason?
>    Thanks.
>                                          Jim

 








--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/99ec8e1b/attachment-0001.html>

From jedbrown at mcs.anl.gov  Wed May 30 08:48:50 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 May 2012 08:48:50 -0500
Subject: [petsc-users] PETSc installed without X windows on this machine
In-Reply-To: <62361f46.125ac.1379dff6439.Coremail.w_ang_temp@163.com>
References: <5526cfd4.12183.1379de7983b.Coremail.w_ang_temp@163.com>
	<CAMYG4GnLDywr8qk9R1FuTsNbTTf1i2GbzwTx08tHw_K4Hq=Fkw@mail.gmail.com>
	<62361f46.125ac.1379dff6439.Coremail.w_ang_temp@163.com>
Message-ID: <CAM9tzSkPrqjY=+LmHjic7d8JdF=p0E_MOWiBkyTtHqMkx3UsLA@mail.gmail.com>

Configure --with-x if you want that.
On May 30, 2012 8:46 AM, "w_ang_temp" <w_ang_temp at 163.com> wrote:

> Hello Matt
>      I am sorry for the vague statements.
>      When use MatView() in the code, it can print the matrix by the CSR
> form in the terminal.
>      When I try to view the matrix with the runtime option -mat_view_draw,
> I get the error message. I just want to draw the matrix nonzero structure.
>       Thanks.
>                                                  Jim
>
>
> >? 2012-05-30 21:27:40?"Matthew Knepley" <knepley at gmail.com> ???
>
> >On Wed, May 30, 2012 at 9:20 AM, w_ang_temp <w_ang_temp at 163.com> wrote:
>
>>  >Hello
>> >    When I try to view a matrix with the option -mat_view_draw, I get :
>> >    [0]PETSC ERROR: PETSc installed without X windows on this machine
>> proceeding without graphics.
>> >    But calling MatView in side the code is ok.
>>
>
> >Are you saying that from within the code, MatView() draws a picture in an
> X-window? I do not see
> >how that can possibly be the case. PETSc did not compile support for
> that, as it says in the error
> >message.
>
> >   Matt
>
>
>>  >    I install the libX11-dev as Matt said in one message in the
>> petsc-users in 2008,  but the problem doesn't solved.
>> >    The system is ubuntu 10.04.
>> >    So what might be the reason?
>> >    Thanks.
>> >                                          Jim
>>
>>
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/87db88ef/attachment.html>

From w_ang_temp at 163.com  Wed May 30 08:54:34 2012
From: w_ang_temp at 163.com (w_ang_temp)
Date: Wed, 30 May 2012 21:54:34 +0800 (CST)
Subject: [petsc-users] PETSc installed without X windows on this machine
In-Reply-To: <CAM9tzSkPrqjY=+LmHjic7d8JdF=p0E_MOWiBkyTtHqMkx3UsLA@mail.gmail.com>
References: <5526cfd4.12183.1379de7983b.Coremail.w_ang_temp@163.com>
	<CAMYG4GnLDywr8qk9R1FuTsNbTTf1i2GbzwTx08tHw_K4Hq=Fkw@mail.gmail.com>
	<62361f46.125ac.1379dff6439.Coremail.w_ang_temp@163.com>
	<CAM9tzSkPrqjY=+LmHjic7d8JdF=p0E_MOWiBkyTtHqMkx3UsLA@mail.gmail.com>
Message-ID: <790e3a57.1283a.1379e06ca63.Coremail.w_ang_temp@163.com>

Hello Jed
Thank you very much. I think I have know the reason.
Thanks.
                         Jim




>At 2012-05-30 21:48:50,"Jed Brown" <jedbrown at mcs.anl.gov> wrote:


>Configure --with-x if you want that.

>On May 30, 2012 8:46 AM, "w_ang_temp" <w_ang_temp at 163.com> wrote:

>Hello Matt
>     I am sorry for the vague statements.
>     When use MatView() in the code, it can print the matrix by the CSR form in the terminal.
>     When I try to view the matrix with the runtime option -mat_view_draw, I get the error message. I just want to draw the matrix nonzero >structure.
>      Thanks.
>                                                 Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/7a3604d6/attachment.html>

From thomas.de-soza at edf.fr  Wed May 30 09:03:49 2012
From: thomas.de-soza at edf.fr (Thomas DE-SOZA)
Date: Wed, 30 May 2012 16:03:49 +0200
Subject: [petsc-users] =?iso-8859-1?q?R=E9f=2E_=3A_Re=3A__PETSc_installed_?=
 =?iso-8859-1?q?without_X_windows_on_this_machine?=
Message-ID: <OFCA49FF9F.06C66303-ONC1257A0E.004D200E-C1257A0E.004D4126@notes.edfgdf.fr>

Hi,

To print the matrix in ASCII form you must use "-mat_view" and not "-mat_view_draw".

Thomas





w_ang_temp at 163.com
Envoy? par : petsc-users-bounces at mcs.anl.gov
30/05/2012 15:46
Veuillez r?pondre ? petsc-users

 
        Pour :  petsc-users at mcs.anl.gov
        cc :    (ccc : Thomas DE-SOZA/A/EDF/FR)
        Objet : Re: [petsc-users] PETSc installed without X windows on this machine


Hello Matt
     I am sorry for the vague statements. 
     When use MatView() in the code, it can print the matrix by the CSR 
form in the terminal.
     When I try to view the matrix with the runtime option -mat_view_draw, 
I get the error message. I just want to draw the matrix nonzero structure.
      Thanks.
                                                 Jim


>? 2012-05-30 21:27:40?"Matthew Knepley" <knepley at gmail.com> ???
>On Wed, May 30, 2012 at 9:20 AM, w_ang_temp <w_ang_temp at 163.com> wrote:
>Hello
>    When I try to view a matrix with the option -mat_view_draw, I get :
>    [0]PETSC ERROR: PETSc installed without X windows on this machine 
proceeding without graphics.
>    But calling MatView in side the code is ok.

>Are you saying that from within the code, MatView() draws a picture in an 
X-window? I do not see
>how that can possibly be the case. PETSc did not compile support for 
that, as it says in the error
>message.

>   Matt
 
>    I install the libX11-dev as Matt said in one message in the 
petsc-users in 2008,  but the problem doesn't solved.
>    The system is ubuntu 10.04.
>    So what might be the reason?
>    Thanks.
>                                          Jim

 





-- 
What most experimenters take for granted before they begin their 
experiments is infinitely more interesting than any results to which their 
experiments lead.
-- Norbert Wiener




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/7de11b1b/attachment.html>
-------------- next part --------------



Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse.

Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message.

Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus.
____________________________________________________

This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval.

If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message.

E-mail communication cannot be guaranteed to be timely secure, error or virus-free.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/7de11b1b/attachment-0001.html>

From mike.hui.zhang at hotmail.com  Wed May 30 09:05:50 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Wed, 30 May 2012 16:05:50 +0200
Subject: [petsc-users] randomness from MATSOLVERPACKAGE
Message-ID: <BLU0-SMTP134476792BC7FBF2E1B0340CB0A0@phx.gbl>

I suspect there are some randomness from external matsolverpackage. I use a parallel matrix
as preconditioner, and the preconditioner is exactly LU factored, then I saw the KSP (gmres)
residuals are different from different runs.

Is it true of randomness from matsolverpackage?  Thanks a lot!

From B.Sanderse at cwi.nl  Wed May 30 09:15:44 2012
From: B.Sanderse at cwi.nl (Benjamin Sanderse)
Date: Wed, 30 May 2012 16:15:44 +0200
Subject: [petsc-users] MatMult
In-Reply-To: <CAM9tzS=BHw277r-5e-0AOQV4OYGOU5LigbxAzEUr6WH-PkobPA@mail.gmail.com>
References: <CAM9tzSkon4iAy05K2ApyJBCG3jx96jRb+2bkWrCr38Ecqi7WHg@mail.gmail.com>
	<8eff2f56-fbe5-4840-86f6-bece772c9e92@zembox02.zaas.igi.nl>
	<CAM9tzS=BHw277r-5e-0AOQV4OYGOU5LigbxAzEUr6WH-PkobPA@mail.gmail.com>
Message-ID: <8B2C6A76-0D04-4154-80AF-9B4844FC917A@cwi.nl>

Hi Jed, 

Moving to the optimized version of Petsc (without debugging) basically removed the issue. Thanks a lot!

Benjamin

Op 30 mei 2012, om 13:43 heeft Jed Brown het volgende geschreven:

> On Wed, May 30, 2012 at 2:23 AM, Benjamin Sanderse <B.Sanderse at cwi.nl> wrote:
> Sorry for forgetting -log_summary. Attached are log_summary for 1 and 2 processors, for both a problem with about 1000 unknowns and one with 125000 unknowns. The summary is for a run of the entire code, which involves many MatMults. I hope this still provides insight on what is going on.
> As you can see there is an extraordinary use of MatGetRow - I am working to change this - but they should not influence speed of the MatMults. Any thoughts?
> 
> 1. What computer is this running on? Specifically, how is its memory hierarchy laid out?  http://www.mcs.anl.gov/petsc/documentation/faq.html#computers  Can you run the benchmarks in src/benchmarks/streams/?
> 
> 2. It's worth heeding this message, the performance will look significantly different. If the parallel version is still much slower, please send that -log_summary.
> 
>       ##########################################################
>       #                                                        #
>       #                          WARNING!!!                    #
>       #                                                        #
>       #   This code was compiled with a debugging option,      #
>       #   To get timing results run ./configure                #
>       #   using --with-debugging=no, the performance will      #
>       #   be generally two or three times faster.              #
>       #                                                        #
>       ##########################################################
> 
>  
> 
> Benjamin
> 
> ----- Original Message -----
> From: "Jed Brown" <jedbrown at mcs.anl.gov>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Tuesday, May 29, 2012 5:56:51 PM
> Subject: Re: [petsc-users] MatMult
> 
> On Tue, May 29, 2012 at 10:52 AM, Benjamin Sanderse <B.Sanderse at cwi.nl>wrote:
> 
> > Hello all,
> >
> > I have a simple question about using MatMult (or MatMultAdd) in parallel.
> >
> > I am performing the matrix-vector multiplication
> >
> > z = A*x + y
> >
> > in my code by using
> >
> > call MatMultAdd(A,x,y,z,ierr); CHKERRQ(ierr)
> >
> > A is a sparse matrix, type MPIAIJ, and x, y, and z have been obtained using
> >
> > call MatGetVecs(A,x,y,ierr); CHKERRQ(ierr)
> > call MatGetVecs(A,PETSC_NULL_OBJECT,z,ierr); CHKERRQ(ierr)
> >
> > x, y, and z are vecs of type mpi.
> >
> > The problem is that in the sequential case the MatMultAdd is MUCH faster
> > than in the parallel case (at least a factor 100 difference).
> >
> 
> 1. Send output of -log_summary
> 
> 2. This matrix is tiny (1000x1000) and very sparse (at most 2 nonzeros per
> row) so you should not expect speedup from running in parallel.
> 
> 
> >
> > As an example, here is the output with some properties of A when using
> > -mat_view_info and -info:
> >
> > 2 processors:
> > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> > -2080374781
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374780
> > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374782
> > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374782
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374780
> > [0] MatStashScatterBegin_Private(): No of messages: 0
> > [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.
> > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 450; storage space: 100
> > unneeded,900 used
> > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 450; storage space: 100
> > unneeded,900 used
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> > [0] Mat_CheckInode(): Found 500 nodes out of 500 rows. Not using Inode
> > routines
> > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> > [1] Mat_CheckInode(): Found 500 nodes out of 500 rows. Not using Inode
> > routines
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374780
> > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374782
> > [0] MatSetUpMultiply_MPIAIJ(): Using block index set to define scatter
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374780
> > [1] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> > -2080374782
> > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter
> > [0] VecScatterCreate(): General case: MPI to Seq
> > [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 0; storage space: 0
> > unneeded,0 used
> > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 500 X 0; storage space: 0
> > unneeded,0 used
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0
> > Matrix Object: 2 MPI processes
> >  type: mpiaij
> >  rows=1000, cols=900
> >  total: nonzeros=1800, allocated nonzeros=2000
> >  total number of mallocs used during MatSetValues calls =0
> >    not using I-node (on process 0) routines
> >
> > 1 processor:
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850688
> > -2080374783
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 1000 X 900; storage space: 200
> > unneeded,1800 used
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 2
> > [0] Mat_CheckInode(): Found 1000 nodes out of 1000 rows. Not using Inode
> > routines
> > Matrix Object: 1 MPI processes
> >  type: seqaij
> >  rows=1000, cols=900
> >  total: nonzeros=1800, allocated nonzeros=2000
> >  total number of mallocs used during MatSetValues calls =0
> >    not using I-node routines
> >
> > When I look at the partitioning of the vectors, I have the following for
> > the parallel case:
> > x:
> >          0         450
> >         450         900
> > y:
> >           0         500
> >         500        1000
> > z:
> >           0         500
> >         500        1000
> >
> > This seems OK to me.
> >
> > Certainly I am missing something in performing this matrix-vector
> > multiplication efficiently. Any ideas?
> >
> > Best regards,
> >
> > Benjamin
> >
> 

-- 
Ir. B. Sanderse
 
Centrum Wiskunde en Informatica
Science Park 123
1098 XG Amsterdam

t: +31 20 592 4161
e: sanderse at cwi.nl

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/be2bfcfc/attachment-0001.html>

From mike.hui.zhang at hotmail.com  Wed May 30 09:21:45 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Wed, 30 May 2012 16:21:45 +0200
Subject: [petsc-users] randomness from MATSOLVERPACKAGE
In-Reply-To: <BLU0-SMTP134476792BC7FBF2E1B0340CB0A0@phx.gbl>
References: <BLU0-SMTP134476792BC7FBF2E1B0340CB0A0@phx.gbl>
Message-ID: <BLU0-SMTP35272A69CCE25339773371FCB0A0@phx.gbl>


> I suspect there are some randomness from external matsolverpackage. I use a parallel matrix
> as preconditioner, and the preconditioner is exactly LU factored, then I saw the KSP (gmres)
> residuals are different from different runs.

Besides, for sequential matrix, there is no such phenomenon, only for parallel matrix.

> 
> Is it true of randomness from matsolverpackage?  Thanks a lot!


From jedbrown at mcs.anl.gov  Wed May 30 09:27:54 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 May 2012 09:27:54 -0500
Subject: [petsc-users] randomness from MATSOLVERPACKAGE
In-Reply-To: <BLU0-SMTP35272A69CCE25339773371FCB0A0@phx.gbl>
References: <BLU0-SMTP134476792BC7FBF2E1B0340CB0A0@phx.gbl>
	<BLU0-SMTP35272A69CCE25339773371FCB0A0@phx.gbl>
Message-ID: <CAM9tzSmks=PGeU55kXYi3127oLjQqXtiEDDvGJJsPhBd8m4WcQ@mail.gmail.com>

How is the matrix assembled?

How different? You should usually not expect deterministic associativity.
If your results are sensitive, you may be able to make it deterministic
(with extra work and choice of package), but you should probably
reformulate the problem.
On May 30, 2012 9:21 AM, "Hui Zhang" <mike.hui.zhang at hotmail.com> wrote:

>
> > I suspect there are some randomness from external matsolverpackage. I
> use a parallel matrix
> > as preconditioner, and the preconditioner is exactly LU factored, then I
> saw the KSP (gmres)
> > residuals are different from different runs.
>
> Besides, for sequential matrix, there is no such phenomenon, only for
> parallel matrix.
>
> >
> > Is it true of randomness from matsolverpackage?  Thanks a lot!
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/369f5a83/attachment.html>

From mike.hui.zhang at hotmail.com  Wed May 30 09:37:11 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Wed, 30 May 2012 16:37:11 +0200
Subject: [petsc-users] randomness from MATSOLVERPACKAGE
In-Reply-To: <CAM9tzSmks=PGeU55kXYi3127oLjQqXtiEDDvGJJsPhBd8m4WcQ@mail.gmail.com>
References: <BLU0-SMTP134476792BC7FBF2E1B0340CB0A0@phx.gbl>
	<BLU0-SMTP35272A69CCE25339773371FCB0A0@phx.gbl>
	<CAM9tzSmks=PGeU55kXYi3127oLjQqXtiEDDvGJJsPhBd8m4WcQ@mail.gmail.com>
Message-ID: <BLU0-SMTP2018E38A996300A3FA2D55ACB0A0@phx.gbl>


> How is the matrix assembled?
> 
In a deterministic way.
> How different?
> 
Not very much. The last residual can vary in its trailing digits, e.g. from 6.5819e-7 to 6.5820e-7
> You should usually not expect deterministic associativity.
> 
Ok, I see.  Thanks!

> If your results are sensitive, you may be able to make it deterministic (with extra work and choice of package), but you should probably reformulate the problem
> 
> On May 30, 2012 9:21 AM, "Hui Zhang" <mike.hui.zhang at hotmail.com> wrote:
> 
> > I suspect there are some randomness from external matsolverpackage. I use a parallel matrix
> > as preconditioner, and the preconditioner is exactly LU factored, then I saw the KSP (gmres)
> > residuals are different from different runs.
> 
> Besides, for sequential matrix, there is no such phenomenon, only for parallel matrix.
> 
> >
> > Is it true of randomness from matsolverpackage?  Thanks a lot!
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/4d713f18/attachment.html>

From jedbrown at mcs.anl.gov  Wed May 30 09:40:25 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 May 2012 09:40:25 -0500
Subject: [petsc-users] randomness from MATSOLVERPACKAGE
In-Reply-To: <BLU0-SMTP2018E38A996300A3FA2D55ACB0A0@phx.gbl>
References: <BLU0-SMTP134476792BC7FBF2E1B0340CB0A0@phx.gbl>
	<BLU0-SMTP35272A69CCE25339773371FCB0A0@phx.gbl>
	<CAM9tzSmks=PGeU55kXYi3127oLjQqXtiEDDvGJJsPhBd8m4WcQ@mail.gmail.com>
	<BLU0-SMTP2018E38A996300A3FA2D55ACB0A0@phx.gbl>
Message-ID: <CAM9tzSnAwZvT-4OxKV9tGN=R-owmwTtKEjHfihjcLLmHmk6i_w@mail.gmail.com>

On Wed, May 30, 2012 at 9:37 AM, Hui Zhang <mike.hui.zhang at hotmail.com>wrote:

> How is the matrix assembled?
>
> In a deterministic way.
>

Do you use ADD_VALUES with off-process entries? If so, the message order
may be different. You can also add -vecscatter_reproduce to make VecScatter
with ADD_VALUES process the messages in a static order instead of in the
order they arrive (somewhat slower).

> How different?
>
> Not very much. The last residual can vary in its trailing digits, e.g.
> from 6.5819e-7 to 6.5820e-7
>

This looks fine.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/6f700121/attachment.html>

From mike.hui.zhang at hotmail.com  Wed May 30 09:48:56 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Wed, 30 May 2012 16:48:56 +0200
Subject: [petsc-users] randomness from MATSOLVERPACKAGE
In-Reply-To: <CAM9tzSnAwZvT-4OxKV9tGN=R-owmwTtKEjHfihjcLLmHmk6i_w@mail.gmail.com>
References: <BLU0-SMTP134476792BC7FBF2E1B0340CB0A0@phx.gbl>
	<BLU0-SMTP35272A69CCE25339773371FCB0A0@phx.gbl>
	<CAM9tzSmks=PGeU55kXYi3127oLjQqXtiEDDvGJJsPhBd8m4WcQ@mail.gmail.com>
	<BLU0-SMTP2018E38A996300A3FA2D55ACB0A0@phx.gbl>
	<CAM9tzSnAwZvT-4OxKV9tGN=R-owmwTtKEjHfihjcLLmHmk6i_w@mail.gmail.com>
Message-ID: <BLU0-SMTP414BD86C14C0F63EDF98C01CB0A0@phx.gbl>


On May 30, 2012, at 4:40 PM, Jed Brown wrote:

> On Wed, May 30, 2012 at 9:37 AM, Hui Zhang <mike.hui.zhang at hotmail.com> wrote:
>> How is the matrix assembled?
>> 
> In a deterministic way.
> 
> Do you use ADD_VALUES with off-process entries? If so, the message order may be different. You can also add -vecscatter_reproduce to make VecScatter with ADD_VALUES process the messages in a static order instead of in the order they arrive (somewhat slower).

Thanks for details. For the moment, it seems not necessary for me.

>> How different?
>> 
> Not very much. The last residual can vary in its trailing digits, e.g. from 6.5819e-7 to 6.5820e-7
> 
> This looks fine.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/acd4d595/attachment.html>

From C.Klaij at marin.nl  Wed May 30 09:50:47 2012
From: C.Klaij at marin.nl (Klaij, Christiaan)
Date: Wed, 30 May 2012 14:50:47 +0000
Subject: [petsc-users] search in archives fails
Message-ID: <A9AE64C525438E45B1D60F4D09AEAA9E0623D3@MAR160N1.marin.local>

I'm trying to find something in the petsc-users mailings list archives but the search function gives the error:

Bug in Mailman version 2.1.14

We're sorry, we hit a bug!

Please inform the webmaster for this site of this problem. Printing of traceback and other system information has been explicitly inhibited, but the webmaster can find this information in the Mailman error logs.


dr. ir. Christiaan Klaij
CFD Researcher
Research & Development
E mailto:C.Klaij at marin.nl
T +31 317 49 33 44

MARIN
2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands
T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl


From keita at cray.com  Wed May 30 10:52:58 2012
From: keita at cray.com (Keita Teranishi)
Date: Wed, 30 May 2012 15:52:58 +0000
Subject: [petsc-users] PETSc with 64 bit integer
Message-ID: <0D7B179941D43E46800EC415D269A65D0F1CB8@CFWEX01.americas.cray.com>

Hi,

Is there any requirement for BLAS/LAPACK to install PETSc with 64-bit integer option?

Thanks,
====================================
Keita Teranishi
Scientific Library Group
Cray Inc.
keita at cray.com
======================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/6b640634/attachment.html>

From jedbrown at mcs.anl.gov  Wed May 30 10:58:20 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 May 2012 10:58:20 -0500
Subject: [petsc-users] PETSc with 64 bit integer
In-Reply-To: <0D7B179941D43E46800EC415D269A65D0F1CB8@CFWEX01.americas.cray.com>
References: <0D7B179941D43E46800EC415D269A65D0F1CB8@CFWEX01.americas.cray.com>
Message-ID: <CAM9tzSnE9ZLhBH2q3Fu3G+D+3Qi88jPBCPsKWnkZ-kEU5v+x5Q@mail.gmail.com>

On Wed, May 30, 2012 at 10:52 AM, Keita Teranishi <keita at cray.com> wrote:

>  Hi,****
>
> ** **
>
> Is there any requirement for BLAS/LAPACK to install PETSc with 64-bit
> integer option?
>

No, PETSc distinguishes between PetscInt and BLAS ints (PetscBLASInt in the
source).


> ****
>
> ** **
>
> Thanks,****
>
> ====================================****
>
> Keita Teranishi****
>
> Scientific Library Group****
>
> Cray Inc.****
>
> keita at cray.com****
>
> ======================================****
>
> ** **
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/58ac6560/attachment-0001.html>

From li at loyno.edu  Wed May 30 11:00:43 2012
From: li at loyno.edu (Xuefeng Li)
Date: Wed, 30 May 2012 11:00:43 -0500 (CDT)
Subject: [petsc-users] Windows 7 32bit installation trouble
In-Reply-To: <490781821.102009281.1336021836522.JavaMail.root@zm09.stanford.edu>
References: <490781821.102009281.1336021836522.JavaMail.root@zm09.stanford.edu>
Message-ID: <Pine.A41.4.61.1205301031070.3743810@loyno.edu>

Dear Petsc team.

I use mpich2/cygwin under my windows 7 (32bit)
computer. Petsc installation process was sucessful,
using the following command in $PETSC_DIR.
   ./configure --download-f-blas-lapack=1
PETSC_DIR and PETSC_ARC were set/exported.
   make all
went without much trouble, with some minor complaints
such as
   win32draw.c: In function .PetscDrawLine_Win32.:
   win32draw.c:279:3: warning: value computed is not used
However, when testing with
   make test
it issues the following error message.
   Running test examples to verify correct installation
   Using PETSC_DIR=/home/Friend/software/petsc-3.2-p7 and PETSC_ARCH=arch-mswin-c-opt
   Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process
   See http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html

What is wrong in the process?

Regards,

--Xuefeng Li, (504)865-3340(phone)
   Like floating clouds, the heart rests easy
   Like flowing water, the spirit stays free
        http://www.loyno.edu/~li/home
   New Orleans, Louisiana (504)865-2051(fax)


From keita at cray.com  Wed May 30 11:01:55 2012
From: keita at cray.com (Keita Teranishi)
Date: Wed, 30 May 2012 16:01:55 +0000
Subject: [petsc-users] PETSc with 64 bit integer
In-Reply-To: <CAM9tzSnE9ZLhBH2q3Fu3G+D+3Qi88jPBCPsKWnkZ-kEU5v+x5Q@mail.gmail.com>
References: <0D7B179941D43E46800EC415D269A65D0F1CB8@CFWEX01.americas.cray.com>
	<CAM9tzSnE9ZLhBH2q3Fu3G+D+3Qi88jPBCPsKWnkZ-kEU5v+x5Q@mail.gmail.com>
Message-ID: <0D7B179941D43E46800EC415D269A65D0F1CF6@CFWEX01.americas.cray.com>

Jed,

Thanks for the reply! How about the other third-party software? Does it convert PETScInt to ?int??

Regards,
Keita

From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown
Sent: Wednesday, May 30, 2012 10:58 AM
To: PETSc users list
Subject: Re: [petsc-users] PETSc with 64 bit integer

On Wed, May 30, 2012 at 10:52 AM, Keita Teranishi <keita at cray.com<mailto:keita at cray.com>> wrote:
Hi,

Is there any requirement for BLAS/LAPACK to install PETSc with 64-bit integer option?

No, PETSc distinguishes between PetscInt and BLAS ints (PetscBLASInt in the source).


Thanks,
====================================
Keita Teranishi
Scientific Library Group
Cray Inc.
keita at cray.com<mailto:keita at cray.com>
======================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/162cac1e/attachment.html>

From jedbrown at mcs.anl.gov  Wed May 30 11:06:15 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 May 2012 11:06:15 -0500
Subject: [petsc-users] PETSc with 64 bit integer
In-Reply-To: <0D7B179941D43E46800EC415D269A65D0F1CF6@CFWEX01.americas.cray.com>
References: <0D7B179941D43E46800EC415D269A65D0F1CB8@CFWEX01.americas.cray.com>
	<CAM9tzSnE9ZLhBH2q3Fu3G+D+3Qi88jPBCPsKWnkZ-kEU5v+x5Q@mail.gmail.com>
	<0D7B179941D43E46800EC415D269A65D0F1CF6@CFWEX01.americas.cray.com>
Message-ID: <CAM9tzSmcTn6K4SYaOkstjsMm2EWQZprVG6tg1CCSUoZOcMhiZg@mail.gmail.com>

On Wed, May 30, 2012 at 11:01 AM, Keita Teranishi <keita at cray.com> wrote:

> Thanks for the reply! How about the other third-party software? Does it
> convert PETScInt to ?int??


Third party software doesn't know about PetscInt.

Those packages that support 64-bit integers must have their own solution
when they call BLAS. We don't, as far as I know, interface to any packages
that claim 64-bit support through hacks like "-i8" compiler flags.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/63b0fedb/attachment.html>

From bsmith at mcs.anl.gov  Wed May 30 11:32:22 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 30 May 2012 11:32:22 -0500
Subject: [petsc-users] PETSc with 64 bit integer
In-Reply-To: <0D7B179941D43E46800EC415D269A65D0F1CF6@CFWEX01.americas.cray.com>
References: <0D7B179941D43E46800EC415D269A65D0F1CB8@CFWEX01.americas.cray.com>
	<CAM9tzSnE9ZLhBH2q3Fu3G+D+3Qi88jPBCPsKWnkZ-kEU5v+x5Q@mail.gmail.com>
	<0D7B179941D43E46800EC415D269A65D0F1CF6@CFWEX01.americas.cray.com>
Message-ID: <DF6B20B7-A23E-47FD-A0CC-AB33F9355A4A@mcs.anl.gov>


On May 30, 2012, at 11:01 AM, Keita Teranishi wrote:

> Jed,
>  
> Thanks for the reply! How about the other third-party software? Does it convert PETScInt to ?int??

   For packages that support 64 bit integers (hypre, superlu_dist, ...) the --download-package option causes the 64 bit integer version to be built automatically.  The flag     self.requires32bitint = 0 in superlu_dist.py (for example) indicates if the package supports 64 bit int. If you build the external package yourself you need to make sure you build the 64 bit integer version.   PETSc assumes that external packages that use BLAS/LAPACK know what they are doing with regard to calling BLAS/LAPACK when using 64 bit integers.

   PETSc built with 64 bit integers does not use external packages that use 32 bit int (except BLAS/LAPACK), that is it does not convert the 64 bit int to 32 bit (except for BLAS/LAPACK). 

   Barry

>  
> Regards,
> Keita
>  
> From: petsc-users-bounces at mcs.anl.gov [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Jed Brown
> Sent: Wednesday, May 30, 2012 10:58 AM
> To: PETSc users list
> Subject: Re: [petsc-users] PETSc with 64 bit integer
>  
> On Wed, May 30, 2012 at 10:52 AM, Keita Teranishi <keita at cray.com> wrote:
> Hi,
>  
> Is there any requirement for BLAS/LAPACK to install PETSc with 64-bit integer option?
>  
> No, PETSc distinguishes between PetscInt and BLAS ints (PetscBLASInt in the source).
>  
>  
> Thanks,
> ====================================
> Keita Teranishi
> Scientific Library Group
> Cray Inc.
> keita at cray.com
> ======================================
>  
>  


From jedbrown at mcs.anl.gov  Wed May 30 11:34:31 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 May 2012 11:34:31 -0500
Subject: [petsc-users] PETSc with 64 bit integer
In-Reply-To: <DF6B20B7-A23E-47FD-A0CC-AB33F9355A4A@mcs.anl.gov>
References: <0D7B179941D43E46800EC415D269A65D0F1CB8@CFWEX01.americas.cray.com>
	<CAM9tzSnE9ZLhBH2q3Fu3G+D+3Qi88jPBCPsKWnkZ-kEU5v+x5Q@mail.gmail.com>
	<0D7B179941D43E46800EC415D269A65D0F1CF6@CFWEX01.americas.cray.com>
	<DF6B20B7-A23E-47FD-A0CC-AB33F9355A4A@mcs.anl.gov>
Message-ID: <CAM9tzSmB4W4dM9nVf5t2Obw+bGichLA8Xyy_7Qn+wHz+2d8obw@mail.gmail.com>

On Wed, May 30, 2012 at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>   For packages that support 64 bit integers (hypre, superlu_dist, ...) the
> --download-package option causes the 64 bit integer version to be built
> automatically.  The flag     self.requires32bitint = 0 in superlu_dist.py
> (for example) indicates if the package supports 64 bit int. If you build
> the external package yourself you need to make sure you build the 64 bit
> integer version.   PETSc assumes that external packages that use
> BLAS/LAPACK know what they are doing with regard to calling BLAS/LAPACK
> when using 64 bit integers.
>
>   PETSc built with 64 bit integers does not use external packages that use
> 32 bit int (except BLAS/LAPACK), that is it does not convert the 64 bit int
> to 32 bit (except for BLAS/LAPACK).
>

And MPI (obviously) and HDF5.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/4672f8c3/attachment.html>

From bsmith at mcs.anl.gov  Wed May 30 11:36:58 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 30 May 2012 11:36:58 -0500
Subject: [petsc-users] PETSc with 64 bit integer
In-Reply-To: <CAM9tzSmB4W4dM9nVf5t2Obw+bGichLA8Xyy_7Qn+wHz+2d8obw@mail.gmail.com>
References: <0D7B179941D43E46800EC415D269A65D0F1CB8@CFWEX01.americas.cray.com>
	<CAM9tzSnE9ZLhBH2q3Fu3G+D+3Qi88jPBCPsKWnkZ-kEU5v+x5Q@mail.gmail.com>
	<0D7B179941D43E46800EC415D269A65D0F1CF6@CFWEX01.americas.cray.com>
	<DF6B20B7-A23E-47FD-A0CC-AB33F9355A4A@mcs.anl.gov>
	<CAM9tzSmB4W4dM9nVf5t2Obw+bGichLA8Xyy_7Qn+wHz+2d8obw@mail.gmail.com>
Message-ID: <595A5655-8DE9-4F44-AD52-357825A2521C@mcs.anl.gov>


On May 30, 2012, at 11:34 AM, Jed Brown wrote:

> On Wed, May 30, 2012 at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>   For packages that support 64 bit integers (hypre, superlu_dist, ...) the --download-package option causes the 64 bit integer version to be built automatically.  The flag     self.requires32bitint = 0 in superlu_dist.py (for example) indicates if the package supports 64 bit int. If you build the external package yourself you need to make sure you build the 64 bit integer version.   PETSc assumes that external packages that use BLAS/LAPACK know what they are doing with regard to calling BLAS/LAPACK when using 64 bit integers.
> 
>   PETSc built with 64 bit integers does not use external packages that use 32 bit int (except BLAS/LAPACK), that is it does not convert the 64 bit int to 32 bit (except for BLAS/LAPACK).
> 
> And MPI (obviously) and HDF5.

   Right.  Kicks self for having no memory or logic circuits still working.

   


From balay at mcs.anl.gov  Wed May 30 10:38:21 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 30 May 2012 10:38:21 -0500 (CDT)
Subject: [petsc-users] Windows 7 32bit installation trouble
In-Reply-To: <Pine.A41.4.61.1205301031070.3743810@loyno.edu>
References: <490781821.102009281.1336021836522.JavaMail.root@zm09.stanford.edu>
	<Pine.A41.4.61.1205301031070.3743810@loyno.edu>
Message-ID: <alpine.LFD.2.02.1205301037060.2160@asterix>

On Wed, 30 May 2012, Xuefeng Li wrote:

> Dear Petsc team.
> 
> I use mpich2/cygwin under my windows 7 (32bit)
> computer. Petsc installation process was sucessful,
> using the following command in $PETSC_DIR.
>   ./configure --download-f-blas-lapack=1
> PETSC_DIR and PETSC_ARC were set/exported.
>   make all
> went without much trouble, with some minor complaints
> such as
>   win32draw.c: In function .PetscDrawLine_Win32.:
>   win32draw.c:279:3: warning: value computed is not used
> However, when testing with
>   make test
> it issues the following error message.
>   Running test examples to verify correct installation
>   Using PETSC_DIR=/home/Friend/software/petsc-3.2-p7 and
> PETSC_ARCH=arch-mswin-c-opt
>   Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI
> process
>   See http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html
> 
> What is wrong in the process?

This message is truncated - so I don't know what errros you've encounterd.

Perhaps you can comple/run examples manually and see if they work.

If you still have issues - send the relavent logs to petsc-maint.

Satish

From balay at mcs.anl.gov  Wed May 30 10:41:51 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 30 May 2012 10:41:51 -0500 (CDT)
Subject: [petsc-users] PETSc installed without X windows on this machine
In-Reply-To: <5526cfd4.12183.1379de7983b.Coremail.w_ang_temp@163.com>
References: <5526cfd4.12183.1379de7983b.Coremail.w_ang_temp@163.com>
Message-ID: <alpine.LFD.2.02.1205301041200.2160@asterix>

On Wed, 30 May 2012, w_ang_temp wrote:

> Hello
>     When I try to view a matrix with the option -mat_view_draw, I get :
>     [0]PETSC ERROR: PETSc installed without X windows on this machine proceeding without graphics.
>     But calling MatView in side the code is ok.
>     I install the libX11-dev as Matt said in one message in the petsc-users in 2008,  but the problem doesn't solved.

Perhaps you didn't re-run configure/make after installing libX11-dev

Satish

>     The system is ubuntu 10.04.
>     So what might be the reason?
>     Thanks.
>                                           Jim
> 
>  


From balay at mcs.anl.gov  Wed May 30 10:52:00 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 30 May 2012 10:52:00 -0500 (CDT)
Subject: [petsc-users] search in archives fails
In-Reply-To: <A9AE64C525438E45B1D60F4D09AEAA9E0623D3@MAR160N1.marin.local>
References: <A9AE64C525438E45B1D60F4D09AEAA9E0623D3@MAR160N1.marin.local>
Message-ID: <alpine.LFD.2.02.1205301051350.2160@asterix>

Thanks for the note. I've forwarded this issue to our sys-admins.

Satish


On Wed, 30 May 2012, Klaij, Christiaan wrote:

> I'm trying to find something in the petsc-users mailings list archives but the search function gives the error:
> 
> Bug in Mailman version 2.1.14
> 
> We're sorry, we hit a bug!
> 
> Please inform the webmaster for this site of this problem. Printing of traceback and other system information has been explicitly inhibited, but the webmaster can find this information in the Mailman error logs.
> 
> 
> dr. ir. Christiaan Klaij
> CFD Researcher
> Research & Development
> E mailto:C.Klaij at marin.nl
> T +31 317 49 33 44
> 
> MARIN
> 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands
> T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl
> 
> 


From balay at mcs.anl.gov  Wed May 30 10:55:55 2012
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 30 May 2012 10:55:55 -0500 (CDT)
Subject: [petsc-users] randomness from MATSOLVERPACKAGE
In-Reply-To: <BLU0-SMTP134476792BC7FBF2E1B0340CB0A0@phx.gbl>
References: <BLU0-SMTP134476792BC7FBF2E1B0340CB0A0@phx.gbl>
Message-ID: <alpine.LFD.2.02.1205301054170.2160@asterix>

On Wed, 30 May 2012, Hui Zhang wrote:

> I suspect there are some randomness from external matsolverpackage. I use a parallel matrix
> as preconditioner, and the preconditioner is exactly LU factored, then I saw the KSP (gmres)
> residuals are different from different runs.
> 
> Is it true of randomness from matsolverpackage?  Thanks a lot!

Which package?

I've seen mumps showing some differences between runs - but not others..

Satish

From neckel at in.tum.de  Wed May 30 13:29:13 2012
From: neckel at in.tum.de (Tobias Neckel)
Date: Wed, 30 May 2012 20:29:13 +0200
Subject: [petsc-users] problems using ilu dt of superlu
Message-ID: <4FC666F9.4060402@in.tum.de>

Hello,

we recently upgraded from 3.0.0-p11 to 3.2-p7 and are facing a problem 
in one of our test cases: We use seqaij matrix format and GMRES with ILU dt.

Knowing that the ILUdt has been replaced by superlu's version, we 
therefore configured our petsc installation with
---
-PETSC_ARCH=petsc-3.2-serial --prefix=/work_fast/liebm/petsc-3.2-serial/ 
--with-debugging=0 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc 
--download-superlu=1 --with-mpi=0
----
resulting in a nicely compiled lib/libsuperlu_4.2.a.

Also, ex52.c works (with ILU as a solver).

Hence, in our code, we replaced the former single-line command
         _ierr = PCFactorSetDropTolerance(...stuff...);
by the code of ex52.c:
         Mat F;
         _ierr = PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU);
         _ierr = PCFactorSetUpMatSolverPackage(pc);/* call 
MatGetFactor() to create F */
         _ierr = PCFactorGetMatrix(pc,&F);
         _ierr = MatSuperluSetILUDropTol(F,1.0e-3);

After some smooting, the compile and link works now fine, but running 
the test case results in a segfault; here is the corresponding valgrind 
output:

==17405== Invalid read of size 4
==17405==    at 0x1B15245: MatGetFactor (in 
/work_fast/liebm/workspace/p1/src/build/debug/dim2/fluid/peano-fluid)
==17405==    by 0x1CDF709: PCFactorSetUpMatSolverPackage_Factor (in 
/work_fast/liebm/workspace/p1/src/build/debug/dim2/fluid/peano-fluid)
==17405==    by 0x1B3F7F9: PCFactorSetUpMatSolverPackage (in 
/work_fast/liebm/workspace/p1/src/build/debug/dim2/fluid/peano-fluid)
==17405==    by 0x19284F5: 
peano::petsc::PETScNonLinearSolver::setupLinearSolver(_p_KSP*&, 
std::string, std::string, double, double, int) 
(PETScNonLinearSolver.cpp:365)
==17405==    by 0x1926DF2: 
peano::petsc::PETScNonLinearSolver::initialize(int, int, int) 
(PETScNonLinearSolver.cpp:242)
==17405==    by 0x901225: 
peano::fluid::NonLinearTrivialgridFluidSimulation::implementSetInitialValue(bool, 
double, int, double) (NonLinearTrivialgridFluidSimulation.cpp:540)
==17405==    by 0x17AE3C8: 
peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid3x3ComputeNavierStokesFunction() 
(SteadyStateIntegrationTestChannel.cpp:114)
==17405==    by 0x17B28A8: 
peano::fluid::SteadyStateIntegrationTestChannel::run() 
(SteadyStateIntegrationTestChannel.cpp:484)
==17405==    by 0x19BDCB3: tarch::tests::TestCaseCollection::run() 
(TestCaseCollection.cpp:42)
==17405==    by 0x19BDCB3: tarch::tests::TestCaseCollection::run() 
(TestCaseCollection.cpp:42)
==17405==    by 0x408A9D: 
peano::IntegrationTestCaseRunner::run(tarch::configuration::IntegrationTestConfiguration 
const&) (IntegrationTestCaseRunner.cpp:127)
==17405==    by 0x412C2E: main (main.cpp:205)
==17405==  Address 0x1e8 is not stack'd, malloc'd or (recently) free'd
==17405==
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, 
and run
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message 
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 
09:30:51 CDT 2012
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 
named atsccs58 by liebm Wed May 30 20:06:30 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial/lib
[0]PETSC ERROR: Configure run at Tue May 29 21:05:34 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial 
--prefix=/work_fast/liebm/petsc-3.2-serial/ --with-debugging=0 
--with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 
--with-mpi=0
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file


Did we miss something (keeping in mind that we use ILU now as a 
preconditioner in combination with GMRES)? Any suggestions are warmly 
welcome ;-).

Thanks in advance and best regards
Tobias

From knepley at gmail.com  Wed May 30 13:35:16 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 May 2012 14:35:16 -0400
Subject: [petsc-users] problems using ilu dt of superlu
In-Reply-To: <4FC666F9.4060402@in.tum.de>
References: <4FC666F9.4060402@in.tum.de>
Message-ID: <CAMYG4GkFjoSK73SR97bqHsDD+eQj4wXW72f3rP-kSA12TjwKkg@mail.gmail.com>

On Wed, May 30, 2012 at 2:29 PM, Tobias Neckel <neckel at in.tum.de> wrote:

> Hello,
>
> we recently upgraded from 3.0.0-p11 to 3.2-p7 and are facing a problem in
> one of our test cases: We use seqaij matrix format and GMRES with ILU dt.
>
> Knowing that the ILUdt has been replaced by superlu's version, we
> therefore configured our petsc installation with
> ---
> -PETSC_ARCH=petsc-3.2-serial --prefix=/work_fast/liebm/**petsc-3.2-serial/
> --with-debugging=0 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc
> --download-superlu=1 --with-mpi=0
> ----
>

I know it involves another build, but turning on debugging is really
essential for tracking down errors like this. Can you
reconfigure with debugging under a different PETSC_ARCH and send the trace?

  Thanks

     Matt


> resulting in a nicely compiled lib/libsuperlu_4.2.a.
>
> Also, ex52.c works (with ILU as a solver).
>
> Hence, in our code, we replaced the former single-line command
>        _ierr = PCFactorSetDropTolerance(...**stuff...);
> by the code of ex52.c:
>        Mat F;
>        _ierr = PCFactorSetMatSolverPackage(**pc,MATSOLVERSUPERLU);
>        _ierr = PCFactorSetUpMatSolverPackage(**pc);/* call MatGetFactor()
> to create F */
>        _ierr = PCFactorGetMatrix(pc,&F);
>        _ierr = MatSuperluSetILUDropTol(F,1.**0e-3);
>
> After some smooting, the compile and link works now fine, but running the
> test case results in a segfault; here is the corresponding valgrind output:
>
> ==17405== Invalid read of size 4
> ==17405==    at 0x1B15245: MatGetFactor (in /work_fast/liebm/workspace/p1/
> **src/build/debug/dim2/fluid/**peano-fluid)
> ==17405==    by 0x1CDF709: PCFactorSetUpMatSolverPackage_**Factor (in
> /work_fast/liebm/workspace/p1/**src/build/debug/dim2/fluid/**peano-fluid)
> ==17405==    by 0x1B3F7F9: PCFactorSetUpMatSolverPackage (in
> /work_fast/liebm/workspace/p1/**src/build/debug/dim2/fluid/**peano-fluid)
> ==17405==    by 0x19284F5: peano::petsc::**PETScNonLinearSolver::**setupLinearSolver(_p_KSP*&,
> std::string, std::string, double, double, int)
> (PETScNonLinearSolver.cpp:365)
> ==17405==    by 0x1926DF2: peano::petsc::**PETScNonLinearSolver::**initialize(int,
> int, int) (PETScNonLinearSolver.cpp:242)
> ==17405==    by 0x901225: peano::fluid::**NonLinearTrivialgridFluidSimul**
> ation::**implementSetInitialValue(bool, double, int, double) (**
> NonLinearTrivialgridFluidSimul**ation.cpp:540)
> ==17405==    by 0x17AE3C8: peano::fluid::**SteadyStateIntegrationTestChan*
> *nel::**testTrivialgrid3x3ComputeNavie**rStokesFunction() (**
> SteadyStateIntegrationTestChan**nel.cpp:114)
> ==17405==    by 0x17B28A8: peano::fluid::**SteadyStateIntegrationTestChan*
> *nel::run() (**SteadyStateIntegrationTestChan**nel.cpp:484)
> ==17405==    by 0x19BDCB3: tarch::tests::**TestCaseCollection::run()
> (TestCaseCollection.cpp:42)
> ==17405==    by 0x19BDCB3: tarch::tests::**TestCaseCollection::run()
> (TestCaseCollection.cpp:42)
> ==17405==    by 0x408A9D: peano::**IntegrationTestCaseRunner::**
> run(tarch::configuration::**IntegrationTestConfiguration const&)
> (IntegrationTestCaseRunner.**cpp:127)
> ==17405==    by 0x412C2E: main (main.cpp:205)
> ==17405==  Address 0x1e8 is not stack'd, malloc'd or (recently) free'd
> ==17405==
> [0]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/**
> petsc-as/documentation/faq.**html#valgrind[0]PETSC<http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC>ERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------**------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51
> CDT 2012
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [0]PETSC ERROR: ./build/debug/dim2/fluid/**peano-fluid on a petsc-3.2
> named atsccs58 by liebm Wed May 30 20:06:30 2012
> [0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-**
> serial/lib
> [0]PETSC ERROR: Configure run at Tue May 29 21:05:34 2012
> [0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial
> --prefix=/work_fast/liebm/**petsc-3.2-serial/ --with-debugging=0
> --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1
> --with-mpi=0
> [0]PETSC ERROR: ------------------------------**
> ------------------------------**------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
>
>
> Did we miss something (keeping in mind that we use ILU now as a
> preconditioner in combination with GMRES)? Any suggestions are warmly
> welcome ;-).
>
> Thanks in advance and best regards
> Tobias
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/f565147e/attachment.html>

From coco at dmi.unict.it  Wed May 30 13:55:35 2012
From: coco at dmi.unict.it (coco at dmi.unict.it)
Date: Wed, 30 May 2012 20:55:35 +0200
Subject: [petsc-users] Multigrid as a preconditioner
Message-ID: <20120530205535.Horde.lcK6Xuph4B9Pxm0nrofi2xA@mbox.dmi.unict.it>

Dear all,

I have chosen the multigrid as a preconditioner of a Richardson kps  
solver and I have set all the multigrid stuff (smoother,  
interpolation, restriction, coarse solver). When I perform two  
iterations of the Richardson ksp solver (with the option -ksp_max_it  
2), which means two cycles of the multigrid, I obtain the same result  
as in the case I perform only one iteraton.
I noticed that when the second iteration starts, the initial guess is  
zero, and then it provides the same solution as in the first  
iteration. Anyway, the norm residual printed by the -ksp_monitor  
option decreases.
How can I investigate the problem?
One more detail: the smoother is implemented by a user-provided  
routine UserSOR() which is registered as MATOP_SOR of a shell matrix,  
which is assigned to the ksp smoother. Even if the UserSOR() takes in  
input a MatSORType flag, I do not take care about it inside the routine.

Thank you.
Best regards,
Armando





From john.fettig at gmail.com  Wed May 30 13:58:00 2012
From: john.fettig at gmail.com (John Fettig)
Date: Wed, 30 May 2012 14:58:00 -0400
Subject: [petsc-users] SAME_PRECONDITIONER vs. DIFFERENT_NONZERO_PATTERN
Message-ID: <CAD8Bu=oGZ+Jxw_edVxmEYEjB+Lep09sRfS01NHPFStJwz6+mFg@mail.gmail.com>

I have a question about the difference between SAME_PRECONDITIONER and
DIFFERENT_NONZERO_PATTERN.  If you solve multiple RHS's (with
different initial guesses) but the matrix does not change, should you
expect to get identical results with SAME_PRECONDITIONER and
DIFFERENT_NONZERO_PATTERN?  I think it should be, but I'm finding that
not to be the case with ML.

John

From jedbrown at mcs.anl.gov  Wed May 30 14:04:46 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 May 2012 14:04:46 -0500
Subject: [petsc-users] SAME_PRECONDITIONER vs. DIFFERENT_NONZERO_PATTERN
In-Reply-To: <CAD8Bu=oGZ+Jxw_edVxmEYEjB+Lep09sRfS01NHPFStJwz6+mFg@mail.gmail.com>
References: <CAD8Bu=oGZ+Jxw_edVxmEYEjB+Lep09sRfS01NHPFStJwz6+mFg@mail.gmail.com>
Message-ID: <CAM9tzS=L+M9nvBMWL9v76h4RZKW+yW+EeeA_74RR=OMb1hj6ag@mail.gmail.com>

On Wed, May 30, 2012 at 1:58 PM, John Fettig <john.fettig at gmail.com> wrote:

> I have a question about the difference between SAME_PRECONDITIONER and
> DIFFERENT_NONZERO_PATTERN.  If you solve multiple RHS's (with
> different initial guesses) but the matrix does not change, should you
> expect to get identical results with SAME_PRECONDITIONER and
> DIFFERENT_NONZERO_PATTERN?


Why are you even calling KSPSetOperators? Just call KSPSolve() for each
right hand side.


> I think it should be, but I'm finding that
> not to be the case with ML.
>

Are you sure the matrix hasn't changed? What kind of differences are you
observing?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/23d9390c/attachment.html>

From jedbrown at mcs.anl.gov  Wed May 30 14:13:34 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 May 2012 14:13:34 -0500
Subject: [petsc-users] Multigrid as a preconditioner
In-Reply-To: <20120530205535.Horde.lcK6Xuph4B9Pxm0nrofi2xA@mbox.dmi.unict.it>
References: <20120530205535.Horde.lcK6Xuph4B9Pxm0nrofi2xA@mbox.dmi.unict.it>
Message-ID: <CAM9tzSmvaSdbZQt0yY7JeCRcaUmXbH9m996M4uv86nVjRG8Csg@mail.gmail.com>

On Wed, May 30, 2012 at 1:55 PM, <coco at dmi.unict.it> wrote:

> Dear all,
>
> I have chosen the multigrid as a preconditioner of a Richardson kps solver
> and I have set all the multigrid stuff (smoother, interpolation,
> restriction, coarse solver). When I perform two iterations of the
> Richardson ksp solver (with the option -ksp_max_it 2), which means two
> cycles of the multigrid, I obtain the same result as in the case I perform
> only one iteraton.
>

What do you mean by this?


> I noticed that when the second iteration starts, the initial guess is
> zero, and then it provides the same solution as in the first iteration.
> Anyway, the norm residual printed by the -ksp_monitor option decreases.
>

Run with -ksp_monitor_true_residual


> How can I investigate the problem?
>

I'd start by checking that UserSOR is really SOR (as opposed to somehow
misusing the vectors). Check that it respects a zero initial guess if that
is part of the flag.


> One more detail: the smoother is implemented by a user-provided routine
> UserSOR() which is registered as MATOP_SOR of a shell matrix, which is
> assigned to the ksp smoother. Even if the UserSOR() takes in input a
> MatSORType flag, I do not take care about it inside the routine.
>

The best thing is to check that the option is whatever you expect it to be
and SETERRQ() if it is not. That will reduce confusion down the road.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/7c8de7e8/attachment-0001.html>

From john.fettig at gmail.com  Wed May 30 14:16:03 2012
From: john.fettig at gmail.com (John Fettig)
Date: Wed, 30 May 2012 15:16:03 -0400
Subject: [petsc-users] SAME_PRECONDITIONER vs. DIFFERENT_NONZERO_PATTERN
In-Reply-To: <CAM9tzS=L+M9nvBMWL9v76h4RZKW+yW+EeeA_74RR=OMb1hj6ag@mail.gmail.com>
References: <CAD8Bu=oGZ+Jxw_edVxmEYEjB+Lep09sRfS01NHPFStJwz6+mFg@mail.gmail.com>
	<CAM9tzS=L+M9nvBMWL9v76h4RZKW+yW+EeeA_74RR=OMb1hj6ag@mail.gmail.com>
Message-ID: <CAD8Bu=rODRBEyk3J8_UCLgPLkY1b6-7rwPt5H5OG7hYYp2SZ0Q@mail.gmail.com>

On Wed, May 30, 2012 at 3:04 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> On Wed, May 30, 2012 at 1:58 PM, John Fettig <john.fettig at gmail.com> wrote:
>>
>> I have a question about the difference between SAME_PRECONDITIONER and
>> DIFFERENT_NONZERO_PATTERN. ?If you solve multiple RHS's (with
>> different initial guesses) but the matrix does not change, should you
>> expect to get identical results with SAME_PRECONDITIONER and
>> DIFFERENT_NONZERO_PATTERN?
>
>
> Why are you even calling KSPSetOperators? Just call KSPSolve() for each
> right hand side.

What I mean is if I run the code twice, once with SAME_PRECONDITIONER
and once with DIFFERENT_NONZERO_PATTERN, the solution for the second
solve in the code is different.  I am sure the matrix does not change.
 There are a couple of calls to MatZeroRows but I have verified that
the diagonal is set to the exact same value both time.  The difference
in the solution is not insignificant (it is probably below the solver
tolerance but still in the 3rd significant digit).

John

From knepley at gmail.com  Wed May 30 14:20:02 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 May 2012 15:20:02 -0400
Subject: [petsc-users] SAME_PRECONDITIONER vs. DIFFERENT_NONZERO_PATTERN
In-Reply-To: <CAD8Bu=rODRBEyk3J8_UCLgPLkY1b6-7rwPt5H5OG7hYYp2SZ0Q@mail.gmail.com>
References: <CAD8Bu=oGZ+Jxw_edVxmEYEjB+Lep09sRfS01NHPFStJwz6+mFg@mail.gmail.com>
	<CAM9tzS=L+M9nvBMWL9v76h4RZKW+yW+EeeA_74RR=OMb1hj6ag@mail.gmail.com>
	<CAD8Bu=rODRBEyk3J8_UCLgPLkY1b6-7rwPt5H5OG7hYYp2SZ0Q@mail.gmail.com>
Message-ID: <CAMYG4G=1oTK8kpBjjutNk56Uh8F5LJWJi7e+uDaGAGvcW_WCAA@mail.gmail.com>

On Wed, May 30, 2012 at 3:16 PM, John Fettig <john.fettig at gmail.com> wrote:

> On Wed, May 30, 2012 at 3:04 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> > On Wed, May 30, 2012 at 1:58 PM, John Fettig <john.fettig at gmail.com>
> wrote:
> >>
> >> I have a question about the difference between SAME_PRECONDITIONER and
> >> DIFFERENT_NONZERO_PATTERN.  If you solve multiple RHS's (with
> >> different initial guesses) but the matrix does not change, should you
> >> expect to get identical results with SAME_PRECONDITIONER and
> >> DIFFERENT_NONZERO_PATTERN?
> >
> >
> > Why are you even calling KSPSetOperators? Just call KSPSolve() for each
> > right hand side.
>
> What I mean is if I run the code twice, once with SAME_PRECONDITIONER
> and once with DIFFERENT_NONZERO_PATTERN, the solution for the second
> solve in the code is different.  I am sure the matrix does not change.
>  There are a couple of calls to MatZeroRows but I have verified that
> the diagonal is set to the exact same value both time.  The difference
> in the solution is not insignificant (it is probably below the solver
> tolerance but still in the 3rd significant digit).


Is the initial residual the same. It would be helpful to see the
-ksp_monitor_true_residual
output

   Matt



>
> John
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/9d25d491/attachment.html>

From jedbrown at mcs.anl.gov  Wed May 30 14:21:27 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Wed, 30 May 2012 14:21:27 -0500
Subject: [petsc-users] SAME_PRECONDITIONER vs. DIFFERENT_NONZERO_PATTERN
In-Reply-To: <CAD8Bu=rODRBEyk3J8_UCLgPLkY1b6-7rwPt5H5OG7hYYp2SZ0Q@mail.gmail.com>
References: <CAD8Bu=oGZ+Jxw_edVxmEYEjB+Lep09sRfS01NHPFStJwz6+mFg@mail.gmail.com>
	<CAM9tzS=L+M9nvBMWL9v76h4RZKW+yW+EeeA_74RR=OMb1hj6ag@mail.gmail.com>
	<CAD8Bu=rODRBEyk3J8_UCLgPLkY1b6-7rwPt5H5OG7hYYp2SZ0Q@mail.gmail.com>
Message-ID: <CAM9tzSn-3dVqa8qua7t4prpFGBF9TQYVoTs0RYi-t0AB=Ac9kg@mail.gmail.com>

On Wed, May 30, 2012 at 2:16 PM, John Fettig <john.fettig at gmail.com> wrote:

> What I mean is if I run the code twice, once with SAME_PRECONDITIONER
> and once with DIFFERENT_NONZERO_PATTERN, the solution for the second
> solve in the code is different.  I am sure the matrix does not change.
>  There are a couple of calls to MatZeroRows but I have verified that
> the diagonal is set to the exact same value both time.  The difference
> in the solution is not insignificant (it is probably below the solver
> tolerance but still in the 3rd significant digit).
>

Why are you calling KSPSetOperators at all if the matrix doesn't change at
all?

Run both cases with -ksp_view_binary (renaming 'binaryoutput' between the
runs), then use src/ksp/ksp/examples/tutorials/ex10 to solve them
separately. You can also check whether the matrices are indeed identical. I
suspect they are different in some minor way.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/ec07fa5b/attachment.html>

From mike.hui.zhang at hotmail.com  Wed May 30 14:48:09 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Wed, 30 May 2012 21:48:09 +0200
Subject: [petsc-users] randomness from MATSOLVERPACKAGE
In-Reply-To: <alpine.LFD.2.02.1205301054170.2160@asterix>
References: <BLU0-SMTP134476792BC7FBF2E1B0340CB0A0@phx.gbl>
	<alpine.LFD.2.02.1205301054170.2160@asterix>
Message-ID: <BLU0-SMTP308EBA371E8192F12FA425FCB0A0@phx.gbl>


On May 30, 2012, at 5:55 PM, Satish Balay wrote:

> On Wed, 30 May 2012, Hui Zhang wrote:
> 
>> I suspect there are some randomness from external matsolverpackage. I use a parallel matrix
>> as preconditioner, and the preconditioner is exactly LU factored, then I saw the KSP (gmres)
>> residuals are different from different runs.
>> 
>> Is it true of randomness from matsolverpackage?  Thanks a lot!
> 
> Which package?

Either mumps or pastix.

> 
> I've seen mumps showing some differences between runs - but not others..
> 
> Satish
> 


From mike.hui.zhang at hotmail.com  Wed May 30 15:35:03 2012
From: mike.hui.zhang at hotmail.com (Hui Zhang)
Date: Wed, 30 May 2012 22:35:03 +0200
Subject: [petsc-users] matptap
Message-ID: <BLU0-SMTP376DDF5F9C93F1DDE7DFCE8CB0A0@phx.gbl>

Is MatPtAP using only transpose but no complex conjugate?

From hzhang at mcs.anl.gov  Wed May 30 17:41:02 2012
From: hzhang at mcs.anl.gov (Hong Zhang)
Date: Wed, 30 May 2012 17:41:02 -0500
Subject: [petsc-users] matptap
In-Reply-To: <BLU0-SMTP376DDF5F9C93F1DDE7DFCE8CB0A0@phx.gbl>
References: <BLU0-SMTP376DDF5F9C93F1DDE7DFCE8CB0A0@phx.gbl>
Message-ID: <CAGCphBvUXULgPRDiw+BtmrShe7UD5N5QAiKwQvX4tSS2QuMSPg@mail.gmail.com>

Hui :

> Is MatPtAP using only transpose but no complex conjugate?
>
Yes,  using only transpose but no complex conjugate.

Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/209f9e63/attachment.html>

From jbakosi at lanl.gov  Wed May 30 18:20:01 2012
From: jbakosi at lanl.gov (Jozsef Bakosi)
Date: Wed, 30 May 2012 17:20:01 -0600
Subject: [petsc-users] Is PCFactorSetZeroPivot() effective?
Message-ID: <20120530232001.GF19605@karman>

Hello,

I have a mesh with very small cell sizes and in a Poisson equation CG solve
with ML preconditioner I get the following error:

[31]PETSC ERROR: --------------------- Error Message ------------------------------------
[31]PETSC ERROR: Detected zero pivot in LU factorization see
http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#ZeroPivot!
[31]PETSC ERROR: Zero pivot row 537 value 9.41111e-13 tolerance 1e-12!

I really would like PETSc to NOT treat matrix values lower than 1e-12 to
be zero, so I'm trying to set the zero pivot to a lower value with
PCFactorSetZeroPivot(), but I still get the same error message: as if
the function call would have no effect. I also tried setting the
zeropivot via PetscOptionsSetValue("-pc_factor_zeropivot", ...), but
still no luck.

Is there another way to set the zero pivot that I'm not aware of?

Is there a way to query the zero pivot value?

Thanks,
J
-- 
Jozsef Bakosi
Computational Physics Group (CCS-2)
MS T006, Los Alamos National Laboratory
Los Alamos, NM 87545, USA
email: jbakosi at lanl.gov
work: 505-663-5607
fax : 505-663-5504

From knepley at gmail.com  Wed May 30 18:27:30 2012
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 30 May 2012 19:27:30 -0400
Subject: [petsc-users] Is PCFactorSetZeroPivot() effective?
In-Reply-To: <20120530232001.GF19605@karman>
References: <20120530232001.GF19605@karman>
Message-ID: <CAMYG4Gkw=9xG5xWGW5ijoJXtGN+_XNWpXQN9GiX3Ut+x4ngcow@mail.gmail.com>

On Wed, May 30, 2012 at 7:20 PM, Jozsef Bakosi <jbakosi at lanl.gov> wrote:

> Hello,
>
> I have a mesh with very small cell sizes and in a Poisson equation CG solve
> with ML preconditioner I get the following error:
>
> [31]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [31]PETSC ERROR: Detected zero pivot in LU factorization see
>
> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#ZeroPivot
> !
> [31]PETSC ERROR: Zero pivot row 537 value 9.41111e-13 tolerance 1e-12!
>
> I really would like PETSc to NOT treat matrix values lower than 1e-12 to
> be zero, so I'm trying to set the zero pivot to a lower value with
> PCFactorSetZeroPivot(), but I still get the same error message: as if
> the function call would have no effect. I also tried setting the
> zeropivot via PetscOptionsSetValue("-pc_factor_zeropivot", ...), but
> still no luck.
>
> Is there another way to set the zero pivot that I'm not aware of?
>

1) Always look at -ksp_view

2) I bet this is a subsolve, so that it would be -sub_pc_factor_zeropivot
<value>. You
    can verify by looking at -ksp_view


> Is there a way to query the zero pivot value?
>

3) Its printed out, 1.0e-12

   Matt


> Thanks,
> J
> --
> Jozsef Bakosi
> Computational Physics Group (CCS-2)
> MS T006, Los Alamos National Laboratory
> Los Alamos, NM 87545, USA
> email: jbakosi at lanl.gov
> work: 505-663-5607
> fax : 505-663-5504
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120530/f2e2bcc4/attachment-0001.html>

From w_ang_temp at 163.com  Wed May 30 21:39:09 2012
From: w_ang_temp at 163.com (w_ang_temp)
Date: Thu, 31 May 2012 10:39:09 +0800 (CST)
Subject: [petsc-users] PETSc installed without X windows on this machine
In-Reply-To: <alpine.LFD.2.02.1205301041200.2160@asterix>
References: <5526cfd4.12183.1379de7983b.Coremail.w_ang_temp@163.com>
	<alpine.LFD.2.02.1205301041200.2160@asterix>
Message-ID: <78812f18.485a.137a0c2ca9b.Coremail.w_ang_temp@163.com>


Hello Satish
       I think I need to reconfigure Petsc with the option --with-X=1.
       Thanks.
                                                      Jim




At 2012-05-30 23:41:51,"Satish Balay" <balay at mcs.anl.gov> wrote:
>On Wed, 30 May 2012, w_ang_temp wrote:
>
>> Hello
>>     When I try to view a matrix with the option -mat_view_draw, I get :
>>     [0]PETSC ERROR: PETSc installed without X windows on this machine proceeding without graphics.
>>     But calling MatView in side the code is ok.
>>     I install the libX11-dev as Matt said in one message in the petsc-users in 2008,  but the problem doesn't solved.
>
>Perhaps you didn't re-run configure/make after installing libX11-dev
>
>Satish
>
>>     The system is ubuntu 10.04.
>>     So what might be the reason?
>>     Thanks.
>>                                           Jim
>> 
>>  
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120531/34fdfdef/attachment.html>

From john.fettig at gmail.com  Wed May 30 22:14:24 2012
From: john.fettig at gmail.com (John Fettig)
Date: Wed, 30 May 2012 23:14:24 -0400
Subject: [petsc-users] [petsc-maint #118299] Re: SAME_PRECONDITIONER vs.
	DIFFERENT_NONZERO_PATTERN
In-Reply-To: <363B5127-770F-40B0-AF5C-0A68B20317F3@mcs.anl.gov>
References: <RT-Ticket-118299@mcs.anl.gov>
	<CAD8Bu=oGZ+Jxw_edVxmEYEjB+Lep09sRfS01NHPFStJwz6+mFg@mail.gmail.com>
	<CAM9tzS=L+M9nvBMWL9v76h4RZKW+yW+EeeA_74RR=OMb1hj6ag@mail.gmail.com>
	<CAD8Bu=rODRBEyk3J8_UCLgPLkY1b6-7rwPt5H5OG7hYYp2SZ0Q@mail.gmail.com>
	<CAM9tzSn-3dVqa8qua7t4prpFGBF9TQYVoTs0RYi-t0AB=Ac9kg@mail.gmail.com>
	<CAD8Bu=r_dHizGQ3sDk6JTynFXPVFKT13NuZ_tYchtpCdsqb+YQ@mail.gmail.com>
	<rt-3.8.7-29418-1338430580-1730.118299-4-0@mcs.anl.gov>
	<9C04F710-BCAE-4640-BF28-8A673ABA1280@mcs.anl.gov>
	<CAD8Bu=o+kqmEUbbPdg-1awAdLutz7fm-h5zf89Shsq_37jvmMA@mail.gmail.com>
	<363B5127-770F-40B0-AF5C-0A68B20317F3@mcs.anl.gov>
Message-ID: <CAD8Bu=pw4ODDvehYC0PU9ajXdriHB43oo1pg38koYmABYf3SHA@mail.gmail.com>

(switching back to petsc-users so that others may see this resolution)

On Wed, May 30, 2012 at 10:47 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> On May 30, 2012, at 9:42 PM, John Fettig wrote:
>> I don't think it does in the PETSc driver, but I'm not certain. ?The
>> only place I find reference to "rand" is in the aggregation object,
>> which has the ability to use a random ordering rather than the natural
>> one. ?This isn't used by the PETSc driver, though.
>
> ? Are you sure? You could run in the debugger and put a break point in rand() to see if it is called.

Ok, it turns out ML has its own random number generator, ML_srandom1,
which is called from ML_random_vec.  It is indeed called during the
call to ML_Gen_MultiLevelHierarchy_UsingAggregation in PCSetUp_ML:

#0  ML_random_vec (u=0x14b1ee0, N=1449, comm=0x12ebeb8)
    at ./Utils/ml_utils.c:1767
#1  0x00002aedf3b060c6 in ML_CG_ComputeEigenvalues (data=0x14b1ee0,
    length=1449, scale_by_diag=19840696) at ./Krylov/ml_cg.c:189
#2  0x00002aedf3ababfb in ML_Krylov_Solve (data=0x14b1ee0, leng=1449,
    invec=0x12ebeb8, outvec=0x2aedf25cfbc8) at ./Krylov/ml_krylov.c:358
#3  0x00002aedf3a7a1e8 in ML_AGG_Gen_Prolongator (ml=0x14b1ee0, level=1449,
    clevel=19840696, data=0x2aedf25cfbc8) at ./Coarsen/ml_agg_genP.c:490
#4  0x00002aedf3a7e316 in ML_MultiLevel_Gen_Prolongator (ml=0x14b1ee0,
    level=1449, clevel=19840696, data=0x2aedf25cfbc8)
    at ./Coarsen/ml_agg_genP.c:2913
#5  0x00002aedf3a7cb33 in ML_Gen_MultiLevelHierarchy (ml=0x14b1ee0,
    fine_level=1449, user_next_level=0x12ebeb8,
    user_gen_restriction=0x2aedf25cfbc8 <hpmp_flag_signals>,
    user_gen_prolongator=0x3, user_data=0x20000)
    at ./Coarsen/ml_agg_genP.c:2557
#6  0x00002aedf3a7befd in ML_Gen_MultiLevelHierarchy_UsingAggregation (
    ml=0x14b1ee0, start=1449, increment_or_decrement=19840696,
    ag=0x2aedf25cfbc8) at ./Coarsen/ml_agg_genP.c:2444
#7  0x00002aedf35f048f in PCSetUp_ML (pc=0x1351a00)
    at /home/jfe/local/petsc-dev/src/ksp/pc/impls/ml/ml.c:692

This must be the reason the different results.  Now that you mention
it, if I try solving the exact same system twice with
DIFFERENT_NONZERO_PATTERN, the residual history and the solution are
different.  If I use SAME_PRECONDITIONER the residual history and the
solution are the same.

Thanks,
John

From neckel at in.tum.de  Thu May 31 08:18:49 2012
From: neckel at in.tum.de (Tobias Neckel)
Date: Thu, 31 May 2012 15:18:49 +0200
Subject: [petsc-users] problems using ilu dt of superlu
In-Reply-To: <CAMYG4GkFjoSK73SR97bqHsDD+eQj4wXW72f3rP-kSA12TjwKkg@mail.gmail.com>
References: <4FC666F9.4060402@in.tum.de>
	<CAMYG4GkFjoSK73SR97bqHsDD+eQj4wXW72f3rP-kSA12TjwKkg@mail.gmail.com>
Message-ID: <4FC76FB9.60903@in.tum.de>

Dear Matt,

 > I know it involves another build, but turning on debugging is really
 > essential for tracking down errors like this. Can you
 > reconfigure with debugging under a different PETSC_ARCH and send the 
trace?

we built the debug version of petsc and now get the attached output 
(with and without valgrind) instead of a segfault for the test under 
consideration.

Interestingly, the code runs now fully (with all the respective error 
messages of petsc) and completes *successfully* the different 5 test 
cases, despite the errors in all of them. The related code lines of the 
PETScNonLinearSolver.cpp (365, 367, and 369) exactly are the newly 
inserted code lines:
        _ierr = PCFactorSetUpMatSolverPackage(pc);
        _ierr = PCFactorGetMatrix(pc,&F);
        _ierr = MatSuperluSetILUDropTol(F,1.0e-3);

What I forgot to mention in the previous email is that we use the ILU 
with GMRES in the linear solve inside a snes non-linear problem ... 
hence, we have a call to SNESCreate() and fetch the ksp from there 
(different to ex52.c).

It looks like setting up the PC is not working correctly even if we 
successfully use it before. Did we miss additional changes in the setup 
from 3.0 to 3.2? Would the full source file help for clarification in 
this case?

Thanks and best regards
Tobias


On 30.05.2012 20:35, Matthew Knepley wrote:
> On Wed, May 30, 2012 at 2:29 PM, Tobias Neckel <neckel at in.tum.de
> <mailto:neckel at in.tum.de>> wrote:
>
>     Hello,
>
>     we recently upgraded from 3.0.0-p11 to 3.2-p7 and are facing a
>     problem in one of our test cases: We use seqaij matrix format and
>     GMRES with ILU dt.
>
>     Knowing that the ILUdt has been replaced by superlu's version, we
>     therefore configured our petsc installation with
>     ---
>     -PETSC_ARCH=petsc-3.2-serial
>     --prefix=/work_fast/liebm/__petsc-3.2-serial/ --with-debugging=0
>     --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1
>     --with-mpi=0
>     ----
>
>
> I know it involves another build, but turning on debugging is really
> essential for tracking down errors like this. Can you
> reconfigure with debugging under a different PETSC_ARCH and send the trace?
>
>    Thanks
>
>       Matt
>
>     resulting in a nicely compiled lib/libsuperlu_4.2.a.
>
>     Also, ex52.c works (with ILU as a solver).
>
>     Hence, in our code, we replaced the former single-line command
>             _ierr = PCFactorSetDropTolerance(...__stuff...);
>     by the code of ex52.c:
>             Mat F;
>             _ierr = PCFactorSetMatSolverPackage(__pc,MATSOLVERSUPERLU);
>             _ierr = PCFactorSetUpMatSolverPackage(__pc);/* call
>     MatGetFactor() to create F */
>             _ierr = PCFactorGetMatrix(pc,&F);
>             _ierr = MatSuperluSetILUDropTol(F,1.__0e-3);
>
>     After some smooting, the compile and link works now fine, but
>     running the test case results in a segfault; here is the
>     corresponding valgrind output:
>
>     ==17405== Invalid read of size 4
>     ==17405==    at 0x1B15245: MatGetFactor (in
>     /work_fast/liebm/workspace/p1/__src/build/debug/dim2/fluid/__peano-fluid)
>     ==17405==    by 0x1CDF709: PCFactorSetUpMatSolverPackage___Factor
>     (in
>     /work_fast/liebm/workspace/p1/__src/build/debug/dim2/fluid/__peano-fluid)
>     ==17405==    by 0x1B3F7F9: PCFactorSetUpMatSolverPackage (in
>     /work_fast/liebm/workspace/p1/__src/build/debug/dim2/fluid/__peano-fluid)
>     ==17405==    by 0x19284F5:
>     peano::petsc::__PETScNonLinearSolver::__setupLinearSolver(_p_KSP*&,
>     std::string, std::string, double, double, int)
>     (PETScNonLinearSolver.cpp:365)
>     ==17405==    by 0x1926DF2:
>     peano::petsc::__PETScNonLinearSolver::__initialize(int, int, int)
>     (PETScNonLinearSolver.cpp:242)
>     ==17405==    by 0x901225:
>     peano::fluid::__NonLinearTrivialgridFluidSimul__ation::__implementSetInitialValue(bool,
>     double, int, double) (__NonLinearTrivialgridFluidSimul__ation.cpp:540)
>     ==17405==    by 0x17AE3C8:
>     peano::fluid::__SteadyStateIntegrationTestChan__nel::__testTrivialgrid3x3ComputeNavie__rStokesFunction()
>     (__SteadyStateIntegrationTestChan__nel.cpp:114)
>     ==17405==    by 0x17B28A8:
>     peano::fluid::__SteadyStateIntegrationTestChan__nel::run()
>     (__SteadyStateIntegrationTestChan__nel.cpp:484)
>     ==17405==    by 0x19BDCB3: tarch::tests::__TestCaseCollection::run()
>     (TestCaseCollection.cpp:42)
>     ==17405==    by 0x19BDCB3: tarch::tests::__TestCaseCollection::run()
>     (TestCaseCollection.cpp:42)
>     ==17405==    by 0x408A9D:
>     peano::__IntegrationTestCaseRunner::__run(tarch::configuration::__IntegrationTestConfiguration
>     const&) (IntegrationTestCaseRunner.__cpp:127)
>     ==17405==    by 0x412C2E: main (main.cpp:205)
>     ==17405==  Address 0x1e8 is not stack'd, malloc'd or (recently) free'd
>     ==17405==
>     [0]PETSC ERROR:
>     ------------------------------__------------------------------__------------
>     [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>     Violation, probably memory access out of range
>     [0]PETSC ERROR: Try option -start_in_debugger or
>     -on_error_attach_debugger
>     [0]PETSC ERROR: or see
>     http://www.mcs.anl.gov/petsc/__petsc-as/documentation/faq.__html#valgrind[0]PETSC
>     <http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC>
>     ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
>     find memory corruption errors
>     [0]PETSC ERROR: configure using --with-debugging=yes, recompile,
>     link, and run
>     [0]PETSC ERROR: to get more information on the crash.
>     [0]PETSC ERROR: --------------------- Error Message
>     ------------------------------__------
>     [0]PETSC ERROR: Signal received!
>     [0]PETSC ERROR:
>     ------------------------------__------------------------------__------------
>     [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15
>     09:30:51 CDT 2012
>     [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>     [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>     [0]PETSC ERROR: See docs/index.html for manual pages.
>     [0]PETSC ERROR:
>     ------------------------------__------------------------------__------------
>     [0]PETSC ERROR: ./build/debug/dim2/fluid/__peano-fluid on a
>     petsc-3.2 named atsccs58 by liebm Wed May 30 20:06:30 2012
>     [0]PETSC ERROR: Libraries linked from
>     /work_fast/liebm/petsc-3.2-__serial/lib
>     [0]PETSC ERROR: Configure run at Tue May 29 21:05:34 2012
>     [0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial
>     --prefix=/work_fast/liebm/__petsc-3.2-serial/ --with-debugging=0
>     --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1
>     --with-mpi=0
>     [0]PETSC ERROR:
>     ------------------------------__------------------------------__------------
>     [0]PETSC ERROR: User provided function() line 0 in unknown directory
>     unknown file
>
>
>     Did we miss something (keeping in mind that we use ILU now as a
>     preconditioner in combination with GMRES)? Any suggestions are
>     warmly welcome ;-).
>
>     Thanks in advance and best regards
>     Tobias
>
-------------- next part --------------
 1338467468   14:31:08     [atsccs58]   debug        /::validateProgramArguments(int,char**)                 Using configuration file configuration-integration-test.xml
 1338467468   14:31:08     [atsccs58]   debug        peano::petsc::PETScFrame::init()                        received parameters: ./build/debug/dim2/fluid/peano-fluid configuration-integration-test.xml 
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Null Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatGetFactor() line 3929 in /work_fast/liebm/petsc-3.2-p7/src/mat/interface/matrix.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage_Factor() line 17 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage() line 26 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factor.c
  14:31:08      info          UserInterface::writeHeader()                             peano - release T2
  14:31:08      info          UserInterface::writeHeader()                             2d, loaded components: stacks geometry plotter trivialgrid fluid fluid-scenario grid petsc ode
  14:31:08      info          UserInterface::writeHeader()                             opts:       
  14:31:08      info          UserInterface::writeHeader()                             grid-opts: d-loop persistent-attr packed 
  14:31:08      info          UserInterface::writeHeader()                             $Revision: 1.88 $
  14:31:08      info          UserInterface::writeHeader()                             2005 - 2008 Tobias Neckel, Tobias Weinzierl, TUM, Informatik V

test testTrivialgrid3x3ComputeNavierStokesFunction
  14:31:08      info          peano::fluid::AbstractCalculateF::AbstractCalculateF()       using gravity with values: 0, 0
  14:31:08      info          peano::trivialgrid::TrivialgridDataContainer::allocateMemory()      allocate grid with 16 vertex dofs and 9 cell dofs
  14:31:08      info          peano::trivialgrid::Trivialgrid::Trivialgrid(...)            Trivialgrid(): number of grid points: 4, 4, domain size: 1, 1, domain offset: 0, 0
  14:31:08      info          peano::fluid::TrivialgridFluidRunner::createJacobianAdapterAndTrivialgrid()      creating Jacobian instances for assembling type: FINITEDIFF (matrix-free=0)
  14:31:08      info          peano::fluid::FluidSimulation::runFluidSimulation()          preparing the grid
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::parseGeometry()      starting geometry parsing
  14:31:08      info          peano::fluid::FluidSimulation::runFluidSimulation()          counting the inner cells and setting boundary conditions
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::updateGrid         updating grid...
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      assembling the matrix
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::createPETScSolver()      PETSc KSP Object info: , type: cg, maximum iterations=10000, rtol=1e-07, atol=1e-07, divtol=10000, left precond.; PETSc PC Object info: , type: jacobi, 
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::assembleMatrix()::Assemble Q iteration      total number of clock ticks within block (cpu-time,calendar-time): (0s,7.9155e-05s)
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      memory consumption after matrix assemblation: 111 MB
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      global time for matrix assemblation:
  14:31:08      info          FluidSimulation::implementSetInitialValue()                  total number of clock ticks within block (cpu-time,calendar-time): (0s,0.00111055s)
  14:31:08      info          UserInterface::writeFluidTimeStepInfoHeader()                || time step || T_sim total   || norm(du/u)_2h || norm(du/u)_max|| norm(du)_2h   || norm(du)_max  || norm(dp/p)_2h || norm(dp/p)_max|| norm(dp)_2h   || norm(dp)_max  || resdiuum      || depth || \#cells || \#dofNodes || cpu sec./step || \#its/LGS-sol.  || cpu sec./LGS-sol. || mem usage MB || f_glob on nr. 4
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::implementSetInitialValue()      starting pseudo-timestepping with 1 steps and specified step size=0.001
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 365 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 365, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 85
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: Matrix not yet factored; call after KSPSetUp() or PCSetUp()!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: PCFactorGetMatrix_Factor() line 176 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorGetMatrix() line 1220 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/interface/precon.c
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 367 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 367, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 58
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Wrong type of object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatSuperluSetILUDropTol() line 455 in /work_fast/liebm/petsc-3.2-p7/src/mat/impls/aij/seq/superlu/superlu.c
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 369 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 369, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 62
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.000112295 (calendar time)

test testTrivialgrid3x3FreeChannelTwoPreSteps(JACOBIAN_ASSEMBLING_ANALYTICAL)
  14:31:08      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid3x3FreeChannelTwoPreSteps(ANALYTICAL)      running test case
  14:31:08      info          peano::fluid::AbstractCalculateF::AbstractCalculateF()       using gravity with values: 0, 0
  14:31:08      info          peano::trivialgrid::TrivialgridDataContainer::allocateMemory()      allocate grid with 16 vertex dofs and 9 cell dofs
  14:31:08      info          peano::trivialgrid::Trivialgrid::Trivialgrid(...)            Trivialgrid(): number of grid points: 4, 4, domain size: 1, 1, domain offset: 0, 0
  14:31:08      info          peano::fluid::TrivialgridFluidRunner::createJacobianAdapterAndTrivialgrid()      creating Jacobian instances for assembling type: FINITEDIFF (matrix-free=0)
  14:31:08      warning       peano::ode::ODESolver::checkSettings()                       no restart writing in use!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Null Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatGetFactor() line 3929 in /work_fast/liebm/petsc-3.2-p7/src/mat/interface/matrix.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage_Factor() line 17 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage() line 26 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factor.c
  14:31:08      info          peano::fluid::FluidSimulation::runFluidSimulation()          preparing the grid
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::parseGeometry()      starting geometry parsing
  14:31:08      info          peano::fluid::FluidSimulation::runFluidSimulation()          counting the inner cells and setting boundary conditions
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::updateGrid         updating grid...
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      assembling the matrix
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::createPETScSolver()      PETSc KSP Object info: , type: cg, maximum iterations=10000, rtol=1e-07, atol=1e-07, divtol=10000, left precond.; PETSc PC Object info: , type: jacobi, 
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::assembleMatrix()::Assemble Q iteration      total number of clock ticks within block (cpu-time,calendar-time): (0s,6.98566e-05s)
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      memory consumption after matrix assemblation: 111 MB
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      global time for matrix assemblation:
  14:31:08      info          FluidSimulation::implementSetInitialValue()                  total number of clock ticks within block (cpu-time,calendar-time): (0s,0.000394344s)
  14:31:08      info          UserInterface::writeFluidTimeStepInfoHeader()                || time step || T_sim total   || norm(du/u)_2h || norm(du/u)_max|| norm(du)_2h   || norm(du)_max  || norm(dp/p)_2h || norm(dp/p)_max|| norm(dp)_2h   || norm(dp)_max  || resdiuum      || depth || \#cells || \#dofNodes || cpu sec./step || \#its/LGS-sol.  || cpu sec./LGS-sol. || mem usage MB || f_glob on nr. 4
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::implementSetInitialValue()      starting pseudo-timestepping with 2 steps and specified step size=0.001
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 365 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 365, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 85
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: Matrix not yet factored; call after KSPSetUp() or PCSetUp()!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: PCFactorGetMatrix_Factor() line 176 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorGetMatrix() line 1220 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/interface/precon.c
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 367 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 367, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 58
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Wrong type of object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatSuperluSetILUDropTol() line 455 in /work_fast/liebm/petsc-3.2-p7/src/mat/impls/aij/seq/superlu/superlu.c
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 369 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 369, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 62
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
  14:31:08      info          peano::ode::ODESolver::solve()                               starting time integration. StartTime=0, endTime=1, startTimeStepNr=1, endTimeStepNr=1
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.000110388 (calendar time)
  14:31:08      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 0: 6.97099e-14; LinearIterations: 0
  14:31:08      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0 (CPU time), 0.00081563 (calendar time)
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.000109434 (calendar time)
  14:31:08      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 1: 6.49184e-16; LinearIterations: 1
  14:31:08      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver completed in 1 iterations; conv. reason=4
  14:31:08      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver converged!
  14:31:08      info          peano::fluid::FluidSimulation::solveNonLinearSystem()        non-linear solution completed with iterations:1
  14:31:08      info          UserInterface::writeFluidTimeStepInfo(...)                   |    1       |    1.00000e-03  |   -nan  |   -nan  |   0.00000e+00  |   0.00000e+00  |   -nan  |   -nan  |   0.00000e+00  |   0.00000e+00  |   1.01140e-13  |   -1    |   9   |   6     |  1.00000e-02     |  3     |  0.00000e+00     |  112  |   0.00000e+00, 0.00000e+00, 
  14:31:08      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid3x3FreeChannelTwoPreSteps()      testTrivialgrid3x3FreeChannelTwoPreSteps(ANALYTICAL) OK

test testTrivialgrid3x3FreeChannelTwoPreSteps(JACOBIAN_ASSEMBLING_FINITEDIFF)
  14:31:08      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid3x3FreeChannelTwoPreSteps(FINITEDIFF)      running test case
  14:31:08      info          peano::fluid::AbstractCalculateF::AbstractCalculateF()       using gravity with values: 0, 0
  14:31:08      info          peano::trivialgrid::TrivialgridDataContainer::allocateMemory()      allocate grid with 16 vertex dofs and 9 cell dofs
  14:31:08      info          peano::trivialgrid::Trivialgrid::Trivialgrid(...)            Trivialgrid(): number of grid points: 4, 4, domain size: 1, 1, domain offset: 0, 0
  14:31:08      info          peano::fluid::TrivialgridFluidRunner::createJacobianAdapterAndTrivialgrid()      creating Jacobian instances for assembling type: FINITEDIFF (matrix-free=0)
  14:31:08      warning       peano::ode::ODESolver::checkSettings()                       no restart writing in use!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Null Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatGetFactor() line 3929 in /work_fast/liebm/petsc-3.2-p7/src/mat/interface/matrix.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage_Factor() line 17 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage() line 26 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factor.c
  14:31:08      info          peano::fluid::FluidSimulation::runFluidSimulation()          preparing the grid
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::parseGeometry()      starting geometry parsing
  14:31:08      info          peano::fluid::FluidSimulation::runFluidSimulation()          counting the inner cells and setting boundary conditions
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::updateGrid         updating grid...
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      assembling the matrix
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::createPETScSolver()      PETSc KSP Object info: , type: cg, maximum iterations=10000, rtol=1e-07, atol=1e-07, divtol=10000, left precond.; PETSc PC Object info: , type: jacobi, 
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::assembleMatrix()::Assemble Q iteration      total number of clock ticks within block (cpu-time,calendar-time): (0s,7.20024e-05s)
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      memory consumption after matrix assemblation: 112 MB
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      global time for matrix assemblation:
  14:31:08      info          FluidSimulation::implementSetInitialValue()                  total number of clock ticks within block (cpu-time,calendar-time): (0s,0.000437021s)
  14:31:08      info          UserInterface::writeFluidTimeStepInfoHeader()                || time step || T_sim total   || norm(du/u)_2h || norm(du/u)_max|| norm(du)_2h   || norm(du)_max  || norm(dp/p)_2h || norm(dp/p)_max|| norm(dp)_2h   || norm(dp)_max  || resdiuum      || depth || \#cells || \#dofNodes || cpu sec./step || \#its/LGS-sol.  || cpu sec./LGS-sol. || mem usage MB || f_glob on nr. 4
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::implementSetInitialValue()      starting pseudo-timestepping with 2 steps and specified step size=0.001
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 365 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 365, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 85
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: Matrix not yet factored; call after KSPSetUp() or PCSetUp()!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: PCFactorGetMatrix_Factor() line 176 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorGetMatrix() line 1220 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/interface/precon.c
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 367 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 367, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 58
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Wrong type of object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatSuperluSetILUDropTol() line 455 in /work_fast/liebm/petsc-3.2-p7/src/mat/impls/aij/seq/superlu/superlu.c
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 369 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 369, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 62
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
  14:31:08      info          peano::ode::ODESolver::solve()                               starting time integration. StartTime=0, endTime=1, startTimeStepNr=1, endTimeStepNr=1
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.000109673 (calendar time)
  14:31:08      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 0: 6.97099e-14; LinearIterations: 0
  14:31:08      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0 (CPU time), 0.000813007 (calendar time)
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.000109196 (calendar time)
  14:31:08      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 1: 6.49184e-16; LinearIterations: 1
  14:31:08      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver completed in 1 iterations; conv. reason=4
  14:31:08      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver converged!
  14:31:08      info          peano::fluid::FluidSimulation::solveNonLinearSystem()        non-linear solution completed with iterations:1
  14:31:08      info          UserInterface::writeFluidTimeStepInfo(...)                   |    1       |    1.00000e-03  |   -nan  |   -nan  |   0.00000e+00  |   0.00000e+00  |   -nan  |   -nan  |   0.00000e+00  |   0.00000e+00  |   1.01140e-13  |   -1    |   9   |   6     |  0.00000e+00     |  3     |  0.00000e+00     |  112  |   0.00000e+00, 0.00000e+00, 
  14:31:08      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid3x3FreeChannelTwoPreSteps()      testTrivialgrid3x3FreeChannelTwoPreSteps(FINITEDIFF) OK

test testTrivialgrid9x9ObstacleInChannelNoPreSteps(JACOBIAN_ASSEMBLING_ANALYTICAL)
  14:31:08      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid9x9ObstacleInChannelNoPreSteps(ANALYTICAL)      running test case
  14:31:08      info          peano::fluid::AbstractCalculateF::AbstractCalculateF()       using gravity with values: 0, 0
  14:31:08      info          peano::trivialgrid::TrivialgridDataContainer::allocateMemory()      allocate grid with 100 vertex dofs and 81 cell dofs
  14:31:08      info          peano::trivialgrid::Trivialgrid::Trivialgrid(...)            Trivialgrid(): number of grid points: 10, 10, domain size: 1, 1, domain offset: 0, 0
  14:31:08      info          peano::fluid::TrivialgridFluidRunner::createJacobianAdapterAndTrivialgrid()      creating Jacobian instances for assembling type: ANALYTICAL (matrix-free=0)
  14:31:08      warning       peano::ode::ODESolver::checkSettings()                       no restart writing in use!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Null Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatGetFactor() line 3929 in /work_fast/liebm/petsc-3.2-p7/src/mat/interface/matrix.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage_Factor() line 17 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage() line 26 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factor.c
  14:31:08      info          peano::fluid::FluidSimulation::runFluidSimulation()          preparing the grid
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::parseGeometry()      starting geometry parsing
  14:31:08      info          peano::fluid::FluidSimulation::runFluidSimulation()          counting the inner cells and setting boundary conditions
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::updateGrid         updating grid...
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      assembling the matrix
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::createPETScSolver()      PETSc KSP Object info: , type: cg, maximum iterations=10000, rtol=1e-07, atol=1e-07, divtol=10000, left precond.; PETSc PC Object info: , type: jacobi, 
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::assembleMatrix()::Assemble Q iteration      total number of clock ticks within block (cpu-time,calendar-time): (0s,0.000596762s)
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      memory consumption after matrix assemblation: 112 MB
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      global time for matrix assemblation:
  14:31:08      info          FluidSimulation::implementSetInitialValue()                  total number of clock ticks within block (cpu-time,calendar-time): (0s,0.000961542s)
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          UserInterface::writeFluidTimeStepInfoHeader()                || time step || T_sim total   || norm(du/u)_2h || norm(du/u)_max|| norm(du)_2h   || norm(du)_max  || norm(dp/p)_2h || norm(dp/p)_max|| norm(dp)_2h   || norm(dp)_max  || resdiuum      || depth || \#cells || \#dofNodes || cpu sec./step || \#its/LGS-sol.  || cpu sec./LGS-sol. || mem usage MB || f_glob on nr. 4
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::implementSetInitialValue()      starting pseudo-timestepping with 0 steps and specified step size=0.001
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 365 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 365, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 85
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: Matrix not yet factored; call after KSPSetUp() or PCSetUp()!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: PCFactorGetMatrix_Factor() line 176 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorGetMatrix() line 1220 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/interface/precon.c
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 367 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 367, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 58
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Wrong type of object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatSuperluSetILUDropTol() line 455 in /work_fast/liebm/petsc-3.2-p7/src/mat/impls/aij/seq/superlu/superlu.c
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 369 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 369, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 62
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          peano::ode::ODESolver::solve()                               starting time integration. StartTime=0, endTime=1, startTimeStepNr=1, endTimeStepNr=1
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.000845671 (calendar time)
  14:31:08      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 0: 3.24893; LinearIterations: 0
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0 (CPU time), 0.00197101 (calendar time)
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.000895262 (calendar time)
  14:31:08      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 1: 0.445035; LinearIterations: 2
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0 (CPU time), 0.00191617 (calendar time)
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.000865698 (calendar time)
  14:31:08      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 2: 6.48912e-05; LinearIterations: 7
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0 (CPU time), 0.00186586 (calendar time)
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.000849247 (calendar time)
  14:31:08      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 3: 3.69354e-13; LinearIterations: 12
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver completed in 3 iterations; conv. reason=3
  14:31:08      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver converged!
  14:31:08      info          peano::fluid::FluidSimulation::solveNonLinearSystem()        non-linear solution completed with iterations:3
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          UserInterface::writeFluidTimeStepInfo(...)                   |    1       |    1.00000e-03  |   0.00000e+00  |   0.00000e+00  |   0.00000e+00  |   0.00000e+00  |   -nan  |   -nan  |   0.00000e+00  |   0.00000e+00  |   -1.00000e+00  |   -1    |   72   |   56     |  3.00000e-02     |  -1     |  -1.00000e+00     |  113  |   8.54262e+01, -1.24345e-14, 
  14:31:08      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid9x9ObstacleInChannelNoPreSteps()      testTrivialgrid9x9ObstacleInChannelNoPreSteps(ANALYTICAL) OK

test testTrivialgrid9x9ObstacleInChannelNoPreSteps(JACOBIAN_ASSEMBLING_FINITEDIFF)
  14:31:08      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid9x9ObstacleInChannelNoPreSteps(FINITEDIFF)      running test case
  14:31:08      info          peano::fluid::AbstractCalculateF::AbstractCalculateF()       using gravity with values: 0, 0
  14:31:08      info          peano::trivialgrid::TrivialgridDataContainer::allocateMemory()      allocate grid with 100 vertex dofs and 81 cell dofs
  14:31:08      info          peano::trivialgrid::Trivialgrid::Trivialgrid(...)            Trivialgrid(): number of grid points: 10, 10, domain size: 1, 1, domain offset: 0, 0
  14:31:08      info          peano::fluid::TrivialgridFluidRunner::createJacobianAdapterAndTrivialgrid()      creating Jacobian instances for assembling type: ANALYTICAL (matrix-free=0)
  14:31:08      warning       peano::ode::ODESolver::checkSettings()                       no restart writing in use!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Null Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatGetFactor() line 3929 in /work_fast/liebm/petsc-3.2-p7/src/mat/interface/matrix.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage_Factor() line 17 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage() line 26 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factor.c
  14:31:08      info          peano::fluid::FluidSimulation::runFluidSimulation()          preparing the grid
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::parseGeometry()      starting geometry parsing
  14:31:08      info          peano::fluid::FluidSimulation::runFluidSimulation()          counting the inner cells and setting boundary conditions
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::updateGrid         updating grid...
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      assembling the matrix
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::createPETScSolver()      PETSc KSP Object info: , type: cg, maximum iterations=10000, rtol=1e-07, atol=1e-07, divtol=10000, left precond.; PETSc PC Object info: , type: jacobi, 
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::assembleMatrix()::Assemble Q iteration      total number of clock ticks within block (cpu-time,calendar-time): (0s,0.000597954s)
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      memory consumption after matrix assemblation: 112 MB
  14:31:08      info          peano::fluid::FluidSimulation::implementSetInitialValue()      global time for matrix assemblation:
  14:31:08      info          FluidSimulation::implementSetInitialValue()                  total number of clock ticks within block (cpu-time,calendar-time): (0s,0.00101686s)
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          UserInterface::writeFluidTimeStepInfoHeader()                || time step || T_sim total   || norm(du/u)_2h || norm(du/u)_max|| norm(du)_2h   || norm(du)_max  || norm(dp/p)_2h || norm(dp/p)_max|| norm(dp)_2h   || norm(dp)_max  || resdiuum      || depth || \#cells || \#dofNodes || cpu sec./step || \#its/LGS-sol.  || cpu sec./LGS-sol. || mem usage MB || f_glob on nr. 4
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::implementSetInitialValue()      starting pseudo-timestepping with 0 steps and specified step size=0.001
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 365 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 365, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 85
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: Matrix not yet factored; call after KSPSetUp() or PCSetUp()!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: PCFactorGetMatrix_Factor() line 176 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorGetMatrix() line 1220 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/interface/precon.c
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 367 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 367, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 58
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: Wrong type of object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatSuperluSetILUDropTol() line 455 in /work_fast/liebm/petsc-3.2-p7/src/mat/impls/aij/seq/superlu/superlu.c
  14:31:08      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Invalid argument!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:08 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 369 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:08      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 369, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:08      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:08      warning       petscError::logPETScError                                    PetscErrorCode is: 62
  14:31:08      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          peano::ode::ODESolver::solve()                               starting time integration. StartTime=0, endTime=1, startTimeStepNr=1, endTimeStepNr=1
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.000849247 (calendar time)
  14:31:08      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 0: 3.24893; LinearIterations: 0
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0 (CPU time), 0.00209475 (calendar time)
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.000848532 (calendar time)
  14:31:08      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 1: 0.445035; LinearIterations: 2
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0 (CPU time), 0.00186658 (calendar time)
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.000863552 (calendar time)
  14:31:08      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 2: 6.48912e-05; LinearIterations: 7
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0.01 (CPU time), 0.00210094 (calendar time)
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.000846863 (calendar time)
  14:31:08      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 3: 3.69354e-13; LinearIterations: 12
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver completed in 3 iterations; conv. reason=3
  14:31:08      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver converged!
  14:31:08      info          peano::fluid::FluidSimulation::solveNonLinearSystem()        non-linear solution completed with iterations:3
  14:31:08      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:08      info          UserInterface::writeFluidTimeStepInfo(...)                   |    1       |    1.00000e-03  |   0.00000e+00  |   0.00000e+00  |   0.00000e+00  |   0.00000e+00  |   -nan  |   -nan  |   0.00000e+00  |   0.00000e+00  |   -1.00000e+00  |   -1    |   72   |   56     |  3.00000e-02     |  -1     |  -1.00000e+00     |  113  |   8.54262e+01, -1.24345e-14, 
  14:31:08      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid9x9ObstacleInChannelNoPreSteps()      testTrivialgrid9x9ObstacleInChannelNoPreSteps(FINITEDIFF) OK
  14:31:08      info          tarch::tests::TestCaseCollection::run()                      running test case collection "fluid" .. ok
  14:31:08      info          tarch::tests::TestCaseCollection::run()                      running test case collection "fluid-parallel" .. ok
  14:31:08      info          tarch::tests::TestCaseCollection::run()                    running test case collection "integration tests" .... ok
-------------- next part --------------
==3762== Memcheck, a memory error detector
==3762== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==3762== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for copyright info
==3762== Command: ./build/debug/dim2/fluid/peano-fluid configuration-integration-test.xml
==3762== 
 1338467464   14:31:04     [atsccs58]   debug        /::validateProgramArguments(int,char**)                 Using configuration file configuration-integration-test.xml
 1338467464   14:31:04     [atsccs58]   debug        peano::petsc::PETScFrame::init()                        received parameters: ./build/debug/dim2/fluid/peano-fluid configuration-integration-test.xml 
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Null Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatGetFactor() line 3929 in /work_fast/liebm/petsc-3.2-p7/src/mat/interface/matrix.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage_Factor() line 17 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage() line 26 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factor.c
  14:31:04      info          UserInterface::writeHeader()                             peano - release T2
  14:31:04      info          UserInterface::writeHeader()                             2d, loaded components: stacks geometry plotter trivialgrid fluid fluid-scenario grid petsc ode
  14:31:04      info          UserInterface::writeHeader()                             opts:       
  14:31:04      info          UserInterface::writeHeader()                             grid-opts: d-loop persistent-attr packed 
  14:31:04      info          UserInterface::writeHeader()                             $Revision: 1.88 $
  14:31:04      info          UserInterface::writeHeader()                             2005 - 2008 Tobias Neckel, Tobias Weinzierl, TUM, Informatik V

test testTrivialgrid3x3ComputeNavierStokesFunction
  14:31:05      info          peano::fluid::AbstractCalculateF::AbstractCalculateF()       using gravity with values: 0, 0
  14:31:05      info          peano::trivialgrid::TrivialgridDataContainer::allocateMemory()      allocate grid with 16 vertex dofs and 9 cell dofs
  14:31:05      info          peano::trivialgrid::Trivialgrid::Trivialgrid(...)            Trivialgrid(): number of grid points: 4, 4, domain size: 1, 1, domain offset: 0, 0
  14:31:05      info          peano::fluid::TrivialgridFluidRunner::createJacobianAdapterAndTrivialgrid()      creating Jacobian instances for assembling type: FINITEDIFF (matrix-free=0)
  14:31:05      info          peano::fluid::FluidSimulation::runFluidSimulation()          preparing the grid
  14:31:05      info          peano::fluid::TrivialgridFluidSimulation::parseGeometry()      starting geometry parsing
  14:31:05      info          peano::fluid::FluidSimulation::runFluidSimulation()          counting the inner cells and setting boundary conditions
  14:31:05      info          peano::fluid::TrivialgridFluidSimulation::updateGrid         updating grid...
  14:31:05      info          peano::fluid::FluidSimulation::implementSetInitialValue()      assembling the matrix
  14:31:06      info          peano::fluid::TrivialgridFluidSimulation::createPETScSolver()      PETSc KSP Object info: , type: cg, maximum iterations=10000, rtol=1e-07, atol=1e-07, divtol=10000, left precond.; PETSc PC Object info: , type: jacobi, 
  14:31:06      info          peano::fluid::TrivialgridFluidSimulation::assembleMatrix()::Assemble Q iteration      total number of clock ticks within block (cpu-time,calendar-time): (0.06s,0.0543659s)
  14:31:06      info          peano::fluid::FluidSimulation::implementSetInitialValue()      memory consumption after matrix assemblation: 277 MB
  14:31:06      info          peano::fluid::FluidSimulation::implementSetInitialValue()      global time for matrix assemblation:
  14:31:06      info          FluidSimulation::implementSetInitialValue()                  total number of clock ticks within block (cpu-time,calendar-time): (0.44s,0.440282s)
  14:31:06      info          UserInterface::writeFluidTimeStepInfoHeader()                || time step || T_sim total   || norm(du/u)_2h || norm(du/u)_max|| norm(du)_2h   || norm(du)_max  || norm(dp/p)_2h || norm(dp/p)_max|| norm(dp)_2h   || norm(dp)_max  || resdiuum      || depth || \#cells || \#dofNodes || cpu sec./step || \#its/LGS-sol.  || cpu sec./LGS-sol. || mem usage MB || f_glob on nr. 4
  14:31:06      info          peano::fluid::NonLinearTrivialgridFluidSimulation::implementSetInitialValue()      starting pseudo-timestepping with 1 steps and specified step size=0.001
  14:31:06      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 365 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:06      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 365, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:06      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:06      warning       petscError::logPETScError                                    PetscErrorCode is: 85
  14:31:06      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: Matrix not yet factored; call after KSPSetUp() or PCSetUp()!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: PCFactorGetMatrix_Factor() line 176 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorGetMatrix() line 1220 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/interface/precon.c
  14:31:06      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 367 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:06      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 367, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:06      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:06      warning       petscError::logPETScError                                    PetscErrorCode is: 58
  14:31:06      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
==3762== Conditional jump or move depends on uninitialised value(s)
==3762==    at 0x1CC43A8: MatSuperluSetILUDropTol (superlu.c:455)
==3762==    by 0x192899D: peano::petsc::PETScNonLinearSolver::setupLinearSolver(_p_KSP*&, std::string, std::string, double, double, int) (PETScNonLinearSolver.cpp:368)
==3762==    by 0x192716E: peano::petsc::PETScNonLinearSolver::initialize(int, int, int) (PETScNonLinearSolver.cpp:242)
==3762==    by 0x901675: peano::fluid::NonLinearTrivialgridFluidSimulation::implementSetInitialValue(bool, double, int, double) (NonLinearTrivialgridFluidSimulation.cpp:540)
==3762==    by 0x17AE3D8: peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid3x3ComputeNavierStokesFunction() (SteadyStateIntegrationTestChannel.cpp:114)
==3762==    by 0x17B28C7: peano::fluid::SteadyStateIntegrationTestChannel::run() (SteadyStateIntegrationTestChannel.cpp:487)
==3762==    by 0x19BE3B3: tarch::tests::TestCaseCollection::run() (TestCaseCollection.cpp:42)
==3762==    by 0x19BE3B3: tarch::tests::TestCaseCollection::run() (TestCaseCollection.cpp:42)
==3762==    by 0x408BCD: peano::IntegrationTestCaseRunner::run(tarch::configuration::IntegrationTestConfiguration const&) (IntegrationTestCaseRunner.cpp:127)
==3762==    by 0x412D5E: main (main.cpp:205)
==3762== 
==3762== Conditional jump or move depends on uninitialised value(s)
==3762==    at 0x1CC43FB: MatSuperluSetILUDropTol (superlu.c:455)
==3762==    by 0x192899D: peano::petsc::PETScNonLinearSolver::setupLinearSolver(_p_KSP*&, std::string, std::string, double, double, int) (PETScNonLinearSolver.cpp:368)
==3762==    by 0x192716E: peano::petsc::PETScNonLinearSolver::initialize(int, int, int) (PETScNonLinearSolver.cpp:242)
==3762==    by 0x901675: peano::fluid::NonLinearTrivialgridFluidSimulation::implementSetInitialValue(bool, double, int, double) (NonLinearTrivialgridFluidSimulation.cpp:540)
==3762==    by 0x17AE3D8: peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid3x3ComputeNavierStokesFunction() (SteadyStateIntegrationTestChannel.cpp:114)
==3762==    by 0x17B28C7: peano::fluid::SteadyStateIntegrationTestChannel::run() (SteadyStateIntegrationTestChannel.cpp:487)
==3762==    by 0x19BE3B3: tarch::tests::TestCaseCollection::run() (TestCaseCollection.cpp:42)
==3762==    by 0x19BE3B3: tarch::tests::TestCaseCollection::run() (TestCaseCollection.cpp:42)
==3762==    by 0x408BCD: peano::IntegrationTestCaseRunner::run(tarch::configuration::IntegrationTestConfiguration const&) (IntegrationTestCaseRunner.cpp:127)
==3762==    by 0x412D5E: main (main.cpp:205)
==3762== 
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Corrupt argument: see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind!
[0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatSuperluSetILUDropTol() line 455 in /work_fast/liebm/petsc-3.2-p7/src/mat/impls/aij/seq/superlu/superlu.c
  14:31:06      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Corrupt argument: see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 369 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:06      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 369, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:06      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:06      warning       petscError::logPETScError                                    PetscErrorCode is: 64
  14:31:06      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
  14:31:07      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0.04 (CPU time), 0.0409479 (calendar time)

test testTrivialgrid3x3FreeChannelTwoPreSteps(JACOBIAN_ASSEMBLING_ANALYTICAL)
  14:31:07      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid3x3FreeChannelTwoPreSteps(ANALYTICAL)      running test case
  14:31:07      info          peano::fluid::AbstractCalculateF::AbstractCalculateF()       using gravity with values: 0, 0
  14:31:07      info          peano::trivialgrid::TrivialgridDataContainer::allocateMemory()      allocate grid with 16 vertex dofs and 9 cell dofs
  14:31:07      info          peano::trivialgrid::Trivialgrid::Trivialgrid(...)            Trivialgrid(): number of grid points: 4, 4, domain size: 1, 1, domain offset: 0, 0
  14:31:07      info          peano::fluid::TrivialgridFluidRunner::createJacobianAdapterAndTrivialgrid()      creating Jacobian instances for assembling type: FINITEDIFF (matrix-free=0)
  14:31:07      warning       peano::ode::ODESolver::checkSettings()                       no restart writing in use!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Null Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatGetFactor() line 3929 in /work_fast/liebm/petsc-3.2-p7/src/mat/interface/matrix.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage_Factor() line 17 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage() line 26 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factor.c
  14:31:07      info          peano::fluid::FluidSimulation::runFluidSimulation()          preparing the grid
  14:31:07      info          peano::fluid::TrivialgridFluidSimulation::parseGeometry()      starting geometry parsing
  14:31:07      info          peano::fluid::FluidSimulation::runFluidSimulation()          counting the inner cells and setting boundary conditions
  14:31:07      info          peano::fluid::TrivialgridFluidSimulation::updateGrid         updating grid...
  14:31:07      info          peano::fluid::FluidSimulation::implementSetInitialValue()      assembling the matrix
  14:31:07      info          peano::fluid::TrivialgridFluidSimulation::createPETScSolver()      PETSc KSP Object info: , type: cg, maximum iterations=10000, rtol=1e-07, atol=1e-07, divtol=10000, left precond.; PETSc PC Object info: , type: jacobi, 
  14:31:07      info          peano::fluid::TrivialgridFluidSimulation::assembleMatrix()::Assemble Q iteration      total number of clock ticks within block (cpu-time,calendar-time): (0s,0.00335407s)
  14:31:07      info          peano::fluid::FluidSimulation::implementSetInitialValue()      memory consumption after matrix assemblation: 283 MB
  14:31:07      info          peano::fluid::FluidSimulation::implementSetInitialValue()      global time for matrix assemblation:
  14:31:07      info          FluidSimulation::implementSetInitialValue()                  total number of clock ticks within block (cpu-time,calendar-time): (0.01s,0.0105739s)
  14:31:07      info          UserInterface::writeFluidTimeStepInfoHeader()                || time step || T_sim total   || norm(du/u)_2h || norm(du/u)_max|| norm(du)_2h   || norm(du)_max  || norm(dp/p)_2h || norm(dp/p)_max|| norm(dp)_2h   || norm(dp)_max  || resdiuum      || depth || \#cells || \#dofNodes || cpu sec./step || \#its/LGS-sol.  || cpu sec./LGS-sol. || mem usage MB || f_glob on nr. 4
  14:31:07      info          peano::fluid::NonLinearTrivialgridFluidSimulation::implementSetInitialValue()      starting pseudo-timestepping with 2 steps and specified step size=0.001
  14:31:07      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 365 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:07      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 365, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:07      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:07      warning       petscError::logPETScError                                    PetscErrorCode is: 85
  14:31:07      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: Matrix not yet factored; call after KSPSetUp() or PCSetUp()!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: PCFactorGetMatrix_Factor() line 176 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorGetMatrix() line 1220 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/interface/precon.c
  14:31:07      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 367 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:07      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 367, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:07      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:07      warning       petscError::logPETScError                                    PetscErrorCode is: 58
  14:31:07      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Corrupt argument: see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind!
[0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatSuperluSetILUDropTol() line 455 in /work_fast/liebm/petsc-3.2-p7/src/mat/impls/aij/seq/superlu/superlu.c
  14:31:07      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Corrupt argument: see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 369 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:07      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 369, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:07      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:07      warning       petscError::logPETScError                                    PetscErrorCode is: 64
  14:31:07      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
  14:31:07      info          peano::ode::ODESolver::solve()                               starting time integration. StartTime=0, endTime=1, startTimeStepNr=1, endTimeStepNr=1
  14:31:07      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0 (CPU time), 0.00469422 (calendar time)
  14:31:07      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 0: 1.18128e-13; LinearIterations: 0
  14:31:07      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:07      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0.08 (CPU time), 0.0804036 (calendar time)
  14:31:08      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0.01 (CPU time), 0.0048914 (calendar time)
  14:31:08      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 1: 7.45947e-16; LinearIterations: 1
  14:31:08      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver completed in 1 iterations; conv. reason=4
  14:31:08      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver converged!
  14:31:08      info          peano::fluid::FluidSimulation::solveNonLinearSystem()        non-linear solution completed with iterations:1
  14:31:08      info          UserInterface::writeFluidTimeStepInfo(...)                   |    1       |    1.00000e-03  |   -nan  |   -nan  |   0.00000e+00  |   0.00000e+00  |   -nan  |   -nan  |   0.00000e+00  |   0.00000e+00  |   2.90332e-13  |   -1    |   9   |   6     |  8.00000e-01     |  3     |  0.00000e+00     |  284  |   0.00000e+00, 0.00000e+00, 
  14:31:08      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid3x3FreeChannelTwoPreSteps()      testTrivialgrid3x3FreeChannelTwoPreSteps(ANALYTICAL) OK

test testTrivialgrid3x3FreeChannelTwoPreSteps(JACOBIAN_ASSEMBLING_FINITEDIFF)
  14:31:08      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid3x3FreeChannelTwoPreSteps(FINITEDIFF)      running test case
  14:31:08      info          peano::fluid::AbstractCalculateF::AbstractCalculateF()       using gravity with values: 0, 0
  14:31:08      info          peano::trivialgrid::TrivialgridDataContainer::allocateMemory()      allocate grid with 16 vertex dofs and 9 cell dofs
  14:31:08      info          peano::trivialgrid::Trivialgrid::Trivialgrid(...)            Trivialgrid(): number of grid points: 4, 4, domain size: 1, 1, domain offset: 0, 0
  14:31:08      info          peano::fluid::TrivialgridFluidRunner::createJacobianAdapterAndTrivialgrid()      creating Jacobian instances for assembling type: FINITEDIFF (matrix-free=0)
  14:31:08      warning       peano::ode::ODESolver::checkSettings()                       no restart writing in use!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Null Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatGetFactor() line 3929 in /work_fast/liebm/petsc-3.2-p7/src/mat/interface/matrix.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage_Factor() line 17 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage() line 26 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factor.c
  14:31:08      info          peano::fluid::FluidSimulation::runFluidSimulation()          preparing the grid
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::parseGeometry()      starting geometry parsing
  14:31:08      info          peano::fluid::FluidSimulation::runFluidSimulation()          counting the inner cells and setting boundary conditions
  14:31:08      info          peano::fluid::TrivialgridFluidSimulation::updateGrid         updating grid...
  14:31:09      info          peano::fluid::FluidSimulation::implementSetInitialValue()      assembling the matrix
  14:31:09      info          peano::fluid::TrivialgridFluidSimulation::createPETScSolver()      PETSc KSP Object info: , type: cg, maximum iterations=10000, rtol=1e-07, atol=1e-07, divtol=10000, left precond.; PETSc PC Object info: , type: jacobi, 
  14:31:09      info          peano::fluid::TrivialgridFluidSimulation::assembleMatrix()::Assemble Q iteration      total number of clock ticks within block (cpu-time,calendar-time): (0s,0.00506377s)
  14:31:09      info          peano::fluid::FluidSimulation::implementSetInitialValue()      memory consumption after matrix assemblation: 288 MB
  14:31:09      info          peano::fluid::FluidSimulation::implementSetInitialValue()      global time for matrix assemblation:
  14:31:09      info          FluidSimulation::implementSetInitialValue()                  total number of clock ticks within block (cpu-time,calendar-time): (0.01s,0.0144298s)
  14:31:09      info          UserInterface::writeFluidTimeStepInfoHeader()                || time step || T_sim total   || norm(du/u)_2h || norm(du/u)_max|| norm(du)_2h   || norm(du)_max  || norm(dp/p)_2h || norm(dp/p)_max|| norm(dp)_2h   || norm(dp)_max  || resdiuum      || depth || \#cells || \#dofNodes || cpu sec./step || \#its/LGS-sol.  || cpu sec./LGS-sol. || mem usage MB || f_glob on nr. 4
  14:31:09      info          peano::fluid::NonLinearTrivialgridFluidSimulation::implementSetInitialValue()      starting pseudo-timestepping with 2 steps and specified step size=0.001
  14:31:09      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 365 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:09      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 365, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:09      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:09      warning       petscError::logPETScError                                    PetscErrorCode is: 85
  14:31:09      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: Matrix not yet factored; call after KSPSetUp() or PCSetUp()!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: PCFactorGetMatrix_Factor() line 176 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorGetMatrix() line 1220 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/interface/precon.c
  14:31:09      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 367 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:09      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 367, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:09      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:09      warning       petscError::logPETScError                                    PetscErrorCode is: 58
  14:31:09      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Corrupt argument: see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind!
[0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatSuperluSetILUDropTol() line 455 in /work_fast/liebm/petsc-3.2-p7/src/mat/impls/aij/seq/superlu/superlu.c
  14:31:09      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Corrupt argument: see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 369 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:09      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 369, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:09      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:09      warning       petscError::logPETScError                                    PetscErrorCode is: 64
  14:31:09      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
  14:31:09      info          peano::ode::ODESolver::solve()                               starting time integration. StartTime=0, endTime=1, startTimeStepNr=1, endTimeStepNr=1
  14:31:09      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0.01 (CPU time), 0.00466967 (calendar time)
  14:31:09      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 0: 1.18128e-13; LinearIterations: 0
  14:31:09      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:09      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0.03 (CPU time), 0.0290048 (calendar time)
  14:31:09      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0.01 (CPU time), 0.00472021 (calendar time)
  14:31:09      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 1: 7.45947e-16; LinearIterations: 1
  14:31:09      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver completed in 1 iterations; conv. reason=4
  14:31:09      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver converged!
  14:31:09      info          peano::fluid::FluidSimulation::solveNonLinearSystem()        non-linear solution completed with iterations:1
  14:31:09      info          UserInterface::writeFluidTimeStepInfo(...)                   |    1       |    1.00000e-03  |   -nan  |   -nan  |   0.00000e+00  |   0.00000e+00  |   -nan  |   -nan  |   0.00000e+00  |   0.00000e+00  |   2.90332e-13  |   -1    |   9   |   6     |  7.00000e-02     |  3     |  1.00000e-02     |  288  |   0.00000e+00, 0.00000e+00, 
  14:31:09      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid3x3FreeChannelTwoPreSteps()      testTrivialgrid3x3FreeChannelTwoPreSteps(FINITEDIFF) OK

test testTrivialgrid9x9ObstacleInChannelNoPreSteps(JACOBIAN_ASSEMBLING_ANALYTICAL)
  14:31:09      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid9x9ObstacleInChannelNoPreSteps(ANALYTICAL)      running test case
  14:31:09      info          peano::fluid::AbstractCalculateF::AbstractCalculateF()       using gravity with values: 0, 0
  14:31:09      info          peano::trivialgrid::TrivialgridDataContainer::allocateMemory()      allocate grid with 100 vertex dofs and 81 cell dofs
  14:31:09      info          peano::trivialgrid::Trivialgrid::Trivialgrid(...)            Trivialgrid(): number of grid points: 10, 10, domain size: 1, 1, domain offset: 0, 0
  14:31:09      info          peano::fluid::TrivialgridFluidRunner::createJacobianAdapterAndTrivialgrid()      creating Jacobian instances for assembling type: ANALYTICAL (matrix-free=0)
  14:31:09      warning       peano::ode::ODESolver::checkSettings()                       no restart writing in use!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Null Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatGetFactor() line 3929 in /work_fast/liebm/petsc-3.2-p7/src/mat/interface/matrix.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage_Factor() line 17 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage() line 26 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factor.c
  14:31:09      info          peano::fluid::FluidSimulation::runFluidSimulation()          preparing the grid
  14:31:09      info          peano::fluid::TrivialgridFluidSimulation::parseGeometry()      starting geometry parsing
  14:31:10      info          peano::fluid::FluidSimulation::runFluidSimulation()          counting the inner cells and setting boundary conditions
  14:31:10      info          peano::fluid::TrivialgridFluidSimulation::updateGrid         updating grid...
  14:31:12      info          peano::fluid::FluidSimulation::implementSetInitialValue()      assembling the matrix
  14:31:12      info          peano::fluid::TrivialgridFluidSimulation::createPETScSolver()      PETSc KSP Object info: , type: cg, maximum iterations=10000, rtol=1e-07, atol=1e-07, divtol=10000, left precond.; PETSc PC Object info: , type: jacobi, 
  14:31:12      info          peano::fluid::TrivialgridFluidSimulation::assembleMatrix()::Assemble Q iteration      total number of clock ticks within block (cpu-time,calendar-time): (0.03s,0.0306334s)
  14:31:12      info          peano::fluid::FluidSimulation::implementSetInitialValue()      memory consumption after matrix assemblation: 289 MB
  14:31:12      info          peano::fluid::FluidSimulation::implementSetInitialValue()      global time for matrix assemblation:
  14:31:12      info          FluidSimulation::implementSetInitialValue()                  total number of clock ticks within block (cpu-time,calendar-time): (0.03s,0.0383279s)
  14:31:12      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:12      info          UserInterface::writeFluidTimeStepInfoHeader()                || time step || T_sim total   || norm(du/u)_2h || norm(du/u)_max|| norm(du)_2h   || norm(du)_max  || norm(dp/p)_2h || norm(dp/p)_max|| norm(dp)_2h   || norm(dp)_max  || resdiuum      || depth || \#cells || \#dofNodes || cpu sec./step || \#its/LGS-sol.  || cpu sec./LGS-sol. || mem usage MB || f_glob on nr. 4
  14:31:12      info          peano::fluid::NonLinearTrivialgridFluidSimulation::implementSetInitialValue()      starting pseudo-timestepping with 0 steps and specified step size=0.001
  14:31:12      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 365 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:12      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 365, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:12      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:12      warning       petscError::logPETScError                                    PetscErrorCode is: 85
  14:31:12      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: Matrix not yet factored; call after KSPSetUp() or PCSetUp()!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: PCFactorGetMatrix_Factor() line 176 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorGetMatrix() line 1220 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/interface/precon.c
  14:31:12      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 367 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:12      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 367, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:12      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:12      warning       petscError::logPETScError                                    PetscErrorCode is: 58
  14:31:12      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Corrupt argument: see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind!
[0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatSuperluSetILUDropTol() line 455 in /work_fast/liebm/petsc-3.2-p7/src/mat/impls/aij/seq/superlu/superlu.c
  14:31:12      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Corrupt argument: see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 369 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:12      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 369, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:12      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:12      warning       petscError::logPETScError                                    PetscErrorCode is: 64
  14:31:12      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
  14:31:12      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:12      info          peano::ode::ODESolver::solve()                               starting time integration. StartTime=0, endTime=1, startTimeStepNr=1, endTimeStepNr=1
  14:31:12      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0.04 (CPU time), 0.0394928 (calendar time)
  14:31:12      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 0: 3.24893; LinearIterations: 0
  14:31:12      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:12      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:12      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0.08 (CPU time), 0.0848253 (calendar time)
  14:31:12      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0.04 (CPU time), 0.0393052 (calendar time)
  14:31:12      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 1: 0.445035; LinearIterations: 2
  14:31:13      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:13      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:13      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0.07 (CPU time), 0.0622749 (calendar time)
  14:31:13      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0.04 (CPU time), 0.0393724 (calendar time)
  14:31:13      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 2: 6.48912e-05; LinearIterations: 7
  14:31:13      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:13      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:13      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0.06 (CPU time), 0.0622993 (calendar time)
  14:31:13      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0.04 (CPU time), 0.0419505 (calendar time)
  14:31:13      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 3: 3.69087e-13; LinearIterations: 12
  14:31:13      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:13      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver completed in 3 iterations; conv. reason=3
  14:31:13      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver converged!
  14:31:13      info          peano::fluid::FluidSimulation::solveNonLinearSystem()        non-linear solution completed with iterations:3
  14:31:13      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:13      info          UserInterface::writeFluidTimeStepInfo(...)                   |    1       |    1.00000e-03  |   0.00000e+00  |   0.00000e+00  |   0.00000e+00  |   0.00000e+00  |   -nan  |   -nan  |   0.00000e+00  |   0.00000e+00  |   -1.00000e+00  |   -1    |   72   |   56     |  8.70000e-01     |  -1     |  -1.00000e+00     |  290  |   8.54262e+01, -7.99361e-15, 
  14:31:13      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid9x9ObstacleInChannelNoPreSteps()      testTrivialgrid9x9ObstacleInChannelNoPreSteps(ANALYTICAL) OK

test testTrivialgrid9x9ObstacleInChannelNoPreSteps(JACOBIAN_ASSEMBLING_FINITEDIFF)
  14:31:13      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid9x9ObstacleInChannelNoPreSteps(FINITEDIFF)      running test case
  14:31:13      info          peano::fluid::AbstractCalculateF::AbstractCalculateF()       using gravity with values: 0, 0
  14:31:13      info          peano::trivialgrid::TrivialgridDataContainer::allocateMemory()      allocate grid with 100 vertex dofs and 81 cell dofs
  14:31:13      info          peano::trivialgrid::Trivialgrid::Trivialgrid(...)            Trivialgrid(): number of grid points: 10, 10, domain size: 1, 1, domain offset: 0, 0
  14:31:13      info          peano::fluid::TrivialgridFluidRunner::createJacobianAdapterAndTrivialgrid()      creating Jacobian instances for assembling type: ANALYTICAL (matrix-free=0)
  14:31:13      warning       peano::ode::ODESolver::checkSettings()                       no restart writing in use!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Null Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatGetFactor() line 3929 in /work_fast/liebm/petsc-3.2-p7/src/mat/interface/matrix.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage_Factor() line 17 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorSetUpMatSolverPackage() line 26 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factor.c
  14:31:13      info          peano::fluid::FluidSimulation::runFluidSimulation()          preparing the grid
  14:31:13      info          peano::fluid::TrivialgridFluidSimulation::parseGeometry()      starting geometry parsing
  14:31:14      info          peano::fluid::FluidSimulation::runFluidSimulation()          counting the inner cells and setting boundary conditions
  14:31:14      info          peano::fluid::TrivialgridFluidSimulation::updateGrid         updating grid...
  14:31:16      info          peano::fluid::FluidSimulation::implementSetInitialValue()      assembling the matrix
  14:31:16      info          peano::fluid::TrivialgridFluidSimulation::createPETScSolver()      PETSc KSP Object info: , type: cg, maximum iterations=10000, rtol=1e-07, atol=1e-07, divtol=10000, left precond.; PETSc PC Object info: , type: jacobi, 
  14:31:16      info          peano::fluid::TrivialgridFluidSimulation::assembleMatrix()::Assemble Q iteration      total number of clock ticks within block (cpu-time,calendar-time): (0.03s,0.0327389s)
  14:31:16      info          peano::fluid::FluidSimulation::implementSetInitialValue()      memory consumption after matrix assemblation: 290 MB
  14:31:16      info          peano::fluid::FluidSimulation::implementSetInitialValue()      global time for matrix assemblation:
  14:31:16      info          FluidSimulation::implementSetInitialValue()                  total number of clock ticks within block (cpu-time,calendar-time): (0.04s,0.0402725s)
  14:31:16      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:16      info          UserInterface::writeFluidTimeStepInfoHeader()                || time step || T_sim total   || norm(du/u)_2h || norm(du/u)_max|| norm(du)_2h   || norm(du)_max  || norm(dp/p)_2h || norm(dp/p)_max|| norm(dp)_2h   || norm(dp)_max  || resdiuum      || depth || \#cells || \#dofNodes || cpu sec./step || \#its/LGS-sol.  || cpu sec./LGS-sol. || mem usage MB || f_glob on nr. 4
  14:31:16      info          peano::fluid::NonLinearTrivialgridFluidSimulation::implementSetInitialValue()      starting pseudo-timestepping with 0 steps and specified step size=0.001
  14:31:16      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 365 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:16      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 365, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:16      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:16      warning       petscError::logPETScError                                    PetscErrorCode is: 85
  14:31:16      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: Matrix not yet factored; call after KSPSetUp() or PCSetUp()!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: PCFactorGetMatrix_Factor() line 176 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/impls/factor/factimpl.c
[0]PETSC ERROR: PCFactorGetMatrix() line 1220 in /work_fast/liebm/petsc-3.2-p7/src/ksp/pc/interface/precon.c
  14:31:16      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Operation done in wrong order!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 367 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:16      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 367, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:16      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:16      warning       petscError::logPETScError                                    PetscErrorCode is: 58
  14:31:16      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Corrupt argument: see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind!
[0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: MatSuperluSetILUDropTol() line 455 in /work_fast/liebm/petsc-3.2-p7/src/mat/impls/aij/seq/superlu/superlu.c
  14:31:16      warning       petscError::logPETScError                                    Attention: no PetscErrorBaseMessage imitated!
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Corrupt argument: see http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 7, Thu Mar 15 09:30:51 CDT 2012 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./build/debug/dim2/fluid/peano-fluid on a petsc-3.2 named atsccs58 by liebm Thu May 31 14:31:04 2012
[0]PETSC ERROR: Libraries linked from /work_fast/liebm/petsc-3.2-serial-debug/lib
[0]PETSC ERROR: Configure run at Thu May 31 10:27:55 2012
[0]PETSC ERROR: Configure options --PETSC_ARCH=petsc-3.2-serial-debug --prefix=/work_fast/liebm/petsc-3.2-serial-debug --with-debugging=1 --with-cxx=gcc --with-fc=gfortran --with-cc=gcc --download-superlu=1 --with-mpi=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: peano::petsc::PETScNonLinearSolver::setupLinearSolver() line 369 in petscpetsc/PETScNonLinearSolver.cpp
  14:31:16      error         petscError::logPetscTraceBackErrorHandler                    Major PETSc error occured in function peano::petsc::PETScNonLinearSolver::setupLinearSolver, line 369, dir. petsc, file petsc/PETScNonLinearSolver.cpp.

  14:31:16      warning       petscError::logPetscTraceBackErrorHandler                    Implementation not yet completed
  14:31:16      warning       petscError::logPETScError                                    PetscErrorCode is: 64
  14:31:16      warning       petscError::logPETScError                                    Attention: no MPI abort for main or unknown function imitated!
  14:31:16      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:16      info          peano::ode::ODESolver::solve()                               starting time integration. StartTime=0, endTime=1, startTimeStepNr=1, endTimeStepNr=1
  14:31:16      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0.04 (CPU time), 0.0396516 (calendar time)
  14:31:16      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 0: 3.24893; LinearIterations: 0
  14:31:16      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:16      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:17      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0.06 (CPU time), 0.0622516 (calendar time)
  14:31:17      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0.04 (CPU time), 0.0400362 (calendar time)
  14:31:17      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 1: 0.445035; LinearIterations: 2
  14:31:17      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:17      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:17      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0.06 (CPU time), 0.0621722 (calendar time)
  14:31:17      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0.03 (CPU time), 0.0395648 (calendar time)
  14:31:17      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 2: 6.48912e-05; LinearIterations: 7
  14:31:17      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:17      info          peano::petsc::PETScNonLinearEquations::jacobianGrid()        iteration started.
  14:31:17      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesJacobian()      runtime for Jacobian: 0.06 (CPU time), 0.0616219 (calendar time)
  14:31:17      info          peano::fluid::NonLinearTrivialgridFluidSimulation::computeNavierStokesFunction()      runtime for B-evaluation: 0.04 (CPU time), 0.0397391 (calendar time)
  14:31:17      info          peano::petsc::PETScNonLinearEquations::monitorNonLinearSolver()      residual norm at iteration 3: 3.69087e-13; LinearIterations: 12
  14:31:17      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:17      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver completed in 3 iterations; conv. reason=3
  14:31:17      info          peano::petsc::PETScNonLinearSolver::solve()                  NLGS solver converged!
  14:31:17      info          peano::fluid::FluidSimulation::solveNonLinearSystem()        non-linear solution completed with iterations:3
  14:31:17      info          peano::fluid::TrivialgridEventHandle2VTKWriter4FluidAdapter::endIteration      
Accumulated and integrated outflow is: 0

  14:31:17      info          UserInterface::writeFluidTimeStepInfo(...)                   |    1       |    1.00000e-03  |   0.00000e+00  |   0.00000e+00  |   0.00000e+00  |   0.00000e+00  |   -nan  |   -nan  |   0.00000e+00  |   0.00000e+00  |   -1.00000e+00  |   -1    |   72   |   56     |  7.70000e-01     |  -1     |  -1.00000e+00     |  290  |   8.54262e+01, -7.99361e-15, 
  14:31:17      info          peano::fluid::SteadyStateIntegrationTestChannel::testTrivialgrid9x9ObstacleInChannelNoPreSteps()      testTrivialgrid9x9ObstacleInChannelNoPreSteps(FINITEDIFF) OK
  14:31:17      info          tarch::tests::TestCaseCollection::run()                      running test case collection "fluid" .. ok
  14:31:17      info          tarch::tests::TestCaseCollection::run()                      running test case collection "fluid-parallel" .. ok
  14:31:17      info          tarch::tests::TestCaseCollection::run()                    running test case collection "integration tests" .... ok
==3762== 
==3762== HEAP SUMMARY:
==3762==     in use at exit: 2,688 bytes in 64 blocks
==3762==   total heap usage: 335,866 allocs, 335,802 frees, 64,575,358 bytes allocated
==3762== 
==3762== LEAK SUMMARY:
==3762==    definitely lost: 312 bytes in 3 blocks
==3762==    indirectly lost: 2,376 bytes in 61 blocks
==3762==      possibly lost: 0 bytes in 0 blocks
==3762==    still reachable: 0 bytes in 0 blocks
==3762==         suppressed: 0 bytes in 0 blocks
==3762== Rerun with --leak-check=full to see details of leaked memory
==3762== 
==3762== For counts of detected and suppressed errors, rerun with: -v
==3762== Use --track-origins=yes to see where uninitialised values come from
==3762== ERROR SUMMARY: 10 errors from 2 contexts (suppressed: 4 from 4)

From recrusader at gmail.com  Thu May 31 13:29:32 2012
From: recrusader at gmail.com (recrusader)
Date: Thu, 31 May 2012 13:29:32 -0500
Subject: [petsc-users] output parallel matrix
Message-ID: <CAA==FctLhamjQP+ywkQbRqMGq5RXZCrFZZjzdT3wD5jhO9xiJQ@mail.gmail.com>

Dear PETSc developers,

I am trying to output a parallel matrix using MatView in ASCII format. To
my understanding, PETSc will first get this matrix at one processor and
output it.
In my case, it is out of memory. I am wondering whether there is any method
to output this matrix.

Thanks a lot.
Yujie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120531/d3542bbf/attachment.html>

From bsmith at mcs.anl.gov  Thu May 31 13:31:00 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 31 May 2012 13:31:00 -0500
Subject: [petsc-users] output parallel matrix
In-Reply-To: <CAA==FctLhamjQP+ywkQbRqMGq5RXZCrFZZjzdT3wD5jhO9xiJQ@mail.gmail.com>
References: <CAA==FctLhamjQP+ywkQbRqMGq5RXZCrFZZjzdT3wD5jhO9xiJQ@mail.gmail.com>
Message-ID: <09A3279F-3B26-4506-A8D6-E4B3698588A9@mcs.anl.gov>


On May 31, 2012, at 1:29 PM, recrusader wrote:

> Dear PETSc developers,
> 
> I am trying to output a parallel matrix using MatView in ASCII format. To my understanding, PETSc will first get this matrix at one processor and output it.
> In my case, it is out of memory. I am wondering whether there is any method to output this matrix.
> 
> Thanks a lot.
> Yujie

    You can never use ASCII output for a large matrix. Can you use the PETSc binary output?

    Barry



From recrusader at gmail.com  Thu May 31 13:33:22 2012
From: recrusader at gmail.com (recrusader)
Date: Thu, 31 May 2012 13:33:22 -0500
Subject: [petsc-users] output parallel matrix
In-Reply-To: <09A3279F-3B26-4506-A8D6-E4B3698588A9@mcs.anl.gov>
References: <CAA==FctLhamjQP+ywkQbRqMGq5RXZCrFZZjzdT3wD5jhO9xiJQ@mail.gmail.com>
	<09A3279F-3B26-4506-A8D6-E4B3698588A9@mcs.anl.gov>
Message-ID: <CAA==Fcu9CY+3_GMEwF6yk7Ee4PrmwxvwvQ8E5ZTpOUi9-6DOAg@mail.gmail.com>

Dear Barry,

Since I need to further process the matrix using my codes. I can easily
check the matrix using ASCII format. How to read a matrix in PETSc binary
format using c/c++?

Thanks a lot,
Yujie

On Thu, May 31, 2012 at 1:31 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On May 31, 2012, at 1:29 PM, recrusader wrote:
>
> > Dear PETSc developers,
> >
> > I am trying to output a parallel matrix using MatView in ASCII format.
> To my understanding, PETSc will first get this matrix at one processor and
> output it.
> > In my case, it is out of memory. I am wondering whether there is any
> method to output this matrix.
> >
> > Thanks a lot.
> > Yujie
>
>     You can never use ASCII output for a large matrix. Can you use the
> PETSc binary output?
>
>    Barry
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120531/74be9750/attachment.html>

From jedbrown at mcs.anl.gov  Thu May 31 13:34:02 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 31 May 2012 13:34:02 -0500
Subject: [petsc-users] output parallel matrix
In-Reply-To: <CAA==Fcu9CY+3_GMEwF6yk7Ee4PrmwxvwvQ8E5ZTpOUi9-6DOAg@mail.gmail.com>
References: <CAA==FctLhamjQP+ywkQbRqMGq5RXZCrFZZjzdT3wD5jhO9xiJQ@mail.gmail.com>
	<09A3279F-3B26-4506-A8D6-E4B3698588A9@mcs.anl.gov>
	<CAA==Fcu9CY+3_GMEwF6yk7Ee4PrmwxvwvQ8E5ZTpOUi9-6DOAg@mail.gmail.com>
Message-ID: <CAM9tzSmsv0HM4petqqBZ1vsrfTVzUw2-UJ1t3qRzPGQr2wc2QA@mail.gmail.com>

On Thu, May 31, 2012 at 1:33 PM, recrusader <recrusader at gmail.com> wrote:

> Since I need to further process the matrix using my codes. I can easily
> check the matrix using ASCII format. How to read a matrix in PETSc binary
> format using c/c++?


MatLoad()
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120531/2d357929/attachment.html>

From recrusader at gmail.com  Thu May 31 13:40:10 2012
From: recrusader at gmail.com (recrusader)
Date: Thu, 31 May 2012 13:40:10 -0500
Subject: [petsc-users] output parallel matrix
In-Reply-To: <CAM9tzSmsv0HM4petqqBZ1vsrfTVzUw2-UJ1t3qRzPGQr2wc2QA@mail.gmail.com>
References: <CAA==FctLhamjQP+ywkQbRqMGq5RXZCrFZZjzdT3wD5jhO9xiJQ@mail.gmail.com>
	<09A3279F-3B26-4506-A8D6-E4B3698588A9@mcs.anl.gov>
	<CAA==Fcu9CY+3_GMEwF6yk7Ee4PrmwxvwvQ8E5ZTpOUi9-6DOAg@mail.gmail.com>
	<CAM9tzSmsv0HM4petqqBZ1vsrfTVzUw2-UJ1t3qRzPGQr2wc2QA@mail.gmail.com>
Message-ID: <CAA==FcvRp1Q1E_8U8sdMxu1R76VMXqqTCu-zcpwN8Dy+5hp37Q@mail.gmail.com>

Dear Jed,

I am sorry that I didn't explain my purpose clearly.
The reason I output the matrix is to reformat the matrix since I need to
import the matrix to another program.
Therefore, I need to first output the matrix and reformat it using other
codes.
if I output the matrix using PETSc Binary format, I need to find a method
to import it to that program not using MatLoad.

thanks,
Yujie

On Thu, May 31, 2012 at 1:34 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Thu, May 31, 2012 at 1:33 PM, recrusader <recrusader at gmail.com> wrote:
>
>> Since I need to further process the matrix using my codes. I can easily
>> check the matrix using ASCII format. How to read a matrix in PETSc binary
>> format using c/c++?
>
>
> MatLoad()
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120531/e2e53666/attachment.html>

From bsmith at mcs.anl.gov  Thu May 31 13:42:17 2012
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 31 May 2012 13:42:17 -0500
Subject: [petsc-users] output parallel matrix
In-Reply-To: <CAA==FcvRp1Q1E_8U8sdMxu1R76VMXqqTCu-zcpwN8Dy+5hp37Q@mail.gmail.com>
References: <CAA==FctLhamjQP+ywkQbRqMGq5RXZCrFZZjzdT3wD5jhO9xiJQ@mail.gmail.com>
	<09A3279F-3B26-4506-A8D6-E4B3698588A9@mcs.anl.gov>
	<CAA==Fcu9CY+3_GMEwF6yk7Ee4PrmwxvwvQ8E5ZTpOUi9-6DOAg@mail.gmail.com>
	<CAM9tzSmsv0HM4petqqBZ1vsrfTVzUw2-UJ1t3qRzPGQr2wc2QA@mail.gmail.com>
	<CAA==FcvRp1Q1E_8U8sdMxu1R76VMXqqTCu-zcpwN8Dy+5hp37Q@mail.gmail.com>
Message-ID: <84716724-234C-4C32-89B1-51DD227AC0EE@mcs.anl.gov>


  Right a small standalone sequential PETSc program that does a MatLoad() and then you can print it out anyway you want from that sequential program.

   Barry

On May 31, 2012, at 1:40 PM, recrusader wrote:

> Dear Jed,
> 
> I am sorry that I didn't explain my purpose clearly.
> The reason I output the matrix is to reformat the matrix since I need to import the matrix to another program. 
> Therefore, I need to first output the matrix and reformat it using other codes.
> if I output the matrix using PETSc Binary format, I need to find a method to import it to that program not using MatLoad.
> 
> thanks,
> Yujie
> 
> On Thu, May 31, 2012 at 1:34 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> On Thu, May 31, 2012 at 1:33 PM, recrusader <recrusader at gmail.com> wrote:
> Since I need to further process the matrix using my codes. I can easily check the matrix using ASCII format. How to read a matrix in PETSc binary format using c/c++?
> 
> MatLoad()
> 


From jedbrown at mcs.anl.gov  Thu May 31 13:42:29 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 31 May 2012 13:42:29 -0500
Subject: [petsc-users] output parallel matrix
In-Reply-To: <CAA==FcvRp1Q1E_8U8sdMxu1R76VMXqqTCu-zcpwN8Dy+5hp37Q@mail.gmail.com>
References: <CAA==FctLhamjQP+ywkQbRqMGq5RXZCrFZZjzdT3wD5jhO9xiJQ@mail.gmail.com>
	<09A3279F-3B26-4506-A8D6-E4B3698588A9@mcs.anl.gov>
	<CAA==Fcu9CY+3_GMEwF6yk7Ee4PrmwxvwvQ8E5ZTpOUi9-6DOAg@mail.gmail.com>
	<CAM9tzSmsv0HM4petqqBZ1vsrfTVzUw2-UJ1t3qRzPGQr2wc2QA@mail.gmail.com>
	<CAA==FcvRp1Q1E_8U8sdMxu1R76VMXqqTCu-zcpwN8Dy+5hp37Q@mail.gmail.com>
Message-ID: <CAM9tzSmhQY7g2rLAV6u+Jjg2McMoYkpMEkM6HhqwhKqm2WV8FA@mail.gmail.com>

On Thu, May 31, 2012 at 1:40 PM, recrusader <recrusader at gmail.com> wrote:

> I am sorry that I didn't explain my purpose clearly.
> The reason I output the matrix is to reformat the matrix since I need to
> import the matrix to another program.
> Therefore, I need to first output the matrix and reformat it using other
> codes.
> if I output the matrix using PETSc Binary format, I need to find a method
> to import it to that program not using MatLoad.
>

Why not have the other program use MatLoad()?


This business of writing the matrix to disk as part of your workflow is
going to be a constant source of pain, limiting scalability, making
everything slow, and limiting what methods you can use.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120531/c2935439/attachment.html>

From jbakosi at lanl.gov  Thu May 31 15:08:21 2012
From: jbakosi at lanl.gov (Jozsef Bakosi)
Date: Thu, 31 May 2012 14:08:21 -0600
Subject: [petsc-users] Is PCFactorSetZeroPivot() effective?
In-Reply-To: <CAMYG4Gkw=9xG5xWGW5ijoJXtGN+_XNWpXQN9GiX3Ut+x4ngcow@mail.gmail.com>
References: <20120530232001.GF19605@karman>
	<CAMYG4Gkw=9xG5xWGW5ijoJXtGN+_XNWpXQN9GiX3Ut+x4ngcow@mail.gmail.com>
Message-ID: <20120531200821.GV19605@karman>

Hi Matt,

Thanks for you answer.

On 05.30.2012 19:27, Matthew Knepley wrote:
> On Wed, May 30, 2012 at 7:20 PM, Jozsef Bakosi <jbakosi at lanl.gov> wrote:
> 
> > Hello,
> >
> > I have a mesh with very small cell sizes and in a Poisson equation CG solve
> > with ML preconditioner I get the following error:
> >
> > [31]PETSC ERROR: --------------------- Error Message
> > ------------------------------------
> > [31]PETSC ERROR: Detected zero pivot in LU factorization see
> >
> > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#ZeroPivot
> > !
> > [31]PETSC ERROR: Zero pivot row 537 value 9.41111e-13 tolerance 1e-12!
> >
> > I really would like PETSc to NOT treat matrix values lower than 1e-12 to
> > be zero, so I'm trying to set the zero pivot to a lower value with
> > PCFactorSetZeroPivot(), but I still get the same error message: as if
> > the function call would have no effect. I also tried setting the
> > zeropivot via PetscOptionsSetValue("-pc_factor_zeropivot", ...), but
> > still no luck.
> >
> > Is there another way to set the zero pivot that I'm not aware of?
> >
> 
> 1) Always look at -ksp_view
> 
> 2) I bet this is a subsolve, so that it would be -sub_pc_factor_zeropivot
> <value>. You
>     can verify by looking at -ksp_view
> 

Using -sub_pc_factor_zeropivot does not seem to make any difference.

> 
> > Is there a way to query the zero pivot value?
> >
> 
> 3) Its printed out, 1.0e-12
> 

Yeah. I mean something like PCFactorGetZeroPivot(), so I can check what
PETSc think I'm trying to use. The above persistent error message seems
to indicate that whatever I'm setting is not being used.

BTW, this error seems to happen only at larger core counts. I wonder if
that gives you a clue as to what might be the problem (e.g. what might
cause a zeropivot).

Thanks,
Jozsef

From jedbrown at mcs.anl.gov  Thu May 31 15:12:54 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 31 May 2012 15:12:54 -0500
Subject: [petsc-users] Is PCFactorSetZeroPivot() effective?
In-Reply-To: <20120531200821.GV19605@karman>
References: <20120530232001.GF19605@karman>
	<CAMYG4Gkw=9xG5xWGW5ijoJXtGN+_XNWpXQN9GiX3Ut+x4ngcow@mail.gmail.com>
	<20120531200821.GV19605@karman>
Message-ID: <CAM9tzSna49Juy3fnPC_BfVxfrp7hcFF38t-zP1Ao86OYbt8kXA@mail.gmail.com>

On Thu, May 31, 2012 at 3:08 PM, Jozsef Bakosi <jbakosi at lanl.gov> wrote:

> Using -sub_pc_factor_zeropivot does not seem to make any difference.
>

Run with -ksp_view and send the output. Also send the FULL error message
when it finds a zero pivot.


>
> >
> > > Is there a way to query the zero pivot value?
> > >
> >
> > 3) Its printed out, 1.0e-12
> >
>
> Yeah. I mean something like PCFactorGetZeroPivot(), so I can check what
> PETSc think I'm trying to use. The above persistent error message seems
> to indicate that whatever I'm setting is not being used.
>
> BTW, this error seems to happen only at larger core counts. I wonder if
> that gives you a clue as to what might be the problem (e.g. what might
> cause a zeropivot).
>

What discretization are you using?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120531/520c1a1b/attachment-0001.html>

From jbakosi at lanl.gov  Thu May 31 15:47:51 2012
From: jbakosi at lanl.gov (Jozsef Bakosi)
Date: Thu, 31 May 2012 14:47:51 -0600
Subject: [petsc-users] Is PCFactorSetZeroPivot() effective?
In-Reply-To: <CAM9tzSna49Juy3fnPC_BfVxfrp7hcFF38t-zP1Ao86OYbt8kXA@mail.gmail.com>
References: <20120530232001.GF19605@karman>
	<CAMYG4Gkw=9xG5xWGW5ijoJXtGN+_XNWpXQN9GiX3Ut+x4ngcow@mail.gmail.com>
	<20120531200821.GV19605@karman>
	<CAM9tzSna49Juy3fnPC_BfVxfrp7hcFF38t-zP1Ao86OYbt8kXA@mail.gmail.com>
Message-ID: <20120531204751.GC19605@karman>

On 05.31.2012 15:12, Jed Brown wrote:
> On Thu, May 31, 2012 at 3:08 PM, Jozsef Bakosi <jbakosi at lanl.gov> wrote:
> 
> > Using -sub_pc_factor_zeropivot does not seem to make any difference.
> >
> 
> Run with -ksp_view and send the output. Also send the FULL error message
> when it finds a zero pivot.

I don't seem to find any additional output from -ksp_view. Am I doing
something wrong when I simply pass -ksp_view to the exectuable?

The full file would be 48M, here is a snippet that keeps coming from all
cores:

[44]PETSC ERROR: --------------------- Error Message -----------------------------------
[44]PETSC ERROR: Detected zero pivot in LU factorization see
http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#ZeroPivot!
[44]PETSC ERROR: Zero pivot row 537 value 9.36276e-13 tolerance 1e-12!
[44]PETSC ERROR:
------------------------------------------------------------------------
[44]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48 CDT 2011
[44]PETSC ERROR: See docs/changes/index.html for recent updates.
[44]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[44]PETSC ERROR: See docs/index.html for manual pages.
[44]PETSC ERROR:
------------------------------------------------------------------------
[44]PETSC ERROR: Unknown Name on a conejo-gn named mu0455.localdomain by jbakosi Thu May 31 14:26:36 2012
[44]PETSC ERROR: Libraries linked from /scratch3/jbakosi/code/hydra/tpl/gnu/lib
[44]PETSC ERROR: Configure run at Mon May 21 09:35:16 2012
[44]PETSC ERROR: Configure options PETSC_DIR=/tmp/temp/petsc-3.1-p8 PETSC_ARCH=conejo-gnu-opt --prefix=/scratch3/jbakosi/code/hydra/tpl/gnu --with-clanguage=c++ --with-x=false --with-x11=false --download-ml=/scratch3/jbakosi/code/hydra/tpl/tarballs/ml-6.2.tar.gz --with-debugging=0 -COPTFLAGS=-O3 -CXXOPTFLAGS=-O3 -FOPTFLAGS=-O3 -FFLAGS=-ffree-line-length-none
[44]PETSC ERROR:
------------------------------------------------------------------------
[44]PETSC ERROR: MatLUFactorNumeric_SeqAIJ() line 574 in src/mat/impls/aij/seq/aijfact.c
[44]PETSC ERROR: MatLUFactorNumeric() line 2587 in src/mat/interface/matrix.c
[44]PETSC ERROR: PCSetUp_LU() line 158 in src/ksp/pc/impls/factor/lu/lu.c
[44]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c
[44]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
[44]PETSC ERROR: PCSetUp_Redundant() line 179 in src/ksp/pc/impls/redundant/redundant.c
[44]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c
[44]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
[44]PETSC ERROR: PCSetUp_MG() line 602 in src/ksp/pc/impls/mg/mg.c
[44]PETSC ERROR: PCSetUp_ML() line 668 in src/ksp/pc/impls/ml/ml.c
[44]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c
[44]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
[44]PETSC ERROR: User provided function() line 275 in "unknowndirectory/"/scratch3/jbakosi/code/hydra/src/LinearAlgebra/PetscSolver.C

> What discretization are you using?

Straightforward Galerkin FEM Poisson solver.

Jozsef

From jedbrown at mcs.anl.gov  Thu May 31 15:58:44 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 31 May 2012 15:58:44 -0500
Subject: [petsc-users] Is PCFactorSetZeroPivot() effective?
In-Reply-To: <20120531204751.GC19605@karman>
References: <20120530232001.GF19605@karman>
	<CAMYG4Gkw=9xG5xWGW5ijoJXtGN+_XNWpXQN9GiX3Ut+x4ngcow@mail.gmail.com>
	<20120531200821.GV19605@karman>
	<CAM9tzSna49Juy3fnPC_BfVxfrp7hcFF38t-zP1Ao86OYbt8kXA@mail.gmail.com>
	<20120531204751.GC19605@karman>
Message-ID: <CAM9tzSkc6pxtUzUdo6DAGVJCq33w+gf66q30BJX91vt8m-tP0A@mail.gmail.com>

On Thu, May 31, 2012 at 3:47 PM, Jozsef Bakosi <jbakosi at lanl.gov> wrote:

> I don't seem to find any additional output from -ksp_view. Am I doing
> something wrong when I simply pass -ksp_view to the exectuable?
>
> The full file would be 48M, here is a snippet that keeps coming from all
> cores:
>
> [44]PETSC ERROR: --------------------- Error Message
> -----------------------------------
> [44]PETSC ERROR: Detected zero pivot in LU factorization see
>
> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#ZeroPivot
> !
> [44]PETSC ERROR: Zero pivot row 537 value 9.36276e-13 tolerance 1e-12!
> [44]PETSC ERROR:
> ------------------------------------------------------------------------
> [44]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48
> CDT 2011
> [44]PETSC ERROR: See docs/changes/index.html for recent updates.
> [44]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [44]PETSC ERROR: See docs/index.html for manual pages.
> [44]PETSC ERROR:
> ------------------------------------------------------------------------
> [44]PETSC ERROR: Unknown Name on a conejo-gn named mu0455.localdomain by
> jbakosi Thu May 31 14:26:36 2012
> [44]PETSC ERROR: Libraries linked from
> /scratch3/jbakosi/code/hydra/tpl/gnu/lib
> [44]PETSC ERROR: Configure run at Mon May 21 09:35:16 2012
> [44]PETSC ERROR: Configure options PETSC_DIR=/tmp/temp/petsc-3.1-p8
> PETSC_ARCH=conejo-gnu-opt --prefix=/scratch3/jbakosi/code/hydra/tpl/gnu
> --with-clanguage=c++ --with-x=false --with-x11=false
> --download-ml=/scratch3/jbakosi/code/hydra/tpl/tarballs/ml-6.2.tar.gz
> --with-debugging=0 -COPTFLAGS=-O3 -CXXOPTFLAGS=-O3 -FOPTFLAGS=-O3
> -FFLAGS=-ffree-line-length-none
> [44]PETSC ERROR:
> ------------------------------------------------------------------------
> [44]PETSC ERROR: MatLUFactorNumeric_SeqAIJ() line 574 in
> src/mat/impls/aij/seq/aijfact.c
> [44]PETSC ERROR: MatLUFactorNumeric() line 2587 in
> src/mat/interface/matrix.c
> [44]PETSC ERROR: PCSetUp_LU() line 158 in src/ksp/pc/impls/factor/lu/lu.c
> [44]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c
> [44]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
> [44]PETSC ERROR: PCSetUp_Redundant() line 179 in
> src/ksp/pc/impls/redundant/redundant.c
> [44]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c
> [44]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
> [44]PETSC ERROR: PCSetUp_MG() line 602 in src/ksp/pc/impls/mg/mg.c
> [44]PETSC ERROR: PCSetUp_ML() line 668 in src/ksp/pc/impls/ml/ml.c
> [44]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c
> [44]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
> [44]PETSC ERROR: User provided function() line 275 in
> "unknowndirectory/"/scratch3/jbakosi/code/hydra/src/LinearAlgebra/PetscSolver.C
>

These are the options you can use to control the shift (assuming you
haven't added your own prefix; you can check for yourself with -help)

 -mg_coarse_redundant_pc_factor_shift_type <NONE> (choose one of) NONE
NONZERO POSITIVE_DEFINITE INBLOCKS (PCFactorSetShiftType)
  -mg_coarse_redundant_pc_factor_shift_amount <0>: Shift added to diagonal
(PCFactorSetShiftAmount)
  -mg_coarse_redundant_pc_factor_zeropivot <2.22045e-14>: Pivot is
considered zero if less than (PCFactorSetZeroPivot)

Due to peculiar setup dependencies, it's going to be quite difficult to
extract the coarse problem produced by ML in time to catch it before the
factorization is attempted.

Matt, this is another example of a setup dependency that I think would be
awkward to intercept as part of a "setup phases" approach.


>
> > What discretization are you using?
>
> Straightforward Galerkin FEM Poisson solver.
>

What boundary conditions? I'm guessing the constant is in the null space,
in which case the coarse space really should be singular and you should not
be trying to solve it with a direct solver.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120531/1c26dc87/attachment.html>

From jbakosi at lanl.gov  Thu May 31 17:26:43 2012
From: jbakosi at lanl.gov (Jozsef Bakosi)
Date: Thu, 31 May 2012 16:26:43 -0600
Subject: [petsc-users] Is PCFactorSetZeroPivot() effective?
In-Reply-To: <CAM9tzSkc6pxtUzUdo6DAGVJCq33w+gf66q30BJX91vt8m-tP0A@mail.gmail.com>
References: <20120530232001.GF19605@karman>
	<CAMYG4Gkw=9xG5xWGW5ijoJXtGN+_XNWpXQN9GiX3Ut+x4ngcow@mail.gmail.com>
	<20120531200821.GV19605@karman>
	<CAM9tzSna49Juy3fnPC_BfVxfrp7hcFF38t-zP1Ao86OYbt8kXA@mail.gmail.com>
	<20120531204751.GC19605@karman>
	<CAM9tzSkc6pxtUzUdo6DAGVJCq33w+gf66q30BJX91vt8m-tP0A@mail.gmail.com>
Message-ID: <20120531222643.GI19605@karman>

On 05.31.2012 15:58, Jed Brown wrote:
> On Thu, May 31, 2012 at 3:47 PM, Jozsef Bakosi <jbakosi at lanl.gov> wrote:
> 
> > I don't seem to find any additional output from -ksp_view. Am I doing
> > something wrong when I simply pass -ksp_view to the exectuable?
> >
> > The full file would be 48M, here is a snippet that keeps coming from all
> > cores:
> >
> > [44]PETSC ERROR: --------------------- Error Message
> > -----------------------------------
> > [44]PETSC ERROR: Detected zero pivot in LU factorization see
> >
> > http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#ZeroPivot
> > !
> > [44]PETSC ERROR: Zero pivot row 537 value 9.36276e-13 tolerance 1e-12!
> > [44]PETSC ERROR:
> > ------------------------------------------------------------------------
> > [44]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17 13:37:48
> > CDT 2011
> > [44]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [44]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [44]PETSC ERROR: See docs/index.html for manual pages.
> > [44]PETSC ERROR:
> > ------------------------------------------------------------------------
> > [44]PETSC ERROR: Unknown Name on a conejo-gn named mu0455.localdomain by
> > jbakosi Thu May 31 14:26:36 2012
> > [44]PETSC ERROR: Libraries linked from
> > /scratch3/jbakosi/code/hydra/tpl/gnu/lib
> > [44]PETSC ERROR: Configure run at Mon May 21 09:35:16 2012
> > [44]PETSC ERROR: Configure options PETSC_DIR=/tmp/temp/petsc-3.1-p8
> > PETSC_ARCH=conejo-gnu-opt --prefix=/scratch3/jbakosi/code/hydra/tpl/gnu
> > --with-clanguage=c++ --with-x=false --with-x11=false
> > --download-ml=/scratch3/jbakosi/code/hydra/tpl/tarballs/ml-6.2.tar.gz
> > --with-debugging=0 -COPTFLAGS=-O3 -CXXOPTFLAGS=-O3 -FOPTFLAGS=-O3
> > -FFLAGS=-ffree-line-length-none
> > [44]PETSC ERROR:
> > ------------------------------------------------------------------------
> > [44]PETSC ERROR: MatLUFactorNumeric_SeqAIJ() line 574 in
> > src/mat/impls/aij/seq/aijfact.c
> > [44]PETSC ERROR: MatLUFactorNumeric() line 2587 in
> > src/mat/interface/matrix.c
> > [44]PETSC ERROR: PCSetUp_LU() line 158 in src/ksp/pc/impls/factor/lu/lu.c
> > [44]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c
> > [44]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
> > [44]PETSC ERROR: PCSetUp_Redundant() line 179 in
> > src/ksp/pc/impls/redundant/redundant.c
> > [44]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c
> > [44]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
> > [44]PETSC ERROR: PCSetUp_MG() line 602 in src/ksp/pc/impls/mg/mg.c
> > [44]PETSC ERROR: PCSetUp_ML() line 668 in src/ksp/pc/impls/ml/ml.c
> > [44]PETSC ERROR: PCSetUp() line 795 in src/ksp/pc/interface/precon.c
> > [44]PETSC ERROR: KSPSetUp() line 237 in src/ksp/ksp/interface/itfunc.c
> > [44]PETSC ERROR: User provided function() line 275 in
> > "unknowndirectory/"/scratch3/jbakosi/code/hydra/src/LinearAlgebra/PetscSolver.C
> >
> 
> These are the options you can use to control the shift (assuming you
> haven't added your own prefix; you can check for yourself with -help)
> 
>  -mg_coarse_redundant_pc_factor_shift_type <NONE> (choose one of) NONE
> NONZERO POSITIVE_DEFINITE INBLOCKS (PCFactorSetShiftType)
>   -mg_coarse_redundant_pc_factor_shift_amount <0>: Shift added to diagonal
> (PCFactorSetShiftAmount)
>   -mg_coarse_redundant_pc_factor_zeropivot <2.22045e-14>: Pivot is
> considered zero if less than (PCFactorSetZeroPivot)
> 

Great. Thanks. When I pass these options on the command line, how can I
get a feedback to see if the option has actually been used or not?

> > > What discretization are you using?
> >
> > Straightforward Galerkin FEM Poisson solver.
> >
> 
> What boundary conditions? I'm guessing the constant is in the null space,
> in which case the coarse space really should be singular and you should not
> be trying to solve it with a direct solver.

I do set the additive constant. I have no problem with this on lower core-counts.

I did another test and set the PETSC_OPTIONS environment variable to
'-options_table -options_left', then I get:

...

[49]PETSC ERROR: Zero pivot row 537 value 9.36276e-13 tolerance 1e-12!

...

#PETSc Option Table entries:
-mg_levels_ksp_norm_type no
-mg_levels_ksp_richardson_scale 0.90
-mg_levels_ksp_type richardson
-mg_levels_pc_type bjacobi
-mg_levels_sub_ksp_norm_type no
-mg_levels_sub_ksp_type preonly
-mg_levels_sub_pc_type icc
-options_left
-options_table
-pc_asm_overlap 1
-pc_mg_cycle_type v
-pc_mg_smoothdown 1
-pc_mg_smoothup 1
-pc_ml_maxCoarseSize 1500
-pc_ml_maxNlevels 10
-pc_type asm
-sub_pc_factor_levels 0
-sub_pc_factor_zeropivot 1e-20
-sub_pc_type ilu
#End of PETSc Option Table entries
There are no unused options.

So how do I tell, which zeropivot setting (the 1e-20 or the 1e-12) is the one petsc is using?

Thanks,
J

From jedbrown at mcs.anl.gov  Thu May 31 18:05:22 2012
From: jedbrown at mcs.anl.gov (Jed Brown)
Date: Thu, 31 May 2012 18:05:22 -0500
Subject: [petsc-users] Is PCFactorSetZeroPivot() effective?
In-Reply-To: <20120531222643.GI19605@karman>
References: <20120530232001.GF19605@karman>
	<CAMYG4Gkw=9xG5xWGW5ijoJXtGN+_XNWpXQN9GiX3Ut+x4ngcow@mail.gmail.com>
	<20120531200821.GV19605@karman>
	<CAM9tzSna49Juy3fnPC_BfVxfrp7hcFF38t-zP1Ao86OYbt8kXA@mail.gmail.com>
	<20120531204751.GC19605@karman>
	<CAM9tzSkc6pxtUzUdo6DAGVJCq33w+gf66q30BJX91vt8m-tP0A@mail.gmail.com>
	<20120531222643.GI19605@karman>
Message-ID: <CAM9tzSmB8suAGqTJqTopJiMFfw61dnXtHYLXOCyUWAQywVd_ag@mail.gmail.com>

On Thu, May 31, 2012 at 5:26 PM, Jozsef Bakosi <jbakosi at lanl.gov> wrote:

> Great. Thanks. When I pass these options on the command line, how can I
> get a feedback to see if the option has actually been used or not?
>

-options_left shows whether the option was queried

-ksp_view will have a line "tolerance for zero pivot" telling you what was
actually used


> > > > What discretization are you using?
> > >
> > > Straightforward Galerkin FEM Poisson solver.
> > >
> >
> > What boundary conditions? I'm guessing the constant is in the null space,
> > in which case the coarse space really should be singular and you should
> not
> > be trying to solve it with a direct solver.
>
> I do set the additive constant. I have no problem with this on lower
> core-counts.
>

What do you mean by "set the additive constant"? How do you enforce it?
Best option is usually to tell the KSP about the null space and use an
inexact coarse level solver. If you use ML with default options, the coarse
levels are generally small enough that you can brute-force it by using
-mg_coarse_redundant_pc_type svd. If you need big coarse spaces, you'll
probably need to use an inexact coarse level solver (possibly use a shift,
as in -pc_coarse_redundant_pc_factor_shift_type nonzero.


>
> I did another test and set the PETSC_OPTIONS environment variable to
> '-options_table -options_left', then I get:
>
> ...
>
> [49]PETSC ERROR: Zero pivot row 537 value 9.36276e-13 tolerance 1e-12!
>

This is an error. How do you get to the output shown below?


>
> ...
>
> #PETSc Option Table entries:
> -mg_levels_ksp_norm_type no
> -mg_levels_ksp_richardson_scale 0.90
> -mg_levels_ksp_type richardson
> -mg_levels_pc_type bjacobi
> -mg_levels_sub_ksp_norm_type no
> -mg_levels_sub_ksp_type preonly
> -mg_levels_sub_pc_type icc
> -options_left
> -options_table
> -pc_asm_overlap 1
> -pc_mg_cycle_type v
> -pc_mg_smoothdown 1
> -pc_mg_smoothup 1
> -pc_ml_maxCoarseSize 1500
> -pc_ml_maxNlevels 10
> -pc_type asm
> -sub_pc_factor_levels 0
> -sub_pc_factor_zeropivot 1e-20
> -sub_pc_type ilu
> #End of PETSc Option Table entries
> There are no unused options.
>
> So how do I tell, which zeropivot setting (the 1e-20 or the 1e-12) is the
> one petsc is using?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120531/dec7c461/attachment.html>