Multiple communicators

[Barry Smith]

On Dec 3, 2009, at 12:59 PM, irfan.khan at gatech.edu wrote:

> Thank you for the response Barry. You are right, the command line  
> options do work.

     Good.

     I'm sorry that we don't have a good method of handling the
1) printing of unused options and
2) printing of global log summaries
when PETSc is only used on a subset of communicators.

This processing is done over PETSC_COMM_WORLD and I cannot wrap my  
head around any way of properly handling these types of things cleanly  
and transparently when using subcommunicators.

    Barry


>
>
>
> ----- Original Message -----
> From: "Barry Smith" <bsmith at mcs.anl.gov>
> To: "PETSc users list" <petsc-users at mcs.anl.gov>
> Sent: Tuesday, November 24, 2009 11:50:02 PM GMT -05:00 US/Canada  
> Eastern
> Subject: Re: Multiple communicators
>
>
> On Nov 24, 2009, at 7:13 PM, irfan.khan at gatech.edu wrote:
>
>> I use seperete communicators for fluid and solid ranks in a fluid-
>> structure interaction code. I use PETSc tools (KSP) to solve for the
>> solid phase (FEA) which is carried out under a different
>> communicator (FEA_Comm).
>>
>> Unless I am doing something wrong, I have found that the command
>> line options: -ksp_type; -mat_partitioning_type, don't work if there
>> are 2 communicators.
>
>    Are you sure that they don't work? Or is it just printing the
> message?
>
> WARNING! There are options you set that were not used!
> WARNING! could be spelling mistake, etc!
> Option left: name:-ksp_type value: cg
> Option left: name:-mat_partitioning_type value: parmetis
>
>    Any option that is not accessed on PROCESS ZERO will be listed
> here as unused, even if it may be used on some process.  I suppose we
> could/should fix this by adding some complicated communication that
> determines just the options that are never used. I never liked this
> warning, only added it after pressure from people who couldn't type.
>
>> But for single communicator they work. Please see the attached files
>> containing output of -log_summary for 2 different codes (scroll down
>> to the end). One with single communicator and the other with two
>> communicators. Both the output have been generated with the same
>> PETSc compilation.
>>
>> Futher -log_summary results are not able to detect all the events
>> that have been registered if they are not in the WORLD communicator.
>
>    Is your concern here that it does not print the name of the event,
> it leaves the name blank? If so this is likely the type of problem as
> the print of options; we don't have a mechanism to get them to process
> 0 to print.
>
>    In summary, yes these are some weaknesses of PETSc. If you switch
> the two communicators you create and use the one that has process 0 of
> MPI_COMM_WORLD for the PETSc stuff then it will look nicer and hide
> these two problems.
>
>    Barry
>
>>
>>
>> Thank you
>> Irfan
>> ----- Original Message -----
>> From: "Matthew Knepley" <knepley at gmail.com>
>> To: "PETSc users list" <petsc-users at mcs.anl.gov>
>> Sent: Tuesday, November 24, 2009 8:03:00 PM GMT -05:00 US/Canada
>> Eastern
>> Subject: Re: Multiple communicators
>>
>> On Tue, Nov 24, 2009 at 6:59 PM, <irfan.khan at gatech.edu> wrote:
>> Hello
>>
>> Does the procedure for providing command line options and to obtain
>> profiling data through -log_summary change if there are multiple MPI
>> communicators in a petsc code?
>>
>> There is always WORLD, so it should be fine unless I misunderstand
>> your question.
>>
>>  Matt
>>
>> If so, can somebody point me to information, references on how to
>> provide command line options if there are multiple communicators?
>>
>> Thank you
>> Irfan
>>
>> -- 
>> PhD Candidate
>> G.W. Woodruff School of Mechanical Engineering
>> Georgia Institute of Technology
>> Atlanta, GA (30332)
>>
>>
>>
>> -- 
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which
>> their experiments lead.
>> -- Norbert Wiener
>>
>>
>> -- 
>> PhD Candidate
>> G.W. Woodruff School of Mechanical Engineering
>> Georgia Institute of Technology
>> Atlanta, GA (30332)
>> <1communicator.out><2communicators.out>
>
>
> -- 
> PhD Candidate
> G.W. Woodruff School of Mechanical Engineering
> Georgia Institute of Technology
> Atlanta, GA (30332)