[Parvis-users] Performance of the parncl.
Jayesh Krishna
jayesh at mcs.anl.gov
Tue Apr 16 11:06:47 CDT 2013
Hi,
I just tried running the script on my laptop (Ubuntu 12.04, x86_64, mpich2 1.5, ParNCL 1.0.0b2 + 128x256x26 24-month dataset) and was able to run the script successfully.
How did you build ParNCL? Have you tried 1.0.0b2?
If your system is Ubuntu, you can try out the ParNCL binaries (http://trac.mcs.anl.gov/projects/parvis/wiki/ParNCLBinaries) and see if it works. You will need to install gcc and mpich2 1.5 before trying out the binaries.
Regards,
Jayesh
----- Original Message -----
From: "Matrix" <sov_matrix at 126.com>
To: "Jayesh Krishna" <jayesh at mcs.anl.gov>
Sent: Tuesday, April 16, 2013 7:57:55 AM
Subject: Re:Re: [Parvis-users] Performance of the parncl.
Hi,
We haven't tried a higher resolution yet, but I will try it later.
I measured the timing by a script :
START=`date +%s%N`;
sync; echo 3 > /proc/sys/vm/drop_caches
mpiexec -n $1 parncl ptest_3.ncl
END=`date +%s%N`;
time=$((END-START))
time=`expr $time / 1000000`
echo proces $1 use $time ms >> time_history
And recently I met with a new problem when I use the ncl script plot_vort.ncl in the website(http://trac.mcs.anl.gov/projects/parvis/wiki/ParNCLDeveloperInfo/SimpleParNCLScripts). There comes out a result in a short time when I use NCL or ParNCL with only one processor. But with two or more processor , it can not get a result no matter how long the time it uses.(We tested both the nc file in the website and the cam package) I considered that some deadlock happened .I was looking into it ,reading the source code, But I still can't u nderstand why the deadlock happens .
Here is the information when I use the pstack to the processes , so we can locate where the deadlock occurs:
#0 0x000000317c80de4d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000317c809ca6 in _L_lock_836 () from /lib64/libpthread.so.0
#2 0x000000317c809ba8 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000018ac77d in PMPI_Send ()
#4 0x00000000012bf511 in Epetra_MpiDistributor::ComputeRecvs_(int, int) ()
#5 0x00000000012bfa25 in Epetra_MpiDistributor::CreateFromSends(int const&, int const*, bool, int&) ()
#6 0x00000000012c06e1 in int Epetra_MpiDistributor::ComputeSends_<int>(int, int const*&, int const*&, int&, int*&, int*&, int) ()
#7 0x00000000012bfe7e in Epetra_MpiDistributor::CreateFromRecvs(int const&, int const*, int const*, bool, int&, int*&, int*&) ()
#8 0x000000000 12794a5 in void Epetra_Import::Construct<int>(Epetra_BlockMap const&, Epetra_BlockMap const&) ()
#9 0x0000000001277a6e in Epetra_Import::Epetra_Import(Epetra_BlockMap const&, Epetra_BlockMap const&) ()
#10 0x000000000123f274 in Epetra_CrsGraphData::MakeImportExport() ()
#11 0x0000000001235a13 in Epetra_CrsGraph::FillComplete(Epetra_BlockMap const&, Epetra_BlockMap const&) ()
#12 0x000000000124b66f in Epetra_CrsMatrix::FillComplete(Epetra_Map const&, Epetra_Map const&, bool) ()
#13 0x000000000126f154 in int Epetra_FECrsMatrix::GlobalAssemble<int>(Epetra_Map const&, Epetra_Map const&, bool, Epetra_CombineMode, bool) ()
#14 0x000000000126c088 in Epetra_FECrsMatrix::GlobalAssemble(Epetra_Map const&, Epetra_Map const&, bool, Epetra_CombineMode, bool) ()
#15 0x0000000000f6ebc5 in pargal::pcalgo::vort_div_cell_center(pargal::pcvar&, pargal::pcvar&, pargal::pcvar&, bool) ()
#16 0x00 00000000f713e4 in pargal::pcalgo::vorticity(pargal::pcvar&, pargal::pcvar&, pargal::pcvar&) ()
#17 0x0000000000fac767 in pargal::vorticity(pargal::pcvar&, pargal::pcvar&, pargal::pcvar&) ()
#18 0x000000000084805e in parncl::ParCALTwoMDIOp(void (*)(pargal::pcvar&, pargal::pcvar&, pargal::pcvar&), void*, void*) ()
#19 0x000000000083e67e in ParCALUVToVort ()
#20 0x000000000091799d in ParCALFuncTempl_W ()
#21 0x0000000000917ae5 in puv2vrA_W ()
#22 0x0000000000856324 in CallINTRINSIC_FUNC_CALL ()
#23 0x000000000086115a in _NclExecute ()
#24 0x0000000000778c2f in yyparse ()
#25 0x0000000000775300 in _NclParse ()
#26 0x0000000000774435 in main ()
###########
#0 0x000000317c80de4d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000317c809ca6 in _L_lock_836 () from /lib64/libpthread.so.0
#2 0x000000317c809ba8 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3& nbsp; 0x00000000018ab222 in PMPI_Recv ()
#4 0x00000000012bf5c1 in Epetra_MpiDistributor::ComputeRecvs_(int, int) ()
#5 0x00000000012bfa25 in Epetra_MpiDistributor::CreateFromSends(int const&, int const*, bool, int&) ()
#6 0x00000000012c06e1 in int Epetra_MpiDistributor::ComputeSends_<int>(int, int const*&, int const*&, int&, int*&, int*&, int) ()
#7 0x00000000012bfe7e in Epetra_MpiDistributor::CreateFromRecvs(int const&, int const*, int const*, bool, int&, int*&, int*&) ()
#8 0x00000000012794a5 in void Epetra_Import::Construct<int>(Epetra_BlockMap const&, Epetra_BlockMap const&) ()
#9 0x0000000001277a6e in Epetra_Import::Epetra_Import(Epetra_BlockMap const&, Epetra_BlockMap const&) ()
#10 0x000000000123f274 in Epetra_CrsGraphData::MakeImportExport() ()
#11 0x0000000001235a13 in Epetra_CrsGraph::FillComplete(Epetra_BlockMap const&, Epetra_Blo ckMap const&) ()
#12 0x000000000124b66f in Epetra_CrsMatrix::FillComplete(Epetra_Map const&, Epetra_Map const&, bool) ()
#13 0x000000000126f154 in int Epetra_FECrsMatrix::GlobalAssemble<int>(Epetra_Map const&, Epetra_Map const&, bool, Epetra_CombineMode, bool) ()
#14 0x000000000126c088 in Epetra_FECrsMatrix::GlobalAssemble(Epetra_Map const&, Epetra_Map const&, bool, Epetra_CombineMode, bool) ()
#15 0x0000000000f6ebc5 in pargal::pcalgo::vort_div_cell_center(pargal::pcvar&, pargal::pcvar&, pargal::pcvar&, bool) ()
#16 0x0000000000f713e4 in pargal::pcalgo::vorticity(pargal::pcvar&, pargal::pcvar&, pargal::pcvar&) ()
#17 0x0000000000fac767 in pargal::vorticity(pargal::pcvar&, pargal::pcvar&, pargal::pcvar&) ()
#18 0x000000000084805e in parncl::ParCALTwoMDIOp(void (*)(pargal::pcvar&, pargal::pcvar&, pargal::pcvar&), void*, void*) ()
#19 0x000000000083e67e in ParCALUVToVort ()
#20 0x000000000091799d in ParCALFuncTempl_W ()
#21 0x0000000000917ae5 in puv2vrA_W ()
#22 0x0000000000856324 in CallINTRINSIC_FUNC_CALL ()
#23 0x000000000086115a in _NclExecute ()
#24 0x0000000000778c2f in yyparse ()
#25 0x0000000000775300 in _NclParse ()
#26 0x0000000000774435 in main ()
I am looking forward to your opinion about it, thank you so much.
Xu Ji.
Tsinghua University.
At 2013-04-16 04:50:04,"Jayesh Krishna" <jayesh at mcs.anl.gov> wrote:
>Hi,
> The timing for the 1 process case looks similar to what I get here (I tried a 128x256x26 24-month dataset) without a parallel file system. However for the 2/3/4/... process cases how are you measuring the timings (Are you measuring real time?) ?
> Have you tried a higher resolution dataset to see if the timings are better (scalability etc)?
> Most of the time in the script is spent reading meta-data and creating the mesh. I will update the ticket mentioned below with my findings and see if we can improve the timings (You should get an email when the ticket is updated).
>
>Regards,
>Jayesh
>
>----- Original Message -----
>From: "Matrix" <sov_matrix at 126.com>
>To: "Jayesh Krishna" <jayesh at mcs.anl.gov>
>Sent: Monday, April 15, 2013 10:00:56 AM
>Subject: Re:Re: [Parvis-users] Performance of the parncl.
>
>
>
>Hi,here is the head of my nc file, thank you very much.
>
>netcdf TH240_CAM-th.cam2.h0.0048-07 {
>dimensions:
>time = UNLIMITED ; // (1 currently)
>lat = 64 ;
>lon = 128 ;
>lev = 26 ;
>ilev = 27 ;
>isccp_prs = 7 ;
>isccp_prstau = 49 ;
>isccp_tau = 7 ;
>variables:
>float CLDHGH(time, lat, lon) ;
>CLDHGH:units = "fraction" ;
>CLDHGH:long_name = "Vertically-integrated high cloud" ;
>CLDHGH:cell_method = "time: mean" ;
>float CLDICE(time, lev, lat, lon) ;
>CLD ICE:units = "kg/kg" ;
>CLDICE:long_name = "Grid box averaged ice condensate amount" ;
>CLDICE:cell_method = "time: mean" ;
>float CLDLIQ(time, lev, lat, lon) ;
>CLDLIQ:units = "kg/kg" ;
>CLDLIQ:long_name = "Grid box averaged liquid condensate amount" ;
>CLDLIQ:cell_method = "time: mean" ;
>float CLDLOW(time, lat, lon) ;
>CLDLOW:units = "fraction" ;
>CLDLOW:long_name = "Vertically-integrated low cloud" ;
>CLDLOW:cell_method = "time: mean" ;
>float CLDMED(time, lat, lon) ;
>CLDMED:units = "fraction" ;
>CLDMED:long_name = " Vertically-integrated mid-level cloud" ;
>CLDMED:cell_method = "time: mean" ;
>float CLDTOT(time, lat, lon) ;
>CLDTOT:units = "fraction" ;
>CLDTOT:long_name = "Vertically-integrated total cloud" ;
>CLDTOT:cell_method = "time: mean" ;
>float CLOUD(time, lev, lat, lon) ;
>CLOUD:units = "fraction" ;
>CLOUD:long_name = "Cloud fraction" ;
>CLOUD:cell_method = "time: mean" ;
>float CMFDQ(time, lev, lat, lon) ;
>CMFDQ:units = "kg/kg/s" ;
>CMFDQ:long_name = "Q tendency - Hack convection" ;
>CMFDQ:cell_method = "time: mean" ;
>float CMFDQR(time, lev, lat, lon) ;
>CMFDQR:units = "kg/kg/s" ;
>CMFDQR:long_name = "Q tendency - shallow convection rainout" ;
>CMFDQR:cell_method = "time: mean" ;
>float CMFDT(time, lev, lat, lon) ;
>CMFDT:units = "K/s" ;
>CMFDT:long_name = "T tendency - Hack convection" ;
>CMFDT:cell_method = "time: mean" ;
>float CMFMC(time, ilev, lat, lon) ;
>CMFMC:units = "kg/m2/s" ;
>CMFMC:long_name = "Moist convection mass flux" ;
>CMFMC:cell_method = "time: mean" ;
>float CONCLD(time, lev, lat, lon) ;
>CONCLD: units = "fraction" ;
>CONCLD:long_name = "Convective cloud cover" ;
>CONCLD:cell_method = "time: mean" ;
>float CPLPRCER(time, lat, lon) ;
>CPLPRCER:units = "kg/m2/s" ;
>CPLPRCER:long_name = "Error in precipitation state (rain or snow) sent to coupler" ;
>CPLPRCER:cell_method = "time: mean" ;
>float CPLRAINC(time, lat, lon) ;
>CPLRAINC:units = "kg/m2/s" ;
>CPLRAINC:long_name = "Convective rainfall sent to coupler" ;
>CPLRAINC:cell_method = "time: mean" ;
>float CPLRAINL(time, lat, lon) ;
>CPLRAINL:units = "kg/m2/s" ;
>CP LRAINL:long_name = "Large-scale rainfall sent to coupler" ;
>CPLRAINL:cell_method = "time: mean" ;
>float CPLSNOWC(time, lat, lon) ;
>CPLSNOWC:units = "kg/m2/s" ;
>CPLSNOWC:long_name = "Convective snowfall sent to coupler" ;
>CPLSNOWC:cell_method = "time: mean" ;
>float CPLSNOWL(time, lat, lon) ;
>CPLSNOWL:units = "kg/m2/s" ;
>CPLSNOWL:long_name = "Large-scale snowfall sent to coupler" ;
>CPLSNOWL:cell_method = "time: mean" ;
>float DCQ(time, lev, lat, lon) ;
>DCQ:units = "kg/kg/s" ;
>DCQ:long_name = "Q tendency due to moist processes" ;
>DCQ:cell_method = "time: mean" ;
>float DTCOND(time, lev, lat, lon) ;
>DTCOND:units = "K/s" ;
>DTCOND:long_name = "T tendency - moist processes" ;
>DTCOND:cell_method = "time: mean" ;
>float DTH(time, lev, lat, lon) ;
>DTH:units = "K/s" ;
>DTH:long_name = "T horizontal diffusive heating" ;
>DTH:cell_method = "time: mean" ;
>float DTV(time, lev, lat, lon) ;
>DTV:units = "K/s" ;
>DTV:long_name = "T vertical diffusion" ;
>DTV:cell_method = "time: mean" ;
>float FICE(time, lev, lat, lon) ;
>& nbsp; FICE:units = "fraction" ;
>FICE:long_name = "Fractional ice content within cloud" ;
>FICE:cell_method = "time: mean" ;
>float FLNS(time, lat, lon) ;
>FLNS:units = "W/m2" ;
>FLNS:long_name = "Net longwave flux at surface" ;
>FLNS:cell_method = "time: mean" ;
>float FLNSC(time, lat, lon) ;
>FLNSC:units = "W/m2" ;
>FLNSC:long_name = "Clearsky net longwave flux at surface" ;
>FLNSC:cell_method = "time: mean" ;
>float FLNSOI(time, lat, lon) ;
>FLNSOI:units = "W/m2" ;
>FLNSOI:long_name = "FLNS over open ocn and ice" ;
>FLNSOI:cell_method = "time: mean" ;
>float FLNT(time, lat, lon) ;
>FLNT:units = "W/m2" ;
>FLNT:long_name = "Net longwave flux at top of model" ;
>FLNT:cell_method = "time: mean" ;
>float FLNTC(time, lat, lon) ;
>FLNTC:units = "W/m2" ;
>FLNTC:long_name = "Clearsky net longwave flux at top of model" ;
>FLNTC:cell_method = "time: mean" ;
>float FLUT(time, lat, lon) ;
>FLUT:units = "W/m2" ;
>FLUT:long_name = "Upwelling longwave flux at top of model" ;
>FLUT:cell_method = "time: mean" ;
>float FLUTC(time, lat, lon) ;
>FLUTC:units = "W/m2" ;
>FLUTC:long_name = "Clearsky upwelling longwave flux at top of model" ;
>FLUTC:cell_method = "time: mean" ;
>float FSDS(time, lat, lon) ;
>FSDS:units = "W/m2" ;
>FSDS:long_name = "Downwelling solar flux at surface" ;
>FSDS:cell_method = "time: mean" ;
>float FSDSC(time, lat, lon) ;
>FSDSC:units = "W/m2" ;
>FSDSC:long_name = "Clearsky downwelling solar flux at surface" ;
>FSDSC:cell_method = "time: mean" ;
>float FSNS(time, lat, lon) ;
>FSNS:units = "W/m2" ;
>FSNS:long_name = "Net solar flux at surface" ;
>FSNS:cell_method = "time: mean" ;
>float FSNSC(time, lat, lon) ;
>FSNSC:units = "W/m2" ;
>FSNSC:long_name = "Clearsky net solar flux at surface" ;
>FSNSC:cell_method = "time: mean" ;
>float FSNSOI(time, lat, lon) ;
>FSNSOI:units = "W/m2" ;
>FSNSOI:long_name = "FSNS over open ocn and ice" ;
>FSNSOI:cell_method = "time: mean" ;
>float FSNT(time, lat, lon) ;
>FSNT:units = "W/m2" ;
>FSNT:long_name = "Net solar flux at top of model" ;
>FSNT:cell_method = "time: mean " ;
>float FSNTC(time, lat, lon) ;
>FSNTC:units = "W/m2" ;
>FSNTC:long_name = "Clearsky net solar flux at top of model" ;
>FSNTC:cell_method = "time: mean" ;
>float FSNTOA(time, lat, lon) ;
>FSNTOA:units = "W/m2" ;
>FSNTOA:long_name = "Net solar flux at top of atmosphere" ;
>FSNTOA:cell_method = "time: mean" ;
>float FSNTOAC(time, lat, lon) ;
>FSNTOAC:units = "W/m2" ;
>FSNTOAC:long_name = "Clearsky net solar flux at top of atmosphere" ;
>FSNTOAC:cell_method = "time: mean" ;
>float GCLDLWP(time, lev, lat, lon) ;
>GCLDLWP:units = "gram/m2" ;
>GCLDLWP:long_name = "Grid-box cloud water path" ;
>GCLDLWP:cell_method = "time: mean" ;
>float ICEFRAC(time, lat, lon) ;
>ICEFRAC:units = "fraction" ;
>ICEFRAC:long_name = "Fraction of sfc area covered by sea-ice" ;
>ICEFRAC:cell_method = "time: mean" ;
>float ICLDIWP(time, lev, lat, lon) ;
>ICLDIWP:units = "gram/m2" ;
>ICLDIWP:long_name = "In-cloud ice water path" ;
>ICLDIWP:cell_method = "time: mean" ;
>float ICLDLWP(time, lev, lat, lon) ;
>ICLDLWP:units = "gram/m2" ;
>ICLDLWP: long_name = "In-cloud cloud water path (liquid and ice)" ;
>ICLDLWP:cell_method = "time: mean" ;
>float LANDFRAC(time, lat, lon) ;
>LANDFRAC:units = "fraction" ;
>LANDFRAC:long_name = "Fraction of sfc area covered by land" ;
>LANDFRAC:cell_method = "time: mean" ;
>float LHFLX(time, lat, lon) ;
>LHFLX:units = "W/m2" ;
>LHFLX:long_name = "Surface latent heat flux" ;
>LHFLX:cell_method = "time: mean" ;
>float LHFLXOI(time, lat, lon) ;
>LHFLXOI:units = "W/m2" ;
>LHFLXOI:long_name = "LHFLX over open ocn and ice" ;
>LHFLXOI:ce ll_method = "time: mean" ;
>float LWCF(time, lat, lon) ;
>LWCF:units = "W/m2" ;
>LWCF:long_name = "Longwave cloud forcing" ;
>LWCF:cell_method = "time: mean" ;
>float OCNFRAC(time, lat, lon) ;
>OCNFRAC:units = "fraction" ;
>OCNFRAC:long_name = "Fraction of sfc area covered by ocean" ;
>OCNFRAC:cell_method = "time: mean" ;
>float OMEGA(time, lev, lat, lon) ;
>OMEGA:units = "Pa/s" ;
>OMEGA:long_name = "Vertical velocity (pressure)" ;
>OMEGA:cell_method = "time: mean" ;
>float OMEGAT(time, lev, lat, lon) ;
>&n bsp; OMEGAT:units = "K Pa/s" ;
>OMEGAT:long_name = "Vertical heat flux" ;
>OMEGAT:cell_method = "time: mean" ;
>float PBLH(time, lat, lon) ;
>PBLH:units = "m" ;
>PBLH:long_name = "PBL height" ;
>PBLH:cell_method = "time: mean" ;
>float PDELDRY(time, lev, lat, lon) ;
>PDELDRY:units = "Pa" ;
>PDELDRY:long_name = "Dry pressure difference between levels" ;
>PDELDRY:cell_method = "time: mean" ;
>float PHIS(time, lat, lon) ;
>PHIS:units = "m2/s2" ;
>PHIS:long_name = "Surface geopotential" ;
>float PRECC(ti me, lat, lon) ;
>PRECC:units = "m/s" ;
>PRECC:long_name = "Convective precipitation rate" ;
>PRECC:cell_method = "time: mean" ;
>float PRECL(time, lat, lon) ;
>PRECL:units = "m/s" ;
>PRECL:long_name = "Large-scale (stable) precipitation rate" ;
>PRECL:cell_method = "time: mean" ;
>float PRECSC(time, lat, lon) ;
>PRECSC:units = "m/s" ;
>PRECSC:long_name = "Convective snow rate (water equivalent)" ;
>PRECSC:cell_method = "time: mean" ;
>float PRECSH(time, lat, lon) ;
>PRECSH:units = "m/s" ;
>&nbs p; PRECSH:long_name = "Shallow Convection precipitation rate" ;
>PRECSH:cell_method = "time: mean" ;
>float PRECSL(time, lat, lon) ;
>PRECSL:units = "m/s" ;
>PRECSL:long_name = "Large-scale (stable) snow rate (water equivalent)" ;
>PRECSL:cell_method = "time: mean" ;
>float PS(time, lat, lon) ;
>PS:units = "Pa" ;
>PS:long_name = "Surface pressure" ;
>PS:cell_method = "time: mean" ;
>float PSDRY(time, lat, lon) ;
>PSDRY:units = "Pa" ;
>PSDRY:long_name = "Surface pressure" ;
>PSDRY:cell_method = "time: mean" ;
>&nb sp; float PSL(time, lat, lon) ;
>PSL:units = "Pa" ;
>PSL:long_name = "Sea level pressure" ;
>PSL:cell_method = "time: mean" ;
>float Q(time, lev, lat, lon) ;
>Q:units = "kg/kg" ;
>Q:long_name = "Specific humidity" ;
>Q:cell_method = "time: mean" ;
>float QC(time, lev, lat, lon) ;
>QC:units = "kg/kg/s" ;
>QC:long_name = "Q tendency - shallow convection LW export" ;
>QC:cell_method = "time: mean" ;
>float QFLX(time, lat, lon) ;
>QFLX:units = "kg/m2/s" ;
>QFLX:long_name = "Surface water flux" ;
>QFLX:cell_method = "time: mean" ;
>float QREFHT(time, lat, lon) ;
>QREFHT:units = "kg/kg" ;
>QREFHT:long_name = "Reference height humidity" ;
>QREFHT:cell_method = "time: mean" ;
>float QRL(time, lev, lat, lon) ;
>QRL:units = "K/s" ;
>QRL:long_name = "Longwave heating rate" ;
>QRL:cell_method = "time: mean" ;
>float QRS(time, lev, lat, lon) ;
>QRS:units = "K/s" ;
>QRS:long_name = "Solar heating rate" ;
>QRS:cell_method = "time: mean" ;
>float RELHUM(time, lev, lat, lon) ;
>& nbsp; RELHUM:units = "percent" ;
>RELHUM:long_name = "Relative humidity" ;
>RELHUM:cell_method = "time: mean" ;
>float SFCLDICE(time, lat, lon) ;
>SFCLDICE:units = "kg/m2/s" ;
>SFCLDICE:long_name = "CLDICE surface flux" ;
>SFCLDICE:cell_method = "time: mean" ;
>float SFCLDLIQ(time, lat, lon) ;
>SFCLDLIQ:units = "kg/m2/s" ;
>SFCLDLIQ:long_name = "CLDLIQ surface flux" ;
>SFCLDLIQ:cell_method = "time: mean" ;
>float SFQ(time, lat, lon) ;
>SFQ:units = "kg/m2/s" ;
>SFQ:long_name = "Q surface flu x" ;
>SFQ:cell_method = "time: mean" ;
>float SHFLX(time, lat, lon) ;
>SHFLX:units = "W/m2" ;
>SHFLX:long_name = "Surface sensible heat flux" ;
>SHFLX:cell_method = "time: mean" ;
>float SHFLXOI(time, lat, lon) ;
>SHFLXOI:units = "W/m2" ;
>SHFLXOI:long_name = "SHFLX over open ocn and ice" ;
>SHFLXOI:cell_method = "time: mean" ;
>float SNOWHICE(time, lat, lon) ;
>SNOWHICE:units = "m" ;
>SNOWHICE:long_name = "Water equivalent snow depth" ;
>SNOWHICE:cell_method = "time: mean" ;
>float SNOWHLND(time, la t, lon) ;
>SNOWHLND:units = "m" ;
>SNOWHLND:long_name = "Water equivalent snow depth" ;
>SNOWHLND:cell_method = "time: mean" ;
>float SOLIN(time, lat, lon) ;
>SOLIN:units = "W/m2" ;
>SOLIN:long_name = "Solar insolation" ;
>SOLIN:cell_method = "time: mean" ;
>float SRFRAD(time, lat, lon) ;
>SRFRAD:units = "W/m2" ;
>SRFRAD:long_name = "Net radiative flux at surface" ;
>SRFRAD:cell_method = "time: mean" ;
>float SWCF(time, lat, lon) ;
>SWCF:units = "W/m2" ;
>SWCF:long_name = "Shortwav e cloud forcing" ;
>SWCF:cell_method = "time: mean" ;
>float T(time, lev, lat, lon) ;
>T:units = "K" ;
>T:long_name = "Temperature" ;
>T:cell_method = "time: mean" ;
>float TAUX(time, lat, lon) ;
>TAUX:units = "N/m2" ;
>TAUX:long_name = "Zonal surface stress" ;
>TAUX:cell_method = "time: mean" ;
>float TAUY(time, lat, lon) ;
>TAUY:units = "N/m2" ;
>TAUY:long_name = "Meridional surface stress" ;
>TAUY:cell_method = "time: mean" ;
>float TGCLDIWP(time, lat, lon) ;
>&nbs p; TGCLDIWP:units = "gram/m2" ;
>TGCLDIWP:long_name = "Total grid-box cloud ice water path" ;
>TGCLDIWP:cell_method = "time: mean" ;
>float TGCLDLWP(time, lat, lon) ;
>TGCLDLWP:units = "gram/m2" ;
>TGCLDLWP:long_name = "Total grid-box cloud liquid water path" ;
>TGCLDLWP:cell_method = "time: mean" ;
>float TMQ(time, lat, lon) ;
>TMQ:units = "kg/m2" ;
>TMQ:long_name = "Total (vertically integrated) precipitatable water" ;
>TMQ:cell_method = "time: mean" ;
>float TREFHT(time, lat, lon) ;
>TREFHT:units = "K" ;
>TREFHT:long_ name = "Reference height temperature" ;
>TREFHT:cell_method = "time: mean" ;
>float TS(time, lat, lon) ;
>TS:units = "K" ;
>TS:long_name = "Surface temperature (radiative)" ;
>TS:cell_method = "time: mean" ;
>float TSMN(time, lat, lon) ;
>TSMN:units = "K" ;
>TSMN:long_name = "Minimum surface temperature over output period" ;
>TSMN:cell_method = "time: minimum" ;
>float TSMX(time, lat, lon) ;
>TSMX:units = "K" ;
>TSMX:long_name = "Maximum surface temperature over output period" ;
>TSMX:cell_method = "time: maximum" ;
>& nbsp; float U(time, lev, lat, lon) ;
>U:units = "m/s" ;
>U:long_name = "Zonal wind" ;
>U:cell_method = "time: mean" ;
>float UU(time, lev, lat, lon) ;
>UU:units = "m2/s2" ;
>UU:long_name = "Zonal velocity squared" ;
>UU:cell_method = "time: mean" ;
>float V(time, lev, lat, lon) ;
>V:units = "m/s" ;
>V:long_name = "Meridional wind" ;
>V:cell_method = "time: mean" ;
>float VD01(time, lev, lat, lon) ;
>VD01:units = "kg/kg/s" ;
>VD01:long_name = "Vertical diffusion of Q" ;< br> VD01:cell_method = "time: mean" ;
>float VQ(time, lev, lat, lon) ;
>VQ:units = "m/skg/kg" ;
>VQ:long_name = "Meridional water transport" ;
>VQ:cell_method = "time: mean" ;
>float VT(time, lev, lat, lon) ;
>VT:units = "K m/s" ;
>VT:long_name = "Meridional heat transport" ;
>VT:cell_method = "time: mean" ;
>float VU(time, lev, lat, lon) ;
>VU:units = "m2/s2" ;
>VU:long_name = "Meridional flux of zonal momentum" ;
>VU:cell_method = "time: mean" ;
>float VV(time, lev, lat, lon) ;
>& nbsp; VV:units = "m2/s2" ;
>VV:long_name = "Meridional velocity squared" ;
>VV:cell_method = "time: mean" ;
>float Z3(time, lev, lat, lon) ;
>Z3:units = "m" ;
>Z3:long_name = "Geopotential Height (above sea level)" ;
>Z3:cell_method = "time: mean" ;
>double gw(lat) ;
>gw:long_name = "gauss weights" ;
>double hyai(ilev) ;
>hyai:long_name = "hybrid A coefficient at layer interfaces" ;
>double hyam(lev) ;
>hyam:long_name = "hybrid A coefficient at layer midpoints" ;
>double hybi(ilev) ;
>hybi:long_name = "hybr id B coefficient at layer interfaces" ;
>double hybm(lev) ;
>hybm:long_name = "hybrid B coefficient at layer midpoints" ;
>double ilev(ilev) ;
>ilev:long_name = "hybrid level at interfaces (1000*(A+B))" ;
>ilev:units = "level" ;
>ilev:positive = "down" ;
>ilev:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ;
>ilev:formula_terms = "a: hyai b: hybi p0: P0 ps: PS" ;
>double isccp_prs(isccp_prs) ;
>isccp_prs:long_name = "Mean ISCCP pressure" ;
>isccp_prs:units = "mb" ;
>isccp_prs:isccp_prs_bnds = 0., 180., 310., 440., 560., 680., 800., 1000. ;
>&n bsp; double isccp_prstau(isccp_prstau) ;
>isccp_prstau:long_name = "Mean pressure (mb).mean optical depth (unitless)/1000" ;
>isccp_prstau:units = "mixed" ;
>double isccp_tau(isccp_tau) ;
>isccp_tau:long_name = "Mean ISCCP optical depth" ;
>isccp_tau:units = "unitless" ;
>isccp_tau:isccp_tau_bnds = 0., 0.3, 1.3, 3.6, 9.4, 23., 60., 379. ;
>double lat(lat) ;
>lat:long_name = "latitude" ;
>lat:units = "degrees_north" ;
>double lev(lev) ;
>lev:long_name = "hybrid level at midpoints (1000*(A+B))" ;
>lev:units = "level" ;
>&nb sp; lev:positive = "down" ;
>lev:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ;
>lev:formula_terms = "a: hyam b: hybm p0: P0 ps: PS" ;
>double lon(lon) ;
>lon:long_name = "longitude" ;
>lon:units = "degrees_east" ;
>int nlon(lat) ;
>nlon:long_name = "number of longitudes" ;
>double time(time) ;
>time:long_name = "time" ;
>time:units = "days since 0001-01-01 00:00:00" ;
>time:calendar = "noleap" ;
>time:bounds = "time_bnds" ;
>int wnummax(lat) ;
>wnummax:long_name = "cutoff Four ier wavenumber" ;
>
>// global attributes:
>:Conventions = "CF-1.0" ;
>:source = "CAM" ;
>:case = "TH240_CAM-th" ;
>:title = "TH240_CAM-th TH240_CAM-th" ;
>:logname = "xxxx" ;
>:host = "xxxx" ;
>:Version = "$Name: ccsm3_0_rel04 $" ;
>:revision_Id = "$Id: history.F90,v 1.26.2.48.4.1 2004/05/20 18:36:01 mvr Exp $" ;
>:history = "Sun Mar 31 00:01:36 2013: ncks -xv P0,ntrm,ntrn,ntrk,ndbase,nsbase,nbdate,mdt,date_written,time_written,nbsec,ndcur,nscur,date,co2vmr,datesec,nsteph,time_bnds /xxxx/TH240_CAM-th.cam2.h0.0048-07.nc /xxxx/xxxx/TH240_CAM-th.cam2.h0.0048-07.nc" ;
>:NCO = "4.0.7" ;
>}
>
>
>
>
>
>
>
>
>
>At 2013-04-15 21:53:11,"Jayesh Krishna" < jayesh at mcs.anl.gov > wrote:
>>Hi,
>> Please let us know more information regarding the dataset that you are using in your scripts (Attach "ncdump -h" of one of the netcdf files) so that we can try out more tests.
>>
>>Regards,
>>Jayesh
>>
>>----- Original Message -----
>>From: "Jayesh Krishna" < jayesh at mcs.anl.gov >
>>To: "Matrix" < sov_matrix at 126.com >
>>Cc: parvis-users at lists.mcs.anl.gov >Sent: Saturday, April 13, 2013 12:49:49 PM
>>Subject: Re: [Parvis-users] Performance of the parncl.
>>
>>Hi,
>> We usually test performance of ParNCL on a system with a parallel file system (like PVFS). I will try some tests on my laptop (with no parallel file system) and let you know the results.
>> Please try the vorticity test, plot_vort.ncl, in the example scripts page (http://trac.mcs.anl.gov/projects/parvis/wiki/ParNCLDeveloperInfo/SimpleParNCLScripts) and let us know the results.
>>
>>Regards,
>>Jayesh
>>
>>----- Original Message -----
>>From: "Matrix" < sov_matrix at 126.com >
>>To: "Jayesh Krishna" < jayesh at mcs.anl.gov >
>>Sent: Saturday, April 13, 2013 12:54:40 AM
>>Subject: Re:Re: [Parvis-users] Performance of the parncl.
>>
>>
>>yes, I did try running with 4-8cores and more.
>>here is my result:
>>
>>1 process uses 7492 ms
>>2 processes use 17531 ms
>>3 processes use 16345 ms
>>4 processes use 16542 ms
>>5 processes use 17814 ms
>>6 processes use 19272 ms
>>7 processes use 22437 ms
>>8 processes use 22507 ms
>>9 processes use 26080 ms
>>10 processes use 25356 ms
>>11 processes use 26817 ms
>>12 processes use 25638 ms
>>13 processes use 29558 ms
>>14 processes use 27958 ms
>>15 processes use 28537 ms
>>16 processes use 30162 ms
>>The running environment: 16 cores,RAID,but we don't have a parallel file system.
>>I don't know why running with only 1 process is the best , and why the more processes I run with , the poorer performance I get. what ' s wrong with my job?
>>I wonder if you have tested the ParNCL on a machine with parallel file system.and can you give me a script calculating vorticity, divergence ? thank you very much.
>>
>>
>>
>>
>>
>>
>>
>>At 2013-04-12 23:37:27,"Jayesh Krishna" < jayesh at mcs.anl.gov > wrote:
>>>Hi,
>>> How many processors are you using to run your script? Did you try running your job with 4-8 cores/processors?
>>> The grid-aware-parallelism in ParNCL/ParGAL comes with its costs (read/distribute data across processes; gather results; etc) and for simpler calculations like dim_avg_n() these costs show up in performance results. For more complex calculations like calculating vorticity, divergence etc ParNCL/ParGAL perform an order of magnitude better than NCL (serial version). So you might have to play with the number of processors to get better results.
>>> Let us know the results.
>>>
>>>(PS: We are also looking into making these simpler calculations more efficient. I have created a ticket on this issue - http://trac.mcs.anl.gov/projects/parvis/ticket/71)
>>>
>>>Regards,
>>>Jayesh
>>>
>>>----- Original Message -----
>>>From: "Matrix" < sov_matrix at 126.com >
>>>To: "Jayesh Krishna" < jayesh at mcs.anl.gov >
>>>Sent: Friday, April 12, 2013 10:12:18 AM
>>>Subject: Re:Re: [Parvis-users] Performance of the parncl.
>>>
>>>
>>>Here is my script .Thank you.
>>>
>>>
>>>load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>>>load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>>>load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>>>load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
>>>begin
>>>fili = systemfunc("cat listname") ; 50 yrs of cam data.
>>>
>>>setfileoption("nc","MissingToFillValue",True) ; alter default mode
>>>
>>>f = addfiles(fili, "r") ; add the files
>>>
>>>t = f[:]->T ; read temp
>>>
>>>t_avg = dim_avg_n(t, 0) ; calc dim_avg
>>>
>>>wks=gsn_open_wks("ps", "parvis_cam_eulspec_plotter_parncl")
>>>gsn_define_colormap(wks,"gui_default")
>>>
>>>res = True
>>>res at cnFillOn = True
>>>res at gsnSpreadColors = True
>>>res at mpOceanFillColor = 0
>>>res at gsnStringFontHeightF = 0.02
>>>res at gsnLeftStringFontHeightF = 0.02
>>>res@ gsnRightStringFontHeightF = 0.02
>>>res at tmXBLabelFontHeightF = 0.017
>>>res at tmYLLabelFontHeightF = 0.017
>>>res at lbLabelFontHeightF = 0.017
>>>res at cnLinesOn = False
>>>res at lbOrientation = "vertical"
>>>res at mpCenterLonF = 180
>>>res at cnLevelSelectionMode = "ExplicitLevels"
>>>
>>>plot=gsn_csm_contour_map_ce(wks, t_avg(0,:,:), res)
>>>
>>>end
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>At 2013-04-12 22:29:45,"Jayesh Krishna" < jayesh at mcs.anl.gov > wrote:
>>>>Hi,
>>>> How are you running ParNCL ? Can you send us the NCL scripts that you are using?
>>>>
>>>>(PS: Note that some computations that are trivial might require higher number of processors for better performance.)
>>>>Regards,
>>>>Jayesh
>>>>
>>>>----- Original Message -----
>>>>From: "Matrix" < sov_matrix at 126.com >
>>>>Cc: parvis-users at lists.mcs.anl.gov >>>Sent: Friday, April 12, 2013 3:44:13 AM
>>>>Subject: [Parvis-users] Performance of the parncl.
>>>>
>>>>
>>>>
>>>>Dear Professor:
>>>>I met with some troubles when using the ParNCL,I've tested the performance of the ParNCL, and it is not better than NCL, I want to know if I made something wrong with the ParNCL. So, I beg you to give me some information about the running environment when you testing the ParNCL. Thank you .
>>>>Best wishes,
>>>>Ji Xu
>>>>Tsinghua University.
>>>>
>>>>
>>>>
>>>>
>>>>_______________________________________________
>>>>Parvis-users mailing list
>>>> Parvis-users at lists.mcs.anl.gov >>>https://lists.mcs.anl.gov/mailman/listinfo/parvis-users
>>>
>>
>>_______________________________________________
>>Parvis-users mailing list
>> Parvis-users at lists.mcs.anl.gov >https://lists.mcs.anl.gov/mailman/listinfo/parvis-users
>
More information about the Parvis-users
mailing list