srun: Warning: can't honor --ntasks-per-node set to 68 which doesn't match the requested tasks 256 with the number of requested nodes 32. Ignoring --ntasks-per-node. -------------------------------------------------------------- MPI_COMM_WORLD communicator : 32 nodes, 256 MPI tasks COMMUNICATOR NODE # [NODE NAME] : (# OF MPI TASKS) TASK # LIST MPI_COMM_WORLD NODE 0 [ nid04185 ] : ( 8 MPI TASKS ) 0-7 MPI_COMM_WORLD NODE 1 [ nid04186 ] : ( 8 MPI TASKS ) 8-15 MPI_COMM_WORLD NODE 2 [ nid04187 ] : ( 8 MPI TASKS ) 16-23 MPI_COMM_WORLD NODE 3 [ nid04188 ] : ( 8 MPI TASKS ) 24-31 MPI_COMM_WORLD NODE 4 [ nid04189 ] : ( 8 MPI TASKS ) 32-39 MPI_COMM_WORLD NODE 5 [ nid04190 ] : ( 8 MPI TASKS ) 40-47 MPI_COMM_WORLD NODE 6 [ nid04191 ] : ( 8 MPI TASKS ) 48-55 MPI_COMM_WORLD NODE 7 [ nid04192 ] : ( 8 MPI TASKS ) 56-63 MPI_COMM_WORLD NODE 8 [ nid04193 ] : ( 8 MPI TASKS ) 64-71 MPI_COMM_WORLD NODE 9 [ nid05532 ] : ( 8 MPI TASKS ) 72-79 MPI_COMM_WORLD NODE 10 [ nid05533 ] : ( 8 MPI TASKS ) 80-87 MPI_COMM_WORLD NODE 11 [ nid05534 ] : ( 8 MPI TASKS ) 88-95 MPI_COMM_WORLD NODE 12 [ nid05535 ] : ( 8 MPI TASKS ) 96-103 MPI_COMM_WORLD NODE 13 [ nid05536 ] : ( 8 MPI TASKS ) 104-111 MPI_COMM_WORLD NODE 14 [ nid05537 ] : ( 8 MPI TASKS ) 112-119 MPI_COMM_WORLD NODE 15 [ nid06136 ] : ( 8 MPI TASKS ) 120-127 MPI_COMM_WORLD NODE 16 [ nid06137 ] : ( 8 MPI TASKS ) 128-135 MPI_COMM_WORLD NODE 17 [ nid06138 ] : ( 8 MPI TASKS ) 136-143 MPI_COMM_WORLD NODE 18 [ nid06139 ] : ( 8 MPI TASKS ) 144-151 MPI_COMM_WORLD NODE 19 [ nid07203 ] : ( 8 MPI TASKS ) 152-159 MPI_COMM_WORLD NODE 20 [ nid07204 ] : ( 8 MPI TASKS ) 160-167 MPI_COMM_WORLD NODE 21 [ nid07205 ] : ( 8 MPI TASKS ) 168-175 MPI_COMM_WORLD NODE 22 [ nid07206 ] : ( 8 MPI TASKS ) 176-183 MPI_COMM_WORLD NODE 23 [ nid07255 ] : ( 8 MPI TASKS ) 184-191 MPI_COMM_WORLD NODE 24 [ nid07256 ] : ( 8 MPI TASKS ) 192-199 MPI_COMM_WORLD NODE 25 [ nid07257 ] : ( 8 MPI TASKS ) 200-207 MPI_COMM_WORLD NODE 26 [ nid07871 ] : ( 8 MPI TASKS ) 208-215 MPI_COMM_WORLD NODE 27 [ nid07872 ] : ( 8 MPI TASKS ) 216-223 MPI_COMM_WORLD NODE 28 [ nid07873 ] : ( 8 MPI TASKS ) 224-231 MPI_COMM_WORLD NODE 29 [ nid07975 ] : ( 8 MPI TASKS ) 232-239 MPI_COMM_WORLD NODE 30 [ nid07976 ] : ( 8 MPI TASKS ) 240-247 MPI_COMM_WORLD NODE 31 [ nid07977 ] : ( 8 MPI TASKS ) 248-255 -------------------------------------------------------------- BUILD DATE: Wed Feb 19 08:38:44 PST 2020 HOST: cori05 BUILT BY: amollen XGC_FLAGS: -DPSPLINE -DUSE_BICUB_MOD -DUSE_ONE_D_I_CUB_MOD -DV_PERP -DSOLVERLU -DUSE_I NQUIRE_DIRECTORY -DITER_GRID -DCAM_TIMERS -DXGCA -DUSE_VEC GIT INFO: origin https://github.com/amollen/XGC-Devel.git (fetch) upstream https://a mollen@github.com/PrincetonUniversity/XGC-Devel.git (fetch) Uncommitted changes exist. Git HEAD commit ID: 4a72f48bbdd41f9420fc9e552076aaa69c5ed49e (t_initf) Read in prof_inparam namelist from: input (t_initf) Using profile_disable= F (t_initf) profile_timer= 4 (t_initf) profile_depth_limit= 99999 (t_initf) profile_detail_limit= 2 (t_initf) profile_barrier= F (t_initf) profile_outpe_num= 1 (t_initf) profile_outpe_stride= 1 (t_initf) profile_single_file= F (t_initf) profile_global_stats= T (t_initf) profile_ovhd_measurement= F (t_initf) profile_add_detail= F (t_initf) profile_papi_enable= F call petsc_init call adios_init staging_read_method_name=BP staging_read_method= 0 # of OMP threads = 8 Total simulation processors : 256 setting up......... 0 Read input file sml_param ptl_param eq_param col_param diag_param smooth_param f0_param mon_param coupling_init Interplane-Major ordering for sml_comm communicator plane mpi communication inter-plane mpi communication adios mpi communication? end of commnunicator setup particles per thread: 5000 particles per rank: 40000 reading...... eq end_flag -1 ptl_num for each MPI rank = 40000 init_interpolation third difinition part eq_x_slope= -2.553035933608407E-011 eq_x2_slope= 0.000000000000000E+000 R range : 1.000000000000000E-002 2.09090920000000 Z range : -1.09090920000000 1.09090920000000 bounce setup setting up grid system..... 0 init_grid: Initializing grid data structure... init_grid: Guess min 1.000000000000000E-002 -1.09090920000000 init_grid: Guess max 2.09090920000000 1.09090920000000 init_grid: open node file (*.node) : ./input_dir/flow7a_coarse.node init_grid: open connectivity file (*.ele) : ./input_dir/flow7a_coarse.ele init_grid: grid%nnode = 6368 init_grid: init_guess_table sum(tr_vol) 19.7744894203433 sum(node_area) 2.99614453625688 sum(node_vol) 19.7756719911409 WARNING: E-field in theta direction set to 0 for node: 6164 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6165 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6166 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6167 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6168 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6169 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6170 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6171 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6172 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6173 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6174 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6175 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6176 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6177 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6178 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6179 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6180 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6181 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6182 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6183 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6184 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6185 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6186 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6187 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6188 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6189 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6190 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6191 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6192 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6193 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6194 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6195 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6196 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6197 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6198 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6199 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6200 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6201 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6202 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6203 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6204 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6205 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6206 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6207 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6208 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6209 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6210 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6211 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6212 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6213 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6214 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6215 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6216 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6217 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6218 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6219 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6220 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6221 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6222 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6223 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6224 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6225 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6226 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6227 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6228 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6229 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6230 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6231 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6232 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6233 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6234 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6235 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6236 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6237 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6238 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6239 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6240 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6241 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6242 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6243 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6244 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6245 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6246 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6247 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6248 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6249 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6250 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6251 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6252 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6253 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6254 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6255 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6256 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6257 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6258 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6259 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6260 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6261 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6262 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6263 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6264 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6265 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6266 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6267 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6268 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6269 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6270 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6271 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6272 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6273 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6274 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6275 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6276 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6277 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6278 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6279 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6280 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6281 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6282 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6283 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6284 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6285 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6286 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6287 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6288 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6289 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6290 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6291 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6292 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6293 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6294 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6295 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6296 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6297 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6298 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6299 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6300 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6301 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6302 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6303 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6304 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6305 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6306 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6307 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6308 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6309 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6310 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6311 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6312 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6313 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6314 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6315 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6316 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6317 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6318 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6319 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6320 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6321 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6322 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6323 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6324 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6325 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6326 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6327 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6328 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6329 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6330 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6331 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6332 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6333 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6334 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6335 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6336 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6337 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6338 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6339 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6340 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6341 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6342 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6343 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6344 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6345 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6346 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6347 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6348 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6349 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6350 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6351 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6352 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6353 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6354 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6355 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6356 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6357 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6358 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6359 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6360 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6361 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6362 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6363 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6364 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6365 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6366 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6367 itr3: -1 itr2: -1 WARNING: E-field in theta direction set to 0 for node: 6368 itr3: -1 itr2: -1 nonzero n-mode, potential boundary extended bd nseg 1 iseg( 1 )= 4616 6368 nonzero n-mode, charge boundary extended bd nseg 1 iseg( 1 )= 4261 6368 n=0-mode, potential boundary extended bd nseg 1 iseg( 1 )= 4616 6368 n=0-mode, charge boundary extended bd nseg 1 iseg( 1 )= 4261 6368 diagnosis_1d init Monte-Carlo volume calculation... smoothing initializing load 0 load: Total number of markers: 10240000 load: Total number of loading triangles: 9786 load: Target ptl/triangle: 1046.39280604946 load: Total loaded volume: 14.1022990420411 load: Max. loading factor: 4.17177577173358 load: Min. loading factor: 0.625682464325871 timestep.dat does not exist. sml_restart is forced to zero uniform_space_dist: valid2/valid = 4.15997324218750 loading end 0 Init poisson Initialize 00 matrix solver - 00 matrix solver memory create create_solver: first vertex in domain: T CREATE_SOLVER: make partitioning with 4615/ 6368 real vertices [54]PETSC ERROR: ------------------------------------------------------------------------ [54]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [54]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [54]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [54]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [54]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [54]PETSC ERROR: to get more information on the crash. [54]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [54]PETSC ERROR: Signal received [54]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [54]PETSC ERROR: Petsc Release Version 3.11.3, unknown [54]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04191 by amollen Wed Feb 19 09:05:25 2020 [54]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [54]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 54 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 54 Failed during initial partitioning [80]PETSC ERROR: ------------------------------------------------------------------------ [80]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [80]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [80]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [80]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [80]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [80]PETSC ERROR: to get more information on the crash. [27]PETSC ERROR: ------------------------------------------------------------------------ [27]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [27]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [27]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [80]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [27]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [27]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [27]PETSC ERROR: to get more information on the crash. [80]PETSC ERROR: Signal received [80]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [28]PETSC ERROR: ------------------------------------------------------------------------ [28]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [80]PETSC ERROR: Petsc Release Version 3.11.3, unknown [27]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [27]PETSC ERROR: Signal received [27]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [27]PETSC ERROR: Petsc Release Version 3.11.3, unknown [109]PETSC ERROR: ------------------------------------------------------------------------ [109]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [109]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [80]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05533 by amollen Wed Feb 19 09:05:25 2020 [28]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [28]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [28]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [28]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [28]PETSC ERROR: to get more information on the crash. [109]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [109]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [109]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [109]PETSC ERROR: to get more information on the crash. [80]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [27]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04188 by amollen Wed Feb 19 09:05:25 2020 [27]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [27]PETSC ERROR: #1 User provided function() line 0 in unknown file [109]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [109]PETSC ERROR: Signal received [109]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [80]PETSC ERROR: #1 User provided function() line 0 in unknown file [28]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [28]PETSC ERROR: Signal received [28]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [109]PETSC ERROR: Petsc Release Version 3.11.3, unknown [109]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05536 by amollen Wed Feb 19 09:05:25 2020 [109]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [109]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 80 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n1] application called MPI_Abort(comm=0x84000004, 59) - process 80 [28]PETSC ERROR: Petsc Release Version 3.11.3, unknown [28]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04188 by amollen Wed Feb 19 09:05:25 2020 [28]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [28]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 27 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 27 Failed during initial partitioning Failed during initial partitioning Rank 109 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 109 Rank 28 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 28 Failed during initial partitioning Failed during initial partitioning [21]PETSC ERROR: ------------------------------------------------------------------------ [21]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [21]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [21]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [21]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [21]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [21]PETSC ERROR: to get more information on the crash. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [21]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [21]PETSC ERROR: Signal received [21]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [21]PETSC ERROR: Petsc Release Version 3.11.3, unknown [21]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04187 by amollen Wed Feb 19 09:05:25 2020 [21]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [1]PETSC ERROR: to get more information on the crash. [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Signal received [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.11.3, unknown [21]PETSC ERROR: #1 User provided function() line 0 in unknown file [1]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04185 by amollen Wed Feb 19 09:05:25 2020 [1]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [1]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 21 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n3] application called MPI_Abort(comm=0x84000002, 59) - process 21 Rank 1 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 1 Failed during initial partitioning Failed during initial partitioning [178]PETSC ERROR: ------------------------------------------------------------------------ [178]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [178]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [178]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [178]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [178]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [178]PETSC ERROR: to get more information on the crash. [178]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [178]PETSC ERROR: Signal received [178]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [178]PETSC ERROR: Petsc Release Version 3.11.3, unknown [178]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07206 by amollen Wed Feb 19 09:05:25 2020 [178]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [178]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 178 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n2] application called MPI_Abort(comm=0x84000002, 59) - process 178 Failed during initial partitioning [82]PETSC ERROR: ------------------------------------------------------------------------ [82]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [45]PETSC ERROR: ------------------------------------------------------------------------ [45]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [45]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [35]PETSC ERROR: ------------------------------------------------------------------------ [35]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [35]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [82]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [82]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [35]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [35]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [35]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [35]PETSC ERROR: to get more information on the crash. [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [82]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [82]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [82]PETSC ERROR: to get more information on the crash. [95]PETSC ERROR: ------------------------------------------------------------------------ [95]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [95]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [226]PETSC ERROR: ------------------------------------------------------------------------ [226]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [226]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [35]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [35]PETSC ERROR: Signal received [35]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [6]PETSC ERROR: to get more information on the crash. [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [6]PETSC ERROR: Signal received [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [79]PETSC ERROR: ------------------------------------------------------------------------ [79]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [79]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [82]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [82]PETSC ERROR: Signal received [95]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [95]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [95]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [95]PETSC ERROR: to get more information on the crash. [116]PETSC ERROR: ------------------------------------------------------------------------ [116]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [116]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [226]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [226]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [226]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [226]PETSC ERROR: to get more information on the crash. [35]PETSC ERROR: Petsc Release Version 3.11.3, unknown [35]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04189 by amollen Wed Feb 19 09:05:25 2020 [35]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [35]PETSC ERROR: #1 User provided function() line 0 in unknown file [6]PETSC ERROR: Petsc Release Version 3.11.3, unknown [6]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04185 by amollen Wed Feb 19 09:05:25 2020 [6]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [6]PETSC ERROR: #1 User provided function() line 0 in unknown file [79]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [79]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [79]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [79]PETSC ERROR: to get more information on the crash. [82]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [82]PETSC ERROR: Petsc Release Version 3.11.3, unknown [82]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05533 by amollen Wed Feb 19 09:05:25 2020 [95]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [95]PETSC ERROR: Signal received [95]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [116]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [116]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [116]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [116]PETSC ERROR: to get more information on the crash. [226]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [226]PETSC ERROR: Signal received [33]PETSC ERROR: ------------------------------------------------------------------------ Rank 6 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 6 [79]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [79]PETSC ERROR: Signal received [79]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [82]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [95]PETSC ERROR: Petsc Release Version 3.11.3, unknown [95]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05534 by amollen Wed Feb 19 09:05:25 2020 [95]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [116]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [116]PETSC ERROR: Signal received [226]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [226]PETSC ERROR: Petsc Release Version 3.11.3, unknown [226]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07873 by amollen Wed Feb 19 09:05:25 2020 [226]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [33]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [33]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger Failed during initial partitioning [79]PETSC ERROR: Petsc Release Version 3.11.3, unknown [79]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05532 by amollen Wed Feb 19 09:05:25 2020 [79]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [82]PETSC ERROR: #1 User provided function() line 0 in unknown file [95]PETSC ERROR: #1 User provided function() line 0 in unknown file [116]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [116]PETSC ERROR: Petsc Release Version 3.11.3, unknown [116]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05537 by amollen Wed Feb 19 09:05:25 2020 [116]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [226]PETSC ERROR: #1 User provided function() line 0 in unknown file [124]PETSC ERROR: ------------------------------------------------------------------------ [124]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [124]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [33]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [33]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [79]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 82 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 82 Rank 95 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 95 [116]PETSC ERROR: #1 User provided function() line 0 in unknown file [230]PETSC ERROR: ------------------------------------------------------------------------ [230]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [203]PETSC ERROR: ------------------------------------------------------------------------ [203]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [203]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [124]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [124]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [124]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [124]PETSC ERROR: to get more information on the crash. [33]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [33]PETSC ERROR: to get more information on the crash. Rank 79 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 79 Failed during initial partitioning Failed during initial partitioning Rank 116 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 116 [230]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [230]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [203]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [203]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [203]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [203]PETSC ERROR: to get more information on the crash. [124]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [124]PETSC ERROR: Signal received [124]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Rank 35 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 35 Failed during initial partitioning Failed during initial partitioning Rank 226 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n1] application called MPI_Abort(comm=0x84000002, 59) - process 226 [203]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [203]PETSC ERROR: Signal received [203]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [180]PETSC ERROR: ------------------------------------------------------------------------ [180]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [124]PETSC ERROR: Petsc Release Version 3.11.3, unknown [124]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06136 by amollen Wed Feb 19 09:05:25 2020 [124]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [124]PETSC ERROR: #1 User provided function() line 0 in unknown file [69]PETSC ERROR: ------------------------------------------------------------------------ [69]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [69]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [33]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [118]PETSC ERROR: ------------------------------------------------------------------------ [118]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [230]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [230]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [203]PETSC ERROR: Petsc Release Version 3.11.3, unknown [203]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07257 by amollen Wed Feb 19 09:05:25 2020 [203]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [180]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [180]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [180]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 124 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n0] application called MPI_Abort(comm=0x84000002, 59) - process 124 [69]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [69]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [69]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [69]PETSC ERROR: to get more information on the crash. [61]PETSC ERROR: ------------------------------------------------------------------------ [61]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [61]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [33]PETSC ERROR: Signal received [33]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [33]PETSC ERROR: Petsc Release Version 3.11.3, unknown [118]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [118]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [118]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Failed during initial partitioning [203]PETSC ERROR: #1 User provided function() line 0 in unknown file [180]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [180]PETSC ERROR: to get more information on the crash. [180]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Failed during initial partitioning [69]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [69]PETSC ERROR: Signal received [69]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [69]PETSC ERROR: Petsc Release Version 3.11.3, unknown [150]PETSC ERROR: ------------------------------------------------------------------------ [150]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [150]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [150]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [61]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [61]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [61]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [61]PETSC ERROR: to get more information on the crash. Failed during initial partitioning [118]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [118]PETSC ERROR: to get more information on the crash. [230]PETSC ERROR: to get more information on the crash. [230]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 203 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 203 [180]PETSC ERROR: Signal received [180]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [180]PETSC ERROR: Petsc Release Version 3.11.3, unknown [69]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04193 by amollen Wed Feb 19 09:05:25 2020 [69]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [69]PETSC ERROR: #1 User provided function() line 0 in unknown file [150]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [150]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [150]PETSC ERROR: to get more information on the crash. [61]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [61]PETSC ERROR: Signal received [61]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [33]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04189 by amollen Wed Feb 19 09:05:25 2020 [33]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [118]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [118]PETSC ERROR: Signal received [118]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [118]PETSC ERROR: Petsc Release Version 3.11.3, unknown [230]PETSC ERROR: Signal received [230]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Failed during initial partitioning [180]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07206 by amollen Wed Feb 19 09:05:25 2020 [180]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Rank 69 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 69 [150]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [150]PETSC ERROR: Signal received [150]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [150]PETSC ERROR: Petsc Release Version 3.11.3, unknown [61]PETSC ERROR: Petsc Release Version 3.11.3, unknown [61]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04192 by amollen Wed Feb 19 09:05:25 2020 [61]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [33]PETSC ERROR: #1 User provided function() line 0 in unknown file [118]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05537 by amollen Wed Feb 19 09:05:25 2020 [118]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [230]PETSC ERROR: Petsc Release Version 3.11.3, unknown [230]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07873 by amollen Wed Feb 19 09:05:25 2020 [180]PETSC ERROR: #1 User provided function() line 0 in unknown file [172]PETSC ERROR: ------------------------------------------------------------------------ [172]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [172]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger Failed during initial partitioning [150]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06139 by amollen Wed Feb 19 09:05:25 2020 [150]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [150]PETSC ERROR: #1 User provided function() line 0 in unknown file [61]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 33 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 33 [118]PETSC ERROR: #1 User provided function() line 0 in unknown file [230]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Rank 180 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n2] application called MPI_Abort(comm=0x84000002, 59) - process 180 [172]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [172]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [172]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [172]PETSC ERROR: to get more information on the crash. Rank 150 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n3] application called MPI_Abort(comm=0x84000002, 59) - process 150 Rank 61 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 61 Failed during initial partitioning Rank 118 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 118 [230]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning [172]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [172]PETSC ERROR: Signal received [172]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [172]PETSC ERROR: Petsc Release Version 3.11.3, unknown Failed during initial partitioning Failed during initial partitioning Failed during initial partitioning [227]PETSC ERROR: ------------------------------------------------------------------------ [227]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [172]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07205 by amollen Wed Feb 19 09:05:25 2020 [172]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [172]PETSC ERROR: #1 User provided function() line 0 in unknown file [34]PETSC ERROR: ------------------------------------------------------------------------ [34]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [113]PETSC ERROR: ------------------------------------------------------------------------ [113]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [227]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [227]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Rank 172 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n1] application called MPI_Abort(comm=0x84000002, 59) - process 172 [34]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [34]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [34]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [113]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [113]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Rank 230 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n1] application called MPI_Abort(comm=0x84000002, 59) - process 230 Failed during initial partitioning Failed during initial partitioning [34]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [34]PETSC ERROR: to get more information on the crash. [34]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [113]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [113]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [227]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [227]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [227]PETSC ERROR: to get more information on the crash. [34]PETSC ERROR: Signal received [34]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [113]PETSC ERROR: to get more information on the crash. [113]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [227]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [34]PETSC ERROR: Petsc Release Version 3.11.3, unknown [34]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04189 by amollen Wed Feb 19 09:05:25 2020 [113]PETSC ERROR: Signal received [227]PETSC ERROR: Signal received [227]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [227]PETSC ERROR: Petsc Release Version 3.11.3, unknown [53]PETSC ERROR: ------------------------------------------------------------------------ [53]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [34]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [34]PETSC ERROR: #1 User provided function() line 0 in unknown file [113]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [113]PETSC ERROR: Petsc Release Version 3.11.3, unknown [227]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07873 by amollen Wed Feb 19 09:05:25 2020 [53]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [53]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [53]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 34 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 34 [113]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05537 by amollen Wed Feb 19 09:05:25 2020 [113]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [227]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [227]PETSC ERROR: #1 User provided function() line 0 in unknown file [53]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [53]PETSC ERROR: to get more information on the crash. [53]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Failed during initial partitioning [113]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 227 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n1] application called MPI_Abort(comm=0x84000002, 59) - process 227 [53]PETSC ERROR: Signal received [53]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Rank 113 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 113 Failed during initial partitioning [53]PETSC ERROR: Petsc Release Version 3.11.3, unknown [53]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04191 by amollen Wed Feb 19 09:05:25 2020 [53]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Failed during initial partitioning [53]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 53 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 53 Failed during initial partitioning [138]PETSC ERROR: ------------------------------------------------------------------------ [138]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [138]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [138]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [138]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [138]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [138]PETSC ERROR: to get more information on the crash. [138]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [138]PETSC ERROR: Signal received [138]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [138]PETSC ERROR: Petsc Release Version 3.11.3, unknown [138]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06138 by amollen Wed Feb 19 09:05:25 2020 [138]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [138]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 138 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n2] application called MPI_Abort(comm=0x84000002, 59) - process 138 Failed during initial partitioning [23]PETSC ERROR: ------------------------------------------------------------------------ [23]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [23]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [23]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [23]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [23]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [23]PETSC ERROR: to get more information on the crash. [23]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [26]PETSC ERROR: ------------------------------------------------------------------------ [26]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [37]PETSC ERROR: ------------------------------------------------------------------------ [37]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [23]PETSC ERROR: Signal received [23]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [23]PETSC ERROR: Petsc Release Version 3.11.3, unknown [23]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04187 by amollen Wed Feb 19 09:05:25 2020 [26]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [26]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [26]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [9]PETSC ERROR: ------------------------------------------------------------------------ [9]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [37]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [37]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [37]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [37]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [23]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [23]PETSC ERROR: #1 User provided function() line 0 in unknown file [26]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [26]PETSC ERROR: to get more information on the crash. [26]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [37]PETSC ERROR: to get more information on the crash. [37]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [26]PETSC ERROR: Signal received [26]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [9]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [9]PETSC ERROR: to get more information on the crash. [37]PETSC ERROR: Signal received [37]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [37]PETSC ERROR: Petsc Release Version 3.11.3, unknown [225]PETSC ERROR: ------------------------------------------------------------------------ [225]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [26]PETSC ERROR: Petsc Release Version 3.11.3, unknown [26]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04188 by amollen Wed Feb 19 09:05:25 2020 [9]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [9]PETSC ERROR: Signal received [241]PETSC ERROR: ------------------------------------------------------------------------ [241]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [241]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [37]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04189 by amollen Wed Feb 19 09:05:25 2020 [37]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Rank 23 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n3] application called MPI_Abort(comm=0x84000002, 59) - process 23 [225]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [225]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [26]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [26]PETSC ERROR: #1 User provided function() line 0 in unknown file [9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [9]PETSC ERROR: Petsc Release Version 3.11.3, unknown [9]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04186 by amollen Wed Feb 19 09:05:25 2020 [37]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning [225]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [225]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [225]PETSC ERROR: to get more information on the crash. [225]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 26 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 26 [9]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [9]PETSC ERROR: #1 User provided function() line 0 in unknown file [247]PETSC ERROR: ------------------------------------------------------------------------ [247]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [247]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [247]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [247]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 37 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 37 [225]PETSC ERROR: Signal received [225]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Failed during initial partitioning Rank 9 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n2] application called MPI_Abort(comm=0x84000002, 59) - process 9 [241]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [241]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [241]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [241]PETSC ERROR: to get more information on the crash. [241]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Failed during initial partitioning [225]PETSC ERROR: Petsc Release Version 3.11.3, unknown [225]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07873 by amollen Wed Feb 19 09:05:25 2020 [225]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [30]PETSC ERROR: ------------------------------------------------------------------------ [30]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range Failed during initial partitioning [141]PETSC ERROR: ------------------------------------------------------------------------ [141]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [247]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [247]PETSC ERROR: to get more information on the crash. [247]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [247]PETSC ERROR: Signal received [106]PETSC ERROR: ------------------------------------------------------------------------ [106]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [225]PETSC ERROR: #1 User provided function() line 0 in unknown file [24]PETSC ERROR: ------------------------------------------------------------------------ [24]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [24]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [141]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [141]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [141]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [241]PETSC ERROR: Signal received [241]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [241]PETSC ERROR: Petsc Release Version 3.11.3, unknown [241]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07976 by amollen Wed Feb 19 09:05:25 2020 [106]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [106]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [106]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [19]PETSC ERROR: ------------------------------------------------------------------------ [19]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range Rank 225 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n1] application called MPI_Abort(comm=0x84000002, 59) - process 225 [30]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [30]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [30]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [30]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [189]PETSC ERROR: ------------------------------------------------------------------------ [189]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [189]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [197]PETSC ERROR: ------------------------------------------------------------------------ [197]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [197]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [141]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [141]PETSC ERROR: to get more information on the crash. [141]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [247]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [247]PETSC ERROR: Petsc Release Version 3.11.3, unknown [247]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07976 by amollen Wed Feb 19 09:05:25 2020 [247]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [247]PETSC ERROR: #1 User provided function() line 0 in unknown file [106]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [106]PETSC ERROR: to get more information on the crash. [19]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [19]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Failed during initial partitioning [24]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [24]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [24]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [24]PETSC ERROR: to get more information on the crash. [205]PETSC ERROR: ------------------------------------------------------------------------ [205]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [179]PETSC ERROR: ------------------------------------------------------------------------ [179]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [189]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [189]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [189]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [189]PETSC ERROR: to get more information on the crash. [197]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [197]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [197]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [197]PETSC ERROR: to get more information on the crash. [141]PETSC ERROR: Signal received [141]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [141]PETSC ERROR: Petsc Release Version 3.11.3, unknown [241]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [241]PETSC ERROR: #1 User provided function() line 0 in unknown file [106]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [106]PETSC ERROR: Signal received [106]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [19]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [19]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [19]PETSC ERROR: to get more information on the crash. [19]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [30]PETSC ERROR: to get more information on the crash. [30]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [30]PETSC ERROR: Signal received [205]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [205]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [179]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [179]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [189]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [189]PETSC ERROR: Signal received [189]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [197]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [197]PETSC ERROR: Signal received [197]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [141]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06138 by amollen Wed Feb 19 09:05:25 2020 [141]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Rank 241 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n0] application called MPI_Abort(comm=0x84000002, 59) - process 241 [7]PETSC ERROR: ------------------------------------------------------------------------ [7]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [106]PETSC ERROR: Petsc Release Version 3.11.3, unknown [106]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05536 by amollen Wed Feb 19 09:05:25 2020 [19]PETSC ERROR: Signal received [19]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [112]PETSC ERROR: ------------------------------------------------------------------------ [112]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [24]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [162]PETSC ERROR: ------------------------------------------------------------------------ [162]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [162]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [248]PETSC ERROR: ------------------------------------------------------------------------ [248]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [205]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [205]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [205]PETSC ERROR: to get more information on the crash. [205]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [179]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [179]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [179]PETSC ERROR: to get more information on the crash. [179]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [189]PETSC ERROR: Petsc Release Version 3.11.3, unknown [189]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07255 by amollen Wed Feb 19 09:05:25 2020 [189]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [189]PETSC ERROR: #1 User provided function() line 0 in unknown file [197]PETSC ERROR: Petsc Release Version 3.11.3, unknown [197]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07256 by amollen Wed Feb 19 09:05:25 2020 [197]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [197]PETSC ERROR: #1 User provided function() line 0 in unknown file [68]PETSC ERROR: ------------------------------------------------------------------------ [68]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [141]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 247 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n0] application called MPI_Abort(comm=0x84000002, 59) - process 247 [216]PETSC ERROR: ------------------------------------------------------------------------ [216]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [216]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [106]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [106]PETSC ERROR: #1 User provided function() line 0 in unknown file [19]PETSC ERROR: Petsc Release Version 3.11.3, unknown [19]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04187 by amollen Wed Feb 19 09:05:25 2020 [112]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [112]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [112]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [30]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [30]PETSC ERROR: Petsc Release Version 3.11.3, unknown [30]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04188 by amollen Wed Feb 19 09:05:25 2020 [30]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [10]PETSC ERROR: ------------------------------------------------------------------------ [162]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [162]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [162]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [162]PETSC ERROR: to get more information on the crash. [250]PETSC ERROR: ------------------------------------------------------------------------ [250]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [250]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [250]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [250]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [250]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [205]PETSC ERROR: Signal received [205]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [179]PETSC ERROR: Signal received [179]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [179]PETSC ERROR: Petsc Release Version 3.11.3, unknown Rank 189 [Wed Feb 19 09:05:35 2020] [c1-3c2s5n3] application called MPI_Abort(comm=0x84000002, 59) - process 189 Rank 197 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n0] application called MPI_Abort(comm=0x84000002, 59) - process 197 [68]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [68]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [68]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 141 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n2] application called MPI_Abort(comm=0x84000002, 59) - process 141 Failed during initial partitioning [216]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [216]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [216]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [216]PETSC ERROR: to get more information on the crash. [36]PETSC ERROR: ------------------------------------------------------------------------ [36]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [7]PETSC ERROR: to get more information on the crash. [7]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [7]PETSC ERROR: Signal received [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Rank 106 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 106 [75]PETSC ERROR: ------------------------------------------------------------------------ [81]PETSC ERROR: ------------------------------------------------------------------------ [81]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [19]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [112]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [112]PETSC ERROR: to get more information on the crash. *** Error in `/global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec': munmap_chunk(): invalid pointer: 0x0000010000a424c0 *** [24]PETSC ERROR: Signal received [24]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [24]PETSC ERROR: Petsc Release Version 3.11.3, unknown [14]PETSC ERROR: ------------------------------------------------------------------------ [14]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [162]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [162]PETSC ERROR: Signal received [248]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [248]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [248]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [248]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [248]PETSC ERROR: to get more information on the crash. [205]PETSC ERROR: Petsc Release Version 3.11.3, unknown [205]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07257 by amollen Wed Feb 19 09:05:25 2020 [205]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [179]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07206 by amollen Wed Feb 19 09:05:25 2020 [179]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Failed during initial partitioning Failed during initial partitioning [68]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [68]PETSC ERROR: to get more information on the crash. [68]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Failed during initial partitioning [146]PETSC ERROR: ------------------------------------------------------------------------ Failed during initial partitioning [216]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [216]PETSC ERROR: Signal received [38]PETSC ERROR: ------------------------------------------------------------------------ [38]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [38]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [7]PETSC ERROR: Petsc Release Version 3.11.3, unknown [7]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04185 by amollen Wed Feb 19 09:05:25 2020 [7]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Failed during initial partitioning [76]PETSC ERROR: ------------------------------------------------------------------------ [76]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [76]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [84]PETSC ERROR: ------------------------------------------------------------------------ [84]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [84]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [84]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [90]PETSC ERROR: ------------------------------------------------------------------------ [90]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [19]PETSC ERROR: #1 User provided function() line 0 in unknown file [112]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [112]PETSC ERROR: Signal received [229]PETSC ERROR: ------------------------------------------------------------------------ [30]PETSC ERROR: #1 User provided function() line 0 in unknown file [10]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [10]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [10]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [10]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [162]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [162]PETSC ERROR: Petsc Release Version 3.11.3, unknown [162]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07204 by amollen Wed Feb 19 09:05:25 2020 [101]PETSC ERROR: ------------------------------------------------------------------------ [101]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [250]PETSC ERROR: to get more information on the crash. [250]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [250]PETSC ERROR: Signal received [209]PETSC ERROR: ------------------------------------------------------------------------ [209]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [209]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [205]PETSC ERROR: #1 User provided function() line 0 in unknown file [179]PETSC ERROR: #1 User provided function() line 0 in unknown file [68]PETSC ERROR: Signal received [147]PETSC ERROR: ------------------------------------------------------------------------ [147]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [147]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [244]PETSC ERROR: ------------------------------------------------------------------------ [244]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [216]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [216]PETSC ERROR: Petsc Release Version 3.11.3, unknown [216]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07872 by amollen Wed Feb 19 09:05:25 2020 [36]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [36]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [36]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [36]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [5]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [73]PETSC ERROR: ------------------------------------------------------------------------ [85]PETSC ERROR: ------------------------------------------------------------------------ [85]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [85]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [85]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [85]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [85]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [90]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [90]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Rank 19 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n3] application called MPI_Abort(comm=0x84000002, 59) - process 19 [112]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. *** Error in `/global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec': munmap_chunk(): invalid pointer: 0x0000010000a424c0 *** [24]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04188 by amollen Wed Feb 19 09:05:25 2020 [24]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [11]PETSC ERROR: ------------------------------------------------------------------------ [11]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [11]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [11]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [11]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [11]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [11]PETSC ERROR: to get more information on the crash. [162]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [162]PETSC ERROR: #1 User provided function() line 0 in unknown file [102]PETSC ERROR: ------------------------------------------------------------------------ [102]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [102]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [102]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [102]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [248]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [248]PETSC ERROR: Signal received [248]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [248]PETSC ERROR: Petsc Release Version 3.11.3, unknown [248]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07977 by amollen Wed Feb 19 09:05:25 2020 [209]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [209]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [209]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [209]PETSC ERROR: to get more information on the crash. Rank 205 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 205 Rank 179 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n2] application called MPI_Abort(comm=0x84000002, 59) - process 179 [126]PETSC ERROR: ------------------------------------------------------------------------ [194]PETSC ERROR: ------------------------------------------------------------------------ [194]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [130]PETSC ERROR: ------------------------------------------------------------------------ [130]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [130]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [64]PETSC ERROR: ------------------------------------------------------------------------ [142]PETSC ERROR: ------------------------------------------------------------------------ [142]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [149]PETSC ERROR: ------------------------------------------------------------------------ [149]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [149]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [149]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [244]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [244]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [244]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [216]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [38]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [38]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [38]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [7]PETSC ERROR: #1 User provided function() line 0 in unknown file [108]PETSC ERROR: ------------------------------------------------------------------------ [108]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [75]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [75]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [75]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [75]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [75]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [75]PETSC ERROR: to get more information on the crash. [81]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [81]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [81]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [81]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [81]PETSC ERROR: to get more information on the crash. [91]PETSC ERROR: ------------------------------------------------------------------------ [91]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [91]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger Failed during initial partitioning [112]PETSC ERROR: Petsc Release Version 3.11.3, unknown [229]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [229]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [229]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [229]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [229]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [24]PETSC ERROR: #1 User provided function() line 0 in unknown file [14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [14]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [14]PETSC ERROR: to get more information on the crash. [14]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 162 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n0] application called MPI_Abort(comm=0x84000002, 59) - process 162 [98]PETSC ERROR: ------------------------------------------------------------------------ [98]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [98]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [98]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [98]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [250]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [250]PETSC ERROR: Petsc Release Version 3.11.3, unknown [250]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07977 by amollen Wed Feb 19 09:05:25 2020 [250]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [250]PETSC ERROR: #1 User provided function() line 0 in unknown file [209]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [209]PETSC ERROR: Signal received [209]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Failed during initial partitioning Failed during initial partitioning [121]PETSC ERROR: ------------------------------------------------------------------------ [121]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [121]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [194]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [194]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [130]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [130]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [130]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [68]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [68]PETSC ERROR: Petsc Release Version 3.11.3, unknown [68]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04193 by amollen Wed Feb 19 09:05:25 2020 [142]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [142]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [146]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [146]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [146]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [146]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [146]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [49]PETSC ERROR: ------------------------------------------------------------------------ [49]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [243]PETSC ERROR: ------------------------------------------------------------------------ [243]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [243]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [243]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [216]PETSC ERROR: #1 User provided function() line 0 in unknown file [39]PETSC ERROR: ------------------------------------------------------------------------ [39]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [39]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [39]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [39]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [39]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [5]PETSC ERROR: to get more information on the crash. [5]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [5]PETSC ERROR: Signal received [110]PETSC ERROR: ------------------------------------------------------------------------ [110]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [110]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [76]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [76]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [76]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [76]PETSC ERROR: to get more information on the crash. [84]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [84]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [84]PETSC ERROR: to get more information on the crash. [84]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [92]PETSC ERROR: ------------------------------------------------------------------------ [92]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [92]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [92]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [92]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [18]PETSC ERROR: ------------------------------------------------------------------------ [18]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [112]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05537 by amollen Wed Feb 19 09:05:25 2020 [233]PETSC ERROR: ------------------------------------------------------------------------ [233]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [229]PETSC ERROR: to get more information on the crash. [229]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 30 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 30 [10]PETSC ERROR: to get more information on the crash. [10]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [10]PETSC ERROR: Signal received [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Failed during initial partitioning [99]PETSC ERROR: ------------------------------------------------------------------------ [99]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [99]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [99]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [99]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [99]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [248]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [248]PETSC ERROR: #1 User provided function() line 0 in unknown file [154]PETSC ERROR: ------------------------------------------------------------------------ [154]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [154]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [209]PETSC ERROR: Petsc Release Version 3.11.3, unknown [209]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07871 by amollen Wed Feb 19 09:05:25 2020 [209]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [204]PETSC ERROR: ------------------------------------------------------------------------ [204]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [177]PETSC ERROR: ------------------------------------------------------------------------ [177]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [190]PETSC ERROR: ------------------------------------------------------------------------ [190]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [125]PETSC ERROR: ------------------------------------------------------------------------ [125]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [125]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [198]PETSC ERROR: ------------------------------------------------------------------------ [198]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [198]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [131]PETSC ERROR: ------------------------------------------------------------------------ [131]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [131]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [64]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [142]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [142]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [142]PETSC ERROR: to get more information on the crash. [147]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [147]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [147]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [147]PETSC ERROR: to get more information on the crash. [49]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [49]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [244]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [244]PETSC ERROR: to get more information on the crash. [244]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [244]PETSC ERROR: Signal received Rank 216 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n0] application called MPI_Abort(comm=0x84000004, 59) - process 216 [36]PETSC ERROR: to get more information on the crash. [36]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [36]PETSC ERROR: Signal received [36]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [36]PETSC ERROR: Petsc Release Version 3.11.3, unknown Rank 7 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 7 Failed during initial partitioning [108]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [108]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [108]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [108]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [108]PETSC ERROR: to get more information on the crash. [73]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [73]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [73]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [73]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [73]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [73]PETSC ERROR: to get more information on the crash. [85]PETSC ERROR: to get more information on the crash. [85]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [85]PETSC ERROR: Signal received [93]PETSC ERROR: ------------------------------------------------------------------------ [93]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [93]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [93]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [93]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [93]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [18]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [18]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [112]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [112]PETSC ERROR: #1 User provided function() line 0 in unknown file [229]PETSC ERROR: Signal received Failed during initial partitioning [11]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [11]PETSC ERROR: Signal received [11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [11]PETSC ERROR: Petsc Release Version 3.11.3, unknown [100]PETSC ERROR: ------------------------------------------------------------------------ [100]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [100]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [100]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [100]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [100]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [100]PETSC ERROR: to get more information on the crash. Rank 250 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n1] application called MPI_Abort(comm=0x84000002, 59) - process 250 [157]PETSC ERROR: ------------------------------------------------------------------------ [157]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [157]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [157]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [157]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [209]PETSC ERROR: #1 User provided function() line 0 in unknown file [204]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [204]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [204]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [204]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [177]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [177]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [190]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [190]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [190]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [126]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [126]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [126]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [126]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [126]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [126]PETSC ERROR: to get more information on the crash. *** Error in `/global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec': free(): invalid pointer: 0x0000010000457dc0 *** [133]PETSC ERROR: ------------------------------------------------------------------------ [133]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [133]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [133]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [133]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [133]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [68]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [68]PETSC ERROR: #1 User provided function() line 0 in unknown file [143]PETSC ERROR: ------------------------------------------------------------------------ [143]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [149]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [149]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [149]PETSC ERROR: to get more information on the crash. [149]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [60]PETSC ERROR: ------------------------------------------------------------------------ [60]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [49]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [49]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [49]PETSC ERROR: to get more information on the crash. [243]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [243]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [243]PETSC ERROR: to get more information on the crash. [243]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [219]PETSC ERROR: ------------------------------------------------------------------------ [219]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [219]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [38]PETSC ERROR: to get more information on the crash. [38]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [38]PETSC ERROR: Signal received [38]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [38]PETSC ERROR: Petsc Release Version 3.11.3, unknown [38]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04189 by amollen Wed Feb 19 09:05:25 2020 [38]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [5]PETSC ERROR: Petsc Release Version 3.11.3, unknown [5]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04185 by amollen Wed Feb 19 09:05:25 2020 [5]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [5]PETSC ERROR: #1 User provided function() line 0 in unknown file [110]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [110]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [110]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [110]PETSC ERROR: to get more information on the crash. [75]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [75]PETSC ERROR: Signal received [75]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [75]PETSC ERROR: Petsc Release Version 3.11.3, unknown [81]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [81]PETSC ERROR: Signal received [81]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [81]PETSC ERROR: Petsc Release Version 3.11.3, unknown [81]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05533 by amollen Wed Feb 19 09:05:25 2020 [90]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [90]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [90]PETSC ERROR: to get more information on the crash. [90]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [90]PETSC ERROR: Signal received [90]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [90]PETSC ERROR: Petsc Release Version 3.11.3, unknown [20]PETSC ERROR: ------------------------------------------------------------------------ [20]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [20]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [114]PETSC ERROR: ------------------------------------------------------------------------ [114]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [237]PETSC ERROR: ------------------------------------------------------------------------ [237]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [237]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [237]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [237]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [229]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [229]PETSC ERROR: Petsc Release Version 3.11.3, unknown Rank 24 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n0] application called MPI_Abort(comm=0x84000004, 59) - process 24 [13]PETSC ERROR: ------------------------------------------------------------------------ [13]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [13]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [13]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [13]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [13]PETSC ERROR: to get more information on the crash. *** Error in `/global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec': free(): invalid pointer: 0x00000100009f9b30 *** [101]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [101]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [101]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [101]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [101]PETSC ERROR: to get more information on the crash. [101]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [101]PETSC ERROR: Signal received [101]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [101]PETSC ERROR: Petsc Release Version 3.11.3, unknown [101]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05535 by amollen Wed Feb 19 09:05:25 2020 Rank 248 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n1] application called MPI_Abort(comm=0x84000004, 59) - process 248 [152]PETSC ERROR: ------------------------------------------------------------------------ [152]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [152]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [152]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [152]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 209 [Wed Feb 19 09:05:35 2020] [c4-3c2s15n3] application called MPI_Abort(comm=0x84000002, 59) - process 209 [202]PETSC ERROR: ------------------------------------------------------------------------ [181]PETSC ERROR: ------------------------------------------------------------------------ [181]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [181]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [181]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [190]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [190]PETSC ERROR: to get more information on the crash. [190]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [121]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [121]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [121]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [121]PETSC ERROR: to get more information on the crash. [121]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [194]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [194]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [194]PETSC ERROR: to get more information on the crash. [194]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [194]PETSC ERROR: Signal received [130]PETSC ERROR: to get more information on the crash. [130]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [130]PETSC ERROR: Signal received [130]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [130]PETSC ERROR: Petsc Release Version 3.11.3, unknown [130]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06137 by amollen Wed Feb 19 09:05:25 2020 [171]PETSC ERROR: ------------------------------------------------------------------------ [171]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [64]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [64]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [64]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [142]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [142]PETSC ERROR: Signal received [142]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [142]PETSC ERROR: Petsc Release Version 3.11.3, unknown [146]PETSC ERROR: to get more information on the crash. [146]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [146]PETSC ERROR: Signal received [146]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [146]PETSC ERROR: Petsc Release Version 3.11.3, unknown [62]PETSC ERROR: ------------------------------------------------------------------------ [62]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [62]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [49]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [49]PETSC ERROR: Signal received [244]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [244]PETSC ERROR: Petsc Release Version 3.11.3, unknown [244]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07976 by amollen Wed Feb 19 09:05:25 2020 Failed during initial partitioning [39]PETSC ERROR: to get more information on the crash. [39]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [39]PETSC ERROR: Signal received [39]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [39]PETSC ERROR: Petsc Release Version 3.11.3, unknown [105]PETSC ERROR: ------------------------------------------------------------------------ [105]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [105]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [76]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [76]PETSC ERROR: Signal received [76]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [76]PETSC ERROR: Petsc Release Version 3.11.3, unknown [76]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05532 by amollen Wed Feb 19 09:05:25 2020 [83]PETSC ERROR: ------------------------------------------------------------------------ [83]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [83]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [83]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [83]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [91]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [91]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [91]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [91]PETSC ERROR: to get more information on the crash. [91]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [91]PETSC ERROR: Signal received [91]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [91]PETSC ERROR: Petsc Release Version 3.11.3, unknown [18]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [18]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [18]PETSC ERROR: to get more information on the crash. [18]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 112 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n1] application called MPI_Abort(comm=0x84000004, 59) - process 112 [238]PETSC ERROR: ------------------------------------------------------------------------ [238]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [238]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [238]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [238]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [238]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [238]PETSC ERROR: to get more information on the crash. [229]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07873 by amollen Wed Feb 19 09:05:25 2020 [229]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [229]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning [14]PETSC ERROR: Signal received [14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [14]PETSC ERROR: Petsc Release Version 3.11.3, unknown [14]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04186 by amollen Wed Feb 19 09:05:25 2020 [14]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [14]PETSC ERROR: #1 User provided function() line 0 in unknown file *** Error in `/global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec': free(): invalid pointer: 0x00000100009f9b30 *** [102]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [102]PETSC ERROR: to get more information on the crash. [102]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [102]PETSC ERROR: Signal received [102]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [102]PETSC ERROR: Petsc Release Version 3.11.3, unknown [102]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05535 by amollen Wed Feb 19 09:05:25 2020 Failed during initial partitioning [154]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [154]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [154]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [154]PETSC ERROR: to get more information on the crash. [154]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [154]PETSC ERROR: Signal received Failed during initial partitioning [204]PETSC ERROR: to get more information on the crash. [204]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [204]PETSC ERROR: Signal received [177]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [177]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [177]PETSC ERROR: to get more information on the crash. [177]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [191]PETSC ERROR: ------------------------------------------------------------------------ [122]PETSC ERROR: ------------------------------------------------------------------------ [122]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [122]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [122]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [122]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [122]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [122]PETSC ERROR: to get more information on the crash. [195]PETSC ERROR: ------------------------------------------------------------------------ [195]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [195]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [195]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [195]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [131]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [131]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [131]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [131]PETSC ERROR: to get more information on the crash. [131]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [131]PETSC ERROR: Signal received [171]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [171]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [64]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [64]PETSC ERROR: to get more information on the crash. [143]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [143]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [143]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [147]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [147]PETSC ERROR: Signal received [147]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [147]PETSC ERROR: Petsc Release Version 3.11.3, unknown [57]PETSC ERROR: ------------------------------------------------------------------------ [57]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [57]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [57]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [57]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [49]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [49]PETSC ERROR: Petsc Release Version 3.11.3, unknown [243]PETSC ERROR: Signal received [243]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [243]PETSC ERROR: Petsc Release Version 3.11.3, unknown [219]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [219]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [36]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04189 by amollen Wed Feb 19 09:05:25 2020 [36]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [36]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 5 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 5 [108]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [108]PETSC ERROR: Signal received [108]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [108]PETSC ERROR: Petsc Release Version 3.11.3, unknown [78]PETSC ERROR: ------------------------------------------------------------------------ [78]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [78]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [78]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [84]PETSC ERROR: Signal received [84]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [84]PETSC ERROR: Petsc Release Version 3.11.3, unknown [84]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05533 by amollen Wed Feb 19 09:05:25 2020 [84]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [84]PETSC ERROR: #1 User provided function() line 0 in unknown file [92]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [92]PETSC ERROR: to get more information on the crash. [92]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [92]PETSC ERROR: Signal received [92]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [92]PETSC ERROR: Petsc Release Version 3.11.3, unknown [92]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05534 by amollen Wed Feb 19 09:05:25 2020 [20]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [20]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [20]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [20]PETSC ERROR: to get more information on the crash. [114]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [114]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [114]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [233]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [233]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [233]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [233]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [233]PETSC ERROR: to get more information on the crash. [233]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 229 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n1] application called MPI_Abort(comm=0x84000002, 59) - process 229 [29]PETSC ERROR: ------------------------------------------------------------------------ [29]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [8]PETSC ERROR: ------------------------------------------------------------------------ [8]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [166]PETSC ERROR: ------------------------------------------------------------------------ [166]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [166]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [166]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [102]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [102]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning [156]PETSC ERROR: ------------------------------------------------------------------------ [156]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [156]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [156]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [156]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [156]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [156]PETSC ERROR: to get more information on the crash. [215]PETSC ERROR: ------------------------------------------------------------------------ [215]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [202]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [202]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [202]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [181]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [181]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [181]PETSC ERROR: to get more information on the crash. [181]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [190]PETSC ERROR: Signal received [190]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [190]PETSC ERROR: Petsc Release Version 3.11.3, unknown [125]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [125]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [125]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [125]PETSC ERROR: to get more information on the crash. [125]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [125]PETSC ERROR: Signal received [125]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [125]PETSC ERROR: Petsc Release Version 3.11.3, unknown [198]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [198]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [198]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [198]PETSC ERROR: to get more information on the crash. [198]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [198]PETSC ERROR: Signal received [198]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [133]PETSC ERROR: to get more information on the crash. [133]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [133]PETSC ERROR: Signal received [133]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [133]PETSC ERROR: Petsc Release Version 3.11.3, unknown [174]PETSC ERROR: ------------------------------------------------------------------------ [64]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [142]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06138 by amollen Wed Feb 19 09:05:25 2020 [142]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [142]PETSC ERROR: #1 User provided function() line 0 in unknown file [149]PETSC ERROR: Signal received [149]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [149]PETSC ERROR: Petsc Release Version 3.11.3, unknown [149]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06139 by amollen Wed Feb 19 09:05:25 2020 [149]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [58]PETSC ERROR: ------------------------------------------------------------------------ [58]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [49]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04191 by amollen Wed Feb 19 09:05:25 2020 [244]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [244]PETSC ERROR: #1 User provided function() line 0 in unknown file [219]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [219]PETSC ERROR: to get more information on the crash. [219]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [38]PETSC ERROR: #1 User provided function() line 0 in unknown file *** Error in `/global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec': free(): invalid pointer: 0x0000010000aa80b0 *** [110]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [110]PETSC ERROR: Signal received [110]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [110]PETSC ERROR: Petsc Release Version 3.11.3, unknown [110]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05536 by amollen Wed Feb 19 09:05:25 2020 [73]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [73]PETSC ERROR: Signal received [73]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [73]PETSC ERROR: Petsc Release Version 3.11.3, unknown [73]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05532 by amollen Wed Feb 19 09:05:25 2020 [73]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [85]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [85]PETSC ERROR: Petsc Release Version 3.11.3, unknown [85]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05533 by amollen Wed Feb 19 09:05:25 2020 [85]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [85]PETSC ERROR: #1 User provided function() line 0 in unknown file [93]PETSC ERROR: to get more information on the crash. [93]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [93]PETSC ERROR: Signal received [93]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [93]PETSC ERROR: Petsc Release Version 3.11.3, unknown [93]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05534 by amollen Wed Feb 19 09:05:25 2020 [93]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [22]PETSC ERROR: ------------------------------------------------------------------------ [22]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [22]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [22]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [22]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [115]PETSC ERROR: ------------------------------------------------------------------------ [115]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [115]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [115]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [115]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [235]PETSC ERROR: ------------------------------------------------------------------------ [235]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [235]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [235]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [235]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [235]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [235]PETSC ERROR: to get more information on the crash. [235]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [235]PETSC ERROR: Signal received [235]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Failed during initial partitioning [29]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [29]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [29]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [10]PETSC ERROR: Petsc Release Version 3.11.3, unknown [10]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04186 by amollen Wed Feb 19 09:05:25 2020 [10]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [10]PETSC ERROR: #1 User provided function() line 0 in unknown file [163]PETSC ERROR: ------------------------------------------------------------------------ [163]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [103]PETSC ERROR: ------------------------------------------------------------------------ [103]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [103]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [103]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [103]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [103]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [103]PETSC ERROR: to get more information on the crash. [103]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [103]PETSC ERROR: Signal received [103]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [103]PETSC ERROR: Petsc Release Version 3.11.3, unknown *** Error in `/global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec': free(): invalid size: 0x000001000046f690 *** [157]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [157]PETSC ERROR: to get more information on the crash. [157]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [157]PETSC ERROR: Signal received [157]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [157]PETSC ERROR: Petsc Release Version 3.11.3, unknown [215]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [215]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [215]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [204]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [204]PETSC ERROR: Petsc Release Version 3.11.3, unknown [204]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07257 by amollen Wed Feb 19 09:05:25 2020 [204]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [177]PETSC ERROR: Signal received [177]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [177]PETSC ERROR: Petsc Release Version 3.11.3, unknown [177]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07206 by amollen Wed Feb 19 09:05:25 2020 [191]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [191]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [191]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [191]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [126]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [126]PETSC ERROR: Signal received [126]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [126]PETSC ERROR: Petsc Release Version 3.11.3, unknown [126]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06136 by amollen Wed Feb 19 09:05:25 2020 [126]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [199]PETSC ERROR: ------------------------------------------------------------------------ [199]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [199]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [199]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [199]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [199]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [199]PETSC ERROR: to get more information on the crash. [130]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [130]PETSC ERROR: #1 User provided function() line 0 in unknown file [168]PETSC ERROR: ------------------------------------------------------------------------ Rank 68 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 68 Failed during initial partitioning [143]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [143]PETSC ERROR: to get more information on the crash. [143]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [146]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06139 by amollen Wed Feb 19 09:05:25 2020 [146]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [146]PETSC ERROR: #1 User provided function() line 0 in unknown file [60]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [60]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [60]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [60]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [60]PETSC ERROR: to get more information on the crash. [60]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [49]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [49]PETSC ERROR: #1 User provided function() line 0 in unknown file [245]PETSC ERROR: ------------------------------------------------------------------------ [245]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [245]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [245]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [245]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [219]PETSC ERROR: Signal received [219]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [219]PETSC ERROR: Petsc Release Version 3.11.3, unknown [39]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04189 by amollen Wed Feb 19 09:05:25 2020 [39]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [39]PETSC ERROR: #1 User provided function() line 0 in unknown file [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [105]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [105]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [105]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [105]PETSC ERROR: to get more information on the crash. [75]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05532 by amollen Wed Feb 19 09:05:25 2020 [75]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [75]PETSC ERROR: #1 User provided function() line 0 in unknown file [81]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [81]PETSC ERROR: #1 User provided function() line 0 in unknown file [90]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05534 by amollen Wed Feb 19 09:05:25 2020 [90]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [90]PETSC ERROR: #1 User provided function() line 0 in unknown file [18]PETSC ERROR: Signal received [18]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [18]PETSC ERROR: Petsc Release Version 3.11.3, unknown [18]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04187 by amollen Wed Feb 19 09:05:25 2020 [18]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [18]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning [237]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [237]PETSC ERROR: to get more information on the crash. [237]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [237]PETSC ERROR: Signal received [237]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [237]PETSC ERROR: Petsc Release Version 3.11.3, unknown [224]PETSC ERROR: ------------------------------------------------------------------------ [224]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [31]PETSC ERROR: ------------------------------------------------------------------------ [31]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [31]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [31]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [11]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04186 by amollen Wed Feb 19 09:05:25 2020 [11]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [11]PETSC ERROR: #1 User provided function() line 0 in unknown file [166]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [166]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [166]PETSC ERROR: to get more information on the crash. [103]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05535 by amollen Wed Feb 19 09:05:25 2020 *** Error in `/global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec': free(): invalid size: 0x000001000046f690 *** [158]PETSC ERROR: ------------------------------------------------------------------------ [158]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [158]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [158]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [158]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [158]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [158]PETSC ERROR: to get more information on the crash. [158]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [158]PETSC ERROR: Signal received [158]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [158]PETSC ERROR: Petsc Release Version 3.11.3, unknown [215]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [215]PETSC ERROR: to get more information on the crash. [202]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [202]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [202]PETSC ERROR: to get more information on the crash. [181]PETSC ERROR: Signal received [181]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [181]PETSC ERROR: Petsc Release Version 3.11.3, unknown [190]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07255 by amollen Wed Feb 19 09:05:25 2020 [190]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [190]PETSC ERROR: #1 User provided function() line 0 in unknown file [121]PETSC ERROR: Signal received [121]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [121]PETSC ERROR: Petsc Release Version 3.11.3, unknown [121]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06136 by amollen Wed Feb 19 09:05:25 2020 [121]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [121]PETSC ERROR: #1 User provided function() line 0 in unknown file *** Error in `/global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec': free(): invalid pointer: 0x0000010000457dc0 *** [131]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [131]PETSC ERROR: Petsc Release Version 3.11.3, unknown [131]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06137 by amollen Wed Feb 19 09:05:25 2020 [131]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [131]PETSC ERROR: #1 User provided function() line 0 in unknown file [171]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [171]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [171]PETSC ERROR: to get more information on the crash. [171]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [171]PETSC ERROR: Signal received [64]PETSC ERROR: Signal received [64]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [139]PETSC ERROR: ------------------------------------------------------------------------ [147]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06139 by amollen Wed Feb 19 09:05:25 2020 [147]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [147]PETSC ERROR: #1 User provided function() line 0 in unknown file [62]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [62]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [62]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [62]PETSC ERROR: to get more information on the crash. [62]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [62]PETSC ERROR: Signal received Rank 49 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 49 [246]PETSC ERROR: ------------------------------------------------------------------------ [246]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [246]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [246]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [246]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [219]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07872 by amollen Wed Feb 19 09:05:25 2020 [219]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Rank 36 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 36 [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [108]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05536 by amollen Wed Feb 19 09:05:25 2020 [108]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [108]PETSC ERROR: #1 User provided function() line 0 in unknown file [76]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [76]PETSC ERROR: #1 User provided function() line 0 in unknown file [83]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [83]PETSC ERROR: to get more information on the crash. [83]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [83]PETSC ERROR: Signal received [83]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [83]PETSC ERROR: Petsc Release Version 3.11.3, unknown [83]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05533 by amollen Wed Feb 19 09:05:25 2020 [91]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05534 by amollen Wed Feb 19 09:05:25 2020 [91]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [91]PETSC ERROR: #1 User provided function() line 0 in unknown file [20]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [20]PETSC ERROR: Signal received [20]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [20]PETSC ERROR: Petsc Release Version 3.11.3, unknown [20]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04187 by amollen Wed Feb 19 09:05:25 2020 [114]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [114]PETSC ERROR: to get more information on the crash. [114]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [114]PETSC ERROR: Signal received [114]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [238]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [238]PETSC ERROR: Signal received [238]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [238]PETSC ERROR: Petsc Release Version 3.11.3, unknown [238]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07975 by amollen Wed Feb 19 09:05:25 2020 [238]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [224]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [224]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [29]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [29]PETSC ERROR: to get more information on the crash. [29]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [13]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [13]PETSC ERROR: Signal received [13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [13]PETSC ERROR: Petsc Release Version 3.11.3, unknown [13]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04186 by amollen Wed Feb 19 09:05:25 2020 [13]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [163]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [163]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [97]PETSC ERROR: ------------------------------------------------------------------------ [97]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [97]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [97]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [97]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [97]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [97]PETSC ERROR: to get more information on the crash. [97]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [97]PETSC ERROR: Signal received [97]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [97]PETSC ERROR: Petsc Release Version 3.11.3, unknown [253]PETSC ERROR: ------------------------------------------------------------------------ [253]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [253]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [158]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07203 by amollen Wed Feb 19 09:05:25 2020 [215]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [204]PETSC ERROR: #1 User provided function() line 0 in unknown file [183]PETSC ERROR: ------------------------------------------------------------------------ [183]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [183]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [183]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [183]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [191]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [191]PETSC ERROR: to get more information on the crash. [191]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [122]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [122]PETSC ERROR: Signal received [122]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [122]PETSC ERROR: Petsc Release Version 3.11.3, unknown [122]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06136 by amollen Wed Feb 19 09:05:25 2020 [122]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [194]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [194]PETSC ERROR: Petsc Release Version 3.11.3, unknown [194]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07256 by amollen Wed Feb 19 09:05:25 2020 [194]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [194]PETSC ERROR: #1 User provided function() line 0 in unknown file [133]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06137 by amollen Wed Feb 19 09:05:25 2020 [133]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [133]PETSC ERROR: #1 User provided function() line 0 in unknown file [174]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [174]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [174]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [174]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [174]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [174]PETSC ERROR: to get more information on the crash. [143]PETSC ERROR: Signal received [143]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [143]PETSC ERROR: Petsc Release Version 3.11.3, unknown [149]PETSC ERROR: #1 User provided function() line 0 in unknown file [56]PETSC ERROR: ------------------------------------------------------------------------ [56]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [56]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [56]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [50]PETSC ERROR: ------------------------------------------------------------------------ [50]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [50]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [240]PETSC ERROR: ------------------------------------------------------------------------ [240]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [219]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 38 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 38 [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [110]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [110]PETSC ERROR: #1 User provided function() line 0 in unknown file [78]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [78]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [78]PETSC ERROR: to get more information on the crash. [78]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [78]PETSC ERROR: Signal received [78]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [78]PETSC ERROR: Petsc Release Version 3.11.3, unknown Rank 81 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 81 [92]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [92]PETSC ERROR: #1 User provided function() line 0 in unknown file [22]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [22]PETSC ERROR: to get more information on the crash. [22]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [22]PETSC ERROR: Signal received [22]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [115]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [115]PETSC ERROR: to get more information on the crash. [115]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [115]PETSC ERROR: Signal received [233]PETSC ERROR: Signal received [233]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [233]PETSC ERROR: Petsc Release Version 3.11.3, unknown [233]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07975 by amollen Wed Feb 19 09:05:25 2020 [233]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [233]PETSC ERROR: #1 User provided function() line 0 in unknown file [224]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [224]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [224]PETSC ERROR: to get more information on the crash. [31]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [31]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [31]PETSC ERROR: to get more information on the crash. [13]PETSC ERROR: #1 User provided function() line 0 in unknown file [166]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [97]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05535 by amollen Wed Feb 19 09:05:25 2020 [97]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [97]PETSC ERROR: #1 User provided function() line 0 in unknown file [253]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [253]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [253]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [253]PETSC ERROR: to get more information on the crash. [152]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [152]PETSC ERROR: to get more information on the crash. [152]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [152]PETSC ERROR: Signal received [152]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [152]PETSC ERROR: Petsc Release Version 3.11.3, unknown [152]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07203 by amollen Wed Feb 19 09:05:25 2020 [215]PETSC ERROR: Signal received [215]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [201]PETSC ERROR: ------------------------------------------------------------------------ [201]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [177]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [177]PETSC ERROR: #1 User provided function() line 0 in unknown file [191]PETSC ERROR: Signal received [191]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [125]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06136 by amollen Wed Feb 19 09:05:25 2020 [125]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [125]PETSC ERROR: #1 User provided function() line 0 in unknown file [195]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [195]PETSC ERROR: to get more information on the crash. [195]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [195]PETSC ERROR: Signal received [195]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [195]PETSC ERROR: Petsc Release Version 3.11.3, unknown [195]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07256 by amollen Wed Feb 19 09:05:25 2020 [129]PETSC ERROR: ------------------------------------------------------------------------ [129]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [175]PETSC ERROR: ------------------------------------------------------------------------ [175]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [175]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [175]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [175]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [175]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [175]PETSC ERROR: to get more information on the crash. [175]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [64]PETSC ERROR: Petsc Release Version 3.11.3, unknown [64]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04193 by amollen Wed Feb 19 09:05:25 2020 [139]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [139]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [139]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Rank 147 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n3] application called MPI_Abort(comm=0x84000002, 59) - process 147 [57]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [57]PETSC ERROR: to get more information on the crash. [57]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [57]PETSC ERROR: Signal received [57]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [57]PETSC ERROR: Petsc Release Version 3.11.3, unknown [57]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04192 by amollen Wed Feb 19 09:05:25 2020 Failed during initial partitioning [242]PETSC ERROR: ------------------------------------------------------------------------ [242]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [242]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [242]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Rank 219 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n0] application called MPI_Abort(comm=0x84000002, 59) - process 219 Failed during initial partitioning [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [2]PETSC ERROR: to get more information on the crash. [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Signal received [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.11.3, unknown [104]PETSC ERROR: ------------------------------------------------------------------------ [104]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [104]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [104]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [73]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 73 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 73 [83]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [83]PETSC ERROR: #1 User provided function() line 0 in unknown file [93]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 93 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 93 [20]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [20]PETSC ERROR: #1 User provided function() line 0 in unknown file [117]PETSC ERROR: ------------------------------------------------------------------------ [117]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [117]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [117]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [117]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [117]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [117]PETSC ERROR: to get more information on the crash. [117]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [234]PETSC ERROR: ------------------------------------------------------------------------ [234]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [234]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [234]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [234]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [234]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [234]PETSC ERROR: to get more information on the crash. [234]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [224]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [29]PETSC ERROR: Signal received [29]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [29]PETSC ERROR: Petsc Release Version 3.11.3, unknown [29]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04188 by amollen Wed Feb 19 09:05:25 2020 [8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [8]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [8]PETSC ERROR: to get more information on the crash. [167]PETSC ERROR: ------------------------------------------------------------------------ [167]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [98]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [98]PETSC ERROR: to get more information on the crash. [98]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [98]PETSC ERROR: Signal received [98]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [98]PETSC ERROR: Petsc Release Version 3.11.3, unknown [98]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05535 by amollen Wed Feb 19 09:05:25 2020 [255]PETSC ERROR: ------------------------------------------------------------------------ [255]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [255]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [255]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [255]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [255]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [255]PETSC ERROR: to get more information on the crash. [152]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [152]PETSC ERROR: #1 User provided function() line 0 in unknown file [215]PETSC ERROR: Petsc Release Version 3.11.3, unknown [202]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [202]PETSC ERROR: Signal received [202]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [181]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07206 by amollen Wed Feb 19 09:05:25 2020 [181]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [181]PETSC ERROR: #1 User provided function() line 0 in unknown file [191]PETSC ERROR: Petsc Release Version 3.11.3, unknown [126]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 126 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n0] application called MPI_Abort(comm=0x84000002, 59) - process 126 [196]PETSC ERROR: ------------------------------------------------------------------------ [196]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [196]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [196]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [196]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [196]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [196]PETSC ERROR: to get more information on the crash. [196]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [196]PETSC ERROR: Signal received [196]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [196]PETSC ERROR: Petsc Release Version 3.11.3, unknown Rank 130 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n1] application called MPI_Abort(comm=0x84000002, 59) - process 130 [168]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [168]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [168]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [168]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [168]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [168]PETSC ERROR: to get more information on the crash. [64]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [143]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06138 by amollen Wed Feb 19 09:05:25 2020 [143]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Rank 146 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n3] application called MPI_Abort(comm=0x84000002, 59) - process 146 [57]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [50]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [50]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [50]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [243]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07976 by amollen Wed Feb 19 09:05:25 2020 [243]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [243]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 243 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n0] application called MPI_Abort(comm=0x84000002, 59) - process 243 Failed during initial partitioning Failed during initial partitioning [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [4]PETSC ERROR: to get more information on the crash. [4]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [4]PETSC ERROR: Signal received [105]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [105]PETSC ERROR: Signal received [105]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [105]PETSC ERROR: Petsc Release Version 3.11.3, unknown [105]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05536 by amollen Wed Feb 19 09:05:25 2020 [105]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [74]PETSC ERROR: ------------------------------------------------------------------------ [74]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [74]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [74]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [74]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [74]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [74]PETSC ERROR: to get more information on the crash. Rank 84 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 84 Failed during initial partitioning [94]PETSC ERROR: ------------------------------------------------------------------------ [94]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [94]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [94]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [94]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [94]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [94]PETSC ERROR: to get more information on the crash. [22]PETSC ERROR: Petsc Release Version 3.11.3, unknown [22]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04187 by amollen Wed Feb 19 09:05:25 2020 [22]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [114]PETSC ERROR: Petsc Release Version 3.11.3, unknown [114]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05537 by amollen Wed Feb 19 09:05:25 2020 [114]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [114]PETSC ERROR: #1 User provided function() line 0 in unknown file [235]PETSC ERROR: Petsc Release Version 3.11.3, unknown [235]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07975 by amollen Wed Feb 19 09:05:25 2020 [235]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [235]PETSC ERROR: #1 User provided function() line 0 in unknown file [224]PETSC ERROR: Signal received [31]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [31]PETSC ERROR: Signal received [31]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Rank 10 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n2] application called MPI_Abort(comm=0x84000002, 59) - process 10 Failed during initial partitioning [160]PETSC ERROR: ------------------------------------------------------------------------ [160]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [160]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [160]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [98]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [98]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 98 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 98 Failed during initial partitioning [253]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [253]PETSC ERROR: Signal received [253]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [153]PETSC ERROR: ------------------------------------------------------------------------ [153]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [153]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [153]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [153]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [153]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [153]PETSC ERROR: to get more information on the crash. [215]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07871 by amollen Wed Feb 19 09:05:25 2020 [215]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [201]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [201]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [201]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [183]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [183]PETSC ERROR: to get more information on the crash. [183]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [183]PETSC ERROR: Signal received Rank 190 [Wed Feb 19 09:05:35 2020] [c1-3c2s5n3] application called MPI_Abort(comm=0x84000002, 59) - process 190 Rank 121 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n0] application called MPI_Abort(comm=0x84000002, 59) - process 121 Failed during initial partitioning [198]PETSC ERROR: Petsc Release Version 3.11.3, unknown [198]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07256 by amollen Wed Feb 19 09:05:25 2020 [198]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [198]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 198 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n0] application called MPI_Abort(comm=0x84000002, 59) - process 198 [129]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [129]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [129]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [171]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [171]PETSC ERROR: Petsc Release Version 3.11.3, unknown [171]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07205 by amollen Wed Feb 19 09:05:25 2020 [171]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [171]PETSC ERROR: #1 User provided function() line 0 in unknown file [64]PETSC ERROR: #1 User provided function() line 0 in unknown file [139]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [139]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [139]PETSC ERROR: to get more information on the crash. Rank 149 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n3] application called MPI_Abort(comm=0x84000002, 59) - process 149 Failed during initial partitioning [58]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [58]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [58]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [58]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [58]PETSC ERROR: to get more information on the crash. [58]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [58]PETSC ERROR: Signal received [58]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [58]PETSC ERROR: Petsc Release Version 3.11.3, unknown [58]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04192 by amollen Wed Feb 19 09:05:25 2020 [50]PETSC ERROR: to get more information on the crash. [50]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 244 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n0] application called MPI_Abort(comm=0x84000002, 59) - process 244 Failed during initial partitioning [217]PETSC ERROR: ------------------------------------------------------------------------ [217]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range Rank 39 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 39 Failed during initial partitioning [2]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04185 by amollen Wed Feb 19 09:05:25 2020 [2]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [2]PETSC ERROR: #1 User provided function() line 0 in unknown file [104]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [104]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [104]PETSC ERROR: to get more information on the crash. Rank 76 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 76 Failed during initial partitioning Rank 85 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 85 Failed during initial partitioning Rank 90 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 90 Failed during initial partitioning [17]PETSC ERROR: ------------------------------------------------------------------------ [17]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [115]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [115]PETSC ERROR: Petsc Release Version 3.11.3, unknown [115]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05537 by amollen Wed Feb 19 09:05:25 2020 [115]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [115]PETSC ERROR: #1 User provided function() line 0 in unknown file [236]PETSC ERROR: ------------------------------------------------------------------------ [236]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [236]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [236]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [236]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [236]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [236]PETSC ERROR: to get more information on the crash. [224]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [224]PETSC ERROR: Petsc Release Version 3.11.3, unknown [25]PETSC ERROR: ------------------------------------------------------------------------ Rank 11 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n2] application called MPI_Abort(comm=0x84000002, 59) - process 11 Failed during initial partitioning [163]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [163]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [163]PETSC ERROR: to get more information on the crash. [163]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [163]PETSC ERROR: Signal received [99]PETSC ERROR: to get more information on the crash. [99]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [99]PETSC ERROR: Signal received [99]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [99]PETSC ERROR: Petsc Release Version 3.11.3, unknown [99]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05535 by amollen Wed Feb 19 09:05:25 2020 [99]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [254]PETSC ERROR: ------------------------------------------------------------------------ [254]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [254]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [254]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [154]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [154]PETSC ERROR: Petsc Release Version 3.11.3, unknown [154]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07203 by amollen Wed Feb 19 09:05:25 2020 [154]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [154]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 154 [Wed Feb 19 09:05:35 2020] [c1-3c1s8n3] application called MPI_Abort(comm=0x84000002, 59) - process 154 [215]PETSC ERROR: #1 User provided function() line 0 in unknown file [202]PETSC ERROR: Petsc Release Version 3.11.3, unknown [202]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07257 by amollen Wed Feb 19 09:05:25 2020 [202]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [183]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [183]PETSC ERROR: Petsc Release Version 3.11.3, unknown [191]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07255 by amollen Wed Feb 19 09:05:25 2020 [191]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [191]PETSC ERROR: #1 User provided function() line 0 in unknown file [122]PETSC ERROR: #1 User provided function() line 0 in unknown file [199]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [199]PETSC ERROR: Signal received [199]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [199]PETSC ERROR: Petsc Release Version 3.11.3, unknown [199]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07256 by amollen Wed Feb 19 09:05:25 2020 [199]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Failed during initial partitioning [174]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [174]PETSC ERROR: Signal received [174]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [174]PETSC ERROR: Petsc Release Version 3.11.3, unknown [174]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07205 by amollen Wed Feb 19 09:05:25 2020 [174]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Rank 64 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n1] application called MPI_Abort(comm=0x84000004, 59) - process 64 Rank 142 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n2] application called MPI_Abort(comm=0x84000002, 59) - process 142 Failed during initial partitioning Failed during initial partitioning [60]PETSC ERROR: Signal received [60]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [60]PETSC ERROR: Petsc Release Version 3.11.3, unknown [60]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04192 by amollen Wed Feb 19 09:05:25 2020 [60]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [60]PETSC ERROR: #1 User provided function() line 0 in unknown file [51]PETSC ERROR: ------------------------------------------------------------------------ [51]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [245]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [245]PETSC ERROR: to get more information on the crash. [245]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [245]PETSC ERROR: Signal received [245]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [245]PETSC ERROR: Petsc Release Version 3.11.3, unknown [245]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07976 by amollen Wed Feb 19 09:05:25 2020 [217]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [217]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [217]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [32]PETSC ERROR: ------------------------------------------------------------------------ [32]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [32]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [4]PETSC ERROR: Petsc Release Version 3.11.3, unknown [4]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04185 by amollen Wed Feb 19 09:05:25 2020 [4]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [4]PETSC ERROR: #1 User provided function() line 0 in unknown file [105]PETSC ERROR: #1 User provided function() line 0 in unknown file [78]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05532 by amollen Wed Feb 19 09:05:25 2020 [78]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [78]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning Rank 91 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 91 Failed during initial partitioning Rank 18 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n3] application called MPI_Abort(comm=0x84000002, 59) - process 18 Failed during initial partitioning [117]PETSC ERROR: Signal received [117]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [117]PETSC ERROR: Petsc Release Version 3.11.3, unknown [117]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05537 by amollen Wed Feb 19 09:05:25 2020 [117]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [237]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07975 by amollen Wed Feb 19 09:05:25 2020 [237]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [237]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 237 [Wed Feb 19 09:05:35 2020] [c5-3c1s9n3] application called MPI_Abort(comm=0x84000002, 59) - process 237 [224]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07873 by amollen Wed Feb 19 09:05:25 2020 [224]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [29]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [29]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 14 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n2] application called MPI_Abort(comm=0x84000002, 59) - process 14 Failed during initial partitioning [165]PETSC ERROR: ------------------------------------------------------------------------ [165]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [165]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [165]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [99]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 99 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 99 Failed during initial partitioning [255]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [255]PETSC ERROR: Signal received [255]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [255]PETSC ERROR: Petsc Release Version 3.11.3, unknown [156]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [156]PETSC ERROR: Signal received [156]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [156]PETSC ERROR: Petsc Release Version 3.11.3, unknown [156]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07203 by amollen Wed Feb 19 09:05:25 2020 [156]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [213]PETSC ERROR: ------------------------------------------------------------------------ [206]PETSC ERROR: ------------------------------------------------------------------------ [206]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [206]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [206]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Rank 177 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n2] application called MPI_Abort(comm=0x84000002, 59) - process 177 Failed during initial partitioning Failed during initial partitioning Rank 125 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n0] application called MPI_Abort(comm=0x84000002, 59) - process 125 Failed during initial partitioning [199]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 131 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n1] application called MPI_Abort(comm=0x84000002, 59) - process 131 Failed during initial partitioning [174]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning [143]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning [62]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [62]PETSC ERROR: Petsc Release Version 3.11.3, unknown [62]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04192 by amollen Wed Feb 19 09:05:25 2020 [62]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [62]PETSC ERROR: #1 User provided function() line 0 in unknown file [52]PETSC ERROR: ------------------------------------------------------------------------ [52]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [52]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [52]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [52]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [245]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [218]PETSC ERROR: ------------------------------------------------------------------------ [32]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [32]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [32]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run Rank 2 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 2 [107]PETSC ERROR: ------------------------------------------------------------------------ [107]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [72]PETSC ERROR: ------------------------------------------------------------------------ [72]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range Rank 83 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 83 Rank 92 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 92 Failed during initial partitioning Rank 20 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n3] application called MPI_Abort(comm=0x84000002, 59) - process 20 Failed during initial partitioning Rank 114 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 114 [238]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 238 [Wed Feb 19 09:05:35 2020] [c5-3c1s9n3] application called MPI_Abort(comm=0x84000002, 59) - process 238 Failed during initial partitioning [224]PETSC ERROR: #1 User provided function() line 0 in unknown file [31]PETSC ERROR: Petsc Release Version 3.11.3, unknown [31]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04188 by amollen Wed Feb 19 09:05:25 2020 [31]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [31]PETSC ERROR: #1 User provided function() line 0 in unknown file [8]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [8]PETSC ERROR: Signal received [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [166]PETSC ERROR: Signal received [166]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [166]PETSC ERROR: Petsc Release Version 3.11.3, unknown [166]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07204 by amollen Wed Feb 19 09:05:25 2020 [166]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [166]PETSC ERROR: #1 User provided function() line 0 in unknown file [100]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [100]PETSC ERROR: Signal received [100]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [100]PETSC ERROR: Petsc Release Version 3.11.3, unknown [100]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05535 by amollen Wed Feb 19 09:05:25 2020 [100]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [253]PETSC ERROR: Petsc Release Version 3.11.3, unknown [253]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07977 by amollen Wed Feb 19 09:05:25 2020 [253]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [253]PETSC ERROR: #1 User provided function() line 0 in unknown file [156]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 156 [Wed Feb 19 09:05:35 2020] [c1-3c1s8n3] application called MPI_Abort(comm=0x84000002, 59) - process 156 Rank 215 [Wed Feb 19 09:05:35 2020] [c4-3c2s15n3] application called MPI_Abort(comm=0x84000002, 59) - process 215 Failed during initial partitioning [207]PETSC ERROR: ------------------------------------------------------------------------ [207]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [207]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [207]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [207]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [207]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [207]PETSC ERROR: to get more information on the crash. Rank 181 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n2] application called MPI_Abort(comm=0x84000002, 59) - process 181 Failed during initial partitioning Rank 191 [Wed Feb 19 09:05:35 2020] [c1-3c2s5n3] application called MPI_Abort(comm=0x84000002, 59) - process 191 Failed during initial partitioning Rank 194 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n0] application called MPI_Abort(comm=0x84000002, 59) - process 194 Failed during initial partitioning Rank 133 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n1] application called MPI_Abort(comm=0x84000002, 59) - process 133 Failed during initial partitioning [175]PETSC ERROR: Signal received [175]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [175]PETSC ERROR: Petsc Release Version 3.11.3, unknown [175]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07205 by amollen Wed Feb 19 09:05:25 2020 [175]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [175]PETSC ERROR: #1 User provided function() line 0 in unknown file [65]PETSC ERROR: ------------------------------------------------------------------------ [65]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [139]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [139]PETSC ERROR: Signal received [139]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [56]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [56]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [56]PETSC ERROR: to get more information on the crash. [56]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [56]PETSC ERROR: Signal received [56]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [56]PETSC ERROR: Petsc Release Version 3.11.3, unknown [56]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04192 by amollen Wed Feb 19 09:05:25 2020 [50]PETSC ERROR: Signal received [50]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [50]PETSC ERROR: Petsc Release Version 3.11.3, unknown [50]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04191 by amollen Wed Feb 19 09:05:25 2020 [50]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [50]PETSC ERROR: #1 User provided function() line 0 in unknown file [246]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [246]PETSC ERROR: to get more information on the crash. [246]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [246]PETSC ERROR: Signal received [246]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [246]PETSC ERROR: Petsc Release Version 3.11.3, unknown [246]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07976 by amollen Wed Feb 19 09:05:25 2020 [220]PETSC ERROR: ------------------------------------------------------------------------ [220]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [32]PETSC ERROR: to get more information on the crash. [32]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Failed during initial partitioning Rank 110 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 110 Failed during initial partitioning Failed during initial partitioning Failed during initial partitioning Failed during initial partitioning [22]PETSC ERROR: #1 User provided function() line 0 in unknown file [117]PETSC ERROR: #1 User provided function() line 0 in unknown file [232]PETSC ERROR: ------------------------------------------------------------------------ [232]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [232]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [232]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [232]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [232]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [232]PETSC ERROR: to get more information on the crash. [232]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 224 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n1] application called MPI_Abort(comm=0x84000004, 59) - process 224 [25]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [25]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [25]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [25]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [25]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run Rank 13 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n2] application called MPI_Abort(comm=0x84000002, 59) - process 13 Failed during initial partitioning [167]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [167]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [167]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [167]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [167]PETSC ERROR: to get more information on the crash. [167]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [100]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 100 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 100 Failed during initial partitioning [254]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [254]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [254]PETSC ERROR: to get more information on the crash. [254]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [157]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07203 by amollen Wed Feb 19 09:05:25 2020 [157]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [157]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 157 [Wed Feb 19 09:05:35 2020] [c1-3c1s8n3] application called MPI_Abort(comm=0x84000002, 59) - process 157 Failed during initial partitioning [213]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [213]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [213]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [213]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [213]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [201]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [201]PETSC ERROR: to get more information on the crash. [201]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [201]PETSC ERROR: Signal received [201]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [201]PETSC ERROR: Petsc Release Version 3.11.3, unknown [183]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07206 by amollen Wed Feb 19 09:05:25 2020 [183]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [183]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning [127]PETSC ERROR: ------------------------------------------------------------------------ [127]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [127]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [127]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [127]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [127]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [127]PETSC ERROR: to get more information on the crash. [127]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [195]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [195]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 195 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n0] application called MPI_Abort(comm=0x84000002, 59) - process 195 [129]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [129]PETSC ERROR: to get more information on the crash. [129]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [168]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [168]PETSC ERROR: Signal received [168]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [168]PETSC ERROR: Petsc Release Version 3.11.3, unknown [168]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07205 by amollen Wed Feb 19 09:05:25 2020 [168]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [66]PETSC ERROR: ------------------------------------------------------------------------ [66]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [66]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [66]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [140]PETSC ERROR: ------------------------------------------------------------------------ [148]PETSC ERROR: ------------------------------------------------------------------------ [148]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [56]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [56]PETSC ERROR: #1 User provided function() line 0 in unknown file [51]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [51]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [51]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [51]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [51]PETSC ERROR: to get more information on the crash. [246]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [246]PETSC ERROR: #1 User provided function() line 0 in unknown file [221]PETSC ERROR: ------------------------------------------------------------------------ [221]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [221]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [32]PETSC ERROR: Signal received Rank 4 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 4 Failed during initial partitioning [104]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [104]PETSC ERROR: Signal received [74]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [74]PETSC ERROR: Signal received [74]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [74]PETSC ERROR: Petsc Release Version 3.11.3, unknown [74]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05532 by amollen Wed Feb 19 09:05:25 2020 [87]PETSC ERROR: ------------------------------------------------------------------------ [94]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [94]PETSC ERROR: Signal received [94]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [94]PETSC ERROR: Petsc Release Version 3.11.3, unknown [94]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05534 by amollen Wed Feb 19 09:05:25 2020 [17]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [17]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [17]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [17]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [17]PETSC ERROR: to get more information on the crash. [119]PETSC ERROR: ------------------------------------------------------------------------ [119]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [119]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [119]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Rank 233 [Wed Feb 19 09:05:35 2020] [c5-3c1s9n3] application called MPI_Abort(comm=0x84000002, 59) - process 233 Failed during initial partitioning Failed during initial partitioning [25]PETSC ERROR: to get more information on the crash. [8]PETSC ERROR: Petsc Release Version 3.11.3, unknown [8]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04186 by amollen Wed Feb 19 09:05:25 2020 [8]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [160]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [160]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [160]PETSC ERROR: to get more information on the crash. [160]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [160]PETSC ERROR: Signal received [101]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [101]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 101 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 101 Failed during initial partitioning [255]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07977 by amollen Wed Feb 19 09:05:25 2020 [255]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [255]PETSC ERROR: #1 User provided function() line 0 in unknown file [158]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [158]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 158 [Wed Feb 19 09:05:35 2020] [c1-3c1s8n3] application called MPI_Abort(comm=0x84000002, 59) - process 158 Failed during initial partitioning [213]PETSC ERROR: to get more information on the crash. [202]PETSC ERROR: #1 User provided function() line 0 in unknown file [182]PETSC ERROR: ------------------------------------------------------------------------ [182]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [185]PETSC ERROR: ------------------------------------------------------------------------ [185]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range Rank 122 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n0] application called MPI_Abort(comm=0x84000002, 59) - process 122 Failed during initial partitioning [196]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07256 by amollen Wed Feb 19 09:05:25 2020 [196]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [196]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 196 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n0] application called MPI_Abort(comm=0x84000002, 59) - process 196 [129]PETSC ERROR: Signal received [129]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [169]PETSC ERROR: ------------------------------------------------------------------------ [169]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [169]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [65]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [65]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [65]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [65]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [139]PETSC ERROR: Petsc Release Version 3.11.3, unknown [139]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06138 by amollen Wed Feb 19 09:05:25 2020 [139]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [148]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [148]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [148]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [148]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [57]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 57 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 57 Failed during initial partitioning [52]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [52]PETSC ERROR: to get more information on the crash. [52]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [52]PETSC ERROR: Signal received [52]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [240]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [240]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [240]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [240]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [240]PETSC ERROR: to get more information on the crash. [240]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [240]PETSC ERROR: Signal received [240]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [240]PETSC ERROR: Petsc Release Version 3.11.3, unknown [217]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [217]PETSC ERROR: to get more information on the crash. [217]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [217]PETSC ERROR: Signal received [32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [32]PETSC ERROR: Petsc Release Version 3.11.3, unknown [32]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04189 by amollen Wed Feb 19 09:05:25 2020 [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [107]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [107]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [107]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [107]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [107]PETSC ERROR: to get more information on the crash. Rank 75 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 75 Failed during initial partitioning [87]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [94]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [94]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 22 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n3] application called MPI_Abort(comm=0x84000002, 59) - process 22 Failed during initial partitioning [234]PETSC ERROR: Signal received [234]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [234]PETSC ERROR: Petsc Release Version 3.11.3, unknown [234]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07975 by amollen Wed Feb 19 09:05:25 2020 [234]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [234]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 29 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 29 [8]PETSC ERROR: #1 User provided function() line 0 in unknown file [163]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [163]PETSC ERROR: Petsc Release Version 3.11.3, unknown [163]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07204 by amollen Wed Feb 19 09:05:25 2020 [163]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [163]PETSC ERROR: #1 User provided function() line 0 in unknown file [103]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [103]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 103 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 103 Failed during initial partitioning [252]PETSC ERROR: ------------------------------------------------------------------------ Rank 152 [Wed Feb 19 09:05:35 2020] [c1-3c1s8n3] application called MPI_Abort(comm=0x84000004, 59) - process 152 Failed during initial partitioning [213]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [213]PETSC ERROR: Signal received Rank 204 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 204 Failed during initial partitioning [182]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [188]PETSC ERROR: ------------------------------------------------------------------------ [188]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [188]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [188]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [127]PETSC ERROR: Signal received [127]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [127]PETSC ERROR: Petsc Release Version 3.11.3, unknown Failed during initial partitioning [129]PETSC ERROR: Petsc Release Version 3.11.3, unknown [129]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06137 by amollen Wed Feb 19 09:05:25 2020 Rank 171 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n1] application called MPI_Abort(comm=0x84000002, 59) - process 171 Failed during initial partitioning [66]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [66]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [66]PETSC ERROR: to get more information on the crash. [140]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [140]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [140]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [140]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [148]PETSC ERROR: to get more information on the crash. [148]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [148]PETSC ERROR: Signal received [58]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [58]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 58 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 58 Failed during initial partitioning [51]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [51]PETSC ERROR: Signal received [51]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [242]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [242]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [242]PETSC ERROR: to get more information on the crash. [242]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [242]PETSC ERROR: Signal received [242]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [242]PETSC ERROR: Petsc Release Version 3.11.3, unknown [242]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07976 by amollen Wed Feb 19 09:05:25 2020 [218]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [218]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [218]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [218]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [218]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [218]PETSC ERROR: to get more information on the crash. [32]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [32]PETSC ERROR: #1 User provided function() line 0 in unknown file [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [3]PETSC ERROR: to get more information on the crash. [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 108 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 108 Failed during initial partitioning [72]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [72]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [72]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [72]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [87]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [87]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [87]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 94 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 94 [17]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 115 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 115 Failed during initial partitioning Rank 235 [Wed Feb 19 09:05:35 2020] [c5-3c1s9n3] application called MPI_Abort(comm=0x84000002, 59) - process 235 Failed during initial partitioning [25]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [25]PETSC ERROR: Signal received [25]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [12]PETSC ERROR: ------------------------------------------------------------------------ [165]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [165]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [165]PETSC ERROR: to get more information on the crash. [165]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [165]PETSC ERROR: Signal received [165]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [165]PETSC ERROR: Petsc Release Version 3.11.3, unknown Rank 97 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 97 Failed during initial partitioning [254]PETSC ERROR: Signal received [254]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [254]PETSC ERROR: Petsc Release Version 3.11.3, unknown [254]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07977 by amollen Wed Feb 19 09:05:25 2020 [254]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [153]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [153]PETSC ERROR: Signal received [153]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [153]PETSC ERROR: Petsc Release Version 3.11.3, unknown [153]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07203 by amollen Wed Feb 19 09:05:25 2020 [153]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [213]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [213]PETSC ERROR: Petsc Release Version 3.11.3, unknown [213]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07871 by amollen Wed Feb 19 09:05:25 2020 [206]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [206]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [206]PETSC ERROR: to get more information on the crash. [206]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [206]PETSC ERROR: Signal received [206]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Rank 183 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n2] application called MPI_Abort(comm=0x84000002, 59) - process 183 [185]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [185]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [185]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [185]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [185]PETSC ERROR: to get more information on the crash. [120]PETSC ERROR: ------------------------------------------------------------------------ Failed during initial partitioning [129]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [173]PETSC ERROR: ------------------------------------------------------------------------ [173]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [173]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [173]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [173]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [173]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [173]PETSC ERROR: to get more information on the crash. [173]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [173]PETSC ERROR: Signal received [173]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [70]PETSC ERROR: ------------------------------------------------------------------------ [70]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [70]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [70]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [70]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [70]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run Rank 143 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n2] application called MPI_Abort(comm=0x84000002, 59) - process 143 Failed during initial partitioning [148]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [148]PETSC ERROR: Petsc Release Version 3.11.3, unknown [148]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06139 by amollen Wed Feb 19 09:05:25 2020 [59]PETSC ERROR: ------------------------------------------------------------------------ [59]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [59]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [59]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [59]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [59]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [59]PETSC ERROR: to get more information on the crash. [59]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [52]PETSC ERROR: Petsc Release Version 3.11.3, unknown [52]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04191 by amollen Wed Feb 19 09:05:25 2020 [52]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [242]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [242]PETSC ERROR: #1 User provided function() line 0 in unknown file [220]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [220]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [220]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [220]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [220]PETSC ERROR: to get more information on the crash. Rank 32 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n1] application called MPI_Abort(comm=0x84000004, 59) - process 32 [3]PETSC ERROR: Signal received [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.11.3, unknown [104]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [104]PETSC ERROR: Petsc Release Version 3.11.3, unknown [104]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05536 by amollen Wed Feb 19 09:05:25 2020 [104]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [74]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [74]PETSC ERROR: #1 User provided function() line 0 in unknown file [87]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [87]PETSC ERROR: to get more information on the crash. Failed during initial partitioning Failed during initial partitioning [119]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [119]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [119]PETSC ERROR: to get more information on the crash. [236]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [236]PETSC ERROR: Signal received [236]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [236]PETSC ERROR: Petsc Release Version 3.11.3, unknown [236]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07975 by amollen Wed Feb 19 09:05:25 2020 [236]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Failed during initial partitioning Rank 8 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n2] application called MPI_Abort(comm=0x84000004, 59) - process 8 [167]PETSC ERROR: Signal received [167]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [167]PETSC ERROR: Petsc Release Version 3.11.3, unknown [167]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07204 by amollen Wed Feb 19 09:05:25 2020 [167]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [167]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 102 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 102 Failed during initial partitioning [249]PETSC ERROR: ------------------------------------------------------------------------ [249]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [249]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [153]PETSC ERROR: #1 User provided function() line 0 in unknown file [213]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [207]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [207]PETSC ERROR: Signal received [207]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [207]PETSC ERROR: Petsc Release Version 3.11.3, unknown [207]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07257 by amollen Wed Feb 19 09:05:25 2020 [182]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [182]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [182]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [188]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [188]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [188]PETSC ERROR: to get more information on the crash. [127]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06136 by amollen Wed Feb 19 09:05:25 2020 [127]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [127]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning [129]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 174 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n1] application called MPI_Abort(comm=0x84000002, 59) - process 174 Failed during initial partitioning [65]PETSC ERROR: to get more information on the crash. [65]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [65]PETSC ERROR: Signal received [65]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [65]PETSC ERROR: Petsc Release Version 3.11.3, unknown [139]PETSC ERROR: #1 User provided function() line 0 in unknown file [148]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Rank 60 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 60 Failed during initial partitioning Rank 50 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 50 Failed during initial partitioning [221]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [221]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [221]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [221]PETSC ERROR: to get more information on the crash. [221]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Failed during initial partitioning [3]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04185 by amollen Wed Feb 19 09:05:25 2020 [3]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [3]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 105 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 105 Failed during initial partitioning Rank 78 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 78 Failed during initial partitioning [86]PETSC ERROR: ------------------------------------------------------------------------ [88]PETSC ERROR: ------------------------------------------------------------------------ [17]PETSC ERROR: Signal received [17]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Rank 117 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 117 Failed during initial partitioning [236]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 31 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 31 Failed during initial partitioning [12]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [12]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [160]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [160]PETSC ERROR: Petsc Release Version 3.11.3, unknown [160]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07204 by amollen Wed Feb 19 09:05:25 2020 [160]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [160]PETSC ERROR: #1 User provided function() line 0 in unknown file [96]PETSC ERROR: ------------------------------------------------------------------------ [252]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [252]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [252]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [252]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Failed during initial partitioning [213]PETSC ERROR: #1 User provided function() line 0 in unknown file [201]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07257 by amollen Wed Feb 19 09:05:25 2020 [201]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [201]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning [185]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [185]PETSC ERROR: Signal received [185]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [120]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [120]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger Rank 199 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n0] application called MPI_Abort(comm=0x84000002, 59) - process 199 Failed during initial partitioning Rank 129 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n1] application called MPI_Abort(comm=0x84000002, 59) - process 129 [168]PETSC ERROR: #1 User provided function() line 0 in unknown file [66]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [66]PETSC ERROR: Signal received [66]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [66]PETSC ERROR: Petsc Release Version 3.11.3, unknown [66]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04193 by amollen Wed Feb 19 09:05:25 2020 [66]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [140]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [140]PETSC ERROR: to get more information on the crash. [140]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [148]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 62 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 62 Failed during initial partitioning [51]PETSC ERROR: Petsc Release Version 3.11.3, unknown [51]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04191 by amollen Wed Feb 19 09:05:25 2020 [51]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [51]PETSC ERROR: #1 User provided function() line 0 in unknown file [245]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 245 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n0] application called MPI_Abort(comm=0x84000002, 59) - process 245 Failed during initial partitioning [217]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [217]PETSC ERROR: Petsc Release Version 3.11.3, unknown [217]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07872 by amollen Wed Feb 19 09:05:25 2020 [217]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [217]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 3 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 3 [107]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [107]PETSC ERROR: Signal received [107]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [72]PETSC ERROR: to get more information on the crash. [72]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [72]PETSC ERROR: Signal received [87]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [87]PETSC ERROR: Signal received [89]PETSC ERROR: ------------------------------------------------------------------------ [89]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [89]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [17]PETSC ERROR: Petsc Release Version 3.11.3, unknown [17]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04187 by amollen Wed Feb 19 09:05:25 2020 [119]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [119]PETSC ERROR: Signal received Failed during initial partitioning [25]PETSC ERROR: Petsc Release Version 3.11.3, unknown [25]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04188 by amollen Wed Feb 19 09:05:25 2020 [25]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [12]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [12]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 163 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n0] application called MPI_Abort(comm=0x84000002, 59) - process 163 Failed during initial partitioning [96]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [96]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger Rank 253 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n1] application called MPI_Abort(comm=0x84000002, 59) - process 253 Failed during initial partitioning Failed during initial partitioning [212]PETSC ERROR: ------------------------------------------------------------------------ Rank 202 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 202 Failed during initial partitioning [182]PETSC ERROR: to get more information on the crash. [182]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [188]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [188]PETSC ERROR: Signal received [188]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [120]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [120]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [192]PETSC ERROR: ------------------------------------------------------------------------ *** Error in `/global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec': corrupted size vs. prev_size: 0x0000010000aa13b0 *** [169]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [169]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [169]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [169]PETSC ERROR: to get more information on the crash. [169]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [70]PETSC ERROR: to get more information on the crash. [70]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [70]PETSC ERROR: Signal received [70]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [70]PETSC ERROR: Petsc Release Version 3.11.3, unknown [70]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04193 by amollen Wed Feb 19 09:05:25 2020 [137]PETSC ERROR: ------------------------------------------------------------------------ [144]PETSC ERROR: ------------------------------------------------------------------------ [144]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [63]PETSC ERROR: ------------------------------------------------------------------------ [63]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [63]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [63]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [63]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [63]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [63]PETSC ERROR: to get more information on the crash. [63]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [63]PETSC ERROR: Signal received [63]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [52]PETSC ERROR: #1 User provided function() line 0 in unknown file [240]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07976 by amollen Wed Feb 19 09:05:25 2020 [240]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [240]PETSC ERROR: #1 User provided function() line 0 in unknown file [218]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [218]PETSC ERROR: Signal received [218]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Failed during initial partitioning [104]PETSC ERROR: #1 User provided function() line 0 in unknown file [72]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [86]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [88]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [88]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [88]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [88]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [17]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [119]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [119]PETSC ERROR: Petsc Release Version 3.11.3, unknown [232]PETSC ERROR: Signal received [232]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [232]PETSC ERROR: Petsc Release Version 3.11.3, unknown [232]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07975 by amollen Wed Feb 19 09:05:25 2020 [232]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [232]PETSC ERROR: #1 User provided function() line 0 in unknown file [25]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning [165]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07204 by amollen Wed Feb 19 09:05:25 2020 [165]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [165]PETSC ERROR: #1 User provided function() line 0 in unknown file [96]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [254]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 153 [Wed Feb 19 09:05:35 2020] [c1-3c1s8n3] application called MPI_Abort(comm=0x84000002, 59) - process 153 [210]PETSC ERROR: ------------------------------------------------------------------------ [206]PETSC ERROR: Petsc Release Version 3.11.3, unknown [206]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07257 by amollen Wed Feb 19 09:05:25 2020 [206]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [206]PETSC ERROR: #1 User provided function() line 0 in unknown file [182]PETSC ERROR: Signal received [185]PETSC ERROR: Petsc Release Version 3.11.3, unknown [185]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07255 by amollen Wed Feb 19 09:05:25 2020 [185]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [120]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [120]PETSC ERROR: to get more information on the crash. [192]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [134]PETSC ERROR: ------------------------------------------------------------------------ [173]PETSC ERROR: Petsc Release Version 3.11.3, unknown [173]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07205 by amollen Wed Feb 19 09:05:25 2020 [173]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [173]PETSC ERROR: #1 User provided function() line 0 in unknown file [71]PETSC ERROR: ------------------------------------------------------------------------ [71]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [71]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [71]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [71]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [71]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [71]PETSC ERROR: to get more information on the crash. [140]PETSC ERROR: Signal received [140]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [140]PETSC ERROR: Petsc Release Version 3.11.3, unknown Rank 148 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n3] application called MPI_Abort(comm=0x84000002, 59) - process 148 Failed during initial partitioning Rank 56 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n0] application called MPI_Abort(comm=0x84000004, 59) - process 56 Failed during initial partitioning Failed during initial partitioning Rank 242 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n0] application called MPI_Abort(comm=0x84000002, 59) - process 242 Failed during initial partitioning [220]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [220]PETSC ERROR: Signal received [220]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [220]PETSC ERROR: Petsc Release Version 3.11.3, unknown [107]PETSC ERROR: Petsc Release Version 3.11.3, unknown [107]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05536 by amollen Wed Feb 19 09:05:25 2020 [107]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [72]PETSC ERROR: Petsc Release Version 3.11.3, unknown [72]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05532 by amollen Wed Feb 19 09:05:25 2020 [87]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [87]PETSC ERROR: Petsc Release Version 3.11.3, unknown [89]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [89]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [89]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [89]PETSC ERROR: to get more information on the crash. [17]PETSC ERROR: #1 User provided function() line 0 in unknown file [119]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05537 by amollen Wed Feb 19 09:05:25 2020 [119]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Rank 234 [Wed Feb 19 09:05:35 2020] [c5-3c1s9n3] application called MPI_Abort(comm=0x84000002, 59) - process 234 Failed during initial partitioning Rank 25 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 25 [12]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [12]PETSC ERROR: to get more information on the crash. Rank 166 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n0] application called MPI_Abort(comm=0x84000002, 59) - process 166 Failed during initial partitioning [96]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 255 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n1] application called MPI_Abort(comm=0x84000002, 59) - process 255 Failed during initial partitioning Failed during initial partitioning [212]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [212]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [212]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [207]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [207]PETSC ERROR: #1 User provided function() line 0 in unknown file [182]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [182]PETSC ERROR: Petsc Release Version 3.11.3, unknown [188]PETSC ERROR: Petsc Release Version 3.11.3, unknown [188]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07255 by amollen Wed Feb 19 09:05:25 2020 [188]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [188]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 127 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n0] application called MPI_Abort(comm=0x84000002, 59) - process 127 [192]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [192]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [134]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [134]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger Rank 175 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n1] application called MPI_Abort(comm=0x84000002, 59) - process 175 Failed during initial partitioning [65]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04193 by amollen Wed Feb 19 09:05:25 2020 [65]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [65]PETSC ERROR: #1 User provided function() line 0 in unknown file [137]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [137]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [137]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [151]PETSC ERROR: ------------------------------------------------------------------------ [151]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [151]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [151]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [151]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 52 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 52 Rank 246 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n0] application called MPI_Abort(comm=0x84000002, 59) - process 246 Failed during initial partitioning [221]PETSC ERROR: Signal received [221]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [221]PETSC ERROR: Petsc Release Version 3.11.3, unknown [221]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07872 by amollen Wed Feb 19 09:05:25 2020 [221]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [221]PETSC ERROR: #1 User provided function() line 0 in unknown file [107]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 74 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 74 Failed during initial partitioning [86]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [86]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [86]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [88]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [88]PETSC ERROR: to get more information on the crash. [88]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 17 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n3] application called MPI_Abort(comm=0x84000002, 59) - process 17 [119]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 236 [Wed Feb 19 09:05:35 2020] [c5-3c1s9n3] application called MPI_Abort(comm=0x84000002, 59) - process 236 Failed during initial partitioning Failed during initial partitioning [12]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 160 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n0] application called MPI_Abort(comm=0x84000004, 59) - process 160 [96]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [249]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [249]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [249]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [249]PETSC ERROR: to get more information on the crash. [249]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [155]PETSC ERROR: ------------------------------------------------------------------------ Rank 213 [Wed Feb 19 09:05:35 2020] [c4-3c2s15n3] application called MPI_Abort(comm=0x84000002, 59) - process 213 Failed during initial partitioning Rank 201 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 201 Failed during initial partitioning [182]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07206 by amollen Wed Feb 19 09:05:25 2020 [182]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [182]PETSC ERROR: #1 User provided function() line 0 in unknown file [185]PETSC ERROR: #1 User provided function() line 0 in unknown file [120]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [192]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [192]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [134]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [134]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 168 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n1] application called MPI_Abort(comm=0x84000004, 59) - process 168 Failed during initial partitioning [66]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 139 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n2] application called MPI_Abort(comm=0x84000002, 59) - process 139 Failed during initial partitioning [144]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [144]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [144]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [59]PETSC ERROR: Signal received [59]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [59]PETSC ERROR: Petsc Release Version 3.11.3, unknown [59]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04192 by amollen Wed Feb 19 09:05:25 2020 [59]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [59]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 51 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 51 Rank 240 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n0] application called MPI_Abort(comm=0x84000004, 59) - process 240 [218]PETSC ERROR: Petsc Release Version 3.11.3, unknown [218]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07872 by amollen Wed Feb 19 09:05:25 2020 [218]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [218]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 104 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n0] application called MPI_Abort(comm=0x84000004, 59) - process 104 [72]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [87]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05533 by amollen Wed Feb 19 09:05:25 2020 [87]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [87]PETSC ERROR: #1 User provided function() line 0 in unknown file [89]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [89]PETSC ERROR: Signal received [89]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [89]PETSC ERROR: Petsc Release Version 3.11.3, unknown Failed during initial partitioning Rank 119 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 119 Rank 232 [Wed Feb 19 09:05:35 2020] [c5-3c1s9n3] application called MPI_Abort(comm=0x84000004, 59) - process 232 [12]PETSC ERROR: Signal received [12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Rank 167 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n0] application called MPI_Abort(comm=0x84000002, 59) - process 167 Failed during initial partitioning [96]PETSC ERROR: to get more information on the crash. [252]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [252]PETSC ERROR: to get more information on the crash. [252]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [252]PETSC ERROR: Signal received [155]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [155]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [155]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [155]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [210]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [210]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [210]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Rank 207 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 207 [176]PETSC ERROR: ------------------------------------------------------------------------ [187]PETSC ERROR: ------------------------------------------------------------------------ Failed during initial partitioning [192]PETSC ERROR: to get more information on the crash. [134]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [134]PETSC ERROR: to get more information on the crash. [169]PETSC ERROR: Signal received [169]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [169]PETSC ERROR: Petsc Release Version 3.11.3, unknown [169]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07205 by amollen Wed Feb 19 09:05:25 2020 [169]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [169]PETSC ERROR: #1 User provided function() line 0 in unknown file [67]PETSC ERROR: ------------------------------------------------------------------------ [67]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [67]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [67]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [67]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [67]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [67]PETSC ERROR: to get more information on the crash. [67]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [140]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06138 by amollen Wed Feb 19 09:05:25 2020 [140]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [140]PETSC ERROR: #1 User provided function() line 0 in unknown file [145]PETSC ERROR: ------------------------------------------------------------------------ [145]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [145]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [145]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [145]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [145]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [145]PETSC ERROR: to get more information on the crash. [63]PETSC ERROR: Petsc Release Version 3.11.3, unknown [63]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04192 by amollen Wed Feb 19 09:05:25 2020 [63]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [63]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning Failed during initial partitioning [220]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07872 by amollen Wed Feb 19 09:05:25 2020 [220]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [220]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning [72]PETSC ERROR: #1 User provided function() line 0 in unknown file [86]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [86]PETSC ERROR: to get more information on the crash. [86]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [88]PETSC ERROR: Signal received [88]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [88]PETSC ERROR: Petsc Release Version 3.11.3, unknown [16]PETSC ERROR: ------------------------------------------------------------------------ Failed during initial partitioning Failed during initial partitioning [12]PETSC ERROR: Petsc Release Version 3.11.3, unknown [12]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04186 by amollen Wed Feb 19 09:05:25 2020 Rank 165 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n0] application called MPI_Abort(comm=0x84000002, 59) - process 165 Failed during initial partitioning [96]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 254 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n1] application called MPI_Abort(comm=0x84000002, 59) - process 254 Failed during initial partitioning [155]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [155]PETSC ERROR: to get more information on the crash. [212]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [212]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [212]PETSC ERROR: to get more information on the crash. Rank 206 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n1] application called MPI_Abort(comm=0x84000002, 59) - process 206 Failed during initial partitioning [176]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [176]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [184]PETSC ERROR: ------------------------------------------------------------------------ [184]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [120]PETSC ERROR: Signal received [120]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [192]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [134]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 173 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n1] application called MPI_Abort(comm=0x84000002, 59) - process 173 Failed during initial partitioning [70]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [70]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 70 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 70 [136]PETSC ERROR: ------------------------------------------------------------------------ [151]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [151]PETSC ERROR: to get more information on the crash. [151]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [151]PETSC ERROR: Signal received Rank 59 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 59 Failed during initial partitioning [223]PETSC ERROR: ------------------------------------------------------------------------ [223]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [223]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger Rank 107 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 107 Rank 72 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n0] application called MPI_Abort(comm=0x84000004, 59) - process 72 [86]PETSC ERROR: Signal received [86]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [89]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05534 by amollen Wed Feb 19 09:05:25 2020 [89]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [89]PETSC ERROR: #1 User provided function() line 0 in unknown file [16]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [12]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Failed during initial partitioning [96]PETSC ERROR: Signal received [249]PETSC ERROR: Signal received [249]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [249]PETSC ERROR: Petsc Release Version 3.11.3, unknown [155]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [210]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [210]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [210]PETSC ERROR: to get more information on the crash. Failed during initial partitioning [176]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [176]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [187]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [187]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [187]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [187]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [120]PETSC ERROR: Petsc Release Version 3.11.3, unknown [120]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06136 by amollen Wed Feb 19 09:05:25 2020 [192]PETSC ERROR: Signal received [134]PETSC ERROR: Signal received [134]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [71]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [71]PETSC ERROR: Signal received [71]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [71]PETSC ERROR: Petsc Release Version 3.11.3, unknown [71]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04193 by amollen Wed Feb 19 09:05:25 2020 [71]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [137]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [137]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [137]PETSC ERROR: to get more information on the crash. [137]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [144]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [144]PETSC ERROR: to get more information on the crash. [144]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [144]PETSC ERROR: Signal received Rank 63 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 63 [48]PETSC ERROR: ------------------------------------------------------------------------ [48]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range Rank 217 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n0] application called MPI_Abort(comm=0x84000002, 59) - process 217 Failed during initial partitioning [111]PETSC ERROR: ------------------------------------------------------------------------ Failed during initial partitioning [86]PETSC ERROR: Petsc Release Version 3.11.3, unknown [86]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05533 by amollen Wed Feb 19 09:05:25 2020 [88]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05534 by amollen Wed Feb 19 09:05:25 2020 [88]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [88]PETSC ERROR: #1 User provided function() line 0 in unknown file [16]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [12]PETSC ERROR: #1 User provided function() line 0 in unknown file [161]PETSC ERROR: ------------------------------------------------------------------------ [96]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [96]PETSC ERROR: Petsc Release Version 3.11.3, unknown [252]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [252]PETSC ERROR: Petsc Release Version 3.11.3, unknown [252]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07977 by amollen Wed Feb 19 09:05:25 2020 [252]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [155]PETSC ERROR: Signal received [155]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [212]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [212]PETSC ERROR: Signal received [212]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [200]PETSC ERROR: ------------------------------------------------------------------------ [200]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [200]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [200]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [176]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [184]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [184]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [120]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [192]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [134]PETSC ERROR: Petsc Release Version 3.11.3, unknown Rank 169 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n1] application called MPI_Abort(comm=0x84000002, 59) - process 169 [71]PETSC ERROR: #1 User provided function() line 0 in unknown file [136]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [145]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [145]PETSC ERROR: Signal received [145]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [145]PETSC ERROR: Petsc Release Version 3.11.3, unknown [145]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06139 by amollen Wed Feb 19 09:05:25 2020 Failed during initial partitioning [48]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [48]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Rank 220 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n0] application called MPI_Abort(comm=0x84000002, 59) - process 220 Failed during initial partitioning [77]PETSC ERROR: ------------------------------------------------------------------------ [86]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [86]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 89 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 89 [16]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Rank 12 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n2] application called MPI_Abort(comm=0x84000002, 59) - process 12 [161]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [161]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [161]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [96]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05535 by amollen Wed Feb 19 09:05:25 2020 [249]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07977 by amollen Wed Feb 19 09:05:25 2020 [249]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [249]PETSC ERROR: #1 User provided function() line 0 in unknown file [155]PETSC ERROR: Petsc Release Version 3.11.3, unknown [155]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07203 by amollen Wed Feb 19 09:05:25 2020 [210]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [200]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [200]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run Rank 182 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n2] application called MPI_Abort(comm=0x84000002, 59) - process 182 Rank 185 [Wed Feb 19 09:05:35 2020] [c1-3c2s5n3] application called MPI_Abort(comm=0x84000002, 59) - process 185 Failed during initial partitioning [120]PETSC ERROR: #1 User provided function() line 0 in unknown file [192]PETSC ERROR: Petsc Release Version 3.11.3, unknown [134]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06137 by amollen Wed Feb 19 09:05:25 2020 [134]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Failed during initial partitioning Rank 65 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 65 Failed during initial partitioning [137]PETSC ERROR: Signal received [137]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [137]PETSC ERROR: Petsc Release Version 3.11.3, unknown [137]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06138 by amollen Wed Feb 19 09:05:25 2020 [151]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [151]PETSC ERROR: Petsc Release Version 3.11.3, unknown [151]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06139 by amollen Wed Feb 19 09:05:25 2020 [151]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [151]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning [48]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [48]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run Rank 221 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n0] application called MPI_Abort(comm=0x84000002, 59) - process 221 Failed during initial partitioning [111]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [111]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [111]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [77]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range Rank 87 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 87 Failed during initial partitioning Rank 88 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n2] application called MPI_Abort(comm=0x84000004, 59) - process 88 [16]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Failed during initial partitioning [161]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [161]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [96]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [96]PETSC ERROR: #1 User provided function() line 0 in unknown file [252]PETSC ERROR: #1 User provided function() line 0 in unknown file [155]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [212]PETSC ERROR: Petsc Release Version 3.11.3, unknown [212]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07871 by amollen Wed Feb 19 09:05:25 2020 [200]PETSC ERROR: to get more information on the crash. [200]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [200]PETSC ERROR: Signal received [176]PETSC ERROR: to get more information on the crash. [176]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [187]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [187]PETSC ERROR: to get more information on the crash. Rank 120 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n0] application called MPI_Abort(comm=0x84000004, 59) - process 120 [192]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07256 by amollen Wed Feb 19 09:05:25 2020 [192]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [134]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 66 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 66 Failed during initial partitioning Rank 140 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n2] application called MPI_Abort(comm=0x84000002, 59) - process 140 Failed during initial partitioning [144]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [144]PETSC ERROR: Petsc Release Version 3.11.3, unknown [144]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06139 by amollen Wed Feb 19 09:05:25 2020 [144]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [48]PETSC ERROR: to get more information on the crash. [222]PETSC ERROR: ------------------------------------------------------------------------ [222]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [222]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [222]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [222]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [222]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [111]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [111]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [77]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [77]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [77]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 86 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n1] application called MPI_Abort(comm=0x84000002, 59) - process 86 Failed during initial partitioning [16]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [16]PETSC ERROR: to get more information on the crash. [161]PETSC ERROR: to get more information on the crash. Rank 96 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n3] application called MPI_Abort(comm=0x84000004, 59) - process 96 Rank 252 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n1] application called MPI_Abort(comm=0x84000002, 59) - process 252 [155]PETSC ERROR: #1 User provided function() line 0 in unknown file [210]PETSC ERROR: Signal received [210]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [210]PETSC ERROR: Petsc Release Version 3.11.3, unknown [200]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Failed during initial partitioning Rank 188 [Wed Feb 19 09:05:35 2020] [c1-3c2s5n3] application called MPI_Abort(comm=0x84000002, 59) - process 188 Failed during initial partitioning Failed during initial partitioning [192]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 134 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n1] application called MPI_Abort(comm=0x84000002, 59) - process 134 [67]PETSC ERROR: Signal received [67]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [67]PETSC ERROR: Petsc Release Version 3.11.3, unknown [67]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04193 by amollen Wed Feb 19 09:05:25 2020 [67]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [67]PETSC ERROR: #1 User provided function() line 0 in unknown file [136]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [136]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [136]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [145]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [145]PETSC ERROR: #1 User provided function() line 0 in unknown file [48]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [48]PETSC ERROR: Signal received [223]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [223]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [223]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [223]PETSC ERROR: to get more information on the crash. [223]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [223]PETSC ERROR: Signal received [111]PETSC ERROR: to get more information on the crash. [77]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [77]PETSC ERROR: to get more information on the crash. Failed during initial partitioning Failed during initial partitioning [16]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [161]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [161]PETSC ERROR: Signal received Failed during initial partitioning Rank 249 [Wed Feb 19 09:05:35 2020] [c5-3c1s10n1] application called MPI_Abort(comm=0x84000002, 59) - process 249 Rank 155 [Wed Feb 19 09:05:35 2020] [c1-3c1s8n3] application called MPI_Abort(comm=0x84000002, 59) - process 155 [211]PETSC ERROR: ------------------------------------------------------------------------ [211]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [211]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [200]PETSC ERROR: Petsc Release Version 3.11.3, unknown [176]PETSC ERROR: Signal received [184]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [184]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [184]PETSC ERROR: to get more information on the crash. [184]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 192 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n0] application called MPI_Abort(comm=0x84000004, 59) - process 192 Failed during initial partitioning Failed during initial partitioning [137]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [137]PETSC ERROR: #1 User provided function() line 0 in unknown file [144]PETSC ERROR: #1 User provided function() line 0 in unknown file [48]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Rank 218 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n0] application called MPI_Abort(comm=0x84000002, 59) - process 218 Failed during initial partitioning [111]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [77]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [16]PETSC ERROR: Signal received [161]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Failed during initial partitioning Failed during initial partitioning [212]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [212]PETSC ERROR: #1 User provided function() line 0 in unknown file [200]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07257 by amollen Wed Feb 19 09:05:25 2020 [176]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [176]PETSC ERROR: Petsc Release Version 3.11.3, unknown [187]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [187]PETSC ERROR: Signal received [187]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [187]PETSC ERROR: Petsc Release Version 3.11.3, unknown [123]PETSC ERROR: ------------------------------------------------------------------------ [123]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access Failed during initial partitioning Rank 71 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 71 Failed during initial partitioning Rank 145 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n3] application called MPI_Abort(comm=0x84000002, 59) - process 145 [48]PETSC ERROR: Petsc Release Version 3.11.3, unknown [48]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04191 by amollen Wed Feb 19 09:05:25 2020 Failed during initial partitioning [111]PETSC ERROR: Signal received [111]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [111]PETSC ERROR: Petsc Release Version 3.11.3, unknown [77]PETSC ERROR: Signal received [77]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [161]PETSC ERROR: Petsc Release Version 3.11.3, unknown [161]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07204 by amollen Wed Feb 19 09:05:25 2020 Failed during initial partitioning [208]PETSC ERROR: ------------------------------------------------------------------------ [208]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [208]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [200]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [176]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07206 by amollen Wed Feb 19 09:05:25 2020 [184]PETSC ERROR: Signal received [123]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [123]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [123]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 67 [Wed Feb 19 09:05:35 2020] [c9-1c2s8n1] application called MPI_Abort(comm=0x84000002, 59) - process 67 [136]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [136]PETSC ERROR: to get more information on the crash. [136]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 151 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n3] application called MPI_Abort(comm=0x84000002, 59) - process 151 Failed during initial partitioning [48]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [0]PETSC ERROR: ------------------------------------------------------------------------ [111]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05536 by amollen Wed Feb 19 09:05:25 2020 [111]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [77]PETSC ERROR: Petsc Release Version 3.11.3, unknown [77]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid05532 by amollen Wed Feb 19 09:05:25 2020 [16]PETSC ERROR: Petsc Release Version 3.11.3, unknown [16]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04187 by amollen Wed Feb 19 09:05:25 2020 [161]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [210]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07871 by amollen Wed Feb 19 09:05:25 2020 [210]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [210]PETSC ERROR: #1 User provided function() line 0 in unknown file [200]PETSC ERROR: #1 User provided function() line 0 in unknown file [176]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [187]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07255 by amollen Wed Feb 19 09:05:25 2020 [187]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [123]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [123]PETSC ERROR: to get more information on the crash. Failed during initial partitioning [136]PETSC ERROR: Signal received Failed during initial partitioning [48]PETSC ERROR: #1 User provided function() line 0 in unknown file [222]PETSC ERROR: to get more information on the crash. [222]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [222]PETSC ERROR: Signal received [222]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [111]PETSC ERROR: #1 User provided function() line 0 in unknown file [77]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [16]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [161]PETSC ERROR: #1 User provided function() line 0 in unknown file [211]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [211]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [211]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [211]PETSC ERROR: to get more information on the crash. [211]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 200 [Wed Feb 19 09:05:35 2020] [c1-3c2s6n1] application called MPI_Abort(comm=0x84000004, 59) - process 200 [176]PETSC ERROR: #1 User provided function() line 0 in unknown file [184]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [184]PETSC ERROR: Petsc Release Version 3.11.3, unknown [184]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07255 by amollen Wed Feb 19 09:05:25 2020 [123]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [123]PETSC ERROR: Signal received [136]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [136]PETSC ERROR: Petsc Release Version 3.11.3, unknown Rank 144 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n3] application called MPI_Abort(comm=0x84000004, 59) - process 144 Rank 48 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n3] application called MPI_Abort(comm=0x84000004, 59) - process 48 [223]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [223]PETSC ERROR: Petsc Release Version 3.11.3, unknown [223]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07872 by amollen Wed Feb 19 09:05:25 2020 [223]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [223]PETSC ERROR: #1 User provided function() line 0 in unknown file [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 111 [Wed Feb 19 09:05:35 2020] [c4-2c2s8n0] application called MPI_Abort(comm=0x84000002, 59) - process 111 [77]PETSC ERROR: #1 User provided function() line 0 in unknown file [16]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 161 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n0] application called MPI_Abort(comm=0x84000002, 59) - process 161 [208]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [208]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Failed during initial partitioning Rank 176 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n2] application called MPI_Abort(comm=0x84000004, 59) - process 176 [187]PETSC ERROR: #1 User provided function() line 0 in unknown file [123]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [123]PETSC ERROR: Petsc Release Version 3.11.3, unknown Rank 137 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n2] application called MPI_Abort(comm=0x84000002, 59) - process 137 Failed during initial partitioning Failed during initial partitioning [222]PETSC ERROR: Petsc Release Version 3.11.3, unknown [222]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07872 by amollen Wed Feb 19 09:05:25 2020 [222]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [222]PETSC ERROR: #1 User provided function() line 0 in unknown file [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.11.3, unknown [0]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04185 by amollen Wed Feb 19 09:05:25 2020 Failed during initial partitioning Rank 77 [Wed Feb 19 09:05:35 2020] [c4-2c2s7n0] application called MPI_Abort(comm=0x84000002, 59) - process 77 Rank 16 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n3] application called MPI_Abort(comm=0x84000004, 59) - process 16 Failed during initial partitioning Rank 210 [Wed Feb 19 09:05:35 2020] [c4-3c2s15n3] application called MPI_Abort(comm=0x84000002, 59) - process 210 Failed during initial partitioning [184]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [184]PETSC ERROR: #1 User provided function() line 0 in unknown file [123]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06136 by amollen Wed Feb 19 09:05:25 2020 [123]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [136]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06138 by amollen Wed Feb 19 09:05:25 2020 [136]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Rank 223 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n0] application called MPI_Abort(comm=0x84000002, 59) - process 223 [0]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [0]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning Failed during initial partitioning [211]PETSC ERROR: Signal received [211]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [211]PETSC ERROR: Petsc Release Version 3.11.3, unknown [211]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07871 by amollen Wed Feb 19 09:05:25 2020 Rank 187 [Wed Feb 19 09:05:35 2020] [c1-3c2s5n3] application called MPI_Abort(comm=0x84000002, 59) - process 187 Failed during initial partitioning [123]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning Rank 222 [Wed Feb 19 09:05:35 2020] [c5-3c0s0n0] application called MPI_Abort(comm=0x84000002, 59) - process 222 Rank 0 [Wed Feb 19 09:05:35 2020] [c9-1c2s6n1] application called MPI_Abort(comm=0x84000004, 59) - process 0 Rank 212 [Wed Feb 19 09:05:35 2020] [c4-3c2s15n3] application called MPI_Abort(comm=0x84000002, 59) - process 212 Failed during initial partitioning Rank 123 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n0] application called MPI_Abort(comm=0x84000002, 59) - process 123 [136]PETSC ERROR: #1 User provided function() line 0 in unknown file Failed during initial partitioning Failed during initial partitioning [214]PETSC ERROR: ------------------------------------------------------------------------ [214]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [214]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [214]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind Rank 184 [Wed Feb 19 09:05:35 2020] [c1-3c2s5n3] application called MPI_Abort(comm=0x84000004, 59) - process 184 Failed during initial partitioning Rank 136 [Wed Feb 19 09:05:35 2020] [c7-2c2s14n2] application called MPI_Abort(comm=0x84000004, 59) - process 136 Failed during initial partitioning [45]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [45]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [45]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [45]PETSC ERROR: to get more information on the crash. [208]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [208]PETSC ERROR: to get more information on the crash. [208]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [208]PETSC ERROR: Signal received Failed during initial partitioning Failed during initial partitioning [45]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [45]PETSC ERROR: Signal received [45]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [45]PETSC ERROR: Petsc Release Version 3.11.3, unknown Failed during initial partitioning [45]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04190 by amollen Wed Feb 19 09:05:25 2020 [45]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [45]PETSC ERROR: #1 User provided function() line 0 in unknown file [211]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [211]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 45 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 45 [214]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [214]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [214]PETSC ERROR: to get more information on the crash. [214]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Failed during initial partitioning [208]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [208]PETSC ERROR: Petsc Release Version 3.11.3, unknown [208]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07871 by amollen Wed Feb 19 09:05:25 2020 [208]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [208]PETSC ERROR: #1 User provided function() line 0 in unknown file [43]PETSC ERROR: ------------------------------------------------------------------------ [43]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [214]PETSC ERROR: Signal received [214]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [214]PETSC ERROR: Petsc Release Version 3.11.3, unknown [43]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [43]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [43]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Rank 211 [Wed Feb 19 09:05:35 2020] [c4-3c2s15n3] application called MPI_Abort(comm=0x84000002, 59) - process 211 Failed during initial partitioning [43]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [43]PETSC ERROR: to get more information on the crash. [214]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07871 by amollen Wed Feb 19 09:05:25 2020 [214]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [214]PETSC ERROR: #1 User provided function() line 0 in unknown file [43]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- Rank 208 [Wed Feb 19 09:05:35 2020] [c4-3c2s15n3] application called MPI_Abort(comm=0x84000004, 59) - process 208 [43]PETSC ERROR: Signal received [43]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Rank 214 [Wed Feb 19 09:05:35 2020] [c4-3c2s15n3] application called MPI_Abort(comm=0x84000002, 59) - process 214 [43]PETSC ERROR: Petsc Release Version 3.11.3, unknown Failed during initial partitioning [43]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04190 by amollen Wed Feb 19 09:05:25 2020 [43]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc Failed during initial partitioning [43]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 43 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 43 Failed during initial partitioning [40]PETSC ERROR: ------------------------------------------------------------------------ [40]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [40]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [40]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [40]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [41]PETSC ERROR: ------------------------------------------------------------------------ [41]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [41]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [41]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [41]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [40]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [40]PETSC ERROR: to get more information on the crash. [40]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [41]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [41]PETSC ERROR: to get more information on the crash. [41]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [40]PETSC ERROR: Signal received [40]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [41]PETSC ERROR: Signal received [41]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [44]PETSC ERROR: ------------------------------------------------------------------------ [44]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [44]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [40]PETSC ERROR: Petsc Release Version 3.11.3, unknown [40]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04190 by amollen Wed Feb 19 09:05:25 2020 [40]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [40]PETSC ERROR: #1 User provided function() line 0 in unknown file [41]PETSC ERROR: Petsc Release Version 3.11.3, unknown [41]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04190 by amollen Wed Feb 19 09:05:25 2020 [41]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [41]PETSC ERROR: #1 User provided function() line 0 in unknown file [44]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [44]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [44]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [44]PETSC ERROR: to get more information on the crash. [42]PETSC ERROR: ------------------------------------------------------------------------ [42]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [44]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [44]PETSC ERROR: Signal received [44]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [44]PETSC ERROR: Petsc Release Version 3.11.3, unknown Rank 40 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n2] application called MPI_Abort(comm=0x84000004, 59) - process 40 Rank 41 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 41 [42]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [42]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [42]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [42]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [44]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04190 by amollen Wed Feb 19 09:05:25 2020 [44]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [44]PETSC ERROR: #1 User provided function() line 0 in unknown file [47]PETSC ERROR: ------------------------------------------------------------------------ [47]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [47]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [47]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [47]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Failed during initial partitioning Failed during initial partitioning [42]PETSC ERROR: to get more information on the crash. [42]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [42]PETSC ERROR: Signal received [42]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [42]PETSC ERROR: Petsc Release Version 3.11.3, unknown [42]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04190 by amollen Wed Feb 19 09:05:25 2020 [47]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [47]PETSC ERROR: to get more information on the crash. [47]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [47]PETSC ERROR: Signal received [42]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [42]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 44 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 44 Failed during initial partitioning [47]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [47]PETSC ERROR: Petsc Release Version 3.11.3, unknown [47]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04190 by amollen Wed Feb 19 09:05:25 2020 [47]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [47]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 42 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 42 Failed during initial partitioning Rank 47 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 47 Failed during initial partitioning [46]PETSC ERROR: ------------------------------------------------------------------------ [46]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [46]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [46]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [46]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [46]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [46]PETSC ERROR: to get more information on the crash. [46]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [46]PETSC ERROR: Signal received [46]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [46]PETSC ERROR: Petsc Release Version 3.11.3, unknown [46]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04190 by amollen Wed Feb 19 09:05:25 2020 [46]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [46]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 46 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n2] application called MPI_Abort(comm=0x84000002, 59) - process 46 Failed during initial partitioning [239]PETSC ERROR: ------------------------------------------------------------------------ [239]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [239]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [239]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [239]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [239]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [239]PETSC ERROR: to get more information on the crash. [239]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [239]PETSC ERROR: Signal received [239]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [239]PETSC ERROR: Petsc Release Version 3.11.3, unknown [239]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07975 by amollen Wed Feb 19 09:05:25 2020 [239]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [239]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 239 [Wed Feb 19 09:05:35 2020] [c5-3c1s9n3] application called MPI_Abort(comm=0x84000002, 59) - process 239 Failed during initial partitioning [186]PETSC ERROR: ------------------------------------------------------------------------ [186]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [186]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [170]PETSC ERROR: ------------------------------------------------------------------------ [170]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [186]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [186]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [170]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [170]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [170]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [170]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [186]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [186]PETSC ERROR: to get more information on the crash. [186]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [170]PETSC ERROR: to get more information on the crash. [170]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [170]PETSC ERROR: Signal received [186]PETSC ERROR: Signal received [186]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [170]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [170]PETSC ERROR: Petsc Release Version 3.11.3, unknown [170]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07205 by amollen Wed Feb 19 09:05:25 2020 [186]PETSC ERROR: Petsc Release Version 3.11.3, unknown [186]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid07255 by amollen Wed Feb 19 09:05:25 2020 [186]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [170]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [170]PETSC ERROR: #1 User provided function() line 0 in unknown file [186]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 170 [Wed Feb 19 09:05:35 2020] [c1-3c1s9n1] application called MPI_Abort(comm=0x84000002, 59) - process 170 Rank 186 [Wed Feb 19 09:05:35 2020] [c1-3c2s5n3] application called MPI_Abort(comm=0x84000002, 59) - process 186 Failed during initial partitioning Failed during initial partitioning [55]PETSC ERROR: ------------------------------------------------------------------------ [55]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [55]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [55]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [55]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [55]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [55]PETSC ERROR: to get more information on the crash. [55]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [55]PETSC ERROR: Signal received [55]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [55]PETSC ERROR: Petsc Release Version 3.11.3, unknown [55]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid04191 by amollen Wed Feb 19 09:05:25 2020 [55]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [55]PETSC ERROR: #1 User provided function() line 0 in unknown file Rank 55 [Wed Feb 19 09:05:35 2020] [c9-1c2s7n3] application called MPI_Abort(comm=0x84000002, 59) - process 55 Failed during initial partitioning [193]PETSC ERROR: ------------------------------------------------------------------------ [236]PETSC ERROR: ------------------------------------------------------------------------ Failed during initial partitioning [73]PETSC ERROR: ------------------------------------------------------------------------ [123]PETSC ERROR: ------------------------------------------------------------------------ [128]PETSC ERROR: ------------------------------------------------------------------------ Failed during initial partitioning Failed during initial partitioning [128]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [128]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [128]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [128]PETSC ERROR: [128]PETSC ERROR: ------------------------------------------------------------------------ [128]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [128]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [128]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [128]PETSC ERROR: [128]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors configure using --with-debugging=yes, recompile, link, and run [128]PETSC ERROR: [128]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run to get more information on the crash. [128]PETSC ERROR: to get more information on the crash. [128]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [128]PETSC ERROR: [128]PETSC ERROR: Signal received --------------------- Error Message -------------------------------------------------------------- [128]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [128]PETSC ERROR: Petsc Release Version 3.11.3, unknown [128]PETSC ERROR: [128]PETSC ERROR: Signal received /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06137 by amollen Wed Feb 19 09:05:25 2020 [152]PETSC ERROR: ------------------------------------------------------------------------ slurmstepd: error: *** STEP 28263517.0 ON nid04185 CANCELLED AT 2020-02-19T09:10:18 DUE TO TIME LIMIT *** Failed during initial partitioning [128]PETSC ERROR: [128]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [128]PETSC ERROR: [128]PETSC ERROR: Petsc Release Version 3.11.3, unknown #1 User provided function() line 0 in unknown file [128]PETSC ERROR: /global/homes/a/amollen/1_tmp_XGC-Devel/XGC-Devel/xgc_build/xgca-vec on a named nid06137 by amollen Wed Feb 19 09:05:25 2020 Rank 128 [Wed Feb 19 09:10:18 2020] [c7-2c2s14n1] application called MPI_Abort(comm=0x84000004, 59) - process 128 [128]PETSC ERROR: Configure options COPTFLAGS=" -g" CXXOPTFLAGS=-g FOPTFLAGS=" -g" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-metis=1 --download-hypre=1 --download-parmetis=1 --known-mpi-shared-libraries --with-debugging=0 --with-mpiexec=srun --with-batch=0 --with-memalign=64 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt64-intel3.11.3 --with-openmp=0 --download-p4est=0 --prefix=/project/projectdirs/m499/Software/petsc/3.11.3/cori_knl/intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc [128]PETSC ERROR: #2 User provided function() line 0 in unknown file Rank 128 [Wed Feb 19 09:10:18 2020] [c7-2c2s14n1] application called MPI_Abort(comm=0x84000004, 59) - process 128 srun: Job step aborted: Waiting up to 32 seconds for job step to finish. srun: got SIGCONT srun: forcing job termination