<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi Barry,<o:p></o:p></p>
<p class="MsoNormal">I used the branch <b><span style="font-size:12.0pt;font-family:Menlo;color:#C814C9">barry/2020-11-10/add-matlab-engine-host.
</span></b>Now it appears that the matlab engine starts can start on any node but it is not able to execute any matlab script. If I revert back to PETSc 3.13.5 the head compute node is able to execute the MATLAB script. See the following error (load_mpc_parameters.m
is a MATLAB script that the code is trying to execute):<b><span style="font-size:12.0pt;font-family:Menlo;color:#C814C9"><o:p></o:p></span></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Starting Matlab engine...<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Matlab engine started<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Storing variables in Matlab workspace...<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Undefined function or variable 'load_mpc_parameters'.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Error using save<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Variable 'a' not found.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[0]PETSC ERROR: Error in external library<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[0]PETSC ERROR: Unable to get array a from matlab<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[0]PETSC ERROR: Petsc Development GIT revision: unknown GIT Date: unknown<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[0]PETSC ERROR: ./main3d on a linux-opt named node35 by kkhedkar Thu Nov 12 02:39:47 2020<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[0]PETSC ERROR: Configure options --CC=/home/kkhedkar/sfw/mpich3.3_mellanox/3.3/bin/mpicc --CXX=/home/kkhedkar/sfw/mpich3.3_mellanox/3.3/bin/mpicxx --FC=/home/kkhedkar/sfw/mpich3.3_mellanox/3.3/bin/mpif90 --COPTFLAGS=-O3
--CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --PETSC_ARCH=linux-opt --with-debugging=0 --download-hypre=1 --with-x=0 --download-fblaslapack=1 --with-matlab-dir=/nas/app/MATLAB/matlab2019a/ --with-matlab-engine=1 --with-matlab-engine-dir=/nas/app/MATLAB/matlab2019a/extern/engines/<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[0]PETSC ERROR: #1 PetscMatlabEngineGetArray() line 407 in /home/kkhedkar/sfw/petsc_matlab_host_fix/petsc/src/sys/classes/matlabengine/matlab.c<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Error using save<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Variable 'h' not found.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[0]PETSC ERROR: #2 PetscMatlabEngineGetArray() line 407 in /home/kkhedkar/sfw/petsc_matlab_host_fix/petsc/src/sys/classes/matlabengine/matlab.c<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Error using save<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Variable 'mpc_start_time' not found.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[0]PETSC ERROR: #3 PetscMatlabEngineGetArray() line 407 in /home/kkhedkar/sfw/petsc_matlab_host_fix/petsc/src/sys/classes/matlabengine/matlab.c<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Error using save<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Variable 'sample_size' not found.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">sample_size = 0<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[0]PETSC ERROR: #4 PetscMatlabEngineGetArray() line 407 in /home/kkhedkar/sfw/petsc_matlab_host_fix/petsc/src/sys/classes/matlabengine/matlab.c<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Error using save<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Variable 'dt_sampling' not found.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Storing up past excitation force samples every 0 time-steps.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in">[0]PETSC ERROR: #5 PetscMatlabEngineGetArray() line 407 in /home/kkhedkar/sfw/petsc_matlab_host_fix/petsc/src/sys/classes/matlabengine/matlab.c<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">IBRedundantInitializer: Deallocating initialization data.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My code is using Petsc matlab engine in this way (“master” is the hostname of the head node where MATLAB is licensed):<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">if(SAMRAI_MPI::getRank() == 0)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> {<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> std::cout << "\nStarting Matlab engine...\n" << std::endl;<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> PetscMatlabEngineCreate(PETSC_COMM_SELF, "master", &(mpc->mengine));<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> std::cout << "Matlab engine started" << std::endl; <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> PetscMatlabEngineEvaluate(mpc->mengine,"clc; clear all; close all; addpath([cd,'/MPC_matlab_code'])");<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> std::cout << "Storing variables in Matlab workspace...\n" << std::endl;<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> PetscMatlabEngineEvaluate(mpc->mengine,"global hydro wave; wave.H_wave = %f ; wave.Tp = %f; m = %f;", input_db->getDouble("HEIGHT"), input_db->getDouble("TIME_PERIOD"), input_db->getDouble("MASS"));<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> PetscMatlabEngineEvaluate(mpc->mengine,"load_mpc_parameters; calculate_mpc_matrices");<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> PetscMatlabEngineGetArray(mpc->mengine, 1, 1, &(mpc->m_plus_Ainf), "a");<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> PetscMatlabEngineGetArray(mpc->mengine, 1, 1, &(mpc->dt_controller), "h");<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> PetscMatlabEngineGetArray(mpc->mengine, 1, 1, &(mpc->mpc_start_time), "mpc_start_time");<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> PetscScalar sample_size;<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> PetscMatlabEngineGetArray(mpc->mengine, 1, 1, &sample_size, "sample_size");<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> mpc->sample_size = int(sample_size);<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> std::cout << "\nsample_size = " << mpc->sample_size << std::endl;<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> mpc->F_past.resize(mpc->sample_size);<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> mpc->t_past.resize(mpc->sample_size);<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> double dt = input_db->getDouble("DT_MAX");<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> mpc->initial_position = input_db->getDouble("ZCOM");<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> mpc->current_position = mpc->initial_position;<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> PetscScalar dt_sampling;<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> PetscMatlabEngineGetArray(mpc->mengine, 1, 1, &dt_sampling, "dt_sampling");<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> mpc->sampling_interval = int(dt_sampling / dt);<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> std::cout << "Storing up past excitation force samples every " << mpc->sampling_interval << " time-steps.\n" << std::endl;<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> }<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Barry Smith <bsmith@petsc.dev><br>
<b>Date: </b>Tuesday, November 10, 2020 at 5:22 PM<br>
<b>To: </b>Kaustubh Khedkar <kkhedkar9879@sdsu.edu><br>
<b>Cc: </b>"petsc-users@mcs.anl.gov" <petsc-users@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-users] Using Petsc-Matlab engine on a cluster<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"> In the git branch <b><span style="font-size:12.0pt;font-family:Menlo;color:#C814C9">barry/2020-11-10/add-matlab-engine-host </span></b> I have added the option -matlab_engine_host hostname <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Note this does require you have ssh access to hostname from where your PETSc program is running which may require setting up some ssh files<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Good luck<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Barry<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Nov 9, 2020, at 5:35 PM, Kaustubh Khedkar <<a href="mailto:kkhedkar9879@sdsu.edu">kkhedkar9879@sdsu.edu</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">Hello,<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">I am using PETSc with Matlab engine. I have built an application that uses Matlab engine through PETSc interface. I am trying to run this application on a cluster which has Matlab installed on a network drive (nas drive). I have made sure
that I am able to access Matlab using my university details on the cluster (by activating Matlab license through<span class="apple-converted-space"> </span><a href="http://mathworks.com/">Mathworks.com</a><span class="apple-converted-space"> </span>account).
I am also able to run this application using mpirun on the head node (master) which starts the Matlab engine but if I try to run this application using slurm on compute node then the Matlab engine does not start. I have also confirmed that if I login into
one of the compute nodes and do a mpirun the Matlab engine does not start there. If I try to launch Matlab on a compute node it asks for Matlab license which the compute node is unable to activate possibly because it is not connected to the internet directly.<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Is there a way start matlab engine on the head node through Petsc interface even though the application is running on compute nodes or is there any other work around this issue?<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Thanks,<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Kaustubh Khedkar<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>