[petsc-users] MatScale returns different results depending on matrix size
Roland Richter
roland.richter at ntnu.no
Wed Jan 6 11:31:04 CST 2021
Hei,
I removed all dependencies to armadillo and other not directly necessary
packages, and attached both CMakeLists.txt and main-file. Even though I
am only having PETSc as main dependence I still have the same issues.
For a scaling factor of 0.1 and a matrix size of [1024, 1024] it works
fine, for a scaling factor of 0.01 on the same matrix the apparent
scaling factor is suddenly 1e-8.
Thank you for your help!
Regards,
Roland
Am 06.01.21 um 17:36 schrieb Matthew Knepley:
> On Wed, Jan 6, 2021 at 11:05 AM Roland Richter <roland.richter at ntnu.no
> <mailto:roland.richter at ntnu.no>> wrote:
>
> Hei,
>
> I ran the program in both versions using "valgrind --tool=memcheck
> --leak-check=full --show-leak-kinds=all <binary> -malloc_debug". I got
>
> ==3059== LEAK SUMMARY:
> ==3059== definitely lost: 12,916 bytes in 32 blocks
> ==3059== indirectly lost: 2,415 bytes in 2 blocks
> ==3059== possibly lost: 0 bytes in 0 blocks
> ==3059== still reachable: 103,511 bytes in 123 blocks
> ==3059== suppressed: 0 bytes in 0 blocks
>
> but none of the leaks is related to the scaling-function itself.
>
> Did I miss something here?
>
> Here is my analysis. It is certainly the case that MatScale() does not
> mysteriously scale by other numbers.
> It is used all over the place in tests, and in the code. Your test
> requires another package. Thus, it seems
> reasonable to guess that a bad interaction with that package (memory
> overwrite, conflicting layout or format, etc.)
> is responsible for the behavior you see.
>
> Thanks,
>
> Matt
>
> Thanks!
>
> Am 06.01.21 um 15:26 schrieb Matthew Knepley:
>> On Wed, Jan 6, 2021 at 2:41 AM Roland Richter
>> <roland.richter at ntnu.no <mailto:roland.richter at ntnu.no>> wrote:
>>
>> Hei,
>>
>> I added one additional function to the code:
>>
>> /void test_scaling_petsc_pointer(const Mat &in_mat,//
>> // Mat &out_mat,//
>> // const PetscScalar
>> &scaling_factor) {//
>> // MatCopy (in_mat, out_mat, SAME_NONZERO_PATTERN);//
>> // PetscScalar *mat_ptr;//
>> // MatDenseGetArray (out_mat, &mat_ptr);//
>> // PetscInt r_0, r_1;//
>> // MatGetLocalSize (out_mat, &r_0, &r_1);//
>> // for(int i = 0; i < r_0 * r_1; ++i)//
>> // *(mat_ptr + i) = (*(mat_ptr + i) * scaling_factor);//
>> //
>> // MatAssemblyBegin (out_mat, MAT_FINAL_ASSEMBLY);//
>> // MatAssemblyEnd (out_mat, MAT_FINAL_ASSEMBLY);//
>> //}/
>>
>> When replacing test function /test_scaling_petsc()/ with
>> /test_scaling_petsc_pointer()/ everything works as it should,
>> but I do not understand why.
>>
>> Do you have any suggestions?
>>
>> The easiest explanation is that you have a memory overwrite in
>> the code somewhere. Barry's suggestion to use
>> valgrind is good.
>>
>> Matt
>>
>> Thanks!
>>
>>
>> Am 05.01.21 um 15:24 schrieb Roland Richter:
>>>
>>> Hei,
>>>
>>> the code I attached to the original mail should work out of
>>> the box, but requires armadillo and PETSc to compile/run.
>>> Armadillo stores the data in column-major order, and
>>> therefore I am transposing the matrices before and after
>>> transferring using .st().
>>>
>>> Thank you for your help!
>>>
>>> Regards,
>>>
>>> Roland
>>>
>>> Am 05.01.21 um 15:21 schrieb Matthew Knepley:
>>>> On Tue, Jan 5, 2021 at 7:57 AM Roland Richter
>>>> <roland.richter at ntnu.no <mailto:roland.richter at ntnu.no>> wrote:
>>>>
>>>> Hei,
>>>>
>>>> I would like to scale a given matrix with a fixed
>>>> scalar value, and
>>>> therefore would like to use MatScale(). Nevertheless, I
>>>> observed an
>>>> interesting behavior depending on the size of the
>>>> matrix, and currently
>>>> I am not sure why.
>>>>
>>>> When running the attached code, I intend to divide all
>>>> elements in the
>>>> matrix by a constant factor of 10. If I have three or
>>>> fewer rows and
>>>> 1024 columns, I get the expected result. If I have four
>>>> or more rows
>>>> (with the same number of columns), suddenly my scaling
>>>> factor seems to
>>>> be 0.01 instead of 0.1 for the PETSc-matrix. The
>>>> armadillo-based matrix
>>>> still behaves as expected.
>>>>
>>>>
>>>> 1) It looks like you assume the storage in your armadillo
>>>> matrix is row major. I would be surprised if this was true.
>>>>
>>>> 2) I think it is unlikely that there is a problem with
>>>> MatScale, so I would guess either you have a memory overwrite
>>>> or are misinterpreting your output. If you send something I
>>>> can run, I will figure out which it is.
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>> I currently do not understand that behavior, but do not
>>>> see any problems
>>>> with the code either. Are there any possible
>>>> explanations for that behavior?
>>>>
>>>> Thank you very much,
>>>>
>>>> regards,
>>>>
>>>> Roland Richter
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin
>>>> their experiments is infinitely more interesting than any
>>>> results to which their experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to
>> which their experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210106/c7cb43fc/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main.cpp
Type: text/x-c++src
Size: 2648 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210106/c7cb43fc/attachment-0001.bin>
-------------- next part --------------
cmake_minimum_required(VERSION 3.12)
set(PROJECT_NAME "armadillo_with_PETSc")
project(${PROJECT_NAME} LANGUAGES CXX C)
include(GNUInstallDirs)
include(CheckLanguage)
if(NOT WIN32)
string(ASCII 27 Esc)
set(ColourReset "${Esc}[m")
set(ColourBold "${Esc}[1m")
set(Red "${Esc}[31m")
set(Green "${Esc}[32m")
set(Yellow "${Esc}[33m")
set(Blue "${Esc}[34m")
set(Magenta "${Esc}[35m")
set(Cyan "${Esc}[36m")
set(White "${Esc}[37m")
set(BoldRed "${Esc}[1;31m")
set(BoldGreen "${Esc}[1;32m")
set(BoldYellow "${Esc}[1;33m")
set(BoldBlue "${Esc}[1;34m")
set(BoldMagenta "${Esc}[1;35m")
set(BoldCyan "${Esc}[1;36m")
set(BoldWhite "${Esc}[1;37m")
endif()
check_language(CUDA)
if(CMAKE_CUDA_COMPILER)
message(STATUS "${BoldGreen}CUDA-support enabled${ColourReset}")
enable_language (CUDA)
set(CMAKE_CUDA_STANDARD 14)
set(CMAKE_CUDA_STANDARD_REQUIRED ON)
add_compile_definitions (USE_CUDA)
else(CMAKE_CUDA_COMPILER)
message(STATUS "${BoldRed}No CUDA support${ColourReset}")
remove_definitions (USE_CUDA)
endif()
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
cmake_policy (SET CMP0074 NEW)
set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake/Modules/")
set(INCLUDE_PATH ${CMAKE_SOURCE_DIR}/include)
set(SOURCE_PATH ${CMAKE_SOURCE_DIR}/source)
set(EXECUTABLE_OUTPUT_PATH ${CMAKE_SOURCE_DIR}/bin)
set(LIBRARY_OUTPUT_PATH ${CMAKE_SOURCE_DIR}/lib)
if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=native -fopenmp-simd")
elseif("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Intel")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -xHost")
endif()
configure_file(downloadFindFFTW.cmake.in findFFTW-download/CMakeLists.txt)
execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}" .
RESULT_VARIABLE result
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/findFFTW-download )
if(result)
message(FATAL_ERROR "CMake step for findFFTW failed: ${result}")
else()
message("CMake step for findFFTW completed (${result}).")
endif()
execute_process(COMMAND ${CMAKE_COMMAND} --build .
RESULT_VARIABLE result
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/findFFTW-download )
if(result)
message(FATAL_ERROR "Build step for findFFTW failed: ${result}")
endif()
set(findFFTW_DIR ${CMAKE_CURRENT_BINARY_DIR}/findFFTW-src)
set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${findFFTW_DIR}")
#Packages
set(ENV{PETSC_DIR} /opt/petsc)
find_package (PETSc REQUIRED)
set(BOOST_ROOT "/opt/boost")
find_package(Boost 1.72 REQUIRED COMPONENTS system filesystem mpi program_options serialization)
pkg_check_modules(PETSc REQUIRED)
find_package(GSL REQUIRED QUIET)
find_package(MKL REQUIRED QUIET)
set(FFTW_ROOT "/opt/fftw3")
find_package(FFTW REQUIRED QUIET)
find_package(OpenMP REQUIRED QUIET)
set(EIGEN3_INCLUDE_DIRS "/usr/include/eigen3")
set(PROJECT_SRC ${CMAKE_SOURCE_DIR}/source/main.cpp)
set(PROJECT_INC "")
INCLUDE_DIRECTORIES(
${INCLUDE_DIRS}
${UPPE_INCLUDE_PATH})
add_executable(${PROJECT_NAME} ${PROJECT_SRC} ${PROJECT_INC})
target_include_directories (${PROJECT_NAME} PRIVATE
${INCLUDE_PATH}
${PETSC_INCLUDE_DIRS}
${FFTW_INCLUDE_DIRS}
${GSL_INCLUDE_DIRS}
${MKL_INCLUDE_DIRS}
${Boost_INCLUDE_DIRS})
target_compile_options(${PROJECT_NAME} PUBLIC -Wall -Wextra -pedantic -Werror -fPIC -flto -fopenmp -O2 -funroll-loops -funroll-all-loops -fstrict-aliasing -mavx -march=native)
set(PROJECT_LINK_LIBRARIES
${EIGEN3_LIBRARIES}
${GSL_LIBRARIES}
${FFTW_LIBRARIES}
/opt/fftw3/lib64/libfftw3_mpi.so
gfortran
/opt/intel/mkl/lib/intel64/libmkl_rt.so
${PETSC_LIBRARY_DIRS}/libpetsc.so
Boost::filesystem
Boost::mpi
Boost::program_options
Boost::serialization
${X11_LIBRARIES}
OpenMP::OpenMP_CXX
)
target_link_libraries(${PROJECT_NAME}
${PROJECT_LINK_LIBRARIES}
)
More information about the petsc-users
mailing list