is it possible to support Complex Double data type ?

[gocanal]

Hi,
I am looking into the code to check if it is feasible to support Complex Double data type for matrix inverse and multiplication.

I realize that you have use a couple of external packages:
BLAS : you use dspr so I can replace it with zspr I presume ?
ARPACK : you use dsaupd and dseupd, I can not find a equivalent method.
Breeze : it supports Complex data type, so should be fine I guess ?

What's your assessment/advice for supporting Complex Double data type ?

many thanks
canal

[xingkungao]

hi @gocanal,
I believe that for Matrix Multiplication Complex Double could be well supported without too much change in the algorithm level, since Breeze itself supports Complex Double data type. But I am not quite sure for Inversion, given that currently the inversion is based on LU Decomposition of the original matrix. If you do need to deal with Complex Matrix Inverse and your use case is not so performance critical, I think you can separate the original Complex Matrix into two parts, Real and Imaginary, each alone using a Double Matrix. Then you can deal with Complex Matrix Inverse at a higher level without a single change in the current codebase.

Best regards.
xingkungao

[gocanal]

Hi,
I assume you mean we can do a 2M * 2M double data inversion, where the original matrix is M * M double complex, and the 2M * 2M matrix is:

real              imaginary
-imaginary            real

Am I correct ?

thanks
canal

[xingkungao]

Hi,
Yes, I mean for a M * M complex matrix (A + iB), consider it's inverse let's say (C + iD), we have
(A+iB)(C+iD) = I, and so

  AC - BD = I    
  AD + BC = 0

So we first get the inverse of 2M * 2M Double matrix

    A     -B            
    B      A     

then multiplies the inverse with the 2M * M matrix:

    I        
    0    

and we get

    C
    D   

which is exactly what we want.

hope it helps!
xingkungao

[gocanal]

Thank you very much for confirming.

I did a quick test, and compared with Scalapack based solution. I am running on one machine, with only 4GB RAM (I3, 3.3GHz, Ubuntu 14.04, 64-bit Java).

For multiplication, Marlin can support up to 5k * 5k. Scalapack can support up to 8k.

For inverse, Marlin can only support up to 2.5k matrix, while Scalapack can still support up to 8k. Beyond 2.5K, marlin hits 'OutOfMemoryError: Java Heap' (the error happened in org.spark_project.protobuf.AbstractMe
ssageLite.toByteArray).

It seems that Marlin (or I should say Hadoop/Spark) consumes a lot more memory ? Or if there is something I did not configure correctly (I set spark.akka.frameSize=1024, executor-memory 2700m).

[Myasuka]

Sorry for late reply.

Actually, by default, 60% memory of Spark executor process is occupied by Spark BlockManager, and 20% memory of Spark executor process is used during shuffle, these two parts are configured by parameter spark.storage.memoryFraction and spark.shuffle.memoryFraction. Besides, JVM-based program consumes more memory than MPI program in general. I think 2.7GB executor memory is a bit small.