Curious about the future of data-driven systems? Join our Data Engineering roundtable and learn how to build scalable data platforms.
Data Engineering: The industry has come a long way from organizing unstructured data to adopting today's modern data pipelines. See how.
Software Architect at iCare
Fort Lauderdale, US
Joined Mar 2016
Konur Unyelioglu is a software architect with iCare (https://www.icare.com/). He has experience in designing and implementing IT solutions for diverse industries including health care, automotive, telecommunications, retail and transportation. His current focus areas include enterprise Java, mobile applications, health IT and cloud technologies.
Stats
| Reputation: | 557 |
| Pageviews: | 232.8K |
| Articles: | 5 |
| Comments: | 3 |
Comments
Dec 18, 2018 · Konur Unyelioglu
Yes. In the example, I used Spark local execution mode (pseudo-cluster) where all the components execute inside a single JVM. In that regard, we could assume everything is running in the driver.
Dec 17, 2018 · Konur Unyelioglu
In terms of scalability, the biggest challenge will be matrix computations, in particular, matrix inversion. Recall that the org.apache.spark.ml.linalg library currently does not even provide a method for matrix inversion. Properly implemented, such a matrix inversion technique should take advantage of parallel computing across cluster nodes. Hence, Update Phase Eq. 5 is one step where computational complexity could be distributed across executors.
The simple example given here utilizes Jama Java Matrix Package, that does not provide parallel computing capabilities out of the box.
Sep 04, 2018 · Konur Unyelioglu
I am glad you find it useful.