Curious about the future of data-driven systems? Join our Data Engineering roundtable and learn how to build scalable data platforms.
Data Engineering: The industry has come a long way from organizing unstructured data to adopting today's modern data pipelines. See how.
Jean-Georges "jgp" Perrin is a technology consultant focusing on building innovative and modern data platforms, president of AIDA User Group, and author of Spark in Action, 2nd edition (Manning). He is passionate about software engineering and all things data. His latest endeavors bring him to more and more data engineering, data governance, industrialization of data science, and his favorite theme, Data Mesh. He is proud to have been recognized as a Lifetime IBM Champion. Jean-Georges shares over 25 years of experience in the IT industry as a presenter and participant at conferences and publishing articles in print and online media. His blog is visible at http://jgp.ai. He enjoys exploring Upstate New York and New England with his wife and kids when not immersed in IT, which he loves.
Stats
| Reputation: | 254 |
| Pageviews: | 108.2K |
| Articles: | 2 |
| Comments: | 6 |
Comments
Jun 16, 2020 · Jean-Georges Perrin
Yeah, that's definitely not the use-case I see for chekpointing. Delta Lake (in the link above) can help.
Jun 15, 2020 · Jean-Georges Perrin
Scala? You're killing me... ;)
I am not sure your use-case is solvable using checkpoints. They are not checkpoints like in VM. Maybe you should look at Delta Lake. Look at https://livebook.manning.com/book/spark-in-action-second-edition/chapter-17?origin=product-toc&a_aid=jgp
Jun 09, 2020 · Jean-Georges Perrin
Look at my link down there, it should help!
Jun 08, 2020 · Jean-Georges Perrin
This article is mostly accurate, but you can also refer to: https://livebook.manning.com/book/spark-in-action-second-edition/chapter-16?a_aid=jgp
Jun 08, 2020 · Jean-Georges Perrin
Indeed, which may also be a security concern...
Jun 08, 2020 · Jean-Georges Perrin
should be good. remember that this will happen on the executor node, not the driver or the master...