https://blog.mxtao.top/tags/spark-notes/ Recent content in Spark-Notes on boboboker~ Hugo -- gohugo.io zh-cn all rights reserved. Thu, 29 Apr 2021 20:30:00 +0800 https://blog.mxtao.top/posts/platform/spark/spark-notes/ Sat, 04 Jul 2020 00:00:00 +0800 https://blog.mxtao.top/posts/platform/spark/spark-notes/ <h1 id="spark-相关内容随记">Spark 相关内容随记 </h1><p>随手记录Spark相关的问题、思考等</p> <p><a class="link" href="https://cloud.tencent.com/developer/article/1038770" target="_blank" rel="noopener" >Spark SQL在100TB上的自适应执行实践</a></p> <p><a class="link" href="http://spark.apache.org/docs/latest/sql-ref-functions-udf-aggregate.html" target="_blank" rel="noopener" >User Defined Aggregate Functions (UDAFs)</a></p> <h2 id="spark-sql---datasource">Spark SQL - DataSource </h2><p>通过实现Spark定义的DataSource接口为Spark新增自定义数据源</p> <p>数据源API目前分V1和V2版本，<del>到目前为止<a class="link" href="https://spark.apache.org/releases/spark-release-3-0-0.html" target="_blank" rel="noopener" ><em>Spark 3.0.0</em></a>似乎还没有完成进化</del>，已在3.0.0版本完成V2版重构</p> <p><a class="link" href="https://issues.apache.org/jira/browse/SPARK-25390" target="_blank" rel="noopener" >Data source V2 API refactoring</a></p> <p>预计将在3.2.0版本将V2版API稳定下来</p> <p><a class="link" href="https://issues.apache.org/jira/browse/SPARK-25186" target="_blank" rel="noopener" >Stabilize Data Source V2 API</a></p> <p><a class="link" href="https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-data-source-api-v2.html" target="_blank" rel="noopener" >https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-data-source-api-v2.html</a></p> <p><a class="link" href="https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-DataSourceV2.html" target="_blank" rel="noopener" >https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-DataSourceV2.html</a></p> <p><a class="link" href="https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-DataSource.html" target="_blank" rel="noopener" >https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-DataSource.html</a></p> <p><a class="link" href="http://blog.madhukaraphatak.com/categories/datasource-v2-series/" target="_blank" rel="noopener" >Category: datasource-v2-series</a></p> <p><a class="link" href="http://blog.madhukaraphatak.com/categories/datasource-v2-spark-three/" target="_blank" rel="noopener" >Category: datasource-v2-spark-three</a></p> <h2 id="spark-sql---csv">Spark SQL - CSV </h2><p>CSV类型文件中，出于各种原因可能导致Spark SQL解析数据会出错。</p> <blockquote> <p>以下问题举例在Hadoop2.6.0-Spark2.1.1-Scala2.10.6-JDK1.7生产环境出现，较新版本中的Spark具体行为暂不可知。该Spark版本已被魔改且无代码，离线环境中只有Spark2.4.4-Scala2.11，尝试看下源代码发现该部分已被重构，抛异常的类都没有了</p> </blockquote> <p>例如，有些字段里面包含了特殊字符，导致Spark SQL解析行数据时出现了字段截断错误，从而导致列错位，有些转换函数直接执行失败，进而导致整个任务失败。</p> <p>问题解决方式是强制指定<code>mode=DROPMALFORMED</code>，直接将问题数据丢弃，这是Spark SQL直接支持的配置，看文档的时候可能看到了，但是无视掉了。。。</p> <p>Spark文档中对于CSV支持的配置有详细介绍。</p> <p>最新版本的参考文档：<a class="link" href="https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/DataFrameReader.html#csv%28paths:String*%29:org.apache.spark.sql.DataFrame" target="_blank" rel="noopener" >DataFrameReader#csv</a></p> <p>Spark 2.4.6参考文档：<a class="link" href="https://spark.apache.org/docs/2.4.6/api/scala/index.html#org.apache.spark.sql.DataFrameReader@csv%28paths:String*%29:org.apache.spark.sql.DataFrame" target="_blank" rel="noopener" >DataFrameReader#csv</a></p> <h2 id="spark-cli">Spark CLI </h2><p>要脱离灵活性太差的自研任务调度服务、逐渐开始习惯用原生CLI进行进行任务的提交</p> <p><code>spark-submit --name JOB-NAME --master yarn --deploy-mode cluster --conf spark.yarn.submit.waitAppCompletion=false --class com.mxtao.App --jars /xxx/xxx.jar,/xxx/xxxx.jar --queue xx --pincipal xxx@DOMAN --keytab xxx.keytab main-class-in-this-jar.jar args-for-main</code></p> <p><a class="link" href="https://spark.apache.org/docs/latest/submitting-applications.html" target="_blank" rel="noopener" >Submitting Applications</a></p> <p><a class="link" href="https://spark.apache.org/docs/latest/running-on-yarn.html#spark-properties" target="_blank" rel="noopener" >Running Spark on YARN - Spark Properties</a></p> <p><a class="link" href="https://spark.apache.org/docs/2.4.6/submitting-applications.html" target="_blank" rel="noopener" >Submitting Applications</a></p> <p><a class="link" href="https://spark.apache.org/docs/2.4.6/running-on-yarn.html#spark-properties" target="_blank" rel="noopener" >Running Spark on YARN - Spark Properties</a></p>