欢迎访问小程序员!

spark出现GC overhead limit exceeded和java heap space

2014-10-14 2条评论 8977次浏览

    spark执行任务时出现java.lang.OutOfMemoryError: GC overhead limit exceeded和java.lang.OutOfMemoryError: java heap space

最直接的解决方式就是在spark-env.sh中将下面两个参数调节的尽量大

    export SPARK_EXECUTOR_MEMORY=6000M
    export SPARK_DRIVER_MEMORY=7000M

注意,此两个参数设置需要注意大小顺序:

    SPARK_EXECUTOR_MEMORY < SPARK_DRIVER_MEMORY< yarn集群中每个nodemanager内存大小


总结一下Spark中各个角色的JVM参数设置:    

(1)Driver的JVM参数:
    -Xmx,-Xms,如果是yarn- client模式,则默认读取spark-env文件中的SPARK_DRIVER_MEMORY值,-Xmx,-Xms值一样大小;如果是yarn- cluster模式,则读取的是spark-default.conf文件中的spark.driver.extraJavaOptions对应的JVM 参数值。
    PermSize,如果是yarn-client模式,则 是默认读取spark-class文件中的JAVA_OPTS="-XX:MaxPermSize=256m $OUR_JAVA_OPTS"值;如果是yarn-cluster模式,读取的是spark-default.conf文件中的 spark.driver.extraJavaOptions对应的JVM参数值。
    GC 方式,如果是yarn-client模式,默认读取的是spark-class文件中的JAVA_OPTS;如果是yarn-cluster模式,则读取 的是spark-default.conf文件中的spark.driver.extraJavaOptions对应的参数值。
以上值最后均可被spark-submit工具中的--driver-java-options参数覆盖。

(2)Executor的JVM参数:
    -Xmx,-Xms,如果是 yarn-client模式,则默认读取spark-env文件中的SPARK_EXECUTOR_MEMORY值,-Xmx,-Xms值一样大小;如果 是yarn-cluster模式,则读取的是spark-default.conf文件中的 spark.executor.extraJavaOptions对应的JVM参数值。
    PermSize,两种模式都是读取的是spark-default.conf文件中的spark.executor.extraJavaOptions对应的JVM参数值。
    GC方式,两种模式都是读取的是spark-default.conf文件中的spark.executor.extraJavaOptions对应的JVM参数值。

(3)Executor数目及所占CPU个数
    如果是yarn-client模式,Executor数目由spark-env中的SPARK_EXECUTOR_INSTANCES指定,每个实例的数目由SPARK_EXECUTOR_CORES指定;如果是yarn-cluster模式,Executor的数目由spark-submit工具的--num-executors参数指定,默认是2个实例,而每个Executor使用的CPU数目由--executor-cores指定,默认为1核。


发表评论

登录,将以游客形式发表

网友最新评论(2)

  1. 1楼 151.249.164.*** [游客] 发表于 2017-08-31 13:34:53 回复此评论
    Hello friends! I am an official representative of private company which deals with all kinds of written work (essay, coursework, dissertation, presentation, report, etc) in short time. We are ready to offer a free accomplishment of written work hoping for further cooperation and honest feedback about our service. Send your work topics to our email: discount@edu-paper.com. This offer has limited quantities!!!
  2. 2楼 151.249.164.*** [游客] 发表于 2017-10-17 11:36:59 回复此评论
    Hello friends! I am an official representative of private company which deals with all kinds of written work (essay, coursework, dissertation, presentation, report, etc) in short time. We are ready to offer a free accomplishment of written work hoping for further cooperation and honest feedback about our service. This offer has limited quantities!!! Details on our website: >>> l2immortal.com