我想停止 spark shell 上的各种消息。
我试图编辑log4j.properties文件以停止这些消息。
log4j.properties
以下是内容log4j.properties
# Define the root logger with appender file log4j.rootCategory=WARN, console log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.target=System.err log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n # Settings to quiet third party logs that are too verbose log4j.logger.org.eclipse.jetty=WARN log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
但是消息仍然显示在控制台上。
以下是一些示例消息
15/01/05 15:11:45 INFO SparkEnv: Registering BlockManagerMaster 15/01/05 15:11:45 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150105151145-b1ba 15/01/05 15:11:45 INFO MemoryStore: MemoryStore started with capacity 0.0 B. 15/01/05 15:11:45 INFO ConnectionManager: Bound socket to port 44728 with id = ConnectionManagerId(192.168.100.85,44728) 15/01/05 15:11:45 INFO BlockManagerMaster: Trying to register BlockManager 15/01/05 15:11:45 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager 192.168.100.85:44728 with 0.0 B RAM 15/01/05 15:11:45 INFO BlockManagerMaster: Registered BlockManager 15/01/05 15:11:45 INFO HttpServer: Starting HTTP Server 15/01/05 15:11:45 INFO HttpBroadcast: Broadcast server star
我该如何阻止这些?
感谢 @AkhlD 和 @Sachin Janani 建议更改.conf文件。
.conf
以下代码解决了我的问题:
1)import org.apache.log4j.{Level, Logger}在导入部分添加
import org.apache.log4j.{Level, Logger}
2)在创建火花上下文对象后添加了以下行,即之后val sc = new SparkContext(conf):
val sc = new SparkContext(conf)
val rootLogger = Logger.getRootLogger() rootLogger.setLevel(Level.ERROR)