This might have taken me a long time to figure out, but fortunately my super awesome colleague (nicknamed Mr.T, he also pities fools) showed me the solution.
We ran into a Kafka Broker which was not starting, and gave the exception that is pasted in the bottom of this post.
FATAL Fatal error during KafkaServer startup. [..] java.lang.NumberFormatException
The solution is that the string (in this example “hs_err_pid19313”) is actually an error log which exists in a topic partition directory. (Re)move this file and Kafka will start without a problem.
(Tip: use find and grep to quickly find the file, go to your Kafka storage directory and run the following command;
find . |grep hs_err_pid19313
The full trace is pasted below:
[2017-08-01 15:56:48,456] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)[2017-08-01 15:56:48,456] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)java.lang.NumberFormatException: For input string: "hs_err_pid19313" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:589) at java.lang.Long.parseLong(Long.java:631) at scala.collection.immutable.StringLike$class.toLong(StringLike.scala:276) at scala.collection.immutable.StringOps.toLong(StringOps.scala:29) at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:195) at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:179) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) at kafka.log.Log.loadSegments(Log.scala:179) at kafka.log.Log.<init>(Log.scala:108) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:151) at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)[2017-08-01 15:56:48,480] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)java.lang.NumberFormatException: For input string: "hs_err_pid19313" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:589) at java.lang.Long.parseLong(Long.java:631) at scala.collection.immutable.StringLike$class.toLong(StringLike.scala:276) at scala.collection.immutable.StringOps.toLong(StringOps.scala:29) at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:195) at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:179) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) at kafka.log.Log.loadSegments(Log.scala:179) at kafka.log.Log.<init>(Log.scala:108) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:151) at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)