FATAL Fatal error during KafkaServer startup, NumberFormatException

This might have taken me a long time to figure out, but fortunately my super awesome colleague (nicknamed Mr.T, he also pities fools) showed me the solution.
We ran into a Kafka Broker which was not starting, and gave the exception that is pasted in the bottom of this post.

FATAL Fatal error during KafkaServer startup. [..] java.lang.NumberFormatException

The solution is that the string (in this example “hs_err_pid19313”) is actually an error log which exists in a topic partition directory. (Re)move this file and Kafka will start without a problem.
(Tip: use find and grep to quickly find the file, go to your Kafka storage directory and run the following command;

find . |grep hs_err_pid19313

Continue reading “FATAL Fatal error during KafkaServer startup, NumberFormatException”

Citrix Receiver on Linux: SSL Error 61 ("You have not chosen to trust")

If you don’t know or understand certificates / root and intermediate certificate authorities, get someone who understands to follow below instructions.

I tried connecting to the company’s citrix server, but kept hitting the same error when I tried to open the connection:

Contact your help desk with the following information:
You have not chosen to trust "INSERT YOUR CA HERE",
the issuer of the server's security certificate (SSL Error 61)

It seems that Citrix has an alternate directory where it stores it’s trusted cert’s / certificate authorities. Even though you can see that the server’s certificate is trusted (by root CA’s) via a web browser, we need to copy those to the correct directory.
In short: Copy the root and intermediate CA’s to this directory: /opt/Citrix/ICAClient/keystore/cacerts
Continue reading “Citrix Receiver on Linux: SSL Error 61 ("You have not chosen to trust")”

This ZooKeeper instance is not currently serving requests

When one of your zookeeper nodes is sending you this message, that means that your Zookeeper cluster hasn’t started in the right order.
Solution: Restart your cluster (node per node), starting from node 1 (as stated in zoo.conf)
This problem is easy to diagnose. When the order was wrong you will get this output:

[myserver:myuser] ~: echo stat | nc localhost 2181
This ZooKeeper instance is not currently serving requests

After you’ve restarted all nodes (in the correct order), you will get this output:

[myserver:myuser] ~: echo stat | nc localhost 2181 |grep Mode
Mode: follower
[myserver:myuser] ~: echo stat | nc localhost 2181 |grep Mode
Mode: leader

Hope this will help you out!

Why no SSL!? Port is open!

Okay, this has taken me too long to not post.. So here it is..:
When your firewall is blocking SSL traffic but allowing HTTP traffic, openssl s_client will show this:

my_host:joris [/etc/stores] openssl s_client -host external_host -port 12345
no peer certificate available
No client certificate CA names sent
SSL handshake has read 0 bytes and written 247 bytes
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE

Continue reading “Why no SSL!? Port is open!”

Exclude grep itself from ps

This is so simple it’s just great 🙂
Solution: use regex in your grep so the grep itself doesn’t show up in the results.

[vagrant@london kafka]$ ps aux |grep kafka
vagrant 5172 0.8 30.3 3178252 309428 ? Sl 07:00 0:06 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/var/log/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/var/log/kafka -Dlog4j.configuration=file:/etc/kafka/log4j.properties -cp :/usr/bin/../share/java/kafka/*:/usr/bin/../share/java/confluent-support-metrics/*:/usr/share/java/confluent-support-metrics/* io.confluent.support.metrics.SupportedKafka /vagrant/config/kafka0.properties
vagrant 5824 0.0 0.0 103316 836 pts/0 R+ 07:13 0:00 grep kafka  <<-- Oh no!
[vagrant@london kafka]$
[vagrant@london kafka]$ ps aux |grep [k]afka
root 5172 0.8 29.6 3178252 302472 ? Sl 07:00 0:04 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/var/log/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/var/log/kafka -Dlog4j.configuration=file:/etc/kafka/log4j.properties -cp :/usr/bin/../share/java/kafka/*:/usr/bin/../share/java/confluent-support-metrics/*:/usr/share/java/confluent-support-metrics/* io.confluent.support.metrics.SupportedKafka /vagrant/config/kafka0.properties
[vagrant@london kafka]$


Using SSH to forward the same local port to multiple external hosts

Okay, this is kinda awesome :-), I got my geek on 🙂
My application is connecting to a cluster of external servers but my application can configure hostname but can’t configure port.
So I wanted to connect to a remote cluster using SSH tunneling, but I was unable to forward everything because the port binding to localhost ( can only be used once.
Then I saw that you can use multiple loopback addresses! See this page: https://en.wikipedia.org/wiki/Loopback
Basically you can bind the portforward to, till, that should provide enough addresses, right!? 🙂
So I can use multiple port forwards from my localhost(s) to the six remote hosts like this:

ssh somedomain.com \
-L \
-L \
-L \
-L \
-L \

Vagrant proxy through CNTLM on Windows using CYGWIN

Wow, talk about a crappy post title, but I just got this working on my corporate network and was quite happy about it.
Reason for this post; Every time I start a new assignment at a (rather big) corporation, I need to follow these steps to be able to access the internet from my vm’s.
(and I keep forgetting the steps, I’m getting old..)
What we’ll use to get this working:

  • Cygwin : https://www.cygwin.com/
  • Vagrant : https://www.vagrantup.com/
  • Virtualbox : https://www.virtualbox.org/
  • Cntlm : http://cntlm.sourceforge.net/

I’m running CentOS vm’s inside Vagrant with Virtualbox provisioning on Cygwin on Windows 7. I’m running Cntlm to create a local proxy for all stuff what I’m doing through cygwin, because I don’t like putting clear text passwords in bashrc or in Win / Bash variables.


  1. Get your corporate proxy URL (Via Google Chrome)
  2. Configure Cntlm
  3. Configure Cygwin
  4. Configure Vagrant
  5. Use teh interwebs from your VM, practicing ninja turtle coding skillz and be instantly awesome!!1!

Continue reading “Vagrant proxy through CNTLM on Windows using CYGWIN”

Comparing sed stream output in linux

Sed is very very powerful, which is a good thing to be aware of.
I was looking to compare the output of a sed command to the original file before I wanted to execute the sed command directly on the file and came across this handy trick.
It works by using temporary named pipes inside the diff command.
Contents of file:
[code title=”contents of numbers.txt:”]
If I just want to remove the line which begins with “Four”, I can check my sed command like this:
[code title=”Terminal output:”]
joris@beanie ~
$ diff <(sed ‘/Four/d’ numbers.txt) numbers.txt
> Four
Awesome possum, now I know my sed command won’t destroy anything.

Weblogic, 1 managed server gives HTTP Error 401 Unauthorized, other works fine.

I encountered this weird behaviour last week where one managed server in a cluster of two servers constantly gave the error “HTTP Error 401 Unauthorized“.
The setup was a standard Oracle Service Bus installation on Oracle Weblogic.
The domain consisted out of one Admin server, one cluster with two managed servers.
Managed Server #1 was acting without problems but all the requests which needed authentication and which where pointed to Managed Server #2 failed and responded with a 401 Unauthorized message.
There was an error in the logs of MS#2, which is displayed at the bottom of this post as well.
#### <> <Inbound http BASIC authentication failed
javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User webhosting javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User webhosting denied

The solution to this problem lies in the internal LDAP configuration of the faulty Managed Server. I did not find an answer to what might have caused this problem, but the solution was to rebuild the LDAP setting of the Managed Server.
This is done by following these steps:

  1. Shut down Managed Server via Weblogic Console
  2. Log in via SSH
  3. Rename the following folder: %domain_directory%/servers/%osb_managed_server_1%/data/ldap
  4. Start the Managed Server via Weblogic Console

These steps will rebuild the LDAP folder, which is the internal LDAP to which Weblogic authenticates.
You can then remove the folder you’ve backed up in step 3.
Continue reading “Weblogic, 1 managed server gives HTTP Error 401 Unauthorized, other works fine.”