Today I wanted to check some stuff on a pod running in my local kubernetes cluster. I’m still getting used to Kubernetes, thus didn’t want something ugly like exposing that pod.
The easiest way of doing it for me has been to spin up a temporary pod using Ubuntu;
kubectl run myshell --rm -i --tty --image ubuntu -- /bin/bashNow you can do whatever you want in the pod and when you log out of myshell, it’ll be removed immediately and we’re back to square one! <3
This might have taken me a long time to figure out, but fortunately my super awesome colleague (nicknamed Mr.T, he also pities fools) showed me the solution.
We ran into a Kafka Broker which was not starting, and gave the exception that is pasted in the bottom of this post.
FATAL Fatal error during KafkaServer startup. [..] java.lang.NumberFormatException
The solution is that the string (in this example “hs_err_pid19313”) is actually an error log which exists in a topic partition directory. (Re)move this file and Kafka will start without a problem.
(Tip: use find and grep to quickly find the file, go to your Kafka storage directory and run the following command;
find . |grep hs_err_pid19313
Important:
If you don’t know or understand certificates / root and intermediate certificate authorities, get someone who understands to follow below instructions.
I tried connecting to the company’s citrix server, but kept hitting the same error when I tried to open the connection:
Contact your help desk with the following information: You have not chosen to trust "INSERT YOUR CA HERE", the issuer of the server's security certificate (SSL Error 61)
It seems that Citrix has an alternate directory where it stores it’s trusted cert’s / certificate authorities. Even though you can see that the server’s certificate is trusted (by root CA’s) via a web browser, we need to copy those to the correct directory.
In short: Copy the root and intermediate CA’s to this directory: /opt/Citrix/ICAClient/keystore/cacerts
(more…)
When one of your zookeeper nodes is sending you this message, that means that your Zookeeper cluster hasn’t started in the right order.
Solution: Restart your cluster (node per node), starting from node 1 (as stated in zoo.conf)
This problem is easy to diagnose. When the order was wrong you will get this output:
[myserver:myuser] ~: echo stat | nc localhost 2181 This ZooKeeper instance is not currently serving requests
After you’ve restarted all nodes (in the correct order), you will get this output:
[myserver:myuser] ~: echo stat | nc localhost 2181 |grep Mode Mode: follower [myserver:myuser] ~: echo stat | nc localhost 2181 |grep Mode Mode: leader
Hope this will help you out!
Okay, this has taken me too long to not post.. So here it is..:
When your firewall is blocking SSL traffic but allowing HTTP traffic, openssl s_client will show this:
my_host:joris [/etc/stores] openssl s_client -host external_host -port 12345 CONNECTED(00000003) write:errno=104 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 247 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE ---
This is so simple it’s just great 🙂
Solution: use regex in your grep so the grep itself doesn’t show up in the results.
Example:
[vagrant@london kafka]$ ps aux |grep kafka vagrant 5172 0.8 30.3 3178252 309428 ? Sl 07:00 0:06 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/var/log/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/var/log/kafka -Dlog4j.configuration=file:/etc/kafka/log4j.properties -cp :/usr/bin/../share/java/kafka/*:/usr/bin/../share/java/confluent-support-metrics/*:/usr/share/java/confluent-support-metrics/* io.confluent.support.metrics.SupportedKafka /vagrant/config/kafka0.properties vagrant 5824 0.0 0.0 103316 836 pts/0 R+ 07:13 0:00 grep kafka <<-- Oh no! [vagrant@london kafka]$ [vagrant@london kafka]$ ps aux |grep [k]afka root 5172 0.8 29.6 3178252 302472 ? Sl 07:00 0:04 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/var/log/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/var/log/kafka -Dlog4j.configuration=file:/etc/kafka/log4j.properties -cp :/usr/bin/../share/java/kafka/*:/usr/bin/../share/java/confluent-support-metrics/*:/usr/share/java/confluent-support-metrics/* io.confluent.support.metrics.SupportedKafka /vagrant/config/kafka0.properties [vagrant@london kafka]$
Okay, this is kinda awesome :-), I got my geek on 🙂
My application is connecting to a cluster of external servers but my application can configure hostname but can’t configure port.
So I wanted to connect to a remote cluster using SSH tunneling, but I was unable to forward everything because the port binding to localhost (127.0.0.1) can only be used once.
Then I saw that you can use multiple loopback addresses! See this page: https://en.wikipedia.org/wiki/Loopback
Basically you can bind the portforward to 127.0.0.2, 127.0.0.3 till 127.255.255.254, that should provide enough addresses, right!? 🙂
So I can use multiple port forwards from my localhost(s) to the six remote hosts like this:
ssh somedomain.com \ -L 127.0.0.1:9042:external-node1.somedomain.com:9042 \ -L 127.0.0.2:9042:external-node2.somedomain.com:9042 \ -L 127.0.0.3:9042:external-node3.somedomain.com:9042 \ -L 127.0.0.4:9042:external-node4.somedomain.com:9042 \ -L 127.0.0.5:9042:external-node5.somedomain.com:9042 \ -L 127.0.0.6:9042:external-node6.somedomain.com:9042
Wow, talk about a crappy post title, but I just got this working on my corporate network and was quite happy about it.
Reason for this post; Every time I start a new assignment at a (rather big) corporation, I need to follow these steps to be able to access the internet from my vm’s.
(and I keep forgetting the steps, I’m getting old..)
What we’ll use to get this working:
I’m running CentOS vm’s inside Vagrant with Virtualbox provisioning on Cygwin on Windows 7. I’m running Cntlm to create a local proxy for all stuff what I’m doing through cygwin, because I don’t like putting clear text passwords in bashrc or in Win / Bash variables.
This post is mostly a bookmark for myself. I’ve been using search engines way too often to find this command..
The command to remove a host from a knownhosts file without seeing the actual hostname in the knownhosts file is the following:
[code]ssh-keygen -R HOSTNAME[/code]
(more…)
Sed is very very powerful, which is a good thing to be aware of.
I was looking to compare the output of a sed command to the original file before I wanted to execute the sed command directly on the file and came across this handy trick.
It works by using temporary named pipes inside the diff command.
Contents of file:
[code title=”contents of numbers.txt:”]
One
Two
Three
Four
Five
[/code]
If I just want to remove the line which begins with “Four”, I can check my sed command like this:
[code title=”Terminal output:”]
joris@beanie ~
$ diff <(sed ‘/Four/d’ numbers.txt) numbers.txt
3a4
> Four
[/code]
Awesome possum, now I know my sed command won’t destroy anything.