Thank me later!

Hadoop filesystem interface has this nice little secret which is not advertised to the outside world directly in any documentation. Your nice, clean FileSystem#get API actually caches the FileSystem objects and returns cached ones, if they are available. This perfectly makes sense till you have a filesystem, which is thread safe to be used between multiple mappers/reducers. When you have a Custom Filesystem implemenation which can't be safely re-used across different threads, there is a possibility you might end up in debugging hell. Unfortunately, I had to to face that ordeal, while implementing and using a custom SFTPFilesystem. I started getting JVM crashes, outOfMemory to create native thread errors etc. So after a day long scrounging through code and hair pulling, I figured out its the cache filesystem which is doing that, and I had to set


in the hadoop configuration. And we lived happily ever after. Phew!

Published at : Wednesday Feb,24 2016 at 23:13 | Tags : hacking, java, hadoop, | View Comments
Mesos is fun!

I was recently involved in writing a mesos framework to autoscale GoCD agents.

My development setup involves:

The framework runs two threads. First thread is the actual framework implementation which listens to messages from mesos-master. The second thread is a Http poller, which polls the GoCD server to find out the demand and supply of agents. If the demand > supply, the framework launches a new go agent.

The Poller part was pretty straight forward to implement. While the framework part is based on standard interface all mesos frameworks follow, there were minor gotchas to get it working.

Setting mesos native lib in the PATH

Exception in thread "main" java.lang.UnsatisfiedLinkError: no mesos in java.library.path

The above error is because the framework can't find the mesos native lib. Explicitly set MESOS_NATIVE_JAVA_LIBRARY=/path/to/mesos/native/lib In Mac OSX, it is usually inside /usr/local/lib/libmesos.dylib, after you brew install mesos.

Making framework visible to the mesos-master

Mesos frameworks are usually run on the same machine as the master or in a machine is publicly accesible from the master. During development there is a high chance that your framework is binding to the localhost ip, which is not visible in the public network. Which will result in very cryptic errors like below.

On framework logs in local, it will be stuck at

sched.cpp:264] No credentials provided. Attempting to register without authentication

In the mesos-master logs you will see messages like:

master.cpp:1423] Received re-registration request from framework GOCD-Mesos-1456332472341 at scheduler-3bfce855-b59c-4ebe-bb04-567770e04f5a@ master.cpp:1474] Re-registering framework GOCD-Mesos-1456332472341 at scheduler-3bfce855-b59c-4ebe-bb04-567770e04f5a@ master.cpp:1501] Framework GOCD-Mesos-1456332472341 failed over hierarchical_allocator_process.hpp:375] Activated framework GOCD-Mesos-1456332472341 master.cpp:3559] Sending 1 offers to framework GOCD-Mesos-1456332472341 master.cpp:725] Framework GOCD-Mesos-1456332472341 disconnected master.cpp:1655] Deactivating framework GOCD-Mesos-1456332472341 hierarchical_allocator_process.hpp:405] Deactivated framework GOCD-Mesos-1456332472341 hierarchical_allocator_process.hpp:563] Recovered cpus():1; mem():378; disk():32808; ports():[31000-32000] (total allocatable: cpus():1; mem():378; disk():32808; ports():[31000-32000]) on slave 20160224-163417-169978048-5050-1266-0 from framework GOCD-Mesos-1456332472341

The problem here is the mesos-master isn't able to communicate back the framework because the framework isn't visible on the public interface. The solution is to set LIBPROCESS_IP=public_interface_ip and restart the framework.

Specify exact user to run command on slaves

The gocd-mesos framework launches go agents as docker containers. When the framework launches a new mesos task, the task needs a explicit user to be specified along with the Task Config. If else it assumes the same user as the framwework is running. This is problematic when your framework and mesos-slaves are running in different machines, which have different uid. To solve this, the mesos-slave should be started with a --switch-user option enabled.

All the above problems are not easily found on a google search. Hope this may help a soul not loose sleepless nights figuring it out. Also mesos framework development is fun. If you are running a instance of GoCD, checkout the progress and if possible support/contribute the development here.

Published at : Wednesday Feb,24 2016 at 22:40 | Tags : hacking, mesos, scala, | View Comments
Finally, I could call myself a runner :P

Last week sunday(07/12/2014), I ran my first half marathon at The Wipro Chennai Marathon'14. My timing was 3:09:23. Which I think is decent for a person of my physique with no proper training whatsoever.

I didn't have any timing expectations to start with. Infact even a week before the event, I was cursing myself that I have did this half marathon registration with overconfidence and I should have just ran a 10K and hoped for my personal best timing. A week before the event, I tried to run a 15K but got exhausted at 11K and had to cut short the run and return home. To top all that, I woke up with a heavy cold, and throbbing head the day previous to the event. Scared and hopeless, I ran to the medical shop and bought a 'Benadryl' syrup and took two servings, hoping that would help me with the situation ;-) Infact it did, the next day my cold was much better.

As usual with the excitement and scare, I slept barely for 3 hours the previous night. So, with all these - I really didn't have much expectations for the run. All I was hoping was to somehow finish the run atleast by crawling the last few kms. It may sound cliche, but seeing that big crowd at 5 in the morning, all waiting to run those 21kms cheering up each other was a big mental boost. The first half of the run was pretty much eventless, I could comfortably complete the 10K mark in my usual pace of 7.30. Right after 11K the legs started to disobey. I reduced the pace and shifted to Run-Walk-Run mode. That worked for sometime but after 15-16K I just couldn't tell my body to run. It flatly refused. I even skipped the last two refreshment points, fearing that once I stop I may not be able to walk further. I strongly felt, marathon as much as it is physical endurance test, it is a mental endurance one as well. Literally at every step I have fight with my body to lift the legs. At some point, it was frustrating and the whole thing looked pointless. I was like, "Why on earth am I going through all this shit?. I should have just ran a 10K and have had a enjoyable experience. This fucking thing is nowhere near enjoyment or fun :-/" I even sweared to myself to never attempt another half marathon in my life :-) At one such frustrating moment, I saw someone holding a board which said - "Pains fade.. Muscles heal.. But the achievement - thats priceless!". Those things WERE HELPFUL atleast to me. Whoever out there who was holding that board, a big thank you to you!

Atlast, when I saw the 1K remaining mark, somehow something happened and I decided to run to completion no matter what. I did RUN the last KM and the moment I reached the finish line - there are only very few moments in life where I have felt extremely happy and prideful - that was one of them. All those pain, agony and frustration went away and I was super active again :) That was surely magical and one hell of a experience!

PS: Thanks to the TWCM team, there is nothing I could possibly point as bad. Only if they provide something other than that stupid burger for the post run refreshment.. Sigh!

PPS: I have already planned my next half marathon :P Discovering running was one of the best things happened to me :)

Published at : Wednesday Dec,10 2014 at 22:00 | Tags : running, | View Comments
Special thanks to my friend who made it awesome!

Email as far as I am concerned is the best way to communicate complex matters both professionally and personally. Best way among the modern day communication channels. Vocal talks, either Face to Face or Through phone/VOIP is too distractive for me. I couldn't tell exactly things I wanted to tell, because there is distraction in some or other way during the talk. Also I suck at choosing right words at the first time, so end up saying something awkward which spoils the whole talk. Email gives the comfort to present the thoughts in a undistracted way. Allows me time to choose the words correctly. And it has always worked, both in personal and professional fronts. And all the above points works for the opposite party as well. There is no instant response expected of you, so you can take time to go through the email, think about it and then reply. Exactly the reason why gtalk/whatsapp/* are not really working for me to do serious conversations. Email has done wonders and has also healed bad wounds for me!

Yesterday was one such day, where I ended up sending a long personal email to a friend and it fucking made my day :D I took 2 hours to draft that 50+ line email. Dumped all thoughts in nice way, and it kind of ended a 3 year long misunderstanding between us. I have tried this many times during vocal communication, or through instant messengers like gtalk/whatsapp/* but everytime it was too distractive to exactly convey the point. So again Email worked wonders, and both of us ended the day yesterday in a really happy note.

I can only imagine the joy people had in conversing through snail mails till the last decade. I decided I should try that once, before it becomes obselete like the telegram :)

PS: This post in addition to being a rant about email, remains here as a marker to the day, I ended a long standing dispute with my bestest ever friend! I've never been so happy recently. Thank you :)

Published at : Wednesday Oct,09 2013 at 15:27 | Tags : happymoments, personal, | View Comments
Git never ceases to amaze me!

At work, we use Subversion for version control. SVN is really good - but it disturbs my workflow which is 'Commit Early, Commit Often'. People create builds three or four times a day for testing and obviously they don't want my Work In Progress commits in that. They expect me to commit only after a feature is completely done. That was handicapping my pace of development.

So, the solution - Enter git-svn :D

git-svn allows me to checkout a subversion repo as a git repository. Since my working copy is a git repo now, I get all the fantastic features of git. At the start, I was using git in a very minimal way - make commits locally, created local branches and seamlessly switched between them. Even that helped greatly to my development pace.

As days pass by, I keep on discovering lot of gems in git - which make me like,

"Holy F, How did I exist without this these many days??" :P

Discovering git squash is one such incident. git bisect was another one. Today is one such day.

At work we use Target Process for project management. Issues, Requests, Feedbacks, Usecases everything is created in TargetProcess and we use the related Id in our commits. We don't commit anything to SVN without mentioning TargetProcess artifact Id.

There is a new feature which came as a request through email - and I have started development on it - made a few commits to my git repo. That feature was not yet added to the TP, so I can't push it to the SVN repo. While this was being done, another important bug fix came and I had to fix it immediatly and push it to the SVN repo. Now the unfinished feature's commits are there and on top of it there is this urgent bug fix commit. I need to push only the bug fix commit to the SVN repo without pushing any of the new feature's commit. I could've done the development of the new feature in a local branch - keeping the master clean. But that is not a option now. Damage is done. What I did is this:

  • I converted both the bugfix and new feature changes into git-am patches.
  • Deleted both the commits and made the master as clean slate.
  • Applied only the bug fix commit.
  • Moved the other feature commits to a separate branch.
  • Pushed to bug fix to the SVN.
  • Be happy!

I know this is trivial. May be some other better way could be there. But, the power and flexibility of git just made me fall in love with it all over again :D git never ceases to amaze me!

Published at : Thursday Mar,28 2013 at 00:20 | Tags : hacking, git, Subversion, svn, | View Comments
Forgive me if I'm bragging too much!

So here I am after running my first ever full 5km run. I should thank a lot of people for inspiring me to start running - most importantly joe. I know 5k is too small to be a significant distance. The 40.53 mins I took for running it is not really great. But above all I still feel very great to have done this. Currently I weigh 3kgs less than a century. Being a lazy bum, I have NEVER done any physical sports activites all through my childhood. So this insignificant run holds a big inspiration for me. Already looking forward to Kaveri Trail Marathon and TCS world 10k, which are coming up in next few months. Here is a photo of myself and karthik after finishing the run.

Click on the pic to get a better view.

I have been recording most of my runs in runkeeper. The 5k run is here

I hope running becomes part of my lifestyle. Wish me luck!

Published at : Sunday Mar,03 2013 at 10:51 | Tags : running, happymoments, | View Comments
Browse Archives
I am..
Azhagu Selvan SP
Atheist, FOSS enthusiast,
Pythonist, Student
Creative Commons License