Thank me later!

Hadoop filesystem interface has this nice little secret which is not advertised to the outside world directly in any documentation. Your nice, clean FileSystem#get API actually caches the FileSystem objects and returns cached ones, if they are available. This perfectly makes sense till you have a filesystem, which is thread safe to be used between multiple mappers/reducers. When you have a Custom Filesystem implemenation which can't be safely re-used across different threads, there is a possibility you might end up in debugging hell. Unfortunately, I had to to face that ordeal, while implementing and using a custom SFTPFilesystem. I started getting JVM crashes, outOfMemory to create native thread errors etc. So after a day long scrounging through code and hair pulling, I figured out its the cache filesystem which is doing that, and I had to set


in the hadoop configuration. And we lived happily ever after. Phew!

Published at : Wednesday Feb,24 2016 at 23:13 | Tags : hacking, java, hadoop, | View Comments
Mesos is fun!

I was recently involved in writing a mesos framework to autoscale GoCD agents.

My development setup involves:

The framework runs two threads. First thread is the actual framework implementation which listens to messages from mesos-master. The second thread is a Http poller, which polls the GoCD server to find out the demand and supply of agents. If the demand > supply, the framework launches a new go agent.

The Poller part was pretty straight forward to implement. While the framework part is based on standard interface all mesos frameworks follow, there were minor gotchas to get it working.

Setting mesos native lib in the PATH

Exception in thread "main" java.lang.UnsatisfiedLinkError: no mesos in java.library.path

The above error is because the framework can't find the mesos native lib. Explicitly set MESOS_NATIVE_JAVA_LIBRARY=/path/to/mesos/native/lib In Mac OSX, it is usually inside /usr/local/lib/libmesos.dylib, after you brew install mesos.

Making framework visible to the mesos-master

Mesos frameworks are usually run on the same machine as the master or in a machine is publicly accesible from the master. During development there is a high chance that your framework is binding to the localhost ip, which is not visible in the public network. Which will result in very cryptic errors like below.

On framework logs in local, it will be stuck at

sched.cpp:264] No credentials provided. Attempting to register without authentication

In the mesos-master logs you will see messages like:

master.cpp:1423] Received re-registration request from framework GOCD-Mesos-1456332472341 at scheduler-3bfce855-b59c-4ebe-bb04-567770e04f5a@ master.cpp:1474] Re-registering framework GOCD-Mesos-1456332472341 at scheduler-3bfce855-b59c-4ebe-bb04-567770e04f5a@ master.cpp:1501] Framework GOCD-Mesos-1456332472341 failed over hierarchical_allocator_process.hpp:375] Activated framework GOCD-Mesos-1456332472341 master.cpp:3559] Sending 1 offers to framework GOCD-Mesos-1456332472341 master.cpp:725] Framework GOCD-Mesos-1456332472341 disconnected master.cpp:1655] Deactivating framework GOCD-Mesos-1456332472341 hierarchical_allocator_process.hpp:405] Deactivated framework GOCD-Mesos-1456332472341 hierarchical_allocator_process.hpp:563] Recovered cpus():1; mem():378; disk():32808; ports():[31000-32000] (total allocatable: cpus():1; mem():378; disk():32808; ports():[31000-32000]) on slave 20160224-163417-169978048-5050-1266-0 from framework GOCD-Mesos-1456332472341

The problem here is the mesos-master isn't able to communicate back the framework because the framework isn't visible on the public interface. The solution is to set LIBPROCESS_IP=public_interface_ip and restart the framework.

Specify exact user to run command on slaves

The gocd-mesos framework launches go agents as docker containers. When the framework launches a new mesos task, the task needs a explicit user to be specified along with the Task Config. If else it assumes the same user as the framwework is running. This is problematic when your framework and mesos-slaves are running in different machines, which have different uid. To solve this, the mesos-slave should be started with a --switch-user option enabled.

All the above problems are not easily found on a google search. Hope this may help a soul not loose sleepless nights figuring it out. Also mesos framework development is fun. If you are running a instance of GoCD, checkout the progress and if possible support/contribute the development here.

Published at : Wednesday Feb,24 2016 at 22:40 | Tags : hacking, mesos, scala, | View Comments
Git never ceases to amaze me!

At work, we use Subversion for version control. SVN is really good - but it disturbs my workflow which is 'Commit Early, Commit Often'. People create builds three or four times a day for testing and obviously they don't want my Work In Progress commits in that. They expect me to commit only after a feature is completely done. That was handicapping my pace of development.

So, the solution - Enter git-svn :D

git-svn allows me to checkout a subversion repo as a git repository. Since my working copy is a git repo now, I get all the fantastic features of git. At the start, I was using git in a very minimal way - make commits locally, created local branches and seamlessly switched between them. Even that helped greatly to my development pace.

As days pass by, I keep on discovering lot of gems in git - which make me like,

"Holy F, How did I exist without this these many days??" :P

Discovering git squash is one such incident. git bisect was another one. Today is one such day.

At work we use Target Process for project management. Issues, Requests, Feedbacks, Usecases everything is created in TargetProcess and we use the related Id in our commits. We don't commit anything to SVN without mentioning TargetProcess artifact Id.

There is a new feature which came as a request through email - and I have started development on it - made a few commits to my git repo. That feature was not yet added to the TP, so I can't push it to the SVN repo. While this was being done, another important bug fix came and I had to fix it immediatly and push it to the SVN repo. Now the unfinished feature's commits are there and on top of it there is this urgent bug fix commit. I need to push only the bug fix commit to the SVN repo without pushing any of the new feature's commit. I could've done the development of the new feature in a local branch - keeping the master clean. But that is not a option now. Damage is done. What I did is this:

  • I converted both the bugfix and new feature changes into git-am patches.
  • Deleted both the commits and made the master as clean slate.
  • Applied only the bug fix commit.
  • Moved the other feature commits to a separate branch.
  • Pushed to bug fix to the SVN.
  • Be happy!

I know this is trivial. May be some other better way could be there. But, the power and flexibility of git just made me fall in love with it all over again :D git never ceases to amaze me!

Published at : Thursday Mar,28 2013 at 00:20 | Tags : hacking, git, Subversion, svn, | View Comments
This exists here for my easy access. May or may not be useful for others.

I paste this gist here, so that I have a easily accessible permalink to refer these. The awesomeness of gist helps me to keep it updated, without ever needing to touch this post again :D

Published at : Thursday May,31 2012 at 01:11 | Tags : ruby, rails, hacking, | View Comments

Today at work I was writing some rspec tests for a Model and found out the test cases failing for some mysterious reason in ActiveRecord. I tried plunging into rails source code to find out the exact point of failure., but then I burnt a whole afternoon with no good result. Thought of writing it here, so that I can try again when I have time later.

The scenario is simple.,

I have a Model named AAA which has a self referential relationship as below,

Imagine this model has only two attributes id and parent_id. Now begins the weirdness. Look at the gist below.

  • I create two objects a and b from model AAA.

  • I am assinging b's parent as a.

  • Then I delete a.

  • Now b points to a parent which is non existent. I then try to assign nil to the 'parent_id' attribute of the b.

  • And then save it!.

  • Rails doesn't complain anything. But when you try to print b again. SEE the parent_id attribute is still set to a's id.

  • No matter what you assign, this doesn't change. No errors nothing.

  • The only way to change is instead of assingning anything to parent_id, assing nil using the association directly. Like., 'b.parent = nil'

  • Now that thing works and I DONT KNOW WHY!!!

Published at : Wednesday May,30 2012 at 18:41 | Tags : ruby, rails, hacking, | View Comments
Trivial but not documented elsewhere.

I was hacking on the way2droid android app - and wanted to add the option to display sim contacts and allow the user to choose from it. This thing seems to be very trivial, but there is no straight forward documentation for it all over the web. I have to google a lot of links and settled on this below way. This piece of code uses a ListActivity to display the list of all sim contacts.

First we should have a Contact class, which can store the name and number of the sim contacts.

The sim contacts can be queried using the uri - "content://icc/adn/". I have created a ContactActivity class which extends the ListActivity and retreived the simcontacts into a ArrayList of Contact type. We can provide the data to the ListActivity through adapters. I have used a ArrayAdapter in this example. The ListActivity has some default layout in which it will display the contact items. If you want you can even supply your own layout. I am not going to explain that part. Every Activity should have a onCreate method, where you will initialise stuff for that. Below is our ContactActivity class.

In the onCreate method, I have first set the custom layout I created for this activity. Then I retreive all the sim contacts through retreiveSimcontacts. Then we assign a TextWatcher for that search field, so that we can filter the results through the contact list as and when the user types in. Assume I have added the searchbox in the layout and it's id is search_box. We have to add a TextWatcher class to this which will monitor the text being typed in the box and call the appropriate methods. The TextWatcher should implement these three methods

  • afterTextChanged
  • beforeTextChanged
  • onTextChanged

Here we are using only the onTextChanged method.

Next thing worth explaining is the onListItemClick method. This method is where we extract the clicked contact details and send it back to the caller. I have set the Result of ContactActivity there and end the activity so that it can return to from where it is called (i.e) the parent activity.

The retreiveSimContacts method is pretty much self explanatory. I am creating a ArrayList and appending all the sim contracts as we retreive them from the above mentioned URI.

Note : This is just the Activity used to retreive and select the contacts. You should call this activity from some where, so that it \ will return a 'name:number' string to you.

To have a look at the complete code - original repo and git mirror.

Published at : Saturday Mar,10 2012 at 18:33 | Tags : hacking, android, | View Comments
No complaining, No cribbing. Just sharing :P

Being from a python background, I just wanted to share this comment about ruby in HN,

" but fundamentally, it's more like the bastard child of a torrid, three-way affair between Lisp, Python and Perl-- the genetic heritage was passed on, but poor baby Ruby got little of the careful childhood upbringing.

The result? Ruby has the syntactical consistency and beauty of Perl, and the blazing speed and low overhead of LispThere are roughly six dozen different ways of doing everything (including really simple things, like delimiting blocks), and bizarre inconsistencies in basic use cases (e.g. most String functions return copies, whereas nearly everything else works by reference), and plenty of long-term code maintainabilit nightmares lurking in the corners. "

No, I don't endorse all the above points. I'm still in early stage of learning ruby and I'm not complaining about the language. But, just wanted to share :P

Published at : Thursday Jul,21 2011 at 17:02 | Tags : ruby, hacking, | View Comments
zc.buildout is EPIC!

Last week we had a nice session of zc.buildout and other ZTK basics by Joe Steeve at our alma mater. To put that listening to good use, I decided to hack my dumBBlogger to use zc.buildout. Another reason is as I don't have a laptop, whenever I go and start using a new machine and want to do a blog, I have to take all the pain to setup my blogging tool manually. I wanted to automate this stuff. zc.buildout proved to be a nice solution for that. Now to the setup story,

My blog tool is basically has a core file which does all the work of finding the posts, rendering it, writing out as html etc. Then there is a configure script which writes out a Makefile( I use the makefile for dependency based building. That is if a blog post is changed only that post, the related tag pages, the archives page and the index page if neccessary are only reloaded. Other posts are left untouched.)

The Makefile will be of form

./build [command] 
   target(output html file) : dependent posts

That './build' is again a simple python script which calls the appropriate core function based on the [command]

Everytime I write a new post, I do

./configure && make

If I modify any post I'll do,


So the dumBBlogger is a set of some scripts which are called one after other. Now the most beautiful thing about zc.buildout is it creates a custom interpreter for us with all the dependency (modules/eggs/scripts) available for import.

To be brief, I created a buildout config to install all the dependencies, create a custom dumbblogger interpreter and used that interepreter path as a 'hashbang' in all my scripts. Voila! It's done :D

There is a wonderful tutorial about using zc.buildout for django by Jacob Kaplan Moss. So I created the initial directory structure and wrote a accordingly. I listed all the dependencies in the That can be found here. Then I wrote the buildout.cfg file as below,

parts = dumbblogger
develop = .
eggs = dumbblogger

recipe = zc.recipe.egg
eggs = ${buildout:eggs}
interpreter = dumbblogger

Now moved the source code into the src/ folder as it was a followed practice while using buildout. Then I ran this,

python && ./bin/buildout

That downloaded and installed all the dependencies as eggs and created a custom interpreter named dumbblogger. Then the last step I had to do is to change the hasbang in my scripts to use this custom interpreter, rather than the global python interpreter. See below,


#-- Mode:Python --

I made the same change in configure script and that's done :) Now wherever I go, I can just checkout my code from bitbucket, cd into it, do 'python && ./bin/buildout' and be done :D

The directory structure after everything is setup looks like below,

|-- bin
|   |-- buildout
|   -- dumbblogger
|-- buildout.cfg
|-- develop-eggs
|-- dumbblogger.egg-link
|-- eggs
|   |-- Mako-0.4.1-py2.6.egg
|   |-- Markdown-2.0.3-py2.6.egg
|   |-- MarkupSafe-0.12-py2.6-linux-x86_64.egg
|   |-- PyRSS2Gen-1.0.0-py2.6.egg
|   |-- python_dateutil-1.5-py2.6.egg
|   |-- setuptools-0.6c12dev_r88846-py2.6.egg
|   |-- zc.buildout-1.5.2-py2.6.egg
|   -- zc.recipe.egg-1.3.2-py2.6.egg
|-- parts
|-- buildout
|-- src
|   |-- dumbblogger
|   `-- dumbblogger.egg-info

You can see all the dependencies installed in the eggs folder, the custom interpreter in the bin folder and application code in the src folder.

I can still think of lot of automation here and will do it as and when the time permits.

Happy hacking :)

Published at : Tuesday Jul,12 2011 at 12:40 | Tags : hacking, python, dumbblogger, | View Comments
Warning! Ugly code ahead

For a django project - I needed a source code highlighter to display code segments in template. Considered markdown with codehilite extension since I have already came across it in HiSlain. It looked simple as described here. But it expects the code to be markdown formatted beforehand. I need to get source code from users and display it. So expecting that to be in markdown (or) formatting it is going be hectic. So decided to settle with the famous pygments library. After looking through some examples, cooked up this following code. This is implemented as a custom template filter for django. So any code can be pygmentified in templates by adding a '| render' filter. Below is that ugly code,

     from pygments import highlight
     from pygments.formatters import HtmlFormatter
     from pygments.lexers import get_lexer_by_name, guess_lexer
     def render(content):
          """Render this content for display."""
          formatter = HtmlFormatter(cssclass=u'source')
        # Guess a lexer by the contents of the block.
            lexer = guess_lexer(content)
          except ValueError, e:
            # Just make it plain text.    
lexer = get_lexer_by_name(u'text', stripnl=True, encoding=u'UTF-8') result = highlight(content, lexer, formatter) return unicode(result)

The above is mostly self explanatory. The decorator above the method corresponds to registering that filter in django.

EDIT: I have added the above snippet and other useful snippets I cooked up here

happy hacking!

Published at : Tuesday May,10 2011 at 20:44 | Tags : hacking, django, | View Comments
Yes I know that name sucks :(

Here comes my first twitter app which I was thinking/devoloping for a long time. To keep is short,

 This app can scrap all your tweets from twitter and render them in a 
 nicely formatted pdf and give it you.

That's about it. I have originally developed it to run on django+app engine. For some reason, it looked very complicated to me. Hence switched to a pure django thing. The public beta is available here -

By no means it's complete. It still can't get all tweets without a fuss. The pdf formatting is ugly. No support for unicode tweets. But still you can use that. Try this and pass on your comments.

Thanks :)

Published at : Tuesday Apr,19 2011 at 22:04 | Tags : hacking, twitter, | View Comments
Inspired by gautham

After seeing my friend's site here, I was very impressed and wish my site would also have such neat, cool and clean design. Actually I hate designing and css stuff. But yet I decided to get my hands over it and try as far as I could. So my website design is in the process of updation.

A tiny part - designing a new header is finished. Have a look :)

Published at : Saturday Apr,24 2010 at 16:46 | Tags : website, hacking, | View Comments
The only thing missing is a beer :)

Today is a perfect day with lot of nice things happened. Ouch! it's not today., but yesterday!! It's already 1.47 midnight.

First I helped my friend to make the software club idea to take off in my college. Then I had a interview - the first time in my life :D It was an phone interview for a internship oppurtunity with a startup. And w0000t, it went perfectly well :)

Then I had this one long conversation with a senior regarding the fate and blockade of TCENet development. Revealed a lot of frigging secrets :P

After all this happened, an call with bala anna., he explained/revealed many awesome things - which added a extra dose of happiness. Finally it's gonna end with some code - went out as patches to TCENet and gone into Remote Admin as local commits.

And know what, me going off to chennai for this weekend :D woohoo!

Published at : Friday Mar,26 2010 at 01:58 | Tags : tcenet, hacking, interview, | View Comments
I have started hacking with HiSlain application. ...

I have started hacking with HiSlain application. I have successfully ported the plaintxtblog theme - my blog currently using it - to the HiSlain blog engine. It works cool in my local setup. You can find the code here - <> :P

Now, am looking into the exported XML file from the wordpress installation - that has all my blog entries. Thinking of writing a script to parse that and write the output as needed by HiSlain to make my transition smoother. This can also pave way for the wordpress -> HiSlain converter tool :D

But not now. I'm done for today. Gonna crash to sleep at once. Grrrrr.

Published at : Thursday Dec,10 2009 at 21:08 | Tags : hislain, hacking, | View Comments
And this is awesome too!

I have sucessfully migrated to the new HiSlain blog from my old WP blog. As i said i have been contributing to this application. I ported the plaintxt theme which i have been using.

Now, I have wrote a script which effectively migrates all the WP posts, pages, tags to HiSlain. I have done my migration with my own-script and am happy :D

During this migration there were problems thrown by unicode strings created by stupid-WP RSS file. I have to fix my script to encode and decode those strings while storing and retreiving from the file.

Current implementation has no built-in FTP. So am manually transferring files from my PC to web server. Will think of intergrating FTP support - yuvi also promised to do that. But i can't wait for that - I have migrated. And you re now in my HiSlain blog.

Now yuvi should also be happy.

Published at : Thursday Oct,22 2009 at 01:28 | Tags : hislain, hacking, happymoments, | View Comments
Host change and design revamp

My website after a host change, dropping of personal blog, complete overhaul of UI etc has went live. :P

Blueprint CSS saved my day. I hate tinkering with css stuff. No mood to try any CMS too. So sat back found that blueprint, and finally brought a somewhat-nice-site up. I am happy. Real happy :)

Published at : Monday Oct,05 2009 at 20:24 | Tags : website, hacking, | View Comments
There comes bala anna..

It's my long wish to contribute code to TCENet , an F/OSS based ERP portal of my college.

I dint have a proper guidance, so i lacked in my progress. And there comes bala anna, he gave lot of suggestions, made me involved in this,encouraged, cleared my doubts without a single fuss! So, things started getting shape, and one fine evening after his approval, i sent the patch to project-mentors. After a round of review and corrections my patch was accepted. I'm damn happy :D All thanks to bala anna.

Later within a week my another patch was also accepted. Of-course both are trivial, stupid patches, But still it encourages me :P

Published at : Wednesday Sep,30 2009 at 21:21 | Tags : hacking, tcenet, | View Comments
Browse Archives
I am..
Azhagu Selvan SP
Atheist, FOSS enthusiast,
Pythonist, Student
Creative Commons License