Thursday, June 30, 2011

Customizing the Python Interpreter

I was working with a friend on debugging an odd pickling issue. In order to really play with it, we started a Python interpreter and ran through the list of imports and sys.path.appends necessary to set up active database and memcache connections, as well as load the project code.

My friend mentioned that he had a python session on his desktop that he never dared to close, since the setup had been so much trouble.

Thinking that it would be great to have this more frequently, and more easily, I created a python file which gets loaded every time I start my python interpreter,

import sys
import os

def getProjectDir(d=os.getcwd()):
projects = os.path.join(os.environ['HOME'], "projects")
if os.path.dirname(d) in ['/home', '/']:
return None
elif projects == os.path.dirname(d):
if os.path.exists(os.path.join(d, "src/pylons/proj")):
return d
return None
return getProjectDir(os.path.dirname(d))

pd = getProjectDir()
if pd:
print "Detected %s project. Setting up database." %(os.path.basename(pd))
sys.path.append(os.path.join(pd, "src/pylons/proj"))
import proj
from proj.lib.helpers import setup_db
factory = setup_db()
dbcon = factory.getConnection()

Now I want this file to run every time the python interpreter is started.

export PYTHONSTARTUP="/home/jsimpson/bin/"

So, what happens is, when the python interpreter starts, it checks the PYTHONSTARTUP variable for a python file to run. That executes the getProjectDir() function, which tests if my current working directory is in a project. (Our workflow includes many branches of the main project.) If I am in a project, it will import the files from that project and setup my interpreter, ready for some interactive work.

Readline Interactions

I previously posted on how Emacs keys work for interaction on many different shells, bash, python, ruby, mysql, etc.

Recently I was trying to improve my python interaction experience when I stumbled across this:

This details some interesting key bindings in bash, as well as how to change the key bindings.

Thursday, May 5, 2011

Custom Tab Complete and Bash Functions

Everyone with even a little experience in Unix has their own special aliases. Maybe you have:

alias wcd="cd ~/workspace"

to take you to the directory where you have your projects. Maybe you have some really sophisticated aliases that change your bash configuration and path when you hit a certain directory.

One of the challenges I found when I moved off of csh to bash was csh aliases had parameters, so you could make niffty aliases that had some value in the middle:

alias fi 'find . -name \!:1 -exec vim {} \;'

(The first alias I ever used, from my friend William Hui). Using this alias, if you run:

fi mycode.c

It will search through the directory tree to find mycode.c and open it in vim. Bash doesn't seem to give you the same level of control over alias parameters. However, recently I've learned that you can create a bash function, and it will run like a command on the command line.

Since fi is a reserved word that closes an if statement in bash, we'll switch from fi to fim. Then create this in a file (

fim() {
test ! -z "$1" && find . -name $1 -exec vim {} \;

and source the file in your current environment:


then this works in bash:

fim mycode.c

Since you are now using a bash function instead of an alias, you can squeeze any amount of bash code you want in there, format it nicely, even (possibly) comment it. Aliases can get pretty crazy sometimes but a bash function should be fairly readable.

So, that's cool, flexible bash functions instead of aliases. Which brings me to my next trick. Everyone loves tab completion. Okay, that may be a fairly sweeping statement, but watching people use the Unix command line, I think I can safely say most people love tab completion.

With newer versions of bash, you can build your own custom tab completion. Lets say you have a directory where you do a lot of your work, and you often want to go there to some project.

wcd() {
cd ~/workspace

That's cool, anywhere you are
takes you there. However, now that it's a bash function, it's really straight forward to augment it a little:

wcd() {
cd ~/workspace/$1

That's cool too. Now, assuming your working directory looks like this:

[jsimpson@jsimpson-lnx1 workspace]$ ls
bookcatalog/ mashup/ mywebsite/ out.html

you can:

wcd bookcatalog

and anywhere you are will take you to ~/workspace/bookcatalog. However, if your workspace looks like mine, it has many projects, some of which haven't been worked on in months or years. Sometimes I need a little help to remember where I'm going. Wouldn't a custom tab complete be great?

wcd() {
cd ~/workspace/$1

_wcd() {
local cur opts
opts=$(cd ~/workspace ; ls -d */. | sed 's|/./||')
COMPREPLY=($(compgen -W "${opts}" -- ${cur}))
complete -F _wcd wcd

The first function we've seen before. Once you press enter on the command line, this is the function that will do the work of changing you to a new directory.

Last line first, complete is a utility that registers a function (-F for function) to be run every time you tab when wcd is the first command on the command line.

_wcd is the function that computes your list of possible completions.

local - sets up some local variables.

cur - gets the last element of the COMP_WORDS array, which is a special bash variable that holds the array of parameters to the command line you are pressing tab on.

opts - this line gets a string of space delimited words that are possible completions, something that will look like this "dir1 dir2 dir3".

COMPREPLY - this is a special bash variable that is an array of all the possible completions. So, if you are here:

wcd m

When you hit tab, the function will run and evaluate like this:

opts="bookcatalog mashup mywebsite"
COMPREPLY=(mashup mywebsite)

and on the command line you will see this:

[jsimpson@jsimpson-lnx1 workspace]$ wcd m
mashup mywebsite

Debian Administrator's Introduction to Bash Completion

Tuesday, March 22, 2011

Consolidated Java Jars

Sometimes I just want my little Java app to be simple. It's suppose to be a command line client, or some JMS utility, and I just want to move it around easily, call it easily and have it work.

But, it's got a bunch of files, and I used something for a network library, and a command line parsing library...

What I would like is a consolidated jar, one jar that has what is needed, it is configured so it can be run with java -jar myjar.jar, and it will work.

<target name="dist" depends="compile"
description="generate the distribution" >
<!-- Create the distribution directory -->
<mkdir dir="${dist}"/>

<jar jarfile="${dist}/Util.jar" basedir="${build}">
<attribute name="Main-Class" value="Main"/>
<zipfileset src="lib/commons-cli-1.2.jar" includes="**/*.class"/>

This ant target will create a jar, and include in the jar, all the class files from commons-cli-1.2.jar. Now, assuming no other dependencies, I should be able to ssh the Util.jar to any destination machine with Java and run it.

Monday, February 14, 2011

Managing Vim Plugins with Mercurial

I always felt a little as though I was risking something each time I installed a new vim plugin. I mean, the vim plugins provide me with awesome power of customization, and my vim setup helps me work, so messing it up would be bad, and each new plugin just throws a bunch of stuff into my vim directories.

I could have tried to organize it based on the plugin name, but that always seemed like too much work.

When I read Steve Yegge's post about his dot-emacs-file, that seems cool. All his configuration files controlled by svn. However, I don't have an offsite svn repo to use.

Then I read the Joel on Software post about distributed version control and Mercurial.

I had used Mercurial before a little, but Joel's post helped me realize that I was stuck thinking about Mercurial as if it was some sort of advanced svn. It's not. I always assumed I had to have some sort of Mercurial repo that served as the 'master' copy, much the way an svn server works.

However, Mercurial is different. You can make any directory into it's own repo. This is fantastic. Now I'm using Mercurial to version control basically any test project I work on, it's fantastic. Every directory is it's own repo. If I need to move it to another computer or give it away to someone, or take some of my test projects to work, no problem. Tar it up and the whole history is there.

The next obvious step was to version control my .vim dir. Now, every time I add a new plugin, it becomes it's own commit in Mercurial. I could go back in history at any time and look at the files that were part of any plugin, or revert any change I make.

To complete the process, I moved my ~/.vimrc file into ~/.vim and renamed it myvimrc. Now, my ~/.vimrc file has only one line:

source ~/.vim/myvimrc

and .vim/myvimrc is version controlled with Mercurial.

Monday, January 31, 2011

Bash, Readline and Emacs

I've known for a while that there are some good bash keyboard shortcuts.

Ctrl-A - cursor to the beginning of the line
Ctrl-E - cursor to the end of the line
Ctrl-R - reverse search for a historical command

Cool, those are great, but recently I realized that bash, by default, uses Emacs key mappings. (You can change it to vi with set -o vi, but this post is about Emacs keys). That opens up a whole new world. Whatever works in Emacs you can try in bash:

Alt-F - forward 1 word
Alt-B - backward 1 word
Alt-Backspace - delete 1 word backwards
Alt-d - delete 1 word forwards
Alt-8 Alt-B - move 8 words backwards

During a search (Ctrl-R), once you have a match, you can hit Ctrl-R to search backwards to the next match, Ctrl-S to search forward to the next match. Any navigation key will drop you out of the search and into editing the command line your search matched.

All of these are standard Emacs behaviors, and I'm sure there is more to be discovered.

Only in the last couple days have I realized that this awesome editing functionality is a product of the readline library from GNU, not a product of bash. Which means, you get all this stuff in any product that uses readline. For example, the mysql command line tool, the python interpreter, the ruby interpreter, the Octave interpreter, Postgresql interpreter. I'm sure there are many more as readline is widely used. Why I never made this connection before, I have no idea.

NOTE: Some keys get intercepted by your OS or terminal. For example, I use Gnome terminal, which, with a default configuration, has a menu bar, and Alt-1 - Alt-0 are configured as hot keys for selecting tab 1 - 10. I edited the keyboard shortcuts (Edit -> Keyboard Shortcuts) and removed the key mappings for selecting a tab with Alt-. Then I right clicked in the terminal and deactivated the Show Menubar setting. I would much rather have the hot keys than the menus that I never use.