Friday, November 12, 2010

Python and Condor

I wanted to see if I could run Python 3.x under Condor as a vanilla job. It seems to work fine. Here's what I did:

1. Built Python and tarred it up:

export PATH=/usr/bin:/usr/sbin:/bin:/sbin
export INSTALL_DIR=/scratch/mick/src/python-3.1.2-x86_64-rhel5
mkdir -p $INSTALL_DIR
tar xjf Python-3.1.2.tar.bz2
cd Python-3.1.2
./configure --prefix=$INSTALL_DIR
make install
cd $INSTALL_DIR/lib/python3.1/lib-dynload
ldd *.so | \
grep '/usr/' | \
gawk '{print $3}' | \
sort -u | \
while read n; do cp $n $INSTALL_DIR/lib; done;
cd `dirname $INSTALL_DIR`
tar czf python-3.1.2-x86_64-rhel5.tar.gz python-3.1.2-x86_64-rhel5

2. Created a wrapper to extract the tarball and run python. I called my wrapper 'python3'



/bin/tar xzf ./$VERSION.tar.gz
exec ./$VERSION/bin/python3 "$@"

3. Submit your job.

universe                = vanilla
executable              = python3
arguments               =
should_transfer_files   = yes
transfer_input_files =, python-3.1.2-x86_64-rhel5.tar.gz
when_to_transfer_output = on_exit
output                  = condor.out.$(CLUSTER).$(PROCESS)
log                     = condor.log.$(CLUSTER).$(PROCESS)
error                   = condor.err.$(CLUSTER).$(PROCESS)

1 comment:

  1. Trying this with Python2.6 I get the following error:

    Python 2.6 (r26:66714, Mar 7 2011, 18:11:51)
    [GCC 4.4.3] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    Traceback (most recent call last):
    File "/etc/pythonstart", line 7, in
    import readline
    ImportError: No module named readline