Script to update godaddy dns A record

Recently, I have setup my raspberry pi as a download machine. The aim was to be able to access the machine from anywhere and be able to add files to download. Now, since the machine is connected via airtel broadband, it is bound to go down and come up as and when the electricity in my location does.

Everytime, when the machine came up, i would be having a different public ip address to access the machine as my ADSL modem+router would also go down.

At first, i setup a cron to be run at boot time which would give me the public ip address.

The corresponding cron entry is
@reboot /home/pi/progs/godaddy.py

I also created a dns entry to access the machine. And then wrote a script to update the dns record in godaddy whenever the machine reboots. Here is the script.

Now whenever my pi reboots, it checks and updates the corresponding DNS record at godaddy. And i can ssh my machine from anywhere.

Deployments

Deployments are a critical phase of any project. In the “stone age”, developers used to simply scp the files required into production. And there used to be issues when you are dealing with multiple http servers. Keeping all the servers in sync was always the issue.

Then capistrano came into picture. It made deployment of ruby/rails apps very easy. Me and a few other people went ahead and modified it to deploy code into production for php apps as well. But it was a tricky job since capistrano was originally designed to work for ruby/rails apps. Here is a sample capistrano code that sucks out code from svn and pushes it to multiple web servers – and then runs some specific post deployment tasks on them

deploy.rb

set :application, “app”
set :repository,  “http:///tags/TAG102”
set :imgpath, “/var/images”

# svn settings
set :deploy_via, :copy
set :scm, :subversion
set :scm_username, “svn_username”
set :scm_password, “svn_password”
set :scm_checkout, “export”
set :copy_cache, true
set :copy_exclude, [“.svn”, “**/.svn”]

# ssh settings
set :user, “server_username”
set :use_sudo, true
default_run_options[:pty] = true

#deployment settings
set :current_dir, “html”
set :deploy_to, “”
set :site_root, “/var/www/#{current_dir}”
set :keep_releases, 3

#web servers
role :web, “192.168.1.1”,”192.168.1.2″,”192.168.1.3″

#the actual script
namespace :deploy do
    desc <<-DESC
  deploy the app
    DESC
    task :update do
      transaction do
        update_code
            symlink
        end
      end

    task :finalize_update do
      transaction do
        sudo “chown -R apache.apache #{release_path}”
        sudo “ln -nfs #{imgpath}/images #{release_path}/images”     
      end
    end

    task :symlink do
      transaction do
            puts “Symlinking #{current_path} to #{site_root}.”
            sudo “ln -nfs #{release_path} #{site_root}”
      end
    end

    task :migrate do
      #do nothing
    end

    task :restart do
      #do nothing
    end   
end

This sucks out the code from the svn repository. creates a tar on local. Scps it to production web servers. Untars it to the specified location. Runs all the tasks specified in finalize_update and finally changes the symlink of “html” directory to the new deployed path. The good point about capistrano is that you are almost blind as to what happens in the backend. The bad point is that since you are blind, you do not know how to do what you want to do. It would need a bit of digging and a bit of tweaking to get your requirements fulfilled by this script.

Now lets check fabric.

Installation is quite easy using pip.

sudo pip install fabric

In case you like the old fashioned way, you can go ahead and download the source code and do a

sudo python setup.py install

To create a fabric script, you need to create a simple fab file with whatever you require. For example, if you need to run a simple command like ‘uname -a’ on all your servers, just create a simple script fabfile.py with the following code

from fabric.api import run

def host_type():
        run(‘uname -a’)

And run the script using the following command

$ fab -f fabfile.py -H localhost,127.0.0.1 host_type

[localhost] Executing task ‘host_type’
[localhost] run: uname -a
[localhost] Login password:
[localhost] out: Linux gamegeek 3.2.0-24-generic #37-Ubuntu SMP Wed Apr 25 08:43:22 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

[127.0.0.1] Executing task ‘host_type’
[127.0.0.1] run: uname -a
[127.0.0.1] out: Linux gamegeek 3.2.0-24-generic #37-Ubuntu SMP Wed Apr 25 08:43:22 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Done.
Disconnecting from localhost… done.
Disconnecting from 127.0.0.1… done.

A simple fabric script which can do whatever the earlier capistrano script was doing is here.

fabfile.py

from __future__ import with_statement
from fabric.api import *
from fabric.operations import local,put

def production():
  env.user = ‘server_username’
  env.hosts = [‘192.168.1.1′,’192.168.1.2′,’192.168.1.3’]
  env.deploy_to = ”
  env.site_root = ‘/var/www/html’
  env.tag_name = ‘tag101’
  env.repository = {  ‘url’:’http:///tags/TAG101′,
            ‘username’: ‘svn_username’,
            ‘password’: ‘svn_password’,
            ‘command’: ‘svn export –force’,
          }
  env.image_path = ‘/var/images’

def deploy():
  checkout()
  pack()
  unpack()
  symlinks()
  makelive()

def checkout():
  local(‘%s –username %s –password %s –no-auth-cache %s /tmp/%s’ %
    (env.repository[‘command’], env.repository[‘username’], env.repository[‘password’], env.repository[‘url’], env.tag_name));

def pack():
  local(‘tar -czf /tmp/%s.tar.gz /tmp/%s’ % (env.tag_name, env.tag_name))

def unpack():
  put(‘/tmp/%s.tar.gz’ % (env.tag_name), ‘/tmp/’)
  with cd(‘%s’ % (env.deploy_to)):
    run(‘tar -xzf /tmp/%s.tar.gz’ % (env.tag_name))

def symlinks():
  run(‘ln -nfs %s/images %s/%s/images’ % (env.image_path, env.deploy_to, env.tag_name))

def makelive():
  run(‘ln -nfs %s/%s %s’ % (env.deploy_to, env.tag_name, env.site_root))

The good point is that i have more control on what i want to do using fabric as compared to capistrano. And it took me a lot less time to cook the fabric recipe as compared to capistrano.

To run this script simply do

fab production deploy

This will execute the tasks production and deploy in that order. You can have separate settings for staging and local in the same script. You can even go ahead and create your own deployment infrastructure and process to do whatever you want without running into any restrictions.

cfengine – a beginners guide A tool to automate infra…

MySQL versus PostgreSQL – part II

My earlier post mysql versus postgresql brought me lots of negative comments – that i did not compare the transactional database of pgsql with the transactional engine (innodb) of mysql. The main reason why i did not do that was because i had found InnoDB to be very slow as compared to MyISAM.

But after all those comments i ran the benchmarks again using the same scripts and the same technology on the same machine (my laptop) and here are the results. I created a new table in both Mysql (using InnoDB engine) and pgsql. And i disabled the binary logging in mysql to speed up insert/update/delete queries. Please refer to the earlier post for the setup information.

Following notification would be used :

<operation(select/insert/update/delete)> : <no_of_threads> X <operations_per_thread>

  • Firstly i ran single thread with inserts both before and after disabling binary logging in mysql
    Mysql Insert : 1 X 100000
    Time : 65.22 Sec (binary logging enabled)
    Time : 32.62 Sec (binary logging disabled)
    So disabling binary logging in mysql would make your insert/update/delete queries take half the time.
    Pgsql Insert : 1 X 100000
    Time : 53.07 Sec
    Inserts in mysql are very fast.
  • Selects : 2 X 100000
    Mysql time : 30.1 Sec
    Pgsql time : 29.92 Sec
    Both are same
  • Updates : 2 X 50000
    Mysql time : 29.38 Sec
    Pgsql time : 36.98 Sec
    Mysql updates are faster
  • Ran 4 threads with different no_of_operations/thread
    Run 1 [Select : 1 X 100000, Insert : 1 X 50000, Update : 1 X 50000, Delete : 1 X 20000]
    Mysql time : 40.86 Sec
    Pgsql time : 45.03 Sec
    Run 2 [Select : 1 X 100000, Insert : 1 X 100000, Update : 1 X 50000, Delete : 1 X 10000]
    Mysql time : 49.91 Sec
    Pgsql time : 63.38 Sec
    Run 3 [Select : 1 X 100000, Insert : 1 X 20000, Update : 1 X 20000, Delete : 1 X 1000]
    Mysql time : 29.83 Sec
    Pgsql time : 29.3 Sec
    It could be seen that increasing the amount of insert/update/delete queries affects the performance of pgsql. Pgsql would perform better if number of selects are very high. Whereas mysql-innodb performs better in all cases
  • Had 4 runs with different no of threads.
    Run 1: 12 threads [Select : 2X30000 + 3X20000, Insert : 1X20000 + 2X10000, Update : 2X10000, Delete : 2X1000]
    Mysql time : 31.16 Sec
    Pgsql time : 30.46 Sec
    Run 2: 12 threads [Select : 2X50000 + 2X40000 + 1X30000, Insert : 1X20000 + 2X15000, Update : 2X15000, Delete : 2X2000]
    Mysql time : 52.25 Sec
    Pgsql time : 53.03 Sec
    Run 3: 20 Threads [Select : 4X50000 + 4X40000 + 2X30000, Insert : 2X20000 + 3X15000, Update : 2X20000 + 1X15000, Delete : 2X5000]
    Mysql time : 169.81 Sec
    Pgsql time : 136.04 Sec
    Run 4: 30 Threads [Select : 2X50000 + 3X40000 + 3X30000 + 3X20000 + 4X10000, Insert : 1X30000 + 2X20000 + 3X10000, Update : 3X20000 + 3X10000, Delete : 1X10000 + 2X5000]
    Mysql time : 200.25 Sec
    Pgsql time : 156.9 Sec
    So, it can be said that for a small system with less concurrency, mysql would perform better. But as concurrency increases, pgsql would perform better. I also saw that while running the pgsql benchmark, the system load was twice than while running mysql benchmark.

Enabling mysql binary logging for replication would ofcourse add an over head. Similarly enabling trigger based replication in pgsql would be another overhead. The fact that replication in mysql is very closely linked with the database server helps in making a high availability system easier. Whereas creating slaves using replication in pgsql is not that easy. All available products for replication in pgsql are external – 3rd party softwares. Still, for a high concurrency system pgsql would be a better choice.

MySQL versus PostgreSQL

I created and ran some simple tests on mysql and postgresql to figure out which one is faster. It is already known that postgresql is more stable and reliable than mysql. pgsql has a rich set of features. It is a complete RDBMS and also supports fulltext search.

All benchmarks were done on my laptop – Intel core 2 duo (2.0 GHz) with 4MB L2 cache & 2 GB ram. I have 64 Bit ubuntu system loaded with MySQL 5.1.24-rc (64 bit binary) and PostgreSQL 8.3.1 (compiled from source).

I used python as a scripting language for writing down my benchmark scripts. I used psycopg2 as a connector from python to postgres and mysql-python as a connector from python to mysql.

The benchmarking was done in phases. Firstly simple Insert, update and select queries were run to check the raw speed of these queries. Then threads were created to run simultaneous insert, update, select and delete queries. I checked the benchmark times for different number of concurrent threads.

I created a simple table on both mysql and pgsql. I used the MyISAM database engine to create table in mysql. :

ABC(id int not null auto_increment primary key, value varchar(250));

Queries that were run are:

Insert(I) : Insert ignore into ABC (id, value) …(For pgsql, a rule has to be created to ignore duplicate inserts)
Update(U) : Update ABC set value=<something> where id=<random_id>
Select(S) : Select * from ABC where id=<random_id>
Delete(D) : Delete from ABC where id=<random_id>

  • Insert – 100000 rows in 1 thread
    Time taken for Mysql : 20.8 seconds
    Time taken for Pgsql : 58.1 seconds
    So, raw insert speed of mysql is much better as compared to pgsql
  • 100000 selects in 1 thread
    Time taken for Mysql : 21.76 seconds
    Time taken for Pgsql : 20.15 seconds
    Raw selects are better in pgsql as compared to mysql
  • Selects – 2 threads of 100000 selects
    Time taken for Mysql : 40.46 seconds
    Time taken for Pgsql : 27.38 seconds
    So, if i increase the concurrency of selects, pgsql perfors much than mysql
  • Update – 2 threads of 50000
    Time taken for Mysql : 23.97 seconds
    Time taken for Pgsql : 34.03 seconds
    Mysql looks better in handling updates here.
  • 4 Threads
    Run 1 : [100000 Selects, 50000 Inserts, 50000 Updates, 20000 Deletes]
    Time taken for Mysql : 45.25 seconds
    Time taken for Pgsql : 54.58 seconds
    Run 2 : [100000 Selects, 100000 Inserts, 50000 Updates, 10000 Deletes]
    Time taken for Mysql : 59.05 seconds
    Time taken for Pgsql : 69.38 seconds
    Run 3 : [100000 Selects, 20000 Inserts, 20000 Updates, 1000 Deletes]
    Time taken for Mysql : 35.54 seconds
    Time taken for Pgsql : 31.23 seconds
    These runs show that Mysql is good when you have very large no of inserts/updates/deletes as compared to selects. But pgsql’s performance surpasses that of mysql when the number of selects are much higher.
  • Finally, lets approach the real life scenario where generally the number of selects are much more than the number of inserts and there are multiple threads performing selects and inserts.
    I will use the following notification here – <no_of_threads> X <no_of_operations(select/insert/update/delete)_per_thread>
    So, for example 3 X 20 Selects = 3 threads of 20 Selects in each thread

    Run 1 : [2 X 30000 selects, 3 X 20000 selects, 1 X 20000 inserts, 2 X 10000 inserts, 2 X 100000 updates, 2 X 1000 deletes] Total – 12 threads
    Time taken for Mysql : 42.28 seconds
    Time taken for Pgsql : 44.28 seconds
    Both Mysql and Pgsql are almost at par.

    Run 2 : [2 X 50000 selects, 2 X 40000 selects, 1 X 30000 selects, 1 X 20000 inserts, 2 X 15000 inserts, 2 X 15000 updates, 2 X 2000 deletes] Total – 12 threads but number of selects are quite high
    Time taken for Mysql : 61.02 seconds
    Time taken for Pgsql : 48.60 seconds
    So, as we increase the number of operations (specially selects) mysql’s performance degrades, whereas pgsql’s performance remains almost the same

    Run 3 : [4 X 50000 selects, 4 X 40000 selects, 2 X 30000 selects, 2 X 20000 inserts, 3 X 15000 inserts, 3 X 15000 updates, 2 X 3000 deletes] Total – 20 threads (10 threads for select, 5 for insert, 3 for update and 2 for delete) Which is the normal trend in database servers.
    Time taken for Mysql : 169.31 seconds
    Time taken for Pgsql : 128.7 seconds
    Bingo, so as concurrency increases pgsql becomes faster than mysql.

My earlier benchmarks with pgsql 7.x was not as good as this one. With postgresql 8.3.1, the speed of serving concurrent requests has increased a lot. So, in a high concurrency environment, i would generally recommend to go ahead with using postgresql rather than mysql.

Please check the comments section. We have some really interesting comments there…

python programming – threading

There are a few options for threading in python. I wont be exploring all the options here. I would simply try to make this blog entry simple, sweet and short.

There are two modules in python which provide threading capabilities – “thread” module and “threading” module. The thread module is very basic. So lets focus on the threading module

To use the threading module, all you have to do is

import threading

class mythread(threading.Thread):
  def run(self):
    <your code>

mythread().start()
mythread().start()

This will create 2 threads running <your code>

That was quick right. Now lets see some basic stuff like Thread Naming, Thread isAlive and join which is used in most threading environments…

Naming a thread:

import threading

class mythread(threading.Thread):
  def run(self):
    print ‘my name is ‘, self.getName()

foo = mythread()
foo.setName(‘Foo’)
foo.start()

bar = mythread()
bar.setName(‘Bar’)
bar.start()

mythread().start()

Run the program:

$ python threadname.py

And see the output

my name is Foo
my name is Bar
my name is Thread-3

Checking if the thread is still alive:

import threading
import time

class mythread(threading.Thread):
  def run(self):
    print ‘my name is ‘, self.getName()

class aliveth(threading.Thread):
  def run(self):
    time.sleep(10)
    print ‘my name is ‘, self.getName()

myt = mythread()
myt.setName(‘mythread’)
myt.start()
if myt.isAlive():
  print ‘myt is alive’
else:
  print ‘myt is dead’

alt = aliveth()
alt.setName(‘aliveth’)
alt.start()
if alt.isAlive():
  print ‘alt is alive’
else:
  print ‘alt is dead’

And check the output

my name is mythread
myt is dead
alt is alive
my name is aliveth

Joining threads:

You can use the thread.join() method to make a thread wait for another thread

import threading
import time

class ThreadOne ( threading.Thread ):
  def run ( self ):
    print ‘Thread’, self.getName(), ‘started.’
    print self.getName(), ‘: sleeping ‘
    time.sleep ( 5 )
    print ‘Thread’, self.getName(), ‘ended.’

class ThreadTwo ( threading.Thread ):
  def run ( self ):
    print ‘Thread’, self.getName(), ‘started.’
    print self.getName(), ‘: waiting for ‘, thingOne.getName()
    thingOne.join()
    print ‘Thread’, self.getName(), ‘ended.’

class ThreadThree (threading.Thread):
  def run(self):
    print ‘Thread’, self.getName(), ‘started’
    print self.getName(),’: Not waiting for any other thread’
    print ‘Thread’, self.getName(), ‘ended.’

thingOne = ThreadOne()
thingOne.start()
thingTwo = ThreadTwo()
thingTwo.start()
thingThree = ThreadThree()
thingThree.start()

And check the output

Thread Thread-1 started.
Thread-1 : sleeping
Thread Thread-2 started.
Thread-2 : waiting for Thread-1
Thread Thread-3 started
Thread-3 : Not waiting for any other thread
Thread Thread-3 ended.
Thread Thread-1 ended.
Thread Thread-2 ended.

This covers most of the stuff for programming threads in python. We will look into thread synchronization issues some other time.

python postgresql connectivity

It is sweet and simple. Got it running in 2 minutes…

  • Download psycopg 2.0 (yup that’s the name) from http://www.initd.org
  • Extract the library
    $ tar -xvzf psycopg2-2.0.7.tar.gz
  • Build the extension
    $ cd psycopg2-2.0.7/
    $ python setup.py build
  • And install the extension
    $ sudo python setup.py install
  • Create a simple script
    $ vim pypgsql.py

    import psycopg2

    try:
        conn = psycopg2.connect(“dbname=’test’ user=’jayant’ host=’localhost’ password=’secret'”)
    except:
        print “unable to connect to db”

    cur = conn.cursor()
    cur.execute(“””select datname from pg_database”””)
    rows = cur.fetchall()

    print “nShow me the databasesn”
    for row in rows:
        print ” “, row[0]

    Save the pypgsql.py file.

  • And finally run the script
    $ python pypgsql.py

    Output:

    Show me the databases

      template1
      template0
      postgres
      test

python on web – getting started : building your first app

I would be focusing here on the following tasks.

  • install pylons.
  • create a helloworld app using pylons.
  • install genshi.
  • change template engine.
  • deploy on apache using mod_wsgi.

Install Pylons

Run the following:

$ easy_install Pylons==0.9.6.1
OR
$ easy_install -f http://pylonshq.com/download/ Pylons==0.9.6.1

More details at http://wiki.pylonshq.com/display/pylonsdocs/Installing+Pylons

Create helloworld app

$ paster create -t pylons helloworld

A directory helloworld with the following structure is created

jayant@jayantbox:~/myprogs/python/helloworld$ ls -lh
total 44K
drwxr-xr-x 4 jayant jayant 4.0K 2008-04-25 17:47 data
-rwxr-xr-x 1 jayant jayant 1.5K 2008-05-15 10:41 development.ini
drwxr-xr-x 2 jayant jayant 4.0K 2008-04-25 17:47 docs
drwxr-xr-x 2 jayant jayant 4.0K 2008-04-25 17:47 ez_setup
drwxr-xr-x 9 jayant jayant 4.0K 2008-05-14 19:52 helloworld
drwxr-xr-x 2 jayant jayant 4.0K 2008-04-25 17:47 helloworld.egg-info
-rwxr-xr-x 1 jayant jayant 79 2008-04-25 17:47 MANIFEST.in
-rwxr-xr-x 1 jayant jayant 463 2008-04-25 17:47 README.txt
-rwxr-xr-x 1 jayant jayant 1.2K 2008-04-25 17:47 setup.cfg
-rwxr-xr-x 1 jayant jayant 865 2008-04-25 17:47 setup.py
-rwxr-xr-x 1 jayant jayant 507 2008-04-25 17:47 test.ini

To run the app, from within the helloworld directory do

jayant@jayantbox:~/myprogs/python/helloworld$ paster serve –reload development.ini
Starting subprocess with file monitor
Starting server in PID 15555.
serving on 0.0.0.0:5000 view at http://127.0.0.1:5000

Create a test.html in the <path_to_helloworld>helloworld/helloworld/public directory

<html>
<body>
Hello World!!
</body>
</html>

And point your browser to http://127.0.0.1:5000/test.html to see the “Hello World!!” page.

Now lets create a controller and try printing info through a template.

jayant@jayantbox:~/myprogs/python/helloworld$ paster controller hello
Creating /home/jayant/myprogs/python/helloworld/helloworld/controllers/hello.py
Creating /home/jayant/myprogs/python/helloworld/helloworld/tests/functional/test_hello.py

Edit the <path_to_helloworld>/helloworld/helloworld/controllers/hello.py file and put in some python code

import logging

from helloworld.lib.base import *

log = logging.getLogger(__name__)

class HelloController(BaseController):

    def index(self):
        # Return a rendered template
        # return render(‘/some/template.mako’)
        # or, Return a response
        return ‘Hello World’
    def serverinfo(self):
        import cgi
        import pprint
        c.pretty_environ = cgi.escape(pprint.pformat(request.environ))
        c.name = ‘Jayant Kumar’
        return render(‘serverinfo.mako’)

Now create a template serverinfo.mako in <path_to_helloworld>/helloworld/helloworld/templates

<h2>
Server info for ${request.host}
</h2>

<p>
The URL you called: ${h.url_for()}
</p>

<p>Hi there ${c.name or c.full_name or “Joe Smith”}</p>

<p>
The name you set: ${c.name}
</p>

<p>The WSGI environ:<br>
<pre>${c.pretty_environ}</pre>
</p>

Edit the <path_to_helloworld>/helloworld/helloworld/config/routing.py file and check if the following code there in the “CUSTOM ROUTES” section

map.connect(”, controller=’hello’, action=’index’)
map.connect(‘:controller/:action/:id’)
map.connect(‘*url’, controller=’template’, action=’view’)

This means that an empty URL is matched to the index action of the hello controller. Otherwise, the route mapper looks for URLs in the form controller/action/id, but if action or controller are not specified the request is routed to the view action of the templates controller (created by the Pylons template). This raises a 404 error by default

Now go to http://localhost:5000/hello/hello and http://localhost:5000/hello/serverinfo and check out the output.

Installing genshi and switching the template engine

$ easy_install Genshi

To enable genshi in your project edit the <path_to_helloworld>/helloworld/helloworld/config/environment.py file and change the template engine from mako to genshi

    config.init_app(global_conf, app_conf, package=’helloworld’,
        template_engine=’genshi’, paths=paths)

Now create a new template in <path_to_helloworld>/helloworld/helloworld/templates, say serverinfo.html and put in the following code

<html xmlns=”http://www.w3.org/1999/xhtml”
xmlns:py=”http://genshi.edgewall.org/”
lang=”en”>
<h2>
Testing…
</h2>

<p>Hi there ${c.name}</p>

<p>
The name you set: ${c.name}
</p>

<p>The WSGI environ:<br/>
<div py:content=”c.pretty_environ”>Pretty Environ</div>
</p>
</html>

And change your <path_to_helloworld>/helloworld/helloworld/controllers/hello.py to render the new template

        return render(‘serverinfo’)

And check the output at http://localhost:5000/hello/hello and http://localhost:5000/hello/serverinfo

Deploy using mod_wsgi

For those who have not yet configured mod_wsgi, you can get mod_wsgi from http://code.google.com/p/modwsgi/.

Simply untar it and do a

./configure –with-apxs=/path/to/apache/bin/apxs
make
sudo make install

Bingo and you have the mod_wsgi.so file in your apache/modules directory

Change the httpd.conf and add the following

LoadModule wsgi_module modules/mod_wsgi.so

to load the wsgi module and then deploy the hello world application

WSGIDaemonProcess hello thread=25
WSGIScriptAlias /hello “/path/to/apache/anydirectory/hello.wsgi
<Location /hello>
        WSGIProcessGroup hello
        WSGIReloadMechanism Process
</Location>

This says that the application helloworld would run as a separate process with 25 threads. And since we have also enabled process reload mechanism available with mod_wsgi-2.0, all that is needed to restart/reload the application is to touch/change the wsgi script modification time.

Wait a minute, we did not create the hello.wsgi script. Create a directory in /path/to/apache or anywhere where apache has read access and where you want to keep your application startup scripts. So what i did was.

$ mkdir /path/to/apache/wsgi

And create a hello.wsgi inside it

cd /path/to/apache/wsgi
vim hello.wsgi

Add the following code here

import os, sys
sys.path.append(‘/path/to/python/application/helloworld’)

from paste.deploy import loadapp

application = loadapp(‘config:/path/to/python/application/helloworld/development.ini’)

And we are done. Please note that the /path/to/python/application should be readable and executable by apache. Or you can do (very unsafe – not recommended on production servers)

chmod -R a+rx /path

Now simply restart apache and point your browser to

http://localhost/hello/
and
http://localhost/hello/hello/serverinfo

To see the output.

Source:
http://wiki.pylonshq.com/display/pylonsdocs/Installing+Pylons
http://wiki.pylonshq.com/display/pylonsdocs/Getting+Started
http://wiki.pylonshq.com/display/pylonsdocs/Using+Other+Template+Languages
http://code.google.com/p/modwsgi/wiki/InstallationInstructions
http://code.google.com/p/modwsgi/wiki/IntegrationWithPylons

Python on web – getting started : selecting the tools

Why am i writing this? Well, because i felt that there are so many options, it is difficult to select an option for yourself. I was confused for quite some time in terms of what framework to use, what templating engine to use, which ORM layer to use and how to deploy the app on the web.

First of all, lets start of with the framework. The major frameworks are Django, TurboGears and pylons. Django is supposed to be the best framework from among the three. But Django is very tightly integrated to its template engine and ORM layer. And any sort of customization is not only difficult but also slows down the app. But it is good thing to start off python-on-web as a beginer. And its tight integration also accounts for its speed. It is the fastest framework among all of them. Take a look at some of the benchmarks on web here
http://www.rkblog.rk.edu.pl/w/p/simple-python-frameworks-benchmark/ And http://www.alrond.com/en/2007/feb/04/in-addition-to-the-test-of-mvc-frameworks/

Django is a good framework to use if you are building a small site with default settings. But when you need to have an enterprise level site with customizations, it is better to use something like Turbogears or Pylons. Turbogears is good and has “Turbogears Widgets” but it lacks proper documentation. TurboGears uses CherryPy whereas Pylons uses paste. I felt pylons to be much easier to use and easier to configure. Pylons has a very small base though it is not as fast as Django. There are lots of comparisons available on the net – just google for it. http://nxsy.org/unscientific-and-biased-comparison-of-django-pylons-and-turbogears gives a detailed comparison of different features of all the 3 frameworks.

After selecting Pylons as my framework, i have tons of options with respect to templating engine. I can use the default mako or kid or genshi or Cheetah (complete list available at http://projects.dowski.com/projects/buffet). I took a look on the performance benchmark and ease of use of all of them (from googlebaba ofcourse). And what i could figure out was that mako is the fastest template engine available. Genshi is a successor of kid and has a very clean xml-based integration. Which means that if i am using genshi, the html designer would not be able to see the code embedded within templates. So, i would prefer to use Mako or Genshi depending on the requirements. A list of different types of available template engines can be obtained from http://wiki.python.org/moin/Templating. Benchmarks are available here http://www.kuwata-lab.com/tenjin/pytenjin-users-guide.html#benchmark. Changing template engine for pylons is described here http://wiki.pylonshq.com/display/pylonsdocs/Using+Other+Template+Languages

I did not do any DB operations, so i did not go into selecting the ORM layer. Anyways, i always prefer to write my own queries instead of letting the ORM layer decide what query to fire. SQLAlchemy seems good though if you plan to use an ORM layer. SQLObject is also there, but it is old.

For deployment few options available are mod_wsgi, mod_python and the default paster. The paster server comes integrated with pylons and ofcourse cannot be used for production level deployment. mod_wsgi looked easy to use and is also supposed to be a small bit faster than mod_python. You can look at http://wiki.pylonshq.com/display/pylonscookbook/Production+deployment+using+mod_python for mod_python or http://code.google.com/p/modwsgi/wiki/IntegrationWithPylons or mod_wsgi integration.

Next we would look at building and deploying the “hello world” application

Python Mysql Connectivity

How do you configure python for use with mysql? You require the MySQLdb module for connecting to mysql server using the python code. This module is used for firing queries to the database server and handling of result sets from python code.

First of all, check if the module is available or not :

jayant@jayantbox:~$ python
Python 2.5.2 (r252:60911, Apr 21 2008, 11:17:30)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import MySQLdb;
Traceback (most recent call last):
File “”, line 1, in
ImportError: No module named MySQLdb
>>>

If you are getting an importError, then the module is not installed. Download the source files from http://sourceforge.net/projects/mysql-python

Compile and install the module:

tar -xvzf MySQL-python-1.2.2.tar.gz
.
.
cd MySQL-python-1.2.2/
.
.
python setup.py build
.
.
sudo python setup.py install
.
.
Installed /usr/lib/python2.5/site-packages/MySQL_python-1.2.2-py2.5-linux-x86_64.egg
Processing dependencies for MySQL-python==1.2.2

I faced a problem during the build process. The setup process was unable to find the mysql_config program in the path. So i did
export PATH=/usr/local/mysql/bin:$PATH
And then i ran the build process and it was successful.

Now again try importing the MySQLdb module in the python environment

jayant@jayantbox:/usr/share/python$ python
Python 2.5.2 (r252:60911, Apr 21 2008, 11:17:30)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import MySQLdb
Traceback (most recent call last):
File “”, line 1, in
File “/usr/lib/python2.5/site-packages/PIL/__init__.py”, line 19, in

File “build/bdist.linux-x86_64/egg/_mysql.py”, line 7, in
File “build/bdist.linux-x86_64/egg/_mysql.py”, line 6, in __bootstrap__
ImportError: libmysqlclient_r.so.16: cannot open shared object file: No such file or directory
>>>

Oops… the library is not able to locate my shared library. So, i added the library path to the LD_LIBRARY_PATH environment variable

export LD_LIBRARY_PATH=/usr/local/mysql/lib:$LD_LIBRARY_PATH

And then try

jayant@jayantbox:~$ python
Python 2.5.2 (r252:60911, Apr 21 2008, 11:17:30)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import MySQLdb
>>>

Bingo, now importing the mysqldb module has been successful.
Lets try running some queries using python

jayant@jayantbox:~$ python
Python 2.5.2 (r252:60911, Apr 21 2008, 11:17:30)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import MySQLdb
>>> db=MySQLdb.connect(host=’localhost’,user=’root’,passwd=’jayant’,db=’mysql’,unix_socket=’/tmp/mysql.sock’)
>>> cursor = db.cursor()
>>> cursor.execute(“show tables”)
23L
>>> result = cursor.fetchall()
>>> for row in result:
… print row[0]

columns_priv
db
event
func
general_log
help_category
help_keyword
help_relation
help_topic
host
ndb_binlog_index
plugin
proc
procs_priv
servers
slow_log
tables_priv
time_zone
time_zone_leap_second
time_zone_name
time_zone_transition
time_zone_transition_type
user
>>>

Now you can create apps backed with mysql db operations using python

python

Yes python. No, I am not refering to the snake, I am refering to the programming language “python”. OOPS!!! Sorry for the confusion.

Python is an interpreted, interactive and object oriented programming language. Often compared to perl, java, ruby etc. As per http://en.wikipedia.org/wiki/Python_programming_language, python was created by some dude named “Guido van Rossum” in 1990. It uses automatic memory management and is a High level language. Oooh.. And python is hot… Why?? Cause it has a very clean syntax and is easy to use and packs a lot of power inside it… Thats why, i think so… So lets get some tits and bits about python…

But before i start, just check out http://www.python.org, the official website of python software foundation the creators and maintainers of python.

The thing which will strike you at the start is the way indentation is important in python. Python groups sentenses on the basis of how it is indented. So i would write a for loop as follows


>>> arr = ['a1','b2','c3','d4','e5','f6'] # defining the array
>>> for x in arr: # starting the for loop
>>> print x # print the element and its length
>>> print len(x)
>>> print 'end' # print end the for loop. Indentation says for loop is over

The best thing i like about this is that developers would automatically indent their code making it good to look at – ummm beautiful may be the right word.

python has a nice set of string operators and regular expressions. You can create and iterate over a range very nicely, write functions, create stacks and queues, do file handling and exception handling. You have complete java style – try->catch level exception handling. And ofcourse you can create classes. Serialize and store objects in file very easily and then restore the objects later. Also there are lots of extensions available which would allow you to interact with the net, create threads, do data compression and decompression. control garbage collection.

Wooo, and the main thing is that all this is possible with very easy and clean syntax . I have seen small and large – in fact very large programs being written in python. And maybe sometime down in future python would replace perl for small programs.

You have got to try it out to see the power of python…