Installing & deploying Python web application using Debian / Apache2 / WSGI

Foreword

I was frustrated recently. I was frustrated of all the half-baked tutorials in regard to Python/Apache/WSGI deployment. Most tutorials or examples miss a huge amount of information i.e. describing the context. The usual approach seems to be to write such a tutorial just for a 15-minutes-of-fame-gig after one mastered those steps to deploy once.

The pity is, most of these tutorial writers seem to have exactly no clue what they did and why they did it. They do not explain why something is done this way and not the other way and they never mention how things might differ in your setup. They describe just their own story like in a „my little diary“-story and simply claim: „works for me.“

Installing an Open Source issue tracker

I wanted to install Roundup an open source issue tracker I would like to use to keep track of bugs and improvement ideas in my projects. Roundup is a python web application working with templates and I like Python. I chose Roundup after I checked out several lists like e.g. „15 Most Popular Bug Tracking Software to Ease Your Defect Management Process“. Other softwares which were in my focus were e.g. „Trac“ or „Mantis“. I thought, well, if this thing is Python, I might be able to easily customize it later on and since I deployed already one Python app successfully I thought this one might be quite easy.

Why apt-get install is a trap

Somewhere in the Debian Wiki I found that apt-get install roundup would just do an installation for me. At least that was what I thought…

Yeah the installation went fine, but this started a python app as a standalone server on some weird port. Oh and it started that server without asking back if I actually wanted it to start at all. I wanted this thing to run smoothly under a nice subdomain like issues.mydomainname.com in my existing apache setup. The need to explicitly put the port number in the url to reach the issue tracker like issues.mydomainname.com:8484/issues/ was not exactly what I had in mind. But the Debian wiki seems to be completely fine with the usual „works for me.“-claim I guess.

I think the person who wrote that tutorial missed the point completely that you usually want to deploy a bug tracking system. Let me tell you something Debian wiki writer no one wants a bug tracker which runs on localhost only for personal use and only reachable through it’s own explicit port. But hey, why not drop such a tutorial there anyway to make things funny for people who want to solve a problem, right?

Why explain it the right way if there is some half-right way?

Another thing most of those nice tutorial writers completely forget about is the most common way of $something. I recognized this when I read one of the better tutorials on the web.

When I am deploying something on some UNIX box, they forget to mention what kind of UNIX that box is running, they also forget to mention the version of the Apache they use and many other things & details which are important to actually reproduce the success they had. They forget to mention they run their Python application using the global system-wide python installation with all it’s installed modules. You can be happy if someone tells you upfront that his setup uses Python 3.x or Python 2.x but most of the time you recognize that painfully later on.

What does it need to succeed?

For me, I just wanted to know how to bring a bug tracking solution to life on a production server. So these are my needs:

  1. Give me screenshots of what the possible solution would look like
    because if it is an ugly piece of crap, I won’t ever spend hours installing it. Bonus, have a running demo of your solution on the web (without fancy registrations).
  2. Give me an easy to find downloadlink of the software
    most solutions hide the download of their package for installation somewhere on the page.
  3. Next to the downloadlink put the link to the installation manual. Keep that manual short and precise and provide it for the three most common use-cases. If you need some kind of rocket science preconditions which would require some neurosurgery on my running production system, then tell me here in big red letters and put a warning sign next to them.
  4. If you ever opt-in to let your software be redistributed through e.g. apt-get make damn sure that you distribute the most recent version in a meaningful way for the most common use-case.
  5. If there are preconditions then talk about those preconditions.
    E.g. tell me that in the process of installation I will need some mySQL server, so I can install that before I start with your tool.
  6. If something runs out of the box like magic, someone kept the very important details about environment variables, searchpaths & preconditions (like e.g. preinstalled global packages e.g. for Python, PHP, etc.) to himself and you will never be able to reproduce this success on a different machine.
    So please: tell me the exact environment variables, searchpaths, versions of packages needed, etc. to reproduce your success.
  7. Actually explain why something is done exactly this way so I can learn from that. Things usually have a reason why they are done this way and not the other way. But especially on UNIX there often are a thousand ways to solve a problem. Tell me why you chose exactly your approach.

How I did install Python app roundup

First things first. So here I will show you all files relevant to us right now.

Apache

Apache2 site-configuration file tasks.meinserver.de which resides in my folder /etc/apache2/sites-available and which was enabled in Apache typing in the command line
$ a2ensite tasks.meinserver.de
After activating that site you need to do a
$ service apache2 restart
which will automatically reload and apply all config changes.

# apache config for tasks.meinserver.de config
<VirtualHost *:80>
  # configure admin email and name of this domain
  ServerAdmin admin@somehost.com
  ServerName  tasks.meinserver.de
  
  # directory where usually all the static html files go
  DocumentRoot /home/www/tasks.meinserver.de/htdocs/
  
  # configure WSGI adapter
  WSGIScriptAlias / /home/www/tasks.meinserver.de/app/tasksapp.wsgi
  WSGIDaemonProcess tasksapp user=www-data group=www-data processes=1 threads=5 
  <Directory /home/www/tasks.meinserver.de/app/>
    WSGIProcessGroup tasksapp
    WSGIApplicationGroup %{GLOBAL}
    WSGIScriptReloading On
    Order deny,allow
    Allow from all
  </Directory>

  # directory where to store log files i.e. errors
  ErrorLog  /home/www/tasks.meinserver.de/logs/error.log
  CustomLog /home/www/tasks.meinserver.de/logs/access.log combined

  # Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
  LogLevel warn
</VirtualHost>

As you can see, in line 08 we refer to the DocumentRoot of this particular website which is served by Apache. Usually you will not need that because your website will be created by the python app we will deploy in a minute.

But, it is nice to have this DocumentRoot and also some simple index.html inside that folder, to easily check if the Apache works like expected. You can check if Apache works by deactivating lines 10 to 19 (all related to WSGI) and what you should then see is what resides in the DocumentRoot.

WSGI

The next file is the .wsgi-file which is passed into the Apache. That file was referenced in line 11 of the siteconfig. It is basically also a python file, but it is actually only a kind of wrapper to kickstart your real Python app.

There are two important lines of code in it. On line 01 we tell Apache which Python-command should be used to execute this WSGI-file. This is important, if we do not define this, some random global python may be used. You never know which version that python is and which modules it may have installed.

So in line 01 we explicitly define that this file should be executed with a python residing in the folder /home/www/tasks.meinserver.de/app/bin we also append a searchpath in line 05 to point to the folder /home/www/tasks.meinserver.de/app. This will tell Python where to search for our .py-file when it tries to import from that file.

Both of these path statements are very important. They tell Apache how this app should be launched and they manifest which Python environment will be used for execution.

#! /home/www/tasks.meinserver.de/app/bin/python
# -*- coding: utf-8 -*-

import sys
sys.path.append('/home/www/tasks.meinserver.de/app')

from tasksapp import app as application

Python app

So tasksapp.wsgi is used to launch tasksapp.py the real application which will then produce an answer to requests coming from the web via Apache:

#! /home/www/tasks.meinserver.de/app/bin/python
# -*- coding: utf-8 -*-

import sys 
sys.path.append("/home/www/tasks.meinserver.de/app/lib/python2.7/site-packages")

from flask import Flask
 
app = Flask( __name__ )
  
@app.route("/")
def hello():
        return "<html><body><h1>Hellow world! Python WSGI app running.</h1></body></html>"

if __name__ == "__main__":
  app.run()

This simple app uses a very common module called Flask which provides a lightweight webapplication framework to easily produce dynamic websites or other responses delivered through the webserver.

But Flask does not belong to the standard python installation it has to be installed separately and someone has to tell our application where to find this module. This is why we again need to define which Python should be used to execute this file (see line 01) and where to search for the modules (see line 05).

Installation

Before we install anything that is a python script, it is best to use virtualenv to build an encapsulated environment for this python application to run in. virtualenv does exactly this, it creates a new folder by creating a full copy of your default/global python installation without all global modules you might have installed. I used following command to create my app folder.

virtualenv --always-copy /home/www/tasks.meinserver.de/app

Check out the help page for virtualenv to adjust the created folder to your needs. Maybe you want a different python version installed.

Read this again

Okay, now read this text again in backward order. I know that still several pieces are missing also in this post but I had the urge to share what I wrote down already. To be continued…

Why do I blog this? I found it not that easy to deploy python in a stable and reproducable way. So this is why I want to document a way that worked for me. I would not recommend deploying anything webserver related through apt-get installation, never do this. This is far to risky and it does not communicate the important things you need to know for mission critical deployments. Stay away from magic you do not understand.

stitch n. (stch), in stitches Informal: Laughing uncontrollably.

Hab ich schon gesagt, dass der Hackerspace Bremen jetzt eine Stickmaschine hat?

First stitch

Gestern hab ich während überall Leute auf einen kleinen weißen Ball geguckt haben der auf grünem Rasen hin und her ging, mal die Stickmaschine in näheren Augenschein genommen.

Nachdem ich ein wenig Recherche gemacht habe welche Software so für den Mac verfügbar ist hab ich mich dran versucht mit einem InkScape-Plugin eine Stickdatei zu erstellen.

Nachfolgend das Ergebnis. Ich bin sehr zufrieden und habe haufenweise Dinge bemerkt und gelernt in Sachen Stickmaschinen-Handling.

Sticken_101_550

Das InkScape-Plugin das ich benutzt habe ist hier im Detail erklärt und nutzt Python für die Erzeugung einer Stickdatei aus einem Vektorpfad einer Vektorgrafik in InkScape.

Das war der erste Versuch mit den Defaulteinstellungen für dieses InkScape-Plugin. Schwierig war eher die Installation von InkScape und dem Plugin selbst als dann die eigentliche Ausführung.

Stickmaschinen-Handling for Dummies

To be continued…

Die Hymne: „Art of the Dress“

MyLittlePony_logoGesungen von Kazumi Evans (video) in Season 1 „Suited for Success“ in der TV Serie My little Pony, hier der zugehörige Songtext:

Thread by thread, stitching it together
Twilight’s dress, cutting out a pattern snip by snip
Making sure the fabric folds nicely
It’s the perfect color and so hip
Always got to keep in mind my pacing
Making sure the cloth’s correctly facing
I’m stitching Twilight’s dress

Yard by yard, fussing on the details.
Jewel neckline. Don’t you know a stitch in time saves nine?
Make her something perfect to inspire
Even though she hates formal attire
Got to mind those intimate details,
Even though she’s more concerned with sales.
It’s Applejack’s new dress.

Dress making’s easy
For Pinkie Pie something pink
Fluttershy something breezy
Blend color and form, do you think it looks cheesy?

Something brash, perhaps quite fetching
Hook and eye, couldn’t you just simply DIE
Making sure it fits forelock and crest
Don’t forget the magic in the dress
Even though it rides high on the flank
Rainbow won’t look like a tank
I’m stitching Rainbow’s dress

Piece by piece, snip by snip
Croup, dock, haunch, shoulders, hip
Thread by thread, primmed and pressed
Yard by yard, never stress
And that’s the art of the dress

Now, the stars on my belt need to be technically accurate.
Orion has three stars on his belt, not four
Stitch by stitch, stitching it together
Deadline looms, don’t you know the client’s always right
Even if my fabric choice was perfect
Gotta get them all done by tonight
Pinkie Pie, the color is too obtrusive
Wait until you see it in the light
I’m sewing them together

Don’t you think my gown would be more me with some lollipops?
Well, I think…
Balloons?
Well…
Do it!
Hour by hour, one more change
I’m sewing them together, take great pains

Fluttershy, you’re putting me in a bind
Rainbowdash, what is on your mind
Oh my gosh, there’s simply not much time
Don’t forget, Applejack’s duds must shine

Dressmaking’s easy
Every customer’s call
Brings a whole new revision
Have to pick up the pace, still hold to my vision

That constellation is Canis Major, not Minor.
French haute couture, please.
Eeeh…
What if it rains? Galoshes!
More balloons! Oh no, that’s too many balloons.
More candy! Oh, less candy.
Oh wait, I know! Streamers!
Streamers?
Whose dress is this?
Streamers it is.
What?
Aren’t you going to tell me to change something too?
No, I just want my dress to be cool.
Do you not like the color?
The color’s fine, just make it look cooler.
Do you not like the shape?
The shape’s fine, just make the whole thing, you know, cooler.
It needs to be about 20% cooler.

All we ever want is indecision
All we really like is what we know
Gotta balance style with adherence
Making sure we make a good appearance
Even if you simply have to fudge it
Make sure that it stays within our budget
Got to overcome intimidation
Remember it’s all in the presentation

Piece by piece, snip by snip
Croup, dock, haunch, shoulders, hip
Bolt by bolt, primmed and pressed
Yard by yard, always stressed
And that’s the art of the dress

Why do I blog this? Weil ich das mit dem Sticken schon immer mal selbst machen wollte. Und ich hab eine ganze Reihe Ideen und Vorhaben damit. Und weil es super Spaß macht! Die Stickmaschine zu dem zu bewegen, was man möchte ist ne tolle Sache. Das Ergebnis hinterher in der Hand halten zu können noch toller! Jedenfalls besser als Software für die man ja immer ein Hilfsgerät braucht, um sie jemandem zu zeigen.

django – some first impressions

django-logoSince I mastered the setup of my apache config to actually find my django project I experience some more fluent development happening.

Actually I learned already a lot about the powerful namespacing pattern for URLs which frees up a lot of capacity you usually spend on fixing broken link experiences for users and wrapping your head around url-paths.

Actually the url-handling via regex is a bit painful at first, but it turns out that this gives you extraordinary flexibility to create wonderfully simple and elegant looking urls which is a thing nowadays if you wanna be found in Google at the top ranks and provide useful permalinks e.g. for products whoich should be found easily.

Also the template engine is very flexible. I can create a kind of loopholes (called blocks) in my base templates to prepare them to get stuff injected from lower level templates which is great. You could build a very complex site in no time and have things like header, navigation, content, footer easily setup and look consistently.

Learning with the best tutorials

After I did the painful 6-part tutorial, I just did another very good tutorial on youtube (and still executing on it). And boy was I happy to actually have learned things the „right way“ before. because in the youtube based tutorial above the guy explaining stuff there totally misses on the concept of url-namespacing. But see for yourself here:



Source: Django Full Website Tutorial by hackedexistence.com

If you know some excellent tutorials, e.g. on how to setup django for SSL-based deployment and the like, let me know in the comments.

I also found some very helpful other links:

Python tricks often appreciated

During all my tinkering I also did used a lot of really useful operations using the manage.py command. E.g. today I just wiped my database to try repopulating it with stuff I dumped before. It was quite easy and which is good to have for experimenting and not loosing all your data. Just be careful you do not change the DB-scheme during these operations.


python manage.py dumpdata > temp_data.json
python manage.py flush
python manage.py loaddata temp_data.json

I also made a lot of use of following command:

python manage.py shell

This loads the whole project or in a way has all the settings needed setup for you. Some quick

>>> from <myApp>.models import <myEntity>
>>> dir(<myEntity>)

Gave me really helpful insights, what the object I have actually is capable of doing, by displaying all the methods and attributes available.

My bootstrapping file for Vagrant now was extended by a lot of useful stuff, but actually I did not destroy my VM recently so if all this works will be seen on the next setup of my VM using vagrant up.

Content I added to my bootstrap.sh file:

# SETUP FOR BOOTSTRAPPING
apt-get update

# INSTALL apache
apt-get install -y apache2

# INSTALL mod_wsgi
apt-get install libapache2-mod-wsgi

# INSTALL django
apt-get install -y python-django

# INSTALL python imaging library
apt-get install python-imaging python-imaging-tk

# INSTALL postgres & python db connector
apt-get install postgresql python-psycopg2

I also backed up my apache config files now to some place external to the VM (shared folder) so I do not loose all config stuff if I destroy the VM.

Further education

To gain some flexibility for my models I just started reading stuff like e.g. „33 projects that make developing django apps awesome“ and „Django database migration tool: south, explained“.

south_logoSouth (start Tutorial here) looks like an elegant solution for my purposes because it provides intelligent schema and data migrations.

The plain vanilla way to do migrations would be Django’s Evolution. But even they point out that South has become a De facto standard. Looks like some stuff was inspired by ActiveRecord from rails.

So some line of
sudo apt-get install python-django-south
just made it fly. I like the ubuntu way of installing software… just to revert the action using
sudo apt-get remove --purge python-django-south
several minutes later because south was not correctly installed and better should have been installed through using pip.

So doing a sudo pip install south to get the e x a c t same result. So must be some other error. Doing some googleing on that one… turns out south needs to be entered in the INSTALLED_APPS entry in the settings.py… nice that you find that on the documentation so easily. So this stackoverflow-line helped me out again:

You probably haven’t added ’south‘ into the list of INSTALLED_APPS of your settings.py file

But at least I now know the exact version of south installed. It’s 0.8.2 the one installed by apt-get was only 0.7.3, so I guess pip is the better way to fly. And now python manage.py did show:

[south]
convert_to_south
datamigration
graphmigrations
migrate
migrationcheck
schemamigration
startmigration
syncdb
test
testserver

Fine! Now that I got that installed the python manage.py syncdb changed its output to some new formatting which involves some south-magic. It now looks like:

Syncing...
Creating tables ...
Creating table south_migrationhistory
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

Synced:
> django.contrib.auth
> django.contrib.contenttypes
> django.contrib.sessions
> django.contrib.sites
> django.contrib.messages
> django.contrib.staticfiles
> django.contrib.admin
> south
> shop
> shopadmin
> tinymce

Not synced (use migrations):
-
(use ./manage.py migrate to migrate these)

I just coninued to put my models under migration control using:
python manage.py schemamigration <myapp> --initial
Worked! But everything after that failed. Even a flush on the database did not help in any way. Hmm…
Adding south to an existing project needs special handling…
python manage.py convert_to_south <myApp>
also learned that deleting the migrations directory is not that evil after all if „things go south“, haha.

Migrations on the server

At the same time I learned that there is a procedure for handling migrations perfectly on the server, its like this:

  1. Install South on server. import south from shell just to make sure you are using the same python env.
  2. Add ’south‘ to INSTALLED_APPS in settings.py
  3. Upload settings.py
  4. Restart (web)server
  5. python manage.py syncdb
  6. Upload new app/models.py and app/migrations/ dir
  7. Restart (web)server
  8. python manage.py migrate app --fake 0001
  9. python manage.py migrate app

Good to know! Since I got some changes to my models waiting, I changed them now and handed over to south:

python manage.py schemamigration myproject.myapp --auto
python manage.py migrate myproject.myapp

Worked like a charme… after fixing some modelissues (adding null=True) and providing default values for some Decimals. Great!

Oh and if you have permission problems with your webserver directory in your VM managed by Vagrant, try adding following line in the Vagrantfile:
config.vm.synced_folder "./", "/vagrant", :owner=> 'vagrant', :group=> 'www-data', :mount_options => ['dmode=775', 'fmode=775']
This adds the directory to the group of the apache-users, so apache is allowed to write into those files. Here it is www-data-group but it may be different for your VM/Webserver combination.

Localization

Today I tried to use translated strings in my templates e.g. using following statement in one template
{% trans 'Willkommen auf meiner Homepage.' %}
you need to add
{% load i18n %} at the top of each template file and below any base-template statements.
using django-admin.py makemessages -l de --all
I tried to create localizationfiles for my app. But that did not really work because I needed to create locale-dirs first in my project.
And then I got the famous error:
CommandError: Error running xgettext. Note that Django internationalization requires GNU gettext 0.15 or newer.
which I fixed using the ubuntu SW installation manager by typing
sudo apt-get install gettext
and adding it to the bootstrap.sh for my VM.

Then after entering
django-admin.py makemessages -l de --all
which created a .po-file in my project
locale/de/LC_MESSAGES/django.po
Repeating the same procedure for english language revealed another folder for en. But…
that alone was not sufficient by far. You need to do a compile on the locale files before you even have a chance of seeing translation happen.
django-admin.py compilemessages and in my case
I also needed to add the LOCALE_PATHS to actually find my locale dir in the project, i have no clue why it does not find that automatically as advertised.

Authentication and sending mails

In the meantime I implemented registration/login/logout and user profile. Following my tutorials I just took django’s default implementation of how to reset passwords for user accounts. That’s when I hit the next wall. But that was really an easy one. If you want to send mails you need a mailserver. After some googeling i went with postfix for testing:
sudo apt-get install postfix
made this work, otherwise I saw an error which pointed me directly towards the problem. I need to remember this for deployment to actually install a maiserver which is stable and configure it in the settings.py according to this and this and during development maybe this.

Actually I would prefer to just setup the following settings:

EMAIL_HOST = 'localhost'
EMAIL_PORT = 1025

and then type
python -m smtpd -n -c DebuggingServer localhost:1025
this prints emails which are normally sent via any configured mailserver to the console to debug its content and headers. Very useful!!!

Other useful stuff I discovered during my voyage… (basically I just store all the open tabs on my browser here, because I need to reboot the machine for some hardware driver I need to install which needs a reboot of my mac.)

Wow, the new device driver fixed my issue with this gaming mouse. The default behaviour of the Cyborg R.A.T. 9 mouse is really pretty ugly. Have a look at this enormous proof of how MadCatz actually doesn’t even care about any Mac customer. Actually you should not buy anything from a company which behaves silly as this. It renders your mousepointer nearly useless. Since Cyborg did never provide any fix for the missing OS X driver and installing the old driver for OS X 10.6 renders completely crazy issues with the mouse… I went with USB Overdrive now. And wow it seems to fix the issues. I cannot manipulate the 3 different setups of the mouse, but I can pretty much freely configure any other button and all scrollwheels now. Great thanks to USB Overdrive! BTW: here is a great review of USB Overdrive.

Taking a REST

Today I started opening up my small website using some REST API. I started installing necessary stuff following instructions over here by typing:

sudo pip install djangorestframework
sudo pip install markdown
sudo pip install django-filter

Wow!!!! Blown away just another time! REST-API support in actually no time. Haha, how cool is that?

By working my way through the tutorial on how to JSONify stuff, I created my own apiviews.py (I did not want that API stuff in my regular views) and then I just setup the urls.py with som additional info. (meanwhile I found myself trying to close a browser tab by typing :q in the window, perhaps it is time for vimperator…). I used the python interactive shell to actually experiment with the stuff in the tutorial. Which is really perfect to get to know the details of how JSONParser and JSONRenderer actually work, i.e. using a stream created with StringIO.

So in the end this took me just one day to grab the concept of JSON-api-ifying my django app to get at least a stable READ-ONLY api. But now I will go on and work on the READ,CREATE,WRITE,DELETE stuff which will involve some validations of incoming JSON and objects created. We will see…

BTW: Meanwhile I just found another nice Video Tutorial on the Python/Django stuff. Looks also very clear in its explanations and also provides some much better input about the background of how django works. So definitely worth a try, though the guy smartly „skips“ (edited out of the video) several very delicate things (like installing the mySQL amd mysql-python, haha).

Update
To make my own experience on video-tutorials as comfortable as possible I just hacked myself a website for one of the tutorial series. It allows to insert any video-id from youtube and will allow free resizing of the video display. Also I preprogrammed the HackedExistence ID’s for you already. So convenience is just 1 click away.

To be continued…