Using JSON logging in Django (and Python in general)

For a while now, I've wanted to output JSON in my Django log files instead of plain text. However, Google searches and the Python and Django documentations fail to provide a single example that contains all of the required bits of information required to make it work.

Specifically, you need to know how to define a custom formatter in a dictionary config for a logger. All the examples are in UML or use the old-style method of instantiating a formatter and assigning it to a handler.

This makes use of the python-json-logger, available here:

https://github.com/madzak/python-json-logger

So, here is a working example. Simply do this in your own code and you'll be all set. Note that in Django you don't have to deal with the dictConfig at all -- just put the right things in your dictionary and Django will do the rest.

#!/usr/bin/env python


import logging
from logging.config import dictConfig
import jsonlogger


log_config = {
    'version': 1,


    'formatters': {
        'json': {
            '()': 'jsonlogger.JsonFormatter',
            'fmt': '%(levelname)s %(asctime)s %(message)s',
        },
    },


    'handlers': {
        'stream': {
            'level': 'DEBUG',
            'formatter': 'json',
            'class': 'logging.StreamHandler',
        },
    },


    'loggers':  {
        'my_logger': {
        'handlers': ['stream'],
        'level': 'DEBUG',
        'propagate': True,
        },
    }


}


dictConfig(log_config)
logger = logging.getLogger('my_logger')


logger.debug('foo')


Django from Scratch

Today I had the pleasure of creating a brand-new Django project from scratch. Here are some things I realized or remembered during my first couple of hours of work after maintaining a large Django project for over a year.

Create a local_settings.py File
Put it in the project folder and import it from settings.py with exception handling so it's okay if it doesn't exist. Make sure your version control ignores this file. Use this to override your database settings, add plugins (like Django Debug Toolbar), and to set DEBUG = True. Always leave DEBUG set to False in the real settings file. That's just one thing you don't want the slightest possibility of moving into production. Also, don't reverse the process and import settings.py in local_settings.py. It causes problems when you have to remember to set or import the proper DJANGO_SETTINGS_MODULE for any given environment. It's so much easier to just let Django use the default settings file hard-coded into manage.py.

Create Initial Fixtures
During development you'll likely create some data. This is useful for obvious reasons. If you create a fixture file with the name initial_data in a path specified in your FIXTURE_DIRS setting, it will be automatically loaded each time you flush the database, or after a syncdb. At the very least it will save you from re-entering the admin super-user info each time you refresh the database.

Use South
This has become a no-brainer for me. South is a database migration tool for Django which is very mature and has become the de facto standard in the community. As tweaks are made to the models, South makes modifying the database completely painless. It makes things a little easier if you create a South migration right from the start, rather than waiting till you make a change. The documentation is very good and explains all. Tip: When you get ready to deploy for the first time, you can always wipe out your South migrations and make a new "initial" migration. Your migrations folder will be cleaner and it'll look like you designed the models right on the first try.

Use setUp in Your TestCases
I could have also titled this one "test one thing at a time." I spent a lot more time today than I should have trying to figure out how to repair a dirty database connection after one test that (correctly) raised an IntegrityError. In the end, I broke out the following test into another function. In hindsight, there was no point in trying to do them both together, except that I was loading a couple of model instances that I wanted to use for both tests. Creating a setUp function in my test class solved the problem nicely, and is much cleaner. If a test ever modifies one of those instances it could introduce a subtle bug into my test suite.

Create Test Fixtures
Have some test data that your tests load. This may seem obvious, but when I first started writing tests I did it the hard way, creating model instances at the beginning of a test because they were dependencies of the model I actually wanted to test.

Create a Test Directory
A fresh Django app will leave a file named tests.py in the new directory. This is to remind you that testing is important. But a single file can become a mess. I much prefer the technique I learned from Eric Holscher's blog. Delete that tests file and create a tests directory. In that directory, create an __init__.py file. Create a bunch of test files, each serving a different purpose, and import them in __init__.py. That way, they'll be nicely organized and still be run automatically when you run ./manage.py test.

Use the related_name Keyword
In models where you use a ForeignKeyField, always specify a related_name. You're going to use it anyway eventually, so you may as well be consistent. Also, it will make your code (and your models) clearer to the maintenance programmer, who will probably be you. Also, adding one later on could break existing code that was using the default syntax.

Don't Use Sqlite
This one upsets me, because I love sqlite. However, it doesn't support certain constraints, which means tests will fail that would pass on the production database. Worse, some constraints do work, but only when a table is created, not when they're added later by a migration. Also, it doesn't support all the functionality required to use South, like the ability to delete columns from a table.

May your next project be a Django project!

--ShawnMilo

A Simple Celery with Django How-To

I've been wanting to give Celery a try for a while now, but every time I tried I ran into a major issue. Either I'm too stupid or the documentation completely fails to demonstrate a simple, end-to-end working example if you're not using RabbitMQ.

So, I finally cobbled something together from countless bits of documentation from the official site and some Google searches. Here it is for your amusement.

Details: I'm using my Django project's database and MongoDB.

Preparation:

• Use pip to install celery and ghettoq.
• Add ghettoq to your INSTALLED_APPS.
• Run syncdb.

Then you'll need a file named celeryconfig.py which exists on your PYTHONPATH. Here's mine:

#!/usr/bin/env python

from django.conf import settings

CARROT_BACKEND = "ghettoq.taproot.Database"

BROKER_HOST = settings.MONGO_HOST
BROKER_PORT = settings.MONGO_PORT

CELERY_RESULT_BACKEND = "mongodb"
CELERY_MONGODB_BACKEND_SETTINGS = {
"host": settings.MONGO_HOST,
"port": settings.MONGO_PORT,
"database": "celery",
"taskmeta_collection": "celery_collection",
}

CELERY_IMPORTS = ("myproject.myapp.tasks", )

Notes on the above:

• I have my MongoDB settings in my settings.py file.
• I've created a file named tasks.py and put it in the folder of one of my Django apps.

The tasks file contains two functions. One will be run whenever I call it from a Django view. The other will run every five minutes, cron-style.

That tasks file looks like this:

from celery.decorators import task
from celery.task.schedules import crontab
from celery.decorators import periodic_task

from myproject.myapp.models import Appointment

from datetime import datetime, timedelta
import time

@task
def sample_task():
 print "gonna do something now and later!"
 time.sleep(10)
 print "It's later!"

@periodic_task(run_every=crontab(minute="*/5"))
def send_appointment_reminders():

 #appointments one day away
 start_date = datetime.now() + timedelta(days = 1)
 end_date = start_date + timedelta(minutes = 5)

 appointments = Appointment.objects.filter(
     scheduled__gte = start_date,
     scheduled__lt = end_date,
 )

 print "%d appointments found." % (appointments.count())

 #send appointment reminder for this appointment now
 pass

Obviously the functions here are incomplete and trivial, but they cover my two basic needs:

• Run something asynchronously from a view.
• Run something on a schedule without decoupling it from my Django project, making it harder to manage and maintain configurations between development and production. (A.K.A. "avoid cron.")

Last, but not least, add some code to one of your Django views:

from myproject.myapp import tasks

result = tasks.sample_task.delay()
print "result was %s" % (result,)

Then just run celeryd -B and you're all set. (The -B tells the daemon to also run the celerybeat daemon, which runs the Celery 'cron' functions. Otherwise you have to run celerybeat separately or the cron jobs are ignored.)

That's it! What's missing here is explaining how to obtain the results of the tasks later on. I haven't gotten that far yet, but if you take "result" in the sample above it contains a UUID. You can save that UUID now and use it subsequently to query the status of your asynchronous function call.

I hope this helps someone.

Second Medium Day

Today's Wednesday, which the RKC Rite of Passage schedule deems a 'medium' day. Last Wednesday I did one four-rung ladder and 15 swings.

Today, I did four four-rung ladders and 30 swings. I don't think I'll make five five-rung ladders on my next 'heavy' day (Saturday), but it won't be much longer.

126 Swings

Dave and I had our little swing competition again. It hasn't happened every Friday as we planned, for various reasons. So, today I made it to 126 swings in the five-minute period. Dave did 100, and his form was a lot more solid this time.

114 Swings

My co-worker Dave and I decided to have a little competition -- kettlebell swings for reps within five minutes. I won, 114 to 100. I maintain the position that I could have easily done more had I realized how little time was remaining.

The important thing is that I maintained a good form throughout. We took a video of the swings so we could count if needed. That's good, because I don't count while swinging, and I walked away thinking I had done fewer than I actually did.

This is going to become a weekly contest, so I'll continue to post the results. It'll also give my daily training some sort of structure, because I'll be working toward a goal. Also, it'll give me something kettlebell-related to post, since I haven't had much to write about the topic lately.

Green Light!

I had the opportunity to have my kettlebell swing evaluated by an RKC-certified trainer yesterday, and it looks like I'm good to go. My form is good. My technique isn't yet perfect, but that will come with repetition.

In any case, I don't have any major new insights to report regarding kettlebells. It's the new year and I'm ready to start training in earnest. I also started to eat a Paleo Diet a couple of days ago. So far I'm feeling great. So, I've been eating better and I haven't lost interest in the kettlebell; I've just been lazy about writing posts.

Look for another update sometime this month.

Jeff Martone

Since I've only mentioned Pavel thus far, I thought I should write about Jeff Martone as well. He has some great video clips available for free on the Crossfit site, and he also has some advanced kettlebell DVDs on his own site. I've learned things just from the free videos.

Pavel gives you all the basics, and it could be reasonably argued that you don't need anything else. However, I think Martone's work is important for several reasons.

First of all, despite the fact that Pavel's name is almost synonymous with kettlebells in the US, he's marketing various fitness products, many of which have nothing to do with kettlebells. However, Martone seems to specialize in kettlebells.

Secondly, while Pavel's instruction is very high-energy and motivational, Jeff's teaching style is more casual and informational. He provides nuances that Pavel doesn't even mention.

Also, Jeff Martone mixes it up, giving little variations to keep things interesting. With Pavel, you can design a perfect workout for life. With Martone, you can easily make every workout a little different, which keeps it interesting. Don't let me give you the impression that Jeff gives you a bunch of new moves and complicates things. Rather, he explains minor variations in the same exercises, and I think that makes them more fun and less repetitive.

I haven't yet seen any of his DVDs, but the video clips I watched were great, and his teaching style was excellent. It could be that the videos were of him teaching other personal trainers and not end-users, but I really felt like I was getting all the in-depth info.

Kettlebell Posture

Okay, so I'm getting the hang of this kettlebell thing. (I refuse to make the easy joke and say I'm 'getting into the swing' of it.) One thing I wanted to report, as I continue to get comfortable with the basic moves and work on perfecting my form is that a kettlebell will improve your posture.

I'm a sloucher. I'm slouching in my chair right now. But you won't catch me slouching after a kettlebell workout. That's because keeping a straight back during the exercises is essential, and afterwards (and the next day), it feels better to keep your back straight.

Interestingly, that changes the way I walk, and improves my balance. As Pavel likes to say, Americans tend to walk by falling forward and catching themselves on their feet, and Russians lead with their stomachs, whether they have them or not.

He also said that his instructors frequently receive the comment from students that "you guys all stand the same way." His response is "so will you." I believe it.

So, just one more plus for kettlebells. In addition to having workouts that actually directly translate to movements you might be called upon to do in an average day of playing with your kids or grocery shopping, it also improves your posture. Time to sell my dumbbells.

First Swings

Well, I finally got a kettlebell this Monday. I don't have too much to report because I'm taking it slow. Pavel advises a new kettlebell user to "practice," rather than "work out" for at least a couple of weeks. It is vital to hone proper form as a beginner, because when you move onto the more complicated moves or higher repetitions, the chance for injury increases if the foundation isn't in place.

I have taken a number of swings and shot video to study my form. At first it was pretty ugly, but I think I'm getting the feel of it now. I'll be on vacation from work next week, and I intend to put in some quality time with the kettlebell and expand my exercises. Thus far I've only done the basic swing and the one-handed swings. I want to start working on the cleans and presses.

I'm still doing research into various diets. And by 'diet' I mean permanent eating habits, not "lose 10 pounds in 10 days" kind of stuff. I'll report on that stuff another time.