Welcome to django-postgres-metrics’s documentation!¶

Background. Why? What?¶

The original idea for this library comes from Craig Kerstiens’s talk “Postgres at any scale” at PyCon Canada 2017. In his talk, Craig pointed out a bunch of metrics one should look at to understand why a PostgreSQL database could be “slow” or not perform as expected.

In many cases, the root cause for “PostgreSQL being slow” is the lack of memory, full table scans, and others. For example, a cache hit ratio of less than 99% means that each of cache miss queries needs to go to the filesystem. Filesystem access is significantly slower than RAM access. Giving the PostgreSQL instance more memory will very likely improve performance.

This project aims at showing these and other metrics in the Django Admin making such bottlenecks visible.

Getting Started¶

Installation¶

Start by installing django-postgres-metrics from PyPI:

$ pip install django-postgres-metrics

Then you need to add postgres_metrics to your INSTALLED_APPS list. Due to the wait postgres_metrics works, you need to include it before the admin app:

INSTALLED_APPS = [
    'postgres_metrics.apps.PostgresMetrics',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

Lastly, you need to add a URL path to your global urls.py before the admin URL patterns.

For Django 2.0 and up:

from django.urls import include, path
urlpatterns = [
    path('admin/postgres-metrics/', include('postgres_metrics.urls')),
    path('admin/', admin.site.urls),
]

For Django 1.11 and before:

from django.conf.urls import include, url
urlpatterns = [
    url(r'^admin/postgres-metrics/', include('postgres_metrics.urls')),
    url(r'^admin/', admin.site.urls),
]

Congratulations, you made it!

When you now browse to the Django Admin with superuser permissions, you’ll see a “PostgreSQL Metrics” section at the bottom left, listing all available metrics.

Designing Your Own Metric¶

First, you need to import a base class that all metrics inherit from. This will take care of fetching the data from the database, implement sorting if you want, and provide per record and per item in a record styling. Furthermore, you need to make your metric know to django-postgres_metrics-metrics:

from django.utils.translation import ugettext_lazy as _

from postgres_metrics.metrics import Metric, registry


class DjangoMigrationStatistics(Metric):
    """
    Count the number of applied Django migrations per app and sort by
    descending count and ascending app name.
    """
    label - _('Migration Statistics')
    slug - 'django-migration-statistics'
    ordering - '-2.1'
    sql - '''
        SELECT
            app, count(*)
        FROM
            django_migrations
        GROUP BY
            app
        {ORDER_BY}
        ;
    '''


registry.register(DjangoMigrationStatistics)

A short explanation of what the metric class attributes do:

docstring:	This will be shown above the metric’s output and allow you to give an explanation what you see.
label:	This is what you see on the Django Admin index page as a metric name.
slug:	A unique identifier for a metric. This will be used as part of the URL.
ordering:	The default ordering you want to use in the metric. Use column indexes (one-indexed) and prefix with `-` for descending sorting.
sql:	The actual SQL you want to run. The `{ORDER_BY}` part is replaced with `ORDER BY 2 DESC, 1` in the example.

Styling Metric Output¶

Styling Records¶

If you want to highlight some of the output rows you can define a get_record_style method on a metric class:

class MyMetric(Metric):
    ...

    def get_record_style(self, record):
        if record[0]:
            if record[1] > 1000:
                return 'critical'
            if record[1] > 100:
                return 'warning'
            if record[1] == 0:
                return 'info'
            return 'ok'

The record parameter is a tuple with all values of a single row in the output. This method will be called for every single row being outputted. Don’t do any expensive calculations here!

django-postgres-metrics ships four pre-defined styles: ok, warning, critical and info that you can return.

Styling Record Items¶

Similarly, you can highlight a single value in the metric output by using the get_record_item_style method on a metric class:

class MyMetric(Metric):
    ...

    def get_record_item_style(self, record, itme, index):
        if index == 2 and record[1]:
            if item > 1000:
                return 'critical'
            if item > 100:
                return 'warning'
            if item == 0:
                return 'info'
            return 'ok'

Along with the record you get the current value or item and the (zero- indexed) position of the item in the record. The item is provided for convenience and is defined as item = record[index].

Contributing¶

Issuing a new Release¶

Install bumpversion with:

$ pip install git+ssh://git@github.com:MarkusH/bumpversion.git@sign#egg=bumpversion

Install twine with:
```
$ pip install twine
```
Determine next version number from the changelog.rst (ensuring to follow SemVer)
Ensure changelog.rst is representative with new version number and commit possible changes.
Update the version number with bumpversion:
```
$ bumpversion $part
```
(instead of $part you can use major, minor, or patch.
git push --tags origin master
Check for TravisCI to complete. If TravisCI fails due to code errors, go back to the start and bump the $part with patch
Build artifacts with:
```
$ python setup.py sdist bdist_wheel
```
Upload artifacts with:
```
$ twine upload -s dist/*$newver*
```
Add likely new version to at the top of changelog.rst

Reference Documentation¶

AppConfig¶

class postgres_metrics.apps.PostgresMetrics(app_name, app_module)[source]¶

Bases: django.apps.config.AppConfig

name = 'postgres_metrics'¶

verbose_name = 'PostgreSQL Metrics'¶

Metrics¶

class postgres_metrics.metrics.MetricRegistry[source]¶

Bases: object

register(metric)[source]¶: Given a class (not class instance) of Metric, adds it to the available list of metrics to show it to a user.

sorted¶: All registered metrics ordered by their label.

unregister(slug)[source]¶: Remove the metric slug from the registry. Raises a KeyError if the metric isn’t registered.

class postgres_metrics.metrics.Metric(ordering=None)[source]¶

Bases: object

The superclass for all Metric implementations. Inherit from this to define your own metric.

If you want to implement your own metric, here’s a gist:

from django.utils.translation import ugettext_lazy as _
from postgres_metrics.metrics import Metric, registry


class DjangoMigrationStatistics(Metric):
    """
    Count the number of applied Django migrations per app and sort by
    descending count and ascending app name.
    """
    label = _('Migration Statistics')
    slug = 'django-migration-statistics'
    ordering = '-2.1'
    sql = '''
        SELECT
            app, count(*)
        FROM
            django_migrations
        GROUP BY
            app
        {ORDER_BY}
        ;
    '''


registry.register(DjangoMigrationStatistics)

description¶

Don’t define this value directly. Instead define a docstring on the metric class.

The docstring will be processed by Python internals to trim leading white spaces and fix newlines. '\r\n' and '\r' line breaks will be normalized to '\n'. Two or more consecutive occurances of '\n' mark a paragraph which will be escaped and wrapped in <p></p> HTML tags. Further, each paragraph will call into Django’s urlize() method to create <a></a> HTML tags around links.

full_sql¶: The sql formatted with get_order_by_clause().

get_data()[source]¶

Iterate over all configured PostgreSQL database and execute the full_sql there.

Returns:	Returns a list of `MetricResult` instances.
Return type:	list

get_order_by_clause()[source]¶

Turn an ordering string like 1.5.-3.-2.6 into the respective SQL.

SQL’s column numbering starts at 1, so do we here. Given self.ordering as 1.5.-3.-2.6 return a string ORDER BY 1 ASC, 5 ASC, 3 DESC, 2 DESC, 6 ASC.

Ensures that each column (excluding the - prefix) is an integer by calling int() on it.

get_record_item_style(record, item, index)[source]¶

Given a single record from MetricResult, the value of the current item within, and the current item’s index, decide how to style it. Most likely to be used with the template tag record_item_style().

By default, django-postgres-metrics supports for styling classes:

ok
warning
critical
info

Override this method and return one of the above strings or None to apply the given style to the entire record. In the Django Admin this will highlight the entire row.

get_record_style(record)[source]¶

Given a single record from MetricResult, decide how to style it. Most likely to be used with the template tag record_style().

By default, django-postgres-metrics supports for styling classes:

ok
warning
critical
info

Override this method and return one of the above strings or None to apply the given style to the entire record. In the Django Admin this will highlight the entire row.

label = ''¶: The label is what is used in the Django Admin views. Consider marking this string as translateable.

ordering = ''¶: The default ordering that should be applied to the SQL query by default. This needs to be a valid ordering string as defined on parsed_ordering.

parsed_ordering¶

Turn an ordering string like 1.5.-3.-2.6 into the respective abstraction.

Given self.ordering as 1.5.-3.-2.6 return a lis of 2-tuples like [('', 1), ('', 5), ('-', 3), ('-', 2), ('', 6)].

slug = ''¶: A URL safe representation of the label and unique across all metrics.

sql = ''¶: The actual SQL statement that is being used to query the database. In order to make use of the ordering, include the string {ORDER_BY} in the query as necessary. For details on that value see get_order_by_clause().

class postgres_metrics.metrics.MetricResult(connection, records)[source]¶

Bases: object

Hold a metric’s data for a single database.

alias¶: The alias under which a database connection is known to Django.

dsn¶: The PostgreSQL connection string per psycopg2.

records¶: The rows returned by a metric for the given database.

class postgres_metrics.metrics.MetricHeader(name, index, ordering)[source]¶

Bases: object

A single column header; mostly takes care of a column’s sorting status.

ascending¶: True if the column is in ascending order False otherwise

static join_ordering(ordering)[source]¶

sort_priority¶: The priority of the columns order. 1 (high), n(low). Default 0.

url_primary¶: Querystring value making this the primary sorting header.

url_remove¶: Querystring value removing this column from sorting.

url_toggle¶: Querystring value toggling ascending/descending for this header.

class postgres_metrics.metrics.AvailableExtensions[source]¶: PostgreSQL can be extended by installing extensions with the CREATE EXTENSION command. The list of available extensions on each database is shown below.

class postgres_metrics.metrics.CacheHitsMetric[source]¶

The typical rule for most applications is that only a fraction of its data is regularly accessed. As with many other things data can tend to follow the 80/20 rule with 20% of your data accounting for 80% of the reads and often times its higher than this. Postgres itself actually tracks access patterns of your data and will on its own keep frequently accessed data in cache. Generally you want your database to have a cache hit rate of about 99%.

(Source: http://www.craigkerstiens.com/2012/10/01/understanding-postgres-performance/)

class postgres_metrics.metrics.IndexSizeMetric[source]¶

class postgres_metrics.metrics.IndexUsageMetric[source]¶

While there is no perfect answer, if you’re not somewhere around 99% on any table over 10,000 rows you may want to consider adding an index. When examining where to add an index you should look at what kind of queries you’re running. Generally you’ll want to add indexes where you’re looking up by some other id or on values that you’re commonly filtering on such as created_at fields.

(Source: http://www.craigkerstiens.com/2012/10/01/understanding-postgres-performance/)

class postgres_metrics.metrics.TableSizeMetric[source]¶

Template Tags¶

postgres_metrics.templatetags.postgres_metrics.get_postgres_metrics()[source]¶: Return an iterable over all registered metrics, sorted by their label. See MetricRegistry.sorted for details.

postgres_metrics.templatetags.postgres_metrics.record_style(context)[source]¶

Return the style to be usd for the metric’s current record.

This expects a template context variable 'metric' referring to the metric instance in question, as well as a context variable 'record' containing the current record to be used for style determination.

This template tag calls into Metric.get_record_style and will—if a return value was specified—prefix that one with 'pgm-'.

postgres_metrics.templatetags.postgres_metrics.record_item_style(context)[source]¶

Return the style to be usd for the metric’s current record specific item.

Like the record_style() template tag, this expects a template context variable 'metric' referring to the metric instance in question, as well as a context variable 'record' containing the current record. Furthermore, the 'forloop.counter0' context is expected with the current column index to be styled. This mean, for styling, unline on the SQL column numbering, columns are zero-indexed.

This template tag calls into Metric.get_record_item_style and will—if a return value was specified—prefix that one with 'pgm-'.

URLs¶

postgres_metrics.urls.re_path(route, view, kwargs=None, name=None, *, Pattern=<class 'django.urls.resolvers.RegexPattern'>)¶

Views¶

postgres_metrics.views.metrics_view(request, name)[source]¶

Changelog¶

0.5.0¶

Added the list of metrics on a metric’s detail page

0.4.0¶

Underscores in metric column headers are now replaced by spaces
The IndexUsageMetric now shows floating point percentages
Added IndexSizeMetric and TableSizeMetric
Added per metric record and record item styling

0.3.0¶

Added description to setup.py
The metric results can now be sorted on any column

0.2.0¶

Switched to Read The Docs theme for docs.
Added “Available Extensions” metric.
Fixed table styling in metric views. The tables now look more like Django Admin’s ChangeList

Welcome to django-postgres-metrics’s documentation!¶

Background. Why? What?¶

Getting Started¶

Installation¶

Designing Your Own Metric¶

Styling Metric Output¶

Styling Records¶

Styling Record Items¶

Contributing¶

Issuing a new Release¶

Reference Documentation¶

AppConfig¶

Metrics¶

Template Tags¶

URLs¶

Views¶

Changelog¶

0.5.0¶

0.4.0¶

0.3.0¶

0.2.0¶

0.1.1¶

0.1.0¶

Indices and tables¶