root labs rdist

November 22, 2011

File upload progress bars with jQuery, Nginx, and Django

Filed under: django,jquery,nginx,python — Murtaza Munaim @ 5:17 am
Tags:

Editor’s note: please welcome Murtaza to the blog. We’re looking forward to him and other team members posting their experiences.

Any web app that supports file uploads can benefit from progress bars. These give users a nice visual status on upload or other processing. When we added this feature to our code search product, we found a nice guide. However, it was last updated in 2008 and no longer worked as written. You should refer to that article for the background, and here’s how we handled it today.

The most straightforward approach to accessing file upload progress is to subclass the Django upload interface and collect stats for each chunk of data. This works fine with Django’s internal webserver. However, Nginx optimizes file uploads by sending them to Django only after they’re complete, so this approach won’t work in production.

The Nginx Progress Handler module is a third-party plugin that exports a REST API for the upload statistics of each file, referenced by a unique progress ID. After getting an ID for the file, we planned to send a file upload POST request to Nginx to begin the transfer. After that, our jQuery callback would query the progress handler module with a GET from http://example.com/progress with the appropriate X-Progress-ID header. However, this Nginx module didn’t work properly with Django and uWSGI.

In ordinary use, Nginx handles a user’s file upload request completely by itself and then passes the uploaded file data and relevant HTTP request meta-information to Django via the uwsgi, FastCGI, or similar protocols. At first, we were using uWSGI as the Nginx/Django interface. However, the Nginx progress handler module does not work with uWSGI. Its wiki entry reads:

WARNING: this directive must be the last directive of the location. It must be in a proxy_pass or fastcgi_pass location.

We decided to change to FastCGI. (We also configured FastCGI for threaded mode for better performance.) Once this was done, we added JavaScript to the upload page to POST the unique Progress ID, initialize the AJAX progress bar (js/jquery.progressbar.js), and start polling and updating the progress bar widget via a timeout function. The result sent back from the Nginx module’s GET method is a JSON string giving the number of bytes received so far, the total number of bytes in the file, and a string describing the state of the current upload.

Here is how we initialized the page by POSTing the user’s upload request and starting the polling function:

$(document).ready(function()
{
    var id = getID();
    $("#X-Progress-ID").val(id);
    var options = {
        dataType: "xml",
        url: "/upload?X-Progress-ID="+$("#X-Progress-ID").val(),
        beforeSubmit: initialize,
        success: finalize
    };
});

function initialize(formData, jqForm, options)
{
    $("#progress").fadeIn();
    $("#progress").progressBar();
    timeout = window.setInterval(updateProgressNginx, freq);
    return true;
};

The polling code then retrieves the number of bytes received so far, computes the fraction completed, and proceeds to the next page if the transfer is complete:

function updateProgressNginx()
{
    var id = $("#X-Progress-ID").val();
    $.ajax({
        url: "/progress",
        type: "GET",
        beforeSend: function(xhr) { xhr.setRequestHeader("X-Progress-ID", id); },
        success: updateNginx,
        async: false
    });
};

function updateNginx(responseText, statusText, xhr)
{
    data = JSON.parse(responseText);
    if (data.state == 'done' || data.state == 'uploading') {
        $("#progress").progressBar(Math.floor(100 * (data.received / data.size)));
    }
    if (data.state == 'done' || data.received >= data.size) {
        window.clearTimeout(timeout);
    }
};

We hope our experience here is useful to other developers facing the problem of out-of-date documentation for jQuery/Nginx/Django.

Edit (Oct 2012): we no longer recommend rolling your own this way. It’s better to use third-party widgets like jQuery File Upload or Plupload.

The Rubric Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 90 other followers