Saturday, July 30, 2011

Project: Basic Web Application List with Real-time Updating (Part 3)

Introduction

In this post, we will setup the basic list item addition and display. We will be covering basic App Engine techniques.

List Item Data Model

To be able to store our list item to the App Engine Datastore, we need to define a data model for the item. The data model is similar to a database model; we indicate the fields and data types. We write all the code for defining the data models in a separate script file, 'model.py', to make our project organized. Create the file, 'model.py' in the root directory of the project folder and put the following code.

from google.appengine.ext import db
from google.appengine.ext.db import polymodel


class Item(polymodel.PolyModel):
    title = db.StringProperty()
    bodytext = db.StringProperty(multiline=True)
    createdate = db.DateTimeProperty(auto_now_add=True)

The data model is very, very basic as our intention is to focus on experimenting with the Channel API. In addition, the above data model can easily be extended if needed.

Handlers to Save and Display Items

We go back now to 'main.py'. We modify it as follows.

#!/usr/bin/env python


from google.appengine.ext import webapp
from google.appengine.ext.webapp import util


import os
from google.appengine.ext.webapp import template
from django.utils import simplejson as json


from model import *


class MainHandler(webapp.RequestHandler):
    def get(self):
        path = os.path.join(os.path.dirname(__file__), 'templates/main.html')
        self.response.out.write(template.render(path, {}))


class SaveHandler(webapp.RequestHandler):
    def post(self):
        itemtitle = self.request.get('itemtitle')
        itemtext = self.request.get('itemtext')
        
        self.response.headers['Content-Type'] = 'text/json'
        
        if itemtext and itemtitle:
            item = Item(title=itemtitle, bodytext=itemtext)
            item.put()
            self.response.out.write(json.dumps({'result': 'success'}))
        else:
            self.response.out.write(json.dumps({'result': 'failure'}))
            
class ListHandler(webapp.RequestHandler):
    def get(self):
        items = Item.all()
        items.order("-createdate")
        
        d = {'rows': []}
        for item in items.fetch(20):
            d['rows'].append({'title': item.title, 'bodytext': item.bodytext})
                
        self.response.headers['Content-Type'] = 'text/json'
        self.response.out.write(json.dumps(d))


def main():
    application = webapp.WSGIApplication([('/', MainHandler),
                                         ('/save', SaveHandler),
                                         ('/list', ListHandler),],
                                         debug=True)
    util.run_wsgi_app(application)




if __name__ == '__main__':
    main()

Highlighted above are the changes. At the top we import all the objects from the 'model.py' script file to give as access to the data model we defined. We also added two request handlers, 'SaveHandler' and 'ListHandler'. The 'SaveHandler' saves a new list item. The 'ListHandler' returns, in JSON representation, the latest 20 items. Lastly, at the bottom, we make the two request handlers accessible through the '/save' and '/list' path.

Changes to main.html Template


We are done with the back-end and would now work on the front-end.

First let us modify the 'main.html' template as follows.

{% extends "base.html" %}


{% block content %}
<div id="itemform" title="Item Form">
  <form id ="itemformmain" action="/save" method="post">
    <ul>
      <li>
        <div class="formlabel">
          <label for="itemtitle">Title</label>
        </div>
        <div class="formwidget">
          <input name="itemtitle"
                 type="text"
                 placeholder="Place Title Here"
                 size="50"
                 required>
        </div>
      </li>
      <li>
        <div class="formlabel">
          <label for="itemtext">Main Text</label>
        </div>
        <div class="formwidget">
          <textarea name="itemtext"
                    cols="50"
                    rows="5"
                    required></textarea>
        </div>
      </li>
    </ul>
  </form>
</div>
<input type="button" name="additem" value="Add New Item"/>
<div id="mainlist">
</div>
{% endblock %}


{% block addscript %}
<script src="/js/main.js"></script>
{% endblock %}

In the content block, we added two div containers. The first div container is going to be used by Jqueryui to instantiate a dialog box.  The dialog box contains a form which will get the information for creating a new list item. The second div container is an empty div that would  contain the Evently widget for displaying the list of items. In between the two div containers is an 'Add New Item' button. This button will open the dialog.

At the bottom, we also added a new script tag inside the 'addscript' block. The script tag references a new Javascript file.

New Item Dialog

To enable the dialog box we defined, we need to do some Javascript. We will be putting all of our Javascript in 'main.js'. So create the 'main.js' file inside the 'js' folder and put the following code inside.

$(function () {
$("#itemform").dialog({
autoOpen: false,
modal: true,
width: 450,
buttons: {
"Save": function() {
var itemtitle = $("#itemformmain input[name='itemtitle']").val();
var itemtext = $("#itemformmain textarea[name='itemtext']").val();
var that = this;
$.post('/save',
      {itemtitle: itemtitle, itemtext: itemtext},
      function () {
      $(that).dialog("close");
      }, 'json');
}, 
"Cancel": function() { 
$(this).dialog("close"); 

}
});

$("input[name='additem']").bind('click',
function () {
$("#itemform").dialog("open");
}
);
});

The code above does two things:

  1. configure the dialog box
  2. and bind a function that opens the dialog box to the Click event of the "Add New Item" button.
In the event handler function for the 'Save' button, we simply make an AJAX POST request to '/save' which we defined to create a new item and save it to the datastore. The other parts of the code is quite simple and is self explanatory.

You can now try accesssing the site and adding new items. The only problem is that we still don't have any code to display the items added. For the meantime, you could just manually check the Datastore through the administrator interface, http://localhost:8080/_ah/admin/datastore.

Some Aesthetics

Notice that our dialog box looks very bad. Aesthetics is important so lets fix the styling a bit. In the 'style' directory, create the 'main.css' file and put the following code.

body {
font-size: 10px;
}

#itemform ul {
padding-left: 0;
}

#itemform li {
display: block;
}

#itemform .formlabel {
float: left;
text-align: right;
width: 60px;
}

#itemform .formwidget {
margin-left: 70px;
}

After creating the CSS file, we need to link it to our HTML page. We do this by inserting a 'link' tag in our 'base.html' template.

<!doctype html>
<html>
<head>
  <meta charset="utf-8"/>
  <title></title>

  <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0"/>
  <link type="text/css" href="/style/smoothness/jquery-ui-1.8.14.custom.css" rel="stylesheet" />
  <link type="text/css" href="/style/main.css" rel="stylesheet" />
</head>

<body>
  <header>
  {% block header %}{% endblock %}
  </header>

  <div id="contentwrapper">
  {% block content %}{% endblock %}
  </div>

  <footer>
  {% block footer %}{% endblock %}
  </footer>

  <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js"></script>
  <script src="/js/jquery.evently.js"></script>
  <script src="/js/jquery.mustache.js"></script>
  <script src="/js/jquery-ui-1.8.14.custom.min.js"></script>
  {% block addscript %}{% endblock %}
</body>

</html>

Highlighted above are the changes.

List Items Display

Now, lets add some Javascript to display the list items. Modify the 'main.js' file as follows.

$(function () {
$("#itemform").dialog({
autoOpen: false,
modal: true,
width: 450,
buttons: {
"Save": function() {
var itemtitle = $("#itemformmain input[name='itemtitle']").val();
var itemtext = $("#itemformmain textarea[name='itemtext']").val();
var that = this;
$.post('/save',
      {itemtitle: itemtitle, itemtext: itemtext},
      function () {
      $(that).dialog("close");
      }, 'json');
}, 
"Cancel": function() { 
$(this).dialog("close"); 
}
});
$("input[name='additem']").bind('click',
function () {
$("#itemform").dialog("open");
}
);

        $("#mainlist").evently({
_init: {
async: function (cb) {
$.getJSON('/list',
function (data) {
cb(data);
});
},
data: function (data) {
return {
items: data['rows']
};
},
mustache: '\
{{#items}}\
<div class="itementry">\
<h2>{{title}}</h2>\
<div>\
{{bodytext}}\
</div>\
</div>\
{{/items}}'
}
});
});

In the above code, we configured the 'mainlist' div container into an Evently Widget. We configured it to display the list items upon loading. The Evently Widget above responds to only one event, the special '_init' event, which is called automatically upon loading. Inside the '_init' object are three methods.

  • async - a function that makes an AJAX request to '/list'. '/list' returns a JSON representation of the 20 latest items.
  • data - callback function for the AJAX request in async. It does some pre-processing for the mustache template.
  • mustache - string template for displaying the object returned by the data method.
Assuming that you already added some items, opening the url, 'http://localhost:8080/' now displays the items in the datastore. However, adding a new list item, the new item is not displayed in the list. This because the widget only requests information upon loading.

Adding New Items to the List

To complete this post, let us finish by completing the basic functionality of displaying new items. The strategy used in displaying new items is chosen in such a way that it would be easy to integrate the Channel API functionality later on.

Modify the 'main.js' file as follows.

$(function () {
$("#itemform").dialog({
autoOpen: false,
modal: true,
width: 450,
buttons: {
"Save": function() {
var itemtitle = $("#itemformmain input[name='itemtitle']").val();
var itemtext = $("#itemformmain textarea[name='itemtext']").val();
var that = this;
$.post('/save',
      {itemtitle: itemtitle, itemtext: itemtext},
      function () {
                                        $("#mainlist").trigger('prependitem', {rows:[{title: itemtitle, bodytext: itemtext}]});
      $(that).dialog("close");
      }, 'json');
}, 
"Cancel": function() { 
$(this).dialog("close"); 
}
});
$("input[name='additem']").bind('click',
function () {
$("#itemform").dialog("open");
}
);

        $("#mainlist").evently({
_init: {
async: function (cb) {
$.getJSON('/list',
function (data) {
cb(data);
});
},
data: function (data) {
return {
items: data['rows']
};
},
mustache: '\
{{#items}}\
<div class="itementry">\
<h2>{{title}}</h2>\
<div>\
{{bodytext}}\
</div>\
</div>\
{{/items}}'
},
                prependitem: {
data: function (e, data) {
return {
items: data['rows']
};
},
mustache: '\
{{#items}}\
<div class="itementry">\
<h2>{{title}}</h2>\
<div>\
{{bodytext}}\
</div>\
</div>\
{{/items}}',
render: 'prepend'
}
});
});

In the above code, we added a 'prependitem' item custom event to the Evently widget. This event inserts at the start of the widget (why its prepend) items passed to the event. By now the 'data' method and 'mustache' property in the 'prepend' event is already quite self-explanatory. Hower, the 'render' property is new. The 'render' property indicates how the mustache template is added to the widget. Its default value is 'replace', meaning it replaces the contents of the widget. Other values are 'prepend' and 'append'.

Near the top of the code above, there is a statement that triggers the 'prependitem' event upon saving a new item. Upon triggering it also passes the information about the new item for display. We will be replacing this trigger statement later on when we integrate the Channel API functionality.

Next Post

In the next post, we will start working with the Channel API.

Thursday, July 28, 2011

Project: Basic Web Application List with Real-time Updating (Part 2)

In this post, I would be extending the boiler-plate app to integrate all of the client-side tools (jquery, jqueryui, evently, mustache.js) that we would be using.

Base Template

To integrate all the tools that we are going to use, we first need to create an html template which links to the  Javascript and CSS files. A convenient way of doing this in Django templates is to define a base template with all of the necessary linkages. The base template will then be extended by other templates to get all linkages.

First, let us create the 'templates' directory.

$ mkdir templates

Inside the 'templates' directory, create the 'base.html' file. Put the following code inside 'base.html'.

<!doctype html>
<html>
<head>
  <meta charset="utf-8"/>
  <title></title>

  <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0"/>
</head>

<body>
  <header>
  {% block header %}{% endblock %}
  </header>

  <div id="contentwrapper">
  {% block content %}{% endblock %}
  </div>

  <footer>
  {% block footer %}{% endblock %}
  </footer>
</body>

</html>

The 'base.html' file is the base Django template which would be extended by all our other templates. Highlighted above are the block areas wherein other templates can fill in their custom code.

Test the Base Template

To be able to test the 'base.html' template, let us make another template that extends it. Also in the 'templates' folder, create the 'main.html' file and put the following code.

{% extends "base.html" %}

{% block content %}
Hello World!!!
{% endblock %}

The above code fills in the defined content code block in 'base.html' with the "Hello World!!!".

Let us then create a request handler to display the 'main.html' template. We do this in 'main.py' in the root directory of our application. We edit the 'main.py' file to look as follows.

#!/usr/bin/env python

from google.appengine.ext import webapp
from google.appengine.ext.webapp import util

import os
from google.appengine.ext.webapp import template


class MainHandler(webapp.RequestHandler):
    def get(self):
        path = os.path.join(os.path.dirname(__file__), 'templates/main.html')
        self.response.out.write(template.render(path, {}))


def main():
    application = webapp.WSGIApplication([('/', MainHandler)],
                                         debug=True)
    util.run_wsgi_app(application)


if __name__ == '__main__':
    main()

Hightlighted in the code above are the changes.

Now running the web application and accessing http://localhost:8080/ would show the result of our changes.

Linking the Javascript Libraries

Now that the base template is working, its time to add integrate the Javascript libraries we would be using. To keep our project folder clean, lets create two folders, one to contain the Javascript files and another to contain the CSS files. Lets name this folders as 'js' and 'style'.

This two folders have to be able to serve static files, so we need to add some configuration directives to 'app.yaml' to enable them to do so.


application: basicrtlist
version: 1
runtime: python
api_version: 1

handlers:
- url: /style
  static_dir: style

- url: /js
  static_dir: js
 
- url: .*
  script: main.py

Now lets add the library files to the two directories. With regards to Jquery, lets just use the Google CDN, so we don't have to download any files. For Evently, download the files from its github page, https://github.com/jchris/evently. The important files for Evently are the following.


  • jquery.evently.js
  • jquery.mustache.js
Put the files above into the 'js' folder.


Note: the jquery.pathbinder.js file should not be included above

For Jqueryui, download the complete package for any theme. In my case, I prefer the smoothness theme. The important files for Jqueryui are the following.
  • jquery-ui-1.8.14.custom.min.js
  • the CSS and images directory
Copy the Javascript file to the 'js' folder and copy the CSS files and the images to the 'style' directory.


After setting up the files, lets modify the 'base.html' to link to all this libraries.

<!doctype html>
<html>
<head>
  <meta charset="utf-8"/>
  <title></title>

  <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0"/>
  <link type="text/css" href="/style/smoothness/jquery-ui-1.8.14.custom.css" rel="stylesheet" />
</head>

<body>
  <header>
  {% block header %}{% endblock %}
  </header>

  <div id="contentwrapper">
  {% block content %}{% endblock %}
  </div>

  <footer>
  {% block footer %}{% endblock %}
  </footer>

  <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js"></script>
  <script src="/js/jquery.evently.js"></script>
  <script src="/js/jquery.mustache.js"></script>
  <script src="/js/jquery-ui-1.8.14.custom.min.js"></script>
  {% block addscript %}{% endblock %}
</body>

</html>

Highlighted above are the necessary changes to link to the libraries. I also added a new block to allow the insertion of some additional Javascript files.


Next Post

That's all for this post. In the next post, we'll start writing code.

Monday, July 25, 2011

Project: Basic Web Application List with Real-time Updating (Part 1)

Introduction

In the past week, I've been experimenting with the Google App Engine (GAE) Channel API. The GAE Channel API allows a client web application to open a persistent connection to allow the server to push information to the client. This allows the real-time updating of the client information without the use of polling, which is very inefficient. The Channel API is very similar to html5 websockets.

To learn about the Channel API, I have decided to make a simple web application that displays a list of messages posted by its users. All of the users sees the same listing of messages. The difference with this simple web application is that the listing of messages is updated, almost instantly or with minimal delay, every time a message is added.  This sort of real-time updating is to be accomplished with the use of the Channel API.

This functionality can also be accomplished by configuring the client to periodically check the server for new messages or polling the server. However, this mechanism would entail a significant overhead and is quite inefficient.

Web Application Stack

To implement this simple web application, the following libraries/technologies were chosen.

  • Google App Engine (Python)
  • Jquery
  • Jquery UI
  • Evently
  • Mustache Templates (Javascript)
A surprising addition to the technologies above is Evently. Evently is a Jquery plug-in which makes the writing of event-based Javascript in a more structured, declarative manner. I learned Evently when I worked on the PEPBrowser Couchapp application. I really think that its paradigm is very sound and would make the Javascript part of the web application more structured and a lot cleaner.

Set-up the GAE Python SDK and the Boiler-Plate App

In this set-up, I am assuming that you are using a Unix-based operating system such as Linux or OS X and that you already have atleast Python 2.5 installed, which is usually the default case in current versions of Linux and OS X.

First download the 'Linux/other platforms' version of the Python SDK from this page. Even if you are using OS X, I still suggest that you use the 'Linux/other platforms' version. After downloading the SDK, unzip the downloaded file into a directory.

$ unzip google_appengine_1.5.2.zip

This will create the 'google_appengine' directory in the current working directory.

$ cd google_appengine
$ cp -r new_project_template/ basicrtlist

Then go into the 'google_appengine' directory and copy the 'new_project_template' folder into a new folder 'basicrtlist'. The 'basicrtlist' folder is now our web application project. It already contains some boiler-plate code which we need to modify.


The boiler-plate web application project contains only three files.
  • app.yaml - the application configuration file
  • index.yaml - the datastore index configuration file
  • main.py - the main Python source file
To complete the boiler-plate web application, we need to make some modifications in app.yaml. We need to change the application name from 'new-project-template' to 'basicrtlist'.

application: basicrtlist
version: 1
runtime: python
api_version: 1

handlers:
- url: .*
  script: main.py

After making the changes, we could try running our barebones web application.

$ ./dev_appserver.py basicrtlist/

We could access the output of the web application by accessing, http://localhost:8080, using a web browser.

Next Post

In the next post, I would extend the basic boiler-plate application and integrate all the tools we would be using. The code for this application could be downloaded from here.

Sunday, July 17, 2011

PEPBrowser: Next Evently Widget (Part II)

Resume Work on the 'pepstatuslist' Widget

Continuing from the previous blog post, lets start improving the 'pepstatuslist' widget. The previous blog post ended with the addition of the some code that enables the 'peplist' widget to trigger a 'setcategory' event when a category is chosen. Lets start adding the code that will listen to the 'setcategory' event.

Inside the 'pepstatuslist' folder, make a script file with the name, 'script.js'. The 'script.js' should contain the following code.


function (e, category) {
  $$(this).category = category;
  $(this).trigger('loadstatuslist');
}

The function in the code above is called when the 'pepstatuslist' widget receives a 'setcategory' event generated by the 'peplist' widget. The function accepts two arguments, the event object and the category value. The category value corresponds to the category clicked/tapped in the 'peplist' widget. The first statement of the function stores a name-value pair to the widget. This is done by the use of the $$ operator. The second statement of the function triggers a 'loadstatuslist' event.

loadstatuslist Handler

The 'loadstatuslist' event handler should load the list of document statuses under the chosen category. Since this event handler is not as simple as the 'setcategory' event handler, we need to make a folder for this event handler. So create a folder inside the 'pepstatuslist' folder with the name, 'loadstatuslist'. Inside the new folder add the javascript file, 'query.js'. The script file should contain the following code.

function () {
  var category = $$(this).category;
  return {
    view: "categorystatus",
    group_level: 2,
    "startkey": [category],
    "endkey": [category, {}]
  };
}

The 'query.js' file contains a function which returns a mapping of query parameters. Remember that in the '_init' event handler of the 'peplist' widget, it contains a 'query.json' file. The 'query.json' contains a mapping literal or JSON string that indicates the query parameters for the event handler. The 'query.json' file has the same purpose as the 'query.js' script. The reason why we use a script is that the query parameters is dynamic, it depends on the value of 'category' name-value pair assigned to the widget through the 'setcategory' event.

Honestly, the above mapping is not that simple and trying to explain it would take quite a long time. If you don't understand the mapping above, I advise you to consult the Views chapter of the "Definitive Guide to CouchDB".

Since, the other parts of the 'loadstatuslist' handler is quite similar to the '_init' handler of the 'peplist' widget. I'll just breeze through the other parts.

The code for 'data.js' follows.


function (data) {
  var l = [];
  for (var i=0; i < data.rows.length; i++) {
    l.push({'category': data.rows[i].key[0], 'status': data.rows[i].key[1], 'count': data.rows[i].value});
  }
  return {'statuslist': l};
};

The code in 'data.js' simply processes the data returned by the query. The function returns a mapping suitable for templating.

The Mustache template follows.

<ul id="pepstatuslist" data-role="listview" data-theme="c" data-dividertheme="b">
  {{#statuslist}}
  <li><a href="#pepdoclist" title="{{category}}|{{status}}">{{status}} ({{count}})</a></li>
  {{/statuslist}}
</ul>

The template simply displays the statuses in a clickable list. It is intended to be clickable so that the user can tap on a status and lead to a list of documents.

Finally, to activate the unordered list generated by the template above into a Jquerymobile list, the following code in 'after.js' is necessary to be executed.

function () {
$('#pepstatuslist').listview();
};



So far we have discussed the common pattern used for the rest of this application. I think that you would be able to understand the rest of the code available in the github repository.