A step-by-step tutorial
=======================

Introduction
------------

This step-by-step tutorial will make you create, document and use webservices
on a basic TurboGears application with TGWebServices.

The services we will create allow a third-party application to do a few
operations on the user definitions, so it can, for example, synchronise
the user definitions with its own one.

Preresquites
------------

* Python : This tutorial was tested with python 2.5 on a machine with
  multiple versions of python installed, so when needed the
  interpreter is explicitely given in the command lines so that another
  version does not interfere.

* setuptools (or distribute) for the version of python you will use (preferably
  python 2.5).

* virtualenv for the version of python you will use::

      easy_install-2.5 virtualenv

Initialise the environment
--------------------------

#. Choose a workspace directory name, for example ~/workspace or C:\workspace
   (we assert you use one of these in this tutorial) and create it.

#. Create the virtualenv and activate it :

   * Unix:

     .. code-block:: bash

         mkdir ~/workspace
         cd ~/workspace
         virtualenv -p python2.5 tutoenv
         source tutoenv/bin/activate

   * Windows

     .. code-block:: bat

         c:
         mdir c:\workspace
         cd \workspace
         C:\Python25\Scripts\virtualenv tutoenv
         tutoenv\Scripts\activate.bat

   Your shell prompt should now look like :

   .. code-block:: bash

        (tutoenv)cdevienne@tryo:~/workspace$

#. Install TurboGears >= 1.1, < 1.1.99:

   .. code-block:: bash

        easy_install -f http://files.turbogears.org/eggs/ "TurboGears >= 1.1, < 1.1.99"

Create the project
------------------

#. Quickstart a TG project named WSTuto with identity activated

   .. code-block:: bash

       tg-admin quickstart --sqlalchemy --identity WSTuto -p wstuto
       cd WSTuto


   .. note::

       From now, for all of the commands we assume the current directory is
       your project root, WSTuto

#. Add the required dependencies in setup.py (around line 57), and also
   specify where to look for the packages::

       install_requires=[
           "TurboGears >= 1.1.1",
           "WebTest",
           "SQLAlchemy >= 0.6.3",
           "TGWebServices >= 1.2.4rc1, < 1.2.99",
           "Sphinx >= 0.6.5, < 0.6.99",
       ],
       dependency_links=['http://files.turbogears.org/eggs/'],

   .. note::

       We depend on Sphinx only to make this tutorial easier to execute. It is
       not really a dependency of your application, but more a tool you need
       when you work on the documentation that you can install manually.


#. Make sure your application has all its dependencies:

   .. code-block:: bash

       python setup.py develop

#. Create your database

   .. code-block:: bash
   
       tg-admin sql create

#. You should now be able to start your turbogears application:

   .. code-block:: bash

       start-wstuto

   And one of the last line of logs on your console should soon be::

       2010-08-25 22:52:31,397 cherrypy.msg INFO HTTP: Serving HTTP on http://0.0.0.0:8080/


   Point your brower on http://localhost:8080/ to see what it looks like for a minute !

Create the webservices controllers
----------------------------------

#. First we need to add a webservices root controller somewhere in
   our controllers tree.
   
   Adding it as "/ws" or "/api" is generally a good idea. Here we'll use
   "/ws".

   Another thing we can determine now is the soap-namespace we will use
   in the wsdl for types and xxx. Here we will use http://wstuto.com/ and
   http://wstuto.com/types/.

   * Create a new directory in wstuto
     for storing all the webservice controllers and name
     it "ws":

     .. code-block:: bash

         mkdir wstuto/ws

   * Add a new file called "__init__.py" in the wstuto/ws/ directory,
     with the following content::

         import logging

         from turbogears import config
         from tgwebservices.controllers import WebServicesRoot, wsexpose, wsvalidate

         log = logging.getLogger(__name__)

         class WSController(WebServicesRoot):
             def __init__(self):
                super(WSController, self).__init__('http://localhost:%s/ws/' % config.get('server.socket_port'),
                                                   'http://wstuto.com/',
                                                   'http://wstuto.com/types/')

   * In wstuto/controllers.py, import your new controller and mount
     it::

         # After all the other imports
         from wstuto.ws import WSController

         # ...
         
         # In the root controller
         class Root(controllers.RootController):
             """The root controller of the application."""

             ws = WSController()

             # ...

   We now have a "/ws" controller that is the root of all the webservices we
   will expose.

   You can already see the wsdl definition of your empty services at
   http://localhost:8080/ws/soap/api.wsdl.

#. Now we will create a dedicated controller for the user management
   functions, so that all the related calls we be done on URLs starting
   with "/ws/user".

   * Create a file wstuto/ws/user.py with the following content::

      # coding: utf-8

      import logging

      from tgwebservices.controllers import WebServicesController, wsexpose, wsvalidate

      log = logging.getLogger(__name__)

      class UserWSController(WebServicesController):
         pass

   * Mount this controller on your webservice root by adding the following code
     to wstuto/ws/__init__.py::

        # After all the other imports
        from wstuto.ws.user import UserWSController

        # ...

        # In the webservice root controller definition :
        class WSController(WebServicesRoot):
            user = UserWSController()

            # ...

   For now the webservice still does nothing, but our controllers tree is
   ready.

Define the apis
---------------

Time to think a bit !

The apis we want to provide should allow a third-party application to
manager our users, groups and permissions.

Custom types
~~~~~~~~~~~~

First we will need to define
complex type to manipulate the user and group definitions at least.

Also, we will certainly manipulate user or group references. Since they
have integer ids in the database, we can directly use integers. But they
will be untyped references. To make things more strict, we will create
custom types for references.

Most importantly, we will document the types right away, so that sphinx
can extract the docstrings for the final documentation.

Add the following code to wstuto/ws/types.py::

    import datetime

    class UserRef(object):
        """A user reference"""

        #: The user unique id
        id = int

        def __init__(self, obj=None):
            self.id = None
            if obj:
                self.id = obj.user_id

    class User(object):
        """A user data"""

        #: The user unique id
        id = int

        #: The user name, used for login.
        name = unicode

        #: The user email address
        email_address = unicode

        #: The user display name
        display_name = unicode

        #: When was this user created
        created = datetime.datetime

        def __init__(self, obj=None):
            self.id = None
            self.name = None
            self.email_address = None
            self.display_name = None
            self.created = None
            if obj:
                self.id = obj.user_id
                self.name = obj.user_name
                self.email_address = obj.email_address
                self.display_name = obj.display_name

    class UserList(object):
        totalcount = int
        offset = int
        users = [User]

        def __init__(self, **kw):
            self.totalcount = kw.get('totalcount')
            self.offset = kw.get('offset')
            self.users = kw.get('users')

    class GroupRef(object):
        """A user group reference"""

        #: The group unique id
        id = int

        def __init__(self, id=None):
            self.id = id

    class Group(object):
        """A user group data"""

        #: The group unique id
        id = int

        #: The group name
        name = unicode

        #: The group display name
        display_name = unicode

        #: When was this group created
        created = datetime.datetime

        def __init__(self):
            self.id = None
            self.name = None
            self.display_name = None
            self.created = None

    class PermissionRef(object):
        """A permission reference"""

        #: The permission unique id
        id = int

        def __init__(self, id=None):
            self.id = id

    class Permission(object):
        """A permission datas"""

        #: The permission unique id
        id = int

        #: The permission name
        name = unicode

        #: The permission description
        description = unicode

        def __init__(self):
            self.id = None
            self.name = None
            self.description = None

.. note::

    It is possible to auto-create these types from the sqlalchemy mapped
    classes. This is not the aim of this tutorial though.

The functions
~~~~~~~~~~~~~

Although we created custom types for user, group and permission handling,
we will concentrate on the user related functions only.

What do we need to provide ? In one 'word': CRUD (Create, Read, Update
and Delete).

This can be done by a few functions:

* 'create' : will create a user, given some datas.

* 'list' : will list all the users, with some pagination to avoid too big
  results to be returned.

* 'query' : will list the users filtered with a few simple criterions.

* 'update' : will update a user datas, given some datas of course.

* 'delete' : will delete a user, given a reference.

In addition, we can have functions to (re)set a user password:

* 'set_password': will change a user password, given the old one.

* 'reset_password': will reset the password.

Let's translate these ideas into tgws definitions. All we need to do is
to expose functions on the `UserWSController` in wstuto/ws/user.py.
We'll also import the needed types from wstuto.ws.types, and
document immediately the api:

.. code-block:: python

    # after all the other imports:
    from wstuto.ws.types import User, UserRef, UserList


    # complete the '/ws/user' controller :
    class UserWSController(WebServicesController):
        @wsexpose(User)
        @wsvalidate(User)
        def create(self, datas):
            """Create a new user.
            
            :param datas: The initial datas of the user. The will be ignored.
            """
            pass

        @wsexpose(UserList)
        @wsvalidate(int, int)
        def list(self, limit=20, offset=0):
            """List all the users, with pagination.

            :param limit: How many users should be returned at most.
            :param offset: From which user should the list start.
            """
            pass

        @wsexpose(UserList)
        @wsvalidate(str, str, str, int, int)
        def query(self, user_name=None, email_address=None, display_name=None,
                  limit=20, offset=0):
            """Returns the users having the given values in their datas.
            The applied operator is "in".

            :param user_name: If non-null, a string that should be found in
                              the user name.
            :param email_address: If non-null, a string that should be found in
                                  the user email address.
            :param display_name: If non-null, a string that should be found in
                                 the user display name.
            """
            pass

        @wsexpose(User)
        @wsvalidate(User)
        def update(self, data):
            """Update a user datas and returns its new datas
            
            :param data: The new values. null values will be ignored. The id
                         must exist in the database.
            """
            pass

        @wsexpose()
        @wsvalidate(UserRef)
        def delete(self, ref):
            """Delete a user.

            :param ref: An existing user reference.
            """
            pass

        @wsexpose()
        @wsvalidate(UserRef, str, str)
        def change_password(self, ref, old_password, new_password):
            """Change a user password.

            :param ref: The reference of the user.
            :param old_password: The user old password.
            :param new_password: The user new password.
            """
            pass

        @wsexpose()
        @wsvalidate(UserRef, str)
        def reset_password(self, ref, password):
            """Reset a user password.

            :param ref: The reference of the user.
            :param password: The new password.
            """
            pass

Have another look to http://localhost:8080/ws/soap/api.wsdl, you should see
a much more complex xml content than before that describe your types and
functions.

Good news, the most difficult part is done ! Now that we properly defined
the types, the functions and the documentation of our web services, the 
next steps are only technical formalities.

Documents
---------

The documentation is actually already written in the code. All we need to 
do is to properly create a sphinx documentation project, install the
tgwsdoc extension and write few directives to reference our code.

Create the doc
~~~~~~~~~~~~~~

.. code-block:: bash

    mkdir doc
    cd doc
    sphinx-quickstart

Sphinx quickstart will ask a few questions. Leave the defaults answers
when there are some. For the others:
 
* Project name: WSTuto

* Author name(s): Your Name Here

* Project version: 1.0

* Install the 'autodoc' extension.

Modify a bit the html target in the generated Makefile (at line 33):

.. code-block:: make

    html:
    	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) ../wstuto/static/doc
    	@echo
    	@echo "Build finished. The HTML pages are in ../wstuto/static/doc"

and make.bat (line 36) if your using windows:

.. code-block:: bat

	if "%1" == "html" (
	    %SPHINXBUILD% -b html %ALLSPHINXOPTS% ../wstuto/static/doc
	    echo.
	    echo.Build finished. The HTML pages are in ../wstuto/static/doc.
	    goto end
	)


Now build this quickstarted project to make sure its working properly:

.. code-block:: bash

    cd doc
    make html

Have a look at http://localhost:8080/static/doc/index.html, and admire your
new (empty) embedded documentation.

Install the extension
~~~~~~~~~~~~~~~~~~~~~

In conf.py:

* Add "tgwsdoc" to the extensions lists

* Add "." to sys.path.

* Add the following line somewhere::

    html_style = 'wstuto.css'

Prepare a few files :

* Download http://tgws.googlecode.com/hg/contrib/tgwsdoc/tgwsdoc.py to the doc
  directory

* Download http://tgws.googlecode.com/hg/contrib/tgwsdoc/toggle.css and 
  http://tgws.googlecode.com/hg/contrib/tgwsdoc/toggle.js to doc/_static.

* Create the file doc/_static/wstuto.css with the following content:

  .. code-block:: css

      @import 'default.css';
      @import 'toggle.css';

For more details, see :mod:`tgwsdoc` (Installing the extension).

.. note::

    The tgwsdoc extension will be more easy to be installed when ported to
    Sphinx 1.0.

Write the ws documentation
~~~~~~~~~~~~~~~~~~~~~~~~~~

Add a doc/ws.rst file, in which we'll document our application webservices:

.. code-block:: rest

    WebServices
    ===========

    Our great application provides webservices thanks to TGWebServices.

    Types
    -----

    .. autotgwstype:: wstuto.ws.types.UserRef

    .. autotgwstype:: wstuto.ws.types.User

    .. autotgwstype:: wstuto.ws.types.UserList

    .. autotgwstype:: wstuto.ws.types.GroupRef

    .. autotgwstype:: wstuto.ws.types.Group

    .. autotgwstype:: wstuto.ws.types.PermissionRef

    .. autotgwstype:: wstuto.ws.types.Permission

    Functions
    ---------

    .. tgwsrootcontroller:: wstuto.ws.WSController

    .. autotgwscontroller:: wstuto.ws.user.UserWSController
        :members:

Now add this file to your index.rst toctree:

.. code-block:: rest

    .. toctree::
        :maxdepth: 2

        ws

That's it !

Generate the documentation
~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: bash

    cd doc
    make html

If everything goes well, you will obtain a nice documentation of your
web services at http://localhost:8080/static/doc/ws.html.

Implements
----------

We have a lot of functions defined, but we will concentrate on 2 of them :
create, and change_password. The other ones are let as an exercise.

.. code-block:: python

    from turbogears.database import session
    from wstuto import model

    #...
    class UserWSController(Controller):

        @wsexpose(User)
        @wsvalidate(User)
        def create(self, datas):
            """Create a new user.
            
            :param datas: The initial datas of the user. The will be ignored.
            """
            user = model.User()
            user.user_name = datas.name
            user.email_address = datas.email_address
            user.display_name = datas.display_name
            session.add(user)
            session.flush()
            return User(user)

        # ...
        @wsexpose()
        @wsvalidate(UserRef, str)
        def reset_password(self, ref, password):
            """Reset a user password.

            :param ref: The reference of the user.
            :param password: The new password.
            """
            user = session.query(model.User).get(ref.id)
            user.password = password
            session.flush()


Security consideration
----------------------

The webservices we just wrote are accessible to anybody on the network, which
might be a security problem.

To address this problem, the solution is very easy, we just need to use
what TurboGears provides, identity.

Let's say we add two permissions:  "userread" for read-only functions,
and "userwrite" for all modifying functions. Give this permission to a
user, and protect your functions with the identity.requires decorator.

For example, the create and query functions definitions will be:

.. code-block:: python

    @wsexpose(User)
    @wsvalidate(User)
    @identity.requires(identity.has_permission("userwrite"))
    def create(self, datas):
        """Create a new user
        """

    @wsexpose(UserList)
    @wsvalidate(str, str, str, int, int)
    @identity.requires(identity.has_permission("userread"))
    def query(self, user_name=None, email_address=None, display_name=None,
              limit=20, offset=0):
        """Returns the users having the given values in their datas.
        """

For more information on this aspect, please refer to the TurboGears
documentation.

Client-side
-----------

Now that we have web services, we will see how they can be accessed with
different protocols and languages.

First you will obviously need to run your application:

.. code-block:: bash

    start-wstuto
    ...
    Listening on http://localhost:8080/

Accessing the web services
~~~~~~~~~~~~~~~~~~~~~~~~~~

REST+XML
''''''''

The functions are callable doing a GET or POST to ws/user/<functionname>,
giving parameters.

For example, for calling user/create, post this code to /ws/user/create:

.. code-block:: xml

    <parameters>
        <datas>
            <name>newuser</name>
            <email_address>newuser@example.com</email_address>
            <display_name>New User</display_name>
        </datas>
    </parameters>

REST+JSON
'''''''''

Same as REST+XML, except that json is used instead of xml. The post or get
should add a "Accept = application/json" header to the request.

For example, to call user/delete, post this code to /ws/user/delete:

.. code-block:: javascript

    {
        'ref': {
            'id': 4
        }
    }

SOAP
''''

A wsdl file is available at /ws/soap/api.wsdl. Any decent soap client will
be able to use this file to auto-create a mapping in the target language.

The functions names will be: userCreate, userQuery etc.

Examples in python
~~~~~~~~~~~~~~~~~~

REST+JSON
'''''''''

.. code-block:: python

    import simplejson
    import liburl2

    url = 'http://localhost:8080/ws/'

    def call(fname, params):
        req = urllib2.Request(url=url+fname, data=simplejson.dumps(params))

        # If authentication is required, the easiest way is to use http 
        # authorization :
        #
        # req.add_header('Authorization',
        #                'Basic %s' % base64.encodestring(
        #                    '%s:%s' % (username, password))[:-1])

        # Set the type of result we want
        req.add_header('Accept', 'text/javascript')

        # Set the type of data we send
        req.add_header('Content-Type', 'application/json')

        # Call the function
        response = urllib2.urlopen(req)
        return simplejson.loads(response)

    call('user/create',
         {
            'datas': {
                'name': 'newuser',
                'email_address': 'newuser@example.com',
                'display_name': 'New User'
            }
         })

SOAP
''''

With suds:

.. code-block:: python

    import datetime

    from suds.client import Client
    import suds

    url = 'http://localhost:8080/ws/soap/api.wsdl'
    client = Client(url, cache=None)

    userdata = client.factory.create('{http://wstuto.com/types/}User')
    userdata.id = 0
    userdata.created = datetime.now()
    userdata.name = 'newuser'
    userdata.email_address = 'newuser@example.com'
    userdata.display_name = 'New User'

    userdata = client.service.userCreate(userdata)

    print "The new user id is:", userdata.id

    userref = client.factory.create('{http://wstuto.com/types/}UserRef')
    userref.id = userdata.id

    client.service.userReset_password(userref, 'test')