Install Open Spending ===================== Installation and setup ---------------------- Steps to set up Open Spending/WDMMG from scratch. This may not be comprehensive. Please edit. Use a virtualenv! ''''''''''''''''' For an installation we recommend highly to use a virtualenv_ environment. The following documentation assumes you do and have activated the virtual environment in `/path/to/env/`. If not create one with:: $ virtualenv --no-site-packages /path/to/env Now activate the environment. Your prompt will be prefixed with the name of the environment:: $ cd /path/to/env /path/to/env$ . /path/to/env/bin/activate (env)/path/to/env$ .. _virtualenv: http://pypi.python.org/pypi/virtualenv Install Mongodb ''''''''''''''' Install `mongodb`_ on your machine. A version >= 1.5.3 is required (for $or queries). .. _mongodb: http://www.mongodb.org/ Check out and install Open Spending and related packages '''''''''''''''''''''''''''''''''''''''''''''''''''''''' Check out wdmmg from the mercurial repository:: (env)/path/to/env $ hg clone http://bitbucket.org/okfn/wdmmg This will create a directory called wdmmg. Inside you will find a pretty standard Pylons application. Change into the directry and install 'wdmmg' and it's dependencies using pip.:: (env)/path/to/env$ cd wdmmg (env)/path/to/env/wdmmg$ pip install -r pip-requirements.txt (env)/path/to/env/wdmmg$ pip install -e . For several things like importing data you need the `wdmmg-ext` package. Unless you install `wdmmg` for production you should install it. (env)/path/to/env/wdmmg $ pip install -e \ > hg+ssh://hg@bitbucket.org/okfn/wdmmg-ext#egg=wdmmg-ext Finally checkout the main wdmmg theme from the mercurial repository into `wdmmg/public/themes/wdmmg`:: (env)/path/to/env/wdmmg$ hg clone http://bitbucket.org/okfn/wdmmg-wp-theme \ > public/themes/wdmmg Create and edit a paster configuration '''''''''''''''''''''''''''''''''''''' Create a development.ini file. This has the config options needed to run the wdmmg:: (env)/path/to/env/wdmmg$ paster make-config wdmmg development.ini Edit development.ini with relevant details for your local machine. The options in the file are commented. Some of the important options in `[app:main]` are:: mongodb.database = wdmmg_dev extra_public_paths = path/to/extra/public/resources /another/extra/public/resource/from/plugin # setup plugins wdmmg.plugins = treemap datatables etc. # Credentials for retrieving data from Google Documents. gdocs_username = gdocs_password = Setup Solr '''''''''' Install Solr and create a solr configuration home directory to use with Open Spending, e.g. copy the example folder of solr installation. Remove schema.xml file and symlink solr/wdmmg_schema.xml from the wdmmg package instead. Start solr with the folder as a parameter. In your development.ini you have to configure the solr url, typically http://localhost:8983/solr :: solr.url = http://localhost:8983/solr Setup Celery '''''''''''' Celery_ is used to manage job queues for background tasks. It is installed as automatically as a dependency of wdmmg. You can find several celery* commands in your virtualenv's /bin directry. Adapt celeryconfig.py to your needs. The default configuration `wdmmg/celeryconfig.py` uses mongodb as a storage and queue backend. Run Celery with:: (env)/path/to/env/wdmmg$ celeryd celeryconfig.py .. _Celery: http://celeryproject.org/ Test the installation --------------------- Run tests, edit anything that is broken.:: (env)/path/to/env/wdmmg$ nosetests Import data into Open Spending ------------------------------ To import data into OpenSpending you need a data :term:`loader`. The Open Spending project ships a set of loaders in the package :mod:`wdmmg-ext` that is required for these steps. These loaders load a :term:`dataset` into the database. The installation of `wdmmg-ext` is described in `Check out and install Open Spending and related packages`_. Loading a big dataset can take a long time. Load a complete dataset ''''''''''''''''''''''' To load a :term:`dataset` you have to first download it. With the installation of `wdmmg` a script `datapkg` was automatically generated in your virtualenv's bin directory. Your development.ini file defines a `getdata_cache` directory. The default is `./pylons_data/getdata` inside the wdmmg package directory. We will now downlad the "cra" data package to that directory:: (env)/path/to/env/wdmmg$ datapkg download ckan://ukgov-finances-cra \ > ./pylons_data/getdata Now you can load the `cra2010` `dataset` into the database.:: (env)/path/to/env/wdmmg$ paster load cra2010 After that you want to update the solr index. We provide a paster command for that:: (env)/path/to/env/wdmmg$ paster solr load cra2010 Load sample data '''''''''''''''' Alternatively you can load a set of sample data and update the solr index for it. Be aware that this will empty the database first.:: (env)/path/to/env/wdmmg$ paster fixtures setup (env)/path/to/env/wdmmg$ paster solr load cofog (env)/path/to/env/wdmmg$ paster solr load cra Run the site ------------ Finally, run the site from development.ini:: (env)/path/to/env/wdmmg$ paster serve --reload development.ini How to upgrade production service --------------------------------- 3 dbs/systems: * data.wheredoesmymoneygo.org - P1 * data.wheredoesmymoneygo.org.2 - P2 * data.staging.wheredoesmymoneygo.org - S1 Suppose P1 = active production, P2 = inactive production * Shut down write to the main system (Hack way: LimitExcept GET in Apache) * Dump active production db and load into inactive production db * Upgrade inactive system and set to use inactive production db * Use it and test it * Switch over from P1 to P2 * In apache move wsgi script to point to the other one and reboot apache2 * If this fails just switch back and you are operational again