Getting Tilestream Working on Ubuntu 15.10 (Wily Werewolf)

This configuration is suitable for a simple tilestream appliance that only serves tiles. It can be quickly configured and uses node.js from chris-lea repositories.

The basic concept:

• install latest Ubuntu 15.10 iso (default settings, I included openssh only), reboot
• install nodejs sym linking it as the node binary
• install npm and git
• clone the mapbox git repo
• edit the package.json file to allow the latest version of connect
• npm install
• run

The basic concept is to link apt-get to the chris-lea repo to install node, download the git repo of the tilestream app from mapbox and adjust the package.json dependency to get a more modern version of connect. Then it should fire up ok.

This list should get you up and running...

$ sudo apt-add-repository ppa:chris-lea/node.js

$ sudo apt-get update

$ sudo apt-get -y install nodejs git-core

$ sudo ln -s /usr/bin/nodejs /usr/bin/node # node binary is the nodejs binary! 

$ sudo apt-get -y install npm # installs lots of dependencies

$ git clone https://github.com/mapbox/tilestream.git && cd tilestream

$ vim package.json # edit the connect version to latest (replace the connect version with a *)

$ npm install

$ ./index.js --host 10.0.0.34

Bam!!

Check out my similar install instructions for Ubuntu 14.04

Check out the config instructions on mapboxes github site

Installing Tilestream on Ubuntu 14.04 (2015 update)

• Install a fresh Ubuntu 14.04 64bit, include only openssh in the setup wizard,
• reboot,
• log in to guest and check the ip address then login from mac host via ssh (so we can copy paste following commands)
• do this as a normal user not root!

1. Follow github instructions from mapbox:

sudo apt-add-repository ppa:chris-lea/node.js

sudo apt-get update

sudo apt-get install nodejs

sudo apt-get install git-core

sudo apt-get update

git clone https://github.com/mapbox/tilestream.git

cd tilestream/ && npm install

And the result is (drum roll)...

./index.js will complain about the stringify module - fail

so to fix this....

    2. Update the dependencies section of the package.json file and allow the latest connect version

    "dependencies": {

        "jsdom":"https://github.com/springmeyer/jsdom-nogyp/tarball/c501ee8b71d8",

        "xmlhttprequest": "1.5.0",

        "backbone":"0.3.3",

        "jquery":"1.6.3",

        "express":"2.5.11",

        "connect":"*",

        "bones": "1.3.29",

        "step": "0.0.5",

        "tilelive": "4.5.3",

        "mbtiles": "0.2.9",

        "sphericalmercator": "1.0.2",

        "underscore": "1.6.0",

        "wax": "6.4.2",

        "modestmaps": "3.3.5"

    },

    3. You have to remove the current node_modules directory, reinstall using the new package config and re launch ./index.js:

charlton@tileserver:~/tilestream$ rm -fR node_modules/ && npm install && ./index.js --host 10.0.0.34
npm WARN package.json tilestream@1.1.0 'repositories' (plural) Not supported. Please pick one as the 'repository' field
npm WARN engine hoek@2.16.3: wanted: {"node":">=0.10.40"} (current: {"node":"0.10.37","npm":"1.4.28"})
npm WARN engine boom@2.10.1: wanted: {"node":">=0.10.40"} (current: {"node":"0.10.37","npm":"1.4.28"})
npm WARN engine cryptiles@2.0.5: wanted: {"node":">=0.10.40"} (current: {"node":"0.10.37","npm":"1.4.28"})
npm WARN deprecated set-immediate@0.1.1: Use `setimmediate` instead
> sqlite3@2.2.7 install /home/charlton/tilestream/node_modules/mbtiles/node_modules/sqlite3
> node-pre-gyp install --fallback-to-build
[sqlite3] Success: "/home/charlton/tilestream/node_modules/mbtiles/node_modules/sqlite3/lib/binding/node-v11-linux-x64/node_sqlite3.node" is installed via remote
sphericalmercator@1.0.2 node_modules/sphericalmercator
step@0.0.5 node_modules/step
xmlhttprequest@1.5.0 node_modules/xmlhttprequest
underscore@1.6.0 node_modules/underscore
jquery@1.6.3 node_modules/jquery
└── htmlparser@1.7.7
express@2.5.11 node_modules/express
├── qs@0.4.2
├── mime@1.2.4
├── mkdirp@0.3.0
└── connect@1.9.2 (formidable@1.0.17)
connect@3.4.0 node_modules/connect
├── utils-merge@1.0.0
├── parseurl@1.3.0
├── debug@2.2.0 (ms@0.7.1)
└── finalhandler@0.4.0 (escape-html@1.0.2, unpipe@1.0.0, on-finished@2.3.0)
backbone@0.3.3 node_modules/backbone
tilelive@4.5.3 node_modules/tilelive
└── optimist@0.6.1 (wordwrap@0.0.3, minimist@0.0.10)
bones@1.3.29 node_modules/bones
├── underscore@1.1.7
├── optimist@0.6.1 (wordwrap@0.0.3, minimist@0.0.10)
└── mirror@0.3.3 (uglify-js@1.0.2)
modestmaps@3.3.5 node_modules/modestmaps
wax@6.4.2 node_modules/wax
jsdom@0.8.3 node_modules/jsdom
├── cssom@0.2.5
├── nwmatcher@1.3.6
├── htmlparser2@3.8.3 (domelementtype@1.3.0, entities@1.0.0, domhandler@2.3.0, readable-stream@1.1.13, domutils@1.5.1)
├── cssstyle@0.2.30 (cssom@0.3.0)
└── request@2.65.0 (aws-sign2@0.6.0, forever-agent@0.6.1, caseless@0.11.0, stringstream@0.0.5, tunnel-agent@0.4.1, oauth-sign@0.8.0, isstream@0.1.2, json-stringify-safe@5.0.1, extend@3.0.0, node-uuid@1.4.7, qs@5.2.0, combined-stream@1.0.5, tough-cookie@2.2.1, mime-types@2.1.7, form-data@1.0.0-rc3, http-signature@0.11.0, bl@1.0.0, hawk@3.1.1, har-validator@2.0.2)
mbtiles@0.2.9 node_modules/mbtiles
├── underscore@1.5.2
└── sqlite3@2.2.7 (set-immediate@0.1.1, nan@1.1.2)
Creating tiles dir /home/charlton/Documents/MapBox/tiles
Started [Server Tile].
Started [Server Core:8888].

Bam!!!

If you have a firewall secretly hiding - don't forget to allow connections from other machines:

charlton@ubuntu:~/tilestream$ sudo ufw allow 8888

[sudo] password for charlton: 

Rules updated

Rules updated (v6)

History

I couldn't find anything about the stringify error anywhere because it is indirectly related but by trial and error I asked for the latest version of each module in the package.json file and found the outdated connect module was the culprit.

Here's the history for proof!

charlton@tileserver:~$ uname -a

Linux tileserver 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul 24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

charlton@tileserver:~/tilestream$ history

    1  ifconfig

    2  sudo reboot

    3  sudo apt-add-repository ppa:chris-lea/node.js

    4  sudo apt-get update

    5  sudo apt-get install nodejs nodejs-dev npm

    6  node -v

    7  sudo apt-get install nodejs

    8  sudo apt-get install git-core

    9  sudo apt-get update

   10  git clone https://github.com/mapbox/tilestream.git

   11  cd tilestream/

   12  npm install

   13  ./index.js 

   14  vim package.json 

   15  ifconfig

   16  rm -fR node_modules/ && npm install && ./index.js --host 10.0.0.34

   17  history


This can be shortened to :

sudo apt-add-repository ppa:chris-lea/node.js

sudo apt-get update

sudo apt-get -y install nodejs git-core

git clone https://github.com/mapbox/tilestream.git && cd tilestream

vim package.json # edit the connect line

npm install && ./index.js --host 10.0.0.34

Hope this saves some hell!!

GPX to postgres bash script

To import a gpx file into postgres do this:

# ogr2ogr -append -f PostgreSQL "PG:dbname=ctbto user=postgres" 2016114.gpx

cool


And a bash script would look like this (compliments of Mapbox from http://www.mapbox.com/guides/postgis-manual/):

files=(

        2016114.gpx

        2016115.gpx

)

for file in ${files[@]}; do

        ogr2ogr -append -f PostgreSQL "PG:dbname=ctbto user=postgres" $file

done


or

for file in /dir/*.gpx
do
     ogr2ogr -append -f PostgreSQL "PG:dbname=ctbto user=postgres" $file
done


Adding a column while importing:

ogrinfo -f PostgreSQL "PG:dbname=your_db user=xxxx password=yyyy" -sql "UPDATE tracks SET track_id = 1234 WHERE ogc_fid = (SELECT max( ogc_fid ) FROM tracks)"

(from: http://gis.stackexchange.com/questions/101825/add-gpx-files-to-postgresql-database-and-add-one-column-with-a-given-value)

Coverting Radio Mobile coverage charts for use with Tilemill and QGIS

Radio mobile can produce radio transmission coverage charts based on a set of particulars related to a transmitter and a receiver. Such a thing is good to theoretically predict signal coverage in various terrain. Since the program can use SRTM data for terrain backgrounds it is reasonably useful when visualized on a GIS. The files produced when creating a coverage plot can be in various formats, .png, .tif, .kml etc but no geo-referencing data is embedded in the file so it cannot be dropped into QGIS or Tilemill for instance over a geo-referenced map background. External files containing the georeferencing data are produced but in my case I needed the data embedded in a single file.

I use the following GDAL commands to translate, warp and embed metadata in the image file so that it can be dropped into QGIS or Tilemill. I used a .png as an output from Radio Mobile which has a native alpha transparency channel so that the white parts of the picture become transparent. You can play with the various color schemes to get a good visual result. For Tilemill I used WGS84.

The radio Mobile setup was:

Picture Properties

Radio Plot Properties

Using the results of a plot named Picture16.png which is 1680 pixels by 1024 and Picture16.geo that was created while saving the picture file, use the the following command sets geographic positions for each corner pixel starting at top left (-gcp 0 0) moving clockwise, and writes to a file called temp.tif:

# gdal_translate -of GTiff -a_ullr 17.07981 49.19989 21.52019 47.40011 -a_srs EPSG:4326 -expand rgba Picture16.png output.tif

I hope someone finds this useful as I messed around for some time getting it to work, I'm sure there are other ways to get it done and don't hesitate to point out any better ways.

Problems with rendering shp files created by qgis

When importing shp files into tilemill I noticed some parts of a layer where not rendering. I went back to qgis and removed any null fields present in the original shp file within qgis.

Bash Script for Bulk Processing DEM files using GDAL and Mapbox

Just making some notes on how to work with some DEM files and create hillshade, slopeshade, color relief and clip via shape files. All is done using GDAL libraries and creates a set of files that can be overlayed in Mapbox's tilemill and then converted to mbtiles to be served by tileserver.

This is my bash script file to burn up a map of Cyprus. If anyone is interested, leave a comment and I'll throw up where to get and install the libraries, ASTERS and the mapbox tools. The basic sequence is as follows:

1. merge all the ASTER DEM files to one big block covering all of Cyprus
2. convert the projection to mercator 
3. create hillshade from the DEM file, and clip
4. create the color relief, and clip
5. create slopeshade, and clip
6. merge the hillshade and color relief, and clip

Note each type of file created is clipped with a shape file of Cyprus which allows me to use any file to overlay in tilemill and serve with tileserver on Ubuntu.

This was done on a Mac but has been used on a linux box (fedora and ubuntu).

source ~/.bash_profile
echo merging multiple aster files ASTGTM2_N34E032_dem.tif  into file [cyprus-aster.tif] ...

gdal_merge.py -o cyprus-aster.tif ASTGTM2_N34E032_dem.tif ASTGTM2_N34E033_dem.tif ASTGTM2_N34E034_dem.tif ASTGTM2_N34E035_dem.tif ASTGTM2_N35E032_dem.tif ASTGTM2_N35E033_dem.tif ASTGTM2_N35E034_dem.tif ASTGTM2_N35E035_dem.tif 

gdalinfo -mm cyprus-aster.tif

echo warping cyprus-aster.tif from 4326 to 3785 into file [cyprus-aster-3785.tif]....

gdalwarp -s_srs EPSG:4326 -t_srs EPSG:3785 -r lanczos cyprus-aster.tif cyprus-aster-3785.tif

gdalinfo -mm cyprus-aster-3785.tif

########### HILLSHADE ######

echo creating hillshade of cyprus-aster-3785.tif into file [hillshade.tif]....

gdaldem hillshade cyprus-aster-3785.tif hillshade.tif -compute_edges

gdalinfo -mm hillshade.tif

echo clipping hillshade inot file [hillshade-clipped.tif]...

rm -f hillshade-clipped.tif

gdalwarp -co compress=deflate -dstnodata 255 -cutline c.shp hillshade.tif hillshade-clipped.tif

############# COLOR RELIEF #############

echo creating color-relief of cyprus-aster.tif into file [relief.tif]

gdaldem color-relief cyprus-aster-3785.tif ramp-dark-white-red-green-khaki.txt relief.tif

echo clipping relief....

rm -f relief-clipped.tif

gdalwarp -co compress=lzw -dstalpha -cutline c.shp relief.tif relief-clipped.tif

################# SLOPE #############

echo making a slope relief

gdaldem slope cyprus-aster-3785.tif slope.tif 

#echo clipping slope....

rm -f slope-clipped.tif

gdalwarp -co compress=deflate -dstnodata 255 -cutline c.shp slope.tif slope-clipped.tif

echo adding slope from slop-ramp.txt into file slopeshade.tif

gdaldem color-relief slope.tif slope-ramp.txt slopeshade.tif

#echo clipping slopeshade....

rm -f slopeshade-clipped.tif

#gdalwarp -co compress=deflate -dstnodata 255 -cutline c.shp slopeshade.tif slopeshade-clipped.tif

gdalwarp -co compress=lzw -dstalpha -cutline c.shp slopeshade.tif slopeshade-clipped.tif

########## RELIEF AND HILLSHADE ##############

echo merging color-relief and hillshade into file [relief-hillshade.tif]

./hsv_merge.py relief.tif hillshade.tif relief-hillshade.tif

########## CLIPPING ################

echo removing any old files before creating the final version

rm -f cyprus-clipped.tif

echo cutting out cyprus and finishing... cutting out relief-hillshade.tif with cyp.shp into file [cyprus-clipped.tif]

gdalwarp -co compress=deflate -dstnodata 255 -cutline c.shp relief-hillshade.tif cyprus-clipped.tif