[Tutorial] Shell script to upload Observations from Mesonet to SensorThings

Owned by Steve Liang
Last updated: Jul 06, 2016
2 min read

Before you start: Precondition

Before you start uploading Observations, you need to create the Thing, Datastream, ObservedProperty, etc.

Table of Content

Introduction

This post describes a very simple Shell script to upload data from Mesonet to SensorThings. Mesonet provides some very nice datasets. From the Mesonet web site:

Mesonet Description
The IEM maintains an ever growing archive of automated airport weather observations from around the world! These observations are typically called 'ASOS' or sometimes 'AWOS' sensors. A more generic term may be METAR data, which is a term that describes the format the data is transmitted as. If you don't get data for a request, please feel free to contact us for help. The IEM also has a one minute interval dataset for Iowa ASOS (2000-) and AWOS (1995-2011) sites. This archive simply provides the as-is collection of historical observations, very little quality control is done. "M" is used to denote missing data.

Data Source: Mesonet

I downloaded data from Iceland weather stations (http://mesonet.agron.iastate.edu/request/download.phtml?network=IS__ASOS), and the specific dataset I downloaded is this link.

After removing the first few lines of metadata, the data format looks like this:

BIAR	2015-01-21 04:00	-18.1002	65.6856	-9.00
BIAR	2015-01-21 05:00	-18.1002	65.6856	-9.00
BIAR	2015-01-21 06:00	-18.1002	65.6856	-11.00
BIAR	2015-01-21 07:00	-18.1002	65.6856	-10.00

And here is the Shell script I used to uploaded observations (don't forget to change the Datastream id and FeatureOfInterest id.

Shell Script

#!/bin/bash
while read f1 f2 f3 f4 f5 f6
  do 
     t="$f2"T"$f3":00Z\
     v=$f6
#     echo $t
#     echo $v
     s="{\"result\":\""$v"\",\"phenomenonTime\":\""$t"\",\"Datastream\":{\"@iot.id\":<id_of_Datastream>}, \"FeatureOfInterest\":{\"id\":<id_of_FeatureOfInterest>}}"
#     echo $s
     curl --request POST \
	     --data "${s}" \
	     --header "Content-Type: application/json" \
	 http://scratchpad.sensorup.com/OGCSensorThings/v1.0/Observations
done < file

Future Work

  • Mesonet in fact offers API for data download.  It would be nice to automatically download daily everyday and upload to a SensorThings API demo (e.g., SensorUp for Environmental Monitoring)??  And this GitHub link provides a python script for automatic download.