Our full technical support staff does not monitor this forum. If you need assistance from a member of our staff, please submit your question from the Ask a Question page.


Log in or register to post/reply in the forum.

CRbasic: can I retrieve data from a remote server with an API request?


DaveMattey Jun 8, 2020 01:47 AM

Slight change of  question - this answer to  above is of course yes, but I am on a steep learning curve...

OK, I want to retrieve and log remote data along with output from local sensors on a CR1000. The remote data is JSON format, from of which I want two variables: "datetime" and "value"   (see below).  I am using HTTPget to call the data from a URI and return as "TextString" 

get_success = HTTPGet( "https://environment.data.gov.uk/flood-monitoring/id/measures/SU73_8-level-groundwater-i-15_min-mAOD/readings?latest", "TextString", "")

Two problems

The HTTPGet call isn't returning anything (the URI works fine elswhere) 

Is there a way to parse the JSON and store only the two variables I need? 

The test code is below, I'd greatly appreciate any advice to get this right, thanks

-----------------------

Public PTemp, batt_volt
Public get_success 'returns 0 if not successful, otherwise it should be a positive number (file handle)
Public TextString As String *1200 'returned string is 1100 characters long

'Define Data Tables

DataTable (Test,1,1000)
DataInterval (0,60,Sec,10)
Minimum (1,batt_volt,FP2,0,False)
Sample (1,PTemp,FP2)
Sample (1,TextString,String)

EndTable

'Main Program

BeginProg

Scan (1,Min,0,0)
PanelTemp (PTemp,60)
Battery (batt_volt)
'HTTPGet Instruction/Function
'get data from an internet server
get_success = HTTPGet( "https://environment.data.gov.uk/flood-monitoring/id/measures/SU73_8-level-groundwater-i-15_min-mAOD/readings?latest", "textString", "")
CallTable Test

NextScan

This is what should be retrieved

{ 
  "@context" : "http://environment.data.gov.uk/flood-monitoring/meta/context.jsonld" ,
  "meta" : { 
    "publisher" : "Environment Agency" ,
    "licence" : "http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/" ,
    "documentation" : "http://environment.data.gov.uk/flood-monitoring/doc/reference" ,
    "version" : "0.9" ,
    "comment" : "Status: Beta service" ,
    "hasFormat" : [ "http://environment.data.gov.uk/flood-monitoring/id/measures/SU73_8-level-groundwater-i-15_min-mAOD/readings.csv?latest", "http://environment.data.gov.uk/flood-monitoring/id/measures/SU73_8-level-groundwater-i-15_min-mAOD/readings.rdf?latest", "http://environment.data.gov.uk/flood-monitoring/id/measures/SU73_8-level-groundwater-i-15_min-mAOD/readings.ttl?latest", "http://environment.data.gov.uk/flood-monitoring/id/measures/SU73_8-level-groundwater-i-15_min-mAOD/readings.html?latest" ]
  }
   ,
  "items" : [ { 
    "@id" : "http://environment.data.gov.uk/flood-monitoring/data/readings/SU73_8-level-groundwater-i-15_min-mAOD/2020-06-08T20-00-00Z" ,
    "dateTime" : "2020-06-08T20:00:00Z" ,
    "measure" : "http://environment.data.gov.uk/flood-monitoring/id/measures/SU73_8-level-groundwater-i-15_min-mAOD" ,
    "value" : 109.3
  }
   ]
}


GaryTRoberts Jun 8, 2020 04:14 PM

It looks like you HTTPGet call is incorrect.

get_success = HTTPGet( "https://environment.data.gov.uk/flood-monitoring/id/measures/SU73_8-level-groundwater-i-15_min-mAOD/readings?latest", "textString", "")

should be

get_success = HTTPGet( "https://environment.data.gov.uk/flood-monitoring/id/measures/SU73_8-level-groundwater-i-15_min-mAOD/readings?latest", textString, "")

textString is a variable, so it should not be in quotes. Also, what does get_success show? Is it returning a number greater than 100?


DaveMattey Jun 9, 2020 04:23 AM

Thanks for reply.  That suggestion, and changing https to http worked!   I now get the JSON string in a table, next challenge is to extract the timestamp and value.  Looks like I can do this with SplitStr and other string functions.  On the road, thanks!


DaveMattey Jun 9, 2020 09:35 AM

Actually, doesn any one have any thoughts on parsing JSON data using the CRbasic functions?

thanks


JCaycho Jun 12, 2020 10:59 PM

Hi DaveMattey,

I have added some sequences to your code to extract the date-time and the value data you require from your string, also that can be saved in the Test1 table.

I have observed that the data is hourly and at the beginning there is a slight delay in connecting to your Http source.

In case the code doesn't work, I suggest you update the CR1000 data logger operating system to the latest version and then reload the code:

 

'CR1000 Series Datalogger

'Declare Public Variables

Public PTemp, Batt_volt
Public get_success 'returns 0 if not successful, otherwise it should be a positive number (file handle)
Public TextString As String *1500 'returned string is 1100 characters long
Public DateAndTime As String * 50
Public Value

'Declare Dim Variables

Dim FilterString1 As String * 20
Dim Start1
Dim MidString1 As String * 100
Dim ResultString(6) As String * 10

Dim FilterString2 As String * 20
Dim Start2
Dim MidString2 As String * 100
Dim ResultString2 As String * 10

'Declare const

Const  MaxLength1 = 32
Const  MaxLength2 = 16


'Define Data Tables

DataTable (Test1,1,1000)
DataInterval (0,1,Min,10)
Minimum (1,batt_volt,FP2,0,False)
Sample (1,PTemp,FP2)
Sample (1,TextString,String)
Sample (1,MidString1,String)
Sample (1,MidString2,String)
Sample (1,DateAndTime,String)
Sample (1,Value,IEEE4)

EndTable

'Main Program

BeginProg
  
Scan (30,sec,0,0)

PanelTemp (PTemp,60)
Battery (batt_volt)
'HTTP Get Instruction/Function
'get data from an internet server
  get_success = HTTPGet( "http://environment.data.gov.uk/flood-monitoring/id/measures/SU73_8-level-groundwater-i-15_min-mAOD/readings?latest", TextString, "")

      'Sequence to extract the characters from the datetime position:
      FilterString1 = "dateTime"
      Start1 = InStr (1,TextString,FilterString1,2)
      MidString1 = Mid (TextString, Start1, MaxLength1)
      SplitStr (ResultString(),MidString1,CHR(32)& CHR(34)& CHR(45)& CHR(84),6,0)
      DateAndTime=ResultString(1)&ResultString(2)&ResultString(3)&" "&ResultString(4)&":"&ResultString(5)&":"&ResultString(6)
      
       'Sequence to extract the characters from the value position:
      FilterString2 = "value"
      Start2 = InStr (1,TextString,FilterString2,2)
      MidString2 = Mid (TextString, Start2, MaxLength2)
      SplitStr (ResultString2,MidString2,CHR(32)& CHR(34)& CHR(58),1,0)
      Value=ResultString2
      

CallTable Test1

NextScan

EndProg

 


DaveMattey Jun 13, 2020 05:30 AM

Really helpful, got it working.  Thanks for replying, best wishes dave

Log in or register to post/reply in the forum.