Can RegExTool read chart data?

[RickyD333]

I don't get a whole lot of time to sit down and work on one particular Vox project every day, but I've been at this one when I can for a few days now. I can't seem to get Vox to read the data within a chart on this website: http://www.uavforecast.com/

I'm just trying to get it to read the information in the current conditions category. I thought after many failed attempts I might finally give in and ask for some help. I just can't seem to find the part within the regextool that gives the actual data. This is what I thought I would have to scrape, <th\sclass="(.*?)">(.*?)</th> , but when I do it only comes up with the title of a category, such as "wind", "temp", "wind chill"... (found within the non-greedy portions I entered in there). If you search for th class=   , you should be able to find the area I'm talking about.

Since I am only getting the titles of the categories, and unless I'm getting this all wrong somehow, I'm wondering if I can at all get Vox to read this data.

[nime5ter]

Can you please post the xml for your command(s) so far?

[RickyD333]

This is just the most recent attempt I tried this morning. I was just trying to get the temp info.

Code: [Select]

<?xml version="1.0" encoding="utf-16"?>
<!--VoxCommando 2.1.4.2-->
<groupCollection open="True" name="Drone Weather Test">
  <commandGroup open="True" name="Weather Conditons" enabled="True" prefix="" priority="0" requiredProcess="" description="">
    <command id="957" name="Drone Flying Conditions" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
      <action>
        <cmdType>Scrape</cmdType>
        <params>
          <param>http://www.uavforecast.com/</param>
        </params>
        <cmdRepeat>1</cmdRepeat>
      </action>
      <action>
        <cmdType>RegExTool.Open</cmdType>
        <params>
          <param>True</param>
        </params>
        <cmdRepeat>0</cmdRepeat>
      </action>
      <action>
        <cmdType>Results.RegEx</cmdType>
        <params>
          <param>&lt;th\sclass="(.*?)"&gt;Temp&lt;/th&gt;</param>
        </params>
        <cmdRepeat>1</cmdRepeat>
      </action>
      <action>
        <cmdType>OSD.ShowText</cmdType>
        <params>
          <param>{Match.1.1}</param>
        </params>
        <cmdRepeat>1</cmdRepeat>
      </action>
      <action>
        <cmdType>TTS.Speak</cmdType>
        <params>
          <param>{Match.1.1}</param>
        </params>
        <cmdRepeat>1</cmdRepeat>
      </action>
    </command>
  </commandGroup>
</groupCollection>

[Haddood]

Ricky, I just took a quick look at the page ... the page handle the tables in a different way. all data is stored in variable ...  then each table starts with "type" for the header ... than various rows ...

so I would scrape the page, isolate this variable ... then perform Regex to read each section ...
I highlighted, underlined and put in italic the different sections I could see... hopefully that clarify how is the data organized
to fully recognize the structure usually I use notepad and break the data into indented lines then try to work the regex

each section starts with a "type": ...data follows are shown in the yellow table header ... then you have "rows": followed by data for each line ... each line starts with "sun_is_up":  and ends with "time":


var initial_forecast_days=   {"days": [{"rows": [{"glonass_kp_20": 6.0, "gps_15": 8.0, "gps_10": 10.0, "glonass_kp_5": 7.5, "cloud_cover": 0.83, "wind_chill": 26.33,
 "glonass_15": 6.0, "gps_5": 11.0, "kp": 1.33, "glonass_kp_0": 8.1, "glonass_10": 7.0, "wind_80m": 11.28, "gps_0": 11.0, "gps_kp_20": 7.1, "glonass_5": 8.0, "glonass_0": 9.0, "visibility":
9.19, "glonass_kp_15": 6.0, "glonass_kp_10": 6.8, "wind_bearing": 203.0, "gps_kp_5": 10.3, "temp": 26.33, "gps_kp_0": 10.3, "glonass_20": 6.0, "gps_20": 7.0, "weather_icon": "partly-
cloudy-day", "precip_prob": 0.0, "gps_kp_10": 9.6, "gps_kp_15": 8.0, "time": "14:34"}], "type": "current", "time": "Saturday 2015-07-18 14:34 EDT"}, {"date":
"Saturday 2015-07-18", "sunset": "20:24", "rows": [{"sun_is_up": true, "glonass_kp_20": 4.2, "gps_15": 8.0, "gps_10": 9.0, "glonass_kp_5": 7.4, "cloud_cover":
0.78, "wind_chill": 26.58, "glonass_15": 6.0, "gps_5": 10.0, "kp": 1.33, "glonass_kp_0": 8.1, "glonass_10": 7.0, "wind_80m": 11.76, "gps_0": 12.0, "gps_kp_20": 7.9, "glonass_5": 8.0,
"glonass_0": 9.0, "visibility": 9.26, "glonass_kp_15": 5.9, "glonass_kp_10": 6.7, "wind_bearing": 202.0, "gps_kp_5": 9.4, "temp": 26.58, "gps_kp_0": 10.7, "glonass_20": 4.0, "gps_20": 8.0,
"weather_icon": "partly-cloudy-day", "precip_prob": 0.0, "gps_kp_10": 8.7, "gps_kp_15": 7.9, "time": "15:00"}, {"sun_is_up": true, "glonass_kp_20": 5.9, "gps_15": 7.0, "gps_10": 9.0,
"glonass_kp_5": 8.4, "cloud_cover": 0.63, "wind_chill": 27.18, "glonass_15": 7.0, "gps_5": 10.0, "kp": 1.33, "glonass_kp_0": 8.4, "glonass_10": 9.0, "wind_80m": 12.73, "gps_0": 10.0,
"gps_kp_20": 5.1, "glonass_5": 9.0, "glonass_0": 9.0, "visibility": 9.55, "glonass_kp_15": 6.8, "glonass_kp_10": 8.4, "wind_bearing": 190.0, "gps_kp_5": 9.1, "temp": 27.18, "gps_kp_0": 9.1,
"glonass_20": 6.0, "gps_20": 5.0, "weather_icon": "rain", "precip_prob": 0.15, "gps_kp_10": 8.4, "gps_kp_15": 6.8, "time": "16:00"}, {"sun_is_up": true, "glonass_kp_20": 6.0, "gps_15":
 6.0, "gps_10": 6.0, "glonass_kp_5": 8.4, "cloud_cover": 0.54, "wind_chill": 27.36, "glonass_15": 7.0, "gps_5": 8.0, "kp": 1.33, "glonass_kp_0": 8.4, "glonass_10": 8.0, "wind_80m": 14.58,
"gps_0": 10.0, "gps_kp_20": 4.2, "glonass_5": 9.0, "glonass_0": 9.0, "visibility": 9.71, "glonass_kp_15": 6.9, "glonass_kp_10": 7.7, "wind_bearing": 186.0, "gps_kp_5": 7.4, "temp": 27.36,
"gps_kp_0": 8.8, "glonass_20": 6.0, "gps_20": 4.0, "weather_icon": "rain", "precip_prob": 0.19, "gps_kp_10": 5.9, "gps_kp_15": 5.9, "time": "17:00"}, {"sun_is_up": true, "glonass_kp_20":
6.0, "gps_15": 5.0, "gps_10": 8.0, "glonass_kp_5": 9.0, "cloud_cover": 0.52, "wind_chill": 27.22, "glonass_15": 6.0, "gps_5": 9.0, "kp": 0.0, "glonass_kp_0": 9.0, "glonass_10": 7.0,
"wind_80m": 13.69, "gps_0": 10.0, "gps_kp_20": 4.2, "glonass_5": 10.0, "glonass_0": 10.0, "visibility": 9.35, "glonass_kp_15": 6.0, "glonass_kp_10": 6.9, "wind_bearing": 183.0, "gps_kp_5":
8.2, "temp": 27.22, "gps_kp_0": 8.9, "glonass_20": 6.0, "gps_20": 4.0, "weather_icon": "rain", "precip_prob": 0.18, "gps_kp_10": 7.5, "gps_kp_15": 5.0, "time": "18:00"}, {"sun_is_up": true,
"glonass_kp_20": 7.0, "gps_15": 7.0, "gps_10": 8.0, "glonass_kp_5": 8.7, "cloud_cover": 0.49, "wind_chill": 26.56, "glonass_15": 8.0, "gps_5": 8.0, "kp": 0.0, "glonass_kp_0": 10.0,
"glonass_10": 9.0, "wind_80m": 13.25, "gps_0": 8.0, "gps_kp_20": 6.1, "glonass_5": 9.0, "glonass_0": 11.0, "visibility": 9.26, "glonass_kp_15": 7.9, "glonass_kp_10": 8.7, "wind_bearing":
182.0, "gps_kp_5": 7.8, "temp": 26.56, "gps_kp_0": 7.8, "glonass_20": 7.0, "gps_20": 6.0, "weather_icon": "rain", "precip_prob": 0.11, "gps_kp_10": 7.8, "gps_kp_15": 7.0, "time": "19:00"},
{"sun_is_up": true, "glonass_kp_20": 5.0, "gps_15": 6.0, "gps_10": 8.0, "glonass_kp_5": 6.4, "cloud_cover": 0.45, "wind_chill": 25.94, "glonass_15": 6.0, "gps_5": 9.0, "kp": 0.0,
"glonass_kp_0": 8.3, "glonass_10": 6.0, "wind_80m": 13.59, "gps_0": 11.0, "gps_kp_20": 5.8, "glonass_5": 7.0, "glonass_0": 10.0, "visibility": 8.92, "glonass_kp_15": 5.8, "glonass_kp_10":
5.8, "wind_bearing": 175.0, "gps_kp_5": 8.0, "temp": 25.94, "gps_kp_0": 9.3, "glonass_20": 5.0, "gps_20": 6.0, "weather_icon": "rain", "precip_prob": 0.41, "gps_kp_10": 7.3, "gps_kp_15":
5.8, "time": "20:00"}, {"sun_is_up": false, "glonass_kp_20": 4.7, "gps_15": 7.0, "gps_10": 8.0, "glonass_kp_5": 5.5, "cloud_cover": 0.44, "wind_chill": 25.53, "glonass_15": 6.0, "gps_5": 10.0,
 "kp": 2, "glonass_kp_0": 6.6, "glonass_10": 6.0, "wind_80m": 11.96, "gps_0": 11.0, "gps_kp_20": 5.4, "glonass_5": 6.0, "glonass_0": 8.0, "visibility": 7.49, "glonass_kp_15": 5.5,
"glonass_kp_10": 5.5, "wind_bearing": 178.0, "gps_kp_5": 8.2, "temp": 25.53, "gps_kp_0": 8.7, "glonass_20": 5.0, "gps_20": 6.0, "weather_icon": "rain", "precip_prob": 0.37, "gps_kp_10":
6.9, "gps_kp_15": 6.2, "time": "21:00"}, {"sun_is_up": false, "glonass_kp_20": 4.6, "gps_15": 7.0, "gps_10": 9.0, "glonass_kp_5": 5.1, "cloud_cover": 0.52, "wind_chill": 25.28, "glonass_15":
5.0, "gps_5": 11.0, "kp": 2, "glonass_kp_0": 6.2, "glonass_10": 5.0, "wind_80m": 10.93, "gps_0": 13.0, "gps_kp_20": 6.4, "glonass_5": 6.0, "glonass_0": 8.0, "visibility": 6.12,
"glonass_kp_15": 4.6, "glonass_kp_10": 4.6, "wind_bearing": 191.0, "gps_kp_5": 8.9, "temp": 25.28, "gps_kp_0": 10.0, "glonass_20": 5.0, "gps_20": 7.0, "weather_icon": "rain",
"precip_prob": 0.33, "gps_kp_10": 7.7, "gps_kp_15": 6.4, "time": "22:00"}, {"sun_is_up": false, "glonass_kp_20": 2.8, "gps_15": 9.0, "gps_10": 10.0, "glonass_kp_5": 5.5, "cloud_cover": 0.52,
 "wind_chill": 25.06, "glonass_15": 4.0, "gps_5": 10.0, "kp": 2, "glonass_kp_0": 6.0, "glonass_10": 6.0, "wind_80m": 10.09, "gps_0": 11.0, "gps_kp_20": 7.2, "glonass_5": 7.0, "glonass_0":
8.0, "visibility": 5.52, "glonass_kp_15": 3.5, "glonass_kp_10": 4.9, "wind_bearing": 197.0, "gps_kp_5": 8.6, "temp": 25.06, "gps_kp_0": 9.1, "glonass_20": 3.0, "gps_20": 8.0, "weather_icon":
"rain", "precip_prob": 0.28, "gps_kp_10": 8.6, "gps_kp_15": 7.9, "time": "23:00"}],"type": "forecast", "sunrise": "05:39"}

[nime5ter]

Your UAV website is particularly complex. Maybe there are other ways to do this, but I would try a RoboBrowser solution.

Here's one possible method:

Code: [Select]

<?xml version="1.0" encoding="utf-16"?>
<!--VoxCommando 2.2.0.1-->
<command id="991" name="Robob Drone flying conditions" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
  <action>
    <cmdType>RoboB.Select</cmdType>
    <params>
      <param>uav</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>RoboB.Navigate</cmdType>
    <params>
      <param>http://www.uavforecast.com/</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>OSD.ShowText</cmdType>
    <params>
      <param>You asked for current {PF.1} conditions for {LongDate} ...</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>TTS.Speak</cmdType>
    <params>
      <param>You asked for current {PF.1} conditions. One moment please.</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>RoboB.Wait</cmdType>
    <params />
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>RoboB.ElementByTag</cmdType>
    <params>
      <param>TBODY</param>
      <param>0</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>RoboB.GetHTML</cmdType>
    <params />
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>RegExTool.Open</cmdType>
    <params />
    <cmdRepeat>0</cmdRepeat>
  </action>
  <action>
    <cmdType>Results.RegEx</cmdType>
    <params>
      <param>{1}</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>OSD.AddText</cmdType>
    <params>
      <param>The forecast says: {Match.1}</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>TTS.Speak</cmdType>
    <params>
      <param> The forecast says: {Match.1}</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <phrase>Current drone conditions for</phrase>
  <payloadFromXML phraseOnly="False" use2partPhrase="False" phraseConnector="by" Phrase2wildcard="anyone" optional="False">payloads\drone-conditions_current.xml</payloadFromXML>
</command>
RoboB plugin must be enabled.

In the above command I get the html for only one part of the web page (the current conditions table only). Since there was no obvious single regular expression pattern that would match all relevant fields, I use a payload XML file, which allows me to ask for a specific condition.

e.g. "Current drone conditions for precipitation | temperature | cloud cover | Visible satellites available ...."

Each payload phrase is associated with its own corresponding regular expression (see attached payload XML file).

Not sure if this is the kind of thing you're interested in, but hopefully it gives an idea of possible approaches if not.

[RickyD333]

Okay cool, thanks guys for helping me out. I'm going to give this stuff a test as soon as I get the chance.

[nime5ter]

Just wanted to mention that although I used regular expressions above, technically with RoboB that isn't necessary.

e.g., the command below demonstrates what happens if you use the RoboB.GetText action to capture text from within specific tags (rather than the GetHTML action):

Code: [Select]

<?xml version="1.0" encoding="utf-16"?>
<!--VoxCommando 2.2.0.1-->
<command id="999" name="robob2 -- no regex at all Drone flying conditions" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
  <action>
    <cmdType>RoboB.Select</cmdType>
    <params>
      <param>uav</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>RoboB.Navigate</cmdType>
    <params>
      <param>http://www.uavforecast.com/</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>RoboB.Wait</cmdType>
    <params />
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>RoboB.Show</cmdType>
    <params />
    <cmdRepeat>0</cmdRepeat>
  </action>
  <action>
    <cmdType>RoboB.ElementByTag</cmdType>
    <params>
      <param>TR</param>
      <param>2</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>RoboB.GetText</cmdType>
    <params />
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>OSD.ShowText</cmdType>
    <params>
      <param>TR2 result: {LastResult}</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>RoboB.ElementByTag</cmdType>
    <params>
      <param>TD</param>
      <param>2</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>RoboB.GetText</cmdType>
    <params />
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>OSD.AddText</cmdType>
    <params>
      <param>Wind speed cell only: {LastResult}</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <phrase>Get text demonstration</phrase>
</command>