VC Knows All About You

[jb5349]

So I have been working on this idea of having VC know all about the person that it is interacting with. I have setup so that depending on who calls in using Skype to VC, it knows who that person is and then stores that info in a variable which is uses to do other tasks. I also set a command to switch user or basically state this is "so and so" and then it also sets this variable that way as well, in case not using Skype or needing to override it for some reason.

So with VC knowing who it is talking to the following things can happen right away.

1. VC can switch MS Speech Recognition Profile so it uses a trained profile for the person it is talking to for better accuracy.
2. VC can now refer to the person it talks to in a variety of command responses, so it seems more human like. Just by including the variable in your TTS string.

Now to take this to a new level, I started playing with the idea of building a database of knowledge about that person. In this case I started by using a simple Map which has a key and value. So by creating a table for that specific user that VC is talking to, I can have VC know about that person, and pull information about that person as needed in any of my other commands or responses.

For example:

If I say..  "Learn that my favorite color is blue" then VC stores a key called color and a value of blue.

Then later I can ask, "What is my favorite color?" and VC says "blue".

Simple enough. Now I can do this for anything really. Remember my birthday is on "November 29th".

Later, I can setup an automatic checking for current date and maybe have VC say "Happy Birthday Jeff" when it is actually my birthday. If not, I can ask, when is my birthday, and it responds.

But lets take this a step further. Not only can this Map include any type of info pertaining to the user, but now we can look at making this info available to other users should they need to know.

For example, I can have a command that, regardless of the user speaking can be  as follow's  "What is Jeff's favorite food" and then it checks the map for Jeff and returns the info to that user. Obviously you want to limit which type of data can be done this way, but it makes VC seem much smarter. "When is my wife's birthday", "What is my wife's favorite movie", etc. Maybe if your other user is a child, spouse, etc. you can then learn more about them through VC. Assuming they interact with VC and provide this information. If nothing else, you don't have to remember it all and can ask your smart home the answer.

Now, the final and ultimate level to this idea is basically to let VC create these "memories" on the fly. So if the Map doesn't exist for a person yet, it should be able to create one, then start adding in the information. "Remember my dad's phone number is 1234567890" and then when dad's map doesn't exist, it says "Ok, I will remember" and then it creates a new map called JeffsDadsInfo and a new key called phone number and value of 1234567890. So next time you ask, what is my dad's phone number, it matches dad's to JeffsDadsInfo and phone number to the map key and then returns the number. Then lets say another user "my wife" is talking with VC and asks "What is Jeff's Dad's phone number" and VC knows to check the map for that entry.  I can see creating the table name on the fly and linking it back with the command being a bit difficult.

I guess with this information you can also have VC use it when needing for other things like..  For Hue users.. VC can have a response "Would you like me to set the light color to blue? I know its your favorite."  Or after a command of "Play some music"  VC can say, would you like to listen to "Favorite Song" because it knows this. It could be favorite artist, band, etc. Or it may say "How about some James Brown" assuming you have that as a favorite artist.

Taking this further.. "I'm getting hungry" and lets assume you said you favorite food was pizza. VC says "Hmm. perhaps we should order in some pizza" or maybe if delivery isn't a choice, it could just be "What should I have for dinner?" and VC says "How about pizza?"

I hope you see the point of this is that you make VC aware of your preferences. You can even do this with things like volume level, light level for different rooms, etc. just be having it remember these things and recall them later. Oh.. Remember that I prefer the temperature to be 68 degrees at night. Or remember that I prefer the temperature to be 72 degrees during the day (sorry using Imperial here). All kinds of routines can be created for VC to know you.. Although I can see that being funny if VC checks each user and one has preferred temp to be 68 while the other is 74. Could be thermostat war!

Anyone have any ideas on generating the map names on the fly and having them work with commands later on. OR maybe this means creating new commands to interact with them on the fly as well? Any feedback or suggestions would be appreciated.

[jb5349]

I know that was a little long winded.. here is a simple way to put it..

Make VC...

Remember that A is B. And store that for each user.

Examples:

favorite color = blue
favorite food = pizza
preferred temp = 68
preferred volume = 25
preferred light level = 75
birthday = november 29th
wife's phone = 1234567890
dad's phone = 5555554555


To be able to create these memories, you have to limit what VC can remember to some degree, just to make it work well. So using a payload XML, you can actually come up with a list of choices. Can be hundreds (maybe thousands as I don't know technical limitations)

List would look something like this

color
book
movie
song
artist
album
music
film
number
food
vacation spot
home address
work address
email address
phone number

etc..

whatever you want to be able to say (i.e. "Remember that my A is B") with A being one of the payload XML and B being Dictation.

I am continuing to experiment and develop this further and if I can get something I think will work for most people, I will post under XML exchange.

[nime5ter]

There will be other challenges to resolve, but you may want to divide your profile attributes into at least a few different commands so that you can customize how you provide the information more, verbally.

For example, I think you'll have difficulty providing phone number information using regular dictation. You'll want to use dictation:spelling for that. The result of the phrase:

My {wife}'s {mobile phone number} is {dictation:spelling "2024534444"}

will end up as: key: mobile phone, value: 2 0 2 4 5 3 4 4 4 4 in map "wife".

You could fix the white space issue with some regex (e.g., http://voxcommando.com/forum/index.php?topic=1645.msg14329#msg14329)

But you don't necessarily want to use dic/spelling payload for everything. So, some forethought about what can be grouped together into natural-feeling command structures that are still functional will probably save some time.

[nime5ter]

The Map.CreateTable action will only create a new map if a map with that name does not yet exist. So, using Map.CreateTable in your commands should still function whether you already have a map or not, but if you're specifying the map name using a payload of some kind this may require some creativity.

I would probably rely on command confirmation for these types of commands. Otherwise you'll probably end up with some really messy map data.

You might also consider following the example of the Our Groceries command structure (http://voxcommando.com/forum/index.php?topic=843.msg13689#msg13689).

These groups use confirmation where appropriate, but also rely on the priority system to favour the "add item" command that uses a payload xml file first; if the grocery item given doesn't fit with anything on the list, it uses the lower priority "add item" command, which uses a dictation payload (see also: http://voxcommando.com/mediawiki/index.php?title=Groups#Group_Properties).

[jb5349]

Ok great, thanks for the tips and suggestions. I will try these out and see what I can come up with.

Right now I have the My Favorites, My Relationships and My Personal Details as separate commands, also have one for preferred settings to be used for things like temp, volume, etc. So I need to create a new one for numbers input.

Is there any way to go back and process words and convert them to numbers. Lets say "Remember my address is one two three oak street" and I want it to take that and make it "123 oak street" before saving to the map. Or doing the same with a phone number for example?

I know there was some kind of payload with numbers that had words and the numerical value. Maybe if I set that as optional before the dictation part, it could work for address entry?

I see that I need to create two commands here. One ready for numerical data based on some key words like phone, etc. and one for address, etc.

[Haddood]

I would store all in payload files ... This way they can be used in commands to reduce the amounts of commands (I think 2 or 3 only)...

One challenge apart from dictation issues is "my" when not speaking through Skype...

A, thinking on a command along

Remember that, learn that
Payload (different people, Jeff, son ....etc.) this will allow multiple names for same person, ie my son and Justin both return justin - this has to be programmed manually or creating a command that create entries in a similar manner
Dictation (things to remember, fav mov, color...etc.) (key is person from first payload, value is things to remember)
Is, are
Dictation (answer) (stores person + things to remember as key, and value is the answer)

To retrieve info,

Remind me,
what, who, when, where ...etc.
is, are
Payload (people)
Payload (things to remember)

This should be armed with event and returns 2 payloads

I love the idea will give it a try and post Back here (though no time to do it for couple of days)

[jb5349]

I am only using Skype at this point since I don't have any other mic setup. But I see your point about when not using Skype. I already created a switch user command.

"This is John speaking" or "Switch user to Susie", etc.

This sets a variable  {var.CUser} to be the first name of that person. That is fine unless you have more than one person in the house with the same first name.. In which case use the first and last name and then have VC shorten it when speaking.

I have everything built for having VC to know all about the user, however I just have to complete the numerical data entry stuff and then I can test it. If it all works well I will post the XML.

[Haddood]

I already created a switch user command.

"This is John speaking" or "Switch user to Susie", etc.

I have same solution with another to make it more natural... I check the commands and if it has "darling" somewhere it mean it is my significant other talking ... If it has "dude" it means my son ...
The down side of this approach is I have to add darling and dude to every command and as suffix or prefix as well where I think they might occur in the command

[nime5ter]

Is there any way to go back and process words and convert them to numbers. Lets say "Remember my address is one two three oak street" and I want it to take that and make it "123 oak street" before saving to the map. Or doing the same with a phone number for example?

I know there was some kind of payload with numbers that had words and the numerical value. Maybe if I set that as optional before the dictation part, it could work for address entry?

If you're a programmer, you could try to write a Python script to do the number conversion for you. But doing the kind of conversion you're speaking of is not so straightforward, since there's a great variety to how we say numbers in English -- particularly for addresses.

Your other option is to use number ranges, along the lines of:

Code: [Select]

<?xml version="1.0" encoding="utf-16"?>
<!--VoxCommando 2.1.0.3-->
<command id="333" name="remember address" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
  <action>
    <cmdType>Map.CreateTable</cmdType>
    <params>
      <param>{1}</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>Results.SetLastResult</cmdType>
    <params>
      <param>{2}{3} {4}</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>Results.RegExReplace</cmdType>
    <params>
      <param>{2}{3} \{4\}</param>
      <param>{2} {3}</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>OSD.ShowText</cmdType>
    <params>
      <param>{1}'s address is {LastResult}.</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>Map.Set</cmdType>
    <params>
      <param>{1}</param>
      <param>address</param>
      <param>{LastResult}</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <payloadList>Bob, Omar, Dana, Celeste</payloadList>
  <phrase>address is</phrase>
  <payloadRange>1,99</payloadRange>
  <payloadRange optional="true">1,99</payloadRange>
  <payloadDictation>payloadDictation: Regular</payloadDictation>
</command>
The above is relatively flexible in allowing variations such as "one ninety-nine" "two oh two" "fifty-four forty" etc.

Within the command, it concatenates the 2 numbers into one, or if it hears only 1, deletes the empty payload.

But you'll probably have to rely on the Alternates technique as well (http://voxcommando.com/mediawiki/index.php?title=Options); that is, have your options set to not automatically execute a command if there are more than 2 or so alternates, and instead choose the correct alternate from the list of possibilities: ("Option 3") etc.

For whatever reason, this is how the Windows SR engines have decided to handle numbers.

... And as mentioned, if you use Dictation: Spelling rather than Dictation: Regular, it will return digits not words, so for phone numbers that is a reasonable option.

[Haddood]

I can't remember where I found this but might be of help with numbers ...
it is a python that receive numbers in words and convert them to digits

load it then call it with
PYexecstring result = txt2int ('fifteen')

I think it can be modified to return the full original phrase with text numbers replaced with digits

Code: [Select]

def txt2int(textnum, numwords={}):
    if not numwords:
      units = [
        "zero", "one", "two", "three", "four", "five", "six", "seven", "eight",
        "nine", "ten", "eleven", "twelve", "thirteen", "fourteen", "fifteen",
        "sixteen", "seventeen", "eighteen", "nineteen",
      ]

      tens = ["", "", "twenty", "thirty", "forty", "fifty", "sixty", "seventy", "eighty", "ninety"]

      scales = ["hundred", "thousand", "million", "billion", "trillion"]

      numwords["and"] = (1, 0)
      for idx, word in enumerate(units):    numwords[word] = (1, idx)
      for idx, word in enumerate(tens):     numwords[word] = (1, idx * 10)
      for idx, word in enumerate(scales):   numwords[word] = (10 ** (idx * 3 or 2), 0)

    current = result = 0
    for word in textnum.split():
        if word not in numwords:
          raise Exception("Illegal word: " + word)

        scale, increment = numwords[word]
        current = current * scale + increment
        if scale > 100:
            result += current
            current = 0

    return result + current

[Haddood]

if ever anybody want to et really fancy about this, the results should be processed with NLP .... here is couple of them for python

in fact if this is achieved it means VC starts to get real AI

http://www.nltk.org/
http://textblob.readthedocs.org/en/dev/

[nime5ter]

I can't remember where I found this but might be of help with numbers ...

You found it on the VC forum. James posted it a few months ago. And yes, it can be modified, but my statement above is still relevant. It's only in certain circumstances that we would always say numbers in our sentences as "five thousand four hundred and thirty", which is the kind of number phrase that the above code will convert to digits. It isn't really versatile enough.

But, you inspired me a bit. jb originally asked how he could convert something like "Remember my address is one two three Rodeo Drive." As long as he's willing to dictate all of his numbers as "digit digit digit", the following adaptation will work (thanks James for helping me punch this out quickly this morning ... and thank you morning coffee!).

Code: [Select]

def text2int(textnum):

    everything=''
    units = {
        "zero":"0", "one":"1", "two":"2", "three":"3", "four":"4", "five":"5", "six":"6", "seven":"7", "eight":"8",
        "nine":"9"}
   
    lastWordWasNum = False
   
    for word in textnum.split():

        if word not in units:
            everything = everything+" "+word
            lastWordWasNum = False
        else:
            if not lastWordWasNum:
                everything = everything + " "
            everything = everything + units[word]
            lastWordWasNum = True                   

    return everything.lstrip()

[nime5ter]

So, using the above, you could do something like the following:

Code: [Select]

<?xml version="1.0" encoding="utf-16"?>
<!--VoxCommando 2.1.0.3-->
<command id="346" name="remember address" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
  <action>
    <cmdType>PY.ExecFile</cmdType>
    <params>
      <param>PY\numDict.py</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>Map.CreateTable</cmdType>
    <params>
      <param>{1}</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>PY.ExecString</cmdType>
    <params>
      <param>result=text2int('{2}')</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>OSD.ShowText</cmdType>
    <params>
      <param>{1}'s address is {LastResult}.</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <action>
    <cmdType>Map.Set</cmdType>
    <params>
      <param>{1}</param>
      <param>address</param>
      <param>{LastResult}</param>
    </params>
    <cmdRepeat>1</cmdRepeat>
  </action>
  <payloadList>Bob, Omar, Dana, Celeste</payloadList>
  <phrase>address is</phrase>
  <payloadDictation>payloadDictation: Regular</payloadDictation>
</command>
(above python, attached to this post -- though it would make more sense to load the python file separately when VC first launches, rather than every time this command is executed)

[jb5349]

Great thank you! I am going to give it a try.

I am really trying to figure out how to give my smart home a personality. I am thinking of a framework of commands which anyone can build upon to customize it for their own use, adjusting personality, etc. I know there are so many different setups out there but I find that in order to get people to use something, they have to be able to talk to it like it was another person, not having to change how they talk for the machine but rather have the machine understand them. I guess this is NLP in a way. But I really like the idea of creating a base personality that others can then take it to another level.

We will see how it goes.. if I come up with something interesting, I will be sure to share it in the XML exchange.

Right now I am trying to limit the commands to the essence of its meaning with all of the possible alternate words as optional. Only adding extra words when needed to keep commands separate from others that are similar.

One question as I haven't found the answer yet in searching..

Can you have a command response be constructed into a variable through a series of IF/Then Else statements in a Macro?

Example..

I give a command but for the response I have a variable called var.TheResponse  and it goes through a series of IF/Then statements building the variable.

So If A = B then
var.TheResponse is "The first part of my response is {1}

Then another If then A= B
var.TheResponse = var.TheResponse + "and this is the second part of the response"  (is there some kind of Append to existing Variable command?)


I am basically wanting a response to be customized based on a set of conditions. Maybe even based on what other variables contain.

Just so it can pull from multiple stored variables to create responses which make it seem more intelligent and human like.

So the end result of a custom build response is then spoken with TTS.Speak var.TheResponse

[jitterjames]

You can concatenate in VC using the concept of "a=a+b" just as you have described, but in the case of TTS it is not really necessary.

Here are two examples to illustrate.  The first concatenates three strings and then speaks.  The second example just speaks using 3 actions:

Code: [Select]

<?xml version="1.0" encoding="utf-16"?>
<!--VoxCommando 2.0.0.5-->
<commandGroup open="True" name="concatenate" enabled="True" prefix="" priority="0" requiredProcess="" description="">
  <command id="245" name="concatenate" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
    <action>
      <cmdType>Results.SetLastResult</cmdType>
      <params>
        <param>one</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>Results.SetLastResult</cmdType>
      <params>
        <param>{LastResult}, two</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>Results.SetLastResult</cmdType>
      <params>
        <param>{LastResult}, buckle my shoe</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>TTS.Speak</cmdType>
      <params>
        <param>{LastResult}</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
  </command>
  <command id="249" name="conspeakenate" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
    <action>
      <cmdType>TTS.Speak</cmdType>
      <params>
        <param>one</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>TTS.Speak</cmdType>
      <params>
        <param>, two</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>TTS.Speak</cmdType>
      <params>
        <param>, buckle my shoe</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
  </command>
</commandGroup>
In my example I am using {LastResult} but you can do the same thing with variables if you want.

[jb5349]

Ah I see. Very clever and not so obvious ways to do this. I will give them both a try and see what I can come up with. Thanks for the examples!

[Haddood]

To create personality, I created a command called feed back, I pass to it a key which will retrieve a value from various possible answers by comma. I store the name of the file in variables ...
I have multiple files with different moods... Happy, sad and normal (more could be added)
The same key in each file will have different phrases based on the mood (and language).... So if it is bad weather, I change the file name variable to be sad .. Good weather normal ...etc.
It is a work in progress (as far as responses) but the base is there and working... I think this approach reduces the multiple ifs

[Nick]

Is there any progress on this?
I was reading the thread and I found it really interesting!