The VoxKinecto Alpha Test

[Auri]

Hey James,

I could not wait to test voxkinecto out and since both my kids are in bed I took some time to set everything up.

First of all I'll give you the specs of my HTPC which is a zotac ZBOX ND22 with an en-US Windows 7 64bit Ultimate.
In my device manager I see the kinect audio array and camera's all correctly installed.

Audio output is HDMI audio through the speakers of my TV, recording device is set to Microsoft Microphone Array (Kinect)

I ran through the speech recognition wizard in Windows 7 and he can hear my voice clearly and all was good to go at the end of the wizard.

I downloaded the latest voxcommando from you website.
I then took you kinecto files and placed them in the same directory to which I installed Vox.

When I try and start voxkinecto.exe I get the following error message :

Code: [Select]

See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Speech, Version=10.2.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.Speech, Version=10.2.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
   at VoxCommando.Main.InitializeSpeechRecognitionEngine()
   at VoxCommando.Main.timer1_Tick(Object sender, EventArgs e)
   at System.Windows.Forms.Timer.OnTick(EventArgs e)
   at System.Windows.Forms.Timer.TimerNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].



************** Loaded Assemblies **************
mscorlib
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.235 (RTMGDR.030319-2300)
    CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
VoxKinecto
    Assembly Version: 0.1.0.0
    Win32 Version: 0.1.0.0
    CodeBase: file:///C:/Users/Auri/Documents/VoxCommando%20892/VoxKinecto.exe
----------------------------------------
System.Windows.Forms
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.235 built by: RTMGDR
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.1 built by: RTMRel
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.232 built by: RTMGDR
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
Interop.SpeechLib
    Assembly Version: 5.4.0.0
    Win32 Version: 5.4.0.0
    CodeBase: file:///C:/Users/Auri/Documents/VoxCommando%20892/Interop.SpeechLib.DLL
----------------------------------------
CoreAudioApi
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.0.0
    CodeBase: file:///C:/Users/Auri/Documents/VoxCommando%20892/CoreAudioApi.DLL
----------------------------------------
System.Xml
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.1 built by: RTMRel
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
System.Speech
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.1 built by: RTMRel
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Speech/v4.0_4.0.0.0__31bf3856ad364e35/System.Speech.dll
----------------------------------------
System.Management
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Management/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Management.dll
----------------------------------------
System.Configuration
    Assembly Version: 4.0.0.0
    Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
    <system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.


When I then try and start VoxCommando.exe I get the following error message :

Code: [Select]

---------------------------
Fatal Error
---------------------------
Error accessing the default audio input device:



Close the application, and check that your recording device is working and set as the default recording device.



-----error text:

System.InvalidOperationException: Cannot find the requested data item, such as a data key or value.

   at System.Speech.Recognition.RecognizerBase.SetInputToDefaultAudioDevice()

   at System.Speech.Recognition.SpeechRecognitionEngine.SetInputToDefaultAudioDevice()

   at VoxCommando.Main.InitializeSpeechRecognitionEngine()
---------------------------
OK   
---------------------------

I reckon this is not a problem with your voxkinecto.exe build but probably a system setting on my end, but I added all this in this new thread since it's the first thing I run against when trying the kinect setup.

Obviously I'm doing something wrong, please advice.

Regards
Auri

[jitterjames]

Hey James,

I could not wait to test voxkinecto out and since both my kids are in bed I took some time to set everything up....


...When I try and start voxkinecto.exe I get the following error message :
...
...
Regards
Auri

you need to install some additional stuff.  visit this link http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/docs/readme.htm
 and scroll down the the section that says

Code: [Select]

For Speech sample (x86 only):
Microsoft Speech Platform Runtime, version 10.2 (x86 edition)
Microsoft Speech Platform - Software Development Kit, version 10.2 (x86 edition)
Kinect for Windows Runtime Language Pack, version 0.9
     (acoustic model from Microsoft Speech Platform for the Kinect for Windows SDK Beta)
You must install the third (acoustic model), I think you need the first (runtime), I'm not sure about the middle one (Microsoft Speech Platform - Software Development Kit, version 10.2 (x86 edition) ), that one might only be needed if you are writing code.

good luck

edit: yes I know you have a 64bit system but install the x86 files anyway.

[Auri]

Got it working now and although it's an out of the box VC (without my custom xml's) it is working very good with my kinect.
Even when sound of a movie or music is playing.  Surely there is much to tweak from my point of things, such as kinect placement etc ... but for the moment with the limited time I have spend playing with it it looks promising and fun.

Will do some more testing tomorrow.

[jitterjames]

do you find that it works better with VoxKinecto than if you just use it as a regular default microphone with VC?

note that the Kinect SDK gives pretty meaningless confidence levels (often 99).  The bottom line is, does it understand your commands, and ignore other sounds.

Can you tell if it learns?  I have no idea if it does, or if it uses the speech profiles in WSR advanced control panel.

[Auri]

I have not yet tried the "regular" VC software.

What I'll do is, I'll do some more testing with louder then average movies, like when you're watching a blockbuster in 1080p dolby true HD and see how voxkintecto and voxcommander react to that (will be tomorrow since my kids are about to go to bed)  and I'll give you my insights in that.

Also I'll post a picture of my kinect setup, it's in front of the tv about 75cm's down which I think is ideal since the sound of my TV speakers will never cross the angle the kinect is listening to.

Stay tuned ...

PS : Perhaps move this thread and rename it like something "the voxkinecto experience" or something

[jitterjames]

you'll notice that the alpha testers folder now contains two different programs.

VoxKinectoAEC.exe
and
VoxKinectoNoAEC.exe

if you run VoxKinectoAEC.exe you must having some audio playing when you first start the program or it won't work.  AEC stands for acoustic echo cancellation, and when it works it seems to work pretty well at low to medium volume levels.

If you are having problems getting VoxKinectoAEC.exe to work then try VoxKinectoNoAEC.exe

[fattybunter]

I installed all of the Speech Platform files and I still get the same error whenever I try to launch VoxKinecto or KinectSimpleTrackerJames.

Code: [Select]

The version of this file is not compatible with the version of Windows you're running.
Check your computer's system information to see whether you need an x86 or x64 version
 of the program, and then contact the software publisher.

[fattybunter]

Ok I just re-installed Windows 7 (x64) and I get the same error! 

[Wanilton]

Do you install the SDK Version ok for 64 bits O.S?
Link for download:
http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/download.aspx

To install the SDK Beta:
 1.Make sure the Kinect device is not plugged in to the USB port on your computer.
 
2.Remove any other drivers for the Kinect device, including earlier versions of the Microsoft Kinect drivers.
 
3.Close Visual Studio.
Note: You must close Visual Studio before installing the SDK and then restart it after installation to pick up the MSRKINECTSDK environment variables that the SDK requires.
 
4.From the download location for the SDK Beta package, double-click the link for your system: 32-bit or 64-bit.

To load the Microsoft Kinect drivers:
1.Make sure the Kinect device is plugged into an external power source.
 
2.After installing the SDK Beta, plug your Kinect device into the PC's USB port.
 
3.All the drivers, including audio, will load seamlessly.

If the driver is installed correctly:
1.You should see a blinking green LED on the Kinect sensor.
 
2.The Kinect sensor should appear in Device Manager as the following nodes under “Microsoft Kinect”:
• Microsoft Kinect Audio Array Control
• Microsoft Kinect Camera
• Microsoft Kinect Device
 
3.The Kinect sensor's microphone array should appear under Sound, Video and Game Controllers in Device Manager as:
• Kinect USB Audio

[jitterjames]

you also need .net 4 but I think you must have it or it would not let you install the sdk.

Wanilton, were you able to run VoxKinnecto?  You haven't said anything about it.

[jitterjames]

look in the dropbox for VoxKinectoNOAECforFatty.exe

see if it works...

[Wanilton]

James,

            Hello J, only today I tested this version, and work fine for me...win7 64 bits, i need more tests, in special with voice recognition
using Kinect, but the VoxKinectoAEC run ok for me, no errors, I need understand concepts, running KinectSimpleTrackerJames work ok too,
red and green alternate moving my left or right hand...
           

Wanilton

[jitterjames]

James,

            Hello J, only today I tested this version, and work fine for me...win7 64 bits, i need more tests, in special with voice recognition
using Kinect, but the VoxKinectoAEC run ok for me, no errors, I need understand concepts, running KinectSimpleTrackerJames work ok too,
red and green alternate moving my left or right hand...
           

Wanilton

ok, good.  Thanks

[Auri]

HI James,

SOrry for the somewhat late reply, specially because I started this thread to trigger a more active test cycle.
Anyways, I found some time for myself today and scheduled some testing time.

First off, It might be me, but I guess there is no "real" documentation about vox kinecto right ? I know it's alpha but I already struggle with all the EXE files and  config files that were in you're shared dropbox dir.
If you could like quickly document what EXE does what, and also whats the don'tdeleteme.wav for ? is it suposed to start together with the AEC exe so it has "sound" to cancelate (is that even a word ? lol) cause if it should then that is not working for me.

Also I keep getting the same error when I try your simpletracker exe :

Here is the eventlog error message it generated. For some reason I cannot copy/paste the adhoc dump when I get the error itself since it'll autojump directly to "close program" or "try to find a solution online" dialog where either choice brings me back to the desktop.

Code: [Select]

Log Name:      Application
Source:        Application Error
Date:          4/07/2011 18:26:28
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      HQ-ZBOX-UP
Description:
Faulting application name: KinectSimpleTrackerJames.exe, version: 1.0.0.0, time stamp: 0x4e0f537b
Faulting module name: KERNELBASE.dll, version: 6.1.7601.17514, time stamp: 0x4ce7bafa
Exception code: 0xe0434352
Fault offset: 0x0000b727
Faulting process id: 0xcc8
Faulting application start time: 0x01cc3a671e057b60
Faulting application path: C:\Users\Auri\Documents\VoxCommando 892\KinectSimpleTrackerJames.exe
Faulting module path: C:\Windows\syswow64\KERNELBASE.dll
Report Id: 5e1c5f20-a65a-11e0-85d8-00012e2bbca4
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-07-04T16:26:28.000000000Z" />
    <EventRecordID>2074</EventRecordID>
    <Channel>Application</Channel>
    <Computer>HQ-ZBOX-UP</Computer>
    <Security />
  </System>
  <EventData>
    <Data>KinectSimpleTrackerJames.exe</Data>
    <Data>1.0.0.0</Data>
    <Data>4e0f537b</Data>
    <Data>KERNELBASE.dll</Data>
    <Data>6.1.7601.17514</Data>
    <Data>4ce7bafa</Data>
    <Data>e0434352</Data>
    <Data>0000b727</Data>
    <Data>cc8</Data>
    <Data>01cc3a671e057b60</Data>
    <Data>C:\Users\Auri\Documents\VoxCommando 892\KinectSimpleTrackerJames.exe</Data>
    <Data>C:\Windows\syswow64\KERNELBASE.dll</Data>
    <Data>5e1c5f20-a65a-11e0-85d8-00012e2bbca4</Data>
  </EventData>
</Event>

I also find that by using the AEC exe the volum needs to be LOW in order to work properly.
I'm still testing with different placings of the kinect and differnet vox exe's (such as the AEC one, the "retail" one and the noAEC, etc ...)

I've made a few pictures of where my kinect is placed.  When I first tried kinect with the retail vox I was greatly impressed by how well he picked up my voice as I said earlier in this thread, but I must have done something wrong or changed something cause I must say atm I just cvan't get him to recognize my voice properly when there is sound playing at medium ... "normal" sound levels.

Taken with my iphone does not great quality!







Hope I can test some more tomorrow !

Regards
Auri

[jitterjames]

First off, It might be me, but I guess there is no "real" documentation about vox kinecto right ? I know it's alpha but I already struggle with all the EXE files and  config files that were in you're shared dropbox dir.
If you could like quickly document what EXE does what, and also whats the don'tdeleteme.wav for ? is it suposed to start together with the AEC exe so it has "sound" to cancelate (is that even a word ? lol) cause if it should then that is not working for me.

Also I keep getting the same error when I try your simpletracker exe :

Here is the eventlog error message it generated. For some reason I cannot copy/paste the adhoc dump when I get the error itself since it'll autojump directly to "close program" or "try to find a solution online" dialog where either choice brings me back to the desktop.

I also find that by using the AEC exe the volum needs to be LOW in order to work properly.
I'm still testing with different placings of the kinect and differnet vox exe's (such as the AEC one, the "retail" one and the noAEC, etc ...)

I've made a few pictures of where my kinect is placed.  When I first tried kinect with the retail vox I was greatly impressed by how well he picked up my voice as I said earlier in this thread, but I must have done something wrong or changed something cause I must say atm I just cvan't get him to recognize my voice properly when there is sound playing at medium ... "normal" sound levels.

you don't need any other files except either:

VoxKinectoAEC.exe or VoxKinectoNoAEC.exe

put them in any existing VoxCommando folder and run them.  There is no documentation, but there isn't any need really.  It works pretty much the same as VC except for the limiitations I have mentioned (like no dictation commands) and the extra angle detection / rejection.

if you use VoxKinectoAEC.exe you need to make sure that audio is playing (from the PC which has Kinect attached) when you start the program.  In theory it should be better (than VoxKinectoNoAEC.exe) with music playing as long as that music is coming from your PC.  I don't' know if it works with hdmi etc. but it seems to sort of work for me using coax (which is digital) as a standard default stereo output.

As for the error with the tracker program, despite its extreme verbosity it doesn't actually tell me anything useful.  Anyway it's just  silly toy really.

[fattybunter]

I'm finally able to load up any of VoxKinectoAEC, NoAEC, and NOAECforFatty.  Results:

• Extremely high positives (~99%) when the room is quiet.
• A lot of false positives (~30-90% confidence) when video is playing
• Looks like the angle is unreliable
• Pretty sure I get better results with plain Vox

I'm going to experiment some more with some Prefixes to see if that gets better results

[jitterjames]

do you have any idea what changed?  Did you install something new?

[fattybunter]

VoxKinecto.exe still gives me the same error I was always getting.  Nope, I didn't change anything

[tranbert]

Hi, from Paris France....

Just testing voxkinecto, and I got a question :

Do you have to use speech models and recognition "made by kinect" or is it possible to use only  the tracking part of SDK and use windows speech recognition ?

When you use kinect as a standard mic, it's only a stereo mic, not array, but it works (on my french speaking/listening voxcommando...)

So using only the array part should be OK... Maybe it's already what voxkinecto does, so why only english speech model ?

I'm a little confused.... 

[jitterjames]

to answer your question I would need to know what you mean by "tracking".

In order to use the features of the array microphone, you need to use the sdk, which means you need to use the kinect speech model (as far as I know), which means English only.  Hurray Microsoft! <not>

If you are talking about tracking with the camera, then yes, you can use that, and keep using the regular version of VoxCommando which supports other languages, profiles, learning etc.

Edit:
it might also be possible to detect the direction that sound is coming from and turn VC on and off from that, but keep using the regular stereo microphone input with regular VC but I'm not sure it is worth the trouble.

[tranbert]

to answer your question I would need to know what you mean by "tracking".

In order to use the features of the array microphone, you need to use the sdk, which means you need to use the kinect speech model (as far as I know), which means English only.  Hurray Microsoft! <not>

Voilà... It was the question.... No array microphone if you don't use speech model...  It's strange, because the array is seen a a device in windows, should be usable as a standalone device... Grrrr

In fact for image tracking, Kinemote is "better" (but 99.999% beta) because it tracks only the hand, so you can be a couch potato and use your kinect ;-)

All this is early development, i'll wait and stay tuned on this (and learn c++ ?)

[jitterjames]

I know, you shouldn't even need to install the SDK to get the benefits of the array!  Anyway, with or without it I haven't found the results to be so great.  I get too many false positives with the Kinect.