prototyper's toolbox: pythonic edition

        I had a really good convo with a co-worker today and I realized I miss working in Python just a wee bit.  For prototyping especially, why wouldn't you use Python?   I'm surprised the thought hasn't really hit me until just now, but you know, the fast turnaround provided by working in an interpreted environment is fairly ideal for rapid prototyping, no?

        As much as I like Unity, it's definitely getting to the point where i'm having to do enough custom implementation that it's putting a damper in my relevant iteration time.  Don't get me wrong, i'm enjoying getting to dip my toes into other languages and features (learning P/Invoke has been pretty cool), but i feel like a Python based prototyping environment might get me a little further a little faster.  More often than not, Python modules work in whatever your Python environment is, and I understand that's not a 1-to-1 comparison as Unity is not a pure .NET environment, but still...

        So in my wanderings last night, I dug up a few different tools that seem like things I'll want to be diving into next.  I went to bed super early, so I haven't spent a ton of time with alot of these resources, but enough to feel like these will all be useful in the future.  Once i get done with this current demo cycle, I'm going to get out of Unityland and start writing standalone apps, just to get more practice writing big software projects.  I feel like I'll have more flexibility in a pure python environment and I'll probably get more done faster. Props to my co-worker Chris for inspiring my search...

---

Modules/Libraries
        Most of these provide the full suite of lower-level functionality you need for prototyping/building media applications (event loop processing, input management, etc), just import whatever else you need alongside them and you're good to go.  Fair warning, some of these are NUI/MT specific, but then, that's really what I'm into these days.  I gotta say it's also made me realize how much of a gadgetwhore I am, but man, I hate spending money on them, which is probably why i don't have one.  That and it also makes me realize how much of a touch interface freak I am.  I can't say I'm huge on gestures, but touch is cool.  I actually found a fairly extensive list of game engines that would probably be make great generic event loop/input/rendering managers too, that's for another blogpost...

Python for Android
I'm not 100% sure why i put this here but it seemed like a cool little aside. I only glanced over the docs a bit, so I can't make any recommendations, but I will say it's definitely more of a power tool than a ready to go development resource. More and more i keep finding reasons to want to get that Android device i keep threatening to get, but then i think, ugh, do i really want to spend money on another gadget? "Another", like i own a bunch of them already...

pygame and PyKinect
I lumped both of these together because most of the PyKinect samples run on pygame anyway, altho nothing precludes you from using your event loop manager (heretoforth referred to as ELMs) of choice. While this isn't necessarily MT, if you want to get started with NUI prototyping, this is honestly the way I'd recommend going. The Kinect SDK is super easy to write code against, you'll be doing crazy things in no time.

VPython
Alright kids, gather around, and let me tell you a tale of POVRay...yes, I just dated myself horribly, but then, i'm constantly getting told that I'm too old for...certain people anyway, so i guess it is what it is. But really, that's about what we have here, even to the point where VPython lets you export to POVRay renderable files. How cool is that? Also includes its own version of IDLE, but I'm going to have to frown on that...

kivy
From what I understand, this is the way to go for Python MT development. I found a few other options (PyMT, etc), but this seems to be the one everyone recommends. If you're an experienced pythonista or have an established development environment, you can skip some of the weird setup requirements and dive right in. Get started now and whip some cool stuff up for their next contest!

pyTUIO
No discussion of MT development would be complete without including some TUIO bindings, so here you go! There are a few dependencies you might want to be aware of too (most notably reacTIVision), but once you've get everything down, this is another great way to get up and running with minimal hardware investment and setup. Make your own Reactable, actually, i've just hit on a plan...

---

Art IDEs
        Sure you could sit in a coffee shop and write a book, but let's be honest, if any chick looks over your shoulder and sees you rockin some cool interactive art in one of these apps, there's an icebreaker. Show her your camera or other input device, get her to play around with it a bit and it's all over except for the part where you embarrass yourself horribly trying to ask her out...Probably better to just get a bunch of buddies together and rock some interactive art jams on your laptops, takes me back to the days of laptop jamming with Live.  Or at least I think it would, i'll know more tomorrow, actually going to a group Processing session of sorts, should be fun. May even be the beginning of my social life...

Nodebox 2
If you're a Maya or Houdini Tech Artist, or just an ICE freak, this one's for you.  I actually came across Nodebox a while back, but got a sad panda because it was MacOS only, altho I suppose it would have been a good use for my still not very used Macbook Pro. But yeah, this is basically Processing with a hypergraph attached to it, how cool is that? And Python to boot, so it's like Maya, but light and stable...ish.

Shoebot
Shoebot is the best-ish of both worlds in that you can run it from its own IDE or you can import it into existing Python projects. It's also a...well, let's say derivative, for lack of a better term, of Nodebox, so most of the docs you find for one apply to the other. Not to down on the work of the Shoebot guys, it's a "derivative" like pyprocessing or pycessing are derivatives of Processing.

---

Additional Reading
        A few last little tidbits to keep your head in the MT game, skip this if you're not interested in this sort of thing.

Txzone
All sorts of really fun musings here, not much of a learning resource, but great for inspiration and keeping up with what all's out there for the MT pythonista...


Getting Started With MT Dev In Python
Some dated and slightly specific tutorials, but definitely great for a quick dip of your toes into the whole world of Python MT/NUI development.


Part 1 | Part 2 | Part 3 | Part 4

---

        I've got some pretty cool interactive art projects in the pipe right now, no ETA on any of them yet, but hopefully I'll be able to post some stuff up in the near future.  Got some longer term "milestones", altho I'm not sure milestone is the proper term since they tend to equate to public releases or showings of some sort...Nothing like a little pressure.

with trees

        Updated to include Teck's suggestions so it looks less like machine and more like man(god?).  Now we get something that looks like this:


Yes, this looks much better methinks.  Only slight changes, just dropping in new terms really:

void Start() { terr = GetComponent<Terrain>(); terr.terrainData.treeInstances = new TreeInstance[]{}; float mapHeight = terr.terrainData.heightmapHeight; float mapWidth = terr.terrainData.heightmapWidth; for(int j=0;j<mapHeight;j+=4) { int treeLine = Random.Range(50,100); for(int i=0;i<mapWidth;i+=4) { TreeInstance t = new TreeInstance(); if(terr.terrainData.GetHeight(i,j)>treeLine) { Vector2 noise = Random.insideUnitCircle*0.01f; t.prototypeIndex = 0; t.position = new Vector3(((float)i/mapWidth)+noise.x,0,((float)j/mapHeight)+noise.y); t.color = Color.white; t.lightmapColor = Color.white; t.heightScale = 1; t.widthScale = 1; terr.AddTreeInstance(t); } } Debug.Log (j); } //terr.terrainData.treeInstances.SetValue (t,terr.terrainData.treeInstances.Length-1); terr.Flush(); }

prototyper's toolbox

        Nothing special here, just a bunch of tools I've been using recently to do rapid prototyping.  It's such an alien concept to me to just throw a bunch of hardware and software into a blender and see what comes out, but it's pretty freakin fun.


Unity
Probably needs no explanation, really good platform for prototyping as it's open enough to do things that aren't necessarily games.


openFrameworks
Everything you need to create rich interactive applications under one roof. This is the model for what easy-to-use SDKs should be.


Processing
It's like an IDE for doing cool graphical stuffs. For you Pythonistas, check out pyprocessing as well.


vvvv
Just found out about this the other day, node based madness.


OpenNI
Definitive framework for natural interaction (gestures, etc). A bit of a learning curve, but good to know.
        So what are some of your favorite rapid prototyping tools?  And if you say UDK, i swear i WILL hunt you down and punch you in the heart...

from my cold, dead, hands...

        Well, ok maybe not quite that dark, but at least I look somewhat civilized when I'm using a Kinect, unlike well...


        Gah, ok ok, I promised myself i'd stop talking smack about former large software companies i may or may not have worked for.  I dunno it's weird, if you're like me, you always feel like once you leave a company in bad blood, you're forever professionally competing with them as long as you're in the same industry.  Not to put too fine a point on anything, or anything, hehe...

        Aaaanyway, so i feel like i'm at a good enough spot with KinectCam to drop some knowledge on some folks if you're interested in playing with this stuff yourself.  I gotta say I might have written Kinect off a bit early, I'm having a good time playing around with it and its ilk.  Granted, none of the games I've ever played on Kinect are doing the sort of stuff I've been doing, I sorta wonder what kind of crazy game you could come up with using the depth stream...Ok, sure, i'm just reaching there, but probably not, I bet someone (Jeff Minter) could come up with some insane depth stream game.  Actually I'm thinking of some sort of pattern recog app that i can feed my CD collection into or maybe just stuff the audio array, you know anything to avoid this sort of thing (Hmm, this would be funnier if you didn't have to squint for the text):



        Alright, well, mindless self indulgence aside, let's get to this MS Kinect SDK wrapper.  Odd how this will probably be the only such post I make that's going to apply to this particular wrapper, therein crawling another bug up my nose, the fact that I wish MS would just release an official Unity wrapper/plugin/whatever.  Truth be told though, it probably wouldn't be terribly hard to compile down my own DLL, it's probably more of a time issue to make sure I expose and marshall everything properly, not to mention proper SDK.  But SDK specifics aside, all the maths herein should be pretty easy to re-apply since it's just a few simple space conversion tricks.

        So recap from my previous post, make sure you install the following softwarez:

• Unity
• Kinect Beta SDK - Now that i'm out of demo land, I'm going to make a concerted effort to figure out release vs beta, there are some interesting jumping off points
• Kinect Wrapper
• Kinect Interface
• DirectX June 2010 SDK
• DirectX End-User Runtime - I feel now like this might be optional, but install it anyway if you're interested in doing independent Kinect/DX development

        I've alluded to the camera control thing for a while, so let's just dive right in and get that taken care of, then I'll do a separate post about the Kinect Wrapper guts.  Again, that probably won't be terribly interesting, mainly it's just for me.

        Alright, so to get stuff working in this environment is pretty simple.  Import the KinectWrapperPackage and you'll have all the scripts and objects you need to get started.  I recommend just starting with a blank scene and build up for there, altho you could pop open the KinectExample scene and make blobs dance around wildly.

        Once you have a new scene (taxing task that that is), all that's required to make it Kinect-aware is to drag a Kinect_Prefab from the Project into the scene, as so:


        This scene doesn't actually do anything yet, obviously, but you can use the Kinect_Prefab to setup your camera.  If you play the scene, you may notice the camera moving around a bit and focusing in odd places (hey buddy, my eyes are UP HERE), which can be fixed by setting some values in the KinectSensor attached to the Kinect_Prefab:


       I've been using the included Skeletal Viewer in the SDK to make sure i'm getting good coverage, once you find some numbers that work, you can always go into the KinectSensor code and set your own defaults if you want.

       Alrighty then, time to get the camera moving.  So, I went the hack route and used Unity's built-in MouseLook script as a jumping off point, as it only required a few minor changes to the camera transform code.  The part that requires a bit of work is getting Kinect skeletal numbers into Unity friendly numbers, and when i say "bit", i mean it, it's actually not that hard.  So the first thing I did was for my own sanity, and that's setup a little GUI indicator that tells me if I'm tracking or not.  Polling the camera for the depth image in this wrapper is a bit annoying because you have to restart Unity every time you stop running.  Might be a fun project to figure that out and re-implement depth and color display as a GUI texture.  Something to keep in mind for my next project.  Anyway, so verifying tracking is pretty simple, i put this all in a script called FromBonePosition:

public class FromBonePosition : MonoBehaviour { public SkeletonWrapper sw; void Update() { if (sw.pollSkeleton()) { if (sw.trackedPlayers[0] > -1 || sw.trackedPlayers[1] > -1) { tracking = true; } else { tracking = false; } } } void OnGUI() { if (tracking) { GUI.Label(new Rect(0, 0, 200, 20), "Tracked"); } } }
        Pretty straightforward, we're just pulling some functionality from the included SkeletonWrapper class, which...well, wraps Skeleton functionality.  Truth be told, I feel like this class is a bit of an extra, but whatever.  Anyway, breaking down the code:

• Really all pollSkeleton() does is call the native NuiSkeletonGetNextFrame(), i'll leave what that probably does as an exercise to the reader.
• trackedPlayers[] is interesting.  The details of how we get there are a bit unimportant at this stage, the important note is that we're grabbing an enum that tells us what we're tracking.  KinectWrapper initializes the values in trackedPlayers[] to -1 which indicates that we haven't even acquired a skeleton.  The possible enums are NUI_SKELETON_NOT_TRACKED, NUI_SKELETON_POSITION_ONLY, and NUI_SKELETON_TRACKED.  So if either of the two players is acquired, we can say we've tracked.
• So once we've tracked, we spit out a string to the UI that tells us what's up.  

        Easy-peasy, yeah?  Well fear not, most of the Kinect polling is that straightforward, it's just flipping the numbers that takes some code.  So let's continue by polling the Kinect for a bone position, which we'll use to drive our camera.  Yeah, yeah, i know, classic Kinect "Hello, World!", recreate a mouse driver, but it's an easy place to start.

        Joints are pretty easy to poll, as the SkeletonWrapper class provides a few different arrays that contain joint information and the SDK provides us a convenient enum for specifying which joint to query for data.  So let's add some code to our class:

public class FromBonePosition : MonoBehaviour { private Vector3 rHandPos; public SkeletonWrapper sw; void Update() { if (sw.pollSkeleton()) { if (sw.trackedPlayers[0] > -1 || sw.trackedPlayers[1] > -1) { tracking = true; rHandPos = sw.bonePos[0,(int)Kinect.NuiSkeletonPositionIndex.WristRight]; } else { tracking = false; } } } void OnGUI() { if (tracking) { GUI.Label(new Rect(0, 0, 200, 20), "Tracked"); GUI.Label(new Rect(0, 20, 200, 20), rHandPos.ToString()); } } }
        Simple, efficient (like the body itself), we add a variable to hold the right hand's position, then we pull a value from the bonePos[] array on the SkeletonWrapper.  You'll notice bonePos[] is a 2D array, the first id is which skeleton to poll (we might be tracking two players), the second id is pretty obviously the joint we want to get the position for.  You can see the full enum definition in the KinectInterop class if you should ever want to poll a different joint, or if you want to update a bunch of joints, or a whole skeleton, or make tea, or something like that.  We output that number to the GUI too, a) for verification purposes and b) so we can set some other numbers later.

       By now we pretty much know all the kinect stuff we need to know to get our camera moving, so let's get our camera look on.  Here's our member list (with comments!):

public class FromBonePosition : MonoBehaviour { //position from kinect private Vector3 rHandPos; //kinect to unity mouse input space private Vector2 inputXY; //final rotation private Vector2 rotationXY; //hand position min and max //we use these to figure out our screen conversion public Vector2 OldX; public Vector2 OldY; //how much do we move between sample points public float MoveStep = 1.0f; //camera move sensitivity public Vector2 DampingXY = new Vector2(1f,1f);
        And now...Maths!  The basic theory behind what we're doing is pretty simple:

• Find the distance (as a percent) between the minimum and maximum x (oldX) that we get from the Kinect
• Repeat for y
• Cast that into (-1,1) so it mimics Unity's mouse input values
• Rotate the camera
• Profit, or at least impress your friends (or your mom)

        This is pretty simple, we can use some of the Unity built-ins to do all the scary maths for us, i'll just post the relevant code here:

//this goes in your Update() //after you've pollSkeleton()'d //you could probably SmoothStep this too //i got good enough results with Lerp rHandPos = Vector3.Lerp(rHandPos, sw.bonePos[0, (int)Kinect.NuiSkeletonPositionIndex.WristRight], Time.deltaTime*MoveStep); inputXY.x = Mathf.SmoothStep(-1f,1f,Mathf.InverseLerp(OldX.x, OldX.y, rHandPos.x); inputXY.y = Mathf.SmoothStep(-1f,1f,Mathf.InverseLerp(OldY.x, OldY.y, rHandPos.y); rotationXY.x = transform.localEulerAngles.y; rotationXY.x += inputXY.x*DampingXY.x; rotationXY.y += inputXY.y*DampingXY.y; transform.localEulerAngles = new Vector3(-rotationXY.y, rotationXY.x, 0);        
        So yeah, we're basically parroting MouseLook's rotation setting and replacing Input.GetAxis() with the data we poll from the Kinect.  This SHOULD work, as it's almost verbatim the code I was using minus things like aim assist and project specific passing.  Play around with the OldX and OldY values, the Kinect SDK says that values for X can lie between (-2.2, 2.2) and Y can be (-1.6,1.6), but that's going to vary a bit depending on where you're hanging out.  For example, the values we ended up using were OldX = (-0.3,0.7) and OldY = (1.3,2.4).  Drop a texture onto the gui and set the rectangle based on hand position, that's a pretty simple way to get some good debug.  It's not too tricky to cast your value into screen space, try it out!
        I think that's all I got for now, going to try and port this to OpenNI/PrimeSense just for fun this weekend too.

sound advice

        Hey gimme a break, i went to bed at like 1 last night and i've been in a bit of crunch this week.  I keep saying that one of these days I'm going to start a job during a project that's NOT in crunch, but you know, it's a fun crunch.  Had the first dog-n-pony with the execs and I gotta say our team is working on some cool shit.  We got to postpone our demo for a week, which is good because I've only been in the code for about a week and a half, and the scope of our demo is pretty impressive.  I mean, not really, but given the time we've had to put it together...


It seemed like a manageable project at the time..!
        So I'm still working on the Kinect camera tutorial, I'm actually going to go in and tighten up the implementation, along with the graphics on level 3, so I'll have a bit of a better presentation.  It's a bit sad that I didn't have more time to really come up with some cool camera control, was hoping to do a little more than just build a mouse driver type thing, but you know, it's a good starting point.  I feel like the implementation I came up with is pretty solid, it's a good hybrid of a few different paradigms that make good sense.  Anyway, hopefully that's a fairly tantalizing preview.  I'll go over the basics of how to setup the zigFu/OpenNI stuff too so you can further hack away on Kinect.  If you haven't messed around with it yet, you should, I'm surprised it took me this long to get into it.


Thompson Eye Phone and Sogo-7s optional...
        Alright, so to some possibly meatier content, although I may be the only person who's run into this issue.  If that's the case, you guys are all jerks for not posting solutions!!  One of the things we've been beating our heads against somewhat for the last couple days is how to manage sound, in particular sound on events, such as collision.  Now, that's a fairly straightforward problem that you could solve in a few different ways, but here are some ways NOT to solve it:

• The rather naive and seemingly obvious audio.Play() in Update(), it only seems obvious at very first glance
• Immediately prior to a seemingly obvious Destroy() call.  Again, it only seems obvious at very first glance

        So, the first method we came up with was a bit brute force I'll admit, but if you know me, that probably comes as no surprise.  We setup a GameObject that's basically an audio container, so actually it's a GameObject and a script that exposes a bunch of public AudioClips.  You probably see where I'm going with this, if you guessed AudioSource.PlayOneShot() or AudioSource.PlayClipAtPoint(), pour yourself an expensive single malt shot.  For some reason, this made the audio sound really wonky, and it wasn't a 3d issue or a playing multiple clips issue.  We could have wrapped either one of these to get a bit more control over the AudioSource, but I felt like that was probably overkill.  For a larger project, I could see the benefit of this though, and it's definitely something I'll be exploring more.  I have a ton of questions about this pattern, performance issues mainly.


Brute Force? Heh, more like ME Force, amirite??
        The second method is about as straightforward, but I had some perf concerns again.  It's funny how I'm still in game developer mode and trying to make the transition to blue sky R&D developer mode.  Performance? Hah, we care not for your framerates and memory budgets!!  Or something...But yeah, basically we ended up attaching multiple AudioSources to the prefab and managing it in script.  The code was pretty simple, something to this effect:

private AudioSource spell_1; private AudioSource spell_2; void Start() { foreach(AudioSource a in GetComponents<AudioSource>()) { if(a.clip.name=="spell_1") spell_1 = a; else if (a.clip.name=="spell_2") spell_2 = a; } } void OnCollisionEnter(Collision collision) { if(condition_1) spell_1.Play() else if(condition_2) spell_2.Play() }
        Obviously this is super naive and you'd want to do a bunch of other checks, but you get the idea, it's sketch code, whadyawant? I'm not very sanguine on this method largely due to the multiple AudioSources on each prefab, again, perf concerns. I may just be paranoid still...

        The solution we ended up going with is not ideal but serves as a bit of a springboard for a pattern I might have used if i had been thinking about it.  For each collision behavior, we exposed an AudioClip for that behavior, then switched out the object's AudioClip in the OnCollisionEnter() based on the name of the colliding object, i.e. We have an object in the world and based on the projectile's name, we swap out the AudioClip and play the AudioSource.  Again, i don't feel like this is super ideal because then we have assets all over the place, but it definitely made life easier for me because I could just concentrate on a behavior at a time. So we're looking at something like this:

private bool testCondition = true; public AudioClip soundClip; void OnCollisionEnter(Collision collision) { ContactPoint contact = collision.contacts[0]; GameObject other = contact.otherCollider.gameObject; if(other.name=="TheProjectile(Clone)") { testCondition = !(testCondition); audio.clip = soundClip; } if(audio&&(!audio.isPlaying)) { audio.Play(); } }
        Again, super-naive, there are better ways to do the specifics, but you get the idea.  This seems to be working for now, so i'm going to just STFG and keep in mind what I've learned.


        I think, in retro, I would have gone with some method that was a hybrid of all of these, maybe have an audio manager type script attached to the prefab that exposed all the AudioClips i wanted to play then have each collision behavior poke into the audio manager.  I like the idea of keeping collision behaviors as separate script just because i hate looking at big scripts that manage everything.  That could just be me, honestly, I ain't to guud at readin teh codez, not bein a real programmer and all...

        Thoughts?  I'd love to hear from anyone else about how you tackle this sort of thing.  At some point i want to go back into library/SDK developer mode and start wrapping up a bunch of functionality like this for future Unity projects...