Hello, everyone. Meet again. I'm Jun Quan.

As a gesture recognition device, Leap Motion has the advantage of accuracy compared with Kniect.

In the development of my graduation project "Scene Rover". The gesture control of Leap Motion is an important link. In this way, let's talk about the implementation of gesture recognition using Leap Motion in development and what we need to pay attention to.

1, Evaluate the ability of Leap Motion

Before setting the gesture. We must know to what extent Leap Motion can achieve, so as not to find it very difficult to achieve after setting the scheme.

This assessment relies on the actual experience of using the device. Mainly from three aspects:

1. Visual gesture recognition interface provided by leap motion

SDK document description

3. APP in leap store

Basically, it can be concluded that:

1.Leap Motion recognition can better recognize horizontal direction or horizontal direction based gestures.

2. There will be errors in the recognition of fist clenching or vertical behavior. Such errors are related to detailed gesture behavior.

3. Don't rely too much on high accuracy. It's right that Leap Motion can detect the millimeter level, but sometimes it will recognize your straight fingers as curved. So prepare for the worst.

2, Actual needs

Move, rotate, click button, zoom and rotate objects, close programs and pause. The main functional requirements are as follows.

There are some principles:

1. Gestures in the same environment should be close and convenient for conversion. The transition between rotation and movement should be designed very naturally.

2. Gestures avoid conflict. It's not a good thing that gestures are too similar.

For example, three straight fingers and four straight fingers should not be designed as two gestures. Of course, this is not absolute. Suppose you make a slow motion and the motion is facing the Leap Motion camera, you should believe it at this time. At least do a separate test for this gesture.

3, Consider the main data structure and the outline of the algorithm

The SDK of Leap Motion was browsed in the first part. At least know the information that Leap Motion can include. From the SDK point of view, this is very rich. Since you design your own gestures, you'd better not rely on the cool gestures of the SKD development kit. Most likely, these gestures are only official for demonstration or show off. Designing the basic data structure of gesture by yourself also has other advantages. For example, the somatosensory device is replaced, but the function is similar. At this time, you only need to change the way to obtain data (from one SDK to another SDK), without changing the algorithm.

The contour of the algorithm has a great relationship with the basic data. Therefore, the data structure must be as concise as possible and agree to change (maybe an algorithm occupies the decisive factor, but it was not considered at the beginning).

public class HandAndFingersPoint : MonoBehaviour 
{
	const int BUFFER_MAX=5;
	Controller m_LeapCtrl;

    <span style="white-space:pre">	</span>public E_HandInAboveView m_AboveView = E_HandInAboveView.None;
    
	//Finger data, [0] for left hand, [1] for right hand
	private Dictionary<Finger.FingerType,FingerData>[] m_FingerDatas = new Dictionary<Finger.FingerType, FingerData>[2];
	//buffer,[0] represents the left hand, [1] represents the right hand, [, n](n belongs to 0,3. It represents the nth cache)
	private Dictionary<Finger.FingerType,FingerData>[,] m_FingerDatasBuffer=new Dictionary<Finger.FingerType, FingerData>[2,BUFFER_MAX];
	private int m_CurBufIndex=0;
	//palm 0: left hand and 1: right hand
	private PointData[] m_PalmDatas = new PointData[2];
	
	private readonly PointData m_DefaultPointData = new PointData(Vector.Zero, Vector.Zero);
        private readonly FingerData m_DefaultFingerData = new FingerData(Vector.Zero,Vector.Zero,Vector.Zero);

The rest of the HandAndFingersPoint class is the methods of filling, clearing and refreshing the data. E_HandInAboveView records which hand first enters the field of view of Leap Motion. Used to set priorities. The other two main data structures are PointData and FingerData:

//The data of a finger includes a fingertip data and the position data of the finger root bone
public struct FingerData
{
    public PointData m_Point;//Position and orientation of fingertips
    public Vector m_Position;//The position of the root bone of the finger is the position of the proximal phalange of the Proximal phalanges for the thumb

    public FingerData(PointData pointData, Vector pos)
    {
        m_Point = pointData;
        m_Position = pos;
    }

    public FingerData(Vector pointPos, Vector pointDir, Vector pos)
    {
        m_Point.m_Position = pointPos;
        m_Point.m_Direction = pointDir;
        m_Position = pos;
    }

    public void Set(FingerData fd)
    {
	m_Point = fd.m_Point;
	m_Position = fd.m_Position;
    }
}
//Data of a point, including direction and position
public struct PointData
{
    public Vector m_Position;//position
    public Vector m_Direction;//direction

	public PointData(Vector pos,Vector dir)
	{
		m_Position = pos;
		m_Direction = dir;
	}

	public void Set(PointData pd)
	{
		m_Position = pd.m_Position;
		m_Direction = pd.m_Direction;
	}

	public void Set(Vector pos,Vector dir)
	{
		m_Position = pos;
		m_Direction = dir;
	}
}

//First seen hand
public enum E_HandInAboveView
{
    None,
    Left,
    Right
}

After the basic data is defined, you'd better confirm that the data filling is OK. Actually through frame = leap Controller. Frame(); To get the latest data.

At this time, there is no hurry to write methods related to basic data. Now what we finally want is the rationality of gesture algorithm. To infer whether it is reasonable, it is best to write an algorithm first.

The simplest is the palm extension gesture. In the control, the horizontal palm extension is used for roaming and the vertical palm extension is used for pause. I found that the palm depends on the finger, and the finger contains two states - straight and curved.

In addition, other gestures are also the straightening or bending of fingers, plus the determination of direction to accumulate various effects. Of course, the bending and straightening judgment algorithm of fingers should be written separately:

/// <summary>
///This method provides an algorithm for single finger matching, such as straightening. bend
///Possible changes in the future: the requirements may be different for different scenarios. The threshold here may change accordingly
/// </summary>
public class FingerMatch
{
	//Angle threshold of bending state
	static readonly float FingerBendState_Radian = Mathf.PI*4f / 18 ;//40 degrees
	//Angle threshold of straightening state
	static readonly float FingerStrightState_Radian = Mathf.PI/12;//15 degrees

	/// <summary>
	///When the deviation between the direction and direction of the root bone fingertip is less than the threshold, the finger is judged to be in the straight state.
	///Note that the invalid direction is a zero vector. First determine the zero vector
	/// </summary>
	///< param name = "adjustborder" > fine tuning of threshold < / param >
	/// <returns></returns>
	public static bool StrightState(FingerData fingerData, float adjustBorder=0f)
	{
		bool isStright =false;
		Vector disalDir = fingerData.m_Point.m_Direction;
		//Assume that the fingertip direction is a 0 vector, indicating invalid data
		if (!disalDir.Equals(Vector.Zero)) 
		{
			Vector fingerDir = fingerData.m_Point.m_Position - fingerData.m_Position;//The vector from the finger root to the fingertip by subtracting the finger root from the fingertip position	        
			float radian = fingerDir.AngleTo(disalDir);
			
			if (radian < FingerStrightState_Radian + adjustBorder)
			{
				isStright = true;
			}
		}
		return isStright;
	}

	/// <summary>
	///Infer whether a finger is bent
	/// </summary>
	///< param name = "fingerdata" > finger data to be judged < / param >
	///< param name = "bandorder" > bending threshold < / param >
	/// <returns></returns>
	public static bool BendState(FingerData fingerData, float adjustBorder=0f)//,out float eulerAugle)
	{
		bool isBend = false;

		//eulerAugle = -1f;
		Vector disalDir = fingerData.m_Point.m_Direction;
		if( !disalDir.Equals(Vector.Zero) )
		{
			Vector fingerDir = fingerData.m_Point.m_Position - fingerData.m_Position;//The fingertip position minus the finger root position refers to the vector following the fingertip

			float radian = fingerDir.AngleTo(disalDir);
			//eulerAugle = radian*180/Mathf.PI;	
			//When the included angle exceeds the defined threshold, it is considered as bending state
			if (radian > FingerBendState_Radian + adjustBorder)
			{
				isBend = true;
			}
		}

		return isBend;
	}

}

The above includes an important concept - threshold. It describes to what extent the narration is straight and what extent it is curved. The determination of threshold value needs actual test.

It's also time for a simple test. After all, the outline of the algorithm has been determined. I didn't even write a judgment algorithm for palm extension. Just make sure it's feasible.

Operations related to basic data structure - HandAndFingersPoint class: Source GitHub link

This class uses basic data. Executing in Unity Editor will show the outline of a palm, and blue indicates the direction of the finger. Red indicates the connection from the finger bone root to the palm and fingertip, and yellow indicates the connection from the palm to fingertip:

4, A brief summary of gesture implementation

Other codes can be in my GitHub:Leap Motion In Unity3D warehouse obtain. In the implementation of gestures, some small skills are also included. For example, for the matching of actions, we should prevent the error caused by the trembling of fingers. Use discrete data sampling - sampling at regular intervals.

How to use and observe these scripts: you can put these scripts in a GameObject. Through Leap Motion, you can see that the attributes of the script will change when the matching is successful. In addition, the script includes the registration function of events, in other words. The external can register an event with any gesture, so that some additional processing can be done when the gesture completes matching or reaches a matching state. These scripts can not directly complete our requirements, such as pause. We need to make further restrictions on these gesture states or actions, such as setting the vertical forward palm as pause and the horizontal palm as translation according to the direction of the palm.

Publisher: full stack programmer, stack length, please indicate the source for Reprint: https://javaforall.cn/115406.html Original link: https://javaforall.cn