Interaction trackers and indicators

As mentioned in previous sections related to interactions, OctoXR's interaction system uses several interfaces whose implementers are used for obtaining scene objects the Interactor can hover and interact with and also for indicating whether the hovering and interactions with obtained objects can start/continue:

IInteractionHoverTracker

Interface that requires implementation of one method:

• void GetInteractionHoveringObjects(ICollection<Interactable> collectionToAddResultsTo) - Method that is used for retrieving all Interactable objects that could be hovered by an Interactor. Resulting Interactables should be added to the collection provided as the method argument. Implementers are not required to clear the specified collection first.

Hover trackers should simply return Interactables that could be hovered based on some criteria, hover trackers are usually implemented in OctoXR as objects that use various configurable spatial queries (ray cast, sphere cast, sphere overlap...) to obtain the objects that have a specific type of Interactable components attached.

IInteractionHoverIndicator

Defines one method:

• bool IsInteractionHoveringObject(Interactable interactable) - Returns true to indicate that the provided Interactable can be hovered by an Interactor.

Objects that implement this interface can be used by Interactors to indicate whether Interactors can actually start hovering an Interactable, as well as whether they can continue hovering them.

IInteractionActivityIndicator

Defines one method:

• bool CanInteractionWithObjectStartOrContinue(Interactable interactable) - Returns true to indicate that the provided Interactable can be interacted with.

Objects that implement this interface can be used by Interactors to indicate whether Interactors can actually start interacting with an Interactable, as well as whether they can continue interacting with them.

Many objects in OctoXR simultaneously implement all three of these interfaces, so the same object types/instances can be used in multiple roles by Interactors.

SpatialQueryBasedInteractionTracker

Component in OctoXR that implements all three previously described interaction interfaces. It performs spatial queries in order to track Interactables that can be hovered/interacted with. There is an abstract generic version of this component defined as SpatialQueryBasedInteractionTracker<TInteractable> whose type parameter is constrained to be a class inheriting from Interactable - this tracker will track only Interactables of a specific type, the one specified by type parameter. Classes that derive from this type of tracker are not required to implement anything.

SpatialQueryBasedInteractionTracker offers many properties in order to configure it, the main ones being a Transform reference that defines an origin of the spatial query, whether the spatial query should consider trigger colliders or not, the layers against which the queries are performed as well as the exact type of spatial query to perform. Spatial queries that can be used are:

• Ray Cast (Single Result) - Performs a ray cast that obtains a single resulting object. Direction of the ray is specified via an assignable Transform component's world-space forward vector.

• Ray Cast (Multi Result) - Performs a ray cast that can return multiple objects. Direction of the ray is specified via the same means as with single-result ray cast.

• Sphere Cast (Single Result) - Performs a sphere cast that returns a single object.

• Sphere Cast (Multi Result) - Performs a sphere cast that can return multiple objects.

• Sphere Overlap - Performs a sphere overlap query.

• Box Cast (Single Result) - Performs a box cast that returns a single object.

• Box Cast (Multi Result) - Performs a box cast that can return multiple objects.

• Box Overlap - Performs a box overlap query.

• Capsule Cast (Single Result) - Performs a capsule cast that returns a single object.

• Capsule Cast (Multi Result) - Performs a capsule cast that can return multiple objects.

• Capsule Overlap - Performs a capsule overlap query.

All stated spatial queries return Interactable components that are attached to the same or parent objects of the colliders that are hit by them. Returned Interactables can also be sorted based on distance of the hit colliders to a desired assigned Transform component (usually the same Transform used as the origin of the query), from the closest to the furthest away or vice-versa and the distance function to use can also be configured. The number of Interactables returned by queries can also be limited to any desired value. Note that the last two functionalities are only relevant to multi-result queries, since the single-result queries will always return just one result anyway.

OctoXR has two implementations of SpatialQueryBasedInteractionTracker<TInteractable>. There is a non-generic SpatialQueryBasedInteractionTracker defined as:

• public class SpatialQueryBasedInteractionTracker : SpatialQueryBasedInteractionTracker<Interactable>

This tracker will track Interactables of any type returned from its spatial queries. However, this is often not very practical as often times there is a need to track only Interactables of one specific type (or types derived from a specific type).

There is another spatial query-based interaction tracker that tracks only GrabInteractables:

• public class SpatialQueryBasedGrabTracker : SpatialQueryBasedInteractionTracker<GrabInteractable>

This tracker is the most commonly used spatial query-based interaction tracker throughout OctoXR.

HandPoseBasedInteractionIndicator

A class in OctoXR that implements IInteractionHoverIndicator and IInteractionActivityIndicator interfaces. It evaluates whether an Interactable can be hovered/interacted with based on the current pose of a hand. Hand pose is read from the Hand Pose Provider property that should be assigned. It is an abstract class that requires implementation of one method:

• protected abstract bool EvaluatePoseWithInteractable(Interactable interactable) - This is called when hover and/or interaction activity is evaluated on HandPoseBasedInteractionIndicator. Return value indicates whether hover/interaction is active. Implementers should evaluate hand pose in some way here to determine the activity of hover/interaction with the provided Interactable.

There is currently one concrete implementation of this class in OctoXR:

HandPoseBasedGrabIndicator

Evaluates the hover/interaction activity based on whether the current hand pose returned from its Hand Pose Provider is recognized as grabbing pose - pose that could best be described as being similar to closed fist. It will always evaluate to false if the Hand Pose Provider property is not assigned.

HandFingerPinchInteractionIndicator

A class in OctoXR that implements IInteractionHoverIndicator and IInteractionActivityIndicator interfaces. It evaluates whether an Interactable can be hovered/interacted with based on whether a pinch gesture is being performed with a specific finger of a hand. Finger pinch is detected using an Input Provider that can be assigned and must be of HandInputDataProvider type. There is also another property called Finger of HandFinger enumeration type that can be used to specify the finger that should be pinching.

UnityXRControllerButtonPressInteractionIndicator

A class in OctoXR that implements IInteractionHoverIndicator and IInteractionActivityIndicator interfaces. It evaluates whether an Interactable can be hovered/interacted with based on whether a specific button on a XR controller is pressed. Button press is detected using an Input Provider that can be assigned and must be of UnityXRControllerInputDataProvider type. The specific button that should be pressed is specified via Button property.

UnityXRControllerButtonTouchInteractionIndicator

A class in OctoXR that implements IInteractionHoverIndicator and IInteractionActivityIndicator interfaces. It evaluates whether an Interactable can be hovered/interacted with based on whether a specific button on a XR controller is being touched. Button touch is detected using an Input Provider that can be assigned and must be of UnityXRControllerInputDataProvider type. The specific button that should be touched is specified via Button property.

GrabPointInteractionTracker

GrabPointInteractionTracker is a component that tracks the grab points of a GrabInteractable that is hovered and/or interacting with the Interactor the GrabPointInteractionTracker is associated with and manages the behavior of interacting hand in a way that should drive the pose of the interacting hand to a pose that is specified by the current active grab point. This assumes that the GrabPointInteractionTracker is setup correctly.

GrabPointInteractionTracker implements only IInteractionActivityIndicator interface, it should be set as Interaction Start Indicator of the Interactor the GrabPointInteractionTracker is associated with - the one that is attached to the same object the GrabPointInteractionTracker is attached to (this Interactor is required by GrabPointInteractionTracker). Usually this would be done by setting GrabPointInteractionTracker as one of the interaction activity indicators in the Logically Composed Interaction Start Tracker where the other interaction activity indicator would be set to any other preferred value (e.g. HandPoseBasedGrabIndicator) with logical operation performed between the two indicators set to And.

GrabPointInteractionTracker drives the interacting hand pose to target poses defined by active grab points indirectly - using several HandSkeletonPoseProviders that should be assigned to it, with one of those HandSkeletonPoseProviders set as Pose Provider for the interacting hand.

Properties defined by GrabPointInteractionTracker include:

• Tracked Hand Pose Provider - HandSkeletonPoseProvider that is assumed by GrabPointInteractionTracker to provide the actual hand pose detected by a tracking system in the back-end. Usually this would be set to an InputDataProvider instance such as HandInputDataProvider.

• Posable Hand Wrist Pose - Transform component of the wrist joint of the hand that gets its pose driven to target grab point poses.

• Posable Hand Pose Provider - reference to an OverrideableHandSkeletonPoseProvider. This HandSkeletonPoseProvider will have its override pose provider set by GrabPointInteractionTracker to the assigned Writeable Pose Provider in order to make the OverrideableHandSkeletonPoseProvider provide hand pose that is written by GrabPointInteractionTracker based on how the detected active grab points are configured. This pose provider would usually be set as Pose Provider for the hand that gets its pose driven to target grab point poses.

• Writeable Pose Provider - reference to a WriteableHandSkeletonPoseProvider. This HandSkeletonPoseProvider is used by the GrabPointInteractionTracker to write the target grab point poses to. When the GrabPointInteractionTracker determines that the interacting hand pose should be controlled by it (based on the presence of nearby grab point), it will set this pose provider as override pose provider to the assigned Posable Hand Pose Provider and write the target hand poses to it. GrabPointInteractionTracker will also restore all the changed state of the pose providers involved when needed, e.g. when there are no more active grab points detected by it.

There are several events the GrabPointInteractionTracker raises when various phases in its process are reached:

• On Grab Point Found - Invoked when GrabPointInteractionTracker has detected nearby grab point that the interacting hand pose should be affected by.

• On Grab Point Grabbed - Invoked when the Interactor associated with the GrabPointInteractionTracker starts interacting with the GrabInteractable at the current tracked grab point that is associated with the GrabInteractable.

• On Grab Point Released - Opposite of the previous; invoked when the Interactor associated with the GrabPointInteractionTracker stops interacting with the GrabInteractable at the current tracked grab point that is associated with the GrabInteractable.

• On Grab Point Lost - Invoked when GrabPointInteractionTracker has lost track of the last grab point it was tracking and the interacting hand no longer has to have its pose manipulated by GrabPointInteractionTracker.

Composite trackers and indicators

There are several interaction trackers and indicators in OctoXR that can be composed from other non-composite and composite trackers/indicators in some way. These can be used to combine results of the trackers/indicators they are composed from in some way to produce the final tracker/indicator results.

InteractableSetComposedInteractionHoverTracker

Class that implements IInteractionHoverTracker interface, it appears in the Interactor's Hover Tracker property drop-down as Set Composed option. It consists of two IInteractionHoverTracker properties, Hover Tracker 1 and Hover Tracker 2, that can be used to assign any pair of hover trackers. It also contains property called Op that can be used to select a binary set operation to perform on the results of the assigned Hover Tracker 1 and Hover Tracker 2 in order to obtain the final list of resulting Interactables that the InteractableSetComposedInteractionHoverTracker returns. Operations that can be performed are:

• Union - Hover tracker returns unique Interactables from both of its child hover trackers.

• Intersect - Hover tracker returns only those Interactables that are contained in results of both of its child hover trackers.

• Except - Hover tracker returns Interactables that are contained in the results of its Hover Tracker 1, but not contained in the results of its Hover Tracker 2.

• SymmetricExcept - Opposite of Intersect; hover tracker returns Interactables that are contained in either the results of Hover Tracker 1 or results of Hover Tracker 2, not the ones contained in both.

LogicallyComposedInteractionHoverIndicator

Class that implements IInteractionHoverIndicator interface, it appears in the Interactor's Hover Start/End Indicator property drop-down as Logically Composed option. It consists of two IInteractionHoverIndicator properties, Hover Indicator 1 and Hover Indicator 2, and a property named Op that specifies the binary logical operation to perform between the results of Hover Indicator 1 and Hover Indicator 2 in order to produce the final result of the LogicallyComposedInteractionHoverIndicator. Operations that can be performed are all binary logical operations - And, Or, Exclusive Or (Xor).

InvertedInteractionHoverIndicator

Class that implements IInteractionHoverIndicator interface, it appears in the Interactor's Hover Start/End Indicator property drop-down as Inverted option. It contains a single IInteractionHoverIndicator property that can be assigned - Hover Indicator. InvertedInteractionHoverIndicator produces result by taking the result of the assigned Hover Indicator and simply inverting it (negating it) - flipping from true to false and vice-versa.

LogicallyComposedInteractionActivityIndicator

Class that implements IInteractionActivityIndicator interface, it appears in the Interactor's Interaction Start/End Indicator property drop-down as Logically Composed option. It consists of two IInteractionActivityIndicator properties, Interaction Activity Indicator 1 and Interaction Activity Indicator 2, and a property named Op that specifies the binary logical operation to perform between the results of Interaction Activity Indicator 1 and Interaction Activity Indicator 2 in order to produce the final result of the LogicallyComposedInteractionActivityIndicator. Operations that can be performed are all binary logical operations - And, Or, Exclusive Or (Xor).

InvertedInteractionActivityIndicator

Class that implements IInteractionActivityIndicator interface, it appears in the Interactor's Interaction Start/End Indicator property drop-down as Inverted option. It contains a single IInteractionActivityIndicator property that can be assigned - Interaction Activity Indicator. InvertedInteractionActivityIndicator produces result by taking the result of the assigned Interaction Activity Indicator and simply inverting it (negating it) - flipping from true to false and vice-versa.

By using composite hover trackers, hover indicator and interaction activity indicators it is possible to create as complex hover/interaction trackers/indicators as needed to determine the state of hovering and interactions between Interactors and various Interactables.