DETAILED ACTION
This office action is responsive to communication(s) filed on 5/2/2024.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Foreign Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested: Three-Dimensional Hand Keypoint Gesture Tracking for Virtual Object Trajectory Simulation and Augmented Reality Display.
Claim Interpretation
The term “hand key points” have not been defined in the Instant Specification, and are herein broadly interpreted as including location reference points of a hand, e.g., joints/fingertips, used to track movement of a hand. Instant Specification Abstract.
Claims Status
Claims 1-15 and 17-21 are pending and are currently being examined.
Claims 1, 17 and 18 are independent.
Claim 16 is canceled.
Claim Objections
Claims 17 and 18 are objected to because of the following informalities:
Claims 17 and 18 improperly utilize the gerund form (“-ing”) following the introductory phrases “cause the electronic device” and “computer to:” To correct this grammatical error, the introductory phrases should be amended or the functional limitations should be amended to the infinitive form (e.g., “recognize,” “determine,” and “simulate”).
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 6, 8, 10-12, 15 and 17-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benson; Nicholas James (hereinafter Benson – US 20190362562 A1) in view of Pastrana Vicente; Israel et al. (hereinafter Pastrana – US 20230100610 A1).
Independent Claim 1:
Benson teaches:
A virtual object display method, comprising:
recognizing, according to three-dimensional coordinates of a hand key point, a trigger gesture of throwing a virtual object from hand images, (The system constructs a 3D model of the hand from video images using capsule elements, allowing for the tracking of hand positions in 3D sensory space to detect real motions, such as a throw ¶¶ 252 and 376. By recognizing the 3D trajectory of these hand points, the system identifies a trigger gesture that manipulates virtual objects, ¶ 376. The tracking is done on the basis of certain points of interest in of the hands [hand key point], e.g., fingertips, ¶ 262. The term “hand key points” has not been defined in the Instant Specification, and is herein broadly interpreted as including location reference points of a hand, e.g., joints/fingertips, used to track movement of a hand. Instant Specification Abstract)
wherein the hand images comprise at least two consecutive frames of first hand images in which the hand key point is relatively stationary, (tracking is by using successive video images [consecutive frames], ¶¶ 122, 140-141,148, 252 and 309, wherein the hands are detected as being relatively stationary –– object has been grasped and user is preparing to throw it, ¶¶ 137 and 384 and ¶¶ 49-54, e.g., in a resting pose, ¶ 140. In these paragraphs, the system maintains a "grabbed" state based on sustained curl metrics, ensuring a throw gesture is registered only after and while the object remains held/grabbed rather than released, ¶¶ 137 and 384).
at least one frame of a second hand image in which the hand key point moves relative to the first hand images […] is the trigger gesture; (a throwing gesture is tracked and recognized when the user has prepared to throw–is grasping the object–and has performed throwing motions, ¶¶ 137 and 384, and tracking is by using consecutive video images [consecutive frames], ¶¶ 140-141 and 252)
in response to the trigger gesture, determining a throwing parameter according to the hand images; (parameters of the object/projectile, such as velocity and direction, are calculated based on the motion reflected in the video images of the hand(s), ¶¶ 252 and 384)
and simulating a motion trajectory to throw the virtual object according to the throwing parameter, and displaying the virtual object in an augmented reality (AR) scenario according to the motion trajectory. (the virtual object is displayed as traveling along a computed trajectory [simulating a motion trajectory] based on velocity and direction information, ¶ 384. the throwing gesture recognition is for virtual or augmented reality, Abstract and ¶¶ 42 and 48)
Benson further teaches that a user can throw an object, ¶ 384. Benson does not appear to expressly teach, but Pastrana teaches:
and a gesture in at least one of the first hand images and the second hand image (a throw gesture includes an initial grab of an object, then releasing the object at the end of the movement, ¶ 274).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Benson to include and a gesture in at least one of the first hand images and the second hand image, as taught by Pastrana.
One would have been motivated to make such a combination in order to maintain a realistic experience by detecting known portions of a throwing gesture, including the release of the object at the end of the throw motion, Pastrana ¶ 274 and Benson ¶ 106.
Claim 2:
The rejection of claim 1 is incorporated. Benson further teaches:
The method of claim 1, wherein recognizing, according to the three-dimensional coordinates of the hand key point, the trigger gesture of throwing the virtual object from the hand images, comprises:
calculating, based on a set angle of field of view, the three-dimensional coordinates of the hand key point in the hand images under a camera coordinate system; (system where optical sensors capture hand motion, using ray tracing, surface tangents, and normal vectors to align a 3D capsule-based model at specific points “P” on the hand surface to calculate its three-dimensional position and shape ¶¶ 50, 248 and 252, and this 3D model is subsequently used to track hand movement and gesture trajectory over time within a 3D sensory space, ¶ 214 [three-dimensional coordinates…camera coordinate system]. the sensors [or capture devices], e.g., on cameras, ¶¶ 123-124, are set up to in set vantage point angles, and by employing stereoscopic vision techniques on images captured from overlapping fields of view, the system utilizes the vantage points and detector parameters of capture devices to calculate the three-dimensional coordinates [positions] of objects within the areas of interest, ¶ 260 and fig. 19)
determining, according to a position relationship of the three-dimensional coordinates of the hand key point with respect to a standard pose skeleton template, a pose of the hand in the hand images; (and identifying specific gestures by comparing this 3D model to, or analyzing its trajectory against, pre-defined templates, ¶¶ 214 and 252. herein, it is broadly interpreted that the gesture templates include a “standard...template” [or reference model] as they are used for comparison against current input. Poses are types of gesture, ¶ 109, and are identified based on bone position modeling, ¶¶ 136 and 140 and figs. 1F and 1H, so herein, it is broadly interpreted that the “standard…template” is a “standard pose skeleton template”)
and recognizing, according to the pose of the hand in the hand images, the trigger gesture. (and identifying specific gestures by comparing this 3D model to, or analyzing its trajectory against, pre-defined templates, ¶¶ 214 and 252.)
Claim 3:
The rejection of claim 2 is incorporated. Benson further teaches:
wherein recognizing the trigger gesture further comprises:
determining a moving direction and a moving speed of relative movement of the hand. (parameters of the object/projectile, such as velocity and direction, are calculated based on [relative to] the motion reflected in the video images of the hand(s), ¶¶ 252, 369, and 384)
Claim 4:
The rejection of claim 3 is incorporated. Benson further teaches:
after determining the moving direction and the moving speed of the relative movement of the hand, the method further comprises: recognizing the thrown virtual object, and determining a throwing object destination position. (the thrown object is not only recognized, but a landing location is also calculated and represented in the animation, ¶¶ 53 and 384)
Claim 6:
The rejection of claim 1 is incorporated. Benson further teaches:
wherein the hand images comprise at least two consecutive frames of third hand images, and the gesture in the third hand images is throwing gesture; (video images include any number of successive images, e.g., third hand images, ¶¶ 122 and 124. a throwing gesture is tracked and recognized when the user has prepared to throw–is grasping the object–and has performed throwing motions, ¶¶ 137 and 384, and tracking is by using consecutive video images [consecutive frames], ¶¶ 140-141 and 252)
determining, according to the hand images, the throwing parameter, comprises:
calculating, based on a set angle of field of view, three-dimensional coordinates of a hand key point in each frame of the third hand images under a camera coordinate system; (system where optical sensors capture hand motion, using ray tracing, surface tangents, and normal vectors to align a 3D capsule-based model at specific points “P” on the hand surface to calculate its three-dimensional position and shape ¶¶ 50, 248 and 252, and this 3D model is subsequently used to track hand movement and gesture trajectory over time within a 3D sensory space, ¶ 214 [three-dimensional coordinates…camera coordinate system]. sensors [or capture devices], e.g., on cameras, ¶¶ 123-124, are set up to in set vantage point angles, and by employing stereoscopic vision techniques on images captured from overlapping fields of view, the system utilizes the vantage points and detector parameters of capture devices to calculate the three-dimensional coordinates [positions] of objects within the areas of interest, ¶ 260 and fig. 19)
and determining, according to the three-dimensional coordinates of the hand key point in each frame of the third hand images, the throwing parameter. (identifying specific gestures by comparing this 3D model to, or analyzing its trajectory against, pre-defined templates, ¶¶ 214 and 252. Throwing parameters of the object/projectile, such as velocity and direction, are calculated based on the motion reflected in the video images of the hand(s), ¶¶ 252 and 384)
Claim 8:
The rejection of claim 6 is incorporated. Benson further teaches:
before determining the throwing parameter according to the hand images, the method further comprises: recognizing a first frame of the third hand images and a last frame of the third hand images in the hand images according to the pose of the hand in each frame of the hand images and the moving speed of the relative movement of the hand in each frame of the hand images with respect to the previous frame of the hand images. (a throwing gesture is tracked and recognized, e.g., when the user has prepared to throw–is grasping the object–and has performed throwing motions, ¶¶ 137 and 384, and tracking is by using consecutive video images [e.g., including a first and last frame], ¶¶ 140-141 and 252. movement is detected based on origin and final positions [a first and last frame], as reflected in different video frames, ¶¶ 140-141 and fig. 1H. Because the gesture takes into account gesture properties, like gesture shape of hand/pose, ¶¶ 109 and 252, gesture path, implying a start and finish point of gesture [a first and last frame], and velocity [which includes speed and direction], and acceleration, which implies relative moving speed from frame to frame, ¶ 282)
Claim 10:
The rejection of claim 1 is incorporated. Benson further teaches:
wherein simulating, according to the throwing parameter, the motion trajectory to throw the virtual object, comprises:
establishing, according to the throwing parameter, a physical motion model of the virtual object; (The system employs a kinematic, parabolic trajectory model to calculate the path of the virtual paintball based on tracked hand velocity and direction, ¶ 384. It is considered a physical motion model because it simulates real-world physics by incorporating initial launch parameters derived from hand motion and the influence of gravity. The model is established by tracking the user’s hand to determine initial velocity and direction upon throwing the virtual object.)
and generating, according to the physical motion model, the motion trajectory of the virtual object. (The trajectory is generated by using these parameters to calculate and display a parabolic arc for the projectile's movement, ¶ 384)
Claim 11:
The rejection of claim 1 is incorporated. Benson further teaches:
wherein the throwing parameter comprises a throwing position, (The system constructs a 3D model of the hand from video images using capsule elements, allowing for the tracking of hand positions [e.g., throwing position] in 3D sensory space to detect real motions, such as a throw ¶¶ 252 and 376. parameters of the object/projectile, such as velocity and direction, are calculated based on the motion reflected in the video images of the hand(s), ¶¶ 252 and 384)
a throwing force (parameters of the object/projectile, such as throwing velocity [includes throwing speed] and direction, are calculated based on the motion reflected in the video images of the hand(s), ¶¶ 252 and 384. It’s common knowledge that Force = mass x acceleration [F=ma], so a throwing speed can be considered a throwing force because, according to physics principles, the speed attained by an object is the direct result of the force applied to it over time. In essence, a higher throwing speed requires greater acceleration, which necessitates a larger force [F=ma] exerted by the thrower on the object)
and a throwing direction; (parameters of the object/projectile, such as throwing velocity [includes throwing speed] and throwing direction, are calculated based on the motion reflected in the video images of the hand(s), ¶¶ 252 and 384)
and in response to determining that the throwing force belongs to a force interval matching the throwing position, and that the throwing direction belongs to a direction interval matching the throwing position, it is determined that the motion trajectory of the virtual object passes through a throwing destination position. (The system maps tracked hand motion to a computed paintball trajectory, ensuring that if the throwing force and direction align with predefined intervals for the throwing position, the virtual object 5103 is rendered to land on a target interface 5303, ¶ 384.)
Claim 12:
The rejection of claim 1 is incorporated. Benson further teaches:
before determining the throwing parameter according to the hand images in response to the trigger gesture of throwing the virtual object from the hand images, the method further comprises: determining an affine transformation relationship of each frame of the hand images with respect to a reference image; and aligning, according to the affine transformation relationship, each frame of the hand images with the reference image. (determining an affine transformation R-T that directly maps a current frame to a reference frame, followed by applying a reverse transformation to align the image frames, ¶ 164. This approach allows for directly calculating a relative motion model between image frames to establish a common coordinate system.)
Claim 15:
The rejection of claim 1 is incorporated. Benson further teaches:
further comprising:
rendering the AR scenario to display at least one of the following in the AR scenario:
illumination in the AR scenario and a shadow formed by the virtual object under the illumination;
texture of the virtual object;
a visual special effect of the AR scenario;
and throwing result information of the virtual object. (both a visual special effect and a throwing result information is reflected in a blooming effect that occurs at the location where the projectile lands [AR scenario], ¶¶ 53 and 384 and figs. 51-52, since visual effect [visual special effect] of displaying a control interface in the landing location, which is a result of a throwing action – a landing [throwing result information])
Independent Claims 17 and 18:
Claim(s) 17 and 18 are directed to an electronic device and computer-readable medium for accomplishing the steps of the method in claim 1, and are rejected using similar rationale(s).
Claims 19 and 21:
The rejection of claims 17 and 18 are incorporated. Claim(s) 19 and 21 are directed to an electronic device and computer-readable medium for accomplishing the steps of the method in claim 2, and are rejected using similar rationale(s).
Claim 20:
The rejection of claim 17 is incorporated. Claim(s) 20 is directed to an electronic device for accomplishing the steps of the method in claim 6, and is rejected using similar rationale(s).
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benson (US 20190362562 A1) in view of Pastrana (US 20230100610 A1), as applied to claim 4 above, and further in view of Boesel; Benjamin H. et al. (hereinafter Boesel – US 20230092282 A1).
Claim 5:
The rejection of claim 4 is incorporated. Benson further teaches:
wherein recognizing, according to the pose of the hand in the hand images, the trigger gesture, comprises:
in response to determining that the at least two consecutive frames of the first hand images are recognized, (tracking is by using successive video images [consecutive frames], ¶¶ 122, 148, 252 and 309)
the hand in the first hand images is in a first throwing pose and the hand key point is relatively stationary, (wherein the hands are detected as being relatively stationary––object has been grasped and user is preparing to throw it [first throwing pose], e.g., in a resting pose, ¶¶ 137, 140 and 384 and ¶¶ 49-54)
and that the at least one frame of the second hand image is recognized after the at least two consecutive frames of the first hand images, the hand in the second hand image is in a second throwing pose
and the hand key point moves relative to the first hand images, (a throwing gesture [second throwing pose and the hand key point moves] is tracked and recognized when the user has prepared to throw–is grasping the object–and has performed throwing motions, ¶¶ 137 and 384, and tracking is by using consecutive video images [consecutive frames], ¶¶ 140-141 and 252. movement is detected based on origin and final positions, as reflected in different video frames [relative to the first hand images], ¶¶ 140-141 and fig. 1H)
determining the moving direction and the moving speed of the relative movement; (the virtual object is displayed as traveling along a computed trajectory [simulating a motion trajectory] based on velocity and direction information, ¶ 384.)
[…].
Benson-Pastrana further teaches:
recognizing, as the trigger gesture, a gesture in the at least two consecutive frames of the first hand images and the at least one frame of the second hand image. (Benson teaches that the system identifies a trigger gesture that manipulates virtual objects, ¶ 376, that a throwing gesture is tracked and recognized when the user has prepared to throw–is grasping the object–and has performed throwing motions, ¶¶ 137 and 384, and tracking is by using consecutive video images [consecutive frames], ¶¶ 140-141 and 252. Pastrana teaches that a throw gesture includes an initial grab of an object, then releasing the object at the end of the movement, ¶ 274).
Benson-Pastrana does not appear to expressly teach, but Boesel teaches:
that the recognizing of the gesture is in response to determining that the moving direction is towards a set range around the throwing object destination position (a system [device 101] uses the direction of the movement to determine if the throw is aimed at a valid target. If the gesture's directional vector is outside the threshold range of the object's destination position, the system does not register it as a targetable throw, ¶ 435)
and that the moving speed exceeds a speed threshold value, (a minimum speed threshold for recognizing a throw, distinguishing it from slower movements [like carrying or moving an object] that do not reach this threshold, ¶ 458)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Benson to include that the recognizing of the gesture is in response to determining that the moving direction is towards a set range around the throwing object destination position and that the moving speed exceeds a speed threshold value, as taught by Boesel.
One would have been motivated to make such a combination in order to improve efficiency of human-machine interface, Boesel ¶ 5. It was well within the capabilities of a person having ordinary skill in the art to have realized that the speed and direction thresholds would help reducing unintended inputs and enhancing accuracy, which support efficient and intuitive user control, Boesel ¶¶ 5, 435 and 458.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benson (US 20190362562 A1) in view of Pastrana (US 20230100610 A1), as applied to claim 6 above, and further in view of Jung; Woo Jin et al. (hereinafter Jung – US 20100216517 A1).
Claim 7:
The rejection of claim 6 is incorporated. Benson further teaches:
wherein the throwing parameter comprises a throwing force (parameters of the object/projectile, such as throwing velocity [includes throwing speed] and direction, are calculated based on the motion reflected in the video images of the hand(s), ¶¶ 252 and 384. It’s common knowledge that Force = mass x acceleration [F=ma], so a throwing speed can be considered a throwing force because, according to physics principles, the speed attained by an object is the direct result of the force applied to it over time. In essence, a higher throwing speed requires greater acceleration, which necessitates a larger force [F=ma] exerted by the thrower on the object)
and a throwing direction; (parameters of the object/projectile, such as velocity and direction, are calculated based on the motion reflected in the video images of the hand(s), ¶¶ 252 and 384)
Benson-Pastrana does not appear to expressly teach, but Jung teaches:
determining, according to the three-dimensional coordinates of the hand key point in each frame of the third hand images, the throwing parameter, comprises: calculating a variation of the three-dimensional coordinates of the hand key point in each frame of the third hand images with respect to the three-dimensional coordinates in the previous frame of hand images; and determining the throwing force according to a peak value of the variation, (A method for user motion analysis that determines input motion strength [force] by calculating the difference between maximum and minimum acceleration values, ¶ 64 and fig. 9. This variance represents the largest difference, essentially acting as “a peak value of the variation.” Herein, it is broadly interpreted that the term “variation”, which was not defined in the Instant Specification, includes any of scalar magnitude of movement, a vector displacement (displacement over time), and other changes in movement characteristics, such as differences in acceleration values. It was well within the capabilities of a person having ordinary skill in the art to have realized that the terms strength and force are often used interchangeably because strength represents the maximum capability to generate or withstand a physical force, meaning a "stronger" entity is simply one capable of producing or enduring higher amounts of force.).
and using, as the throwing direction, a direction of the variation corresponding to the peak value (By analyzing the acceleration data for specific peaks (maximum and minimum) in a snapping motion, the data analysis unit 172 determines input direction based on the axis and sign of the acceleration variance, while the resulting difference value represents the magnitude of that motion's intensity, ¶ 64)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Benson to include determining, according to the three-dimensional coordinates of the hand key point in each frame of the third hand images, the throwing parameter, comprises: calculating a variation of the three-dimensional coordinates of the hand key point in each frame of the third hand images with respect to the three-dimensional coordinates in the previous frame of hand images; and determining the throwing force according to a peak value of the variation, and using, as the throwing direction, a direction of the variation corresponding to the peak value, as taught by Jung.
One would have been motivated to make such a combination in order to improve the accuracy of the throwing gesture recognition, Jung ¶¶ 10 and 64. It was well within the capabilities of a person having ordinary skill in the art, to have realized that combining Jung with Benson-Pastrana allowed a person of ordinary skill in the art to improve throwing gesture recognition, e.g., by analyzing release snap intensity and calculating force-proportional difference values rather than relying solely on arm speed.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benson (US 20190362562 A1) in view of Pastrana (US 20230100610 A1), as applied to claim 8 above, and further in view of Boesel; Benjamin H. et al. (hereinafter Boesel – US 20230092282 A1) and Monnin; Brian et al. (hereinafter Monnin – US 20180021630 A1).
Claim 9:
The rejection of claim 8 is incorporated. Benson-Pastrana further teaches that the recognition engine employs gesture properties such as velocity to identify variations in control object motion and interaction with virtual constructs, allowing for the interpretation of commands and related information to control a machine, Benson ¶ 282, and the detecting of end of a throw, Pastrana ¶ 274.
Benson-Pastrana does not appear to expressly teach, but Boesel teaches:
wherein recognizing, according to the pose of the hand in each frame of the hand images and the moving speed of the relative movement of the hand in each frame of the hand images with respect to the previous frame of the hand images, the first frame of the third hand images and the last frame of the third hand images in the hand images, comprises: in response to determining that the hand in one frame of the hand images is in the throwing pose and the moving speed of the relative movement with respect to the previous frame of the hand images exceeds a first speed threshold value are recognized, using, as the first frame of the third hand images, a frame of the hand images in which the hand is in the throwing pose and the moving speed of the relative movement with respect to the previous frame of the hand images exceeds the first speed threshold value (a minimum speed threshold for recognizing a throw, distinguishing it from slower movements [like carrying or moving an object] that do not reach this threshold, ¶ 458).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Benson to include wherein recognizing, according to the pose of the hand in each frame of the hand images and the moving speed of the relative movement of the hand in each frame of the hand images with respect to the previous frame of the hand images, the first frame of the third hand images and the last frame of the third hand images in the hand images, comprises: in response to determining that the hand in one frame of the hand images is in the throwing pose and the moving speed of the relative movement with respect to the previous frame of the hand images exceeds a first speed threshold value are recognized, using, as the first frame of the third hand images, a frame of the hand images in which the hand is in the throwing pose and the moving speed of the relative movement with respect to the previous frame of the hand images exceeds the first speed threshold value, as taught by Boesel.
One would have been motivated to make such a combination in order to improve efficiency of human-machine interface, Boesel ¶ 5. It was well within the capabilities of a person having ordinary skill in the art to have realized that the speed and direction thresholds would help reducing unintended inputs and enhancing accuracy, which support efficient and intuitive user control, Boesel ¶¶ 5, 435 and 458.
Benson-Pastrana- Boesel does not appear to expressly teach, but Monnin teaches:
and in response to determining that the hand in at least one frame of the hand images is in the throwing pose (gestures, e.g., grasping a ball are identified, see mapping to Benson above)
and the moving speed of the relative movement with respect to the previous frame of the hand images is lower than a second speed threshold value are recognized, using, as the last frame of the third hand images, a last frame of the hand images in the at least one frame of the hand images in which the hand is in the throwing pose and the moving speed of the relative movement with respect to the previous frame of the hand images is lower than the second speed threshold value (a system identifies the end of a throwing gesture by detecting when the hand's relative movement speed drops below a specific threshold following a peak acceleration, thereby defining the final frame of the gesture by analyzing when the hand slows down [point 2224] to accurately segment the motion and calculate the throw's duration [2124] and initial velocity, Monnin ¶ 276).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Benson to include and in response to determining that the hand in at least one frame of the hand images is in the throwing pose and the moving speed of the relative movement with respect to the previous frame of the hand images is lower than a second speed threshold value are recognized, using, as the last frame of the third hand images, a last frame of the hand images in the at least one frame of the hand images in which the hand is in the throwing pose and the moving speed of the relative movement with respect to the previous frame of the hand images is lower than the second speed threshold value, as taught by Monnin.
One would have been motivated to make such a combination in order to implement the recognizing of releasing of an object at the end of the movement, Pastrana ¶ 274, in a known and effective way, Monnin ¶ 276.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benson (US 20190362562 A1) in view of Pastrana (US 20230100610 A1), as applied to claim 12 above, and further in view of Daniilidis, Konstantinos et al. (hereinafter Daniilidis -- US 20200005469 A1).
Claim 13:
The rejection of claim 12 is incorporated. Benson-Pastrana does not appear to expressly teach, but Daniilidis teaches:
wherein determining the affine transformation relationship of each frame of the hand images with respect to the reference image comprises: calculating, based on an optical flow method, a coordinate deviation between a corner point of the hand in each frame of the hand images and a corresponding corner point of the reference image; and according to the coordinate deviation, determining the affine transformation relationship of each frame of the hand images with respect to the reference (The text describes using an Expectation-Maximization Iterative Closest Point (EM-ICP) algorithm to determine affine transformations by minimizing the squared distance between corner points pj in a frame and a reference image, ¶ 49. By iteratively minimizing this cost function through E-step associations and M-step parameter updates, the method calculates the optimal linear A and translation b transformation parameters to align frames, ¶¶ 50-51. The algorithm integrates frame-based inertial measurement unit (IMU) data, provided at discrete time intervals, with high-speed, asynchronous, purely event-based tracking measurements via an Extended Kalman Filter to provide 6-DOF pose estimation, ¶¶ 78, 82 and 83. Focusing on corners solves the aperture problem in event-based tracking by providing edge data in multiple directions, unlike straight lines, ¶ 125. It was well within the capabilities of a person having ordinary skill in the art to have realized that in applying these concepts to Benson-Pastrana, the images would hand images in a 3D coordinate environment).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Benson to include wherein determining the affine transformation relationship of each frame of the hand images with respect to the reference image comprises: calculating, based on an optical flow method, a coordinate deviation between a corner point of the hand in each frame of the hand images and a corresponding corner point of the reference image; and according to the coordinate deviation, determining the affine transformation relationship of each frame of the hand images with respect to the reference, as taught by Benson-Pastrana.
One would have been motivated to make such a combination in order to improve the tracking capabilities by enabling longer tracks, Daniilidis Abstract and Daniilidis ¶ 4, and tracking persistence, Daniilidis ¶ 76.
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benson (US 20190362562 A1) in view of Pastrana (US 20230100610 A1), as applied to claim 1 above, and further in view of Hikawa, H., et al (hereinafter Hikawa – Non-Patent Literature, “Dynamic Gesture Recognition System with Gesture Spotting Based on Self-Organizing Maps”, published 2/22/2021).
Claim 14:
The rejection of claim 1 is incorporated. Benson-Pastrana does not appear to expressly teach, but Hikawa teaches:
before determining the throwing parameter according to the hand images in response to the trigger gesture of throwing the virtual object from the hand images, the method further comprises: (it was well within the capabilities of a person having ordinary skill in the art to have realized that the smoothing discussed below would occur before determining the throwing parameter because frame by frame is naturally noisy/jittery and trying to calculate would make calculation difficult, see Page 8)
collecting a plurality of frames of the hand images by an image sensor, (a sequence of video frames of dynamic hand gestures, collected using a camera, are fed/input into the system, Pgs 4 and 9)
and performing mean filtering (a moving average [mean filtering] is computed, Pg 8)
on a plurality of consecutive frames of the hand images (the moving average is of samples of consecutive video frames of hand motions, Pgs 8-9)
according to a set step length (moving average over a sample size [L], see formula 17, Pg 8, the sample size serves as a sliding window [set step length] for the moving average).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Benson to include wherein before determining the throwing parameter according to the hand images in response to the trigger gesture of throwing the virtual object from the hand images, the method further comprises: collecting a plurality of frames of the hand images by an image sensor, and performing mean filtering on a plurality of consecutive frames of the hand images according to a set step length, as taught by Hikawa.
One would have been motivated to make such a combination in order to improve the method’s detection accuracy, Hikawa Pg 11.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Below is a list of these references, including why they are pertinent:
Lacey; Paul US 20210263593 A1, is pertinent to claim 1 for disclosing allowing users to interact with virtual objects using their hands and a plurality of associated hand keypoints, Abstract and ¶ 6.
Langridge; Adam Jethro et al. US 20120309535 A1, is pertinent to claim 1 for disclosing a system that analyzes a user’s physical, gesture-based movements, such as a one-handed or two-handed throwing motion, to control in-game actions like casting spells with magnitudes proportional to the speed and distance of the movement, ¶¶ 57, 68 and 6].
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GABRIEL S MERCADO whose telephone number is (408)918-7537. The examiner can normally be reached Mon-Fri 8am-5pm (Eastern Time).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kieu Vu can be reached at (571) 272-4057. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Gabriel Mercado/Primary Examiner, Art Unit 2171
Prosecution Insights