Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is responsive to the application filed June 25, 2025, claims 1-20 are presented for examination. Claims 1, 9 and 17 are independent claims.
Priority
Acknowledgment is made of applicant’s claim for continuation priority based on application # 18638273 which is a continuation of 17941301 September 9, 2022 Patent 11995780.
Oath/Declaration
The Office acknowledges receipt of a properly signed Oath/Declaration submitted June 25, 2025
Drawings
The drawings filed June 25, 2025 are accepted by the examiner.
Abstract
The abstract filed June 25, 2025 is accepted by the examiner.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the "right to exclude" granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory obviousness-type double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428,46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046,29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Omum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969). A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the conflicting application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. Effective January 1, 1994, a registered attorney or agent of record may sign a terminal disclaimer. A terminal disclaimer signed by the assignee must fully comply with 37 CPR 3.73(b).
Claims 1-20 are rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claims 1-18 of application No. 18638273 Patent 12361653 B2. Although the conflicting claims are not identical, they are not patentably distinct from each other because the claims recites from detecting, using one or more hardware processors, from video frame data captured by a camera, a firearm-like hand gesture comprising a thumb and an index finger indicating a shooting hand sign; determining, using the one or more hardware processors, a direction that the index finger is pointing based on a location and position of the index finger in the video frame data; performing, using the one or more hardware processors, a raycasting technique to determine a vector path from the index finger to a point along an outer boundary of a scene comprising a representation of a real world environment; in response to detecting a change in the location and position of the index finger, updating the vector path based on the changed direction of the index finger; generating, using the one or more hardware processors, a virtual projectile object at the location of the index finger; rendering, using the one or more hardware processors, movement of the virtual projectile object along the updated vector path within the scene; and providing for display the rendered movement of the virtual projectile object along the updated vector path, therefore the same limitations as claimed in application No. 18638273 Patent 12361653 B2.
This is an obviousness-type double patenting rejection.
US Application No. 19249032
No. . 18638273 Patent 12361653 B2
1. 1. A method, comprising: detecting, using one or more hardware processors, from video frame data captured by a camera, a firearm-like hand gesture comprising a thumb and an index finger indicating a shooting hand sign; determining, using the one or more hardware processors, a direction that the index finger is pointing based on a location and position of the index finger in the video frame data; performing, using the one or more hardware processors, a raycasting technique to determine a vector path from the index finger to a point along an outer boundary of a scene comprising a representation of a real world environment; in response to detecting a change in the location and position of the index finger, updating the vector path based on the changed direction of the index finger; generating, using the one or more hardware processors, a virtual projectile object at the location of the index finger; rendering, using the one or more hardware processors, movement of the virtual projectile object along the updated vector path within the scene; and providing for display the rendered movement of the virtual projectile object along the updated vector path.
Similar claims 9 and 17.
1. A method, comprising: detecting, using one or more hardware processors, from a set of frames, a first gesture corresponding to an open trigger finger gesture; detecting, using the one or more hardware processors, from a second set of frames, a second gesture corresponding to a closed trigger finger gesture; generating, using the one or more hardware processors, a first virtual object based at least in part on a location and a position of a representation of a finger; rendering, using the one or more hardware processors, a movement of the first virtual object along a vector away from the location and the position of the representation of the finger within a first scene, the vector away from the location and the position of the representation of the finger being determined based on raycasting a path from the location and the position of the representation of the finger to a point along a particular outer boundary of the first scene, wherein rendering the movement of the first virtual object comprises: while rendering the movement of the first virtual object, generating a second virtual object based at least in part on a second location and a second position of the representation of the finger; rendering a movement of the second virtual object along a second vector away from the location and the position of the representation of the finger within a first scene, wherein the first scene comprises a first representation of a real world scene, the first virtual object, and the second virtual object; and providing for display the rendered movement of the second virtual object along the second vector within the first scene.
Similar claims 10 and 18.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5, 8-10, 13 and 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Langridge et al. (US 20140235350 Al) in view of Wu et al. (US 20130303247 A1) in further view of Pinchon (US 11294475 B1)
As to Claim 1:
Langridge et al. discloses a method (Langridge, see Abstract, where Langridge discloses gestures of a computer user are observed with a depth camera. A throwing gesture of the computer user is identified and an aiming vector is calculated from a path of a hand during the throwing gesture. An interface action is directed along the aiming vector within an interactive interface), comprising: detecting, using one or more hardware processors (Langridge, see figure 2), from video frame data (Langridge, see frames t0 through t3 in figure 13B) captured by a camera (Langridge, see 44 in figure 3 where Langridge discloses an acknowledge ready posture and paragraph [0049], where Langridge discloses that at 44, method 42 includes acknowledging a ready posture of a game player. In some embodiments, the game player may be observed by a depth camera and modeled with a virtual skeleton, as described above. A position of one or more joints of the virtual skeleton may be translated/interpreted as a ready posture or any of the other gestures described herein depending on the relative joint positions and joint movement from frame to frame), a hand gesture comprising a thumb and an index finger indicating a shooting hand sign (Langridge, see 18 and 16 in figure 1); determining, using the one or more hardware processors, a based on a location and position of the fingers in the video frame data (Langridge, see paragraphs [0105], [0043] and [0044], where Langridge discloses that when a video camera is used, it may be used to provide target tracking data, confirmation data for error correction of target tracking, image capture, face recognition, high-precision tracking of fingers (or other small features), light sensing, and/or other functions and that during skeletal modeling 34, one or more depth images ( e.g., depth map 32) of a world space scene including a computer user ( e.g., game player 18) are obtained from the depth camera. A virtual skeleton may take the form of a data structure including one or more parameters for each of a plurality of skeletal joints ( e.g., a joint matrix including an x position, a y position, a z
position, and a rotation for each joint). In some embodiments, other types of virtual skeletons may be used (e.g., a wireframe, a set of shape primitives, etc.); generating, using the one or more hardware processors, a virtual projectile object (Langridge, see 40 and 38 in figure 2) at the location of the fingers (Langridge, see 34 in figure 2, figure 15A); rendering, using the one or more hardware processors, movement of the virtual projectile object along the updated vector path within the scene (Langridge, see t0, t1, t2 and t3 in figures 13A and 13B); and providing for display the rendered movement of the virtual projectile object along the updated vector path (Langridge, see t0, t1, t2 and t3 in figures 13A and 13B and figure 17A).
PNG
media_image1.png
522
676
media_image1.png
Greyscale
PNG
media_image2.png
888
930
media_image2.png
Greyscale
PNG
media_image3.png
708
570
media_image3.png
Greyscale
PNG
media_image4.png
842
1082
media_image4.png
Greyscale
Langridge differs from the claimed subject in that Langridge does not explicitly disclose a fire-arm like hand gesture as it pertains to a firearm-like hand gesture comprising a thumb and an index finger indicating a shooting hand sign; performing, using the one or more hardware processors, a raycasting technique to determine a vector path from the index finger to a point along an outer boundary of a scene comprising a representation of a real world environment; in response to detecting a change in the location and position of the index finger, updating the vector path based on the changed direction of the index finger.
However in an analogous art, Wu discloses a fire-arm like hand gesture as it pertains to a firearm-like hand gesture comprising a thumb and an index finger indicating a shooting hand sign (Wu, see claim mapping in figure 4A).
PNG
media_image5.png
493
694
media_image5.png
Greyscale
It would have been obvious to one of ordinary skill in the art to modify the invention of Langridge with Wu. One would be motivated to modify Langridge by disclosing a fire-arm like hand gesture as it pertains to a firearm-like hand gesture comprising a thumb and an index finger indicating a shooting hand sign as taught by Wu, and thereby providing an interaction display system and method utilizing both front-facing and rear-facing cameras simultaneously in a mobile device therefore improving real and virtual interaction with the mobile device (Wu, see paragraph [0002]). Wu does not explicitly disclose performing, using the one or more hardware processors, a raycasting technique to determine a vector path from the index finger to a point along an outer boundary of a scene comprising a representation of a real world environment; in response to detecting a change in the location and position of the index finger, updating the vector path based on the changed direction of the index finger.
However in an analogous art, Pinchon discloses performing, using the one or more hardware processors, a raycasting technique to determine a vector path (Pinchon, see column 10 lines 39-49, where Pinchon discloses that the control point can be the point the ray is cast out from the user ( e.g., one of the user's fingertip). In an example, when the ray casting gesture is a pinch or open pinch, the control point can be the point where thumb tip touches another (e.g., index) fingertip or a point between the tip of one of the user's fingers and the tip of the user's thumb when held apart. The interaction point can be the point at the end of the line ( e.g., a particular distance from the control point) that the user can control by moving the origin and control points to touch/contact/ interact with the target object) from the index finger to a point along an outer boundary of a scene (Pinchon, see 704, 706 and 708 in figure 7) comprising a representation of a real world environment (Pinchon, see column 3 lines 54-58, where Pinchon discloses that "Virtual reality" or "VR," as used herein, refers to an immersive experience where a user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to systems where a user views images of the real world after they have passed through a computing system); in response to detecting a change in the location and position of the index finger (Pinchon, see 704, 706 and 708 in figure 7), updating the vector path based on the changed direction (Pinchon, see 704, 706 and 708 in figure 7 and 705, 706 in figure 8) of the index finger (Pinchon, see figures 7 and 8 and column 5 lines 34-41, where Pinchon discloses that short ray input mode can provide rays that extend, for a particular distance, along a line that intersects an origin point ( e.g., in the user's palm) and a control point (e.g., between the user's thumb tip and index finger tip), extending to an interaction point ( e.g., point of the short ray that can interact with a target object). This short ray input mode enables a user to perform precise interactions with a target object that is near the user, without occluding the target objects).
PNG
media_image6.png
944
960
media_image6.png
Greyscale
PNG
media_image7.png
762
1042
media_image7.png
Greyscale
PNG
media_image8.png
896
1027
media_image8.png
Greyscale
It would have been obvious to one of ordinary skill in the art to modify the invention of Langridge and Wu with Pinchon. One would be motivated to modify Langridge and Wu by disclosing a performing, using the one or more hardware processors, a raycasting technique to determine a vector path from the index finger to a point along an outer boundary of a scene comprising a representation of a real world environment; in response to detecting a change in the location and position of the index finger, updating the vector path based on the changed direction of the index finger as taught by Pinchon, and thereby providing an improved system having different types of rays, such as straight rays, curved rays, or rays that emanate from different body parts or other user-controlled elements (Pinchon, see column 1 lines 46-50).
As to Claim 2:
Langridge in view of Wu in further view of Pinchon discloses that the method of claim 1, wherein detecting the firearm-like hand gesture comprises: detecting a particular gesture including the thumb and the index finger indicating a firearm or shooting hand sign in an open trigger finger gesture; and detecting a transition to a closed trigger finger gesture where the thumb and the index finger are pointing in a same direction Wu, see figure 4A claim mapping and figure 3D).
PNG
media_image9.png
337
343
media_image9.png
Greyscale
PNG
media_image10.png
484
402
media_image10.png
Greyscale
As to Claim 5:
Langridge in view of Wu in further view of Pinchon discloses that the method of claim 1, wherein the virtual projectile object comprises a representation of a projectile or missile to be fired from a firearm (Wu, see figure 4A and paragraph [0035], where Wu discloses FIGS. 4A-4B illustrate a diagram of the shooting game application according to yet another embodiment of the invention. In yet another embodiment, the projectile in the shooting game application can be shot in various ways. For example, the user's hand may be in a "pistol" shape, wherein the orientation of the forefinger indicates the shooting direction of the projectile. In the embodiment, the mobile device 100 may comprise a microphone 420 coupled to the processing unit 230 for receiving sounds from the user. Accordingly, the processing unit 230 may further control the projectile in the shooting game application by the received sounds of the user ( e.g. a specific word such as "BANG") and the gestures in the captured images from the front-facing camera 210. That is, the processing unit 230 may further compute and create the interactions according to the received sounds and gestures. In yet another embodiment, the user may shoot the projectile by using specific hand/body postures, or facial expressions (e.g. twinkling eyes, smiling, etc.), as illustrated in FIG. 4 B. The specific hand postures are similar to the hand postures in FIG. 3A. The processing unit 230 may detect the connected point 410 of the fingertips in the captured images from the front-facing camera 210. When the forefinger quickly moves away from the thumb to throw the projectile, the processing unit 230 may detect the moving speed and the orientation of the forefinger, thereby calculating the trajectory of the projectile in the interaction images. In yet another embodiment, the user may throw the projectile by using the gesture analogous to throwing a dart. The processing unit 230 may detect the moving speed and orientation of the user's hand, thereby calculating the trajectory of the projectile in the interaction images. It should be noted that the application is not limited to the aforementioned embodiments to operate the projectile).
As to Claim 8:
Langridge in view of Wu in further view of Pinchon discloses that the method of claim 1, wherein rendering movement of the virtual projectile object comprises animating the virtual projectile object to move across a particular outer boundary of the scene and outside a view of the scene (Langridge, see figures 17A through 17D).
As to Claim 9:
Langridge et al. discloses a system (Langridge, see Abstract, where Langridge discloses gestures of a computer user are observed with a depth camera. A throwing gesture of the computer user is identified and an aiming vector is calculated from a path of a hand during the throwing gesture. An interface action is directed along the aiming vector within an interactive interface) comprising: a processor; and a memory including instructions that, when executed by the processor (Langridge, see figure 2), cause the processor to perform operations comprising: detecting, using one or more hardware processors, from video frame data (Langridge, see frames t0 through t3 in figure 13B) captured by a camera (Langridge, see 44 in figure 3 where Langridge discloses an acknowledge ready posture and paragraph [0049], where Langridge discloses that at 44, method 42 includes acknowledging a ready posture of a game player. In some embodiments, the game player may be observed by a depth camera and modeled with a virtual skeleton, as described above. A position of one or more joints of the virtual skeleton may be translated/interpreted as a ready posture or any of the other gestures described herein depending on the relative joint positions and joint movement from frame to frame), a hand gesture comprising a thumb and an index finger indicating a shooting hand sign (Langridge, see 18 and 16 in figure 1); determining, using the one or more hardware processors, a direction that the index finger is pointing based on a location and position of the fingers in the video frame data Langridge, see paragraphs [0105], [0043] and [0044], where Langridge discloses that when a video camera is used, it may be used to provide target tracking data, confirmation data for error correction of target tracking, image capture, face recognition, high-precision tracking of fingers (or other small features), light sensing, and/or other functions and that during skeletal modeling 34, one or more depth images ( e.g., depth map 32) of a world space scene including a computer user ( e.g., game player 18) are obtained from the depth camera. A virtual skeleton may take the form of a data structure including one or more parameters for each of a plurality of skeletal joints ( e.g., a joint matrix including an x position, a y position, a z position, and a rotation for each joint). In some embodiments, other types of virtual skeletons may be used (e.g., a wireframe, a set of shape primitives, etc.); generating, using the one or more hardware processors, a virtual projectile object (Langridge, see 40 and 38 in figure 2) at the location of the fingers (Langridge, see 34 in figure 2, figure 15A); rendering, using the one or more hardware processors, movement of the virtual projectile object along the updated vector path within the scene (Langridge, see t0, t1, t2 and t3 in figures 13A and 13B); and providing for display the rendered movement of the virtual projectile object along the updated vector path (Langridge, see t0, t1, t2 and t3 in figures 13A and 13B and figure 17A).
PNG
media_image1.png
522
676
media_image1.png
Greyscale
PNG
media_image2.png
888
930
media_image2.png
Greyscale
PNG
media_image3.png
708
570
media_image3.png
Greyscale
PNG
media_image4.png
842
1082
media_image4.png
Greyscale
Langridge differs from the claimed subject in that Langridge does not explicitly disclose a fire-arm like hand gesture as it pertains to a firearm-like hand gesture comprising a thumb and an index finger indicating a shooting hand sign; performing, using the one or more hardware processors, a raycasting technique to determine a vector path from the index finger to a point along an outer boundary of a scene comprising a representation of a real world environment; in response to detecting a change in the location and position of the index finger, updating the vector path based on the changed direction of the index finger.
However in an analogous art, Wu discloses a fire-arm like hand gesture as it pertains to a firearm-like hand gesture comprising a thumb and an index finger indicating a shooting hand sign (Wu, see claim mapping in figure 4A).
PNG
media_image5.png
493
694
media_image5.png
Greyscale
It would have been obvious to one of ordinary skill in the art to modify the invention of Langridge with Wu. One would be motivated to modify Langridge by disclosing a fire-arm like hand gesture as it pertains to a firearm-like hand gesture comprising a thumb and an index finger indicating a shooting hand sign as taught by Wu, and thereby providing an interaction display system and method utilizing both front-facing and rear-facing cameras simultaneously in a mobile device therefore improving real and virtual interaction with the mobile device (Wu, see paragraph [0002]). Wu does not explicitly disclose performing, using the one or more hardware processors, a raycasting technique to determine a vector path from the index finger to a point along an outer boundary of a scene comprising a representation of a real world environment; in response to detecting a change in the location and position of the index finger, updating the vector path based on the changed direction of the index finger.
However in an analogous art, Pinchon discloses performing, using the one or more hardware processors, a raycasting technique to determine a vector path (Pinchon, see column 10 lines 39-49, where Pinchon discloses that the control point can be the point the ray is cast out from the user ( e.g., one of the user's fingertip). In an example, when the ray casting gesture is a pinch or open pinch, the control point can be the point where thumb tip touches another (e.g., index) fingertip or a point between the tip of one of the user's fingers and the tip of the user's thumb when held apart. The interaction point can be the point at the end of the line ( e.g., a particular distance from the control point) that the user can control by moving the origin and control points to touch/contact/ interact with the target object) from the index finger to a point along an outer boundary of a scene (Pinchon, see 704, 706 and 708 in figure 7) comprising a representation of a real world environment (Pinchon, see column 3 lines 54-58, where Pinchon discloses that "Virtual reality" or "VR," as used herein, refers to an immersive experience where a user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to systems where a user views images of the real world after they have passed through a computing system); in response to detecting a change in the location and position of the index finger (Pinchon, see 704, 706 and 708 in figure 7), updating the vector path based on the changed direction (Pinchon, see 704, 706 and 708 in figure 7 and 705, 706 in figure 8) of the index finger (Pinchon, see figures 7 and 8 and column 5 lines 34-41, where Pinchon discloses that short ray input mode can provide rays that extend, for a particular distance, along a line that intersects an origin point ( e.g., in the user's palm) and a control point (e.g., between the user's thumb tip and index finger tip), extending to an interaction point ( e.g., point of the short ray that can interact with a target object). This short ray input mode enables a user to perform precise interactions with a target object that is near the user, without occluding the target objects).
PNG
media_image6.png
944
960
media_image6.png
Greyscale
PNG
media_image7.png
762
1042
media_image7.png
Greyscale
PNG
media_image8.png
896
1027
media_image8.png
Greyscale
It would have been obvious to one of ordinary skill in the art to modify the invention of Langridge and Wu with Pinchon. One would be motivated to modify Langridge and Wu by disclosing a performing, using the one or more hardware processors, a raycasting technique to determine a vector path from the index finger to a point along an outer boundary of a scene comprising a representation of a real world environment; in response to detecting a change in the location and position of the index finger, updating the vector path based on the changed direction of the index finger as taught by Pinchon, and thereby providing an improved system having different types of rays, such as straight rays, curved rays, or rays that emanate from different body parts or other user-controlled elements (Pinchon, see column 1 lines 46-50).
As to Claim 10:
Langridge in view of Wu in further view of Pinchon discloses that the system of claim 9, wherein detecting the firearm-like hand gesture comprises: detecting a particular gesture including the thumb and the index finger indicating a firearm or shooting hand sign in an open trigger finger gesture; and detecting a transition to a closed trigger finger gesture where the thumb and the index finger are pointing in a same direction Wu, see figure 4A claim mapping and figure 3D).
PNG
media_image9.png
337
343
media_image9.png
Greyscale
PNG
media_image10.png
484
402
media_image10.png
Greyscale
As to Claim 13:
Langridge in view of Wu in further view of Pinchon discloses that the system of claim 9, wherein the virtual projectile object comprises a representation of a projectile or missile to be fired from a firearm (Wu, see figure 4A and paragraph [0035], where Wu discloses FIGS. 4A-4B illustrate a diagram of the shooting game application according to yet another embodiment of the invention. In yet another embodiment, the projectile in the shooting game application can be shot in various ways. For example, the user's hand may be in a "pistol" shape, wherein the orientation of the forefinger indicates the shooting direction of the projectile. In the embodiment, the mobile device 100 may comprise a microphone 420 coupled to the processing unit 230 for receiving sounds from the user. Accordingly, the processing unit 230 may further control the projectile in the shooting game application by the received sounds of the user ( e.g. a specific word such as "BANG") and the gestures in the captured images from the front-facing camera 210. That is, the processing unit 230 may further compute and create the interactions according to the received sounds and gestures. In yet another embodiment, the user may shoot the projectile by using specific hand/body postures, or facial expressions (e.g. twinkling eyes, smiling, etc.), as illustrated in FIG. 4 B. The specific hand postures are similar to the hand postures in FIG. 3A. The processing unit 230 may detect the connected point 410 of the fingertips in the captured images from the front-facing camera 210. When the forefinger quickly moves away from the thumb to throw the projectile, the processing unit 230 may detect the moving speed and the orientation of the forefinger, thereby calculating the trajectory of the projectile in the interaction images. In yet another embodiment, the user may throw the projectile by using the gesture analogous to throwing a dart. The processing unit 230 may detect the moving speed and orientation of the user's hand, thereby calculating the trajectory of the projectile in the interaction images. It should be noted that the application is not limited to the aforementioned embodiments to operate the projectile).
As to Claim 16:
Langridge in view of Wu in further view of Pinchon discloses that the system of claim 9, wherein rendering movement of the virtual projectile object comprises animating the virtual projectile object to move across a particular outer boundary of the scene and outside a view of the scene (Langridge, see figures 17A through 17D).
As to Claim 17:
Langridge et al. discloses a non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations (Langridge, see Abstract, where Langridge discloses gestures of a computer user are observed with a depth camera. A throwing gesture of the computer user is identified and an aiming vector is calculated from a path of a hand during the throwing gesture. An interface action is directed along the aiming vector within an interactive interface) comprising: detecting, using one or more hardware processors (Langridge, see figure 2), from video frame data (Langridge, see frames t0 through t3 in figure 13B) captured by a camera (Langridge, see 44 in figure 3 where Langridge discloses an acknowledge ready posture and paragraph [0049], where Langridge discloses that at 44, method 42 includes acknowledging a ready posture of a game player. In some embodiments, the game player may be observed by a depth camera and modeled with a virtual skeleton, as described above. A position of one or more joints of the virtual skeleton may be translated/interpreted as a ready posture or any of the other gestures described herein depending on the relative joint positions and joint movement from frame to frame), a hand gesture comprising a thumb and an index finger indicating a shooting hand sign (Langridge, see 18 and 16 in figure 1); determining, using the one or more hardware processors, a direction that the fingers are pointing based on a location and position of the fingers in the video frame data (Langridge, see paragraphs [0105], [0043] and [0044], where Langridge discloses that when a video camera is used, it may be used to provide target tracking data, confirmation data for error correction of target tracking, image capture, face recognition, high-precision tracking of fingers (or other small features), light sensing, and/or other functions and that during skeletal modeling 34, one or more depth images ( e.g., depth map 32) of a world space scene including a computer user ( e.g., game player 18) are obtained from the depth camera. A virtual skeleton may take the form of a data structure including one or more parameters for each of a plurality of skeletal joints ( e.g., a joint matrix including an x position, a y position, a z position, and a rotation for each joint). In some embodiments, other types of virtual skeletons may be used (e.g., a wireframe, a set of shape primitives, etc); performing, using the one or more hardware processors,; generating, using the one or more hardware processors, a virtual projectile object (Langridge, see 40 and 38 in figure 2) at the location of the fingers (Langridge, see 34 in figure 2, figure 15A); rendering, using the one or more hardware processors, movement of the virtual projectile object along the updated vector path within the scene (Langridge, see t0, t1, t2 and t3 in figures 13A and 13B); and providing for display the rendered movement of the virtual projectile object along the updated vector path (Langridge, see t0, t1, t2 and t3 in figures 13A and 13B and figure 17A).
PNG
media_image1.png
522
676
media_image1.png
Greyscale
PNG
media_image2.png
888
930
media_image2.png
Greyscale
PNG
media_image3.png
708
570
media_image3.png
Greyscale
PNG
media_image4.png
842
1082
media_image4.png
Greyscale
Langridge differs from the claimed subject in that Langridge does not explicitly disclose a fire-arm like hand gesture as it pertains to a firearm-like hand gesture comprising a thumb and an index finger indicating a shooting hand sign; performing, using the one or more hardware processors, a raycasting technique to determine a vector path from the index finger to a point along an outer boundary of a scene comprising a representation of a real world environment; in response to detecting a change in the location and position of the index finger, updating the vector path based on the changed direction of the index finger.
However in an analogous art, Wu discloses a fire-arm like hand gesture as it pertains to a firearm-like hand gesture comprising a thumb and an index finger indicating a shooting hand sign (Wu, see claim mapping in figure 4A).
PNG
media_image5.png
493
694
media_image5.png
Greyscale
It would have been obvious to one of ordinary skill in the art to modify the invention of Langridge with Wu. One would be motivated to modify Langridge by disclosing a fire-arm like hand gesture as it pertains to a firearm-like hand gesture comprising a thumb and an index finger indicating a shooting hand sign as taught by Wu, and thereby providing an interaction display system and method utilizing both front-facing and rear-facing cameras simultaneously in a mobile device therefore improving real and virtual interaction with the mobile device (Wu, see paragraph [0002]). Wu does not explicitly disclose performing, using the one or more hardware processors, a raycasting technique to determine a vector path from the index finger to a point along an outer boundary of a scene comprising a representation of a real world environment; in response to detecting a change in the location and position of the index finger, updating the vector path based on the changed direction of the index finger.
However in an analogous art, Pinchon discloses performing, using the one or more hardware processors, a raycasting technique to determine a vector path (Pinchon, see column 10 lines 39-49, where Pinchon discloses that the control point can be the point the ray is cast out from the user ( e.g., one of the user's fingertip). In an example, when the ray casting gesture is a pinch or open pinch, the control point can be the point where thumb tip touches another (e.g., index) fingertip or a point between the tip of one of the user's fingers and the tip of the user's thumb when held apart. The interaction point can be the point at the end of the line ( e.g., a particular distance from the control point) that the user can control by moving the origin and control points to touch/contact/ interact with the target object) from the index finger to a point along an outer boundary of a scene (Pinchon, see 704, 706 and 708 in figure 7) comprising a representation of a real world environment (Pinchon, see column 3 lines 54-58, where Pinchon discloses that "Virtual reality" or "VR," as used herein, refers to an immersive experience where a user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to systems where a user views images of the real world after they have passed through a computing system); in response to detecting a change in the location and position of the index finger (Pinchon, see 704, 706 and 708 in figure 7), updating the vector path based on the changed direction (Pinchon, see 704, 706 and 708 in figure 7 and 705, 706 in figure 8) of the index finger (Pinchon, see figures 7 and 8 and column 5 lines 34-41, where Pinchon discloses that short ray input mode can provide rays that extend, for a particular distance, along a line that intersects an origin point ( e.g., in the user's palm) and a control point (e.g., between the user's thumb tip and index finger tip), extending to an interaction point ( e.g., point of the short ray that can interact with a target object). This short ray input mode enables a user to perform precise interactions with a target object that is near the user, without occluding the target objects).
PNG
media_image6.png
944
960
media_image6.png
Greyscale
PNG
media_image7.png
762
1042
media_image7.png
Greyscale
PNG
media_image8.png
896
1027
media_image8.png
Greyscale
It would have been obvious to one of ordinary skill in the art to modify the invention of Langridge and Wu with Pinchon. One would be motivated to modify Langridge and Wu by disclosing a performing, using the one or more hardware processors, a raycasting technique to determine a vector path from the index finger to a point along an outer boundary of a scene comprising a representation of a real world environment; in response to detecting a change in the location and position of the index finger, updating the vector path based on the changed direction of the index finger as taught by Pinchon, and thereby providing an improved system having different types of rays, such as straight rays, curved rays, or rays that emanate from different body parts or other user-controlled elements (Pinchon, see column 1 lines 46-50).
As to Claim 18:
Langridge in view of Wu in further view of Pinchon discloses that the non-transitory computer-readable medium of claim 17, wherein detecting the firearm-like hand gesture comprises: detecting a particular gesture including the thumb and the index finger indicating a firearm or shooting hand sign in an open trigger finger gesture; and detecting a transition to a closed trigger finger gesture where the thumb and the index finger are pointing in a same direction (Wu, see figure 4A claim mapping and figure 3D).
PNG
media_image9.png
337
343
media_image9.png
Greyscale
PNG
media_image10.png
484
402
media_image10.png
Greyscale
Allowable Subject Matter
Claims 3, 4, 6, 7, 11, 12, 14, 15, 19 and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Referring to claims 3, 11 and 19 the following is a statement of reasons for the indication of allowable subject matter: the prior art fail to suggest limitations “wherein determining the direction that the index finger is pointing comprises: analyzing a location and position of a representation of the thumb and a second location and second position of a representation of the index finger in the video frame data; and calculating an angular orientation based on a relative positions of the thumb and the index finger”.
Referring to claims 4, 12 and 20 the following is a statement of reasons for the indication of allowable subject matter: the prior art fail to suggest limitations “wherein the change in the location and position of the index finger comprises detecting movement of the index finger from a first pointing direction to a second pointing direction while maintaining the firearm-like hand gesture”.
Referring to claims 6 and 14, the following is a statement of reasons for the indication of allowable subject matter: the prior art fail to suggest limitations “wherein rendering movement of the virtual projectile object comprises: while rendering the movement of the virtual projectile object along the updated vector path, generating a second virtual projectile object based on a subsequent change in the location and position of the index finger; and rendering the movement of the second virtual projectile object along a second vector path determined by the subsequent change in direction”.
Referring to claims 7 and 15, the following is a statement of reasons for the indication of allowable subject matter: the prior art fail to suggest limitations “wherein the second virtual projectile object and the virtual projectile object are a same type of virtual object, or the second virtual projectile object and the virtual projectile object are different types of virtual objects”.
Conclusion
The prior art made of record and not relied upon is considered pertinent to
applicant's disclosure. Ravasz (US 10802600 B1) discloses that when a user wishes to interact with an object outside her immediate reach, she can use a projection to select, move, or otherwise interact with the distant object. The present technology also includes object selection techniques for identifying and disambiguating between objects, allowing a user to select objects both near and distant from the user. Yet further aspects of the present technology include techniques for interpreting various bimanual (two-handed) gestures for interacting with objects.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NELSON ROSARIO whose telephone number is (571)270-1866. The examiner can normally be reached on Monday through Friday, 7:30am- 5:00pm EST. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Eason can be reached on (571) 270-7230. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NELSON M ROSARIO/ Primary Examiner, Art Unit 2624
Prosecution Insights