DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
The limitations of “displaying a virtual tough interface, wherein the virtual touch interface is associated with a position of a target plane in a physical space in which an electronic device is located;” is considered something that cannot be practically performed in the human mine and not a mental process.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 7, 8, 9, 15, 16, 17 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Beith et al. (US 2021/0065455)(Hereinafter referred to as Beith)
Regarding claim 1, Beith teaches A virtual operating method (Systems, apparatuses (or devices), methods, and computer-readable media are provided for generating virtual content. For example, a device (e.g., an extended reality device) can obtain an image of a scene of a real-world environment, wherein the real-world environment is viewable through a display of the extended reality device as virtual content is displayed by the display. The device can detect at least a part of a physical hand of a user in the image. The device can generate a virtual keyboard based on detecting at least the part of the physical hand. The device can determine a position for the virtual keyboard on the display of the extended reality device relative to at least the part of the physical hand. The device can display the virtual keyboard at the position on the display. See abstract),
wherein the method comprises:
displaying a virtual touch interface, wherein the virtual touch interface is associated with a position of a target plane in a physical space in which an electronic device is located (In some examples the virtual keyboard 228 can be generated and displayed in response to detecting the hands 226 of the user in the field of view of a camera of the AR device 202 (e.g., camera 108 shown in FIG. 1 or other camera). For instance, a processor of the AR device 202 (e.g., processor 114 shown in FIG. 1 or other component) can detect the actual (or physical) hands 226 of the user in an image captured by the camera of the AR device 202. See paragraph [0060]);
recognizing hand gesture information of a user (In some examples, an initial position of the virtual keyboard 228 can be determine using the locations of the landmark points of the hands 226 (and in some cases using points in the real-world scene or environment). In some cases, after the initial position of the virtual keyboard 228 is determined, the virtual keyboard 228 can move as the hands 226 of the user move. In some examples, the virtual keyboard 228 can maintain a same location or position relative to the fingertip location(s) as the hands 226 move. The virtual keyboard 228 may not move if the fingertips are within a threshold distance (e.g., 0.5 inch, 1 inch, 2 inches, and/or other distance) of selecting nearby keys. For instance, the virtual keyboard 228 can remain stationary when the fingertips are within the threshold distance of selecting nearby keys. See paragraph [0082]); and
performing a target operation in a case that position information, of a first region of the virtual touch interface, in the physical space and the hand gesture information meet a preset condition (In some examples, an initial position of the virtual keyboard 228 can be determine using the locations of the landmark points of the hands 226 (and in some cases using points in the real-world scene or environment). In some cases, after the initial position of the virtual keyboard 228 is determined, the virtual keyboard 228 can move as the hands 226 of the user move. In some examples, the virtual keyboard 228 can maintain a same location or position relative to the fingertip location(s) as the hands 226 move. The virtual keyboard 228 may not move if the fingertips are within a threshold distance (e.g., 0.5 inch, 1 inch, 2 inches, and/or other distance) of selecting nearby keys. For instance, the virtual keyboard 228 can remain stationary when the fingertips are within the threshold distance of selecting nearby keys. See paragraph [0082]).
Regarding claim 7, Beith teaches the method according to claim 1, wherein the recognizing the hand gesture information of the user comprises:
obtaining a fourth image of the physical space, wherein the fourth image comprises an image of a user hand (Image of hand at a different point in time); and
performing user hand gesture recognition on the fourth image to obtain at least one piece of hand joint point information (The different joints of the fingers of the hand 326 allow for different degrees of movement, as illustrated in the legend 339. As illustrated by the diamond shapes (e.g., diamond 333) in FIG. 3, the base of each finger (corresponding to the metacarpophalangeal joint (MCP) between the proximal phalanx and the metacarpal) has two degrees of freedom (2DOF) corresponding to flexion and extension as well as abduction and adduction. As illustrated by the circle shapes (e.g., circle 331) in FIG. 3, each of the upper joints of each finger (corresponding to the interphalangeal joints between the distal, middle, and proximal phalanges) has one degree of freedom (2DOF) corresponding flexion and extension. As a result, the hand 326 provides 26 degrees of freedom (26DOF) from which to register and track the virtual keyboard 228. See paragraph [0074]).
Regarding claim 8, Beith teaches The method according to claim 7, wherein the at least one piece of hand joint point information comprises at least one first coordinate of at least one hand joint in a second coordinate system (The different joints of the fingers of the hand 326 allow for different degrees of movement, as illustrated in the legend 339. As illustrated by the diamond shapes (e.g., diamond 333) in FIG. 3, the base of each finger (corresponding to the metacarpophalangeal joint (MCP) between the proximal phalanx and the metacarpal) has two degrees of freedom (2DOF) corresponding to flexion and extension as well as abduction and adduction. As illustrated by the circle shapes (e.g., circle 331) in FIG. 3, each of the upper joints of each finger (corresponding to the interphalangeal joints between the distal, middle, and proximal phalanges) has one degree of freedom (2DOF) corresponding flexion and extension. As a result, the hand 326 provides 26 degrees of freedom (26DOF) from which to register and track the virtual keyboard 228. See paragraph [0074]); and
the performing the target operation in the case that the position information, of the first region of the virtual touch interface, in the physical space and the hand gesture information meet the preset condition comprises: determining the first region of the virtual touch interface in a case that a spatial distance in the physical space between the at least one first coordinate and the virtual touch interface meets the preset condition; and performing the target operation based on operation information corresponding to the first region (In some examples, an initial position of the virtual keyboard 228 can be determine using the locations of the landmark points of the hands 226 (and in some cases using points in the real-world scene or environment). In some cases, after the initial position of the virtual keyboard 228 is determined, the virtual keyboard 228 can move as the hands 226 of the user move. In some examples, the virtual keyboard 228 can maintain a same location or position relative to the fingertip location(s) as the hands 226 move. The virtual keyboard 228 may not move if the fingertips are within a threshold distance (e.g., 0.5 inch, 1 inch, 2 inches, and/or other distance) of selecting nearby keys. For instance, the virtual keyboard 228 can remain stationary when the fingertips are within the threshold distance of selecting nearby keys. See paragraph [0082]).
Regarding claim 9, Beith teaches An electronic device (Systems, apparatuses (or devices), methods, and computer-readable media are provided for generating virtual content. For example, a device (e.g., an extended reality device) can obtain an image of a scene of a real-world environment, wherein the real-world environment is viewable through a display of the extended reality device as virtual content is displayed by the display. The device can detect at least a part of a physical hand of a user in the image. The device can generate a virtual keyboard based on detecting at least the part of the physical hand. The device can determine a position for the virtual keyboard on the display of the extended reality device relative to at least the part of the physical hand. The device can display the virtual keyboard at the position on the display. See abstract), comprising
a processor and a memory, wherein a program or instructions are stored in the memory and executable on the processor, and the program or instructions, when executed by the processor, cause the electronic device to (while one processor 114 and one memory 112 are shown in FIG. 1, the AR device 102 can include multiple processors and/or multiple memory devices in some implementations. The processor 114 and the memory 112 can store and execute instructions used to perform the techniques described herein. In implementations where the AR device 102 is in communication (wired or wirelessly) with the memory 112 and the processor 114, a device housing the memory 112 and the processor 114 may be a computing device, such as a desktop computer, a laptop computer, a mobile phone, a tablet, a game console, or other suitable device. The AR device 102 also includes or is in communication with (wired or wirelessly) an input device 116. The input device 116 can include any suitable input device, such as a touchscreen, a pen or other pointer device, a keyboard, a mouse a button or key, a microphone for receiving voice commands, a gesture input device for receiving gesture commands, any combination thereof, and/or other input device. In some cases, the camera 108 can capture images that can be processed for interpreting gesture commands. See paragraph [0048]) perform:
displaying a virtual touch interface, wherein the virtual touch interface is associated with a position of a target plane in a physical space in which the electronic device is located (In some examples the virtual keyboard 228 can be generated and displayed in response to detecting the hands 226 of the user in the field of view of a camera of the AR device 202 (e.g., camera 108 shown in FIG. 1 or other camera). For instance, a processor of the AR device 202 (e.g., processor 114 shown in FIG. 1 or other component) can detect the actual (or physical) hands 226 of the user in an image captured by the camera of the AR device 202. See paragraph [0060]);
recognizing hand gesture information of a user (In some examples, an initial position of the virtual keyboard 228 can be determine using the locations of the landmark points of the hands 226 (and in some cases using points in the real-world scene or environment). In some cases, after the initial position of the virtual keyboard 228 is determined, the virtual keyboard 228 can move as the hands 226 of the user move. In some examples, the virtual keyboard 228 can maintain a same location or position relative to the fingertip location(s) as the hands 226 move. The virtual keyboard 228 may not move if the fingertips are within a threshold distance (e.g., 0.5 inch, 1 inch, 2 inches, and/or other distance) of selecting nearby keys. For instance, the virtual keyboard 228 can remain stationary when the fingertips are within the threshold distance of selecting nearby keys. See paragraph [0082]); and
performing a target operation in a case that position information, of a first region of the virtual touch interface, in the physical space and the hand gesture information meet a preset condition (In some examples, an initial position of the virtual keyboard 228 can be determine using the locations of the landmark points of the hands 226 (and in some cases using points in the real-world scene or environment). In some cases, after the initial position of the virtual keyboard 228 is determined, the virtual keyboard 228 can move as the hands 226 of the user move. In some examples, the virtual keyboard 228 can maintain a same location or position relative to the fingertip location(s) as the hands 226 move. The virtual keyboard 228 may not move if the fingertips are within a threshold distance (e.g., 0.5 inch, 1 inch, 2 inches, and/or other distance) of selecting nearby keys. For instance, the virtual keyboard 228 can remain stationary when the fingertips are within the threshold distance of selecting nearby keys. See paragraph [0082]).
Regarding claim 15, Beith teaches The electronic device according to claim 9, wherein the program or instructions, when executed by the processor, cause the electronic device to perform: obtaining a fourth image of the physical space, wherein the fourth image comprises an image of a user hand (Image of hand at a different point in time); and
performing user hand gesture recognition on the fourth image to obtain at least one piece of hand joint point information (The different joints of the fingers of the hand 326 allow for different degrees of movement, as illustrated in the legend 339. As illustrated by the diamond shapes (e.g., diamond 333) in FIG. 3, the base of each finger (corresponding to the metacarpophalangeal joint (MCP) between the proximal phalanx and the metacarpal) has two degrees of freedom (2DOF) corresponding to flexion and extension as well as abduction and adduction. As illustrated by the circle shapes (e.g., circle 331) in FIG. 3, each of the upper joints of each finger (corresponding to the interphalangeal joints between the distal, middle, and proximal phalanges) has one degree of freedom (2DOF) corresponding flexion and extension. As a result, the hand 326 provides 26 degrees of freedom (26DOF) from which to register and track the virtual keyboard 228. See paragraph [0074]).
Regarding claim 16, Beith teaches The electronic device according to claim 15, wherein the at least one piece of hand joint point information comprises at least one first coordinate of at least one hand joint in a second coordinate system (The different joints of the fingers of the hand 326 allow for different degrees of movement, as illustrated in the legend 339. As illustrated by the diamond shapes (e.g., diamond 333) in FIG. 3, the base of each finger (corresponding to the metacarpophalangeal joint (MCP) between the proximal phalanx and the metacarpal) has two degrees of freedom (2DOF) corresponding to flexion and extension as well as abduction and adduction. As illustrated by the circle shapes (e.g., circle 331) in FIG. 3, each of the upper joints of each finger (corresponding to the interphalangeal joints between the distal, middle, and proximal phalanges) has one degree of freedom (2DOF) corresponding flexion and extension. As a result, the hand 326 provides 26 degrees of freedom (26DOF) from which to register and track the virtual keyboard 228. See paragraph [0074]); and the program or instructions, when executed by the processor, cause the electronic device to perform: determining the first region of the virtual touch interface in a case that a spatial distance in the physical space between the at least one first coordinate and the virtual touch interface meets the preset condition; and performing the target operation based on operation information corresponding to the first region (In some examples, an initial position of the virtual keyboard 228 can be determine using the locations of the landmark points of the hands 226 (and in some cases using points in the real-world scene or environment). In some cases, after the initial position of the virtual keyboard 228 is determined, the virtual keyboard 228 can move as the hands 226 of the user move. In some examples, the virtual keyboard 228 can maintain a same location or position relative to the fingertip location(s) as the hands 226 move. The virtual keyboard 228 may not move if the fingertips are within a threshold distance (e.g., 0.5 inch, 1 inch, 2 inches, and/or other distance) of selecting nearby keys. For instance, the virtual keyboard 228 can remain stationary when the fingertips are within the threshold distance of selecting nearby keys. See paragraph [0082]).
Regarding claim 17, Beith teaches A non-transitory readable storage medium, wherein a program or instructions are stored in the non-transitory readable storage medium, and the program or instructions, when executed by a processor of an electronic device, cause the electronic device (Systems, apparatuses (or devices), methods, and computer-readable media are provided for generating virtual content. For example, a device (e.g., an extended reality device) can obtain an image of a scene of a real-world environment, wherein the real-world environment is viewable through a display of the extended reality device as virtual content is displayed by the display. The device can detect at least a part of a physical hand of a user in the image. The device can generate a virtual keyboard based on detecting at least the part of the physical hand. The device can determine a position for the virtual keyboard on the display of the extended reality device relative to at least the part of the physical hand. The device can display the virtual keyboard at the position on the display. See abstract) (while one processor 114 and one memory 112 are shown in FIG. 1, the AR device 102 can include multiple processors and/or multiple memory devices in some implementations. The processor 114 and the memory 112 can store and execute instructions used to perform the techniques described herein. In implementations where the AR device 102 is in communication (wired or wirelessly) with the memory 112 and the processor 114, a device housing the memory 112 and the processor 114 may be a computing device, such as a desktop computer, a laptop computer, a mobile phone, a tablet, a game console, or other suitable device. The AR device 102 also includes or is in communication with (wired or wirelessly) an input device 116. The input device 116 can include any suitable input device, such as a touchscreen, a pen or other pointer device, a keyboard, a mouse a button or key, a microphone for receiving voice commands, a gesture input device for receiving gesture commands, any combination thereof, and/or other input device. In some cases, the camera 108 can capture images that can be processed for interpreting gesture commands. See paragraph [0048]) to perform:
displaying a virtual touch interface, wherein the virtual touch interface is associated with a position of a target plane in a physical space in which the electronic device is located (In some examples the virtual keyboard 228 can be generated and displayed in response to detecting the hands 226 of the user in the field of view of a camera of the AR device 202 (e.g., camera 108 shown in FIG. 1 or other camera). For instance, a processor of the AR device 202 (e.g., processor 114 shown in FIG. 1 or other component) can detect the actual (or physical) hands 226 of the user in an image captured by the camera of the AR device 202. See paragraph [0060]);
recognizing hand gesture information of a user (In some examples, an initial position of the virtual keyboard 228 can be determine using the locations of the landmark points of the hands 226 (and in some cases using points in the real-world scene or environment). In some cases, after the initial position of the virtual keyboard 228 is determined, the virtual keyboard 228 can move as the hands 226 of the user move. In some examples, the virtual keyboard 228 can maintain a same location or position relative to the fingertip location(s) as the hands 226 move. The virtual keyboard 228 may not move if the fingertips are within a threshold distance (e.g., 0.5 inch, 1 inch, 2 inches, and/or other distance) of selecting nearby keys. For instance, the virtual keyboard 228 can remain stationary when the fingertips are within the threshold distance of selecting nearby keys. See paragraph [0082]); and
performing a target operation in a case that position information, of a first region of the virtual touch interface, in the physical space and the hand gesture information meet a preset condition (In some examples, an initial position of the virtual keyboard 228 can be determine using the locations of the landmark points of the hands 226 (and in some cases using points in the real-world scene or environment). In some cases, after the initial position of the virtual keyboard 228 is determined, the virtual keyboard 228 can move as the hands 226 of the user move. In some examples, the virtual keyboard 228 can maintain a same location or position relative to the fingertip location(s) as the hands 226 move. The virtual keyboard 228 may not move if the fingertips are within a threshold distance (e.g., 0.5 inch, 1 inch, 2 inches, and/or other distance) of selecting nearby keys. For instance, the virtual keyboard 228 can remain stationary when the fingertips are within the threshold distance of selecting nearby keys. See paragraph [0082]).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 6, 14 are rejected under 35 U.S.C. 103 as being unpatentable over Beith et al. (US 2021/0065455)(Hereinafter referred to as Beith) in view of Wang et al. (“Superpixel-Based Hand Gesture Recognition with Kinect Depth Camera”, IEEE. 2015)(Hereinafter referred to as Wang).
Regarding claim 6, Beith teaches the method according to claim 1, but is silent to wherein before the displaying the virtual touch interface, the method further comprises: obtaining a third image of the physical space, wherein the third image comprises an image of a user hand;
based on the third image, calculating a third point cloud of the physical space in a first coordinate system, wherein the third point cloud comprises position and posture information of the user hand; determining a third plane in the third point cloud based on the position and posture information of the user hand; and determining a screen corresponding to the third plane in the physical space as the target plane; wherein the first coordinate system is a coordinate system corresponding to the physical space.
Wang teaches capturing an image and projecting the 3D point cloud onto an image plane and performing graphical interfacing with hand gestures (This paper presents a new superpixel-based hand gesture recognition system based on a novel superpixel earth mover’s distance metric, together with Kinect depth camera. The depth and skeleton information from Kinect are effectively utilized to produce markerless hand extraction. The hand shapes, corresponding textures and depths are represented in the form of superpixels, which effectively retain the overall shapes and color of the gestures to be recognized. Based on this representation, a novel distance metric, superpixel earth mover’s distance (SP-EMD), is proposed to measure the dissimilarity between the hand gestures. This measurement is not only robust to distortion and articulation, but also invariant to scaling, translation and rotation with proper preprocessing. The effectiveness of the proposed distance metric and recognition algorithm are illustrated by extensive experiments with our own gesture dataset as well as two other public datasets. Simulation results show that the proposed system is able to achieve high mean accuracy and fast recognition speed. Its superiority is further demonstrated by comparisons with other conventional techniques and two real-life applications. See abstract)(Meanwhile, moderate out-of-plane rotation of the hand is compensated with the aid of the depth map. More precisely, the palm plane is first estimated from the 3D point cloud of the hand, which is then rotated to ensure that the palm plane is parallel to the image plane. Finally, the rotated point cloud is projected back to image plane, which is then utilized for the later steps of superpixel generation and hand gesture recognition. See page 32, third paragraph).
Beith and Wang teach of virtual interfaces and Wang teaches that out-of-plane rotation can be compensated by estimating the palm plane from a 3D point cloud and rotating and projecting, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of Beith with the out-of-plane rotation compensation techniques of Wang such that could properly capture the individuals intended operation in a wider range.
Regarding claim 14, Beith teaches the electronic device according to claim 9, but is silent to wherein the program or instructions, when executed by the processor, cause the electronic device to further perform: obtaining a third image of the physical space, wherein the third image comprises an image of a user hand; based on the third image, calculating a third point cloud of the physical space in a first coordinate system, wherein the third point cloud comprises position and posture information of the user hand; determining a third plane in the third point cloud based on the position and posture information of the user hand; and determining a screen corresponding to the third plane in the physical space as the target plane; wherein the first coordinate system is a coordinate system corresponding to the physical space.
Wang teaches capturing an image and projecting the 3D point cloud onto an image plane and performing graphical interfacing with hand gestures (This paper presents a new superpixel-based hand gesture recognition system based on a novel superpixel earth mover’s distance metric, together with Kinect depth camera. The depth and skeleton information from Kinect are effectively utilized to produce markerless hand extraction. The hand shapes, corresponding textures and depths are represented in the form of superpixels, which effectively retain the overall shapes and color of the gestures to be recognized. Based on this representation, a novel distance metric, superpixel earth mover’s distance (SP-EMD), is proposed to measure the dissimilarity between the hand gestures. This measurement is not only robust to distortion and articulation, but also invariant to scaling, translation and rotation with proper preprocessing. The effectiveness of the proposed distance metric and recognition algorithm are illustrated by extensive experiments with our own gesture dataset as well as two other public datasets. Simulation results show that the proposed system is able to achieve high mean accuracy and fast recognition speed. Its superiority is further demonstrated by comparisons with other conventional techniques and two real-life applications. See abstract)(Meanwhile, moderate out-of-plane rotation of the hand is compensated with the aid of the depth map. More precisely, the palm plane is first estimated from the 3D point cloud of the hand, which is then rotated to ensure that the palm plane is parallel to the image plane. Finally, the rotated point cloud is projected back to image plane, which is then utilized for the later steps of superpixel generation and hand gesture recognition. See page 32, third paragraph).
Beith and Wang teach of virtual interfaces and Wang teaches that out-of-plane rotation can be compensated by estimating the palm plane from a 3D point cloud and rotating and projecting, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of Beith with the out-of-plane rotation compensation techniques of Wang such that could properly capture the individuals intended operation in a wider range.
Allowable Subject Matter
Claims 2-5, 10-13, 18-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The closest prior art of record is Beith et al. (US 2021/0065455)(Hereinafter referred to as Beith) in view of Wang et al. (“Superpixel-Based Hand Gesture Recognition with Kinect Depth Camera”, IEEE. 2015)(Hereinafter referred to as Wang).
Beith teaches interacting with a virtual keyboard (Systems, apparatuses (or devices), methods, and computer-readable media are provided for generating virtual content. For example, a device (e.g., an extended reality device) can obtain an image of a scene of a real-world environment, wherein the real-world environment is viewable through a display of the extended reality device as virtual content is displayed by the display. The device can detect at least a part of a physical hand of a user in the image. The device can generate a virtual keyboard based on detecting at least the part of the physical hand. The device can determine a position for the virtual keyboard on the display of the extended reality device relative to at least the part of the physical hand. The device can display the virtual keyboard at the position on the display. See abstract).
Wang teaches capturing an image and projecting the 3D point cloud onto an image plane and performing graphical interfacing with hand gestures (This paper presents a new superpixel-based hand gesture recognition system based on a novel superpixel earth mover’s distance metric, together with Kinect depth camera. The depth and skeleton information from Kinect are effectively utilized to produce markerless hand extraction. The hand shapes, corresponding textures and depths are represented in the form of superpixels, which effectively retain the overall shapes and color of the gestures to be recognized. Based on this representation, a novel distance metric, superpixel earth mover’s distance (SP-EMD), is proposed to measure the dissimilarity between the hand gestures. This measurement is not only robust to distortion and articulation, but also invariant to scaling, translation and rotation with proper preprocessing. The effectiveness of the proposed distance metric and recognition algorithm are illustrated by extensive experiments with our own gesture dataset as well as two other public datasets. Simulation results show that the proposed system is able to achieve high mean accuracy and fast recognition speed. Its superiority is further demonstrated by comparisons with other conventional techniques and two real-life applications. See abstract)(Meanwhile, moderate out-of-plane rotation of the hand is compensated with the aid of the depth map. More precisely, the palm plane is first estimated from the 3D point cloud of the hand, which is then rotated to ensure that the palm plane is parallel to the image plane. Finally, the rotated point cloud is projected back to image plane, which is then utilized for the later steps of superpixel generation and hand gesture recognition. See page 32, third paragraph).
Beith and Wang alone or in combination are silent to the limitations “determining, from the first point cloud, a target point cloud located in a target coordinate range, wherein the target coordinate range is determined based on coordinate information of the electronic device in the first coordinate system; and determining a target plane based on the target point cloud; wherein the first coordinate system is a coordinate system corresponding to the physical space.” Of claim 2 when read in light of the rest of the limitations in claim 2 and the claims to which claim 2 depends and thus claim 2 contains allowable subject matter.
Claims 3-5 contain allowable subject matter because they depend on a claim containing allowable subject matter.
The prior art of record alone or in combination is silent to the limitations “determining, from the first point cloud, a target point cloud located in a target coordinate range, wherein the target coordinate range is determined based on coordinate information of the electronic device in the first coordinate system; and determining a target plane based on the target point cloud; wherein the first coordinate system is a coordinate system corresponding to the physical space.” Of claim 10 when read in light of the rest of the limitations in claim 10 and thus claim 10 contains allowable subject matter.
Claims 11-13 contain allowable subject matter because they depend on a claim containing allowable subject matter.
The prior art of record alone or in combination is silent to the limitations “determining, from the first point cloud, a target point cloud located in a target coordinate range, wherein the target coordinate range is determined based on coordinate information of the electronic device in the first coordinate system; and determining a target plane based on the target point cloud; wherein the first coordinate system is a coordinate system corresponding to the physical space. ” Of claim 18 when read in light of the rest of the limitations in claim 18 and thus claim 18 contains allowable subject matter.
Claims 19-20 contain allowable subject matter because they depend on a claim containing allowable subject matter.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kinstner et al. (US 2017/0330378)(Hereinafter referred to as Kinstner), generally teaches a point cloud representation of a user’s hand interacting with interfaces based on a gesture (See paragraph [0076])
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS R WILSON whose telephone number is (571)272-0936. The examiner can normally be reached M-F 7:30-5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (572)-272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICHOLAS R WILSON/Primary Examiner, Art Unit 2611
Prosecution Insights