DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The IDS filed 28 January 2025 has been received and considered.
Claims 1 – 20 are pending.
Claims 1 – 5, 11 – 15, and 17 – 20 are rejected.
Claims 6 – 10 and 16 are objected to.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 – 2, 5, 11 – 12, 15, and 17 – 18 are rejected under 35 U.S.C. 103 as being unpatentable over US Publication No. 2012/0105613 A1 to Weng et al. (hereinafter Weng) in view of US Publication No. 2026/0170815 A1 to Denney in further view of US Publication No. 2011/0110560 A1 to Adhikari.
Claim 1
Regarding Claim 1, an independent method claim, Weng teaches a method for recognizing a gesture, wherein the gesture comprises pose information and gesture command information, and the method comprises: acquiring a plurality of frames of images comprising a hand object within a first preset duration before a current time ("The invention may provide a system that serves as a backbone for an in-car hand gesture recognition system. In order to operate this system, one or more video acquisition devices such as camera(s) may be used to capture hand gestures performed by the vehicle operator on a background surface. The background surface may be the surface of the steering wheel or another designated surface that will not distract the operator. The inventive system may process the acquired video stream in real-time, and can be used to send the desired in-car commands that are indicated by the hand gestures to several devices.", Paragraph [0037]);
respectively performing feature extraction on the plurality of frames of images to obtain a feature vector corresponding to each of the plurality of frames of images ("After the hand region is identified in each video frame in hand region detection module 142, feature extraction module 144 may be used to extract features to model the hand motion trajectory for gesture recognition by handwriting and gesture recognition module 118. In one embodiment, two types of features are extracted by feature extraction module 144. The first feature is the center of mass (COM) of the hand region. This parameter may be extracted because COM is very robust and insensitive to the misclassification of the skin pixels near the boundary of the detected hand region. The COM values may be computed by evaluating the zero order moment of all skin pixels inside the detected hand region.". Paragraph [0073]; "Fingertip position may be chosen as the second feature to be extracted by feature extraction module 144 because it may be more precise to use fingertip position to represent the handwriting trajectories. A K-curvature algorithm may be used to detect the fingertip position in each video frame. This algorithm may assume that the finger is the sharpest point in the hand contour.", Paragraph [0074]; "Once the COM and the fingertip position are computed for each frame, the COM and the fingertip position may be normalized and combined/concatenated to a feature vector. The sequence of these feature vectors may be used as the input to gesture recognition module 118. The combination of COM and fingertip position is a novel way of modeling the hand motion trajectory, and this combination provides a novel and effective method of gesture recognition.", Paragraph [0075]);
for a current frame of image, determining at least one frame of target image before the current frame of image ("Once the COM and the fingertip position are computed for each frame, the COM and the fingertip position may be normalized and combined/concatenated to a feature vector. The sequence of these feature vectors may be used as the input to gesture recognition module 118.", Paragraph [0075]).
a feature vector corresponding to each of the at least one frame of target image (Rejected as applied directly above, i.e. Paragraph [0075]), wherein the feature vectors are determined for all of the frames (current and previous frames “target frames” of a sequence of frames).
Weng does not explicitly teach determining pose information corresponding to the hand object according to a feature vector corresponding to the current frame of image and a preset deep learning model.
However, Denney teaches a preset deep learning model ("According the present disclosure, a gesture recognition system that uses a deep learning hand skeleton model to recognize static hand gestures is described herein. The present disclosure describes a hand skeleton detector that obtains information about the skeletal position of a hand in a captured image. A hand feature vector generation module performs processing to detect, from the skeleton detection results, a feature of a hand within the captured image data. The detected feature of the hand represents it's position in the particular frame and, based on that, can be used to predict whether the current state of the hand is performing one or more gestures that may be used to control an operation of a computing system. The hand feature is provided as a hand feature vector as an input to a static gesture learning module that has been trained according to images of hands that are in one of the various positions associated with a particular gesture. A corresponding gesture inference module performs a first classification of the state of the hand based on the hand feature vector to provide a first likelihood value that the hand is performing the gesture or not performing the gesture. The first likelihood value is provided as in input to a final gesture determination module which outputs a final prediction that the hand in the captured image is performing the gesture or not performing the gesture based on the current hand gesture inference module results and the prior final gesture determination module results. Thus, the described gesture detection processing advantageously uses the transition information that exists between a current prediction and previous final prediction to increase the confidence of the system in identifying whether or not a person in the image is intending to (or actually performing) a gesture that can be used to control a computing system.", Paragraph [0015]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Weng to incorporate utilizing a Deep Learning Model, as disclosed by Denney. The suggestion/motivation for doing so would have been to train the deep learning model to perform the functions by the various modules disclosed in Weng for better efficiency and accuracy.
Neither Weng, or Denney, or the combination teach determining pose information corresponding to the hand object according to a feature vector corresponding to the current frame of image.
However, Adhikari teaches determining pose information corresponding to the hand object according to a feature vector corresponding to the current frame of image (Paragraphs [0098 - 0116], with emphasis on [0099]), wherein the tracking of pose information from frame to frame using feature vectors is disclosed.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the teachings of Weng, in view of Denney, to incorporate determining pose information on all the frames of a sequence of frames, as disclosed by Adhikari. The suggestion/motivation for doing so would have been to use the pose information from a current frame, along with the previous frames, to determine a directional movement (or motion trajectory) of the movement of the hand. Furthermore, the deep learning model could further be trained to perform the functions of the algorithm(s) used to track the hand pose information between frames, allowing for even greater efficiency.
Weng, in view of Denney and Adhikari, further teaches determining gesture command information corresponding to the hand object according to the feature vector corresponding to each of the plurality of frames of images and the preset deep learning model ("The inventive system may process the acquired video stream in real-time, and can be used to send the desired in-car commands that are indicated by the hand gestures to several devices. By using the framework of this inventive system, the driver can give commands to the in-car devices in a very natural and effortless way while driving.", Paragraph [0037]; "Referring now to the drawings, and particularly to FIG. 1, there is shown one embodiment of a gesture-based information and command entry system 10 of the present invention, including a surface 12 on which a user may "air-write" by sliding or moving his index finger 14 across surface 12. Finger 14 may maintain contact with surface 12 throughout the writing process, may contact surface 12 intermittently throughout the writing process, or may not contact surface 12 at all throughout the writing process.", Paragraph [0043]; and Paragraph [0059]), wherein it would have been obvious to one skilled in the art to utilize the deep learning model, disclosed by Denney (further trained on the already discussed modules), to increase the efficiency of the Command entry system of Weng.
Claim 2
Regarding Claim 2, dependent on claim 1, Weng, in view of Denney and Adhikari, teach the invention as claimed in claim 1.
Weng further wherein the respectively performing feature extraction on the plurality of frames of images to obtain the feature vector corresponding to each of the plurality of frames of images, comprises: respectively performing feature extraction on the plurality of frames of images according to an image feature extraction model, to obtain the feature vector corresponding to each of the plurality of frames of images (Rejected as applied to claim 1, specifically Paragraphs [0073 - 0075]).
Claim 5
Regarding Claim 5, dependent on claim 1, Weng, in view of Denney and Adhikari, teach the invention as claimed in claim 1.
Weng further teaches wherein a total number of the plurality of frames of images is M, the current frame of image is an M-th frame of image, and M is a positive integer greater than 1 ("The inventive system may process the acquired video stream in real-time, and can be used to send the desired in-car commands that are indicated by the hand gestures to several devices.", Paragraph [0037]; ""After the hand region is identified in each video frame in hand region detection module 142, feature extraction module 144 may be used to extract features to model the hand motion trajectory for gesture recognition by handwriting and gesture recognition module 118.", Paragraph [0073]), wherein one skilled in the art would know that the number of frames in a sequence of frames would be the total amount a frames "M", the last frame in the sequence a frames (especially if in real-time) would be the current frame "M-th frame", and that the total amount of frames would be greater than one as a sequence of frames is further known to be 3 or more; and
for the current frame of image, the determining at least one frame of target image before the current frame of image comprises: for the current frame of image, determining that previous N frames of image before the current frame of image are target images, wherein N is a positive integer, and M is greater than N (Rejected as applied directly above), wherein one skilled in the art would also know that the previous frames before the current frame (when tracking hand regions from frame to frame in a sequence of frames) would be the target images "N" and M would have to be greater than N since M (being the total frames) would include the previous frames "target frames" AND the current frame.
Claim 11, an independent electronic device claim, is rejected for the same reasons as applied to claim 1.
Claim 17, an independent non-transitory computer readable storage medium claim, is rejected for the same reasons as applied to claim 1.
Claims 12, 15, and 18 are rejected for the same reasons as applied to the above claims (same claim language).
Claims 3 – 4, 13 – 14, and 19 – 20 are rejected under 35 U.S.C. 103 as being unpatentable over US Publication No. 2012/0105613 A1 to Weng et al. (hereinafter Weng) in view of US Publication No. 2026/0170815 A1 to Denney in further view of US Publication No. 2011/0110560 A1 to Adhikari in further view of US Publication No. 2020/0387698 A1 to YI et al. (hereinafter YI).
Claim 3
Regarding Claim 3, dependent on claim 1, Weng, in view of Denney and Adhikari, teach the invention as claimed in claim 1.
Neither Weng, or Denney, or the combination teach wherein the preset deep learning model comprises a pose fusion module; and
the determining the pose information corresponding to the hand object according to the feature vector corresponding to the current frame of image, the feature vector corresponding to each of the at least one frame of target image and the preset deep learning model, comprises: determining the pose information corresponding to the hand object according to the feature vector corresponding to the current frame of image, the feature vector corresponding to each of the at least one frame of target image and the pose fusion module,
wherein the pose information comprises coordinate information of a plurality of key points corresponding to the hand object, and
rotation angle information of each key point among the plurality of key points.
However, Adhikari further teaches wherein the preset deep learning model comprises a pose fusion module ("A hand tracking and pose processing module 216 is also illustrated. The hand tracking and pose processing module 216 provides processing capabilities for the core processing module 200 to perform the automated real time hand tracking, pose classification, and interface control, as described above and in more detail below. The camera 112 is illustrated and again provides image capture capabilities for the core processing module 200.", Paragraph [0035]); and
the determining the pose information corresponding to the hand object according to the feature vector corresponding to the current frame of image, the feature vector corresponding to each of the at least one frame of target image and the preset deep learning model, comprises: determining the pose information corresponding to the hand object according to the feature vector corresponding to the current frame of image, the feature vector corresponding to each of the at least one frame of target image and the pose fusion module (Paragraphs [0098 - 0116]), wherein the tracking of pose information from frame to frame is disclosed
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the teachings of Weng, in view of Denney and Adhikari, to incorporate determining pose information on all the frames of a sequence of frames, as disclosed by Adhikari. The suggestion/motivation for doing so would have been to use the pose information from a current frame, along with the previous frames, to determine a directional movement (or motion trajectory) of the movement of the hand.
Neither Weng, or Denney, or Adhikari, or the combination teach wherein the pose information comprises coordinate information of a plurality of key points corresponding to the hand object, and
rotation angle information of each key point among the plurality of key points.
However, YI teaches wherein the pose information comprises coordinate information of a plurality of key points corresponding to the hand object ("a hand key-point recognition model based on a deep neural network is generally used for hand key-point recognition.", Paragraph [0049]); " In one embodiment of the present disclosure, the model training device trains the hand key-point recognition model by using a deep learning method of coupling the two-dimensional heatmap to the three-dimensional key-point coordinates.", Paragraph [0138]), and
rotation angle information of each key point among the plurality of key points ("The gyroscope sensor 1012 may detect a body direction and a rotation angle of the hand key-point recognition device 1000.", Paragraph [0214]), wherein one skilled in the art would know that detecting the body direction and the hand rotation angle can allow you to estimate or calculate the rotation angles of a plurality of other keypoints from frame to frame.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the teachings of Weng, in view of Denney and Adhikari, to incorporate determining the coordinate information and the rotation angle of the plurality of keypoints, as disclosed by YI. The suggestion/motivation for doing so would have been to gather that information to potentially be used in determining the gesture of the hand in a sequence of frames.
Claim 4
Regarding Claim 4, dependent on claim 3, Weng, in view of Denney, Adhikari, and YI, teach the invention as claimed in claim 3.
Weng, in view of Denney, Adhikari, and YI, further teaches wherein the preset deep learning model further comprises a command fusion module ("The information and commands denoted by the hand gestures may be captured by a gesture capturing module, recognized by a recognition module, confirmed via a confirmation module, and then transmitted to corresponding devices within the vehicle.", Paragraph [0036]), wherein it is obvious to one skilled in the art that the various modules cited in the prior art used in the claim 1 rejection could be performed by a further trained deep learning model cited by Denney in the claim 1 rejection; and
wherein the gesture command information comprises a gesture command with an interactive function ("The output of gesture recognition module 18 may be in the form of a word, letter or number that corresponds to a possible selection, command, input, or piece of information provided by the user. For example, the output of gesture recognition module 18 may be in the form of the letter "B", or the numeral "3" corresponding to particular respective menu selections. The output of gesture recognition module 18 may also be in the form of a command such as "find" for a navigation system, "play" for an entertainment system, or "defrost" for an HVAC system, for example. Associated with such commands may be input data that the user also writes on surface 12, before or after the command. For example, associated with the command "find" may be the address "10 Elm Street". As another example, associated with the command "play" may be "Beethoven's Fifth Symphony", or a radio station identification such as "WGN" or "720 AM". As yet another example, associated with a "heat" command for an HVAC system may be a temperature level such as "75 F" and/or a location such as "driver".", Paragraph [0049]), wherein the interactive function are the commands in which the user is telling the vehicle to perform.
Claims 13 – 14 and 19 – 20 are rejected for the same reasons as applied to the above claims (same claim language).
Allowable Subject Matter
Claims 6 – 10 and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: While it would have been known to one skilled in the art to train a deep learning model to perform the functions of all of the modules cited in the prior art used for the rejection of the claims in the current application, the specific method used to train said deep learning model would not have been so easily known. Specifically, the claim limitations of claim 6, which states a training process of the preset deep learning model comprises: a first stage: acquiring a plurality of frames of sample images comprising the hand object within a second preset duration, and for each sample image, training an initial pose fusion module by taking the sample image and at least one frame of target image before the sample image as training samples to obtain a trained pose fusion module; and
a second stage: training an initial command fusion module using the plurality of frames of sample images comprising the hand object within the second preset duration to obtain a trained command fusion module.
The Examiner was unable to find the specific claim language cited above as being disclosed by Weng, Denney, Adhikari, and YI, either alone or in combination, and the remaining prior art of record fails to cure these deficiencies. Therefore, the Examiner has objected to claim 6. Claims 7 – 10 and 16 have similar claim language and have also been objected to.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ronde Miller whose telephone number is (703) 756-5686 The examiner can normally be reached Monday-Friday 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Gregory Morse can be reached on (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RONDE LEE MILLER/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights