DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: "module" in claims 1, 3-5, 7, 11, 13-15, and 17.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Objections
Claims 1 and 20 objected to because of the following informalities:
Claim 1: “a continuous image” in the limitation “an image capture module, installed in the operational area, configured to capture a continuous image of the at least one hand movement of the operator during the operation period” should be “continuous images”.
Claim 1: “the continuous image” in the limitation “a 2D joint recognition model, configured to recognize a hand joint position of the operator based on the continuous image” should be “the continuous images”.
Claim 1: “continuous images” in the limitation “configured to generate a hand joint position coordinate of the operator based on a plurality of continuous images from various perspectives captured by a plurality of image capture modules in the operational area during the operation period” should be “the continuous images”.
Claim 11: “a continuous image” in the limitation “an image capture module capturing a continuous image of the at least one hand movement of the operator during the operation period” should be “continuous images”.
Claim 11: “the continuous image” in the limitation “a 2D joint recognition model of a recognition module recognizing a hand joint position of the operator based on the continuous image” should be “the continuous images”.
Claim 11: “continuous images” in the limitation “a joint 3D coordinate recognition model of the recognition module generating a hand joint position coordinate of the operator based on a plurality of continuous images from various perspectives captured by a plurality of image capture modules in the operational area during the operation period” should be “the continuous images”.
Appropriate correction is required.
Drawings
The drawings are objected to because the drawings, particularly Figs. 1-4, should reflect the correct number of "image capture modules" as further explained below in the 112 Rejection. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
It is unclear whether there is one “image capture module” or “a plurality of image capture modules” as recited in claim 1 and 10. Claims dependent on claims 1 and 10 should also be clear whether there are one or a plurality of “image capture modules”, particularly in claims 2-3, 8, 12-13, and 18.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 8, 10-15, 18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sankai et al. (US-20250174047-A1) and further in view of Mao et al. (US-20180024641-A1).
Regarding claim 1, Sankai teaches:
A system for hand movements recognition and analysis (“a hand-motion monitoring apparatus of the present invention includes an imaging unit that sequentially obtains RGB images and depth images by imaging an upper body including both hands of a subject person,” Para [0015]) for evaluating the correctness of at least one hand movement performed by an operator in an operational area during an operation period (“a behavior checking unit that checks the hand-motion behavior recognized by the behavior recognition unit” Para [0015]), the system comprising:
an image capture module, installed in the operational area, configured to capture a continuous image of the at least one hand movement of the operator during the operation period (“a hand-motion monitoring apparatus of the present invention includes an imaging unit that sequentially obtains RGB images and depth images by imaging an upper body including both hands of a subject person,” Para [0015]);
a recognition module, coupled with the image capture module (“a skeleton information acquisition unit that obtains pieces of temporally continuous three-dimensional skeleton information on mainly the both hands of the subject person,” Para [0015]), the recognition module comprising:
a 2D joint recognition model, configured to recognize a hand joint position of the operator based on the continuous image (“The coordinates extraction unit 11 sequentially extracts from the image coordinates, which have been set by the image coordinates setting unit 10, image coordinates related to a hand-washing behavior of the subject person,” Para [0047]);
a joint 3D coordinate recognition model, coupled with the 2D joint recognition model, and configured to generate a hand joint position coordinate of the operator based on a three-dimensional skeleton information on mainly the both hands of the subject person by temporally synchronizing the image coordinates sequentially extracted by the coordinates extraction unit with the depth images sequentially obtained from the imaging unit,” Para [0015]); and
wherein the 2D joint recognition model, the joint 3D coordinate recognition model,
a movement analysis module, coupled with the recognition module, the movement analysis module comprising a hand movement analysis model configured to receive the hand joint position coordinate (“The behavior recognition unit 13 sequentially recognizes the relevant local actions from the pieces of three-dimensional skeleton information sequentially obtained by the skeleton information acquisition unit 12,” Para [0058]), analyze and compare the hand joint position coordinate of the operator with a standard hand movement parameter set (“action recognition pattern set”; “The behavior recognition unit 13 sequentially recognizes the relevant local actions from the pieces of three-dimensional skeleton information… with reference to a hand-washing-action recognition model constructed through deep learning by using as training data an action recognition pattern set for each local action,” Para [0058]),
and generate an analysis and comparison result to evaluate the correctness of the at least one hand movement performed by the operator during the operation period (“The behavior checking unit 14 checks the hand-washing behavior recognized by the behavior recognition unit 13 by determining if the recognition accuracy of each local action of the hand-washing behavior is lower than a predetermined threshold,” Para [0063]),
wherein the hand movement analysis model is trained on a standard operational hand movement dataset by a second machine learning method (“The behavior recognition unit 13 sequentially recognizes the relevant local actions from the pieces of three-dimensional skeleton information… with reference to a hand-washing-action recognition model constructed through deep learning by using as training data an action recognition pattern set for each local action,” Para [0058]).
Sankai is not relied upon to teach the following limitations. Mao, however, further teaches:
a joint 3D coordinate recognition model, coupled with the 2D joint recognition model, and configured to generate a hand joint position coordinate of the operator based on a plurality of continuous images from various perspectives captured by a plurality of image capture modules (that obtain “stereo images”) in the operational area during the operation period, wherein the hand joint position coordinate is a 3D coordinate (“the one or more images may comprise two stereo images of the portion of the object, and the two stereo images may be captured by two cameras that are used to generate 3D positions of the hand joints,” Para [0019]); and
a hand skeletal joint position global optimization model, coupled with the joint 3D coordinate recognition model, and configured to optimize the hand joint position coordinate based on a finger bone length constraint (“bone length”) and a finger bone joint constraint (“joint angles”) of the operator through a global optimization processing to improve the accuracy of the hand joint position coordinate (“the method may further comprise subjecting the obtained 3D positions of the one or more locations (e.g., hand joints) to one or more constraints to obtain refined 3D positions of the one or more locations,” Para [0019]; where the constraints are “human bone-joint constraints such as bone lengths and joint angles,” Para [0044])),
wherein the 2D joint recognition model, the joint 3D coordinate recognition model, and the hand skeletal joint position global optimization model are trained on a hand movement dataset by a first machine learning method (“training the detection model and the extraction model may comprises various steps, such as training a detection model, refining the detection model, training an extraction model, and refining the extraction model, Para [0020]).
Mao is considered to be analogous to the claimed invention because they are both in the field of 3D hand tracking. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Mao into Sankai for the benefit of more accurate 3D hand identification and tracking, which will lead to more accurate hand posture/gesture analysis.
Regarding claim 2, the rejection of claim 1 is incorporated herein. Sankai in view of Mao teaches the system of claim 1, and Mao further teaches:
wherein the 3D coordinate is calculated by triangulating an intersection of a ray for each of the image capture modules (“obtaining the 3D positions of the one or more locations on the tracked portion of the object based at least in part on the obtained 2D positions comprises obtaining the 3D positions of the one or more locations on the tracked portion of the object through a triangulation method,” Para [0025]), or by using a midpoint calculation of the shortest distance of each ray from the image capture modules as an approximate intersection point.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Mao into Sankai for the benefit of more accurate 3D hand identification and tracking, which will lead to more accurate hand posture/gesture analysis.
Regarding claim 3, the rejection of claim 1 is incorporated herein. Sankai in view of Mao teaches the system of claim 1, and Sankai further teaches:
wherein the movement analysis module further comprises:
a movement classification recognition model, configured to recognize and classify different types of hand movements (“the following eight local actions (A) to (H) have been defined as a hand-washing behavior as illustrated in FIG. 3. (A) Wet hands, (B) Apply soap, (C) Rub to palm, (D) Rub back of each hand, (E) Rub fingers, (F) Rub between each finger, (G) Rub thumb, and (H) Rub wrist,” Para [0057]) based on the hand joint position coordinate in the continuous images captured by the image capture module through a third machine learning method (“The behavior recognition unit 13 sequentially recognizes the relevant local actions from the pieces of three-dimensional skeleton information sequentially obtained by the skeleton information acquisition unit 12 with reference to a hand-washing-action recognition model constructed through deep learning by using as training data an action recognition pattern set for each local action,” Para [0058]).
Regarding claim 4, the rejection of claim 1 is incorporated herein. Sankai in view of Mao teaches the system of claim 1, and Sankai further teaches:
wherein the movement analysis module further comprises: a movement and sequence accuracy recognition model, configured to analyze and compare the hand joint position coordinate with the standard hand movement parameter set (“a hand-washing-action recognition model constructed through deep learning by using as training data an action recognition pattern set for each local action,” Para [0058]), and determine whether the at least one hand movement of the operator and an operation sequence are correct (“The behavior checking unit 14 checks the hand-washing behavior recognized by the behavior recognition unit 13 by determining if the recognition accuracy of each local action of the hand-washing behavior is lower than a predetermined threshold,” Para [0063]).
Regarding claim 5, the rejection of claim 1 is incorporated herein. Sankai in view of Mao teaches the system of claim 1, and Sankai further teaches:
a notification module, coupled with the movement analysis module, the notification module configured to send a notification message to alert the operator based on the analysis and comparison result generated by analyzing and comparing the hand joint position coordinate of the operator in the operational area during the operation period with the standard hand movement parameter set (“Based on the results of the check performed by the behavior checking unit 14, the action improvement proposal unit 15 gives the subject person feedback about a proposal for improving a local action with recognition accuracy lower than the threshold among the local actions of the hand-washing behavior,” Para [0069]).
Regarding claim 8, the rejection of claim 1 is incorporated herein. Sankai in view of Mao teaches the system of claim 1, and Sankai further teaches:
wherein the image capture module comprises a camera installed in the operational area (“the RGB-D sensor 3 is disposed to image the upper body including both hands of the subject person with reference to the standing position of the subject person and the faucet in the hand-washing facility, and sequentially obtains RGB images and depth images as a result of imaging,” Para [0042]).
Regarding claim 10, the rejection of claim 3 is incorporated herein. Sankai in view of Mao teach the system of claim 3, and Sankai further teaches:
wherein the convolutional neural network (CNN) layer,” Para [0059]) and the third machine learning method (“the behavior recognition unit 13 applies a hand-washing-behavior recognition model including three modules, which include a convolutional neural network (CNN) layer,” Para [0059]) further comprise at least one of the following: an Artificial Neural Network (ANN), a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Decision Tree, a Support Vector Machine (SVM), a Random Forest, a K-Nearest Neighbors (KNN) algorithm, K-Means Clustering, Principal Component Analysis (PCA), Linear Regression, Logistic Regression, Gradient Boosting Machines, a Deep Belief Network (DBN), a Recursive Neural Network (RecNN), Reinforcement Learning, an Autoencoder, Gaussian Processes, and a Complex Neural Network.
Sankai is not relied upon to teach the following limitation. However, Mao further teaches:
the first machine learning method (“system 100 may train a machine learning model with the dataset and the ground truth to obtain a hand detection model. The machine learning model may include a random forest method, a convolution neural network (CNN) method, etc,” Para [0061])… further comprise[s] at least one of the following: an Artificial Neural Network (ANN), a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Decision Tree, a Support Vector Machine (SVM), a Random Forest, a K-Nearest Neighbors (KNN) algorithm, K-Means Clustering, Principal Component Analysis (PCA), Linear Regression, Logistic Regression, Gradient Boosting Machines, a Deep Belief Network (DBN), a Recursive Neural Network (RecNN), Reinforcement Learning, an Autoencoder, Gaussian Processes, and a Complex Neural Network.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Mao into Sankai for the benefit of more accurate 3D hand identification and tracking, which will lead to more accurate hand posture/gesture analysis.
Regarding claims 11-15, 18, and 20, the claims are considered to be corresponding method claims to system claims 1-5, 8, and 10. Therefore, the rejection of the system claims applies to the method claims.
Claim(s) 6-7 and 16-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sankai in view of Mao as applied to claims 1 and 10 above, and further in view of Yi et al. (JP-2021174535-A).
Regarding claim 6, the rejection of claim 5 is incorporated herein. Sankai in view of Mao teaches the system of claim 5, and Sankai further teaches:
wherein the notification message further comprises an audio notification (“The video/audio output unit 4, which serves as a feedback means, outputs a video for prompting an improvement in the subject person local action, which has been selected by the action improvement proposal unit 15, to a display screen of the monitor, and also outputs audio for prompting an improvement in the local action through the speaker,” Para [0070])
Sankai and Mao are not relied upon to teach the following limitation. Yi, however, further teaches:
wherein the notification message further comprises an audio notification and a light signal notification (“the notification information Noti includes an evaluation of the handwashing actions of the person washing their hands. The evaluation may include at least one of the following: a judgment on whether the handwashing motion performed by the person washing their hands was correct… The display module may display notification information in the form of light, sound, still images, videos, or a combination thereof at the display unit,” Para [0049]).
Yi is considered to be analogous to the claimed invention because they are in the same field of hand monitoring and hand procedure compliance. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Yi into Sankai and Mao for the benefit of the user to more easily know when their compliance to a procedure is noncompliant.
Regarding claim 7, the rejection of claim 6 is incorporated herein. Sankai in view of Mao and Yi teach the system of claim 6, and Sankai further teaches:
wherein if the analysis and comparison result between the hand joint position coordinate of the operator in the operational area during the operation period and the standard hand movement parameter set does not match (“Based on the results of the check performed by the behavior checking unit 14, the action improvement proposal unit 15 gives the subject person feedback about a proposal for improving a local action with recognition accuracy lower than the threshold among the local actions of the hand-washing behavior,” Para [0069]), the notification module generates the audio notification, wherein the audio notification is an error alert warning (“The video/audio output unit 4, which serves as a feedback means, outputs a video for prompting an improvement in the subject person local action… outputs audio for prompting an improvement in the local action through the speaker,” Para [0070]).
Regarding claims 16-17, the claims are corresponding method claims to system claims 6-7. Therefore, the rejection of the method claims applies to the system claims.
Claim(s) 9 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sankai in view of Mao as applied to claims 1 and 10 above, and further in view of Shmueli (US-20210027360-A1).
Regarding claim 9, the rejection of claim 1 is incorporated herein. Sankai in view of Mao teach the system of claim 1, and Shmueli further teaches:
wherein the operational area comprises an experimental operation platform (a “lab”), a sterile operation platform, and a cell handling station (“the disclosed subject matter may be utilized for monitoring other types of work spaces, such as in a Lab 400c,” Para [0176]; and Fig. 4C).
PNG
media_image1.png
282
594
media_image1.png
Greyscale
Shmueli is considered to be analogous to the claimed invention because they are both in the field of hand monitoring and procedure compliance. And, although Shmueli does not explicitly teach “a sterile operation platform, and a cell handling station”, it would be reasonable to find these locations in a laboratory environment, as labs often conduct experiments involving sterile matter and/or cells. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Shmueli into Sankai and Mao for the benefit of increased safety and compliance in laboratory environments.
Regarding claim 19, the claim is a corresponding method claim for system claim 9. Therefore, the rejection of claim 9 applies to claim 19.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Jarvis et al. (US-20220080060-A1) teaches a hand-washing monitoring and compliance system.
Park et al. (US-20250336086-A1) teaches a method for obtaining 3D coordinates of a person’s hands.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RACHEL A OMETZ whose telephone number is (571)272-2535. The examiner can normally be reached 6:45am-4:00pm ET Monday-Thursday, 6:45am-1:00pm ET every other Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at 571-272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Rachel Anne Ometz/ Examiner, Art Unit 2668 7/13/26
Rachel.ometz@uspto.gov
/VU LE/ Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights