DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see Remarks page 7, filed 01/30/2026, with respect to the interpretations of claims 1 and 9-10 under 35 U.S.C. 112(f) have been fully considered and are persuasive. The interpretations of claims 1 and 9-10 have been withdrawn.
Applicant’s arguments, see Remarks page 7, filed 01/30/2026, with respect to the rejections of claims 1-2 and 4-10 under 35 U.S.C. 101 have been fully considered and are persuasive. The rejections of claims 1-2 and 4-10 have been withdrawn.
Applicant’s arguments, see Remarks pages 7-10, filed 01/30/2026, with respect to the rejection of amended claim(s) 1 and 9-10 under 35 U.S.C. 103 have been fully considered and are moot in view of the new grounds of rejection (detailed in the rejections below) necessitated by Applicant’s amendment to the claim(s).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-2, and 4-8 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "the display device”. There is insufficient antecedent basis for this limitation in the claim. For the purposes of examination, the limitation is interpreted as “a display device”.
Regarding claims 2 and 4-8, it/they is/are rejected under 112b for inheriting and failing to cure the deficiencies of the parent claim 8.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4-6, and 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (CN109726668A), hereinafter referenced as Wang, in view of Johnson (US6038331A) and Galkin et al. (A Vision-Based Hygiene Monitoring System: Using Deep Learning to Assess Handwashing Procedure in Real Time) hereinafter referenced as Galkin, and Paliath-Pathiyal et al. (US2021312788A1) hereinafter referenced as Paliath-Pathiyal.
Regarding claim 1, Wang discloses: A handwash monitoring system (Wang: 0002: “to a method for automatically detecting the standardization of hand washing and disinfection processes based on computer vision.”) comprising: an imaging device (Wang: 0012: “Fix the camera in a position that can ensure that the entire pool area is captured.”); a display device (Wang: 0006: “The present invention mainly utilizes computer vision methods to detect each process in the hand washing and disinfection process in sequence and reflects the detection results of each step on the display interface in real time, thereby playing a role in supervising the standardization of the hand washing process.”); and a processor configured to extract a hand region corresponding to a hand of a user from an image of a plurality of image frames obtained on a time-series basis captured by the imaging device (Wang: 0013: “Take one frame every four frames from the video captured by the camera as the image to be processed and automatically detect the process standardization”; 0026-0031: “The 6 ROI areas specifically include…a5: Foam detection area, the selected area includes the active area of both hands during hand washing;”), detect a detergent foam region (Wang: 0071-0072: “(1) extracting area a5 of the image to be processed according to the coordinate data of area a5; (2) The foam part and the non-foam part in the grayscale histogram of the image to be processed are distinguished according to the threshold III preset by the illumination conditions of the scene.”), calculate a proportion of the detergent foam region to the hand region based on the number of pixels in the hand region, compare the calculated proportion to a predetermined threshold, and decide that a hand-washing motion of the user satisfies a specified condition when the calculated proportion is greater than the predetermined threshold based on the proportion of the detergent foam region to the hand region (Wang: 0072-0073: “After the image in area a5 is converted into a binary image, the pixel value of the foam part is 1, and the pixel value of the non-foam part is 0; (3) The following formula is used to calculate the proportion of the foam part in the binary image of area a5:
PNG
media_image1.png
82
140
media_image1.png
Greyscale
”;
0076-0077: “(4) Compare S<sub>foam</sub> with the preset threshold IV and save the comparison result: ① If S<sub>foam</sub>＞threshold IV, it means that the amount of foam in the current image to be processed meets the requirement;”;
0079: “Count whether the current image to be processed and the previous N<sub>2</sub>frame images have generated sufficient foam detection results…if the foam amount in all the statistical detection results reaches the required number and is greater than the threshold V, the corresponding detection result is that sufficient foam has been generated; otherwise, the detection result is that insufficient foam has been generated;”),
wherein the processor is further configured to detect which operation step among a plurality of operation steps determined in advance the user is performing, based on a shape of the hand region (Wang: 0021-0023: “① Detect whether the hands are rubbing against each other during the washing process; ② Check whether the hands are rubbed with their fingers crossed during the washing process; ③ Check whether the backs of the left and right hands are rubbed during the washing process”; 0109-0110: “① If Nfinger ≥ 4, it means that the ten fingers of both hands are in a crossed state during the scrubbing process in the current image to be processed, and rcross＝1; ② If Nfinger<4, it means that the ten fingers of both hands are not in a crossed state during the scrubbing process in the current image to be processed, and rcross = 0;”).
Wang fails to disclose: detect a detergent foam region having a color component designated in advance in the hand region, count a number of pixels having the color component in the hand region of the image, calculate a proportion of the detergent foam region to the hand region based on the number of pixels having the color component in the hand region.
Johnson discloses: detecting a detergent foam region having a color component designated in advance in the hand region (Johnson: Col 1: Lines 10-13: “A soap with a particular feature (e.g., color and opaqueness) that allows it to be differentiated from a user's hands and a background is also disclosed.”; Col 3: Lines 25-27: “Since human hands 14 are primarily red or brown in color, background 24 is preferably blue and soap 16 green or vice versa.”; Wherein the detergent and its foam may be differentiated based upon their color), count a number of pixels having the color component in the hand region of the image, calculate a proportion of the detergent foam region to the hand region based on the number of pixels having the color component in the hand region (Johnson Col 2: Lines 20-23: “…calculates the amount of area to which soap has been applied compared with the area of the hands to which soap has not been applied and compares that with a predetermined level of required coverage.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the algorithm used to detect the number of pixels containing soap through color taught in Johnson into the soap foam detection process disclosed in Wang. The suggestion/motivation for doing so would have been the usage of color in addition to techniques, such as texture, increases the soap detection accuracy (Johnson: Col 3: Lines 14-16). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Wang in view of Johnson does not disclose expressly: wherein the processor is further configured to detect, using image classification deep learning, which operation step among a plurality of operation steps determined in advance the user is performing, based on a shape of the hand region.
Galkin discloses: a posture detector, using image classification deep learning, to detect which operation step among a plurality of operation steps determined in advance the user is performing, based on a shape of the hand region (Galkin: Abstract: “In a healthcare environment, workers must follow proper handwashing protocol to prevent the transmission of pathogens. The World Health Organization (WHO) recommends six handwashing motions to ensure that all parts of the hands are cleaned properly. This project aimed to apply deep learning to identify whether a subject is correctly performing handwashing motions in compliance with the WHO protocol. Video footage from ten subjects was used to train a ResNet-18 neural network capable of identifying the six different hand poses.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the algorithms used for detecting the hand posture during the handwashing process disclosed by Wang in view of Johnson, with the handwashing pose classification neural network disclosed by Galkin. The suggestion/motivation for doing so would have been “It also handled the visual display that allowed a user to see live inferences. The system only displayed the predicted hand pose at the top of the screen for confidence values over a threshold of 70% so that no classification would be shown if no motion was being performed. Each of the poses was listed to the right of the live video feed with their corresponding confidence percentages” (Galkin: III. EXPERIMENTAL PROCEDURE). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results.
Wang in view of Johnson and Galkin does not disclose expressly: and wherein a display device outputs a decision result including: a timing at which the hand-washing motion does not satisfy the specific condition during the operation steps, and an indication of what needs to be improved.
Paliath-Pathiyal discloses: a display device which outputs a decision result including: a timing at which the hand-washing motion does not satisfy the specific condition during the operation steps, and an indication of what needs to be improved (Paliath-Pathiyal: Figure 4b:
PNG
media_image2.png
109
481
media_image2.png
Greyscale
;
0028: “The device, system, and method described herein describe techniques to help encourage compliance with a recommended handwashing protocol by instructing the user on the protocol steps by displaying appropriate instructions in real-time when the user is about to wash his/her hands. Depending on the embodiment of the present disclosure, various advantages may result. ”;
0079: “FIG. 4b lists examples of how the device modifies its output according to one or more embodiments. Depending on the recommended step and the type of deviation, reinforcement may include outputting the importance of the particular step's compliance to ensure good hand hygiene; outputting the deviation in such a manner to “call out” the deviation to the user and to others in the washroom; repeating the display of the recommended step with additional urgency to comply with the step; or some combination thereof. When the device 300 detects that the user appears to not comply with a recommended hand wash, the device 300 highlights non-compliance to the user by the use of an audio warning, a visual warning, or some combination thereof. The audio warning could be a voice recording, “You are not quite done your handwash!” or some suitable equivalent that may reference the type of non-compliance. ”; Wherein when a handwashing step is not being complied, a display outputs the non-compliance and how it may be corrected.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the known technique of displaying real-time messages based on a handwashing step non-compliance detection taught by Paliath-Pathiyal for the displaying of compliance of each handwashing step disclosed by Wang in view of Johnson and Galkin. The suggestion/motivation for doing so would have been “There are, in fact, several actions a user may take that deviate from the recommended handwashing protocol. Static signage describing the protocol is sometimes not sufficiently understood, and users are not held accountable in real-time to comply when they deviate from the recommended handwashing protocol. One or more goals of the present disclosure are to increase hand hygiene compliance, namely: (1) to encourage a user to wash his/her hands in a particular environment where the device is situated, and (2) to guide users to wash their hands in the recommended protocol.” (Paliath-Pathiyal: 0026). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wang in view of Johnson and Galkin with Paliath-Pathiyal to obtain the invention as specified in claim 1.
Regarding claim 2, Wang in view of Johnson, Galkin, and Paliath-Pathiyal discloses: The handwash monitoring system according to claim 1, wherein the processor detects the detergent foam region having the color component corresponding to a detergent designated in advance (Johnson: Col 1: Lines 10-13: “A soap with a particular feature (e.g., color and opaqueness) that allows it to be differentiated from a user's hands and a background is also disclosed.”; Col 3: Lines 25-27: “Since human hands 14 are primarily red or brown in color, background 24 is preferably blue and soap 16 green or vice versa.”)
Regarding claim 4, Wang in view of Johnson, Galkin, and Paliath-Pathiyal discloses: The handwash monitoring system according to claim 1, wherein the processor decides that the hand-washing motion of the user satisfies the specified condition by using the proportion threshold corresponding to the detected operation step (Wang: 0067: “Furthermore, when the detection result of whether the hand sanitizer is collected corresponding to the current image to be processed is Rsoap = 3, whether sufficient foam is generated during the hand washing process is detected, which specifically includes the following steps:”; 0079: “Count whether the current image to be processed and the previous N2 frame images have generated enough foam detection results, where N2 is a preset value; if the foam amount in all the statistical detection results reaches the required number and is greater than the threshold V, the corresponding detection result is that sufficient foam has been generated.”; Wherein the operating step is detected by the handwashing pose classification neural network disclosed by Galkin).
Regarding claim 5, Wang in view of Johnson, Galkin, and Paliath-Pathiyal discloses: The handwash monitoring system according to claim 1, wherein the processor counts a number of repeated motions of the user based on a motion of the hand region and the processor decides that the hand-washing motion of the user satisfies the specified condition when the proportion is greater than the specified proportion threshold and the number of repeated motions of the user is greater than a specified number threshold (Wang: 0262-0263: “Count whether the number of rmove=1 corresponding to the current image to be processed and the previous N4 frame images is greater than a preset threshold VIII… if it is greater than the threshold VIII, the corresponding detection result is that the action of interlacing and rubbing the fingers together has been completed…if the user requests a mutual scrubbing time of 5 seconds or longer and the fps is 30 frames per second, then threshold VIII = 5/5*30 = 30,”; Wherein the frames compared to threshold correspond to the amount of time spent crossing and rubbing fingers).
Regarding claim 6, Wang in view of Johnson, Galkin, and Paliath-Pathiyal discloses: The handwash monitoring system according to claim 5, wherein the processor counts the number of repeated motions when a cycle of a repeating motion of the user is identical or nearly identical to a designated cycle (Wang: 0097-0099: “(5) Count the total distance of the relative movement of the two hands in the five consecutive frames of the image to be processed…If the total distance is greater than the threshold VI, it means that the two hands rubbed each other during the scrubbing process, and rmove=1…(6) Count the number of rmove=1 corresponding to the current image to be processed and the previous N3 frames… if the number exceeds the preset threshold VII, the corresponding detection result is that the mutual scrubbing action has been completed; otherwise, the detection result is that the mutual scrubbing action has not been completed;”; Wherein the distance between the relative movement is detected in order to detect a cycle of scrubbing hands).
As per claim(s) 9, arguments made in rejecting claim(s) 1 are analogous.
As per claim(s) 10, arguments made in rejecting claim(s) 1 are analogous. Additionally, Paragraph 0139 of Wang discloses a computer vision based method executed by the computer for handwash monitoring which implies the use of a computer-readable non-transitory recording medium storing the handwash monitoring program.
Claim(s) 7 and 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Johnson, Galkin, and Paliath-Pathiyal, and further in view of Simonovsky (WO2020044351A1).
Regarding claim 7, Wang in view of Johnson, Galkin, and Paliath-Pathiyal discloses: The handwash monitoring system according to claim 1.
Wang in view of Johnson, Galkin, and Paliath-Pathiyal does not disclose expressly: wherein the threshold is determined based on a type of detergent used by the user.
Simonovsky discloses: the dispensing of a reagent according to the type of reagent and the hand washing sequence (Simonovsky: Page 7: Lines 11-25: “In order to prepare a predefined pattern set of parameters, machine learning tools are used to train the system’s algorithms with various images/videos of hands movements (both conforming to requirements and not conforming to requirements). The predefined pattern set of parameters can include…• Reagents - types (Plain Soap, Antimicrobial Soap, Biodegradable soap, Scrub Soap, Foam, Alcohol-based disinfection agents, Bio-Based disinfection agents, Iodine or another disinfection agent) and volume.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the algorithms for dispensing reagents throughout the handwashing process taught by Simonovsky into the handwash monitoring system disclosed by Wang in view of Johnson, Galkin, and Paliath-Pathiyal. The suggestion/motivation for doing so would have been due to the fact that different user kinds of users require different reagents with varying amounts of the reagent (Simonovsky: Page 11: Lines 7-10). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wang in view of Johnson, Galkin, and Paliath-Pathiyal with Simonovsky to obtain the invention as specified in claim 7.
Regarding claim 8, Wang in view of Johnson, Galkin, Paliath-Pathiyal, and Simonovsky discloses: The handwash monitoring system according to claim 7, wherein the processor estimates the type of detergent used by the user based on the image captured by the imaging device (Johnson: Col 4: Lines 61-64: “The reference feature of soap 16 is determined by processing an image which is comprised of background and mostly soap. To image properly, the soap must have a certain amount of opaqueness and a color quality that allows it to be adequately distinguished on a hand by input device 12.”; Col 4: Lines 20-25: “Starting from the most common color space or HSY value based upon occurrence from step C, clustering the color space or HSY values by allowing small percentage differences in color space or HSY. Building up a list of ranges/clusters which include all soap colors.”; Wherein different reference features may be used to distinguish different type of detergent).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672