DETAILED ACTION
Response to Arguments
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. The new reference of Castelli et al. (US Pub. No. 2017/0286766 A1) has been introduced to teach the new limitations of the claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7, 9, 11-17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Au et al. (US Pub. No. 2013/0282609 A1) in view of Vukicevic et al. (“Generic compliance of industrial PPE by using deep learning techniques”) and in further view of Castelli et al. (US Pub. No. 2017/0286766 A1).
Regarding claim 1, Au discloses, a device comprising: a camera; (See Au ¶23, “In an embodiment, the image capture device 110 comprises an electronic device (e.g., a camera or other image capturing device) used to capture an image of an individual.”)
an output system including a display; (See Au ¶89, “I/O devices 690 may include printers, video monitors, liquid crystal displays (LCDs), touch screen displays.”)
and a computer-based analysis subsystem configured to perform operations including: (See Au ¶26, “The image recognition process system 140 may comprise a set of instructions that may be implemented on a computer comprising a processing and a non-transitory computer readable medium acting as a memory.”)
receiving images captured by the camera, (See Au ¶28, “The image extraction component 141 may be configured to receive the image data.”)
the images including a first image depicting a subject performing an ongoing task; (See Au ¶24, “The trigger device 120 is configured to initiate the image recognition process through the generation of a trigger signal. In an embodiment, the trigger device may take the form of an entry way device to a work area such as a badge reader (e.g., a wireless badge reader), keypad, door handle, etc. … This embodiment may be useful in verifying proper use of the PPE for the particular device at the time the device is activated.”)
analyzing the first image
determining, based on the analyzing, that the subject’s performance of the ongoing task is not in accordance with a compliance requirement of the ongoing task; (See Au ¶48, “At step 316, a decision is made to determine if the identified PPE is being properly worn. In an embodiment, the PPE positioning component 145 may retrieve information from the PPE information store 151 to determine the proper positioning of the PPE relative to the identified body part in order to identify what positioning of the PPE is the proper positioning on the person. If the PPE is properly positioned, the image recognition process may continue to step 318, otherwise the image recognition process may proceed to step 324 in which entry into the work area and/or authorization to access the access device is denied.”)
and causing the display to display an indication of why the subject’s performance of the ongoing task is not in accordance with the compliance requirement. (See Au ¶71, “The data displayed on the output device 470 may comprise an indication that the person has complied with the PPE standards, has not complied with the PPE standards, instructions on complying with the PPE standards (e.g., proper positioning information, proper PPE kind/type information, etc.), incident report information, and/or the like.”)
Au discloses the above limitations but he fails to disclose analyzing the first image by applying a machine-learned model to segment the image, isolate segments depicting the subject, and extract features from the segments.
However, Vukicevic discloses, analyzing the first image by applying a machine-learned model to segment the image (See Vukicevic p. 3 left col, Section 2.1, “The first step is detection/ identification of an employee in the workspace (Fig. 2a). For these purposes, we used 2D pose estimation algorithms as they enabled us to simultaneously detect body landmark points (Fig. 2b). By using the body landmark points, we defined the region of interest (ROI) that should be the subject of PPE compliance (Fig. 2c). Particularly, we divide PPE into five groups (following Fig. 2d): 1) head-mounted PPE (e.g. hardhats, glasses, earmuffs); 2) upper body PPE (e.g. wets); 3) Hands (e.g. Gloves); 4) Legs (e.g. boots, safety shoes); and 5) whole-body (i.e. work suit).” Further see Vukicevic p. 3 left col, Section 2.2, “The pose estimation was done using the HigherHRNet.” Whereby the ROIs are considered to be the segments.)
isolate segments depicting the subject, (See Vukicevic p. 4 left col lines 8-9, “The ROI cropping was performed following Fig. 2.”)
and extract features from the segments; (See Vukicevic p. 3 left col. Section 2.3, “The PPE compliance was considered as a classification problem, where the classification inputs are previously cropped ROIs. In this study we considered the following deep learning classifiers.” Whereby the classifiers will extract feature maps when doing convolution.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the neural networks that provide pose estimation and segmentation of body parts and PPE as suggested by Vukicevic to Au’s segmentation of body parts and PPE. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so because the machine learning pose estimation and segmentation methods handle challenging, real-world conditions better, including variations in lighting, clothing, camera angles, occlusions such as hidden body parts, and crowded scenes with multiple people.
Au and Vukicevic disclose the above limitations, but they fail to disclose for each step of an ongoing task for wearing protective gear: receiving images captured by the camera, including an image depicting a subject performing the step of the ongoing task; analyzing the image; determining, based on the analyzing, whether the subject's performance of the step of the ongoing task is not in accordance with a compliance requirement of the ongoing task; and in response to determining that the subject's performance of the step is not in accordance with the compliance requirement: causing the output system to generate a command directing the subject through the step of the ongoing task; and causing the display to display an indication of why the subject's performance of the step of the ongoing task is not in accordance with the compliance requirement.
However, Castelli discloses, for each step of an ongoing task (See Castelli ¶59, Moreover, in an embodiment, step 560 involves determining when a user has completed a given step. Further, in an embodiment, step 560 involves determining whether the user is having difficulty with any of the steps.”)
for wearing protective gear: (Based on the combination with Au and Vukicevic with Castelli, the method of Castelli can be applied to the step of gowning or wearing protective gear.)
receiving images captured by the camera including an image depicting a subject performing the step of the ongoing task; (See Castelli ¶58, “At step 550, monitor the user and capture images of the user performing any actions relating to the documentation or representation thereof.”)
analyzing the image (See Castelli ¶40 The gesture recognition system 230 recognizes gestures. The gesture recognition system 230 can, for example, recognize actions taken by a user (grabbing a wrench, turning clockwise instead of counter-clockwise, and so forth).
determining, based on the analyzing, whether the subject's performance of the step of the ongoing task is not in accordance with a compliance requirement of the ongoing task; (See Castelli ¶59, “At step 560, evaluate the user performed actions against expected actions specified in the documentation. In an embodiment, step 560 involves determining whether the user is correctly following the documentation. Moreover, in an embodiment, step 560 involves determining when a user has completed a given step. Further, in an embodiment, step 560 involves determining whether the user is having difficulty with any of the steps.”)
and in response to determining that the subject's performance of the step is not in accordance with the compliance requirement: causing the output system to generate a command directing the subject through the step of the ongoing task; and causing the display to display an indication of why the subject's performance of the step of the ongoing task is not in accordance with the compliance requirement. (See Castelli ¶61, “At step 570, provide (visual and/or audible) feedback to the user regarding the user performed actions. For example, either confirm or correct the user performed actions. Any visual and/or auditory feedback (e.g., an indication) can be provided (e.g., display a green light, have the system state “continue”, and/or so forth). If the user is doing something incorrect, then the corrective action (e.g., audibly state “do not turn the screw counter-clockwise, instead turn it clockwise”, “Careful—rotate the screw clockwise—the other way” and/or so forth) can be specified.” Further see Castelli ¶61, “In an embodiment, step 570 can include providing on-line and/or otherwise publicly available information on the display and/or through the speaker in order to assist the user.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the analyzing images for each step of a procedure and providing feedback for each step as suggested by Castelli to Au and Vukicevic’s determining compliance for wearing protective gear. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so as disclosed by Castelli in ¶25 is “One advantage of the present principles is that it is not limited to computer-based procedures, and does not require specially constructed documentation. Another advantage of the present principles is that they do not rely on an existing model.
Regarding claim 2, Au, Vukicevic, and Castelli disclose, the device of claim 1, wherein analyzing the image comprises: identifying and locating body parts and the protective gear worn by the subject; (See Vukicevic Fig. 2(d) which shows the body part ROI segments that are identified, i.e. head, upper body hands, and feet. Further see 2(f) where the PPE classes of each segment or ROI is classified.)
grouping the identified body parts and the protective gear of the subject into segments that correspond to different body parts and the protective gear, (See Vukicevic Fig. 2(d), where the body parts are grouped together, and in Fig. 2(f) where the classified PPE are grouped.)
and estimating positions, orientations, and relationships between different segments to create a representation of the subject's body and the protective gear in the image. (See Vukicevic Fig. 2(b) and 2(c) for the pose estimation which shows the positions and orientation of the ROI body parts relative to each other. Further see Fig. 2(d) and 2(f) which shows a representation of the body parts and PPE relative to each other.)
Regarding claim 3, Au, Vukicevic, and Castelli disclose, the device of claim 2, wherein determining whether the subject’s performance of the step of the ongoing task is not in accordance with a compliance requirement comprises determining, based on data generated by the analyzing of the first image, whether the protective gear covers the body of the subject; (See Au ¶48, “When the PPE is detected at step 312, the method may proceed to step 314 where the PPE positioning in the image is detected. In an embodiment, the relative positioning of the identified body portion of the person may be analyzed with respect to any PPE identified at step 310. For example, any identified goggles may be compared to the person's head to determine if the goggles are covering the person's eyes. Such a placement of the goggles relative to the person's head would indicate that the goggles were properly positioned. Should the goggles be identified above the person's eyes, for example pushed up on the person's head, then the identification of the positioning would indicate that while the goggles were present, they were not properly positioned.”)
and generating an indication of the subject’s performance of the step of the ongoing task. (See Au ¶48, “At step 316, a decision is made to determine if the identified PPE is being properly worn. In an embodiment, the PPE positioning component 145 may retrieve information from the PPE information store 151 to determine the proper positioning of the PPE relative to the identified body part in order to identify what positioning of the PPE is the proper positioning on the person. If the PPE is properly positioned, the image recognition process may continue to step 318, otherwise the image recognition process may proceed to step 324 in which entry into the work area and/or authorization to access the access device is denied. Alternatively, or in addition to denying access, a message may be transmitted to the output device 170 to instruct the person of the proper PPE positioning.”)
Regarding claim 4, Au, Vukicevic, and Castelli disclose, the device of claim 3, wherein determining whether the protective gear covers the body of the subject comprises verifying that the protective gear covers the face, torso, arms, and legs of the subject. (See Au ¶27,” Any of the algorithms may be used to process image data in a manner that identifies the visual features of an item of PPE and/or a person (e.g., eyes, head, arms, hands, and/or other body parts).”
Further see Au ¶31. “The PPE positioning component 145 may be configured to analyze image data and identify the positioning of any PPE present relative to the body of the person.”
Further see Au ¶22, “In an embodiment, the image recognition system 100 may be used to verify compliance with the PPE standards for one or more kinds of PPE including, but not limited to, glasses, goggles, ear plugs, ear muffs, face masks, respirators, hair nets, hard hats, wrist bands, gloves, skirts, gowns, aprons, shoes, boots, safety harnesses, safety suits, chemical suits, and any combinations thereof.”)
Regarding claim 5, Au, Vukicevic, and Castelli disclose, the device of claim 3, wherein determining whether the protective gear covers the body of the subject comprises any one of: verifying that the subject is wearing a hairnet; verifying that the subject’s beard is covered; verifying that the subject’s shirt is tucked in; verifying that the subject is wearing gloves; and verifying that the subject is wearing boots. (See the rejection of claim 4 as it is equally applicable for claim 5 as well.)
Regarding claim 6, Au, Vukicevic, and Castelli disclose, the device of claim 3, wherein determining whether the protective gear covers the body of the subject comprises: determining an optimal fit of the protective gear on the subject; and comparing the optimal fit to an actual fit to determine if the protective gear is properly worn by the subject. (See Au ¶48, “At step 316, a decision is made to determine if the identified PPE is being properly worn. In an embodiment, the PPE positioning component 145 may retrieve information from the PPE information store 151 to determine the proper positioning of the PPE relative to the identified body part in order to identify what positioning of the PPE is the proper positioning on the person. If the PPE is properly positioned, the image recognition process may continue to step 318, otherwise the image recognition process may proceed to step 324 in which entry into the work area and/or authorization to access the access device is denied. Alternatively, or in addition to denying access, a message may be transmitted to the output device 170 to instruct the person of the proper PPE positioning.”)
Regarding claim 7, Au, Vukicevic, and Castelli disclose, the device of claim 1, wherein the output system further includes a speaker and the command includes a voice command. (See Au ¶71, “The output device may be configured to receive data from the image recognition processing system 440 and output the information to one or more people. The output device may comprise any number of device such as an audio device for issuing a warning signal, a lighting device for issuing a visual indicator, an audio/visual device for presenting information to the person.” Further see Au ¶48, “Alternatively or in addition to denying access, a message may be transmitted to the output device 170 to instruct the person of the proper PPE positioning.”)
Regarding claim 9, Au, Vukicevic, and Castelli disclose, the device of claim 1, wherein the indication that is displayed on the display indicates an explanation of how to fix a problem detected by the computer-based analysis subsystem or an instruction to start the ongoing task again if the problem is not fixable. (See Au ¶71, “The data displayed on the output device 470 may comprise an indication that the person has complied with the PPE standards, has not complied with the PPE standards, instructions on complying with the PPE standards (e.g., proper positioning information, proper PPE kind/type information, etc.), incident report information, and/or the like.”)
Regarding claim 11, Au discloses, a method comprising: receiving images (See Au ¶28, “The image extraction component 141 may be configured to receive the image data.”)
including a first image depicting a subject performing an ongoing task; (See Au ¶24, “The trigger device 120 is configured to initiate the image recognition process through the generation of a trigger signal. In an embodiment, the trigger device may take the form of an entry way device to a work area such as a badge reader (e.g., a wireless badge reader), keypad, door handle, etc. … This embodiment may be useful in verifying proper use of the PPE for the particular device at the time the device is activated.”)
analyzing the first image
determining, based on the analyzing, that the subject’s performance of the ongoing task is not in accordance with a compliance requirement of the ongoing task; (See Au ¶48, “At step 316, a decision is made to determine if the identified PPE is being properly worn. In an embodiment, the PPE positioning component 145 may retrieve information from the PPE information store 151 to determine the proper positioning of the PPE relative to the identified body part in order to identify what positioning of the PPE is the proper positioning on the person. If the PPE is properly positioned, the image recognition process may continue to step 318, otherwise the image recognition process may proceed to step 324 in which entry into the work area and/or authorization to access the access device is denied.”)
and causing a display to display an indication of why the subject’s performance of the ongoing task is not in accordance with the compliance requirement. (See Au ¶71, “The data displayed on the output device 470 may comprise an indication that the person has complied with the PPE standards, has not complied with the PPE standards, instructions on complying with the PPE standards (e.g., proper positioning information, proper PPE kind/type information, etc.), incident report information, and/or the like.”)
Au discloses the above limitations but he fails to disclose analyzing the first image by applying a machine-learned model to segment the image, isolate segments depicting the subject, and extract features from the segments.
However, Vukicevic discloses, analyzing the first image by applying a machine-learned model to segment the image (See Vukicevic p. 3 left col, Section 2.1, “The first step is detection/ identification of an employee in the workspace (Fig. 2a). For these purposes, we used 2D pose estimation algorithms as they enabled us to simultaneously detect body landmark points (Fig. 2b). By using the body landmark points, we defined the region of interest (ROI) that should be the subject of PPE compliance (Fig. 2c). Particularly, we divide PPE into five groups (following Fig. 2d): 1) head-mounted PPE (e.g. hardhats, glasses, earmuffs); 2) upper body PPE (e.g. wets); 3) Hands (e.g. Gloves); 4) Legs (e.g. boots, safety shoes); and 5) whole-body (i.e. work suit).” Further see Vukicevic p. 3 left col, Section 2.2, “The pose estimation was done using the HigherHRNet.” Whereby the ROIs are considered to be the segments.)
isolate segments depicting the subject, (See Vukicevic p. 4 left col lines 8-9, “The ROI cropping was performed following Fig. 2.”)
and extract features from the segments; (See Vukicevic p. 3 left col. Section 2.3, “The PPE compliance was considered as a classification problem, where the classification inputs are previously cropped ROIs. In this study we considered the following deep learning classifiers.” Whereby the classifiers will extract feature maps when doing convolution.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the neural networks that provide pose estimation and segmentation of body parts and PPE as suggested by Vukicevic to Au’s segmentation of body parts and PPE. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so because the machine learning pose estimation and segmentation methods handle challenging, real-world conditions better, including variations in lighting, clothing, camera angles, occlusions such as hidden body parts, and crowded scenes with multiple people.
Au and Vukicevic disclose the above limitations, but they fail to disclose for each step of an ongoing task for wearing protective gear: receiving images including an image depicting a subject performing the step of the ongoing task; analyzing the image; determining, based on the analyzing, whether the subject's performance of the step of the ongoing task is not in accordance with a compliance requirement of the ongoing task; and in response to determining that the subject's performance of the step is not in accordance with the compliance requirement: causing the output system to generate a command directing the subject through the step of the ongoing task; and causing a display to display an indication of why the subject's performance of the step of the ongoing task is not in accordance with the compliance requirement.
However, Castelli discloses, for each step of an ongoing task (See Castelli ¶59, Moreover, in an embodiment, step 560 involves determining when a user has completed a given step. Further, in an embodiment, step 560 involves determining whether the user is having difficulty with any of the steps.”)
for wearing protective gear: (Based on the combination with Au and Vukicevic with Castelli, the method of Castelli can be applied to the step of gowning or wearing protective gear.)
receiving images including an image depicting a subject performing the step of the ongoing task; (See Castelli ¶58, “At step 550, monitor the user and capture images of the user performing any actions relating to the documentation or representation thereof.”)
analyzing the image (See Castelli ¶40 The gesture recognition system 230 recognizes gestures. The gesture recognition system 230 can, for example, recognize actions taken by a user (grabbing a wrench, turning clockwise instead of counter-clockwise, and so forth).
determining, based on the analyzing, whether the subject's performance of the step of the ongoing task is not in accordance with a compliance requirement of the ongoing task; (See Castelli ¶59, “At step 560, evaluate the user performed actions against expected actions specified in the documentation. In an embodiment, step 560 involves determining whether the user is correctly following the documentation. Moreover, in an embodiment, step 560 involves determining when a user has completed a given step. Further, in an embodiment, step 560 involves determining whether the user is having difficulty with any of the steps.”)
and in response to determining that the subject's performance of the step is not in accordance with the compliance requirement: causing the output system to generate a command directing the subject through the step of the ongoing task; and causing a display to display an indication of why the subject's performance of the step of the ongoing task is not in accordance with the compliance requirement. (See Castelli ¶61, “At step 570, provide (visual and/or audible) feedback to the user regarding the user performed actions. For example, either confirm or correct the user performed actions. Any visual and/or auditory feedback (e.g., an indication) can be provided (e.g., display a green light, have the system state “continue”, and/or so forth). If the user is doing something incorrect, then the corrective action (e.g., audibly state “do not turn the screw counter-clockwise, instead turn it clockwise”, “Careful—rotate the screw clockwise—the other way” and/or so forth) can be specified.” Further see Castelli ¶61, “In an embodiment, step 570 can include providing on-line and/or otherwise publicly available information on the display and/or through the speaker in order to assist the user.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the analyzing images for each step of a procedure and providing feedback for each step as suggested by Castelli to Au and Vukicevic’s determining compliance for wearing protective gear. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so as disclosed by Castelli in ¶25 is “One advantage of the present principles is that it is not limited to computer-based procedures, and does not require specially constructed documentation. Another advantage of the present principles is that they do not rely on an existing model.
Regarding claim 12, Au, Vukicevic, and Castelli disclose, the method of claim 11, wherein analyzing the image comprises: identifying and locating body parts and the protective gear worn by the subject; grouping the identified body parts and protective gear of the subject into segments that correspond to different body parts and protective gear, and estimating positions, orientations, and relationships between different segments to create a representation of the subject's body and the protective gear in the image. (See the rejection of claim 2 as it is equally applicable for claim 12 as well.)
Regarding claim 13, Au, Vukicevic, and Castelli disclose, the method of claim 12, wherein determining the subject’s performance of the step of the ongoing task is not in accordance with a compliance requirement comprises determining, based on data generated by the analyzing of the image, whether the protective gear covers the body of the subject; and generating an indication of the subject’s performance of the step of the ongoing task. (See the rejection of claim 3 as it is equally applicable for claim 13 as well.)
Regarding claim 14, Au, Vukicevic, and Castelli disclose, the method of claim 13, wherein determining whether the protective gear covers the body of the subject comprises verifying that the protective gear covers the face, torso, arms, and legs of the subject. (See the rejection of claim 4 as it is equally applicable for claim 14 as well.)
Regarding claim 15, Au, Vukicevic, and Castelli disclose, the method of claim 13, wherein determining whether the protective gear covers the body of the subject comprises any one of: verifying that the subject is wearing a hairnet; verifying that the subject’s beard is covered; verifying that the subject’s shirt is tucked in; verifying that the subject is wearing gloves; and verifying that the subject is wearing boots. (See the rejection of claim 5 as it is equally applicable for claim 15 as well.)
Regarding claim 16, Au, Vukicevic, and Castelli disclose, the method of claim 13, wherein determining whether the protective gear covers the body of the subject comprises: determining an optimal fit of the protective gear on the subject; and comparing the optimal fit to an actual fit to determine if the protective gear is properly worn by the subject. (See the rejection of claim 6 as it is equally applicable for claim 16 as well.)
Regarding claim 17, Au, Vukicevic, and Castelli disclose, the method of claim 11, wherein the output device includes a speaker and the command includes a voice command. (See the rejection of claim 7 as it is equally applicable for claim 17 as well.)
Regarding claim 20, Au discloses, a computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a computing system, cause the computing system to perform operations including: (See Au ¶94, “In an embodiment, some or all of the functionality disclosed above may be provided as a computer program product. The computer program product may comprise one or more computer readable storage medium having computer usable program code embodied therein to implement the functionality disclosed above.”)
receiving images (See Au ¶28, “The image extraction component 141 may be configured to receive the image data.”)
including a first image depicting a subject performing an ongoing task; (See Au ¶24, “The trigger device 120 is configured to initiate the image recognition process through the generation of a trigger signal. In an embodiment, the trigger device may take the form of an entry way device to a work area such as a badge reader (e.g., a wireless badge reader), keypad, door handle, etc. … This embodiment may be useful in verifying proper use of the PPE for the particular device at the time the device is activated.”)
analyzing the first image
determining, based on the analyzing, that the subject’s performance of the ongoing task is not in accordance with a compliance requirement of the ongoing task; (See Au ¶48, “At step 316, a decision is made to determine if the identified PPE is being properly worn. In an embodiment, the PPE positioning component 145 may retrieve information from the PPE information store 151 to determine the proper positioning of the PPE relative to the identified body part in order to identify what positioning of the PPE is the proper positioning on the person. If the PPE is properly positioned, the image recognition process may continue to step 318, otherwise the image recognition process may proceed to step 324 in which entry into the work area and/or authorization to access the access device is denied.”)
and causing a display to display an indication of why the subject’s performance of the ongoing task is not in accordance with the compliance requirement. (See Au ¶71, “The data displayed on the output device 470 may comprise an indication that the person has complied with the PPE standards, has not complied with the PPE standards, instructions on complying with the PPE standards (e.g., proper positioning information, proper PPE kind/type information, etc.), incident report information, and/or the like.”)
Au discloses the above limitations but he fails to disclose analyzing the first image by applying a machine-learned model to segment the image, isolate segments depicting the subject, and extract features from the segments.
However, Vukicevic discloses, analyzing the first image by applying a machine-learned model to segment the image (See Vukicevic p. 3 left col, Section 2.1, “The first step is detection/ identification of an employee in the workspace (Fig. 2a). For these purposes, we used 2D pose estimation algorithms as they enabled us to simultaneously detect body landmark points (Fig. 2b). By using the body landmark points, we defined the region of interest (ROI) that should be the subject of PPE compliance (Fig. 2c). Particularly, we divide PPE into five groups (following Fig. 2d): 1) head-mounted PPE (e.g. hardhats, glasses, earmuffs); 2) upper body PPE (e.g. wets); 3) Hands (e.g. Gloves); 4) Legs (e.g. boots, safety shoes); and 5) whole-body (i.e. work suit).” Further see Vukicevic p. 3 left col, Section 2.2, “The pose estimation was done using the HigherHRNet.” Whereby the ROIs are considered to be the segments.)
isolate segments depicting the subject, (See Vukicevic p. 4 left col lines 8-9, “The ROI cropping was performed following Fig. 2.”)
and extract features from the segments; (See Vukicevic p. 3 left col. Section 2.3, “The PPE compliance was considered as a classification problem, where the classification inputs are previously cropped ROIs. In this study we considered the following deep learning classifiers.” Whereby the classifiers will extract feature maps when doing convolution.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the neural networks that provide pose estimation and segmentation of body parts and PPE as suggested by Vukicevic to Au’s segmentation of body parts and PPE. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so because the machine learning pose estimation and segmentation methods handle challenging, real-world conditions better, including variations in lighting, clothing, camera angles, occlusions such as hidden body parts, and crowded scenes with multiple people.
Au and Vukicevic disclose the above limitations, but they fail to disclose for each step of an ongoing task for wearing protective gear: receiving images including an image depicting a subject performing the step of the ongoing task; analyzing the image; determining, based on the analyzing, whether the subject's performance of the step of the ongoing task is not in accordance with a compliance requirement of the ongoing task; and in response to determining that the subject's performance of the step is not in accordance with the compliance requirement: causing the output system to generate a command directing the subject through the step of the ongoing task; and causing a display to display an indication of why the subject's performance of the step of the ongoing task is not in accordance with the compliance requirement.
However, Castelli discloses, for each step of an ongoing task (See Castelli ¶59, Moreover, in an embodiment, step 560 involves determining when a user has completed a given step. Further, in an embodiment, step 560 involves determining whether the user is having difficulty with any of the steps.”)
for wearing protective gear: (Based on the combination with Au and Vukicevic with Castelli, the method of Castelli can be applied to the step of gowning or wearing protective gear.)
receiving images including an image depicting a subject performing the step of the ongoing task; (See Castelli ¶58, “At step 550, monitor the user and capture images of the user performing any actions relating to the documentation or representation thereof.”)
analyzing the image (See Castelli ¶40 The gesture recognition system 230 recognizes gestures. The gesture recognition system 230 can, for example, recognize actions taken by a user (grabbing a wrench, turning clockwise instead of counter-clockwise, and so forth).
determining, based on the analyzing, whether the subject's performance of the step of the ongoing task is not in accordance with a compliance requirement of the ongoing task; (See Castelli ¶59, “At step 560, evaluate the user performed actions against expected actions specified in the documentation. In an embodiment, step 560 involves determining whether the user is correctly following the documentation. Moreover, in an embodiment, step 560 involves determining when a user has completed a given step. Further, in an embodiment, step 560 involves determining whether the user is having difficulty with any of the steps.”)
and in response to determining that the subject's performance of the step is not in accordance with the compliance requirement: causing the output system to generate a command directing the subject through the step of the ongoing task; and causing a display to display an indication of why the subject's performance of the step of the ongoing task is not in accordance with the compliance requirement. (See Castelli ¶61, “At step 570, provide (visual and/or audible) feedback to the user regarding the user performed actions. For example, either confirm or correct the user performed actions. Any visual and/or auditory feedback (e.g., an indication) can be provided (e.g., display a green light, have the system state “continue”, and/or so forth). If the user is doing something incorrect, then the corrective action (e.g., audibly state “do not turn the screw counter-clockwise, instead turn it clockwise”, “Careful—rotate the screw clockwise—the other way” and/or so forth) can be specified.” Further see Castelli ¶61, “In an embodiment, step 570 can include providing on-line and/or otherwise publicly available information on the display and/or through the speaker in order to assist the user.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the analyzing images for each step of a procedure and providing feedback for each step as suggested by Castelli to Au and Vukicevic’s determining compliance for wearing protective gear. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so as disclosed by Castelli in ¶25 is “One advantage of the present principles is that it is not limited to computer-based procedures, and does not require specially constructed documentation. Another advantage of the present principles is that they do not rely on an existing model.
Claims 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Au et al. (US Pub. No. 2013/0282609 A1) in view of Vukicevic et al. (“Generic compliance of industrial PPE by using deep learning techniques”) in view of Castelli et al. (US Pub. No. 2017/0286766 A1) and in further view of Bohannon et al. (US Pub. No. 2020/0410444 A1).
Regarding claim 8, Au, Vukicevic, and Castelli disclose, the device of claim 7, wherein the voice command comprises instructions but they fail to disclose, requiring the subject to perform a visual self-inspection to verify that the protective gear is worn properly.
However, Bohannon discloses, requiring the subject to perform a visual self-inspection to verify that the protective gear is worn properly. (See Bohannon ¶134, “In some examples, PPEIS 6 may prompt worker 10A to compare the one or more articles of PPE on avatar 52 with the articles of PPE he or she is wearing (192). In some such examples, PPEIS 6 may optionally alert worker 10A to input confirmation of the one or more articles of PPE worn by avatar 52 (e.g., to indicate that he or she is wearing the required PPE).”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the requires a PPE inspection confirmation by a worker as suggested by Bohannon to Au, Vukicevic, and Castelli’s PPE fitting visual inspection. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so is to improve worker safety, ensuring compliance, and fostering a strong safety culture.
Regarding claim 18, Au, Vukicevic, Castelli, and Bohannon disclose, the method of claim 17, wherein the voice command comprises instructions requiring the subject to perform a visual self-inspection to verify that the protective gear is worn properly. (See the rejection of claim 8 as it is equally applicable for claim 18 as well.)
Claims 10 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Au et al. (US Pub. No. 2013/0282609 A1) in view of Vukicevic et al. (“Generic compliance of industrial PPE by using deep learning techniques”) in view of Castelli et al. (US Pub. No. 2017/0286766 A1).and in further view of Howell et al. (US Pub. No. 2025/0218436 A1).
Regarding claim 10, Au, Vukicevic, and Castelli disclose, the device of claim 1, but they fail to disclose, wherein the device further comprises a microphone and the operations further comprise: receiving a recording, generated by the microphone, of an utterance made by the subject; analyzing the utterance to determine it includes a voice command; and implementing the voice command.
However, Howell discloses, wherein the device further comprises a microphone and the operations further comprise: receiving a recording, generated by the microphone, of an utterance made by the subject; (See Howell ¶28, “For example, if a user 14 utters a voice command (e.g., “Take a snapshot picture”), microphone 20 may detect the audio, and may send the audio (which may be processed/filtered by microphone 20 and/or other circuitry, such as processing circuitry 34) to STT engine 42. STT engine 42 may determine that the user has uttered the string “Take a snapshot picture”, and provides that string to intent determiner 44.”)
analyzing the utterance to determine it includes a voice command; (See Howell ¶8, “Intent determiner 44 parses the string to determine that user 14 desires to execute the “Take snapshot picture” command, which may be one of several available commands of head worn device 12.”)
and implementing the voice command. (See Howell ¶28,” and causes image sensor 31 to capture one or more images (e.g., of the environment, of user 14, etc.).”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the voice command and its implementation as suggested by Howell to Au, Vukicevic, and Castelli’s PPE compliance determination. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so is in order to provide hands-free efficiency, allowing workers to report issues or get guidance while keeping hands and eyes on tasks, reducing errors, improving reporting speed,
Regarding claim 19, Au, Vukicevic, Castelli, and Howell disclose, the method of claim 11, further comprising: receiving a recording, generated by a microphone, of an utterance made by the subject; analyzing the utterance to determine it includes a voice command; and implementing the voice command. (See the rejection of claim 9 as it is equally applicable for claim 19 as well.)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID PERLMAN whose telephone number is (571) 270-1417.
The examiner can normally be reached on Monday - Friday; 10:00am -6:30pm.
Examiner interviews are available via telephone and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is
(571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at (866) 217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call (800) 786-9199 (IN USA OR CANADA) or (571) 272-1000.
/DAVID PERLMAN/Primary Examiner, Art Unit 2673
Prosecution Insights