DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 2/2/2026 have been fully considered but they are not persuasive.
PNG
media_image1.png
145
644
media_image1.png
Greyscale
PNG
media_image2.png
772
662
media_image2.png
Greyscale
On pages 9 and 10, Applicant argues,
Examiner respectfully disagrees.
PNG
media_image1.png
145
644
media_image1.png
Greyscale
In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “On the other hand, the subject matter of claim 1 is directed to identifying a specific operator based on the skeletal movements of operators who are performing a series of tasks. In other words, the presently amended claim 1 is configured to identify a particular subject who is performing certain motions associated with particular tasks. It is submitted that the motions selected and performed to complete the same task may vary among operators… Furthermore, the time consumed in each step also varies from operator to operator. Focusing on such differences among operators, the claimed subject matter is capable of identifying a target operator from among a plurality of operators.”) are not recited in the rejected claim. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Baek, column 4, lines 12-30, teaches a system which detects and tracks individual operators, among multiple simultaneously imaged, to produce individual risk assessment reports for repetitive motions of the operators. Columns 9 and 10, lines 45-52 and 48-54, respectively, teaches performing detection by generating a bounding box for each worker and performing risk assessment using a computed skeleton for the worker according to each bounding box. The relevance here being that there is a plurality of skeleton information, for each joint of the multiple operators, corresponding to a single image captured of the operators during work, and each operator’s skeleton is analyzed for individual risk assessment (see also fig. 7). Column 20 and 21, lines 50-67 and 1-22, teaches that waveforms for each worker are compared to reference waveforms for the risk assessment, to decide if ergonomic risk is high or low.
Under broadest reasonable interpretation, the claim limitation “a target identification to identify, among a plurality of skeleton information about the plurality of operators during work, detected by the skeleton information detection, skeleton information of a target operator indicating an operation satisfying a target requirement that a difference between the reference operation information and inspection subject information being operation information of the operator during work falls within a preset allowable operation range” merely requires the identification of a specific operator, among the multiple, whose operation falls within a preset allowable range. In Baek, column 21, lines 3-7, the “reference operation information” is explicitly taught as the “existing known waveform parameters”, which resemble waveforms of “safer workers”. The system then compares each operator’s waveform, or the “inspection target information”, to this reference. This comparison would necessarily distinguish workers whose operations fall within the acceptable range (low risk) from those who do not (high risk), such as through risk assessment reporting. This act of distinguishing a worker based on this comparison is the “target identification” required by the claim. Furthermore, Baek explicitly teaches identifying workers whose waveforms match the reference as targets for training, which is a direct example of identifying a target operator whose operation satisfies the requirement of being in a preset allowable range. Claim 1 does not require anything beyond this comparison for target operator identification, such as what the operator is being identified as a target for, thus, Baek teaches the limitation above, and the combination of Baek in view of Yong teaches all the limitation according to claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Baek et al. (US 11,328,239 B2) in view of Yong et al. (KR 101245231 B1), (hereinafter, Yong).
Regarding claim 1, Baek teaches a skeleton detection system (Baek, “Described herein is a prevention and safety management (PSM) system 10 that utilizes a non-intrusive imaging sensor 12 ( e.g. surveillance cameras, smartphone cameras) and a computer vision system to record videos of manufacturing workers performing tasks during their shifts (see FIG. 1). The videos are then analyzed using a deep machine learning algorithm for detecting the kinematic activities (set of predetermined body joint positions and angles) of the workers and recognizing various physical activities (walk/posture, lift, push, pull, reach, force, repetition, duration etc.).”, column 7, lines 13-18), comprising:
a first storage device configured to store reference operation information related to at least one cycle of a reference operation being a repetitive operation (Baek, “The motion manifold is estimated from data sets where 3D motion measurements of people are available. The Carnegie Mellon University Motion Capture (CMU MoCap) data set is preferred, but other relevant data sets can also be used, as long as the data set contains the 3D kinematics measurement and their corresponding (time-synced) video recording data.” column 18, lines 38-44, “Joint posture categories may be defined as recommended by NIOSH. In some cases, the user can enter joint posture categorization schemes customized to their needs. “, column 19, lines 65-67, Operator motion data is used to generate a motion manifold. This includes storing reference motion data of operators performing repetitive operations and using the data to train a neural network for 3D pose estimation.);
at least one processor (Baek, “The computer 601 may comprise one or more processors 603…”, column 21, lines 29-30); and
at least one memory comprising instructions that, when executed, cause the at least one processor to perform (Baek, “a system memory 612…”, column 21, lines 30-31);
a skeleton information detection to detect skeleton information for each of the plurality of operators from a working video obtained by capturing each of the plurality of operators during work (Baek, “a computer vision system to record videos of manufacturing workers performing tasks during their shifts”, column 7, lines 13-18 “Each new worker identified in the video stream is compared to the feature map database to determine whether that person has an existing employee profile. If the employee is already in the database, the tracking information is added to that employee's profile. If the feature map of a new bounding box does not match an existing employee profile, a new profile is created.”, column 20, lines 25-31, “FIG. 8 shows an algorithm that can be used to perform an ergonomic assessment of worker risk based on the positions, angles, and velocities of the various joints and landmarks of tracked workers. The positions of the landmarks and joints are used to create the kinematic configuration for each worker, as shown in FIGS. 6 and 7.”, column 20, lines 51-56, see Figs. 6 and 7, Joints and landmarks are estimated and stored for both new and existing workers who are present in the video stream.); and
a target identification to identify, among a plurality of skeleton information about the plurality of operators during work, detected by the skeleton information detection, skeleton information of a target operator indicating an operation satisfying a target requirement that a difference between the reference operation information and inspection subject information being operation information of the operator during work falls within a preset allowable operation range (Baek, “Multiple workers 14 can also be monitored simultaneously if needed.”, column 8, lines 1-2, “In essence, the 2-D coordinates of the landmarks and joints can be used to detect a 2-D posture for each person in each frame. The 2-D posture coordinates are compared to a 3-D motion capture database to estimate the likely 3-D posture of each worker in each frame, based on the 2-D coordinates of the landmarks and joints. Based on the likely 3-D postures of each worker in each frame, the joint angles and changes in joint angles over time can be calculated and recorded in that worker's database record. The system generates time-series continuous data that enables analysis of risk data.”, column 20, lines 51-62, “Waveform graphs, as shown in FIGS. 9, 10, and 11 may be created and visually displayed for selected joints or markers. These waveform graphs can be compared with existing known waveform parameters to determine when workers are being exposed to higher risks… The waveform data might also be useful for optimizing performance of workers. For examples the waveforms of high production or safer workers might be studied and used as examples for training new workers.”, column 21, lines 3-22, see Figure 7, Multiple workers can be simultaneously tracked for ergonomic risk assessment per frame of the video stream. For each workers skeleton, a repetitive motion cycle is represented as waveform graphs. The system compares these waveforms with ergonomic thresholds to assess whether posture falls in an allowable operation range for risk assessment. Target workers with waveforms that reflect low-risk are identified among the multiple tracked skeletons for risk assessment reporting and can also be used as a reference for optimizing performance of workers, such as through training.).
Baek does not teach storing reference operation information related to at least one cycle of a reference operation being a repetitive operation which is performed by a plurality of operators and whose contents are the same for the plurality of operators
However, Yong teaches storing reference operation information related to at least one cycle of a reference operation being a repetitive operation which is performed by a plurality of operators and whose contents are the same for the plurality of operators (Yong, “the main technical configuration of the present invention is configured to measure a work time using a video obtained by continuously photographing a motion of a worker performing the same repetitive work on a production line”, pg. 4, paragraph 0005, lines 1-5, “The present invention relates to a motion analysis method… which can innovatively improve productivity by standardizing a work management system by precisely observing and analyzing specific time and motions included in machine work and human work through motion analysis using a video to remove waste and set the order or combination of better work.”, pgs. 9-10, paragraph 0022, lines 1-6, “In addition, since there may be a difference in time required for the same work process depending on the worker or the machine used by the worker, it is preferable to photograph each work operation performed by a plurality of workers as much as possible. Next, the program installation step (s20) relates to a step of storing the video captured in the video capturing step (s10) in a storage device such as a computer, and installing a motion analysis program for checking and analyzing a work motion included in the captured video in the computer.”, pg. 10 , paragraph 0026, lines 1-4, Motion data is stored for multiple operators on a production line performing the same repetitive operation, and this data then is analyzed to standardize the operation across workers.).
Baek teaches storing reference information to form a motion manifold, used to train a network for detecting repetitive operations of multiple operators in a video stream (Baek, “FIG. 8 shows an algorithm that can be used to perform an ergonomic assessment of worker risk based on the positions, angles, and velocities of the various joints and landmarks of tracked workers. The positions of the landmarks and joints are used to create the kinematic configuration for each worker, as shown in FIGS. 6 and 7.”, column 20, lines 51-56, see Figs. 6-8). Baek does not teach storing reference information of repetitive operations cycles corresponding to multiple operators performing the same repetitive operation. Yong teaches storing the same repetitive operation performed by multiple operators on a production line, for standardization across workers (see above). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the motion manifold of Baek to include stored videos of multiple operators performing the same operation, as taught by Yong (Yong, pgs. 9-10, paragraph 0022, lines 1-6 and pg. 10 , paragraph 0026, lines 1-4), thus training the network to detect repetitive operations of multiple operators performing the same operation in the video stream. The motivation for doing so would have been to standardize the same operation across multiple workers (e.g., on a production line), thereby improving productivity of the workers (as suggested by Yong, “…the work motion performed by the derived improvement plan by a photographing means again, thereby minimizing waste included in the work motion, setting a better order or combination of work, and finally standardizing a work management system, thereby innovatively improving productivity.”, pg. 20, paragraph 0072). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Baek with Yong to obtain the invention according to claim 1.
Regarding claim 2, Baek in view Yong of teaches the skeleton detection system according to claim 1, wherein the first storage device stores the reference operation information, about the target operator, obtained from a reference video obtained by capturing a reference operation of the target operator (Baek, “The motion manifold is estimated from data sets where 3D motion measurements of people are available. The Carnegie Mellon University Motion Capture (CMU MoCap) data set is preferred, but other relevant data sets can also be used, as long as the data set contains the 3D kinematics measurement and their corresponding (time-synced) video recording data.”, column 18, lines 38-44, The motion manifold is established by storing reference motion data of operators skeletons performing repetitive operations in video data.)
Regarding claim 3, Baek in view Yong teaches the skeleton detection system according to claim 2, further comprising:
a second storage device configured to store the inspection subject information (Baek, “The computing device is adapted to detect workers within the image data. The detected workers movements are tracked within the image data. Workers are identified and assigned a file identity within a database such that all tracked movements made by a single worker are 40 saved in a record within the database associated with that worker”, column 20, lines 25-29, A feature map database of each individual worker is maintained for storing detected feature map information during working operations.).
Regarding claim 4, Baek in view Yong teaches the skeleton detection system according to claim 1, wherein the at least one processor is further caused to perform
a candidate selection to select a candidate satisfying a predetermined candidate requirement from the plurality of skeleton information about the plurality of operators detected from the working video (Baek, “Individual workers 14 are detected by using a region-based frame-by-frame analysis of the video stream. In particular, bounding boxes are generated around each person detected in the image data.”, column 9, lines 45-49, various workers are selected as candidates for motion tracking contingent on whether or not they are in the frame of the video data),
wherein the target identification device identifies skeleton information about the operator selected by the candidate selection, indicating an operation satisfying a target requirement that a difference between the reference operation information and the inspection subject information about the operator falls within a preset allowable operation range, as the skeleton information of the target operator (Baek, “the 2-D coordinates of the landmarks and joints can be used to detect a 2-D posture for each person in each frame. The 2-D posture coordinates are compared to a 3-D motion capture database to estimate the likely 3-D posture of each worker in each frame”, column 20, lines 51-62, “Waveform graphs, as shown in FIGS. 9, 10, and 11 may be created and visually displayed for selected joints or markers. These waveform graphs can be compared with existing known waveform parameters to determine when workers are being exposed to higher risks.”, column 21, lines 3-7, See analysis of claim 1, Each candidate of the frames are individually processed for motion comparison to determine if the operations satisfy a waveform parameter established by the reference data.).
Regarding claim 7, Baek in view of Yong teaches the skeleton detection system according to claim 4, wherein the candidate requirement includes a condition that the skeleton information is located in a predetermined region in a capturing range indicated by the working video (Baek, “Individual workers 14 are detected by using a region-based frame-by-frame analysis of the video stream. In particular, bounding boxes are generated around each person detected in the image data.”, column 9, lines 45-49, The predetermined region can be interpreted broadly as the capturing range of the video frame, therefore, the candidates being selected only if they are within the frame satisfies the candidate requirement of a predetermined region in a capturing range of the video.).
Regarding claim 8, Baek in view of Yong teaches the skeleton detection system according to claim 1, wherein the target identification device identifies the target operator, based on an operation indicated by the skeleton information represented by a change in each of a first position and a second position separated on both sides sandwiching a center line of a body of the operator detected from the working video, and a third position separated from each of the first position and the second position of the body (Baek, “The feature map preferably tracks several landmarks and joints (represented by dots in FIGS. 6 and 7) on each worker to aid in ergonomic analyses. In a preferred embodiment the following joints and landmarks may be tracked: left hip; right hip; chest; neck; left shoulder; right shoulder; left elbow; right elbow; left wrist; right wrist; left knee; right knee; left ankle; right ankle; nose; left ear; right ear; left eye; and right eye… Each of the landmarks and joints are tracked and compared for each frame to generate a kinematic configuration of each person. Examples of maps of the kinematic configuration of workers can be seen in FIGS. 6 and 7.”, columns 10 and 11, lines 67 and 1-13, respectively, “We set the head as the root node for the advantage of tracking, as the head region tends to provide stronger and richer visual cues and features effective for detection and tracking.”, column 17, lines 9-12, Various left and right landmark positions in relation to a point on the head of the workers is utilized in the skeleton information to effectively track the change in the points for motion detection of the workers. Figs. 6 and 7 illustrates a center line or “bone” of the skeleton which is between respective left and right landmarks).
Regarding claim 9, Baek in view of Yong teaches the skeleton detection system according to claim 8, wherein the positional data of the first position to the third position in the working video is included in the inspection subject information (Baek, “The positions of the landmarks and joints are used to create the kinematic configuration for each worker, as shown in FIGS. 6 and 7.”, column 20, lines 54-56, The established landmarks, including right, left and head points are used to determine motions corresponding to the operations of the workers).
Regarding claim 10, Baek in view of Yong teaches the skeleton detection system according to claim 1, wherein the skeleton information includes data indicating a position of at least either of a plurality of portions of the operator or a line that connects two or more portions of the plurality of portions of the operator (Baek, “The feature map preferably tracks several landmarks and joints (represented by dots in FIGS. 6 and 7)… left hip; right hip; chest; neck; left shoulder; right shoulder; left elbow; right elbow; left wrist; right wrist; left knee; right knee; left ankle; right ankle; nose; left ear; right ear; left eye; and right eye”, columns 10 and 11, lines 67 and 1-13, respectively, various landmarks and joints of the operator is determined for the skeleton, the landmarks are connected by lines as illustrated in Figs. 6 and 7).
Regarding claim 11, Baek in view of Yong teaches a work management device, comprising:
the skeleton detection system according to claim 1 (see analysis of claim 1);
wherein the at least one processor is further caused to perform a pass/fail determination that determines a pass or a fail of the operation of the target operator, based on whether a difference between the inspection subject information indicated by the target operator and the reference operation information corresponding to the target operator falls within a preset reference range (Baek, “The tracked movements are analyzed using ergonomic analysis tools to generate worker risk assessments.” Column 9, lines 42-43, “Waveform graphs, as shown in FIGS. 9, 10, and 11 may be created and visually displayed for selected joints or markers. These waveform graphs can be compared with existing known waveform parameters to determine when workers are being exposed to higher risks.”, column 21, lines 3-7, Waveform graphs for selected joints of the skeleton can be determined for the workers and compared to reference data in order to determine if the motion of the worker falls within the preset range.); and
an output device configured to output a result of the pass/fail determination (Baek, “This data is also useful for creating epidemiological data that can be used to study and measure the risks of various activities for the population of workers. This is especially useful for being data generated in the actual workplace rather than in an artificial laboratory setting.”, column 21, lines 8-12, The risk assessment result is output and used to study generalized risks of various operations of the workers.).
Regarding claim 12, Baek in view of Yong teaches the skeleton detection system according to claim 1, further comprising:
a second storage device configured to store the inspection subject information (Baek, “The computing device is adapted to detect workers within the image data. The detected workers movements are tracked within the image data. Workers are identified and assigned a file identity within a database such that all tracked movements made by a single worker are 40 saved in a record within the database associated with that worker”, column 20, lines 25-29, A feature map database of each individual worker is maintained for storing detected feature map information during working operations.).
Regarding claim 13, Baek in view of Yong teaches the skeleton detection system according to claim 2, wherein the at least one processor is further caused to perform
a candidate selection to select a candidate satisfying a predetermined candidate requirement from the plurality of skeleton information about the plurality of operators detected from the working video (Baek, “Individual workers 14 are detected by using a region-based frame-by-frame analysis of the video stream. In particular, bounding boxes are generated around each person detected in the image data.”, column 9, lines 45-49, Various workers are selected as candidates for motion tracking contingent on whether or not they are in the frame of the video data.),
wherein the target identification device identifies skeleton information about the operator selected by the candidate selection, indicating an operation satisfying a target requirement that a difference between the reference operation information and the inspection subject information about the operator falls within a preset allowable operation range, as the skeleton information of the target operator (Baek, “the 2-D coordinates of the landmarks and joints can be used to detect a 2-D posture for each person in each frame. The 2-D posture coordinates are compared to a 3-D motion capture database to estimate the likely 3-D posture of each worker in each frame”, column 20, lines 51-62, “Waveform graphs, as shown in FIGS. 9, 10, and 11 may be created and visually displayed for selected joints or markers. These waveform graphs can be compared with existing known waveform parameters to determine when workers are being exposed to higher risks.”, column 21, lines 3-7, See analysis of claim 1, Each candidate of the frames are individually processed for motion comparison to determine if the operations satisfy a waveform parameter established by the reference data.).
Regarding claim 14, Baek in view of Yong teaches the skeleton detection system according to claim 3, wherein the at least one processor is further caused to perform
a candidate selection to select a candidate satisfying a predetermined candidate requirement from the plurality of skeleton information about the plurality of operators detected from the working video (Baek, “Individual workers 14 are detected by using a region-based frame-by-frame analysis of the video stream. In particular, bounding boxes are generated around each person detected in the image data.”, column 9, lines 45-49, Various workers are selected as candidates for motion tracking contingent on whether or not they are in the frame of the video data.),
wherein the target identification device identifies skeleton information about the operator selected by the candidate selection, indicating an operation satisfying a target requirement that a difference between the reference operation information and the inspection subject information about the operator falls within a preset allowable operation range, as the skeleton information of the target operator (Baek, “the 2-D coordinates of the landmarks and joints can be used to detect a 2-D posture for each person in each frame. The 2-D posture coordinates are compared to a 3-D motion capture database to estimate the likely 3-D posture of each worker in each frame”, column 20, lines 51-62, “Waveform graphs, as shown in FIGS. 9, 10, and 11 may be created and visually displayed for selected joints or markers. These waveform graphs can be compared with existing known waveform parameters to determine when workers are being exposed to higher risks.”, column 21, lines 3-7, See analysis of claim 1, Each candidate of the frames are individually processed for motion comparison to determine if the operations satisfy a waveform parameter established by the reference data.).
Regarding claim 15, Baek in view of Yong teaches the skeleton detection system according to claim 2, wherein the target identification identifies the target operator, based on an operation indicated by the skeleton information represented by a change in each of a first position and a second position separated on both sides sandwiching a center line of a body of the operator detected from the working video, and a third position separated from each of the first position and the second position of the body (Baek, “The feature map preferably tracks several landmarks and joints (represented by dots in FIGS. 6 and 7) on each worker to aid in ergonomic analyses. In a preferred embodiment the following joints and landmarks may be tracked: left hip; right hip; chest; neck; left shoulder; right shoulder; left elbow; right elbow; left wrist; right wrist; left knee; right knee; left ankle; right ankle; nose; left ear; right ear; left eye; and right eye… Each of the landmarks and joints are tracked and compared for each frame to generate a kinematic configuration of each person. Examples of maps of the kinematic configuration of workers can be seen in FIGS. 6 and 7.”, columns 10 and 11, lines 67 and 1-13, respectively, “We set the head as the root node for the advantage of tracking, as the head region tends to provide stronger and richer visual cues and features effective for detection and tracking.”, column 17, lines 9-12, Various left and right landmark positions in relation to a point on the head of the workers is utilized in the skeleton information, to effectively track the change in the points for motion detection of the workers. Figs. 6 and 7 illustrates a center line or “bone” of the skeleton which is between respective left and right landmarks.).
Regarding claim 16, Baek in view of Yong teaches the skeleton detection system according to claim 4, wherein the target identification unit identifies the target operator, based on an operation indicated by the skeleton information represented by a change in each of a first position and a second position separated on both sides sandwiching a center line of a body of the operator detected from the working video, and a third position separated from each of the first position and the second position of the body (Baek, “The feature map preferably tracks several landmarks and joints (represented by dots in FIGS. 6 and 7) on each worker to aid in ergonomic analyses. In a preferred embodiment the following joints and landmarks may be tracked: left hip; right hip; chest; neck; left shoulder; right shoulder; left elbow; right elbow; left wrist; right wrist; left knee; right knee; left ankle; right ankle; nose; left ear; right ear; left eye; and right eye… Each of the landmarks and joints are tracked and compared for each frame to generate a kinematic configuration of each person. Examples of maps of the kinematic configuration of workers can be seen in FIGS. 6 and 7.”, columns 10 and 11, lines 67 and 1-13, respectively, “We set the head as the root node for the advantage of tracking, as the head region tends to provide stronger and richer visual cues and features effective for detection and tracking.”, column 17, lines 9-12, Various left and right landmark positions in relation to a point on the head of the workers is utilized in the skeleton information, to effectively track the change in the points for motion detection of the workers. Figs. 6 and 7 illustrates a center line or “bone” of the skeleton which is between respective left and right landmarks.).
Regarding claim 18, Baek in view of Yong teaches the skeleton detection system according to claim 7, wherein the target identification identifies the target operator, based on an operation indicated by the skeleton information represented by a change in each of a first position and a second position separated on both sides sandwiching a center line of a body of the operator detected from the working video, and a third position separated from each of the first position and the second position of the body (Baek, “The feature map preferably tracks several landmarks and joints (represented by dots in FIGS. 6 and 7) on each worker to aid in ergonomic analyses. In a preferred embodiment the following joints and landmarks may be tracked: left hip; right hip; chest; neck; left shoulder; right shoulder; left elbow; right elbow; left wrist; right wrist; left knee; right knee; left ankle; right ankle; nose; left ear; right ear; left eye; and right eye… Each of the landmarks and joints are tracked and compared for each frame to generate a kinematic configuration of each person. Examples of maps of the kinematic configuration of workers can be seen in FIGS. 6 and 7.”, columns 10 and 11, lines 67 and 1-13, respectively, “We set the head as the root node for the advantage of tracking, as the head region tends to provide stronger and richer visual cues and features effective for detection and tracking.”, column 17, lines 9-12, Various left and right landmark positions in relation to a point on the head of the workers is utilized in the skeleton information, to effectively track the change in the points for motion detection of the workers. Figs. 6 and 7 illustrates a center line or “bone” of the skeleton which is between respective left and right landmarks.).
Regarding claim 19, Baek in view of Yong teaches a work management device, comprising:
the skeleton detection system according to claim 2 (see analysis of claim 2);
wherein the at least one processor is further caused to perform a pass/fail determination that determines a pass or a fail of the operation of the target operator, based on whether a difference between the inspection subject information indicated by the target operator and the reference operation information corresponding to the target operator falls within a preset reference range (Baek, “The tracked movements are analyzed using ergonomic analysis tools to generate worker risk assessments.” Column 9, lines 42-43, “Waveform graphs, as shown in FIGS. 9, 10, and 11 may be created and visually displayed for selected joints or markers. These waveform graphs can be compared with existing known waveform parameters to determine when workers are being exposed to higher risks.”, column 21, lines 3-7, Waveform graphs for selected joints of the skeleton can be determined for the workers and compared to reference data in order to determine if the motion of the worker falls within the preset range.); and
an output device configured to output a result of the pass/fail determination (Baek, “This data is also useful for creating epidemiological data that can be used to study and measure the risks of various activities for the population of workers. This is especially useful for being data generated in the actual workplace rather than in an artificial laboratory setting.”, column 21, lines 8-12, The risk assessment result is output and used to study generalized risks of various operations of the workers.).
Regarding claim 20, Baek in view of Yong teaches a work management device, comprising:
the skeleton detection system according to claim 4 (see analysis of claim 4);
wherein the at least one processor is further caused to perform a pass/fail determination that determines a pass or a fail of the operation of the target operator, based on whether a difference between the inspection subject information indicated by the target operator and the reference operation information corresponding to the target operator falls within a preset reference range (Baek, “The tracked movements are analyzed using ergonomic analysis tools to generate worker risk assessments.” Column 9, lines 42-43, “Waveform graphs, as shown in FIGS. 9, 10, and 11 may be created and visually displayed for selected joints or markers. These waveform graphs can be compared with existing known waveform parameters to determine when workers are being exposed to higher risks.”, column 21, lines 3-7, Waveform graphs for selected joints of the skeleton can be determined for the workers and compared to reference data in order to determine if the motion of the worker falls within the preset range defined by the reference waveform graphs.); and
an output device configured to output a result of the pass/fail determination (Baek, “This data is also useful for creating epidemiological data that can be used to study and measure the risks of various activities for the population of workers. This is especially useful for being data generated in the actual workplace rather than in an artificial laboratory setting.”, column 21, lines 8-12, The risk assessment result is output and used to study generalized risks of various operations of the workers.).
Claims 5 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Baek et al. (US 11,328,239 B2) in view of Yong et al. (KR 101245231 B1) and further in view of Segawa et al. (US 10,474,876 B2), (Hereinafter, Segawa).
Regarding claim 5, Baek in view of Yong teaches the skeleton system according to claim 4, but the combination does not teach wherein the candidate requirement includes a condition that a size of a skeleton indicated by the skeleton information detected from the working video is largest among the plurality of skeleton information about the plurality of operators detected from the working video.
However, Segawa teaches wherein the candidate requirement includes a condition that a size of a person largest detected from the working video (Segawa, “When faces have been detected by face detection, a face with a larger detection frame is more likely than not on this side of a face with a smaller detection frame, and therefore, faces with larger detection frames may be placed at higher positions in the order of priority (that is, given higher degrees of priority).”, columns 4 and 5, lines 65-67 and 1-3, respectively, the largest detection frame corresponding to the detected humans is used to identify candidates with highest priority for future analysis).
Baek in view of Yong teaches detecting multiple bounding box regions of workers which contains skeleton information for each box (Baek, “Each of the identified workers 14 within a bounding box 18 is then tracked by the computing device 16 using the algorithm shown illustrated in FIG. 5. The preferred framework for tracking the workers is DeepSORT… The DeepSORT framework extracts a feature map within each bounding box. As a result, each worker has a unique feature map that can be used to identify and track the worker and his or her features within the video stream.”, column 10, lines 48-61). Segawa teaches a candidate selection condition for the largest detection region (Segawa, columns 4 and 5, lines 65-67 and 1-3, respectively). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Baek in view of Yong to include a candidate selection condition corresponding to the largest detected person as taught by Segawa (columns 4 and 5, lines 65-67 and 1-3), thereby selecting a candidate based on the largest skeleton information of the determined bounding boxes (Baek, column 10, lines 48-61). The motivation for doing so would have been to implement detection conditions into a multi-person detection method, thereby reducing the number of parameters and computational load on the system (As suggested by Segawa, “Moreover, when the poses of a plurality of people are to be estimated simultaneously, the number of parameters increases further, leading to a still larger amount of computation required.”, column 1, lines 41-45). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine the teachings of Baek in view of Yong with Segawa to obtain the invention as specified in claim 5.
Regarding claim 17, Baek in view of Yong and further in view of Segawa teaches the skeleton detection system according to claim 5, wherein the target identification identifies the target operator, based on an operation indicated by the skeleton information represented by a change in each of a first position and a second position separated on both sides sandwiching a center line of a body of the operator detected from the working video, and a third position separated from each of the first position and the second position of the body (Baek, “The feature map preferably tracks several landmarks and joints (represented by dots in FIGS. 6 and 7) on each worker to aid in ergonomic analyses. In a preferred embodiment the following joints and landmarks may be tracked: left hip; right hip; chest; neck; left shoulder; right shoulder; left elbow; right elbow; left wrist; right wrist; left knee; right knee; left ankle; right ankle; nose; left ear; right ear; left eye; and right eye… Each of the landmarks and joints are tracked and compared for each frame to generate a kinematic configuration of each person. Examples of maps of the kinematic configuration of workers can be seen in FIGS. 6 and 7.”, columns 10 and 11, lines 67 and 1-13, respectively, “We set the head as the root node for the advantage of tracking, as the head region tends to provide stronger and richer visual cues and features effective for detection and tracking.”, column 17, lines 9-12, Various left and right landmark positions in relation to a point on the head of the workers is utilized in the skeleton information, to effectively track the change in the points for motion detection of the workers. Figs. 6 and 7 illustrates a center line or “bone” of the skeleton which is between respective left and right landmarks.).
Allowable Subject Matter
Claim 6 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CONNOR LEVI HANSEN whose telephone number is (703)756-5533. The examiner can normally be reached Monday-Friday 9:00-5:00 (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached on (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CONNOR L HANSEN/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights