DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-17 are pending regarding this application.
Priority
The present application claims foreign priority benefits from KR10-2021-0113406 filed on
08/26/2021. The certified copies of the priority documents were electronically retrieved on 03/16/2024.
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/23/2024, 04/11/2025, and 11/18/2025 are considered and attached.
Claim Objections
Claim 14 is objected to because of the following informalities:
In claim 14, please amend the phrase “a posture of the operator” to recite “the posture of the operator” as “a posture of an operator” is already introduced in line 9 of claim 10, upon which claim 14 depends.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites “a region of interest in a workspace” in line 1 and “a region of interest corresponding to a machine tool” in line 5. Here, it is unclear whether the region of interest as recited in line 1 is equivalent to or distinct from the region of interest as recited in line 5. Applicant’s specification discusses the region of interest; however, no sections of the specification specifically clarify whether there are two different regions of interest are claimed in claim 1, or whether the two regions of interest are the same. As such, claim 1 is rejected for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention. One suggestion to resolve this issue may be to amend line 5 of claim 1 to recite “setting the region of interest”.
Corresponding claim 10 is similarly rejected. Claims 2-9 and 11-17 are rejected due to their dependence upon claims 1 and 10.
Regarding claim 4, claim 4 recites “determining whether point cloud data corresponding to the operator exists in the emergency region and sub-observation regions distinguished by body parts of the operator; and determining the posture of the operator based on a combination of regions in which the point cloud data corresponding to the operator exists among the emergency region and the sub-observation regions”. However, it is unclear how one can determine whether point cloud data corresponding to the operator exists in […] sub-observation regions distinguished by body parts of the operator, if it is already inherent that these sub-observation regions contain point cloud data corresponding to the operator. Applicant discusses the above subject matter in the specification in para. [0053]-[0061]. However, none of these sections clarify what is being claimed in the above subject matter. More specifically, applicant’s specification never clarifies how point cloud data corresponding to the operator can be determined to exist in sub-observation regions corresponding to body parts of the operator. As a result, claim 4 is rejected for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Corresponding claim 13 is similarly rejected.
Claims 1-17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being incomplete for omitting essential elements, such omission amounting to a gap between the elements. See MPEP § 2172.01. The omitted elements are: “setting a region of interest corresponding to a machine tool placed in the workplace”.
In the above limitation (found in line 5 of claims 1 and 10), a region of interest is set corresponding to a machine tool placed in the workplace. However, from the claim language, it is unclear where that region of interest is set. Said differently, applicant’s specification clarifies that the region of interest corresponds to a region within a three-dimensional space, wherein the regions of interest may be set “in a spatial information map generated based on the point cloud data with respect to the workspace” as shown in para. [0085]-[0086]. However, the limitation of “setting a region of interest corresponding to a machine tool placed in the workplace” fails to convey the element of where the region of interest exists. As such, one solution may be to amend the claim language to recite “setting a region of interest within a spatial information map generated based on the point cloud data with respect to the workspace corresponding to a machine tool placed in the workplace”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 8-11, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Zahand (U.S. Publication No. 2015/0228078) in view of Denenberg et al. (U.S. Publication No. 2021/0379763 A1), hereinafter Denenberg.
Regarding claim 1, Zahand teaches a method for monitoring a region of interest in a workspace (Zahand teaches “systems and methods for monitoring a workstation region of a manufacturing line” in para. [0002]), the method comprising:
obtaining (Zahand teaches that “a depth camera 44 may generate depth image data 46 that is provided to the computing device 22” in para. [0015]. Here, the depth camera is interpreted as the claimed 3D sensor that detects a 3D space. Zahand further teaches that “the depth camera 44 may determine, for each pixel, the depth of a surface in the observed scene relative to the depth camera”, wherein “the manufacturing line 18 as well as other equipment, components and features within the workstation region 14 may also be imaged by depth camera 44” along with the operator 304 as shown in para. [0020] and FIG. 2. The depth values are used to generate a depth map (see para. [0021] which is interpreted as equivalent to the claimed data);
setting a region of interest corresponding to a machine tool placed in the workspace (Zahand teaches specifically analyzing a nip region 302 of a conveyor belt which is interpreted as the claimed region of interest corresponding to a machine tool placed in the workspace in para. [0028]. Furthermore, Zahand teaches that the user may set a second defined alert region 84 encapsulating the nip region (region between the rollers 320 and conveyor belt 314) as shown in para. [0039]; See also para. [0030] (here, the robotic arm 340 is the region of interest));
tracking a posture of an operator operating the machine tool based on the (Zahand teaches “utilizing the depth image data 46 and corresponding series of movements of the operator 304, the line monitoring program 26 may determine that the operator 304 is within a predetermined distance of the nip 402” as shown in para. [0028]. Here, the operator 304 is tracked with respect to the nip 402 (region of interest). See also para. [0029]-[0030]. Zahand further teaches specifically tracking movements of the operator by using depth image data 46 to generate “a skeletal model of an operator 204 in the workstation region 14 based on a captured temporal sequence of images of the operator” as shown in para. [0018] and FIG. 2); and
detecting an emergency situation in the workspace, based on the tracked posture of the operator (Zahand teaches “utilizing the depth image data 46 and corresponding series of movements of the operator 304, the line monitoring program 26 may determine that the operator 304 is within a predetermined distance of the nip 402” in para. [0028], wherein, when “the operator's hand 330 is determined to have entered the nip 230[, the incident] may be classified as an Alert 4 in which the flashing beacon 342 is activated, the operator's manager 344 is notified, a siren in the workstation region 14 is sounded and the manufacturing line 18 is shut down” as shown in para. [0032]. See also FIG. 1, para. [0030] (here, the robotic arm 340 is the region of interest)).
Zahand fails to specifically teach obtaining point cloud data of the workspace.
However, Denenberg teaches obtaining point cloud data of the workspace (Denenberg teaches “to model the robot dynamics and/or human activities in the workspace 100 and map the safe and/or unsafe regions, in various embodiments, the control system 112 first computationally generates a 3D spatial representation (e.g., as voxels) of the workspace 100 where the machinery (including the robot 106 and auxiliary equipment), workpiece and human operator are based on, for example, the scanning data acquired by the sensor system 101”, wherein , using this model, “a spatial POE [“potential occupancy envelopes”] of the machinery can be estimated” and “the POE may be represented in any computationally convenient form, e.g., as a cloud of points” as shown in para. [0069]. Here, the POE is interpreted as equivalent to the claimed point cloud data, as it is derived from data captured using a 3D sensor. See also para. [0059], wherein the sensors are ToF cameras or 3D LIDAR sensors, which typically capture data in the form of point clouds).
Zahand and Denenberg are both considered to be analogous to the claimed invention because they are in the same field of analyzing workspace environments using 3D information to promote safety between operators and machinery in the workplace. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zahand to incorporate the teachings of Denenberg and include “obtaining point cloud data of the workspace”. The motivation for doing so would have been that “it may be desired to model and/or compute, in real time, the robot dynamics and/or human activities and provide safety mapping of the robot and/or human in the workspace”, as suggested by Denenberg in para. [0068]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zahand with Denenberg to obtain the invention specified in claim 1.
Regarding claim 2, Zahand and Denenberg teach the method of claim 1,
wherein the region of interest comprises an emergency region of the machine tool (Zahand teaches that “a depth map generated by one or more of the depth cameras 44 may identify the location of the nip 402 in the 3D space of the workstation region 14” in para. [0028]. Here, the nip 402 and the robotic arm 340 (see para. [0030]) are interpreted as equivalent to the claimed emergency region and may also be classified as an alert region (emergency region) of the machine tool as shown in para. [0039]) and
an observation region where the body of the operator is located when operating the machine tool (Zahand teaches observing a region where the operator is located within the workspace region, and tracking their movements with respect to the machine tool(s) in para. [0018]. See also para. [0039]).
Regarding claim 8, Zahand and Denenberg teach the method of claim 1, further comprising:
predicting a movement of the operator according to a change in the tracked posture of the operator (Denenberg teaches “a spatial POE of the human operator that characterizes the spatial region potentially occupied by any portion of the human operator is based on any possible or anticipated movements of the human operator” as shown in para. [0079], wherein “the POE 602 of the human operator is refined by acquiring more information about the operator. For example, the sensor system 101 may acquire a series of scanning data (e.g., images) within a time interval Δt. By analyzing the operator's positions and poses in the scanning data and based on the time period Δt, the operator's moving direction, velocity and acceleration can be determined” as shown in para. [0080]); and
predicting an emergency situation in the workspace according to the predicted movement (Denenberg teaches initiating an emergency stop in the workplace according to the predicted movement of both the machinery and a human operator as shown in para. [0102]. Here, this initiation is interpreted as equivalent to predicting an emergency situation, as the initiation is an emergency action carried out based on predicted movement of the operator).
Similar motivation as applied to claim 1 can be applied here to claim 8.
Regarding claim 9, Zahand and Denenberg teach the non-transitory computer-readable storage medium storing instructions, when executed by one or more processors, configured to cause the one or more processors to perform the method of claim 1 (Zahand teaches “the logic subsystem may include [one or more processors that are configured to execute software instructions and/or] one or more hardware or firmware logic machines configured to execute hardware or firmware instructions” as shown in para. [0055]).
Regarding claim 10, Zahand teaches sensing device for monitoring a region of interest in a workspace (Zahand teaches “the monitoring system 10 may also include one or more other sensors for receiving input from the workstation region 14 of the manufacturing line 18” in para. [0017]), the sensing device comprising:
a sensor configured to obtain (Zahand teaches that “a depth camera 44 may generate depth image data 46 that is provided to the computing device 22” in para. [0015]. Here, the depth camera is interpreted as the claimed 3D sensor that detects a 3D space. Zahand further teaches that “the depth camera 44 may determine, for each pixel, the depth of a surface in the observed scene relative to the depth camera”, wherein “the manufacturing line 18 as well as other equipment, components and features within the workstation region 14 may also be imaged by depth camera 44” along with the operator 304 as shown in para. [0020] and FIG. 2. The depth values are used to generate a depth map (see para. [0021] which is interpreted as equivalent to the claimed data);
a memory storing one or more instructions; and a processor configured to execute the one or more instructions (Zahand teaches “the monitoring system 10 may comprise a computing device 22 that includes a line monitoring program 26 stored in mass storage 30 of the computing device 22. The line monitoring program 26 may be loaded into memory 34 and executed by a processor 38 of the computing device 22 to perform one or more of the methods and processes” in para. [0011]) to:
set a region of interest corresponding to a machine tool placed in the workspace (Zahand teaches specifically analyzing a nip region 302 of a conveyor belt which is interpreted as the claimed region of interest corresponding to a machine tool placed in the workspace in para. [0028]. Furthermore, Zahand teaches that the user may set a second defined alert region 84 encapsulating the nip region (region between the rollers 320 and conveyor belt 314) as shown in para. [0039]; See also para. [0030] (here, the robotic arm 340 is the region of interest)),
track a posture of an operator operating the machine tool based on the (Zahand teaches “utilizing the depth image data 46 and corresponding series of movements of the operator 304, the line monitoring program 26 may determine that the operator 304 is within a predetermined distance of the nip 402” as shown in para. [0028]. Here, the operator 304 is tracked with respect to the nip 402 (region of interest). See also para. [0029]-[0030]. Zahand further teaches specifically tracking movements of the operator by using depth image data 46 to generate “a skeletal model of an operator 204 in the workstation region 14 based on a captured temporal sequence of images of the operator” as shown in para. [0018] and FIG. 2), and
detect an emergency situation in the workspace, based on the tracked posture of the operator (Zahand teaches “utilizing the depth image data 46 and corresponding series of movements of the operator 304, the line monitoring program 26 may determine that the operator 304 is within a predetermined distance of the nip 402” in para. [0028], wherein, when “the operator's hand 330 is determined to have entered the nip 230[, the incident] may be classified as an Alert 4 in which the flashing beacon 342 is activated, the operator's manager 344 is notified, a siren in the workstation region 14 is sounded and the manufacturing line 18 is shut down” as shown in para. [0032]. See also FIG. 1, para. [0030] (here, the robotic arm 340 is the region of interest)).
Zahand fails to specifically teach obtaining point cloud data of the workspace.
However, Denenberg teaches obtaining point cloud data of the workspace (Denenberg teaches “to model the robot dynamics and/or human activities in the workspace 100 and map the safe and/or unsafe regions, in various embodiments, the control system 112 first computationally generates a 3D spatial representation (e.g., as voxels) of the workspace 100 where the machinery (including the robot 106 and auxiliary equipment), workpiece and human operator are based on, for example, the scanning data acquired by the sensor system 101”, wherein , using this model, “a spatial POE [“potential occupancy envelopes”] of the machinery can be estimated” and “the POE may be represented in any computationally convenient form, e.g., as a cloud of points” as shown in para. [0069]. Here, the POE is interpreted as equivalent to the claimed point cloud data, as it is derived from data captured using a 3D sensor. See also para. [0059], wherein the sensors are ToF cameras or 3D LIDAR sensors, which typically capture data in the form of point clouds).
Zahand and Denenberg are both considered to be analogous to the claimed invention because they are in the same field of analyzing workspace environments using 3D information to promote safety between operators and machinery in the workplace. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zahand to incorporate the teachings of Denenberg and include “obtaining point cloud data of the workspace”. The motivation for doing so would have been that “it may be desired to model and/or compute, in real time, the robot dynamics and/or human activities and provide safety mapping of the robot and/or human in the workspace”, as suggested by Denenberg in para. [0068]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zahand with Denenberg to obtain the invention specified in claim 10.
Regarding claim 11, Zahand and Denenberg teach the sensing device of claim 10,
wherein the region of interest comprises an emergency region of the machine tool (Zahand teaches that “a depth map generated by one or more of the depth cameras 44 may identify the location of the nip 402 in the 3D space of the workstation region 14” in para. [0028]. Here, the nip 402 and the robotic arm 340 (see para. [0030]) are interpreted as equivalent to the claimed emergency region and may also be classified as an alert region (emergency region) of the machine tool as shown in para. [0039]) and
an observation region where the body of the operator is located when operating the machine tool (Zahand teaches observing a region where the operator is located within the workspace region, and tracking their movements with respect to the machine tool(s) in para. [0018]. See also para. [0039]).
Regarding claim 17, Zahand and Denenberg teach the sensing device of claim 10, wherein the processor is further configured to
predict a movement of the operator according to a change in the tracked posture of the operator (Denenberg teaches “a spatial POE of the human operator that characterizes the spatial region potentially occupied by any portion of the human operator is based on any possible or anticipated movements of the human operator” as shown in para. [0079], wherein “the POE 602 of the human operator is refined by acquiring more information about the operator. For example, the sensor system 101 may acquire a series of scanning data (e.g., images) within a time interval Δt. By analyzing the operator's positions and poses in the scanning data and based on the time period Δt, the operator's moving direction, velocity and acceleration can be determined” as shown in para. [0080]); and
predict an emergency situation in the workspace according to the predicted movement (Denenberg teaches initiating an emergency stop in the workplace according to the predicted movement of both the machinery and a human operator as shown in para. [0102]. Here, this initiation is interpreted as equivalent to predicting an emergency situation, as the initiation is an emergency action carried out based on predicted movement of the operator).
Similar motivation as applied to claim 10 can be applied here to claim 17.
Claims 3 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Zahand (U.S. Publication No. 2015/0228078) in view of Denenberg et al. (U.S. Publication No. 2021/0379763), hereinafter Denenberg and Luo et al. (CN 110232320 A, see attached English translation), hereinafter Luo.
Regarding claim 3, Zahand and Denenberg teach the method of claim 2,
wherein setting the region of interest comprises setting, in a spatial information map generated based on the point cloud data with respect to the workspace (Dennenberg teaches a spatial map represented by a point cloud with respect to a workspace 100 in para. [0069]), and
the observation region determined according to body information based on a user profile of the operator (Zahand teaches that “biometric information of an operator may be used to identify the operator” as shown in para. [0042]).
Similar motivations as applied to claim 1 can be applied here to claim 3.
While Zahand teaches determining a hazardous area for specific types of machine tools in para. [0027] and [0028], Zahand and Denenberg fail to teach the emergency region that is preset according to a type of the machine tool.
However, Luo teaches the emergency region that is preset according to a type of the machine tool (Luo teaches that, “since the present invention considers the working state of the machine, and the YOLOv2 model can recognize the type of the machine, it is easy to perform compatible calculation with the setting of the dangerous area of the machine, that is, on the basis of the present invention, further consideration can be given to different types of machinery. The characteristics of the shape, working characteristics and other factors, the specific range of dynamic hazard areas of a specific type of machinery is specified, and the threshold of the dangerous distance is obtained” as shown in para. [0034]).
Zahand, Denenberg, and Luo are all considered to be analogous to the claimed invention because they are in the same field of analyzing workspace environments using 3D information to promote safety between operators and machinery in the workplace. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zahand (as modified by Denenberg) to incorporate the teachings of Luo and include “the emergency region that is preset according to a type of the machine tool”. The motivation for doing so would have been “so that the issued warning information is more scientific and reliable”, as suggested by Luo in para. [0034]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zahand and Denenberg with Luo to obtain the invention specified in claim 3.
Regarding claim 12, Zahand and Denenberg teach the sensing device of claim 11,
wherein the processor is further configured to set, in a spatial information map generated based on the point cloud data with respect to the workspace (Dennenberg teaches a spatial map represented by a point cloud with respect to a workspace 100 in para. [0069]), and
the observation region determined according to body information based on a user profile of the operator (Zahand teaches that “biometric information of an operator may be used to identify the operator” as shown in para. [0042]).
Similar motivations as applied to claim 1 can be applied here to claim 3.
While Zahand teaches determining a hazardous area for specific types of machine tools in para. [0027] and [0028], Zahand and Denenberg fail to teach the emergency region that is preset according to a type of the machine tool.
However, Luo teaches the emergency region that is preset according to a type of the machine tool (Luo teaches that, “since the present invention considers the working state of the machine, and the YOLOv2 model can recognize the type of the machine, it is easy to perform compatible calculation with the setting of the dangerous area of the machine, that is, on the basis of the present invention, further consideration can be given to different types of machinery. The characteristics of the shape, working characteristics and other factors, the specific range of dynamic hazard areas of a specific type of machinery is specified, and the threshold of the dangerous distance is obtained” as shown in para. [0034]).
Zahand, Denenberg, and Luo are all considered to be analogous to the claimed invention because they are in the same field of analyzing workspace environments using 3D information to promote safety between operators and machinery in the workplace. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zahand (as modified by Denenberg) to incorporate the teachings of Luo and include “the emergency region that is preset according to a type of the machine tool”. The motivation for doing so would have been “so that the issued warning information is more scientific and reliable”, as suggested by Luo in para. [0034]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zahand and Denenberg with Luo to obtain the invention specified in claim 12.
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Zahand (U.S. Publication No. 2015/0228078) in view of Denenberg et al. (U.S. Publication No. 2021/0379763), hereinafter Denenberg and Brooks et al. (U.S. Publication No. 2014/0067121 A1), hereinafter Brooks.
Regarding claim 4, Zahand and Denenberg teach the method of claim 2, wherein tracking the posture of the operator comprises:
determining whether point cloud data corresponding to the operator exists in the emergency region (Denenberg teaches determining whether point cloud data corresponding to an operator is a minimum separation distance a part from a piece of machinery in the workplace as shown in para. [0084]. See para. [0069], wherein the POE may be a point cloud) (Zahand additionally teaches “utilizing the depth image data 46 and corresponding series of movements of the operator 304, the line monitoring program 26 may determine that the operator 304 is within a predetermined distance of the nip 402” in para. [0028], wherein the nip 402 is interpreted as equivalent to the claimed emergency region) and
sub-observation regions distinguished by body parts of the operator (Zahand teaches “Virtual skeletons in accordance with the present disclosure may include virtually any number of joints, each of which can be associated with virtually any number of parameters including, but not limited to, three dimensional joint position, joint rotation, and body posture of a corresponding body part (e.g., arm extended, hand open, hand closed, etc.)” in para. [0023], wherein the hand [sub-observation region) may be specifically analyzed as shown in para. [0030]. In other situations, different limbs may be analyzed as well (see, for example, para. [0036] and [0034]).
While Zahand teaches determining the posture of the operator (see para. [0018]) and Denenberg teaches utilizing point clouds to visualize overlap between machinery and humans, Zahand and Denenberg fail to teach determining the posture of the operator based on a combination of regions in which the point cloud data corresponding to the operator exists among the emergency region and the sub-observation regions.
However, Brooks teaches determining the posture of the operator based on a combination of regions in which the point cloud data corresponding to the operator exists among the emergency region and the sub-observation regions (Brooks teaches that “by virtue of the sonar sensor arrangement and the resulting shape of the detection zone, discrimination between a person's head and upper body and his hands is achieved incidentally to detection” as shown in para. [0035] in order to determine the position of the person based on the overlap between a danger zone and sub-parts of the human’s body (sub-observation regions)).
Zahand, Denenberg, and Brooks are all considered to be analogous to the claimed invention because they are in the same field of analyzing workspace environments using 3D information to promote safety between operators and machinery in the workplace. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zahand (as modified by Denenberg) to incorporate the teachings of Brooks and include “determining the posture of the operator based on a combination of regions in which the point cloud data corresponding to the operator exists among the emergency region and the sub-observation regions”. The motivation for doing so would have been “to provide systems and methods for robot safety that are straightforwardly implemented while avoiding unnecessary interruptions and [declines] of robot operation”, as suggested by Brooks in para. [0005]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zahand and Denenberg with Brooks to obtain the invention specified in claim 4.
Regarding claim 13, Zahand and Denenberg teach the sensing device of claim 11, wherein the processor is further configured to
determine whether point cloud data corresponding to the operator exists in the emergency region (Denenberg teaches determining whether point cloud data corresponding to an operator is a minimum separation distance a part from a piece of machinery in the workplace as shown in para. [0084]. See para. [0069], wherein the POE may be a point cloud) (Zahand additionally teaches “utilizing the depth image data 46 and corresponding series of movements of the operator 304, the line monitoring program 26 may determine that the operator 304 is within a predetermined distance of the nip 402” in para. [0028], wherein the nip 402 is interpreted as equivalent to the claimed emergency region) and
sub-observation regions distinguished by body parts of the operator (Zahand teaches “Virtual skeletons in accordance with the present disclosure may include virtually any number of joints, each of which can be associated with virtually any number of parameters including, but not limited to, three dimensional joint position, joint rotation, and body posture of a corresponding body part (e.g., arm extended, hand open, hand closed, etc.)” in para. [0023], wherein the hand [sub-observation region) may be specifically analyzed as shown in para. [0030]. In other situations, different limbs may be analyzed as well (see, for example, para. [0036] and [0034]).
While Zahand teaches determining the posture of the operator (see para. [0018]) and Denenberg teaches utilizing point clouds to visualize overlap between machinery and humans, Zahand and Denenberg fail to teach determining the posture of the operator based on a combination of regions in which the point cloud data corresponding to the operator exists among the emergency region and the sub-observation regions.
However, Brooks teaches determining the posture of the operator based on a combination of regions in which the point cloud data corresponding to the operator exists among the emergency region and the sub-observation regions (Brooks teaches that “by virtue of the sonar sensor arrangement and the resulting shape of the detection zone, discrimination between a person's head and upper body and his hands is achieved incidentally to detection” as shown in para. [0035] in order to determine the position of the person based on the overlap between a danger zone and sub-parts of the human’s body (sub-observation regions)).
Zahand, Denenberg, and Brooks are all considered to be analogous to the claimed invention because they are in the same field of analyzing workspace environments using 3D information to promote safety between operators and machinery in the workplace. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zahand (as modified by Denenberg) to incorporate the teachings of Brooks and include “determining the posture of the operator based on a combination of regions in which the point cloud data corresponding to the operator exists among the emergency region and the sub-observation regions”. The motivation for doing so would have been “to provide systems and methods for robot safety that are straightforwardly implemented while avoiding unnecessary interruptions and [declines] of robot operation”, as suggested by Brooks in para. [0005]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zahand and Denenberg with Brooks to obtain the invention specified in claim 13.
Claims 5 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Zahand (U.S. Publication No. 2015/0228078) in view of Denenberg et al. (U.S. Publication No. 2021/0379763), hereinafter Denenberg and Ren et al. (CN 112749671 A, see attached English translation for citations), hereinafter Ren.
Regarding claim 5, Zahand and Denenberg teach the method of claim 2, wherein tracking the posture of the operator comprises:
determining the posture of the operator based on body parts of the operator classified according to joint positions of the skeleton information (Zahand teaches “Virtual skeletons in accordance with the present disclosure may include virtually any number of joints, each of which can be associated with virtually any number of parameters including, but not limited to, three dimensional joint position, joint rotation, and body posture of a corresponding body part (e.g., arm extended, hand open, hand closed, etc.)” in para. [0023]. See also para. [0036]) and the emergency region (Zahand teaches determining that the posture of the operator is inappropriate based on the hazard region as shown in para. [0027]-[0028] and [0030]).
Zahand and Denenberg fail to teach obtaining skeleton information from point cloud data corresponding to the operator by using a machine learning-based neural network model.
However, Ren teaches obtaining skeleton information from point cloud data corresponding to the operator by using a machine learning-based neural network model (Ren teaches obtaining depth information, “converting the depth map sequence into three-dimensional point cloud”, and “extracting the three-dimensional convolutional neural network characteristic diagram from the motion trajectory to obtain a space-time characteristic diagram, and generating a corresponding heat map according to the position information of the skeleton point” as shown in para. [0028] and [0038]. Here, the point cloud data is used to generate skeletal information using a three-dimensional convolutional neural network characteristic diagram which is interpreted as equivalent to the machine learning-based neural network model).
Zahand, Denenberg, and Ren are all considered to be analogous to the claimed invention because they are in the same field of analyzing human behavior. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zahand (as modified by Denenberg) to incorporate the teachings of Ren and include “obtaining skeleton information from point cloud data corresponding to the operator by using a machine learning-based neural network model”. The motivation for doing so would have been “so that the behavior recognition effect is effectively improved, meanwhile, the important characteristics can be extracted, and the human body behavior recognition has better discrimination”, as suggested by Ren in para. [0034]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zahand and Denenberg with Ren to obtain the invention specified in claim 5.
Regarding claim 14, Zahand and Denenberg teach the sensing device of claim 11, wherein the processor is further configured to
determine a posture of the operator based on body parts of the operator classified according to joint positions of the skeleton information (Zahand teaches “Virtual skeletons in accordance with the present disclosure may include virtually any number of joints, each of which can be associated with virtually any number of parameters including, but not limited to, three dimensional joint position, joint rotation, and body posture of a corresponding body part (e.g., arm extended, hand open, hand closed, etc.)” in para. [0023]. See also para. [0036]) and the emergency region (Zahand teaches determining that the posture of the operator is inappropriate based on the hazard region as shown in para. [0027]-[0028] and [0030]).
Zahand and Denenberg fail to teach obtaining, by using a machine learning-based neural network model, skeleton information from point cloud data corresponding to the operator.
However, Ren teaches obtaining, by using a machine learning-based neural network model, skeleton information from point cloud data corresponding to the operator (Ren teaches obtaining depth information, “converting the depth map sequence into three-dimensional point cloud”, and “extracting the three-dimensional convolutional neural network characteristic diagram from the motion trajectory to obtain a space-time characteristic diagram, and generating a corresponding heat map according to the position information of the skeleton point” as shown in para. [0028] and [0038]. Here, the point cloud data is used to generate skeletal information using a three-dimensional convolutional neural network characteristic diagram which is interpreted as equivalent to the machine learning-based neural network model).
Zahand, Denenberg, and Ren are all considered to be analogous to the claimed invention because they are in the same field of analyzing human behavior. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zahand (as modified by Denenberg) to incorporate the teachings of Ren and include “obtaining, by using a machine learning-based neural network model, skeleton information from point cloud data corresponding to the operator”. The motivation for doing so would have been “so that the behavior recognition effect is effectively improved, meanwhile, the important characteristics can be extracted, and the human body behavior recognition has better discrimination”, as suggested by Ren in para. [0034]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zahand and Denenberg with Ren to obtain the invention specified in claim 14.
Claims 6, 7, 15, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Zahand (U.S. Publication No. 2015/0228078) in view of Denenberg et al. (U.S. Publication No. 2021/0379763), hereinafter Denenberg and Kim et al. (KR 10-2275723 B1, see attached English translation for citations), hereinafter Kim.
Regarding claim 6, Zahand and Denenberg teach the method of claim 1, wherein detecting the emergency situation in the workspace comprises:
determining a level of the emergency (Zahand teaches “a hierarchy of alerts 72 may be utilized. Each of the alerts may be associated with one or more operator activities that may be classified according to a risk level to an operator. As shown in FIG. 5, two or more operator activities may be classified as presenting a similar risk level, and accordingly may be paired with the same alert. Further, as the risk level of the operator activity increases, the magnitude of the corresponding alert similarly increases” as shown in para. [0031], wherein the magnitude of the alert is interpreted as equivalent to the claimed level of the emergency situation. Additionally, the magnitude may be determined in part based on the distance between the operator’s hand and the alert region(s) as shown in para. [0032], as, inherently, the closer the operator’s hand is to dangerous machinery, the greater the risk posed to the operator/machinery. See also FIG. 5); and
determining whether the emergency situation occurred, according to the determined emergency situation level (Zahand teaches determining that an emergency situation eliciting a specific response has occurred based on an emergency situation magnitude as shown in para. [0031]-[0032] and FIG. 5).
While Zahand teaches tracking whether the operator comes within a predetermined distance of a hazard and Denenberg teaches a degree to which a volume of the operator corresponding to the [predicted] posture of the operator and a [predicted] volume of the machine tool overlap with each other in para. [0085], Zahand and Denenberg fail to teach a degree to which a volume of the operator corresponding to the tracked posture of the operator and a volume of the machine tool overlap with each other.
However, Kim specifically teaches a degree to which a volume of the operator corresponding to the tracked posture of the operator and a volume of the machine tool overlap with each other and determining whether the emergency situation occurred, according to the determined emergency situation level (Kim teaches “the safety accident risk notification device 200 calculates an overlap rate using the worker position determined in step S302 and the common area width of the work area determined in step S306 (S308). When the calculated overlap rate has a value greater than or equal to a preset threshold, the safety accident risk notification device 200 uses an alerting module to inform that the operator is located in the danger zone, and induces movement outside the danger zone do (S310)” in para. [0044]. Here, the overlap rate (also defined as a degree in para. [0024]) is interpreted as equivalent to the claimed degree. Kim additionally teaches determining whether a safety accident risk notification (emergency situation) need be output, based on the degree of overlap).
Zahand, Denenberg, and Kim are all considered to be analogous to the claimed invention because they are in the same field of analyzing human behavior. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zahand (as modified by Denenberg) to incorporate the teachings of Kim and include “a degree to which a volume of the operator corresponding to the tracked posture of the operator and a volume of the machine tool overlap with each other”. The motivation for doing so would have been that “it is possible to accurately calculate the working area by tracking the position of the continuously changing container in the course of the work, and has the effect of preventing safety accidents in the port”, as suggested by Kim in para. [0064]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zahand and Denenberg with Kim to obtain the invention specified in claim 6.
Regarding claim 7, Zahand, Denenberg, and Kim teach the method of claim 6,
further comprising notifying a server providing an emergency situation management service about the detected emergency situation to (Kim teaches “the communication unit 202 may transmit/receive data related to safety accident records in a port with an external data storage such as a cloud data server” in para. [0036]) (Zahand further teaches notifying a manager (emergency situation management service) about the detected emergency situation as shown in FIG. 5 and para. [0032]), wherein the notifying comprises notifying the server that a potential emergency situation is detected in response to the determined level of emergency situation being greater than or equal to a first reference value (Zahand teaches notifying the user through a flashing beacon 342 that a potential emergency situation is detected in response to the operator’s hand reaching 0.5 meters from the emergency region in para. [0030]. Here, the risk level of the above situation qualifies for the event to be classified as alert level 1 (first reference value)) (While Zahand and Denenberg both teach detecting a potential emergency situation (see Denenberg, para. [0083]-[0085]), Kim specifically teaches notifying the server about emergency situations. The teachings of Zahand and Denenberg can be combined with the teachings of Kim to teach the above limitation, as it would have been obvious to combine Zahand’s teaching of notifying a user of a potential emergency situation in the workplace with Kim’s teaching of specifically notifying a server of an emergency situation in the workplace), and notifying the server that an emergency situation has occurred in response to the determined level of emergency situation being greater than a second reference value greater than the first reference value (Zahand teaches notifying the user through a flashing beacon 342 and notifying the manager that an emergency situation is detected in response to the operator’s hand entering the nip (emergency region) in para. [0032]. Here, the risk level of the above situation is high enough to be classified as a level 4 alert value (second reference value), which is inherently higher than the risk level of the potential emergency situation (first reference value) wherein the alert value is classified as alert 1) (While Zahand teaches detecting an emergency situation, Kim specifically teaches notifying the server about emergency situations. The teachings of Zahand and Denenberg can be combined with the teachings of Kim to teach the above limitation, as it would have been obvious to combine Zahand’s teaching of notifying a manager/user of an emergency situation in the workplace with Kim’s teaching of specifically notifying a server of an emergency situation in the workplace).
The motivation for doing so would have been to “transmit/receive data related to safety accident records in a port with an external data storage such as a cloud data server”, as suggested by Kim in para. [0036]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zahand and Denenberg with Kim to obtain the invention specified in claim 7.
Regarding claim 15, Zahand and Denenberg teach the sensing device of claim 10, wherein the processor is further configured to
determine a level of the emergency (Zahand teaches “a hierarchy of alerts 72 may be utilized. Each of the alerts may be associated with one or more operator activities that may be classified according to a risk level to an operator. As shown in FIG. 5, two or more operator activities may be classified as presenting a similar risk level, and accordingly may be paired with the same alert. Further, as the risk level of the operator activity increases, the magnitude of the corresponding alert similarly increases” as shown in para. [0031], wherein the magnitude of the alert is interpreted as equivalent to the claimed level of the emergency situation. Additionally, the magnitude may be determined in part based on the distance between the operator’s hand and the alert region(s) as shown in para. [0032], as, inherently, the closer the operator’s hand is to dangerous machinery, the greater the risk posed to the operator/machinery. See also FIG. 5); and
determine whether the emergency situation occurred, according to the determined emergency situation level (Zahand teaches determining that an emergency situation eliciting a specific response has occurred based on an emergency situation magnitude as shown in para. [0031]-[0032] and FIG. 5).
While Zahand teaches tracking whether the operator comes within a predetermined distance of a hazard and Denenberg teaches a degree to which a volume of the operator corresponding to the [predicted] posture of the operator and a [predicted] volume of the machine tool overlap with each other in para. [0085], Zahand and Denenberg fail to teach a degree to which a volume of the operator corresponding to the tracked posture of the operator and a volume of the machine tool overlap with each other.
However, Kim specifically teaches a degree to which a volume of the operator corresponding to the tracked posture of the operator and a volume of the machine tool overlap with each other and determining whether the emergency situation occurred, according to the determined emergency situation level (Kim teaches “the safety accident risk notification device 200 calculates an overlap rate using the worker position determined in step S302 and the common area width of the work area determined in step S306 (S308). When the calculated overlap rate has a value greater than or equal to a preset threshold, the safety accident risk notification device 200 uses an alerting module to inform that the operator is located in the danger zone, and induces movement outside the danger zone do (S310)” in para. [0044]. Here, the overlap rate (also defined as a degree in para. [0024]) is interpreted as equivalent to the claimed degree. Kim additionally teaches determining whether a safety accident risk notification (emergency situation) need be output, based on the degree of overlap).
Zahand, Denenberg, and Kim are all considered to be analogous to the claimed invention because they are in the same field of analyzing human behavior. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zahand (as modified by Denenberg) to incorporate the teachings of Kim and include “a degree to which a volume of the operator corresponding to the tracked posture of the operator and a volume of the machine tool overlap with each other”. The motivation for doing so would have been that “it is possible to accurately calculate the working area by tracking the position of the continuously changing container in the course of the work, and has the effect of preventing safety accidents in the port”, as suggested by Kim in para. [0064]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zahand and Denenberg with Kim to obtain the invention specified in claim 15.
Regarding claim 16, Zahand, Denenberg, and Kim teach the sensing device of claim 15, further comprising a communication interface configured to notify a server providing an emergency situation management service about the detected emergency situation to (Kim teaches “the communication unit 202 may transmit/receive data related to safety accident records in a port with an external data storage such as a cloud data server” in para. [0036]) (Zahand further teaches notifying a manager (emergency situation management service) through an interface about the detected emergency situation as shown in FIG. 5 and para. [0032]), wherein the processor is further configured to notify the server, through the communication interface, that a potential emergency situation is detected in response to the determined level of emergency situation being greater than a first reference value (Zahand teaches notifying the user through a flashing beacon 342 that a potential emergency situation is detected in response to the operator’s hand reaching 0.5 meters from the emergency region in para. [0030]. Here, the risk level of the above situation qualifies for the event to be classified as alert level 1 (first reference value)) (While Zahand and Denenberg both teach detecting a potential emergency situation (see Denenberg, para. [0083]-[0085]), Kim specifically teaches notifying the server about emergency situations. The teachings of Zahand and Denenberg can be combined with the teachings of Kim to teach the above limitation, as it would have been obvious to combine Zahand’s teaching of notifying a user of a potential emergency situation in the workplace with Kim’s teaching of specifically notifying a server of an emergency situation in the workplace), and notify the server that an emergency situation has occurred in response to the determined level of emergency situation being greater than a second reference value greater than the first reference value (Zahand teaches notifying the user through a flashing beacon 342 and notifying the manager that an emergency situation is detected in response to the operator’s hand entering the nip (emergency region) in para. [0032]. Here, the risk level of the above situation is high enough to be classified as a level 4 alert value (second reference value), which is inherently higher than the risk level of the potential emergency situation (first reference value) wherein the alert value is classified as alert 1) (While Zahand teaches detecting an emergency situation, Kim specifically teaches notifying the server about emergency situations. The teachings of Zahand and Denenberg can be combined with the teachings of Kim to teach the above limitation, as it would have been obvious to combine Zahand’s teaching of notifying a manager/user of an emergency situation in the workplace with Kim’s teaching of specifically notifying a server of an emergency situation in the workplace).
The motivation for doing so would have been to “transmit/receive data related to safety accident records in a port with an external data storage such as a cloud data server”, as suggested by Kim in para. [0036]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zahand and Denenberg with Kim to obtain the invention specified in claim 16.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to KYLA G ALLEN whose telephone number is (703)756-5315. The examiner can
normally be reached M-F 7:30am - 4:30pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use
the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
John Villecco can be reached on (571) 272-7319. The fax phone number for the organization where this
application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users. To
file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or
571-272-1000.
/Kyla Guan-Ping Tiao Allen/
Examiner, Art Unit 2661
/XUEMEI G CHEN/Primary Examiner, Art Unit 2661
Prosecution Insights