Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This communication is a first office action, non-final rejection on the merits. Claims 1-10 as filed, are currently pending and have been considered below.
Specification
The disclosure is objected to because of the following informalities:
0056: Inconsistencies with the element number of the robot. Figure 2 and multiple paragraphs (starting on 0045) identify the robot as element 100 and the camera as element 200. In 0056 the robot is identified as element 200. Suggest changing “the optimal path of the robot (200)” to “the optimal path of the robot (100)”.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories (process, machine, manufacture, or composition of matter) of patent eligible subject matter because the claim is directed as a “program” that is reasonably interpreted as a software, and software claims are not patentable. Based on the provided claim language, the claimed “program” is not supported by hardware such as tangible computer storage medium or execution engine, which would enable one skill in the art to construe that the apparatus is built from tangible product to carry out any functionality being conveyed from the claim. Thus, the “program” is software per se and therefore is not being embodied in a manner as to be executable. Therefore, Applicant is suggested to amend claim 10 to recite for example, A computer program product for ……, the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code comprising the programming instructions for…..”.
Accordingly, appropriated correction and/or clarification is requested. Claim 10 will be examined in light of prior art as if the suggested corrections have been made.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1 and 10 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Khawaja et al. (Khawaja, F.I.; Kanazawa, A.; Kinugawa, J.; Kosuge, K. A Human-Following Motion Planning and Control Scheme for Collaborative Robots Based on Human Motion Prediction. Sensors 2021, 21, 8229. hereinafter “Khawaja”).
Regarding Claim 1, Khawaja discloses
A skeleton-based industrial site monitoring method comprising:
generating skeleton data, by a control server, by recognizing joint points of a worker based on a captured video of the industrial site where a robot is present;
Khawaja pertains to a method of controlling an industrial robot based on motion tracking of a worker and details “These terms are calculated from the worker’s skeletal data observed by the 3-D vision sensor in real-time” (Fig 1, pg. 4). As described in the background provided by Khawaja, for those of ordinary skill in the art of motion tracking, generating skeletal data from a 3-D vision sensor is comparable to generating skeletal data from video captured from a camera, specifically stating “One of the first human-following approaches was proposed by Nagumo et al., in which an LED device carried by the human was detected and tracked by the robot using a camera [20]” (pg 3). Khawaja further discloses
estimating, by the control server, an expected movement path of the worker based on the generated skeleton data;
in the section entitled “3.3 Worker’s Motion Prediction” beginning on page 6. Khawaja further discloses in figure 1
searching, by the control server, for an optimal path for the robot to avoid collision with the worker based on the estimated expected movement path of the worker; and
further specifying, “This scheme plans the collision-free robot trajectory that follows the moving worker efficiently under the safety cost constraint and the robot’s velocity and acceleration constraints” (pg 8). Khawaja further discloses in figure 6
controlling, by the control server, the robot according to the searched optimal path
further specifying, “The worker moves to the working position for Task 2 and the robot follows him. The worker returns the bolt tightening tool to the tool holder (Figure 6c) and picks up three grommets from the parts tray” (pg 10).
Regarding Claim 10, Khawaja discloses
A computer program recorded on a recording medium, combined with a computing device comprising:
a memory;
a transceiver; and
a processor that processes instructions stored in the memory, wherein the processor is configured to:
Khawaja specifies “The proposed scheme was installed in a computer with an Intel Core i7-3740QM (Quadcore processor, 2.7 GHz) with 16GB memory. All calculations were done within 30 ms, the sampling interval of the sensing system that tracks the position of the human worker” (pg 10). Additionally in figures 4 and 5 Khawaja discloses the use of a Microsoft Kinect™ sensing system, which is known to those of ordinary skill in the art of motion capture to include a transceiver.
The remainder of claim 10 is rejected in a similar manner as claim 1 above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2-4, and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Khawaja in view of Kanazawa et al (Kanazawa, A., Kinugawa, J., Kosuge, K. “Adaptive Motion Planning for a Collaborative Robot Based on Prediction Uncertainty to Enhance Human Safety and Work Efficiency”. IEEE Transactions on Robotics, vol. 35, no. 4, August 2019, 817-832 hereinafter “Kanazawa”).
Regarding Claim 2, Khawaja discloses all of the limitations of claim 1, and Kanazawa further discloses in part III section B. Predicting the Worker Trajectory beginning on page 820
estimating movement direction of the worker and movement speed of the worker based on changes in positions of body feature points based on the generated skeleton data; and
estimating the expected movement path based on the estimated movement direction of the worker and the movement speed of the worker.
Kanazawa, an earlier publication by the same research group as Khawaja, pertains to the details of the worker trajectory prediction not included in the Khawaja publication. Kanazawa further specifies “the worker’s motion trajectory is modeled by GMR. When using human’s velocity or acceleration information for modeling, the precise prediction becomes difficult due to the effect of the sensor noise and the error of the difference calculus. In the proposed algorithm, we choose the autoregression model which uses the worker’s position history to consider the worker’s temporal behavior and improve the prediction performance as reported in related work [22]” (820). Therefore, it would have been known to one of ordinary skill in the art to use the details of the worker trajectory prediction provided by Kanazawa for the control of an industrial robot of Khawaja according to the known methods to yield predictable results. Please see MPEP 2143 for further support of this rationale.
Regarding Claim 3, Khawaja in view of Kanazawa discloses all of the limitations of claim 2, and Kanazawa further discloses
estimating a type of task based on the estimated expected movement path; and
modifying the estimated expected movement path based on the estimated type of task.
Kanazawa employs an adaptive learning method to distinguish between multiple worker tasks (equations 1-3) before predicting the worker’s trajectory (equations 4-13), specifying “Using the current worker position x(t)worker and the constructed GMM, the current task i is estimated. At the same time, the worker positions when the worker’s velocity is less than a predefined value vth are extracted and stored as sample data” (pg 820).
Regarding Claim 4, Khawaja in view of Kanazawa discloses all of the limitations of claim 3, and Kanazawa further discloses in figures 6 and 9
matching tracked movement of the worker (x(t)worker) with the estimated expected movement path (N(t+Tp)) in real time by tracking the movement of the worker; and
Specifically in figure 9, the current worker position is represented by the purple dots and the predicted worker trajectory is represented with the red line. Kanazawa further discloses
determining the movement of the worker as abnormal in case that distance between the movement of the worker and the expected movement path exceeds a preset value.
Kanazawa predicts the motion of irregular worker patterns, using a variance as a preset value. When this variance is set to a constant, the system does a poor job of predicting abnormal movements resulting in the robot getting close to the worker. When this variance is allowed to be dynamic, the system does a better job of predicting abnormal movements and the robot maintains a safe distance from the worker. This is specified as “Fig. 18 shows the results for the distance of the closest approach when each worker moves in an irregular pattern. The simulation results of the trajectory considering the constant variance confirm that the robot approached within 0.3 m of the worker in many cases. On the other hand, the results of the trajectory considering the predicted variance confirm that the robot remains about 0.3 m from the worker” (828).
Regarding Claim 8, Khawaja discloses all of the limitations of claim 1, and Khawaja further discloses in figure 1
tracking the movement of the worker based on the skeleton data;
The use of skeletal data to track the movement of the worker was addressed in the rejection of claim 1 above. Khawaja is silent on the details of the worker path prediction, providing those details in their earlier work Kanazawa. Kanazawa discloses
estimating a risk level based on the difference between the expected movement path and the movement of the worker; and
Kanazawa pertains to an industrial robot control method and refers the risk between the movement path of the robot and the movement of a worker as collision risk, stating “By planning the robot trajectory in the expanded temporal space that includes future worker states and its uncertainty, the proposed method explicitly incorporates temporal requirements and collision risk into a trajectory optimization problem” (pg 818). The estimated risk level is incorporated into equation 26, specifically, “The term L2(q(k)) serves to avoid a collision with the worker based on the distribution of the predicted worker position. w is a weighting coefficient of this cost function. Using the Mahalanobis distance between the distributions of the predicted worker position N(μ(k)worker,Σ(k)worker) at step k and each joint position Kj(θ(k)), an artificial potential field is constituted as shown in Fig. 3. Using the Mahalanobis distance, the artificial potential is wider in the direction of larger variance in the predicted position. Therefore, in this direction of high uncertainty in the worker position, it is possible to widely avoid the worker and decrease the risk of the collision” (823). Kanazawa further discloses in figure 6
resetting the expected movement path in case that the estimated risk level exceeds a preset value.
The prediction method disclosed by Kanazawa is an iterative process, with the prediction at time t being reset by the prediction at time t+1, this is true of worker path N(t+Tp) and robot trajectory q(t+T0). The specific preset value of risk level is incorporated in the term L2(q(k)) of equation 26 as detailed above. Therefore, it would have been known to one of ordinary skill in the art to use the details of the collision avoidance method provided by Kanazawa for the control of an industrial robot of Khawaja according to the known methods to yield predictable results. Please see MPEP 2143 for further support of this rationale.
Claims 5-7 are rejected under 35 U.S.C. 103 as being unpatentable over Khawaja in view of Kanazawa, further in view of Lim et al (US patent application 20190347803, hereinafter “Lim”).
Regarding Claim 5, Khawaja in view of Kanazawa discloses all of the limitations of claim 4, but fails to teach the segmentation of the video. However, Lim discloses
dividing the captured video into a dynamic region set based on the expected movement path and a static region corresponding to remaining area in the captured video excluding the dynamic region, and
tracking the movement of the worker by analyzing the dynamic region.
Lim pertains to the identification of workers in a video feed, their skeletonization, and the division of the video feed into two regions (fore/background). Lim does not specifically predict the workers path, but does predict worker motion via known techniques “Techniques for motion estimation and segmentation include, but are not limited to optical flow techniques, block-based matching techniques, Maximum A posteriori Probability (MAP) techniques, and/or simultaneous motion estimation and segmentation technique” (0044). Lim details image segmentation in figure 1, detailing “determine when a subject has come into proximity to the video capture system 200, and/or identify discontinuities in a depth image and related depth image data used to perform image segmentation for a subject” (0032). Additionally, Lim further describes the background as static “In some examples, the background image 352 may be a static image” (0049). Therefore, it would have been known to one of ordinary skill in the art of motion capture to use the worker path projections of Kanazawa to fully predict worker motion when employing the image segmentation process of Lim to reduce the computational load when performing industrial robot control.
Regarding Claim 6, Khawaja in view of Kanazawa and Lim discloses all of the limitations of claim 5, and Lim further discloses in figures 3 and 8A-D
generating a virtual worker (327) of a preset size based on the skeleton data (832); and
determining the dynamic region (303) based on movement of the generated virtual worker (334) along the .
As detailed in the rejection of claim 5 above, Lim predicts general worker motion, not expected paths. As detailed in the rejections of claims 2-4, Kanazawa predicts the worker’s expected path. Therefore, it would have been known to one of ordinary skill in the art of motion capture to use the worker path projections of Kanazawa to fully predict worker motion when employing the image segmentation via skeletonization process of Lim to reduce the computational load when performing industrial robot control.
Regarding Claim 7, Khawaja in view of Kanazawa and Lim discloses all of the limitations of claim 6, and Lim further discloses in figure 3 wherein the controlling comprises: performing masking on the static region (302).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Khawaja in view of Kanazawa, further in view of Imazawa et al (US patent application 20160253618, hereinafter “Imazawa”).
Regarding Claim 9, Khawaja in view of Kanazawa discloses all of the limitations of claim 8, and Kanazawa teaches in figure 1
estimating a risk level based on the difference between the reset expected movement path and the movement of the worker; and
As covered in the rejection of claim 8 above, the prediction method disclosed by Kanazawa is an iterative process, with the prediction at time t being reset by the prediction at time t+1, this is true of worker path N(t+Tp), robot trajectory q(t+T0), and risk estimate L2(q(k)) . Kanazawa does not teach the use of a warning signal. However, Imazawa teaches in figure 14
displaying a warning signal in case that the difference between the risk level corresponding to the reset expected movement path and the risk level corresponding to the expected movement path before resetting is less than a preset value.
Imazawa pertains to a method of industrial worker path prediction for quality control and discloses the use of a warning signal to alert when a worker’s path deviates and becomes an unusual path, including a warning threshold value (0074). Therefore, it would have been known to one of ordinary skill in the art of industrial robot control to apply the warning signal of Imazawa to the collision risk level of Kanazawa to improve worker safety.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Nathan Daniel Neckel whose telephone number is (571)272-9537. The examiner can normally be reached M-F, 7-3.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wade Miles can be reached at 571-270-7777. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NATHAN DANIEL NECKEL/Examiner, Art Unit 3656
/WADE MILES/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights