DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
1. Applicant’s arguments filed 8-21-26 have been fully considered but are moot in view of the new ground(s) of rejection(s).
Claim Rejections - 35 USC § 103
2. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
3. Claims 21-22, 25, and 41 are rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan et al., US 10,981,272 in view of Mathew et al., US 2015/0347908, and in further view of Palumbo et al, US 2019/0182272.
Regarding claim 21, Nagarajan teaches of an apparatus (See Fig.1, 120; Fig.2, 270) comprising:
interface circuitry (See col.22-23);
computer readable instructions (See col.22-23); and
at least one processor circuit to be programmed based on the instructions (See col.22-23) to:
cause a first machine learning model to be deployed to a first device, the first machine learning model trained with first training data to detect at least one event depicted in first video segments (See Figs.1-3; col.1-4; col.6-8; col.11-13; and col.17-19);
second training data (See Figs.1-3; col.1-4; col.6-8; col.11-13; and col.17-19), iteratively retrain the first machine learning model based on respectively different portions of the second training data (See Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19 produces the second output data/results of a second video and the updated model is based on the updated reported data through updated the model through an iterative process, the second training data is of different segments/images/sequences) to obtain a second machine learning model, the second training data based on output data from the first device, the output data associated with execution of the first machine learning model by the first device (See Figs.1-3; col.1-4; col.6-8; col.11-13; and col.17-19); and
cause the second machine learning model to be deployed to a second device different from the first device (See Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19).
Nagarajan is silent with respect to where retraining occurs after availability of at least a threshold amount of data.
However, in the same field of endeavor, Mathew teaches of where retraining occurs after availability of at least a threshold amount of data (See [0024]).
It would have been obvious to one of ordinary skill in the art before the time effective filing date of the claimed invention to have modified the teachings of Nagarajan to have incorporated the teachings of Mathew for the mere benefit of more efficient use of bandwidth and resources.
The combination of Nagarajan and Mathew is silent with respect to the training/retraining until at least one of a threshold rate of false event detections is achieved or a threshold rate of missed event detections is achieved.
However, in the same field of endeavor, Palumbo teaches of the training/retraining until at least one of a threshold rate of false event detections is achieved or a threshold rate of missed event detections is achieved (See [0015], [0021] and [0074]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Nagarajan and Mathew to have incorporated the teachings of Palumbo for the mere benefit of ensuring more accurate data.
Regarding claim 22, the combination teaches the apparatus of claim 21, wherein the output data is from a plurality of devices that respectively executed the first machine learning model, the plurality of devices including the first device (See Nagarajan, Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19 output data is from the robots/devices that executed the grasp model).
Regarding claim 25, the combination teaches the apparatus of claim 21, wherein one or more of the at least one processor circuit is to train the first machine learning model to output values representative of respective likelihoods that corresponding events are depicted in an input video segment (See Nagarajan, Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19).
Regarding claim 41, the combination teaches the apparatus of claim 21, wherein one or more of the at least one processor circuit is to at least one of:
determine whether the threshold rate of the false event detections is achieved based on first reference video segments of the second training data for which the first machine learning model during retraining, incorrectly inferred the at least one event was present; or
determine whether the threshold rate of missed event detections is achieved based on second reference video segments of the second training data for which the first machine learning model during retraining incorrectly inferred the at least one event was not present (See Nagarajan, Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19 produces the second output data/results of a second video and the updated model is based on the updated reported data through updated the model through an iterative process, the second training data is of different segments/images/sequences of retrained data in an iterative training process; Palumbo, See [0015], [0021] and [0074] threshold rates achieved for error/false event or missed event for security breaches).
4. Claims 26-27 are rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan et al., US 10,981,272 in view of Mathew et al., US 2015/0347908, in further view of Palumbo et al, US 2019/0182272, and in further view of Tusch, US 2018/0018508.
Regarding claim 26, the combination of Nagarajan, Mathew, and Palumbo teaches the apparatus of claim 21.
The combination silent with respect to wherein the least one event corresponds to arrival of a package.
However, in the same field of endeavor, Tusch teaches of the at least one event corresponding to arrival of a package (See [0961] table 3).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combined teachings of Nagarajan, Mathew, and Palumbo to have incorporated the teachings of Tusch for the mere benefit being able to recognize and classify different types of objects/events for different types of applications.
Regarding claim 27, the combination teaches the apparatus of claim 21, further wherein the least one event corresponds to a presence of an individual (See Tusch, Figs. 4-14; [0961 tables 3-5).
5. Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan et al., US 10,981,272 in view of Mathew et al., US 2015/0347908, in further view of Palumbo et al, US 2019/0182272, and in further view of Cao, US 2019/0348152.
Regarding claim 23, the combination of Nagarajan, Mathew, and Palumbo teaches the apparatus of claim 21, wherein one or more of the at least one processor circuit is to retrain the first machine learning model (See Nagarajan, analysis of claim 21; Mathew, [0024]) after (i) the availability of at least the threshold amount of second training data (See Mathew, [0024]).
The combination is silent with respect to updating after an (ii) expiration of a periodic interval.
However, in the same field of endeavor, Cao teaches of updating after an (ii) expiration of a periodic interval (See [0090]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Nagarajan, Mathew, and Palumbo to have incorporated the teachings of Cao for the mere benefit of having a more efficient way of updating the model with respect to the training data.
6. Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan et al., US 10,981,272 in view of Mathew et al., US 2015/0347908, in further view of Palumbo et al, US 2019/0182272, in further view of Cao, US 2019/0348152, and in further view of Kasaragod et al., US 2019/0036716.
Regarding claim 24, the combination of Nagarajan, Mathew, Palumbo, and Cao teaches the apparatus of claim 21, wherein one or more of the at least one processor circuit is to retrain the first machine learning model after (i) the availability of at least the threshold amount of second training data, (ii) expiration of a periodic interval (See analysis of claim 23.
The combination is silent with respect to the updating after (iii) receipt of a user input.
However, in the same field of endeavor, Kasaragod teaches of updating after (iii) receipt of a user input (See [0126]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Nagarajan, Mathew, Palumbo, and Cao to have incorporated the teachings of Kasaragod for the mere benefit of having a more efficient way of updating the model with respect to the training data.
7. Claims 28 and 32-33 are rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan et al., US 10,981,272 in view of JP 2017522066 (English translation).
Regarding claim 28, Nagarajan teaches of an apparatus (See Figs.1-3, robot) comprising:
interface circuitry to download a machine learning model (See Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19 downloading of the model);
computer readable instructions (See Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19); and
at least one processor circuit to be programmed based on the instructions (See Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19) to:
execute the machine learning model on a first input video segment from a video feed to produce first output data corresponding to the first input video segment, the first input video segment obtained from the video feed based on an image sensor, the image sensor to capture the video feed, the machine learning model trained based on first training data to detect at least one event depicted in training video segments (See Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19 robot executes the model according to the image sensor and object detection event in a location for grasping based on trained segments and data);
cause the interface circuitry to report second training data, the second training data based on the first input video segment, the first output data and a label specifying whether the first input video segment depicts the at least one event (See Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19 robots report and send updated data/second training data which is based on the first output and of labeling/annotated/verifying the event); and
execute a retrained instance of the machine learning model on a second input video segment from the video feed to produce second output data corresponding to the second input video segment, the second input video segment obtained from the video feed based on the image sensor, the retrained instance of the machine learning model based on the second training data (See Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19 the robot executes the updated/retrained model which produces the second output data/results of a second video and the updated model is based on the updated reported data through updated the model through an iterative process).
Nagarajan is silent with respect to the feeds based on a sweep rate of an image sensor.
However, in the same field of endeavor, JP 2017522066 teaches of the feeds based on a sweep rate of an image sensor (See Pages 40-43).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Nagarajan to have incorporated the teachings of JP 2017522066 for the mere benefit of accounting for imaging for different depths.
Regarding claim 32, the combination teaches the apparatus of claim 28, wherein the label specifying whether the first input video segment depicts the at least one event is based on a user input (See Nagarajan, Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19 annotating and labeling whether the event as a successful grasp by a user).
Regarding claim 33, the combination teaches the apparatus of claim 32, wherein one or more of the at least one processor circuit is to: cause the first input video segment to be presented on a display; and generate the label based on the user input (See Nagarajan, Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19 viewer/human views the recording, thereby having being displayed and presented and annotating labeling by the user).
8. Claim 31 is rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan et al., US 10,981,272 in view of JP 2017522066, and in view of Amato et al., US 2019/0065901.
Regarding claim 31, Nagarajan in view of JP 2017522066 teaches the apparatus of claim 28 wherein the one or more of the at least one processor circuit is to segment the video feed based on the sweep rate of the image sensor to obtain the first input video segment and the second input video segment (See analysis of claim 21 and claim 28 with respect to Nagarajan and JP 2017522066).
Nagarajan in view of JP 2017522066 is silent with respect to also being based on a capture rate.
However, in the same field of endeavor, Amato teaches of wherein the characteristic includes a capture rate of the image sensor (See [0055]).
It would have been obvious to one of ordinary skill in the art before effective filing date of the claimed invention to have modified the teachings of Nagarajan and JP 2017522066 to have incorporated the teachings of Amato for the mere benefit of providing a more in depth training model.
9. Claims 34 are rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan et al., US 10,981,272 in view of Fridental et al., US 2018/0293442.
Regarding claim 34, Nagarajan teaches the apparatus of claim 28, wherein the first output data includes an indication of whether the machine learning model inferred that the first input video segment depicts the at least one event (See Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-20 grasp attempt data being data indicative of being a success of failure), and one or more of the at least one processor circuit is to determine the second training data based on (i) the indication of whether the machine learning model inferred that the first input video segment depicts the at least one event and (ii) the label specifying whether the first input video segment depicts the at least one event (See Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-20 labeling and annotating of the event by the user for verification based on second training data of the labeling).
Nagarajan is silent with respect to data being based on a comparison of the inferred event and the label.
However, in the same field of endeavor, Fridental teaches of data being based on a comparison of the inferred event and the label (See [0076]-[0077], and [0092]-[0098]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Nagarajan to have incorporated the teachings of Fridental for the mere benefit of being able to provide corrected data from supervised training for a more accurate model.
10. Claims 35-36, and 39 are rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan et al., US 10,981,272 in view of Zeiler et al., US 2018/0091832.
Regarding claim 35, Nagarajan teaches of at least one non-transitory computer readable medium comprising computer readable instruction to cause at least one processor circuit to at least:
evaluate a download machine learning model to produce first output data corresponding to a first input video segment, the machine learning model trained with first training data to detect at least one event depicted in training video segments (See Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19 robot executes the model according to the image sensor and object detection event in a location for grasping based on trained segments and data);
cause second training data to be reported, the second training data based on the first input video segment, the first output data and the label (See Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19 robots report and send updated data/second training data which is based on the first output and of labeling/annotated/verifying the event); and
execute a retrained instance of the machine learning model to produce second output data corresponding to a second input video segment, the retained instance of the machine learning model based on the second training data (See Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19 the robot executes the updated/retrained model which produces the second output data/results of a second video and the updated model is based on the updated reported data through updated the model through an iterative process).
Nagarajan is silent with respect to based on a determination that no user input indicating whether the first input video segment depicts the at least one event was received during a presentation of the first input video segment, generate a label that indicates the first input video segment does not depict the at least one event.
However, in the same field of endeavor, Zeiler teaches of based on a determination that no user input indicating whether the first input video segment depicts the at least one event was received during a presentation of the first input video segment, generate a label that indicates the first input video segment does not depict the at least one event (See Zeiler, [0030]-[0031] and [0074]-[0076] absence of concept/event is based on having on input/button pressed where the absence/negative associations are labeled/stored).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Nagarajan to have incorporated the teachings of Zeiler for the mere benefit of not having to manually input every instance.
Regarding claim 36, the combination teaches the claim 35, wherein one or more of the at least one processor circuit is to segment a video feed based on a characteristic of an image sensor to obtain the first input video segment and the second input video segment, the image sensor to capture the video feed (See Nagarajan, Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-19 wherein the image sensor has a characteristic of at least a placement on the robot which obtains the first and second video segments and segments the video with bounding boxes for the object).
Regarding claim 39, the claim has been analyzed and rejected for the same reasons set forth in the rejection of claim 33.
11. Claim 37 is rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan et al., US 10,981,272 in view of Zeiler et al., US 2018/0091832, and in further view of JP 2017522066 (English translation).
Regarding claim 37, Nagarajan in view of Zeiler teaches the claim of 36.
Nagarajan in view of Zeiler is silent with respect to wherein the characteristic includes a sweep rate of the image sensor.
However, in the same field of endeavor, JP 2017522066 teaches of wherein the characteristic includes a sweep rate of the image sensor (See Pages 40-43).
It would have been obvious to one of ordinary skill in the art before effective filing date of the claimed invention to have modified the teachings of Nagarajan and Zieler to have incorporated the teachings of JP 2017522066 for the mere benefit of providing a more in depth training model.
12. Claim 38 is rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan et al., US 10,981,272 in view of Zeiler et al., US 2018/0091832, and in further view of Amato et al., US 2019/0065901.
Regarding claim 38, Nagarajan in view of Zeiler teaches the claim of 36.
Nagarajan in view of Zeiler is silent with respect to wherein the characteristic includes a capture rate of the image sensor.
However, in the same field of endeavor, Amato teaches of wherein the characteristic includes a capture rate of the image sensor (See [0055]).
It would have been obvious to one of ordinary skill in the art before effective filing date of the claimed invention to have modified the teachings of Nagarajan and Zeiler to have incorporated the teachings of Amato for the mere benefit of providing a more in depth training model.
13. Claim 40 is rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan et al., US 10,981,272 in view of Zeiler et al., US 2018/0091832, and in further view of .
Regarding claim 40, the combination of Nagarajan and Zeiler teaches the claim of 35, wherein the first output data includes an indication of whether the machine learning model inferred that the first input video segment depicts the at least one event (See Nagarajan, Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-20 grasp attempt data being data indicative of being a success of failure), and one or more of the at least one processor circuit is to determine the second training data based on (i) the indication of whether the machine learning model inferred that the first input video segment depicts the at least one event and (ii) the label specifying whether the first input video segment depicts the at least one event (See Nagarajan, Figs. 1-3; col.1-4; col.6-8; col.11-13; and col.17-20 labeling and annotating of the event by the user for verification based on second training data of the labeling).
Nagarajan is silent with respect to data being based on a comparison of the inferred event and the label.
However, in the same field of endeavor, Fridental teaches of data being based on a comparison of the inferred event and the label (See [0076]-[0077], and [0092]-[0098]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Nagarajan and Zeiler to have incorporated the teachings of Fridental for the mere benefit of being able to provide corrected data from supervised training for a more accurate model.
Conclusion
14. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Contact
15. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ricky Chin whose telephone number is 571-270-3753. The examiner can normally be reached on M-F 8:30-6:00.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Benjamin Bruckart can be reached on 571-272-3982. The fax phone number for the organization where this application or proceeding is assigned is 703-872-9306.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/Ricky Chin/
Primary Examiner
AU 2424
(571) 270-3753
Ricky.Chin@uspto.gov
Prosecution Insights