DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
This Application is a Continuation of US Application 17/829,484 filed 1 June 2022, now US Patent No 12,296,846.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 17 July 2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 5-11, 13, 15, 16, 18 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over US PGPub 2023/0051565 to Cower et al (hereafter Cower) in view of US Patent No 11,157,813 to Je et al (hereafter Je).
Referring to claim 1, Feng discloses a computer implemented method comprising:
generating, by a computing system, a classification model [prediction model/neural network] of a plurality of classification models associated with a multi-task machine learning model on a vehicle, each classification model associated with a respective scenario of interest [scene] (see [0022]-[0024] – The system obtains one or more sensor data inputs depicting the same scene in an environment. The system processes the one or more sensor data inputs using one or more trained prediction models to generate a plurality of predictions about a characteristic of an object of the scene. The one or more trained prediction models can be trained on the machine learning task using training data sampled from the log data, or using a different set of training data.), wherein the generating comprises:
training the classification model [prediction model/task neural network] on the first training data reflective of the scenario of interest (see [0021]; [0023]; [0049]; and [0052] – Classification of images depicting a scene in an environment. The system uses the training dataset to train the task neural network. Once the task neural network has been fully trained, the task neural network can be deployed on an autonomous vehicle so that image classification or object detection may be performed by an on-board computer system of the autonomous vehicle as the vehicle navigates through the environment.);
training the classification model on second training data determined with a threshold confidence level by the classification model to be reflective of the scenario of interest (see Fig 1; [0021]; [0024]; [0030]; [0042]; [0050]; [0052] – The one or more trained prediction models are the same as the task neural network. The respective prediction may assign a score to each object category of a set of object categories, with each score representing an estimated likelihood that the object of the scene depicted in the particular input image belonging to the respective object category. The system then adds the hard examples to the training dataset and uses the updated dataset to re-train the task neural network. The re-trained neural network is considered to represent the second trained classification model. Once the task neural network has been fully trained, the task neural network can be deployed on an autonomous vehicle so that image classification or object detection may be performed by an on-board computer system of the autonomous vehicle as the vehicle navigates through the environment.);
determining, by the computing system, an occurrence of the scenario of interest based on the classification model (see [0052] – Once the task neural network (or the multiple task neural networks) has been fully trained, the task neural network(s) can be, for example, deployed on an autonomous vehicle so that image classification or object detection may be performed by an on-board computer system of the autonomous vehicle as the autonomous vehicle navigates through the environment.)
While Cower teaches determining an occurrence of the scenario of interest based on the classification, Cower fails to explicitly teach the further limitation of providing, by the computing system, a representation of the scenario of interest to a remote server to update as associated classification model. Je teaches the classification of images/video from a vehicle using a classification model, including the further limitations of
training the classification model on second training data determined with a threshold confidence level by the classification model to be reflective of the scenario of interest (Je: see column 10, line 48 – column 11, line 9);
determining, by the computing system, an occurrence of the scenario of interest based on the classification model (see column 8, lines 1-7); and
providing, by the computing system, a representation of the scenario of interest to a remote server to update an associated classification model (see column 9, lines 29-37).
Cower and Je are analogous art since they both relate to the selection of data for training a classification model and the actual training of the classification model with relation to a vehicle. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to utilize the process of Je for updating the classification model of Cower. One would have been motivated to do so in order to effectively update a perception network (Cower: see [0020]).
Referring to claim 3, the combination of Cower and Je (hereafter Cower/Je) teaches the method of claim 1, wherein the representation of an occurrence of the scenario of interest is a video frame stored in memory of the vehicle (Je: see column 9, lines 48-61).
Referring to claim 5, Cower/Je teaches the method of claim 1, wherein the first training data is based on a query performed on a set of image data, the query specifying the scenario of interest [scene] (Je: see column 9, line 29 – column 10, line 32 and column 11, lines 20-60).
Referring to claim 6, Cower/Je teaches the method claim 5, wherein a text-based search is performed in response to the query (Je: see column 9, line 29 – column 10, line 32 and column 11, lines 20-60 – Using scene codes).
Referring to claim 7, Cower/Je teaches the non-transitory, processor-readable medium of claim 3, wherein the query performs an image-based search (Je: see column 9, line 29 – column 10, line 32 and column 11, lines 20-60 – Using object detection information).
Referring to claim 8, Cower/Je teaches the method of claim 1, wherein the classification model is trained based on text-image co-occurrence. (Je: see column 9, line 29 – column 10, line 32 and column 11, lines 20-60).
Referring to claim 9, Cower/Je teaches the method of claim 1, wherein the first training data and the second training data include frames of image data, and the first training data has fewer frames than the second training data (Cower: see Fig 1; [0021]; [0024]; [0030]; [0042]; [0050]; [0052] – The one or more trained prediction models are the same as the task neural network. The respective prediction may assign a score to each object category of a set of object categories, with each score representing an estimated likelihood that the object of the scene depicted in the particular input image belonging to the respective object category. The system then adds the hard examples to the training dataset and uses the updated dataset to re-train the task neural network. The re-trained neural network is considered to represent the second trained classification model. Once the task neural network has been fully trained, the task neural network can be deployed on an autonomous vehicle so that image classification or object detection may be performed by an on-board computer system of the autonomous vehicle as the vehicle navigates through the environment.).
Referring to claim 10, Cower/Je teaches the method of claim 1, wherein the scenario of interest is associated with at least one of slippery road surface, rainy day [rain], sun glare, road ice, traffic accident in front of vehicle, emergency vehicle, road work, and scattered debris (Je: see column 8, lines 36-55).
Referring to claim 11, Cower discloses a system comprising:
at least one processor (see [0069]); and
a memory storing instructions that, when executed by the at least one processor (see [0074]), cause the system to perform operations comprising:
generating a classification model [prediction model/neural network] of a plurality of classification models associated with a multi-task machine learning model on a vehicle, each classification model associated with a respective scenario of interest [scene] (see [0022]-[0024] – The system obtains one or more sensor data inputs depicting the same scene in an environment. The system processes the one or more sensor data inputs using one or more trained prediction models to generate a plurality of predictions about a characteristic of an object of the scene. The one or more trained prediction models can be trained on the machine learning task using training data sampled from the log data, or using a different set of training data.), wherein the generating comprises:
training the classification model [prediction model/task neural network] on the first training data reflective of the scenario of interest (see [0021]; [0023]; [0049]; and [0052] – Classification of images depicting a scene in an environment. The system uses the training dataset to train the task neural network. Once the task neural network has been fully trained, the task neural network can be deployed on an autonomous vehicle so that image classification or object detection may be performed by an on-board computer system of the autonomous vehicle as the vehicle navigates through the environment.);
training the classification model on second training data determined with a threshold confidence level by the classification model to be reflective of the scenario of interest (see Fig 1; [0021]; [0024]; [0030]; [0042]; [0050]; [0052] – The one or more trained prediction models are the same as the task neural network. The respective prediction may assign a score to each object category of a set of object categories, with each score representing an estimated likelihood that the object of the scene depicted in the particular input image belonging to the respective object category. The system then adds the hard examples to the training dataset and uses the updated dataset to re-train the task neural network. The re-trained neural network is considered to represent the second trained classification model. Once the task neural network has been fully trained, the task neural network can be deployed on an autonomous vehicle so that image classification or object detection may be performed by an on-board computer system of the autonomous vehicle as the vehicle navigates through the environment.);
determining an occurrence of the scenario of interest based on the classification model (see [0052] – Once the task neural network (or the multiple task neural networks) has been fully trained, the task neural network(s) can be, for example, deployed on an autonomous vehicle so that image classification or object detection may be performed by an on-board computer system of the autonomous vehicle as the autonomous vehicle navigates through the environment.)
While Cower teaches determining an occurrence of the scenario of interest based on the classification, Cower fails to explicitly teach the further limitation of providing a representation of the scenario of interest to a remote server to update as associated classification model. Je teaches the classification of images/video from a vehicle using a classification model, including the further limitations of
training the classification model on second training data determined with a threshold confidence level by the classification model to be reflective of the scenario of interest (Je: see column 10, line 48 – column 11, line 9);
determining an occurrence of the scenario of interest based on the classification model (see column 8, lines 1-7); and
providing a representation of the scenario of interest to a remote server to update an associated classification model (see column 9, lines 29-37).
Cower and Je are analogous art since they both relate to the selection of data for training a classification model and the actual training of the classification model with relation to a vehicle. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to utilize the process of Je for updating the classification model of Cower. One would have been motivated to do so in order to effectively update a perception network (Cower: see [0020]).
Referring to claim 13, Cower/Je teaches the system of claim 11, wherein the representation of an occurrence of the scenario of interest is a video frame stored in memory of the vehicle (Je: see column 9, lines 48-61).
Referring to claim 15, Cower/Je teaches the system of claim 11, wherein the first training data is based on a query performed on a set of image data, the query specifying the scenario of interest [scene] (Je: see column 9, line 29 – column 10, line 32 and column 11, lines 20-60).
Referring to claim 16, Cower discloses a non-transitory, computer-readable storage medium including instructions that, when executed by at least one processor of a computing system (see [0068]), cause the computing system to perform operations comprising:
generating a classification model [prediction model/neural network] of a plurality of classification models associated with a multi-task machine learning model on a vehicle, each classification model associated with a respective scenario of interest [scene] (see [0022]-[0024] – The system obtains one or more sensor data inputs depicting the same scene in an environment. The system processes the one or more sensor data inputs using one or more trained prediction models to generate a plurality of predictions about a characteristic of an object of the scene. The one or more trained prediction models can be trained on the machine learning task using training data sampled from the log data, or using a different set of training data.), wherein the generating comprises:
training the classification model [prediction model/task neural network] on the first training data reflective of the scenario of interest (see [0021]; [0023]; [0049]; and [0052] – Classification of images depicting a scene in an environment. The system uses the training dataset to train the task neural network. Once the task neural network has been fully trained, the task neural network can be deployed on an autonomous vehicle so that image classification or object detection may be performed by an on-board computer system of the autonomous vehicle as the vehicle navigates through the environment.);
training the classification model on second training data determined with a threshold confidence level by the classification model to be reflective of the scenario of interest (see Fig 1; [0021]; [0024]; [0030]; [0042]; [0050]; [0052] – The one or more trained prediction models are the same as the task neural network. The respective prediction may assign a score to each object category of a set of object categories, with each score representing an estimated likelihood that the object of the scene depicted in the particular input image belonging to the respective object category. The system then adds the hard examples to the training dataset and uses the updated dataset to re-train the task neural network. The re-trained neural network is considered to represent the second trained classification model. Once the task neural network has been fully trained, the task neural network can be deployed on an autonomous vehicle so that image classification or object detection may be performed by an on-board computer system of the autonomous vehicle as the vehicle navigates through the environment.);
determining an occurrence of the scenario of interest based on the classification model (see [0052] – Once the task neural network (or the multiple task neural networks) has been fully trained, the task neural network(s) can be, for example, deployed on an autonomous vehicle so that image classification or object detection may be performed by an on-board computer system of the autonomous vehicle as the autonomous vehicle navigates through the environment.)
While Cower teaches determining an occurrence of the scenario of interest based on the classification, Cower fails to explicitly teach the further limitation of providing a representation of the scenario of interest to a remote server to update as associated classification model. Je teaches the classification of images/video from a vehicle using a classification model, including the further limitations of
training the classification model on second training data determined with a threshold confidence level by the classification model to be reflective of the scenario of interest (Je: see column 10, line 48 – column 11, line 9);
determining an occurrence of the scenario of interest based on the classification model (see column 8, lines 1-7); and
providing a representation of the scenario of interest to a remote server to update an associated classification model (see column 9, lines 29-37).
Cower and Je are analogous art since they both relate to the selection of data for training a classification model and the actual training of the classification model with relation to a vehicle. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to utilize the process of Je for updating the classification model of Cower. One would have been motivated to do so in order to effectively update a perception network (Cower: see [0020]).
Referring to claim 18, Cower/Je teaches the non-transitory, computer-readable medium of claim 16, wherein the representation of an occurrence of the scenario of interest is a video frame stored in memory of the vehicle (Je: see column 9, lines 48-61).
Referring to claim 20, Cower/Je teaches the non-transitory, computer-readable medium of claim 16, wherein the first training data is based on a query performed on a set of image data, the query specifying the scenario of interest [scene] (Je: see column 9, line 29 – column 10, line 32 and column 11, lines 20-60).
Claim(s) 2, 4, 12, 14, 17 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over US PGPub 2023/0051565 to Cower et al (hereafter Cower) in view of US Patent No 11,157,813 to Je et al (hereafter Je) as applied to claims 1, 11 and 16 above, and further in view of US PGPub 2020/0074266 to Peake et al (hereafter Peake).
Referring to claims 2, 12 and 17, while Cower/Je teaches a classification model updated based on a vehicle, Cower/Je fails to explicitly teach the further limitation wherein the remote server includes a plurality of classification models including the associated classification model, the plurality of classification models updated based on information from a fleet including the vehicle. Peake teaches the use of classification models with an autonomous vehicle, including the further limitation wherein the remote server includes a plurality of classification models including the associated classification model, the plurality of classification models updated based on information from a fleet including the vehicle (see [0034]).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to use a plurality of vehicles to update the classification model of Cower/Je as taught by Peake. One would have been motivated to do so in order to have a plurality of data to use in the updating of the model (Peake: see [0004]).
Referring to claims 4, 14 and 19, Cower/Je fails to teach the further limitation of a scenario occurrence counter. Peake teaches the use of classification models with an autonomous vehicle, including the further limitation wherein a scenario occurrence counter of the vehicle includes a numerical count of instances of video frames associated with the scenario of interest (see [0043]).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed inventio to use the occurrence calculator of Peake with the process of Cower/Je. One would have been motivated to do so to use the information for efficiently updating the model (Peake: see [0004]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US PGPub 2022/0114476 to Bui et al – Paragraph [0122] teaches how many iterations are used for training
US PGPub 2015/0305663 – teaches recall/precision
US PGPub 2018/0232601 to Feng et al - Feng discloses a computer implemented method comprising: training the classification model [neural network] on first training data [initial set of labeled images] reflective of scenario of interest (see [0044]; [0045]; [0051]; and Fig 1C, S1 – An initial labeled training dataset is provided and the neural network is trained by using the dataset.); training the classification model on second training data determined with a threshold confidence level by the classification model to be reflective of the scenario of interest (see [0045]; [0052]; [0053]; [0054]; [0055]; Fig 1C, steps S3 and S4 – Given the score obtained in step S2, images with the top K highest scores are selected for labeling by the annotation device. The selected images with newly annotated labels are added into the current (latest) labeled training set to get a new training dataset. The network is retrained based on the new training dataset.).
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KIMBERLY LOVEL WILSON whose telephone number is (571)272-2750. The examiner can normally be reached 8-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aleksandr Kerzhner can be reached at 571-270-1760. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KIMBERLY L WILSON/Primary Examiner, Art Unit 2165
Prosecution Insights