DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Applicant(s) Response to Official Action
The response filed on 3/30/2026 has been entered and made of record.
Response to Arguments/Amendments
Presented arguments have been fully considered, but some are rendered moot in view of the new ground(s) of rejection necessitated by amendment(s) initiated by the applicant(s). Examiner fully addresses below any arguments that were not rendered moot.
Claim Rejections - 35 USC § 103
Summary of Arguments:
Regarding claims 1-5, 8, 10-17 and 20 Applicant argues that Choe was filed on March 31, 2021. Choe claims priority to provisional application 63/091,938 filed on October 15, 2020. Upon review, the text passages of Choe that the Examiner relied upon (e.g., pertaining to evaluation module 210 - see page 4 of the Office Action) were not present in provisional application 63/091,938.
Choe teaches 'domain randomization', which relates to failure cases. See Choe, paragraph 24: Training data corresponding to these failure cases can be provided to a synthetic data generation network, for example, which can generate synthetic training data that imitates or mimics these failure cases, but differ in one or more ways, such as may be accomplished using domain randomization. Choe however fails to teach or to fairly suggest, processing of stored data by the Al module by adding a random component.
Finally, Ariel fails to teach or to fairly suggest: determining a state in which a system operated by the Al module is not in use for its primary purpose. While Ariel, as the Examiner correctly pointed out, teaches in paragraph 75 that the learning stage may be performed during idle time background operations (BKOPs) for pre- processing and model learning, no determining of a state in which a system operated by the Al module is not in use for its primary purpose is taught. Ariel's suggested performing of a learning stage during idle time background operations does not automatically teach the claimed determination.
Examiner’s Response:
Examiner respectfully disagrees. Regarding claims 1, 3 and 5, Examiner contends that Tae Eun Choe et al. [US 20220122001 A1] is well supported by provisional application 63/091,938. Page 1 of the provisional application discloses:
Summary and Description
Data is the most important part for the development of machine learning algorithms. The data collected in the real world may not always be representative or balanced for training machine learning modules such as Deep Neural Network (DNN). Embodiments of the present disclosure provide a technique to generate synthetic data and train a network with it to fortify the dataset. First, the current system is evaluated and failure cases such as false positive and false negative detections are identified. Secondly, data imitating such failure cases is collected to provide a more abundant dataset. This data may include generated synthetic data or collected real data. Thirdly, a network is retrained with the existing data and the newly added synthetic data. In one or more embodiments these steps may be repeated until the evaluation metric converges. This cyclic development flow is illustrated in Figure 1.
¶0036 define the function determined in by the “evaluation module 210 or process” as “loss function” of Choe as “l is a loss function with lϵ[0,1], with a value of 0 if detection is a true positive or true negative, and a value of 1 if detection is a false positive or false negative in a binary loss function”. This is consistent with the evaluation disclosed in the provisional application. Therefore, ¶0036 of Choe is well supported by its provisional application.
The addition of a random component to the stored data, as claimed, is patentably indistinguishable from the “domain randomization” disclosed in Choe. Applicant’s arguments fail to comply with 37 CFR 1.111(b) because they amount to a general allegation that the claims define a patentable invention without specifically pointing out how the language of the claims patentably distinguishes them from the references.
It is inherent and logical that Choe’s require a way to determine when/if the system is idle in order to initiate machine learning or training in flash systems using BKOPs in idle time. This is not an optional feature/function that may occur or be present, there is simply no other way.
Accordingly, Examiner maintains the rejections.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-5, 8, 10-17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Tae Eun Choe et al. [US 20220122001 A1: already of record] in view of Ariel Navon et al. [US 20200110536 A1: already of record].
Regarding claim 1, Choe teaches and/or suggests:
1. A method for operating an AI module (i.e. Approaches presented herein provide for the generation of synthetic data to fortify a dataset for use in training a network via imitation learning- Abstract), comprising:
processing of stored data (i.e. a set of initial real data is provided that serves as a source of labeled ground truth data 202- ¶0031) by the AI module by adding a random (i.e. such as may be accomplished using domain randomization- ¶0024) component (i.e. The evaluation module 210 can also determine various error or failure cases from the network inferences, and can provide information about these failure cases to an image synthesizer 206. This information can include at least information identifying images for which one or more incorrect inferences were identified. This information may also include information about the incorrect inferences, such as the type of inference, incorrect inference data (e.g., label or location), and so on. The image synthesizer 206 can then obtain the corresponding training image to use as a basis for one or more synthetic images, and can determine one or more parameter variations or perturbations to apply for each synthetic image to be generated- ¶0032);
evaluating a result of the processing (i.e. The same network can then be retrained, or a different network trained, using the initial training data and the newly-added synthetic data. These steps can be repeated until an end criterion is satisfied, such as where an evaluation metric converges- ¶0024… For each iteration, synthetic data will be generated in at least one embodiment only based on failures or inference errors detected during that specific training pass- ¶0030); and
adapting the AI module depending on the evaluation (i.e. These inferences can be passed, along with the relevant ground truth data, to an evaluation module 210 or process, which can determine errors in the inferences, loss values, or other error or performance-related data. In this example an evaluation module 210 can perform multiple tasks. First, the evaluation module 210 can determine the loss values for the network, using an appropriate loss function, and can provide this loss data to one or more network parameter adjustment modules 212 that can perform back-propagation through the network and determine one or more network weight adjustments to be applied to the DNN 208 as part of the training process of the training module 206- ¶0031).
However, Choe does not teach explicitly:
determining a state in which a system operated by the AI module is not in use for its primary purpose.
In the same field of endeavor, Ariel teaches and/or suggests:
determining a state in which a system operated by the AI module is not in use for its primary purpose (i.e. the learning stage may be performed during idle time background operations (BKOPs) for pre-processing and model learning and provide better performance in foreground forward operation- ¶0075).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Choe with the teachings of Ariel to provide better performance in foreground forward operation (Ariel- ¶0075).
Regarding claim 2, Choe and Ariel teach all the limitations of claim 1 and Choe further teaches and/or suggests:
wherein the stored data comprise data from earlier situations processed by the AI module (i.e. FIG. 1 illustrates example image data that can be utilized or produced in an example training process for an object detection network. As mentioned, an example training process can start with a set of labeled real data, or ground truth data. This data may have been captured with one or more cameras, for example, and then at least partially manually labeled in order to identify locations in a given image that correspond to an object, of a set of object types, that the network is trained to recognize- -¶0025).
Regarding claim 3, Choe and Ariel teach all the limitations of claim 1 and Choe further teaches and/or suggests:
wherein the random component is applied to the stored data (i.e. In a first step, an existing system or network is evaluated using a set of training data, and failure cases such as false positive and false negative detections are identified. Training data corresponding to these failure cases can be provided to a synthetic data generation network… such as may be accomplished using domain randomization- ¶0024).
Regarding claim 4, Choe and Ariel teach all the limitations of claim 3 and Choe further teaches and/or suggests:
wherein the random component causes random perturbations of the stored data (i.e. Training data corresponding to these failure cases can be provided to a synthetic data generation network, for example, which can generate synthetic training data that imitates or mimics these failure cases, but differ in one or more ways, such as may be accomplished using domain randomization- ¶0024… In the figures, it is illustrated that a first example synthesized image 120 with one or more perturbations includes a representation of the object corresponding to the incorrect inference that is larger and in a different location. The selection of other objects, both of interest and otherwise, is also varied. A second perturbed synthetic image 130 is generated that shows a similar type of object with a different orientation, in a different location. Various other changes or perturbations can be utilized as well, as may include variations in lighting, contrast, weather conditions, time of day, season, motion, geometry, background, and the like- ¶0029… Example game engines for such simulation include Unity, Unreal Engine, and GTA. A simulator was selected that was based on the Unreal Engine, which provided realistic rendering with diffused shadows, reflection of light, water droplets on lenses or windshields, diffused lights, and sun glare. This simulator was able to reproduce any target scenario with various weather conditions, locations, and time for domain randomization- ¶0044…The imitator was therefore caused to create multiple images with partially visible trucks at various places, times of day, and weather conditions for domain randomization- ¶0051).
Regarding claim 5, Choe and Ariel teach all the limitations of claim 1 and Choe further teaches and/or suggests:
wherein the random component is applied to at least one algorithm or at least one parameter of the AI module (i.e. The newly trained model was evaluated on an evaluation dataset and the cases of false positive and false negative detections were reported and the cycle was repeated. In another example, an existing DNN trained on real data had an issue in detecting partially-visible trucks. The imitator was therefore caused to create multiple images with partially visible trucks at various places, times of day, and weather conditions for domain randomization- ¶0051).
Regarding claim 8, Choe and Ariel teach all the limitations of claim 1 and Choe further teaches and/or suggests:
wherein the AI module is used in a locomotion means (i.e. FIG. 16A illustrates an example of an autonomous vehicle, according to at least one embodiment- ¶0020-0023, fig. 16a-16d… In at least one embodiment, the trained neural network can then be deployed or used for inferencing as part of an application, system, or service. One such system 400 is illustrated in FIG. 4. In this example, a DNN 406 is used for object recognition as part of an autonomous, or semi-autonomous, navigation system. In this example, a vehicle may include one or more cameras 402 and/or one or more sensors 404 that are able to capture image data, or additional data relating to captured image data, as may include position, motion, or distance data- ¶0035).
Regarding claim 10, Choe and Ariel teach all the limitations of claim 8 and Choe further teaches and/or suggests:
wherein the AI module is used for computer vision or autonomous driving tasks (i.e. FIG. 13 is an example data flow diagram for a process 1300 of generating and deploying an image processing and inferencing pipeline, in accordance with at least one embodiment. In at least one embodiment, process 1300 may be deployed for use with imaging devices, processing devices, and/or other device types at one or more facilities 1302. Process 1300 may be executed within a training system 1304 and/or a deployment system 1306. In at least one embodiment, training system 1304 may be used to perform training, deployment, and implementation of machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.)- ¶0116… In at least one embodiment, controller(s) 1636, which may include, without limitation, one or more system on chips (“SoCs”) (not shown in FIG. 16A) and/or graphics processing unit(s) (“GPU(s)”), provide signals (e.g., representative of commands) to one or more components and/or systems of vehicle 1600. For instance, in at least one embodiment, controller(s) 1636 may send signals to operate vehicle brakes via brake actuator(s) 1648, to operate steering system 1654 via steering actuator(s) 1656, and/or to operate propulsion system 1650 via throttle/accelerator(s) 1652. Controller(s) 1636 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals, and output operation commands (e.g., signals representing commands) to enable autonomous driving and/or to assist a human driver in driving vehicle 1600. In at least one embodiment, controller(s) 1636 may include a first controller 1636 for autonomous driving functions, a second controller 1636 for functional safety functions, a third controller 1636 for artificial intelligence functionality (e.g., computer vision), a fourth controller 1636 for infotainment functionality, a fifth controller 1636 for redundancy in emergency conditions, and/or other controllers. In at least one embodiment, a single controller 1636 may handle two or more of above functionalities, two or more controllers 1636 may handle a single functionality, and/or any combination thereof.- ¶0167).
Regarding claim 11, Choe and Ariel teach all the limitations of claim 8 and Choe further teaches and/or suggests:
wherein the AI module is used in a robot, a medical device, or a consumer device (i.e. In at least one embodiment, vehicle 1a00 may be an airplane, robotic vehicle, or other kind of vehicle- ¶0163).
Regarding claim 12, computer-readable medium storing instructions claim 12 corresponds to the same method as claimed in claim 1, and therefore is also rejected for the same reasons of obviousness as listed above.
Regarding claim 13, apparatus claim 13 is drawn to the apparatus using/performing the same method as claimed in claim 1. Therefore, apparatus claim 13 corresponds to method claim 1, and is rejected for the same reasons of obviousness as used above.
Regarding claim 14, Choe and Ariel teach all the limitations of claim 8 and Choe further teaches and/or suggests:
wherein the AI module is used in a robot, a medical device, or a consumer device (i.e. In at least one embodiment, to aid in processing or execution of applications or containers in pipelines, services 1320 may be leveraged. In at least one embodiment, services 1320 may include compute services, artificial intelligence (AI) services, visualization services, and/or other service types. In at least one embodiment, services 1320 may provide functionality that is common to one or more applications in software 1318, so functionality may be abstracted to a service that may be called upon or leveraged by applications- ¶0127).
Regarding claim 15, Choe and Ariel teach all the limitations of claim 14 and Choe further teaches and/or suggests:
15. A locomotion, device, the locomotion device has an AI module according to claim 14 (i.e. FIG. 16A illustrates an example of an autonomous vehicle, according to at least one embodiment- ¶0020-0023, fig. 16a-16d… In at least one embodiment, the trained neural network can then be deployed or used for inferencing as part of an application, system, or service. One such system 400 is illustrated in FIG. 4. In this example, a DNN 406 is used for object recognition as part of an autonomous, or semi-autonomous, navigation system. In this example, a vehicle may include one or more cameras 402 and/or one or more sensors 404 that are able to capture image data, or additional data relating to captured image data, as may include position, motion, or distance data- ¶0035).
Regarding claim 16, Choe and Ariel teach all the limitations of claim 2 and Choe further teaches and/or suggests:
wherein the random component is applied to the stored data (i.e. In a first step, an existing system or network is evaluated using a set of training data, and failure cases such as false positive and false negative detections are identified. Training data corresponding to these failure cases can be provided to a synthetic data generation network… such as may be accomplished using domain randomization- ¶0024).
Regarding claim 17, Choe and Ariel teach all the limitations of claim 2 and Choe further teaches and/or suggests:
wherein the random component is applied to at least one algorithm or at least one parameter of the AI module (i.e. The newly trained model was evaluated on an evaluation dataset and the cases of false positive and false negative detections were reported and the cycle was repeated. In another example, an existing DNN trained on real data had an issue in detecting partially-visible trucks. The imitator was therefore caused to create multiple images with partially visible trucks at various places, times of day, and weather conditions for domain randomization- ¶0051).
Regarding claim 20, Choe and Ariel teach all the limitations of claim 13 and Choe further teaches and/or suggests:
20. A locomotion device, wherein the locomotion device has a device according to claim 13 (i.e. FIG. 16A illustrates an example of an autonomous vehicle, according to at least one embodiment- ¶0020-0023, fig. 16a-16d… In at least one embodiment, the trained neural network can then be deployed or used for inferencing as part of an application, system, or service. One such system 400 is illustrated in FIG. 4. In this example, a DNN 406 is used for object recognition as part of an autonomous, or semi-autonomous, navigation system. In this example, a vehicle may include one or more cameras 402 and/or one or more sensors 404 that are able to capture image data, or additional data relating to captured image data, as may include position, motion, or distance data- ¶0035).
Claims 6, 18 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Tae Eun Choe et al. [US 20220122001 A1: already of record] in view of Ariel Navon et al. [US 20200110536 A1: already of record] and further in view of Zhenqiang Li [Towards Visually Explaining Video Understanding Networks with Perturbation: already of record].
Regarding claims 6, 18 and 19, Choe and Ariel teach and/or suggest all the limitation of claims 1, 2, 3 respectively:
However, Choe and Ariel do not teach explicitly:
wherein correlations in data sets are detected during the evaluation of results of the processing.
In the same field of endeavor, Zhenqiang teaches and/or suggests:
wherein correlations in data sets are detected during the evaluation of results of the processing (i.e. Fig. 3 illustrates two groups of visualization results including the original frames and importance maps generated by different visual explanation approaches. Each group corresponds to one example video, and only 5 frames are sampled out of 16 input frames for visualization. For perturbation-based methods, we visualize the results generated under the preservation ratio constrain of 0.1… There is a part of regions highly correlated with the ground-truth label in the two videos, e.g., the first two frames of the left video example, and the middle three frames of the right example- Section 4.3.1… Figure 3. Qualitative comparison of visual explanation results generated by baseline methods and the perturbation-based methods. The importance maps generated by our method could smoothly preserve the regions associated with the ground-truth label and remove areas with weaker correlations- fig. 3).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Choe and Ariel with the teachings of Zhenqiang to visually explain a black-box network by directly operating on the input and locating the area that affects the output most in a forward manner (Zhenqiang- fig. 1).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Tae Eun Choe et al. [US 20220122001 A1: already of record] in view of Ariel Navon et al. [US 20200110536 A1: already of record] and further in view of Angelo Dalli et al. [US 20210232915 A1].
Regarding claim 7, Choe and Ariel teach and/or suggest all the limitation of claim 8.
However, Choe and Ariel do not teach explicitly:
wherein adapting the AI module comprises a reduction of the input variables to be processed for a task.
In the same field of endeavor, Brent teaches and/or suggests:
wherein adapting the AI module comprises a reduction of the input variables to be processed for a task (i.e. a CNN-XNN may be used to directly utilize pre-processed or partially processed input data and correctly perform a combination of fusion, routing, transformation, dimensionality reduction and/or flatten operations, taking advantage of the white-box nature of CNN-XNNs to do so correctly and efficiently- ¶0083).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Choe and Ariel with the teachings of Angelo to take advantage of the white-box nature of CNN-XNNs to do so correctly and efficient (Angelot- ¶083).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Tae Eun Choe et al. [US 20220122001 A1: already of record] in view of Ariel Navon et al. [US 20200110536 A1: already of record] and further in view of Brent Justin Goldman et al. [US 20190340928 A1: already of record].
Regarding claim 9, Choe and Ariel teach and/or suggest all the limitation of claim 8.
However, Choe and Ariel do not teach explicitly:
wherein a state in which the system operated by the AI module is not in use for its primary purpose is determined when the locomotion means is parked or not in motion.
In the same field of endeavor, Brent teaches and/or suggests:
wherein a state in which the system operated by the AI module is not in use for its primary purpose is determined when the locomotion means is parked or not in motion (i.e. For example, a computing system (e.g., an operations computing system of a service entity) can obtain data associated with one or more autonomous vehicles that are online with a service entity (e.g., and in an idle state)… the computing system re-position the autonomous vehicle(s) so that the processing, memory, and power resources of the vehicle's computing system are more likely to be utilized for performing vehicle services (as opposed to vehicle idling)- ¶0057).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Choe and Ariel with the teachings of Brent leads to a more effective use of a vehicle's computational resource, while reducing the need for autonomous vehicle's to go offline to replenish such resources (Brent- ¶0057).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CLIFFORD HILAIRE whose telephone number is (571)272-8397. The examiner can normally be reached 5:30-1400.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SATH V PERUNGAVOOR can be reached at (571)272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
CLIFFORD HILAIRE
Primary Examiner
Art Unit 2488
/CLIFFORD HILAIRE/Primary Examiner, Art Unit 2488
Prosecution Insights