DETAILED ACTION
This office action is responsive to the response filed 2/5/2026. The application contains claims 1-20, all examined and rejected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. § 101 because the instant application is directed to non-patentable subject matter. Specifically, the claims are directed toward at least one judicial exception without reciting additional elements that amount to significantly more than the judicial exception. The rationale for this determination is in accordance with the guidelines of USPTO, applies to all statutory categories, and is explained in detail below.
When considering subject matter eligibility under 35 U.S.C. 101, (1) it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter. If the claim does fall within one of the statutory categories, (2a) it must then be determined whether the claim is directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea), and if so (2b), it must additionally be determined whether the claim is a patent-eligible application of the exception. If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim amounts to significantly more than the abstract idea itself. Examples of abstract ideas include certain methods of organizing human activities; a mental processes; and mathematical concepts, (2019 PEG)
STEP 1.
Per Step 1, the claims are determined to include a process, machine, manufacture as in independent Claim 1, 10, and 17 in the therefrom dependent claims. Therefore, the claims are directed to a statutory eligibility category.
Step 2A:
The invention is directed to training to be able to detect entities within different domains which is akin to a mental process and a mathematical concept. As such, the claims include an abstract idea. When considering the limitations individually and as a whole the limitations directed to abstract idea are:
forming a set of training parameters applicable to detection of two or more entities between and/or among a distribution of digitally stored entities from a plurality of digitally stored observations (Mental process, user can initially set a set of training parameters);
modifying one or more training parameters of the set of training parameters to define a translation applicable to detection of real-world entities of a first environment corresponding to the two or more entities between and/or among the distribution of digitally stored entities from the plurality of digitally stored observations; the translation to be defined based (Mental Process, user can select specific training parameters), at least in part, on:
a first process of generating the two or more entities between and/or among the distribution of digitally stored entities from the plurality of digitally stored observations (Mental process);
a second process of discriminating between and/or among the generated two or more entities based, at least in part, on the modified one or more training parameters (Mental process, user can mentally differentiate between real and generated content)
sampling, during operation in a second environment, real-world observations to determine a distribution of physical entities present in the second environment (Mental process); and
detecting, using the translation, at least one physical entity in the second environment (Mental process);
further modifying the one or more training parameters based at least in part on the (i) determined distribution of the one or more real-world entities of the second environment and (ii) detection performance associated with the detected at least one physical entity so as to improve detection of physical entity across differing environments (Mental process).
This judicial exception is not integrated into a practical application. The elements are recited at a high level of generality, i.e. a generic computing system performing generic functions including generic processing of data. Accordingly the additional elements do not integrate the abstract into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Therefore the claims are directed to an abstract idea. (2019 Revised Patent Subject Matter Eligibility Guidance ("2019 PEG"). Thus, under Step 2A of the Mayo framework, the Examiner holds that the claims are directed to concepts identified as abstract.
Step 2A Prong Two:
The claims recite additional elements as “a computing device”, “computing device via one or more sensors”, “by the computing device” are limitation that invokes computers or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f),
“automatically further modifying the one or more training parameters in a closed- loop manner” high-generic computer software process of training data. These limitations do not amount to significantly more than the judicial exception, see MPEP 2106.05 (f);
are limitation that invokes computers or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), in addition does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it". See Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44 (Fed. Cir. 2016) (MPEP 2106.05(f)(1)).
The elements are recited at a high level of generality, i.e. a generic computing system performing generic functions including generic processing and receiving of data. Accordingly the additional elements do not integrate the abstract into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Therefore the claims are directed to an abstract idea. (2019 Revised Patent Subject Matter Eligibility Guidance ("2019 PEG"). Thus, under Step 2A of the Mayo framework, the Examiner holds that the claims are directed to concepts identified as abstract.
STEP 2B.
Because the claims include one or more abstract ideas, the examiner now proceeds to Step 2B of the analysis, in which the examiner considers if the claims include individually or as an ordered combination limitations that are "significantly more" than the abstract idea itself. This includes analysis as to whether there is an improvement to either the "computer itself," "another technology," the "technical field," or significantly more than what is "well-understood, routine, or conventional" in the related arts.
The instant application includes in Claim 1 additional steps to those deemed to be abstract idea.
“a computing device”, “computing device via one or more sensors”, “by the computing device” are limitation that invokes computers or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f)(2), in addition does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it". does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it".
“automatically further modifying the one or more training parameters in a closed- loop manner” high-generic computer software process of training data. These limitations do not amount to significantly more than the judicial exception, see MPEP 2106.05 (f).
In the instant case, Claim 1 is directed to above mentioned abstract idea. Technical functions such as receiving, and processing data are common and basic functions in computer technology. The individual limitations are recited at a high level and do not provide any specific technology or techniques to perform the functions claimed.
Looking to MPEP 2106.05 (d), based on court decisions well understood, routine and conventional computer functions or mere instruction and/or insignificant activity have been identified to include: Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321,120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); In Bilski referring to Flook, where Flook determined that an insignificant post-solution activity does not makes an otherwise patent ineligible claim patent eligible. In Bilski, the court added to Flook that pre-solution (such as data gathering) and insignificant step in the middle of a process (such as receiving user input) to be equally ineffective. The claims does not provide any specific process with respect to the additional elements that would transform the function beyond what is well understood. Like as found in Electric Power Group, Bilski, the technical process to implement the input and display functions are conventional and well understood.
In addition, when the claims are taken as a whole, as an ordered combination, the combination of steps does not add "significantly more" by virtue of considering the steps as a whole, as an ordered combination. The instant application, therefore, still appears only to implement the abstract idea to the particular technological environments using what is well-understood, routine, and conventional in the related arts. The steps are still a combination made to the abstract idea. The additional steps only add to those abstract ideas using well-understood and conventional functions, and the claims do not show improved ways of, for example, an unconventional non-routine functions for analyzing documents or receiving user input or generating output that could then be pointed to as being "significantly more" than the abstract ideas themselves. Moreover, Examiner was not able to identify any "unconventional" steps, which, when considered in the ordered combination with the other steps, could have transformed the nature of the abstract idea previously identified. The instant application, therefore, still appears to only implement the abstract ideas to the particular technological environments using what is well-understood, routine, and conventional in the related arts.
Further, note that the limitations, in the instant claims, are done by the generically recited computing devices. The limitations are merely instructions to implement the abstract idea on a computing device that is recited in an abstract level and require no more than a generic computing devices to perform generic functions.
CONCLUSION
It is therefore determined that the instant application not only represents an abstract idea identified as such based on criteria defined by the Courts and on USPTO examination guidelines, but also lacks the capability to bring about "Improvements to another technology or technical field" (Alice), bring about "Improvements to the functioning of the computer itself" (Alice), "Apply the judicial exception with, or by use of, a particular machine" (Bilski), "Effect a transformation or reduction of a particular article to a different state or thing" (Diehr), "Add a specific limitation other than what is well-understood, routine and conventional in the field" (Mayo), "Add unconventional steps that confine the claim to a particular useful application" (Mayo), or contain "Other meaningful limitations beyond generally linking the use of the judicial exception to a particular technological environment" (Alice), transformed a traditionally subjective process performed by humans into a mathematically automated process executed on computers (McRO), or limitations directed to improvements in computer related technology, including claims directed to software (Enfish).
Claim 10 recites a system comprising “a processor coupled to at least one memory device” configured to perform the same method as set forth in claim 1, the added element of “a processor coupled to at least one memory device” do not transform the judicial exception into a practical application because they are tantamount to a mere instruction to apply the judicial exception to a generic computer. The additional elements are also not sufficient to amount to significantly more than the judicial exception because the action of implementing the method on a general purpose computer with at least one processor and at least one memory is tantamount to a mere instruction to apply the judicial exception to a computer.
Claim 10 is therefore rejected according to the same findings and rationale as provided above.
Claim 17 recites a system comprising “non-transitory storage medium, having instructions stored thereon, which, when executed by a computer processor coupled to at least one memory” configured to perform the same method as set forth in claim 1, the added element of “non-transitory storage medium, having instructions stored thereon, which, when executed by a computer processor coupled to at least one memory” do not transform the judicial exception into a practical application because they are tantamount to a mere instruction to apply the judicial exception to a generic computer. The additional elements are also not sufficient to amount to significantly more than the judicial exception because the action of implementing the method on a general purpose computer with at least one processor and at least one memory is tantamount to a mere instruction to apply the judicial exception to a computer.
Claim 17 is therefore rejected according to the same findings and rationale as provided above.
The dependent claims, when considered individually and as a whole, likewise do not provide “significantly more” than the abstract idea for similar reasons as the independent claim. For example claim 2 disclose using generative adversarial network (GAN) as a discriminator (provide nothing more than mere instructions to implement an abstract idea on a generic computer. See MPEP 2106.05(f)) that doesn't integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 3, list an algorithms CycleGAN (provide nothing more than mere instructions to implement an abstract idea on a generic computer. See MPEP 2106.05(f)) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 4 disclose “wherein the two or more entities between and/or among the distribution of digitally stored entities correspond to entities detectable within visual sensor or audio sensor data” (data description, which is directed to generally linking the use of a judicial exception to a particular technological environment or type or source of data or field of use MPEP 2106.05(h)) and does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it". See Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44 (Fed. Cir. 2016) (MPEP 2106.05(f)), claim 5, disclose “wherein detecting the at least one physical entity comprises classifying the at least one physical entity using a trained neural network” (data description, which is directed to generally linking the use of a judicial exception to a particular technological environment or type or source of data or field of use MPEP 2106.05(h)) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 6, disclose “automatically further modifying the training parameters comprises narrowing one or more parameter ranges responsive to incorrect classification of the detected at least one physical entity” (data description, which is directed to generally linking the use of a judicial exception to a particular technological environment or type or source of data or field of use MPEP 2106.05(h)) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 7, “automatically further modifying comprises weighting the training parameters based on frequencies within the determined distribution” (provide nothing more than mere instructions to implement an abstract idea on a generic computer. See MPEP 2106.05(f)) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 8, “sampling real-world observations includes querying a real world entities database” sending and receiving data (insignificant extra-solution activity that the courts recognized as well-understood, routine, conventional activity) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 9, “first environment comprises an office environment and the second environment comprises a domestic environment” defining applicable domains (provide nothing more than mere instructions to implement an abstract idea on a generic computer. See MPEP 2106.05(f)) and does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words "apply it". See Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44 (Fed. Cir. 2016) (MPEP 2106.05(f)),
The dependent claims which impose additional limitations also fail to claim patent-eligible subject matter because the limitations cannot be considered statutory. The dependent claim(s) have been examined individually and in combination with the preceding claims, however they do not cure the deficiencies of claim 1; where all claims are directed to the same abstract idea, "addressing each claim of the asserted patents [is] unnecessary." Content Extraction &. Transmission LLC v, Wells Fargo Bank, Natl Ass'n, 776 F.3d 1343, 1348 (Fed. Cir. 2014). If applicant believes the dependent claims are directed towards patent eligible subject matter, they are invited to point out the specific limitations in the claim that are directed towards patent eligible subject matter. Claims for the other statutory classes are similarly analyzed and rejected under similar rationale.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over “Self-Supervised CycleGAN for Object-Preserving Image-to-Image Domain Adaptation” hereinafter D1 Published on 2020 in view of Mellado Bataller et al. [US 10902551 B1, hereinafter D2].
With regard to Claim 1,
D1 teach a method, comprising:
forming, by a computing device, a set of training parameters applicable to detection of two or more entities between and/or among a distribution of digitally stored entities from a plurality of digitally stored observations (D1, P.1, Introduction, “Deep learning networks have shown impressive successes on various computer vision tasks such as image classification [9,19,28] and semantic segmentation”, P.3, “CycleGAN has two paired generator-discriminator modules, which are capable of learning two mappings, i.e., from domain A to domain B {GAB, DB} and the inverse B to A {GBA, DA} . The generators (GAB, GBA) translate images between the source and target domains, while the discriminators (DA, DB) aim to distinguish the original data from the translated ones. Thereby, the generators and discriminators are gradually updated during this adversarial competition. As shown in Fig. 1, the original CycleGAN is supervised by two losses, i.e., adversarial loss Ladv and cycle-consistency loss Lcyc.”, Training a CycleGAN to detect entities as (e.g. cars, building, trees from two domains day and night) See at least Fig. 1, model is trained using parameters to allow objects detection in translated images (weights, adversarial loss and cycle-consistency loss), Fig. 2-3, P.5-6, 4.1, “Self-supervision formulation. As no preexistent label information is available for the self-supervised siamese network, the supervision is derived from the image data itself … During the training stage, the framework randomly selects two patches from the patch pool as the paired input of the siamese network … relative position of two patches can be used to supervise the proxy task that extracts features with content information, while the provenance information of the patches can be used to formulate the proxy task as a domain classification”, using self-supervised learning the system use provided images to identify features including domain features)
modifying, by the computing device, one or more training parameters of the set of training parameters to define a translation applicable to detection of real-world entities of a first environment corresponding to the two or more entities between and/or among the distribution of digitally stored entities from the plurality of digitally stored observations (Fig. 4, P.3, “CycleGAN has two paired generator-discriminator modules, which are capable of learning two mappings, i.e., from domain A to domain B {GAB, DB} and the inverse B to A {GBA, DA} . The generators (GAB, GBA) translate images between the source and target domains …”, “the generators and discriminators are gradually updated during this adversarial competition”, the continuous update is a modification to the initial training parameters, P.5, 4, “Similar to the original CycleGAN, our OP-GAN involves the adversarial and cycle-consistency losses to achieve unpaired I2I translation”, P.7, “minimum content distortion is a mandatory requirement in our domain adaptation task, the image-objects in source and translated images should be geometrically consistent (i.e., maintaining the shape and position of objects). Hence, we formulate the content consistency loss (Lcc) using the two content attention maps (~p) in L2 norm”, Also modifying CycleGAN model to include self-supervised constraint for maintaining the consistency of image-objects is parameters modification, P.9, “optimization of LS is performed in the same manner of Ladv fixing the siamese network (S) and DA/DB to optimize GBA/GAB first, and then optimize S and DA/DB respectively, with GBA/GAB fixed”, P.10, “generator, discriminator, and siamese network are iteratively trained for 200 epochs with the Adam solver”), the translation (convert images from cloudy to sunny Fig. 4) to be based, at least in part, on:
a first process of generating the two or more entities between and/or among the distribution of digitally stored entities from the plurality of digitally stored observations (Fig. 1, Fig. 3, generating images by the CycleGAN or OP-GAN that include similar entities but within different conditions (translation) as shown in Fig. 4 (Cloudy-to-sunny, night to day, etc.), Fig. 1, P.3, “CycleGAN has two paired generator-discriminator modules, which are capable of learning two mappings, i.e., from domain A to domain B {GAB, DB} and the inverse B to A {GBA, DA} . The generators (GAB, GBA) translate images between the source and target domains”, P.8, 4.2, “the proposed OP-GAN has cyclic generators (GAB, GBA)”, P.9, 4.3, “The optimization of LS is performed in the same manner of Ladv| fixing the siamese network (S) and DA/DB to optimize GBA/GAB first, and then optimize S and DA/DB respectively, with GBA/GAB fixed. Therefore, similar to the discriminators, our siamese network can directly pass the knowledge of image-objects to the generators”); and
a second process of discriminating between and/or among the generated two or more entities based, at least in part, on the modified one or more training parameters (discriminator distinguish between real and generated image (between domains), P.3, “CycleGAN has two paired generator-discriminator modules, which are capable of learning two mappings, i.e., from domain A to domain B {GAB, DB} and the inverse B to A {GBA, DA} . The generators (GAB, GBA) translate images between the source and target domains, while the discriminators (DA, DB) aim to distinguish the original data from the translated ones. Thereby, the generators and discriminators are gradually updated during this adversarial competition”, P.8, 4.2, “OP-GAN has cyclic generators (GAB, GBA) and corresponding discriminators (DB, DA), which have the same architectures as described in [36]. The … discriminators adopt PatchGAN [12,18] to provide patch-wise predictions of given image being real or fake, rather than classifying the whole image”, P.9, 4.3, “The optimization of LS is performed in the same manner of Ladv| fixing the siamese network (S) and DA/DB to optimize GBA/GAB first, and then optimize S and DA/DB respectively, with GBA/GAB fixed. Therefore, similar to the discriminators, our siamese network can directly pass the knowledge of image-objects to the generators”);
detecting, by the computing device using the translation, at least one physical entity in the second environment (P. 2, “maintaining the consistency of image-objects”, P.3, “The generators (GAB, GBA) translate images between the source and target domains”); and
automatically further modifying the one or more training parameters in a closed loop manner based at least in part on (i) the determined distribution of physical entities present in the second environment (Fig. 1, Fig. 3, P.3, “the generators and discriminators are gradually updated during this adversarial”, P. 10, ¶4, “The generator, discriminator, and siamese network are iteratively trained”); so as to improve detection of physical entities across differing environments (Fig. 4, (Cloudy-to-sunny, night to day, etc.), P. 2, “maintaining the consistency of image-objects”).
D1 does not explicitly teach sampling, by the computing device via one or more sensors during operation in a second environment, real-world observations to determine a distribution of physical entities present in the second environment; and detecting performance associated with the detected at least one physical entity.
D2 teach sampling, by the computing device via one or more sensors during operation in a second environment real-world observations (Col. 5, lines 10-18, “robotic system may include sensors for capturing information of the environment in which the robotic system is operating”, Col. 5, lines 23-24, “robotic system may capture images of the environment and may store the captured images for later use”, Col. 5, lines 29-34, Col. 8, lines 26-36) to determine a distribution of physical entities present in the second environment (Col. 14, lines 1-3, “prediction module 150 could detect a class imbalance in training database 540 or augmented image(s) 532”, Col. 16, lines 25-35, Col. 15, lines 45-47, “Prediction module 150 could then determine the frequency at which each of the provided object classes appears in augmented image(s) 532”, Col. 15, lines 51-59, “prediction module 150 may determine the frequency at which “fork” objects appear in augmented image(s) 532 and may determine the frequency at which “spoon” objects appear in augmented image(s) 532. After this, prediction module 150 could determine whether the frequency r any object class is below a threshold … based on the frequency at which each of the provided object classes appears in augmented image(s) 532”, Col. 16, lines 30-35, “prediction module 150 determining the frequency at which each of the provided object classes appears in the initial set of training data”);
Detecting, by the computing device using the translation, at least one physical entity in the second environment (Col. 5, lines 18-21, “robotic system may use the captured sensor information as input into the aforementioned predictive models, which may assist the robotic system with classifying/identifying objects in its environment”); and
automatically further modifying the one or more training parameters in a closed loop manner based responsive at least in part on to (i) the determination of the determined distribution of physical entities present in the second environment (Fig. 6, Col. 16, lines 10-12, “prediction module 150 identifies imbalances in its training data, temporarily suspends its training, and requests additional augmented images to balance its training data.”, Col. 17, lines 3-5, “prediction module 150 may resume training using the initial images from block 602 in addition to the augmented image(s) received at block 616”, Col. 4-5, lines 64-7, “upon detecting a class imbalance, the described system could pause or otherwise halt the training process of a predictive model … apply the aforementioned transformation module to generate augmented image(s) using the segmented object(s). These augmented image(s) could be added to the training data to create augmented training data. The described system could later resume the training process with the augmented training data”, system modify the training data which automatically change the training parameters (e.g. adjusting the input distribution which will impact the learned weights) based on the detected imbalance in the training data) and (ii) detection performance associated with the detected at least one physical entity so as to improve detection of physical entities across differing environments (Col. 17, lines 8-15, “ If the trained predictive models perform poorly on a particular class of objects (e.g., an area under ROC curve below 0.5 or an accuracy below 0.5), prediction module 150 may request from segmentation module 130/transformation module 140 additional augmented image(s) for the poorly performing class. Prediction module 150 may retrain the predictive models with these additional augmented image(s) to increase the overall performance of the predictive models”, Col. 14, lines 33-35, “ background database 520 may contain background images taken/captured from parks, offices, streets, playgrounds, beaches, homes, and so on”, Col. 5, lines 30-32, “the augmented images can help the robotic system identify objects and otherwise operate in previously unseen environments”).
D1 and D2 are analogous art to the claimed invention because they are from a similar field of endeavor of generating data to improve machine learning models by increasing data diversity. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify D1 resulting in resolutions as disclosed by D2 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify D1 as described above to prevent overfitting and address this lack of diversity is to generate more varied training data without becoming unduly time consuming and inefficient (D2 Col. 1, lines 5-6, Col. 3, lines 42-52).
With regard to Claim 2,
D1-D2 teach the method of claim 1, wherein the first process and form a generative adversarial network (GAN) process (P.2, ¶2, “GAN-based unpaired I2I domain adaptation methods, e.g., CycleGAN [36], DiscoGAN [13], and DualGAN [32] were recently proposed, where a cycle consistency constraint was applied to encourage bidirectional image translations with regularized structural output”).
The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 3,
D1-D2 teach the method of claim 2, further comprising iteratively repeating the first process and the second process to form a cycle-GAN process (P.2, ¶2, “GAN-based unpaired I2I domain adaptation methods, e.g., CycleGAN [36], DiscoGAN [13], and DualGAN [32] were recently proposed, where a cycle consistency constraint was applied to encourage bidirectional image translations with regularized structural output”).
The same motivation to combine for claim 2 equally applies for current claim.
With regard to Claim 4,
D1-D2 teach the method of claim 1, wherein the two or more entities between and/or among the distribution of digitally stored entities correspond to entities detectable within visual sensor data (D1, Figs. 3-4, D2, Col. 5, lines 10-18, “robotic system may include sensors for capturing information of the environment in which the robotic system is operating”, Col. 5, lines 23-24, “robotic system may capture images of the environment and may store the captured images for later use”) or audio sensor data.
The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 5,
D1-D2 teach the method of claim 1, wherein detecting the at least one physical entity comprises classifying the at least one physical entity using a trained neural network (D2, Col. 5, lines 18-21, “robotic system may use the captured sensor information as input into the aforementioned predictive models, which may assist the robotic system with classifying/identifying objects in its environment”, Col. 15, lines 45-47, “Prediction module 150 could then determine the frequency at which each of the provided object classes appears in augmented image(s) 532”, Col. 15, lines 51-59, “prediction module 150 may determine the frequency at which “fork” objects appear in augmented image(s) 532 and may determine the frequency at which “spoon” objects appear in augmented image(s) 532. After this, prediction module 150 could determine whether the frequency r any object class is below a threshold …”, Col. 16, lines 30-35, “prediction module 150 determining the frequency at which each of the provided object classes appears in the initial set of training data”, Col. 13, lines 14-16, “Prediction module 150 may contain one or more predictive models including, but not limited to: an artificial neural network”, Col. 13, lines 20-23, “During a training phase, the predictive models of prediction module 150 may be trained …“).
The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 6,
D1-D2 teach the method of claim 1, wherein automatically further modifying the training parameters comprises narrowing one or more parameter ranges responsive to incorrect classification of the detected at least one physical entity (D2, Fig. 6, Col. 16, lines 10-12, “prediction module 150 identifies imbalances in its training data, temporarily suspends its training, and requests additional augmented images to balance its training data.”, Col. 17, lines 8-15, “ If the trained predictive models perform poorly on a particular class of objects (e.g., an area under ROC curve below 0.5 or an accuracy below 0.5), prediction module 150 may request from segmentation module 130/transformation module 140 additional augmented image(s) for the poorly performing class. Prediction module 150 may retrain the predictive models with these additional augmented image(s) to increase the overall performance of the predictive models”, retraining based on poor classification refine and adjust learned parameter values and decision boundaries to reduce future classification errors).
The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 7,
D1-D2 teach the method of claim 1, wherein automatically further modifying comprises weighting the training parameters based on frequencies within the determined distribution Col. 15, lines 45-47, “Prediction module 150 could then determine the frequency at which each of the provided object classes appears in augmented image(s) 532”, Col. 15, lines 51-59, “prediction module 150 may determine the frequency at which “fork” objects appear in augmented image(s) 532 and may determine the frequency at which “spoon” objects appear in augmented image(s) 532. After this, prediction module 150 could determine whether the frequency r any object class is below a threshold …”, Col. 16, lines 30-35, “prediction module 150 determining the frequency at which each of the provided object classes appears in the initial set of training data”, Col. 17, lines 8-15, “ If the trained predictive models perform poorly on a particular class of objects (e.g., an area under ROC curve below 0.5 or an accuracy below 0.5), prediction module 150 may request from segmentation module 130/transformation module 140 additional augmented image(s) for the poorly performing class. Prediction module 150 may retrain the predictive models with these additional augmented image(s) to increase the overall performance of the predictive models”).
The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 8,
D1-D2 teach the method of claim 1, wherein sampling real world observations include querying a real word entities database (D2, Fig. 4A, Col. 14, 6-10, “transformation module 140 could receive (i) ground truth object properties 512 from ground truth object database 510”, Col. 14, lines 12-14, “Ground truth object database 510 may include one or more ground truth tables, such as ground truth table 400, each containing ground truth property values for objects”, Col. 4, lines 33-40, Col. 12, lines 1-6, Col. 5, lines 10-18, “robotic system may include sensors for capturing information of the environment in which the robotic system is operating. For example, the sensors may monitor the environment in real time, and detect obstacles, elements of the terrain, weather conditions, temperature, or other aspects of the environment”, Col. 5, lines 23-24, “robotic system may capture images of the environment and may store the captured images for later use”, Col. 14, 17-25, “image analysis system can provide the ground truth property values to populate ground truth object database 510. Such an image analysis system may be operable to receive a set of labeled images and responsively analyze objects in the set of labeled images to determine ground truth property values for each object”).
The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 9,
D1-D2 teach the method of claim 1, wherein first environment comprises an office environment and the second environment comprises a domestic environment (D1, Fig. 4, cloudy to sunny, night to day, multicenter colonoscopy, P.1, Introduction, “Deep learning networks have shown impressive successes on various computer vision tasks such as image classification [9,19,28] and semantic segmentation”, P.3, “CycleGAN has two paired generator-discriminator modules, which are capable of learning two mappings, i.e., from domain A to domain B {GAB, DB} and the inverse B to A {GBA, DA} . The generators (GAB, GBA) translate images between the source and target domains, while the discriminators (DA, DB) aim to distinguish the original data from the translated ones. Thereby, the generators and discriminators are gradually updated during this adversarial competition” P.7, “minimum content distortion is a mandatory requirement in our domain adaptation task, the image-objects in source and translated images should be geometrically consistent (i.e., maintaining the shape and position of objects). Hence, we formulate the content consistency loss (Lcc) using the two content attention maps (~p) in L2 norm””, D2, Col. 5, lines 29-34, “the robotic system may operate in a limited set of environments—and thus only captures images from the limited set of environments”, D2, Col. 4, lines 49-59 “transformation module may contain background images taken/captured from parks, offices, streets, playgrounds, beaches, homes, and so on … for training predictive model”, Col. 3, lines 43-52, “to address this lack of diversity is to generate more varied training data. In the field of object classification and detection, this would typically involve collecting images from a wide variety of environments (e.g., living rooms, dining rooms, outdoors, offices spaces, conference rooms, etc.)”, Col. 4, lines 33-38 “Background database 520 could include background images taken/captured from a wide variety of environments. For instance, background database 520 may contain background images taken/captured from parks, offices, streets, playgrounds, beaches, homes, and so on. The variability images in background database 520 helps to further increase the diversity of augmented image(s)”).
The same motivation to combine for claim 1 equally applies for current claim.
The Examiner further notes that the [office environment and domain environment] is non-functional descriptive material and is not functionally involved in the steps recited. Thus, this descriptive material will not distinguish the claimed invention from the prior art in terms of patentability. See In re Gulack, 703 F.2d 1381, 218 USPQ 401, 403 (Fed. Cir. 1983); In re Lowry, 32 F.3d 1579, 32 USPQ2d 1031 (Fed. Cir. 1994).
With regard to Claim 10,
Claim 10 is similar in scope to claim 1; therefore it is rejected under similar rationale. D1 further disclose An apparatus, comprising: a processor coupled to at least one memory device (D1, P.1, Introduction, “Deep learning networks have shown impressive successes on various computer vision tasks such as image classification [9,19,28] and semantic segmentation”).
The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 11,
Claim 11 is similar in scope to claim 2; therefore it is rejected under similar rationale.
With regard to Claim 12,
Claim 12 is similar in scope to claim 4; therefore it is rejected under similar rationale. With regard to Claim 13,
Claim 13 is similar in scope to claim 5; therefore it is rejected under similar rationale. With regard to Claim 14,
Claim 14 is similar in scope to claim 7; therefore it is rejected under similar rationale. With regard to Claim 15,
Claim 15 is similar in scope to claim 8; therefore it is rejected under similar rationale.
With regard to Claim 16,
Claim 16 is similar in scope to claim 9; therefore it is rejected under similar rationale.
With regard to Claim 17,
Claim 17 is similar in scope to claim 1; therefore it is rejected under similar rationale. D1 further disclose An article, comprising: a non-transitory storage medium, having instructions stored thereon, which, when executed by a computer processor coupled to at least one memory, are operable (D1, P.1, Introduction, “Deep learning networks have shown impressive successes on various computer vision tasks such as image classification [9,19,28] and semantic segmentation”).
With regard to Claim 18,
Claim 18 is similar in scope to claim 2; therefore it is rejected under similar rationale.
With regard to Claim 19,
Claim 19 is similar in scope to claim 4; therefore it is rejected under similar rationale.
With regard to Claim 20,
Claim 20 is similar in scope to claim 9; therefore it is rejected under similar rationale.
Response to Arguments
Applicant argue that the recited claims do not fall into any of the abstract ideas groupings. As the limitations pertaining to sensors obtaining real-world observations, detecting physical entities, automatically modifying training parameters in a closed-loop manner, and improving detection across environments.
Examiner respectfully disagrees, The argued limitations is not part of the abstract idea as using a sensor is considered as merely using a computer as a tool to perform an abstract idea MPEP 2106.04(d) and training a model is a high-generic computer software process of training data. These limitations do not amount to significantly more than the judicial exception, see MPEP 2106.05 (f). However, detecting physical entities in captured data is a mental process that a human is capable of doing.
Applicant argue that the current amendments is an improvement to technology by improving detection accuracy across environments.
Examiner respectfully disagrees, the arguments does not clarify the current state of the art and how the claims in view of the specification provide such improvement.
Applicant argue that claims as amended goes beyond, in a meaningful way, generally linking the use of a judicial exception to a particular technological environment, such that the claims as a whole are more than a drafting effort designed to monopolize the exception, as is clearly shown in the patent application, as filed. For example, amended independent claim 1 recites a detailed description of how to detect real-world entities in at least two environments, including sensors obtaining real-world observations, detecting physical entities, automatically modifying training parameters in a closed-loop manner, and improving detection across environments. This is not insignificant activity and is not meant to monopolize a technical environment.
Examiner respectfully disagrees, the claims includes limitations that human mind is capable of doing as detecting real world entities in at least two environments, detecting physical entities, modifying training parameters and the extra elements of using image sensors is considered as merely using a computer as a tool to perform an abstract idea MPEP 2106.04(d) and training a model to improve is a high-generic computer software process of training data. These limitations do not amount to significantly more than the judicial exception, see MPEP 2106.05 (f).
Applicant argue that claimed subject matter represents improvements over existing technology, and thus, claimed subject matter is not merely well-known, routine or conventional. For example, the limitations of amended claim 1, such as those discussed above and further discussed below, are not well known, routine and/or conventional. claimed subject matter of an ordered combination of elements provides a non-conventional architecture: sensor detection performance evaluation parameter update that (a) improves the accuracy and efficiency of detection of real-world entities; (b) improves the accuracy of such detections; and (c) simplifies the computational complexity of such detections. Specific examples are provided in the specification of the present patent application in the areas of autonomous automobile navigation/operation and automatic housecleaning devices, although subject matter is not limited in scope in these respects. In addition, the recited limitations are part of the abstract idea and not extra elements. The improvement to a technological field need to be based on the extra elements and not the abstract idea (MPEP 2106.05(h), “Another consideration when determining whether a claim integrates the judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than generally linking the use of a judicial exception to a particular technological environment or field of use”).
Examiner respectfully disagrees,
the current claims do not specify any field of use. In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., autonomous automobile navigation/operation and automatic housecleaning devices) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Applicant arguments related to the absence of motivation to combine art is not persuasive as the examiner provided a clear motivations in the body of the rejection. The examiner refer the applicant to the rejection for the detailed motivations. In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, D1 and D2 are analogous art to the claimed invention because they are from a similar field of endeavor of generating data to improve machine learning models by increasing data diversity. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify D1 resulting in resolutions as disclosed by D2 with a reasonable expectation of success. One of ordinary skill in the art would be motivated to modify D1 as described above to prevent overfitting and address this lack of diversity is to generate more varied training data without becoming unduly time consuming and inefficient (D2 Col. 1, lines 5-6, Col. 3, lines 42-52).
Applicant argue that D1 does not state that it shows "sampling, by the computing device via one or more sensors during operation in a second environment, real-world observations to determine a distribution of physical entities present in the second environment; detecting, by the computing device using the translation, at least one physical entity in the second environment; and automatically further modifying the one or more training parameters in a closed-loop manner based at least in part on (i) the determined distribution of physical entities present in the second environment and (ii) detection performance associated with the detected at least one physical entity so as to improve detection of physical entities across differing environments" as recited in claim 1, as amended.
Examiner respectfully disagrees,
The provided rejection is an obvious rejection that rely on D2 to modify D1 to teach the whole claim limitations. D1 teach detecting, by the computing device using the translation, at least one physical entity in the second environment (P. 2, “maintaining the consistency of image-objects”, P.3, “The generators (GAB, GBA) translate images between the source and target domains”); and automatically further modifying the one or more training parameters in a closed loop manner based at least in part on (i) the determined distribution of physical entities present in the second environment (Fig. 1, Fig. 3, P.3, “the generators and discriminators are gradually updated during this adversarial”, P. 10, ¶4, “The generator, discriminator, and siamese network are iteratively trained”); so as to improve detection of physical entities across differing environments (Fig. 4, (Cloudy-to-sunny, night to day, etc.), P. 2, “maintaining the consistency of image-objects”).
D1 does not explicitly teach sampling, by the computing device via one or more sensors during operation in a second environment, real-world observations to determine a distribution of physical entities present in the second environment; and detecting performance associated with the detected at least one physical entity.
D2 teach sampling, by the computing device via one or more sensors during operation in a second environment real-world observations (Col. 5, lines 10-18, “robotic system may include sensors for capturing information of the environment in which the robotic system is operating”, Col. 5, lines 23-24, “robotic system may capture images of the environment and may store the captured images for later use”, Col. 5, lines 29-34, Col. 8, lines 26-36) to determine a distribution of physical entities present in the second environment (Col. 14, lines 1-3, “prediction module 150 could detect a class imbalance in training database 540 or augmented image(s) 532”, Col. 16, lines 25-35, Col. 15, lines 45-47, “Prediction module 150 could then determine the frequency at which each of the provided object classes appears in augmented image(s) 532”, Col. 15, lines 51-59, “prediction module 150 may determine the frequency at which “fork” objects appear in augmented image(s) 532 and may determine the frequency at which “spoon” objects appear in augmented image(s) 532. After this, prediction module 150 could determine whether the frequency r any object class is below a threshold … based on the frequency at which each of the provided object classes appears in augmented image(s) 532”, Col. 16, lines 30-35, “prediction module 150 determining the frequency at which each of the provided object classes appears in the initial set of training data”);
Detecting, by the computing device using the translation, at least one physical entity in the second environment (Col. 5, lines 18-21, “robotic system may use the captured sensor information as input into the aforementioned predictive models, which may assist the robotic system with classifying/identifying objects in its environment”); and
automatically further modifying the one or more training parameters in a closed loop manner based responsive at least in part on to (i) the determination of the determined distribution of physical entities present in the second environment (Fig. 6, Col. 16, lines 10-12, “prediction module 150 identifies imbalances in its training data, temporarily suspends its training, and requests additional augmented images to balance its training data.”, Col. 17, lines 3-5, “prediction module 150 may resume training using the initial images from block 602 in addition to the augmented image(s) received at block 616”, Col. 4-5, lines 64-7, “upon detecting a class imbalance, the described system could pause or otherwise halt the training process of a predictive model … apply the aforementioned transformation module to generate augmented image(s) using the segmented object(s). These augmented image(s) could be added to the training data to create augmented training data. The described system could later resume the training process with the augmented training data”, system modify the training data which automatically change the training parameters (e.g. adjusting the input distribution which will impact the learned weights) based on the detected imbalance in the training data) and (ii) detection performance associated with the detected at least one physical entity so as to improve detection of physical entities across differing environments (Col. 17, lines 8-15, “ If the trained predictive models perform poorly on a particular class of objects (e.g., an area under ROC curve below 0.5 or an accuracy below 0.5), prediction module 150 may request from segmentation module 130/transformation module 140 additional augmented image(s) for the poorly performing class. Prediction module 150 may retrain the predictive models with these additional augmented image(s) to increase the overall performance of the predictive models”, Col. 14, lines 33-35, “ background database 520 may contain background images taken/captured from parks, offices, streets, playgrounds, beaches, homes, and so on”, Col. 5, lines 30-32, “the augmented images can help the robotic system identify objects and otherwise operate in previously unseen environments”). D1 and D2 are analogous art to the claimed invention because they are from a similar field of endeavor of generating data to improve machine learning models by increasing data diversity. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify D1 resulting in resolutions as disclosed by D2 with a reasonable expectation of success. One of ordinary skill in the art would be motivated to modify D1 as described above to prevent overfitting and address this lack of diversity is to generate more varied training data without becoming unduly time consuming and inefficient (D2 Col. 1, lines 5-6, Col. 3, lines 42-52). In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Applicant arguments (Remarks P. 20) regarding the objective difference between D1 and the current invention is mainly related to the intended use of the devices and a claim containing a "recitation with respect to the manner in which a claimed apparatus is intended to be employed does not differentiate the claimed apparatus from a prior art apparatus" if the prior art apparatus teaches all the structural limitations of the claim. Ex parte Masham, 2 USPQ2d 1647 (Bd. Pat. App. & Inter. 1987) (MPEP 2114(II)).
Examiner respectfully disagrees,
The provided rejection is an obvious rejection that rely on D2 to modify D1 to teach the whole claim limitations. D1 teach detecting, by the computing device using the translation, at least one physical entity in the second environment (P. 2, “maintaining the consistency of image-objects”, P.3, “The generators (GAB, GBA) translate images between the source and target domains”); and automatically further modifying the one or more training parameters in a closed loop manner based at least in part on (i) the determined distribution of physical entities present in the second environment (Fig. 1, Fig. 3, P.3, “the generators and discriminators are gradually updated during this adversarial”, P. 10, ¶4, “The generator, discriminator, and siamese network are iteratively trained”); so as to improve detection of physical entities across differing environments (Fig. 4, (Cloudy-to-sunny, night to day, etc.), P. 2, “maintaining the consistency of image-objects”).
D1 does not explicitly teach sampling, by the computing device via one or more sensors during operation in a second environment, real-world observations to determine a distribution of physical entities present in the second environment; and detecting performance associated with the detected at least one physical entity.
D2 teach sampling, by the computing device via one or more sensors during operation in a second environment real-world observations (Col. 5, lines 10-18, “robotic system may include sensors for capturing information of the environment in which the robotic system is operating”, Col. 5, lines 23-24, “robotic system may capture images of the environment and may store the captured images for later use”, Col. 5, lines 29-34, Col. 8, lines 26-36) to determine a distribution of physical entities present in the second environment (Col. 14, lines 1-3, “prediction module 150 could detect a class imbalance in training database 540 or augmented image(s) 532”, Col. 16, lines 25-35, Col. 15, lines 45-47, “Prediction module 150 could then determine the frequency at which each of the provided object classes appears in augmented image(s) 532”, Col. 15, lines 51-59, “prediction module 150 may determine the frequency at which “fork” objects appear in augmented image(s) 532 and may determine the frequency at which “spoon” objects appear in augmented image(s) 532. After this, prediction module 150 could determine whether the frequency r any object class is below a threshold … based on the frequency at which each of the provided object classes appears in augmented image(s) 532”, Col. 16, lines 30-35, “prediction module 150 determining the frequency at which each of the provided object classes appears in the initial set of training data”);
Detecting, by the computing device using the translation, at least one physical entity in the second environment (Col. 5, lines 18-21, “robotic system may use the captured sensor information as input into the aforementioned predictive models, which may assist the robotic system with classifying/identifying objects in its environment”); and
automatically further modifying the one or more training parameters in a closed loop manner based responsive at least in part on to (i) the determination of the determined distribution of physical entities present in the second environment (Fig. 6, Col. 16, lines 10-12, “prediction module 150 identifies imbalances in its training data, temporarily suspends its training, and requests additional augmented images to balance its training data.”, Col. 17, lines 3-5, “prediction module 150 may resume training using the initial images from block 602 in addition to the augmented image(s) received at block 616”, Col. 4-5, lines 64-7, “upon detecting a class imbalance, the described system could pause or otherwise halt the training process of a predictive model … apply the aforementioned transformation module to generate augmented image(s) using the segmented object(s). These augmented image(s) could be added to the training data to create augmented training data. The described system could later resume the training process with the augmented training data”, system modify the training data which automatically change the training parameters (e.g. adjusting the input distribution which will impact the learned weights) based on the detected imbalance in the training data) and (ii) detection performance associated with the detected at least one physical entity so as to improve detection of physical entities across differing environments (Col. 17, lines 8-15, “ If the trained predictive models perform poorly on a particular class of objects (e.g., an area under ROC curve below 0.5 or an accuracy below 0.5), prediction module 150 may request from segmentation module 130/transformation module 140 additional augmented image(s) for the poorly performing class. Prediction module 150 may retrain the predictive models with these additional augmented image(s) to increase the overall performance of the predictive models”, Col. 14, lines 33-35, “ background database 520 may contain background images taken/captured from parks, offices, streets, playgrounds, beaches, homes, and so on”, Col. 5, lines 30-32, “the augmented images can help the robotic system identify objects and otherwise operate in previously unseen environments”). D1 and D2 are analogous art to the claimed invention because they are from a similar field of endeavor of generating data to improve machine learning models by increasing data diversity. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify D1 resulting in resolutions as disclosed by D2 with a reasonable expectation of success. One of ordinary skill in the art would be motivated to modify D1 as described above to prevent overfitting and address this lack of diversity is to generate more varied training data without becoming unduly time consuming and inefficient (D2 Col. 1, lines 5-6, Col. 3, lines 42-52). In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Applicant argue that D1 describes adversarial learning used to translate images between domains. For example, the Examiner contends "CycleGAN has two paired generator-discriminator modules... The generators translate images between the source and target domains, while the discriminators aim to distinguish the original data from the translated ones." See, for example, page 14 of the Office Action. Therefore, D1 appears to teach offline model training. Assignee respectfully submits that D1 does not teach or suggest adaptation based on deployed sensing, for example.
Examiner respectfully disagrees, D1 is not relied upon alone to teach the complete translation framework, as D1 expressly teach generator and discriminator translation models that iteratively update during adversarial training across different environments. D2 is relied upon to modify D1 to include sensor based sampling of real world observations and determination of distribution of physical entities in the second environment See e.g. captured images stored for later use, and resuming retraining responsive to a detected imbalance or poor performance. Thus the combination disclose the argued limitation. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Applicant argue that D2 also does not state that it shows "sampling, by the computing device via one or more sensors during operation in a second environment, real-world observations to determine a distribution of physical entities present in the second environment; detecting, by the computing device using the translation, at least one physical entity in the second environment; and automatically further modifying the one or more training parameters in closed-loop manner based at least in part on (i) the determined distribution of physical entities present in the second environment and (ii) detection performance associated with the detected at least one physical entity SO as to improve detection of physical entities across differing environments" as recited in claim 1, as amended.
Examiner respectfully disagrees,
The provided rejection is an obvious rejection that rely on D2 to modify D1 to teach the whole claim limitations. D1 teach detecting, by the computing device using the translation, at least one physical entity in the second environment (P. 2, “maintaining the consistency of image-objects”, P.3, “The generators (GAB, GBA) translate images between the source and target domains”); and automatically further modifying the one or more training parameters in a closed loop manner based at least in part on (i) the determined distribution of physical entities present in the second environment (Fig. 1, Fig. 3, P.3, “the generators and discriminators are gradually updated during this adversarial”, P. 10, ¶4, “The generator, discriminator, and siamese network are iteratively trained”); so as to improve detection of physical entities across differing environments (Fig. 4, (Cloudy-to-sunny, night to day, etc.), P. 2, “maintaining the consistency of image-objects”). D1 does not explicitly teach sampling, by the computing device via one or more sensors during operation in a second environment, real-world observations to determine a distribution of physical entities present in the second environment; and detecting performance associated with the detected at least one physical entity.
D2 teach sampling, by the computing device via one or more sensors during operation in a second environment real-world observations (Col. 5, lines 10-18, “robotic system may include sensors for capturing information of the environment in which the robotic system is operating”, Col. 5, lines 23-24, “robotic system may capture images of the environment and may store the captured images for later use”, Col. 5, lines 29-34, Col. 8, lines 26-36) to determine a distribution of physical entities present in the second environment (Col. 14, lines 1-3, “prediction module 150 could detect a class imbalance in training database 540 or augmented image(s) 532”, Col. 16, lines 25-35, Col. 15, lines 45-47, “Prediction module 150 could then determine the frequency at which each of the provided object classes appears in augmented image(s) 532”, Col. 15, lines 51-59, “prediction module 150 may determine the frequency at which “fork” objects appear in augmented image(s) 532 and may determine the frequency at which “spoon” objects appear in augmented image(s) 532. After this, prediction module 150 could determine whether the frequency r any object class is below a threshold … based on the frequency at which each of the provided object classes appears in augmented image(s) 532”, Col. 16, lines 30-35, “prediction module 150 determining the frequency at which each of the provided object classes appears in the initial set of training data”); detecting, by the computing device using the translation, at least one physical entity in the second environment (Col. 5, lines 18-21, “robotic system may use the captured sensor information as input into the aforementioned predictive models, which may assist the robotic system with classifying/identifying objects in its environment”); and
automatically further modifying the one or more training parameters in a closed loop manner based responsive at least in part on to (i) the determination of the determined distribution of physical entities present in the second environment (Fig. 6, Col. 16, lines 10-12, “prediction module 150 identifies imbalances in its training data, temporarily suspends its training, and requests additional augmented images to balance its training data.”, Col. 17, lines 3-5, “prediction module 150 may resume training using the initial images from block 602 in addition to the augmented image(s) received at block 616”, Col. 4-5, lines 64-7, “upon detecting a class imbalance, the described system could pause or otherwise halt the training process of a predictive model … apply the aforementioned transformation module to generate augmented image(s) using the segmented object(s). These augmented image(s) could be added to the training data to create augmented training data. The described system could later resume the training process with the augmented training data”, system modify the training data which automatically change the training parameters (e.g. adjusting the input distribution which will impact the learned weights) based on the detected imbalance in the training data) and (ii) detection performance associated with the detected at least one physical entity so as to improve detection of physical entities across differing environments (Col. 17, lines 8-15, “ If the trained predictive models perform poorly on a particular class of objects (e.g., an area under ROC curve below 0.5 or an accuracy below 0.5), prediction module 150 may request from segmentation module 130/transformation module 140 additional augmented image(s) for the poorly performing class. Prediction module 150 may retrain the predictive models with these additional augmented image(s) to increase the overall performance of the predictive models”, Col. 14, lines 33-35, “ background database 520 may contain background images taken/captured from parks, offices, streets, playgrounds, beaches, homes, and so on”, Col. 5, lines 30-32, “the augmented images can help the robotic system identify objects and otherwise operate in previously unseen environments”). D1 and D2 are analogous art to the claimed invention because they are from a similar field of endeavor of generating data to improve machine learning models by increasing data diversity. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify D1 resulting in resolutions as disclosed by D2 with a reasonable expectation of success. One of ordinary skill in the art would be motivated to modify D1 as described above to prevent overfitting and address this lack of diversity is to generate more varied training data without becoming unduly time consuming and inefficient (D2 Col. 1, lines 5-6, Col. 3, lines 42-52). In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Applicant argue that D2 appears to show a transformation module 140 that may request and receive background images from a background database from which augmented images may be generated. Such images may show backgrounds from different environments that might be appropriate for particular objects. See, for example, column 14, lines 39-56 of D2. However, it is respectfully submitted that D2 does not state that it shows sampling observations via sensors during operation, detecting physical entities in the environment, and automatically modifying parameters in a closed-loop feedback process based on detection performance.
Examiner respectfully disagrees, D2 is not relied upon alone to teach the complete translation framework, as D1 expressly teach generator and discriminator translation models that iteratively update during adversarial training across differing environments. D2 is relied upon to modify D1 to include sensor based sampling of real world observations and determination of distribution of physical entities in the second environment See e.g. captured images stored for later use, and resuming retraining responsive to a detected imbalance or poor performance. Thus the combination disclose the argued limitation. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
As to the remaining dependent claims, applicant argue that they are allowable due to their respective direct and indirect dependencies upon one of the aforementioned Independent claims. The examiner respectfully disagrees, Independent claims were not allowable as stated in the paragraph above in this “Response to Arguments” section in this office action.
Conclusion
The prior art made of record and not relied upon is considered pertinent to the applicant’s disclosure.
US Patent Application Publication No. 2022/0405530 filed by Metwaly et al . that disclose the ability to use a generative adversarial network (GAN) that predicts color intensities within a low-light image for synthetic domain See at least Abstract, ¶18, Fig. 3-5.
Examiner has pointed out particular references contained in the prior arts of record in the body of this action for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and Figures may apply as well. It is respectfully requested from the applicant, in preparing the response, to consider fully the entire references as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior arts or disclosed by the examiner. It is noted that any citation to specific pages, columns, figures, or lines in the prior art references any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331-33, 216 USPQ 1038-39 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)).
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMED ABOU EL SEOUD whose telephone number is (303)297-4285. The examiner can normally be reached Monday-Thursday 9:00am-6:00pm MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMED ABOU EL SEOUD/Primary Examiner, Art Unit 2148
Prosecution Insights