DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 10/08/2025 has been entered.
Response to Arguments
Applicant's arguments filed 10/08/2025 have been fully considered but they are not persuasive.
Regarding the 103 rejections, applicant's arguments filed with respect to the prior art rejections have been fully considered but they are moot. Applicant has amended the claims to recite new combinations of limitations. Applicant's arguments are directed at the amendment. Please see below for new grounds of rejection, necessitated by Amendment.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 12-15, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Sohn, et al., US Pre-Grant Publication 2019/0065853A1 (“Sohn”) in view of Yasutomi, et al., US Pre-Grant Publication 2018/0349741A1 (“Yasutomi”) and further in view of Rothberg, et al., US Pre-Grant Publication 2019/0347523A1 (“Rothberg”).
Regarding claim 1 and analogous claims 19 and 20, Sohn discloses:
A learning apparatus comprising a processor configured with a program to perform operations comprising: (Sohn, claim 1, “an object recognition system in communication with the camera, the object recognition system including a processor for executing: [A learning apparatus comprising a processor configured with a program to perform operations comprising:]”).
operation as a data acquiring unit configured to acquire a plurality of learning data sets that are each constituted by a combination of training data, metadata indicating an attribute regarding acquisition of the training data, (Sohn, ⁋39, “The system/method can process input images from more than one perspective of an environment. For example, an imaging device, such as, e.g., a camera, can capture images [operation as a data acquiring unit configured to acquire] of a first set of images of an environment having a first perspective or viewpoint, such as, e.g., a first angle of elevation, and a second set of images of the environment having a second perspective or viewpoint. Therefore, the system can include surveillance images including a first surveillance image 10 a and a second surveillance image 10 b along with training images [a plurality of learning data sets that are each constituted by a combination of training data,] such as, e.g., a source image 10 c, where the first surveillance image 10 a and the second surveillance image 10 b have different viewpoints; whether the image comes from the first or second perspective is interpreted as metadata/domain [metadata indicating an attribute regarding acquisition of the training data,].”).
and correct answer data indicating a feature included in the training data that corresponds to a correct answer to an estimation task with respect to the training data; and (Sohn, ⁋29, “the object recognition system 500 is trained with widely available and cheap labelled images [and correct answer data indicating a feature included in the training data]” and Sohn, ⁋35, “The domain adaptation module 520 can then generate classifications corresponding to vehicle makes and/or modules based on the training from the labeled images of the database 630; labeled images are interpreted as the correct answer data and classification is interpreted as a estimation task that corresponds to labeled images, or correct answer data, because classification training requires trying to match an input to one of the labeled images [that corresponds to a correct answer to an estimation task with respect to the training data; and].”).
operation as a learning processing unit configured to perform machine learning of a learning model (Sohn, ⁋72, “Thus, the domain adaptation module 520 [operation as a learning processing unit] trains the feature extractor 300 to extract the same features from the surveillance image 10 and the style and view augmented source image 13 when each of the surveillance image 10 and the style and view augmented source image 13 contain the same objects [configured to perform machine learning of a learning model].”).
including a first…model for extracting domain-common features, a second…model for extracting domain-specific features, (Sohn, ⁋73, “To train the feature extractor 300, a convolutional neural network (CNN) 301 [including a first…model] extracts surveillance features 20 [for extracting domain-common features,] from the surveillance image 10 while another CNN 302 [a second… model] concurrently extracts augmented source features 23 [for extracting domain-specific features,] from the style and view augmented source image 13. Each of the CNN 301 and 302 can be, e.g., any neural network for extracting feature representations form input images.”).
a first metadata identifier, a second metadata identifier, and an estimator, wherein (Sohn, ⁋75, “According to the joint parameterization, the domain adaptation training unit 400; element 400 is interpreted as the estimator as it produces predicted outputs [and an estimator, wherein] includes a classifier 401 for determining both classification and domain discrimination [a first metadata identifier,] for the surveillance features 20, and a classifier 402 for determining both classification and domain discrimination [a second metadata identifier,] for the augmented source features 23.”).
the first…model is configured to convert input data that is given into it to a first feature amount of the domain-common features, the second…model is configured to convert the input data to a second feature amount of the domain-specific features, (Sohn, ⁋73, “To train the feature extractor 300, a convolutional neural network (CNN) 301 [the first…model] extracts surveillance features 20 from the surveillance image 10 [is configured to convert input data that is given into it to a first feature amount of the domain-common features,] while another CNN 302 [a second…model] concurrently extracts augmented source features 23 from the style and view augmented source image 13 [is configured to convert the input data to a second feature amount of the domain-specific features,]. Each of the CNN 301 and 302 can be, e.g., any neural network for extracting feature representations form input images.”).
the first metadata identifier is configured to receive an output value of the first…model, and identify an attribute regarding acquisition of the input data from the first feature amount, (Sohn. ⁋76, “the classifiers 401 [the first metadata identifier] and 402 can each generate outputs that include, e.g., entries for class scores corresponding to each classification as well as an additional entry for a domain classification score corresponding to domain discrimination; domain discrimination is interpreted as identifying an attribute regarding where the input data came from [and identify an attribute regarding acquisition of the input data from the first feature amount,]. Thus, the classifier 401 and the classifier 402 will each include, e.g., parameters for determining both feature classifications and domain classification for the surveillance features 20 and the augmented source features, respectively [is configured to receive an output value of the first…model].”).
the second metadata identifier is configured to receive an output value of the second…model, and identify an attribute regarding acquisition of the input data from the second feature amount, (Sohn. ⁋76, “the classifiers 401 and 402 [the second metadata identifier] can each generate outputs that include, e.g., entries for class scores corresponding to each classification as well as an additional entry for a domain classification score corresponding to domain discrimination; domain discrimination is interpreted as identifying an attribute regarding where the input data came from [and identify an attribute regarding acquisition of the input data from the second feature amount,]. Thus, the classifier 401 and the classifier 402 will each include, e.g., parameters for determining both feature classifications and domain classification for the surveillance features 20 and the augmented source features, respectively [is configured to receive an output value of the second…model].”).
the estimator is configured to receive output values of the first…model and the second…model, and estimate a feature included in the input data from the first feature amount and the second feature amount, (Sohn, see Figure 5., The estimator is interpreted as being element 400 and element 400 takes the output values, or feature amounts, from the first and second models to produce predicted class labels [the estimator is configured to receive output values of the first…model and the second…model, and estimate a feature included in the input data from the first feature amount and the second feature amount,]).
performing the machine learning comprises: first training the second…model and the second metadata identifier such that, with respect to each learning data set, an identification result obtained from the second metadata identifier by giving the training data to the second…model (Sohn, ⁋83, “Thus, the classifiers 401 and 402 are trained via cross entropy loss, and the loss of the classifiers 401 and 402 can be used to then jointly train the parameters of the CNNs 301 and 302 [performing the machine learning comprises: first training the second…model and the second metadata identifier such that, an identification result obtained from the second metadata identifier by giving the training data to the second…model].”).
matches the metadata; (Sohn, ⁋78, “Thus, the loss function for the classifier parameters θC include both a source domain classifier 402 term, as well as an N+1 entry of the target domain classifier 401. The N+1 entry in each classifier 401 and 402 is the entry provided for domain discrimination classification. Thus, optimizing the parameters for the classifiers 401 and 402 includes optimizing for a domain class; optimizing for a domain class is interpreted as training when matching the metadata/domain and training the classifiers in turn trains the CNNs, which are interpreted as the encoders [matches the metadata;] in addition to label classes y.”).
second training the first…model, the second…model, and the estimator such that, (Sohn, ⁋83, “Thus, the classifiers 401 and 402; classifiers 401 and 402 make up element 400 which is interpreted as the estimator [and the estimator such that,] are trained via cross entropy loss, and the loss of the classifiers 401 and 402 can be used to then jointly train the parameters of the CNNs 301 and 302 [second training the first…model, the second…model]”).
with respect to each learning data set, an estimation result obtained from the estimator by giving the training data to the first…model and the second…model matches the correct answer data; (Sohn, ⁋76, “As a result of using classifiers without discriminators, the classifiers 401 and 402 [estimator] can each generate outputs that include, e.g., entries for class scores corresponding to each classification as well as an additional entry for a domain classification score corresponding to domain discrimination. Thus, the classifier 401 and the classifier 402 will each include, e.g., parameters for determining both feature classifications [with respect to each learning data set, an estimation result obtained from the estimator…matches the correct answer data;] and domain classification for the surveillance features 20 and the augmented source features, respectively; the surveillance and augmented source features are outputted from the first and second models after being given training data [by giving the training data to the first…model and the second…model].”).
third training the first metadata identifier such that, with respect to each learning data set, an identification result obtained from the first metadata identifier by giving the training data to the first…model (Sohn, ⁋83, “Thus, the classifiers 401 [third training the first metadata identifier] and 402 are trained via cross entropy loss, and the loss of the classifiers 401 and 402 can be used to then jointly train the parameters of the CNNs 301 and 302 [such that, with respect to each learning data set, an identification result obtained from the first metadata identifier by giving the training data to the first…model].”).
matches the metadata; (Sohn, ⁋78, “Thus, the loss function for the classifier parameters θC include both a source domain classifier 402 term, as well as an N+1 entry of the target domain classifier 401. The N+1 entry in each classifier 401 and 402 is the entry provided for domain discrimination classification. Thus, optimizing the parameters for the classifiers 401 and 402 includes optimizing for a domain class; optimizing for a domain class is interpreted as training when matching the metadata/domain and training the classifiers trains the CNNs, which are interpreted as the encoders [matches the metadata;] in addition to label classes y.”).
fourth training the first…model such that, with respect to each learning data set, an identification result obtained from the first metadata identifier by giving the training data to the first…model does not match the metadata, (Sohn, ⁋83, “Thus, the classifiers 401 [the first metadata identifier] and 402 are trained via cross entropy loss, and the loss of the classifiers 401 and 402 can be used to then jointly train the parameters of the CNNs 301 and 302 [fourth training the first… model such that, with respect to each learning data set, an identification result obtained from the first metadata identifier by giving the training data to the first…model]. As a result, the CNNs 301 and 302 are trained in an adversarial technique to fool a domain class; fooling the domain class is interpreted as outputs that do not match the domain [does not match the metadata,] of the classifiers 401 and 402 without the use of a separate discriminator.”).
and the third training and the fourth training are alternatingly and repeatedly executed. (Sohn, ⁋35, “The domain adaptation module 520 can then generate classifications corresponding to vehicle makes and/or modules based on the training from the labeled images of the database 630. Training can be performed concurrently with classification of the capture images, or separately. Training can also be performed continually to constantly improve the accuracy of the object recognition module 500 [the third training and the fourth training are alternatingly and repeatedly executed.].”).
While Sohn teaches the use of multiple models for domain adaptation, Sohn does not explicitly teach:
using encoders
and fifth training the first encoder and the second encoder such that, with respect to each learning data set, a mutual information amount decreases between an output value obtained, as the first feature amount, from the first encoder by giving the training data to the first encoder and an output value obtained, as the second feature amount, from the second encoder by giving the training data to the second encoder,
Yasutomi teaches using encoders (Yasutomi, ⁋12, “The process includes extracting feature for input data utilizing an encoder [using encoders], the input data including labeled data and unlabeled data;”).
Sohn and Yasutomi are both in the same field of endeavor (i.e. object detection). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Sohn and Yasutomi to teach the above limitation(s). The motivation for doing so is that encoders enhances generalization performance (cf. Yasutomi, ⁋4, “Thus, in recent years, there is known semi-supervised learning of enhancing generalization performance by using a small amount of supervised data and a large number of unsupervised data, and an autoencoder is known as semi-supervised learning of a class classification problem using deep learning.”).
While Sohn in view of Yasutomi teaches the use of multiple encoders for domain adaptation, the combination does not explicitly teach:
and fifth training the first encoder and the second encoder such that, with respect to each learning data set, a mutual information amount decreases between an output value obtained, as the first feature amount, from the first encoder by giving the training data to the first encoder and an output value obtained, as the second feature amount, from the second encoder by giving the training data to the second encoder,
Rothberg teaches
and fifth training the first encoder and the second encoder such that, with respect to each learning data set, (Rothberg, ⁋171, “The latent random variables T1 and T2 for first and second modalities may be defined as the outputs of encoders for the first and second modalities, respectively [and fifth training the first encoder and the second encoder such that, with respect to each learning data set,].”).
a mutual information amount decreases between an output value obtained, as the first feature amount, from the first encoder by giving the training data to the first encoder and an output value obtained, as the second feature amount, from the second encoder by giving the training data to the second encoder, (Rothberg, ⁋173, “Therefore, while we are compressing, we also want to make sure that the compressed representations T1 [between an output value obtained, as the first feature amount, from the first encoder by giving the training data to the first encoder] and T2 [and an output value obtained, as the second feature amount, from the second encoder by giving the training data to the second encoder,] are as informative about one other as possible. This equation indicates that we should maximally compress X1 and X2 by minimizing the mutual information between X1, T1 and X2, T2 [a mutual information amount decreases]”).
Sohn, in view of Yasutomi, and Rothberg are both in the same field of endeavor (i.e. feature comparison). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Sohn, in view of Yasutomi, and Rothberg to teach the above limitation(s). The motivation for doing so is that considering mutual information when training feature extractors improves the system’s generalization abaility (cf. Rothberg, ⁋174, “Intuitively, by learning a cross-modality driven compressed representations, we are leveraging the labeled (or paired) data across many modalities, which reduces the generalization gap.”).
Regarding claim 2, Sohn in view of Yasutomi and Rothberg teaches the learning apparatus according to claim 1. Sohn further teaches wherein at least one first learning data set and at least one second learning data set that are included in the plurality of learning data sets are obtained from different domains such that the attribute indicated by the metadata of the at least one first learning data set differs from that of the at least one second learning data set. (Sohn, ⁋45, “the source domain includes the source image 10 c, along with the adjusted views of the source image 10 c generated by the viewpoint synthesizer 100 and photometric adjuster 200. The target domain can include surveillance images to be analyzed by the feature extractor 300, including, e.g., the first surveillance image 10 a and the second surveillance image 10 b [wherein at least one first learning data set and at least one second learning data set that are included in the plurality of learning data sets are obtained from different domains]. Because the source domain and the target domain are not identical, the feature extractor 300 can be better trained for feature extraction if the domains are adapted [such that the attribute indicated by the metadata of the at least one first learning data set differs from that of the at least one second learning data set.].”).
Regarding claim 3, Sohn in view of Yasutomi and Rothberg teaches the learning apparatus according to claim 1. Sohn further teaches:
wherein, in the first training, the second encoder is trained such that the second feature amount includes a component corresponding to the attribute regarding acquisition of the training data indicated by the metadata, and (Sohn, ⁋73, “To train the feature extractor 300, a convolutional neural network (CNN) 301 extracts surveillance features 20 from the surveillance image 10 while another CNN 302 [wherein, in the first training, the second encoder is trained] concurrently extracts augmented source features 23 from the style and view augmented source image 13 [such that the second feature amount includes a component corresponding to the attribute regarding acquisition of the training data indicated by the metadata,]. Each of the CNN 301 and 302 can be, e.g., any neural network for extracting feature representations form input images.”).
in the fourth training, the first encoder is trained such that the first feature amount includes a component corresponding to information that appears in common across domains from which the training data of the learning data sets are acquired. (Sohn, ⁋46, “According to aspects of the present invention, the domain adaption can include, e.g. an adversarial technique to adapt the source and target domains to extract similar features corresponding to similar objects despite the different domains [such that the first feature amount includes a component corresponding to information that appears in common across domains from which the training data of the learning data sets are acquired.]…The adversarial technique of the DANN can accurately compare the source domain and the target domain for accurate training of the feature extractor 300 [in the fourth training, the first encoder is trained].”).
Regarding claim 12, Sohn in view of Yasutomi and Rothberg teaches the learning apparatus according to claim 1. Sohn further teaches:
wherein the training data is sensing data obtained by sensors that observe vehicles moving on a road, (Sohn, ⁋29, “For example, the object recognition system 500 can be provided with front view images of cars of a variety of makes and/or models to train recognition of make and/or model of the vehicles 611 [wherein the training data is sensing data obtained by sensors that observe vehicles moving on a road,].”).
the metadata indicates, as the attribute regarding acquisition, an attribute of the road, observation angles of the sensors, installation intervals of the sensor, or types of the sensors, or a combination of these, (Sohn, ⁋39, “For example, an imaging device, such as, e.g., a camera, can capture images of a first set of images of an environment having a first perspective or viewpoint, such as, e.g., a first angle of elevation, and a second set of images of the environment having a second perspective or viewpoint [the metadata indicates, as the attribute regarding acquisition, an attribute of…observation angles of the sensors,].”).
and the correct answer data indicates information regarding traffic conditions of the road, as the feature. (Sohn, ⁋36, “he classified images can be labeled with labels corresponding to the classifications, such as, e.g., labels for the make and/or model of the vehicle 611 capture by the camera 600 [and the correct answer data indicates information regarding traffic conditions of the road, as the feature.].”).
Regarding claim 13, Sohn in view of Yasutomi and Rothberg teaches the learning apparatus according to claim 1. Sohn further teaches:
wherein the training data is sensing data obtained by a sensor that observes a state of an examinee, (Sohn, ⁋48, “Additionally, class labels of the labeled target image 30 can be used by an automatic response system 530. The automatic response system 530 can include, e.g., a parking lot ticketing system that automatically generates a ticket for illegally parked vehicles, or a traffic camera that automatically generates a ticket for vehicles that violate traffic laws, such as, e.g., running a red light [wherein the training data is sensing data obtained by a sensor that observes a state of an examinee,].”).
the metadata indicates, as the attribute regarding acquisition, identification information of the examinee, an attribute regarding the time at which the sensing data was obtained, an attribute regarding installation conditions of the sensor, or an installed location of the sensor, or a combination of these, (Sohn, ⁋39, “For example, an imaging device, such as, e.g., a camera, can capture images of a first set of images of an environment having a first perspective or viewpoint, such as, e.g., a first angle of elevation, and a second set of images of the environment having a second perspective or viewpoint [the metadata indicates, as the attribute regarding acquisition…installation conditions of the sensor].”).
and the correct answer data indicates the state of the examinee as the feature. (Sohn, ⁋48, “Additionally, class labels of the labeled target image 30 [and the correct answer data] can be used by an automatic response system 530. The automatic response system 530 can include, e.g., a parking lot ticketing system that automatically generates a ticket for illegally parked vehicles, or a traffic camera that automatically generates a ticket for vehicles that violate traffic laws, such as, e.g., running a red light [indicates the state of the examinee as the feature.].”).
Regarding claim 14, Sohn in view of Yasutomi and Rothberg teaches the learning apparatus according to claim 1. Sohn further teaches:
wherein the training data is image data of an image of a product, (Sohn, ⁋48, “Additionally, class labels of the labeled target image 30 can be used by an automatic response system 530. The automatic response system 530 can include, e.g., a parking lot ticketing system that automatically generates a ticket for illegally parked vehicles; vehicles are interpreted as products [wherein the training data is image data of an image of a product,], or a traffic camera that automatically generates a ticket for vehicles that violate traffic laws, such as, e.g., running a red light.”).
the metadata indicates, as the attribute regarding acquisition, an attribute of the product, shooting conditions of the product, or an attribute of a factory for producing the product, or a combination of these, (Sohn, ⁋39, “For example, an imaging device, such as, e.g., a camera, can capture images of a first set of images of an environment having a first perspective or viewpoint, such as, e.g., a first angle of elevation, and a second set of images of the environment having a second perspective or viewpoint [the metadata indicates, as the attribute regarding acquisition…shooting conditions of the product].”).
and the correct answer data indicates that state of the product as the feature. (Sohn, ⁋48, “Additionally, class labels of the labeled target image 30 [and the correct answer data] can be used by an automatic response system 530. The automatic response system 530 can include, e.g., a parking lot ticketing system that automatically generates a ticket for illegally parked vehicles, or a traffic camera that automatically generates a ticket for vehicles that violate traffic laws, such as, e.g., running a red light [indicates that state of the product as the feature.].”).
Regarding claim 15, Sohn in view of Yasutomi and Rothberg teaches the learning apparatus according to claim 1. Sohn further teaches:
An estimation apparatus comprising a second processor configured with a second program to perform operations comprising: (Sohn, ⁋46, “The domain adaptation unit 400 can be, e.g., stored in a memory or a storage and executed using a processor using a DANN stored in a memory or a storage and executed by the processor [An estimation apparatus comprising a second processor configured with a second program to perform operations comprising:].”).
operation as a data acquiring unit configured to acquire object data; (Sohn, ⁋39, “The system/method can process input images from more than one perspective of an environment. For example, an imaging device, such as, e.g., a camera, can capture images [operation as a data acquiring unit configured to acquire object data;] of a first set of images of an environment having a first perspective or viewpoint, such as, e.g., a first angle of elevation, and a second set of images of the environment having a second perspective or viewpoint.”).
operation as an estimating unit configured to estimate, using the learning apparatus according to claim 1, a feature included in acquired object data using the first encoder, the second encoder, and the estimator that were trained by the learning apparatus; (Sohn, see Figure 5., The estimating unit is interpreted as being element 400 and element 400 takes the output values, from the first and second models/encoders to produce predicted features [operation as an estimating unit configured to estimate, using the learning apparatus according to claim 1, a feature included in acquired object data using the first encoder, the second encoder, and the estimator that were trained by the learning apparatus;]).
and operation as an output unit configured to output information regarding the result of estimating the feature. (Sohn, ⁋76, “As a result of using classifiers without discriminators, the classifiers 401 and 402 [and operation as an output unit configured to] can each generate outputs that include, e.g., entries for class scores corresponding to each classification [output information regarding the result of estimating the feature.] as well as an additional entry for a domain classification score corresponding to domain discrimination.”).
Claims 4 and 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over Sohn, et al., US Pre-Grant Publication 2019/0065853A1 (“Sohn”) in view of Yasutomi, et al., US Pre-Grant Publication 2018/0349741A1 (“Yasutomi”) and further in view of Rothberg, et al., US Pre-Grant Publication 2019/0347523A1 (“Rothberg”) and Poole, et al., Non-Patent Literature “Analyzing noise in autoencoders and deep networks” (“Poole”).
Regarding claim 4, Sohn in view of Yasutomi and Rothberg teaches the learning apparatus according to claim 1. While the combination teaches the learning apparatus according to claim 1, the combination fails to explicitly teach wherein, the first, second, third, and fourth trainings are performed by inputting noise along with the training data to the encoders.
Poole teaches wherein, the first, second, third, and fourth trainings are performed by inputting noise along with the training data to the encoders. (Poole, pg. 2 Section 2.2, “Injecting noise into both the inputs and hidden representations of autoencoders [wherein, the first, second, third, and fourth trainings] has been proposed for linear networks in prior work by [11], but has not been analyzed in detail for nonlinear representations. We parameterize the noise in the NAE as a tuple (eI, eH, eZ) that characterizes the distribution of the noises corrupting the input, hidden unit inputs, and hidden activations [are performed by inputting noise along with the training data to the encoders.]”).
Sohn, in view of Yasutomi and Rothberg, and Poole are both in the same field of endeavor (i.e. machine learning). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Sohn, in view of Yasutomi and Rothberg, and Poole to teach the above limitation(s). The motivation for doing so is that injecting noise into machine learning models improves generalization (cf. Poole, pg. 8 Section 5, “Thus noise on internal representations leads to a spreading of representations of different classes, and a contraction within classes that may be beneficial for generalization.”).
Regarding claim 8, Sohn in view of Yasutomi and Rothberg teaches the learning apparatus according to claim 1. While the combination teaches the first encoder and the second encoder and the first feature amount and the second feature amount as seen in claim 1, the combination does not explicitly teach: wherein the learning model further includes a decoder configured to decode the input data from the…feature amount…and performing the machine learning further includes fifth training the…encoder…and the decoder such that, with respect to each learning data set, decoded data obtained by the decoder by giving the training data to the…encoder…encoder matches the training data.
Poole teaches:
wherein the learning model further includes a decoder configured to decode the input data from the…feature amount…and (Poole, pg. 2 Section 2.1, “The autoencoder consists of an encoder that maps inputs to a hidden representation: f(x) = sf (W x+b), and a decoder [wherein the learning model further includes a decoder configured to decode the input data from the] that maps the hidden representation [feature amount] back to the inputs: g(h) = sg(W0h + d).”).
performing the machine learning further includes fifth training the…encoder…and the decoder such that, with respect to each learning data set, decoded data obtained by the decoder by giving the training data to the…encoder…encoder matches the training data. (Poole, pg. 2 Section 2.1, “The composition of the encoder and decoder yield the reconstruction function: r(x) = g(f(x)). The typical training criterion for autoencoders [performing the machine learning further includes fifth training the…encoder…and the decoder such that] is minimizing the reconstruction error, P x∈X L(x, r(x)) with respect to some loss L, typically either squared error or the binary cross-entropy; cross-entropy is interpreted determining whether the decoded representation matches the input data [with respect to each learning data set, decoded data obtained by the decoder by giving the training data to the…encoder…encoder matches the training data.] [2].”).
Sohn, in view of Yasutomi Rothberg, and Poole are both in the same field of endeavor (i.e. machine learning). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Sohn, in view of Yasutomi and Rothberg, and Poole to teach the above limitation(s). The motivation for doing so is that decoding the encoded feature representations improves object recognition (cf. Poole, pg. 2 Section 2.1, “the autoencoder tends to learn a more interesting representation of the input data that can be useful in other tasks such as object recognition.”).
Regarding claim 9, Sohn in view of Yasutomi, Rothberg, and Poole teaches the learning apparatus according to claim 8. The combination further teaches teaches wherein, in the first, the second, and the fifth training, an output value is acquired, as the second feature amount, from the second encoder by giving the training data to the second encoder, as seen in claim 8.
Poole further teaches:
and the trainings are executed by inputting noise to the second metadata identifier, the estimator, (Poole, pg. 4 Section 3.3, “Recent work in neuroscience has shown that single-layer models trained with input and output noise and optimized to maximize mutual information yield receptive fields resembling those found in biological systems [7, 12]. Similar to our work, these models show the importance of input noise and hidden activation noise on learning representations; other models are interpreted as the metadata identifier and the estimator [and the trainings are executed by inputting noise to the second metadata identifier, the estimator,].”).
and the decoder along with the acquired output value. (Poole, pg. 2 Section 2.2, “Inspired by the recent work on dropout, we extend denoising autoencoders to allow for the injection of additional noise at the input and output [and the decoder along with the acquired output value] of the hidden units.”).
Sohn, Yasutomi, Rothberg, and Poole are all in the same field of endeavor (i.e. machine learning). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Sohn, Yasutomi, Rothberg, and Poole to teach the above limitation(s). The motivation for doing so is that training with noise improves robustness of the model.
Regarding claim 10, Sohn in view of Yasutomi, Rothberg, and Poole teaches the learning apparatus according to claim 8. The combination teaches wherein the data acquiring unit acquires, after the learning processing unit has performed machine learning of the learning model, an output value from the first encoder as the first feature amount by giving at least one training data of the plurality of learning data sets to the first encoder, acquires an output value from the second encoder as the second feature amount by giving the training data to the second encoder, and acquires output data from the decoder as the decoded data by inputting the output value acquired from the first encoder to the decoder as seen in claim 8.
Poole further teaches and inputting noise along with the output value obtained from the second encoder to the decoder, (Poole, pg. 2 Section 2.2, “Injecting noise into both the inputs and hidden representations of autoencoders has been proposed for linear networks in prior work by [11], but has not been analyzed in detail for nonlinear representations. We parameterize the noise in the NAE as a tuple (eI, eH, eZ) that characterizes the distribution of the noises corrupting the input, hidden unit inputs, and hidden activations [and inputting noise along with the output value obtained from the second encoder to the decoder,]”).
Sohn, Yasutomi, Rothberg, and Poole are all in the same field of endeavor (i.e. machine learning). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Sohn, Yasutomi, Rothberg, and Poole to teach the above limitation(s). The motivation for doing so is that training with noise improves robustness of the model.
Sohn further teaches and the learning processing unit again performs machine learning of the learning model using the acquired output data as new training data. (Sohn, see Figure 5, In Figure 5, the output of the classifiers is fed back into the first and second models for continued training therefore, the output data is used as new training data [and the learning processing unit again performs machine learning of the learning model using the acquired output data as new training data.]).
Claims 6 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Sohn, et al., US Pre-Grant Publication 2019/0065853A1 (“Sohn”) in view of Yasutomi, et al., US Pre-Grant Publication 2018/0349741A1 (“Yasutomi”) and further in view of Rothberg, et al., US Pre-Grant Publication 2019/0347523A1 (“Rothberg”) and Mariani, et al., Non-Patent Literature “BAGAN: Data Augmentation with Balancing GAN” (“Mariani”).
Regarding claim 6, Sohn in view of Yasutomi and Rothberg teaches the learning apparatus according to claim 1. Sohn further teaches wherein, in the fourth training,…and performing training of the first encoder such that the identification result does not match the metadata includes performing training of the first encoder such that an identification result obtained from the first metadata identifier by giving the training data to the first encoder matches the…metadata. (Sohn, ⁋83, “Thus, the classifiers 401 [the first metadata identifier] and 402 are trained via cross entropy loss, and the loss of the classifiers 401 and 402 can be used to then jointly train [wherein, in the fourth training,…and performing training of the first encoder such that the identification result] the parameters of the CNNs 301 [includes performing training of the first encoder such that an identification result obtained from the first metadata identifier by] and 302. As a result, the CNNs 301 and 302 are trained in an adversarial technique to fool a domain class; fooling the domain class is interpreted as outputs that do not match the domain, or fake values [does not match the metadata,] [by giving the training data to the first encoder matches the…fake metadata.] of the classifiers 401 and 402 without the use of a separate discriminator.”).
While Sohn in view of Yasutomi and Rothberg teaches the use of fake values for adversarial learning, the combination does not explicitly teach with respect to each learning data set, dummy metadata that corresponds to the metadata, and has a value that is different from that of the corresponding metadata is acquired
Mariani teaches with respect to each learning data set, dummy metadata that corresponds to the metadata, and has a value that is different from that of the corresponding metadata is acquired (Mariani, pg. 1 col. 1-2, “In this work we propose a balancing generative adversarial network (BAGAN) as an augmentation tool to restore the dataset balance by generating new minority-class images [with respect to each learning data set, dummy metadata that corresponds to the metadata,]. Since these images are scarce in the initial dataset, it is challenging to train a GAN for generating new ones…the proposed methodology includes in the adversarial training all data from minority and majority classes at once. This enables BAGAN to learn underlying features of the specific classification problem starting from all images and then to apply these features for the generation of new minority-class images [and has a value that is different from that of the corresponding metadata is acquired].”).
Sohn, in view of Yasutomi and Rothberg, and Mariani are both in the same field of endeavor (i.e. adversarial learning). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Sohn, in view of Yasutomi and Rothberg, and Mariani to teach the above limitation(s). The motivation for doing so is that augmenting the target domain with dummy examples improves training (cf. Mariani, abstract, “In this work we propose balancing GAN (BAGAN) as an augmentation tool to restore balance in imbalanced datasets. This is challenging because the few minority-class images may not be enough to train a GAN. We overcome this issue by including during the adversarial training all available images of majority and minority classes.”).
Regarding claim 7, Sohn in view of Yasutomi, Rothberg, and Mariani teaches the learning apparatus according to claim 6. Mariani further teaches wherein the dummy metadata is constituted by metadata of a learning data set that is different from the corresponding learning data set. (Mariani, pg. 1 col. 1-2, “In this work we propose a balancing generative adversarial network (BAGAN) as an augmentation tool to restore the dataset balance by generating new minority-class images; generating new class images from the GAN is interpreted as different data than the initial learning data as it is generated during training [wherein the dummy metadata is constituted by metadata of a learning data set that is different from the corresponding learning data set.].”).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Mariani with the teachings of Sohn, Yasutomi, and Rothberg for the same reasons disclosed in claim 6.
Claims 11 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Sohn, et al., US Pre-Grant Publication 2019/0065853A1 (“Sohn”) in view of Yasutomi, et al., US Pre-Grant Publication 2018/0349741A1 (“Yasutomi”) and further in view of Rothberg, et al., US Pre-Grant Publication 2019/0347523A1 (“Rothberg”) and Agaian, et al., US Pre-Grant Publication 2016/0253466A1 (“Agaian”).
Regarding claim 11, Sohn in view of Yasutomi and Rothberg teaches the learning apparatus according to claim 1. While the combination teaches of the first encoder and the first feature amount as seen in claim 1, the combination does not explicitly teach wherein the learning model further includes an additional estimator configured to receive an output value…and to estimate a feature included in the input data…and the performing the machine learning further includes fifth training the first…and the additional estimator such that, with respect to each learning data set, an estimation result obtained from the additional estimator by giving the training data to the first…matches the correct answer data or a different correct answer data indicating a different feature included in the training data.
Agaian teaches:
wherein the learning model further includes an additional estimator configured to receive an output value…and to estimate a feature included in the input data…and (Agaian, see Figure 6, In Figure 6, the multiple grading classifiers are interpreted as the additional estimator [wherein the learning model further includes an additional estimator configured to receive an output value…and to estimate a feature included in the input data…and]).
the performing the machine learning further includes fifth training the first…and the additional estimator such that, with respect to each learning data set, (Agaian, ⁋147, “multi-classifier ensemble is trained according to the following procedure: (1) n Tier-1 classifiers are trained [the performing the machine learning further includes fifth training the first…] and validated, based on a cross-validation partition of the available training data, which correspond to feature vectors computed using a different method per classifier; (2) m Tier-2 grading classifiers are trained [and the additional estimator such that, with respect to each learning data set,], one or more for each base-level classifier.”).
an estimation result obtained from the additional estimator by giving the training data to the first…matches the correct answer data or a different correct answer data indicating a different feature included in the training data. (Agaian, ⁋146 and Figure 6, “With the assistance of grading meta-classifiers (one per base-level classifier) [an estimation result obtained from the additional estimator by giving the training data to the first], the proposed system is able to predict for each of the original base-level learning methods whether its prediction for a particular training example is correct or not [matches the correct answer data or a different correct answer data indicating a different feature included in the training data.]. Only the predictions of base-level classifiers which are more likely to be right will be fused at the next stage.”).
Sohn, in view of Yasutomi and Rothberg, and Agaian are both in the same field of endeavor (i.e. image classification). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Sohn, in view of Yasutomi and Rothberg, and Agaian to teach the above limitation(s). The motivation for doing so is that comparing different multiple predictions aids in the robustness of a classification system (cf. Agaian, ⁋24, “robust multi-classifier systems are built based on the fusion or analysis of the predictions of base-level classifiers through one or more layers of meta-classifiers or refinement classifiers.”).
Regarding claim 16, Sohn in view of Yasutomi, Rothberg, and Agaian teaches the learning apparatus according to claim 11. Sohn further teaches:
An estimation apparatus comprising a second processor configured with a second program to perform operations comprising: (Sohn, ⁋46, “The domain adaptation unit 400 can be, e.g., stored in a memory or a storage and executed using a processor using a DANN stored in a memory or a storage and executed by the processor [An estimation apparatus comprising a second processor configured with a second program to perform operations comprising:].”).
operation as a data acquiring unit configured to acquire object data; (Sohn, ⁋39, “The system/method can process input images from more than one perspective of an environment. For example, an imaging device, such as, e.g., a camera, can capture images [operation as a data acquiring unit configured to acquire object data;] of a first set of images of an environment having a first perspective or viewpoint, such as, e.g., a first angle of elevation, and a second set of images of the environment having a second perspective or viewpoint.”).
operation as an estimating unit configured to estimate, using the learning apparatus according to claim 11, a feature included in acquired object data using the first encoder (Sohn, see Figure 5., The estimating unit is interpreted as being element 400, and element 400 takes the output values, from the first and second models/encoders to produce predicted features [operation as an estimating unit configured to estimate, using the learning apparatus according to claim 11, a feature included in acquired object data using the first encoder])
…and operation as an output unit configured to output information regarding the result of estimating the feature. (Sohn, ⁋76, “As a result of using classifiers without discriminators, the classifiers 401 and 402 [and operation as an output unit configured to] can each generate outputs that include, e.g., entries for class scores corresponding to each classification [output information regarding the result of estimating the feature.] as well as an additional entry for a domain classification score corresponding to domain discrimination.”).
Agaian further teaches:
and the additional estimator that were trained by the learning apparatus; (Agaian, see Figure 6, In Figure 6, the multiple grading classifiers are interpreted as the additional estimator [and the additional estimator that were trained by the learning apparatus;]).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Agaian with the teachings of Sohn, Yasutomi, and Rothberg for the same reasons disclosed in claim 11.
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Sohn, et al., US Pre-Grant Publication 2019/0065853A1 (“Sohn”) in view of Yasutomi, et al., US Pre-Grant Publication 2018/0349741A1 (“Yasutomi”) and further in view of Rothberg, et al., US Pre-Grant Publication 2019/0347523A1 (“Rothberg”) and Cao, et al., Non-Patent Literature “Partial Adversarial Domain Adaptation” (“Cao”).
Regarding claim 17, Sohn in view of Yasutomi and Rothberg teaches the estimation apparatus according to claim 15. While the combination teaches using the second encoder and the second metadata identifier that were trained by the learning apparatus, as seen on claim 15, the combination does not explicitly teach: further comprising: operation as an evaluating unit configured to identify the attribute regarding acquisition of the object data…and determine whether or not the result of estimating the feature is adopted based on the identification result.
Cao teaches further comprising: operation as an evaluating unit configured to identify the attribute regarding acquisition of the object data…and determine whether or not the result of estimating the feature is adopted based on the identification result. (Cao, pg. 10 Section 4.2, “But there are classes in the source domain that do not exist in the target domain, a.k.a. outlier source classes. This explains their weak performance for partial domain adaptation. Not surprisingly, PADA [operation as an evaluating unit configured to identify the attribute regarding acquisition of the object data] outperforms all the comparison methods by large margins, indicating that PADA can effectively avoid negative transfer by eliminating the influence of outlier source classes irrelevant to the target domain; eliminating data points that negatively influence model performance is interpreted as determining whether to adopt/use outputs [and determine whether or not the result of estimating the feature is adopted based on the identification result.].”).
Sohn, in view of Yasutomi and Rothberg, and Cao are both in the same field of endeavor (i.e. domain adaptation). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Sohn, in view of Yasutomi and Rothberg, and Cao to teach the above limitation(s). The motivation for doing so is determining which samples have a negative impact on model performance improves knowledge transfer in domain adaptation (cf. Cao, pg. 3 Section 1, “identifies the irrelevant source data belonging to the outlier source classes and down-weighs their importance automatically. The key improvement over previous methods is the capability to simultaneously promote positive transfer of relevant source data and alleviate negative transfer of irrelevant source data.”).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Sohn, et al., US Pre-Grant Publication 2019/0065853A1 (“Sohn”) in view of Yasutomi, et al., US Pre-Grant Publication 2018/0349741A1 (“Yasutomi”) and further in view of Rothberg, et al., US Pre-Grant Publication 2019/0347523A1 (“Rothberg”), Poole, et al., Non-Patent Literature “Analyzing noise in autoencoders and deep networks” (“Poole”), and Cao, et al., Non-Patent Literature “Partial Adversarial Domain Adaptation” (“Cao”).
Regarding claim 18, Sohn in view of Yasutomi, Rothberg, and Poole teaches the learning apparatus according to claim 8. Sohn further teaches:
A data generation apparatus comprising a second processor configured with a second program to perform operations comprising: (Sohn, ⁋46, “The domain adaptation unit 400 can be, e.g., stored in a memory or a storage and executed using a processor using a DANN stored in a memory or a storage and executed by the processor [A data generation apparatus comprising a second processor configured with a second program to perform operations comprising:].”).
a data acquiring unit configured to acquire object data; (Sohn, ⁋39, “The system/method can process input images from more than one perspective of an environment. For example, an imaging device, such as, e.g., a camera, can capture images [a data acquiring unit configured to acquire object data;] of a first set of images of an environment having a first perspective or viewpoint, such as, e.g., a first angle of elevation, and a second set of images of the environment having a second perspective or viewpoint.”).
a data generating unit configured to, using the learning apparatus according to claim 8, acquire an output value from the first encoder as the first feature amount by giving the object data to the first encoder trained by the learning apparatus, (Sohn, ⁋73, “To train the feature extractor 300 [a data generating unit configured to, using the learning apparatus according to claim 8,], a convolutional neural network (CNN) 301 [the first encoder] extracts surveillance features 20 from the surveillance image 10 [acquire an output value from the first encoder as the first feature amount by giving the object data to the first encoder trained by the learning apparatus,] while another CNN 302 concurrently extracts augmented source features 23 from the style and view augmented source image 13. Each of the CNN 301 and 302 can be, e.g., any neural network for extracting feature representations form input images.”).
…and a saving processing unit configured to save the generated decoded data in a predetermined storage area. (Sohn, ⁋40, “Thus, the feature extractor 300 can be, e.g., stored in a memory or a storage [and a saving processing unit configured to save the generated decoded data in a predetermined storage area.] and executed using a processor.”).
Poole further teaches and generate decoded data by decoding the object data from the output value acquired from the first encoder using the trained decoder (Poole, pg. 2 Section 2.1, “The autoencoder consists of an encoder that maps inputs to a hidden representation: f(x) = sf (W x+b), and a decoder [and generate decoded data by decoding the object data] that maps the hidden representation [from the output value acquired from the first encoder using the trained decoder] back to the inputs: g(h) = sg(W0h + d).”).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Poole with the teachings of Sohn, Yasutomi, and Rothberg for the same reasons disclosed in claim 8.
While Sohn in view of Yasutomi, Rothberg, and Poole teaches domain adaptation using encoders and decoders, the combination does not explicitly teach without giving an output value acquired from the second encoder;
Cao teaches without giving an output value acquired from the second encoder; (Cao, pg. 10 Section 4.2, “But there are classes in the source domain that do not exist in the target domain, a.k.a. outlier source classes. This explains their weak performance for partial domain adaptation. Not surprisingly, PADA outperforms all the comparison methods by large margins, indicating that PADA can effectively avoid negative transfer by eliminating the influence of outlier source classes irrelevant to the target domain; eliminating data points that negatively influence model performance is interpreted as determining whether to use outputs [without giving an output value acquired from the second encoder;].”).
Sohn, in view of Yasutomi, Rothberg, and Poole, and Cao are both in the same field of endeavor (i.e. domain adaptation). It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Sohn, in view of Yasutomi, and Rothberg, and Poole, and Cao to teach the above limitation(s). The motivation for doing so is determining which samples have a negative impact on model performance improves knowledge transfer in domain adaptation (cf. Cao, pg. 3 Section 1, “identifies the irrelevant source data belonging to the outlier source classes and down-weighs their importance automatically. The key improvement over previous methods is the capability to simultaneously promote positive transfer of relevant source data and alleviate negative transfer of irrelevant source data.”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Zhou, et al., US20190220977A1 discloses cross-domain medical image analysis and cross-domain medical image synthesis using deep image-to-image networks and adversarial networks. A first encoder and the second encoder are trained together based on a similarity of feature maps generated by the first encoder from training images from a first domain and feature maps generated by the second encoder from training images from a second domain, and a decoder is trained to generate output images from feature maps generated by the first encoder or the second encoder.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS S WU whose telephone number is (571)270-0939. The examiner can normally be reached Monday - Friday 8:00 am - 4:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached on 571-431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.S.W./Examiner, Art Unit 2148
/MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148