DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 32-35, 37-38, 40-47, and 50-51 are presented for examination.
Claim Rejections - 35 USC § 101
Claims 32-35, 37-38, 40-47, and 50-51 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50 (“2019 PEG”).
Claim 32
Step 1: The claim recites a method; therefore, it is directed to the statutory category of processes.
Step 2A Prong 1: The claim recites, inter alia:
[S]electing, … as second training images: images or portions of images of a first training dataset that lie within a neighborhood of [a] target domain of the at least one target image, wherein the target domain differs from or is a sub-domain of a source domain defined by the first training dataset, and portions or full target images, of the at least one target image, of which output values are predicted by a global domain mathematical model with a level of confidence equal to or above a predetermined confidence threshold: This limitation could encompass mentally observing the output values of the model, mentally comparing them to a threshold, mentally defining a distance metric between images, mentally calculating the distances between the images in the dataset and the target image, and mentally selecting the images that either meet the threshold or are within less than a predetermined distance from the target image.
[G]enerating a second training dataset with the second training images: This limitation could encompass the mental generation of the training dataset by mentally selecting which images should be in the dataset.
[R]etraining [a] global domain mathematical model with the second training dataset to obtain a domain-adapted predictive model; and analyzing one or more images captured by the imaging system to assign … continuous or discrete values to the one or more images …; wherein the global domain mathematical model is a mathematical model trained, by executing a machine learning algorithm, with images of a first training dataset to reduce a global error measured across the first training dataset; wherein retraining the global domain mathematical model comprises adjusting parameters of the mathematical model executing the machine learning algorithm to obtain the domain-adapted predictive model that is optimized for analyzing images within the target domain: Given that the model is described as “mathematical” and training and retraining a machine learning algorithm involve a series of mathematical manipulations including adjusting parameters, these limitations are directed to mathematical concepts. Assigning values to an image can be performed in the mind.
[T]he neighborhood is computed by looking at a distance between feature vectors or a combination of feature vectors of the at least one target image and feature vectors or a combination of feature vectors of the images in the first training dataset: This limitation could encompass the mental computation of the neighborhood by mentally calculating the distance between vectors. Alternatively, this is a mathematical calculation.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim additionally recites “obtaining at least one target dataset, wherein the at least one target dataset comprises at least one target image that belongs to a target domain”. However, this limitation amounts to the insignificant extra-solution activity of mere data gathering and output. MPEP § 2106.05(g). The further recitation that the method is “computer-implemented” and that certain functions are performed “using the domain-adapted predictive model” and by an “imaging system comprising a lens and a sensor array” are mere instructions to apply the judicial exception using a generic computer programmed with a generic class of computer algorithm and/or generically recited image capturing technology. MPEP § 2106.05(f). Finally, the recitation that the method is “for optimizing an imaging system … automatically [to] analyze scenes within a target observation domain”, in addition to being an intended-use clause within the preamble not entitled to patentable weight, merely restricts the field of use of the judicial exception to scene analysis in an observation domain. MPEP § 2106.05(h).
Step 2B: The claim does not contain significantly more than the judicial exception. The obtaining limitation, in addition to being insignificant extra-solution activity, also recites the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP § 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Otherwise, the analysis mirrors that of step 2A, prong 2. As an ordered whole, the claim is directed to the abstract idea of training a mathematical model. Nothing in the claim provides significantly more than this. As such, the claim is not patent eligible.
Claim 33
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the same judicial exceptions as in claim 32.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that “the feature vectors or the combination of feature vectors are totally or partially derived from a pre-trained machine learning model”. However, this amounts to a mere instruction to apply the judicial exception using a generic computer programmed with a generic class of computer algorithm. MPEP § 2106.05(f).
Step 2B: The claim does not contain significantly more than the judicial exception. The claim further recites that “the feature vectors or the combination of feature vectors are totally or partially derived from a pre-trained machine learning model”. However, this amounts to a mere instruction to apply the judicial exception using a generic computer programmed with a generic class of computer algorithm. MPEP § 2106.05(f).
Claim 34
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
[G]enerating the feature vectors or the combination of feature vectors for each pixel or set of pixels of each of the at least one target image of the at least one target dataset: This limitation could encompass the mental generation of the image feature vectors.
[G]enerating the feature vectors or the combination of feature vectors for each pixel or set of pixels of each image of the first training dataset: This limitation could encompass the mental generation of the image feature vectors.
[C]omputing the distance between the feature vectors or the combination of feature vectors: Computing a distance is a mathematical concept.
[S]electing pixels or sets of pixels of the images of the first training dataset that have a distance value lower than a threshold distance to the pixels or sets of pixels of the at least one target image of the at least one target dataset: This limitation could encompass the mental selection of the pixels that have the lower than threshold distance.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. See claim 33 analysis.
Step 2B: The claim does not contain significantly more than the judicial exception. See claim 33 analysis.
Claim 35
Step 1: A process, as above.
Step 2A Prong 1: The claim recites that “an individual feature vector of the feature vectors is the result of combining different feature vectors comprising histograms of gradient orientations (HOG), red-green-blue (RGB) color histograms, texture histograms, response to wavelets filters, artificial neural networks, deep neural network features extracted from a pre-trained model, and their combinations; or … the feature vectors, the way the feature vectors are combined, and a function that measures the distance between the feature vectors, are selected depending on one or more image transformation invariances, wherein the one or more image transformation invariances include any combination of translations, rotations, scaling, shear, image blur, or image brightness and contrast changes.” This limitation could encompass the mental combination of the enumerated types of vectors.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. See claim 34 analysis.
Step 2B: The claim does not contain significantly more than the judicial exception. See claim 34 analysis.
Claim 37
Step 1: A process, as above.
Step 2A Prong 1: The claim recites that “the predetermined confidence threshold is defined in relation to a level of accuracy in a prediction of an identification, classification or labeling process of the at least one target image of the at least one target dataset.” Training the network with the second training dataset remains a mathematical concept under these further assumptions.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. See claim 36 analysis.
Step 2B: The claim does not contain significantly more than the judicial exception. See claim 36 analysis.
Claim 38
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia, that “the portions or full target images are … selected using their pixel-wise confidence levels”. This limitation could encompass the mental selection of the images using the confidence levels.
The claim further recites that the “threshold value per class is predetermined, and … the prediction from the global domain mathematical model in the pixels of the portions or full target images is above the predetermined confidence threshold.” Predetermining a threshold value per class is mentally performable, and training the network with the second training dataset remains a mathematical concept under the further assumption that the prediction from the global domain mathematical model is above the threshold.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that “the portions or full target images are obtained by using a semi-supervised machine learning method”. However, this is a mere instruction to apply the judicial exception using a generic computer programmed with a generic class of computer algorithm. MPEP § 2106.05(f).
Step 2B: The claim does not contain significantly more than the judicial exception. The claim further recites that “the portions or full target images are obtained by using a semi-supervised machine learning method”. However, this is a mere instruction to apply the judicial exception using a generic computer programmed with a generic class of computer algorithm. MPEP § 2106.05(f).
Claim 40
Step 1: A process, as above.
Step 2A Prong 1: The claim recites that “the second training dataset further comprises manually labeled full images or portions of images of the at least one target dataset that: were classified by the global domain mathematical model with the level of confidence below the predetermined confidence threshold; or the distance between the feature vectors or the combination of feature vectors of the images or portions of target images of the at least one target dataset to the feature vectors or the combination of feature vectors of the images of the first training dataset is equal to or larger than a threshold value; or were classified by the global domain mathematical model with the level of confidence below the predetermined confidence threshold and the distance between the feature vectors or the combination of feature vectors of the target images or portions of images of the at least one target dataset the feature vectors or the combination of feature vectors of the images of the first training dataset is equal to or larger than a threshold value.” Training the model using the second training dataset remains a mathematical concept under these further assumptions.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. See claim 32 analysis.
Step 2B: The claim does not contain significantly more than the judicial exception. See claim 32 analysis.
Claim 41
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the same judicial exceptions as in claim 32.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that “at least one target image and/or the one or more images are captured by the imaging system”. This limitation recites the insignificant extra-solution activity of mere data gathering. MPEP § 2106.05(g).
The claim further recites that the imaging device is “all or partially onboard an aerial vehicle, wherein the aerial vehicle is a satellite, a spacecraft, an aircraft, a plane, an unmanned aerial vehicle, UAV, or a drone.” This amounts to a mere restriction of the judicial exception to the field of use of aerial vehicles. MPEP § 2106.05(h).
Step 2B: The claim does not contain significantly more than the judicial exception. The capturing limitation, in addition to being insignificant extra-solution activity, is also directed to the well-understood, routine, and conventional activity of storing and retrieving information in memory. MPEP § 2106.05(d)(II); Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015). The recitation that the device is onboard an aerial vehicle is a mere restriction of the judicial exception to a field of use for the same reasons as stated above.
Claim 42
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia, “predict[ing] continuous or discrete values from imagery content.” This limitation could encompass the mental prediction of the values.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “learn[ing] segmentation of images of the at least one target dataset comprising aerial or satellite images based on land use classes”. This amounts to a mere restriction of the judicial exception to the field of use of aerial or satellite imagery. MPEP § 2106.05(h). The claim further recites that the prediction occurs using a “global domain mathematical model [that] is trained and retrained to” perform the prediction. However, this is a mere instruction to apply the judicial exception using a generic computer programmed with a generic class of computer algorithm. MPEP § 2106.05(f).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step is the same as in step 2A, prong 2.
Claim 43
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia, “segment[ing] image contents with image content labels including water bodies, rivers, lakes, dams, forests, bare lands, waste dumps, buildings, roads, crop types, crop growth, soil composition, mines, and/or oil and gas infrastructure.” This limitation could encompass the mental segmentation of the image contents with the enumerated labels.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that the segmentation occurs using a “global domain mathematical model”. However, this is a mere instruction to apply the judicial exception using a generic computer programmed with a generic class of computer algorithm. MPEP § 2106.05(f).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step is the same as in step 2A, prong 2.
Claim 44
Step 1: A process, as above.
Step 2A Prong 1: The claim recites the same judicial exceptions as in claim 32.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that the “training and retraining the global domain mathematical model, and generating the second training dataset are performed using at least one of artificial neural networks, deep learning techniques, non- supervised machine learning methods, semi-supervised machine learning methods, or convolutional neural networks.” However, this is a mere instruction to apply the judicial exception using a generic computer programmed with a generic class of computer algorithm. MPEP § 2106.05(f).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step is the same as in step 2A, prong 2.
Claim 45
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
[S]electing, as selected pixels or sets of pixels, pixels or sets of pixels from the at least one target image of the at least one target dataset that have a distance value that is equal to or larger than a threshold distance value to pixels or sets of pixels of the images of the first training dataset: This limitation could encompass the mental selection of the pixels with above a threshold distance to pixels of images of the first training set.
[M]anually annotating a label or assigning a value to the selected pixels or sets of pixels to obtain one or more first labeled images: This limitation could encompass the mental assignment of a value to pixels to obtain labeled images.
[A]dding the one or more first labeled images of the at least one target dataset to the second training dataset: This limitation could encompass the mental addition of the labeled image to the dataset.1
Step 2A Prong 2: This judicial exception is not integrated into a practical application. See claim 37 analysis.
Step 2B: The claim does not contain significantly more than the judicial exception. See claim 37 analysis.
Claim 46
Step 1: The claim is directed to an imaging device comprising a lens and a sensor array and a computer program; therefore, the claim is directed to the statutory category of machines.
Step 2A Prong 1: The claim recites, inter alia:
[S]elect[ing], as second training images: images or portions of images of the first training dataset that lie within a neighborhood of [a] target domain of the at least one target image, wherein the target domain differs from or is a sub-domain of a source domain defined by the first training dataset, and portions or full target images, of the at least one target image, of which output values are predicted by the global domain mathematical model with a level of confidence equal to or above a predetermined confidence threshold: This limitation could encompass mentally observing the output values of the model, mentally comparing them to a threshold, mentally defining a distance metric between images, mentally calculating the distances between the images in the dataset and the target image, and mentally selecting the images that either meet the threshold or are within less than a predetermined distance from the target image.
[G]enerat[ing] a second training dataset with the second training images: This limitation could encompass the mental generation of the dataset by mentally deciding which images to add to it.
[R]etrain[ing] the global domain mathematical model with the second training dataset to obtain a domain-adapted predictive model; … wherein retraining the global domain mathematical model comprises adjusting parameters of the global domain mathematical model executing the machine learning algorithm to obtain the domain-adapted predictive model that is optimized for analyzing images within the target domain: Given that training comprises a series of mathematical operations, the specific mathematical operation of adjusting parameters is claimed, and the model trained is described as “mathematical”, the retraining recites a mathematical concept.
[A]nalyz[ing] one or more images captured by the imaging device to assign … continuous or discrete values to the one or more images captured by the imaging device: This limitation could encompass mentally assigning values to the image.
[T]he neighborhood is computed by looking at a distance between feature vectors or a combination of feature vectors of the at least one target image and feature vectors or a combination of feature vectors of the images in the first training dataset: This limitation could encompass the mental computation of the neighborhood by mentally calculating the distance between vectors. Alternatively, this is a mathematical calculation.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “an imaging device configured to capture at least one target image, the imaging device comprising a lens and a sensor array; a global domain mathematical model trained with a first training dataset; and a processor …; wherein training the global domain mathematical model comprises executing a machine learning algorithm” and that the assignment is performed “using the domain-adapted predictive model”. These limitations amount to mere instructions to apply the judicial exception using a generic computer programmed with a generic class of computer algorithm and/or using a generically recited imaging device. MPEP § 2106.05(f).
The claim further recites “obtain[ing] at least one target dataset, wherein the at least one target dataset comprises the at least one target image that belongs to a target domain”. This limitation recites the insignificant extra-solution activity of mere data gathering. MPEP § 2106.05(g).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of step 2A prong 2, except insofar as the obtaining limitation, in addition to being insignificant extra-solution activity, also recites the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP § 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). As an ordered whole, the claim is directed to an abstract idea of training a mathematical model with a training dataset. Nothing in the claim provides significantly more than this. As such, the claim is not patent eligible.
Claim 47
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites the same judicial exceptions as in claim 46.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that “the first training dataset comprises a collection of images containing a plurality of images having characteristics which have been assigned semantic labels.” However, this limitation recites a mere instruction to apply the judicial exception using a generic computer. MPEP § 2106.05(f).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step is the same as at step 2A prong 2.
Claim 50
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites, inter alia, “generat[ing] the second training dataset comprising manually annotated full target images or portions of target images: classified by the global domain mathematical model with the level of confidence below the predetermined confidence threshold; or that the distance between the feature vectors or the combination of feature vectors of the images or portions of target images of the at least one target dataset to the feature vectors or the combination of feature vectors of the images of the first training dataset is equal to or larger than a threshold value.” This limitation could encompass the generation of the images using a pen and paper.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that the generation is performed using a “computer program”. However, this limitation recites a mere instruction to apply the judicial exception using a generic computer. MPEP § 2106.05(f).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step is the same as at step 2A prong 2.
Claim 51
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites the same judicial exceptions as in claim 46.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites that “the system is all or partially on-board an aerial vehicle, or a ground-based or separate aerial vehicle, with such ground- based or separate aerial vehicle in communication with a portion of the system; and the aerial vehicle is an aircraft, a spacecraft, a drone, a plane, an unmanned aerial vehicle, UAV, or a satellite.” However, this limitation merely restricts the judicial exception to the field of use of aerial photography captured with aerial vehicles. MPEP § 2106.05(h).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step is the same as at step 2A prong 2.
Claim Rejections - 35 USC § 103
Claims 32-33, 42, and 44 are rejected under 35 U.S.C. 103 as being unpatentable over Krishnamurthy et al. (US 20180060701) (“Krishnamurthy”) in view of Tsuchida et al. (US 20120030157) (“Tsuchida”) and further in view of Bhatt et al. (US 20160253597) (“Bhatt”).
Regarding claim 32, Krishnamurthy discloses “[a] computer-implemented method for optimizing an imaging system … automatically [to] analyze scenes within a target observation domain, the method comprising:
obtaining at least one target dataset, wherein the at least one target dataset comprises at least one target image that belongs to a target domain (dataset of images [target dataset comprising target images belonging to a target domain] is received [obtained]; dataset may include a gallery of stock photos stored as image files in a storage device accessible to a neural network – Krishnamurthy, paragraph 64); …
generating a second training dataset with … second training images (images are augmented [to create second training images]; augmenting the images involves placing cutout objects from the masked images onto various background images; augmenting the images distorts the objects – Krishnamurthy, paragraph 67; the dataset is then updated to include the augmented images [updated dataset = second dataset] – id. at paragraph 69);
retraining [a] global domain mathematical model with the second training dataset to obtain a domain-adapted predictive model (neural network [global domain mathematical model] is trained using the updated dataset [second dataset]; neurons of the network are trained on location-sensitive objective functions derived from the masks such as bounding box regression [so the model is mathematical in nature]; updated dataset is used automatically to extract a mask of additional images in the dataset to expand further the number of images in the dataset [i.e., the system can then be retrained based on the further expanded dataset that includes the updated/second dataset; note also that the model trained on the expanded dataset becomes a “domain-adapted” model relative to the original model insofar as it is trained on a wider range of data than the original data and therefore handles images from a wider domain than the original] – Krishnamurthy, paragraph 70);
analyzing one or more images captured by the imaging system to assign, using the domain-adapted predictive model, continuous or discrete values to the one or more images, the imaging system comprising a lens and a sensor array (network may be trained to perform semantic segmentation, which includes the task of labeling each pixel of an image with the label of the object class to which the pixel belongs [i.e., assign discrete values to the image]; the automated extraction of masks allows for re-training of the neural network on updated information input into the network [i.e., the network performing the segmentation may be re-trained] – Krishnamurthy, paragraph 70; see also Figs. 6-7 and 12 (depicting digital images, i.e., images captured by a camera with a lens and a sensor array));
wherein the global domain mathematical model is a mathematical model trained, by executing a machine learning algorithm, with images of the first training dataset (process for training [using a machine learning algorithm] the neural network [global domain mathematical model] includes convolving representations for a query side and a search side of the network, then reshaping and normalizing the convolved set [normalized set = first training dataset]; error with respect to the true mask or heat map is then minimized [reduced] by feeding the mask or heat map produced using the reshaped convolved set into a loss function [i.e., the error is measured across the first dataset] – Krishnamurthy, paragraphs 52-60; see also Fig. 8) ….”
Krishnamurthy appears not to disclose explicitly the further limitations of the claim. However, Tsuchida discloses “selecting, as second training [data points]: images or portions of [data points] of a first training dataset that lie within a neighborhood of the target domain of the at least one target [data point] (training data generation unit uses a distance between training data candidates and a representative point of a cluster as a criterion for the degree of cluster membership of the training data candidates [i.e., it measures the distance between the data points and the representative/target point and selects those that are within a neighborhood of the target] – Tsuchida, paragraph 144; training data generation unit obtains the distribution of the labels of the training data candidates and identifies [selects] training data candidates that meet a preset condition based on the identified distribution – id. at paragraph 140), … and portions or full target [data points], of the at least one target [data point], of which output values are predicted by a global domain mathematical model with a level of confidence equal to or above a predetermined confidence threshold (for training data candidates that are not assigned the specified labels, the training data generation unit obtains the degree of membership in the clusters, or the confidence level; the training data generation unit treats training data candidates whose confidence level is equal to or greater than a threshold value as training data [i.e., selects them as training data] – Tsuchida, paragraph 67; see also paragraph 35 (indicating that k-means may be used to perform the clustering, i.e., the predictions of cluster membership are performed by a mathematical model)); …
wherein the neighborhood is computed by looking at a distance between feature vectors or a combination of feature vectors of the at least one target [data point] and feature vectors or a combination of feature vectors of the [data points] in the first training dataset (training data generation unit uses a distance between training data candidates and a representative point of a cluster as a criterion for the degree of cluster membership of the training data candidates [i.e., it measures the distance between the data points [feature vectors in the first dataset] and the representative/target point and selects those that are within a neighborhood of the target] – Tsuchida, paragraph 144; see also paragraph 55 (disclosing that the training data candidates may be represented as vectors)).”
Tsuchida and the instant application both relate to manipulation of training data for machine learning and are analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Krishnamurthy to add points to a training dataset based on their distance to a target data point and based on their confidence values exceeding a threshold, as disclosed by Tsuchida, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would increase the accuracy of the resulting model by expanding the training data available to train it. See Tsuchida, paragraph 10.
Neither Krishnamurthy nor Tsuchida appears to disclose explicitly the further limitations of the claim. However, Bhatt discloses that “the target domain differs from or is a sub-domain of a source domain defined by the first training dataset (assumption that training and test data belong to the same domain does not always hold true, and performance is reduced when the data distribution in the test domain differs from that in the training domain – Bhatt, paragraph 3) …; [and]
retraining the global domain mathematical model comprises adjusting parameters of the mathematical model executing the machine learning algorithm to obtain the domain-adapted predictive model that is optimized for analyzing images within the target domain (confidently predicted images in an iteration are used to re-train a target-specific [optimized for the target domain] classifier [machine learning algorithm] and update [adjust] ensemble [mathematical model executing the ML algorithm] weights [parameters]; this process is performed until all unlabeled instances are confidently predicted or a predefined maximum number of iterations is exhausted – Bhatt, paragraph 73; objects to be classified in the target domain can be images – id. at paragraph 26) ….”
Bhatt and the instant application both relate to domain adaptation for machine learning and are analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Krishnamurthy and Tsuchida to retrain the model by adjusting the parameters of the model to optimize it for the target domain, as disclosed by Bhatt, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would eliminate the need to relearn a new model from scratch each time for different test data. See Bhatt, paragraph 5.
Regarding claim 33, the rejection of claim 32 is incorporated. Krishnamurthy further discloses that “the feature vectors or the combination of feature vectors are totally or partially derived from a pre-trained machine learning model (neural network generates a heat map identifying a location of pixels corresponding to a query object within a visual medium; the representations include a vector having values defining the features of the query object and the visual medium [image] – Krishnamurthy, paragraph 5 [note that the fact that the network is capable of producing these vectors implies that it was pre-trained to do so]).”
Regarding claim 42, Krishnamurthy discloses that “the global domain mathematical model is trained and retrained to learn segmentation of images of the at least one target dataset comprising aerial or satellite images based on land use classes; or wherein the global domain mathematical model is trained and retrained to automatically predict continuous or discrete values from imagery content to identify elements on the surface of the Earth and/or extract data (expanded dataset improves the accuracy of automated classification of objects [prediction of discrete values, wherein the prediction is the datum extracted] in visual media [imagery content] – Krishnamurthy, paragraph 24; see rejection of claim 32 supra for description of the training and retraining of the model).”
Regarding claim 44, Krishnamurthy discloses that “training and retraining the global domain mathematical model, and generating the second training dataset are performed using at least one of artificial neural networks, deep learning techniques, non-supervised machine learning methods, semi-supervised machine learning methods, or convolutional neural networks (neural network [artificial neural network] is trained using the updated dataset [second dataset]; neurons of the network are trained on location-sensitive objective functions derived from the masks such as bounding box regression [so the model is mathematical in nature]; updated dataset is used automatically to extract a mask of additional images in the dataset to expand further the number of images in the dataset [i.e., the system can then be retrained based on the further expanded dataset that includes the updated/second dataset; note also that the model trained on the expanded dataset becomes a “global” model relative to the original model insofar as it is trained on a wider range of data than the original data] – Krishnamurthy, paragraph 70).”
Claims 41, 46-47, and 51 are rejected under 35 U.S.C. 103 as being unpatentable over Krishnamurthy in view of Tsuchida and further in view of Mithal et al. (US 20180130193) (“Mithal”) and Bhatt.
Regarding claim 41, Krishnamurthy/Tsuchida/Bhatt appears not to disclose explicitly the further limitations of the claim. However, Mithal discloses that “the at least one target image and/or the one or more images are captured by an imaging system all or partially onboard an aerial vehicle, wherein the aerial vehicle is a satellite, a spacecraft, an aircraft, a plane, an unmanned aerial vehicle, UAV, or a drone (system that automatically labels satellite data to determine the extent of bodies of water includes a satellite that contains multiple sensors that collect frames of sensor values [target dataset] – Mithal, paragraph 116; see also paragraph 30 (disclosing that the system is for satellite image classification, i.e., the sensors are imaging devices)).”
Mithal and the instant application both relate to image classification using satellites and are analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Krishnamurthy/Tsuchida/Bhatt to capture the image data using a satellite, as disclosed by Mithal, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would allow the system to analyze images that cover a wider area than would be available with only ground-based imagery. See Mithal, paragraph 116.
Regarding claim 46, Krishnamurthy discloses “[a] … system comprising:
an imaging device configured to capture at least one target image (the phrase “query object” refers to an image; such an image includes a photograph [implying the existence of a camera or imaging device to capture the photograph] – Krishnamurthy, paragraph 26), the imaging device comprising a lens and a sensor array (Krishnamurthy Figs. 6-7 and 12 depict digital images, i.e., images captured by a camera with a lens and a sensor array);
a global domain mathematical model trained with a first training dataset (process for training the neural network [global domain mathematical model] includes convolving representations for a query side and a search side of the network, then reshaping and normalizing the convolved set [normalized set = first training dataset]; error with respect to the true mask or heat map is then minimized [reduced] by feeding the mask or heat map produced using the reshaped convolved set into a loss function [i.e., the error is measured across the first dataset] – Krishnamurthy, paragraphs 52-60; see also Fig. 8); and
a processor (Krishnamurthy Fig. 13, processor 1302 that runs instructions) configured to:
obtain at least one target dataset, wherein the at least one target dataset comprises the at least one target image that belongs to a target domain (dataset of images [target dataset comprising images belonging to a target domain] is received [obtained]; dataset may include a gallery of stock photos stored as image files in a storage device accessible to a neural network – Krishnamurthy, paragraph 64); …
generate a second training dataset with … second training images (images are augmented [to generate second training images]; augmenting the images involves placing cutout objects from the masked images onto various background images; augmenting the images distorts the objects – Krishnamurthy, paragraph 67; the dataset is then updated to include the augmented images [updated dataset = second dataset] – id. at paragraph 69); and
retrain the global domain mathematical model with the second training dataset to obtain a domain-adapted predictive model (neural network [global domain mathematical model] is trained using the updated dataset [second dataset]; neurons of the network are trained on location-sensitive objective functions derived from the masks such as bounding box regression [so the model is mathematical in nature]; updated dataset is used automatically to extract a mask of additional images in the dataset to expand further the number of images in the dataset [i.e., the system can then be retrained based on the further expanded dataset that includes the updated/second dataset; note also that the model trained on the expanded dataset becomes a “domain-adapted” model relative to the original model insofar as it is trained on a wider range of data than the original data and therefore handles images from a wider domain than the original] – Krishnamurthy, paragraph 70); and
analyze one or more images captured by the imaging device to assign, using the retrained domain-adapted predictive model, continuous or discrete values to the one or more images captured by the imaging device (network may be trained to perform semantic segmentation, which includes the task of labeling each pixel of an image with the label of the object class to which the pixel belongs [i.e., assign discrete values to the image]; the automated extraction of masks allows for re-training of the neural network on updated information input into the network [i.e., the network performing the segmentation may be re-trained] – Krishnamurthy, paragraph 70);
wherein training the global domain mathematical model comprises executing a machine learning algorithm (process for training [using a machine learning algorithm] the neural network [global domain mathematical model] includes convolving representations for a query side and a search side of the network, then reshaping and normalizing the convolved set – Krishnamurthy, paragraphs 52-60; see also Fig. 8) ….”
Krishnamurthy appears not to disclose explicitly the further limitations of the claim. However, Tsuchida discloses “select[ing], as second training [data points]: images or portions of [data points] of a first training dataset that lie within a neighborhood of [a] target domain of the at least one target [data point] (training data generation unit uses a distance between training data candidates and a representative point of a cluster as a criterion for the degree of cluster membership of the training data candidates [i.e., it measures the distance between the data points and the representative/target point and selects those that are within a neighborhood of the target] – Tsuchida, paragraph 144; training data generation unit obtains the distribution of the labels of the training data candidates and identifies [selects] training data candidates that meet a preset condition based on the identified distribution – id. at paragraph 140), … and
portions or full target [data points], of the at least one target [data point], of which output values are predicted by the global domain mathematical model with a level of confidence equal to or above a predetermined confidence threshold (for training data candidates that are not assigned the specified labels, the training data generation unit obtains the degree of membership in the clusters, or the confidence level; the training data generation unit treats training data candidates whose confidence level is equal to or greater than a threshold value as training data [i.e., selects them as training data] – Tsuchida, paragraph 67; see also paragraph 35 (indicating that k-means may be used to perform the clustering, i.e., the predictions of cluster membership are performed by a mathematical model)); …
wherein the neighborhood is computed by looking at a distance between feature vectors or a combination of feature vectors of the at least one target [data point] and feature vectors or a combination of feature vectors of the [data points] in the first training dataset (training data generation unit uses a distance between training data candidates and a representative point of a cluster as a criterion for the degree of cluster membership of the training data candidates [i.e., it measures the distance between the data points [feature vectors in the first dataset] and the representative/target point and selects those that are within a neighborhood of the target] – Tsuchida, paragraph 144; see also paragraph 55 (disclosing that the training data candidates may be represented as vectors)).” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Krishnamurthy to add points to a training dataset based on their distance to a target data point and based on their confidence values exceeding a threshold, as disclosed by Tsuchida, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would increase the accuracy of the resulting model by expanding the training data available to train it. See Tsuchida, paragraph 10.
Neither Krishnamurthy nor Tsuchida appears to disclose explicitly the further limitations of the claim. However, Mithal discloses that the system is “aerial or satellite-based (Mithal paragraph 30 discloses that the system is for satellite image classification) ….” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Krishnamurthy to capture the image data using a satellite, as disclosed by Mithal, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would allow the system to analyze images that cover a wider area than would be available with only ground-based imagery. See Mithal, paragraph 116.
Neither Krishnamurthy, Mithal, nor Tsuchida appears to disclose explicitly the further limitations of the claim. However, Bhatt discloses that “the target domain differs from or is a sub-domain of a source domain defined by the first training dataset (assumption that training and test data belong to the same domain does not always hold true, and performance is reduced when the data distribution in the test domain differs from that in the training domain – Bhatt, paragraph 3) …; [and]
retraining the global domain mathematical model comprises adjusting parameters of the mathematical model executing the machine learning algorithm to obtain the domain-adapted predictive model that is optimized for analyzing images within the target domain (confidently predicted images in an iteration are used to re-train a target-specific [optimized for the target domain] classifier [machine learning algorithm] and update [adjust] ensemble [mathematical model executing the ML algorithm] weights [parameters]; this process is performed until all unlabeled instances are confidently predicted or a predefined maximum number of iterations is exhausted – Bhatt, paragraph 73; objects to be classified in the target domain can be images – id. at paragraph 26) ….” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Krishnamurthy, Mithal, and Tsuchida to retrain the model by adjusting the parameters of the model to optimize it for the target domain, as disclosed by Bhatt, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would eliminate the need to relearn a new model from scratch each time for different test data. See Bhatt, paragraph 5.
Regarding claim 47, Krishnamurthy/Tsuchida/Mithal/Bhatt discloses that “the first training dataset comprises a collection of images containing a plurality of images having characteristics which have been assigned semantic labels (network may be trained to produce a semantic segmentation of an input image; semantic segmentation includes the task of labeling each pixel of an image with the label of the object class to which the pixel belongs; the set of pixels in a single label forms a mask of the object [i.e., the first training dataset]; automated extraction of masks [i.e., multiple images] allows for retraining the neural network on updated information – Krishnamurthy, paragraph 70).”
Regarding claim 51, Krishnamurthy, as modified by Tsuchida, Bhatt, and Mithal, discloses that “the system is all or partially on-board an aerial vehicle, or a ground-based or separate aerial vehicle, with such ground-based or separate aerial vehicle in communication with a portion of the system; and the aerial vehicle is an aircraft, a spacecraft, a drone, a plane, an unmanned aerial vehicle, UAV, or a satellite (system that automatically labels satellite data to determine the extent of bodies of water includes a satellite that contains multiple sensors that collect frames of sensor values [target dataset] – Mithal, paragraph 116; see also paragraph 30 (disclosing that the system is for satellite image classification, i.e., the sensors are imaging devices), Figure 13 (showing that the satellite is in communication with a receiving dish that transmits the data to the classifier)).” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Krishnamurthy to capture the image data using a satellite, as disclosed by Mithal, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would allow the system to analyze images that cover a wider area than would be available with only ground-based imagery. See Mithal, paragraph 116.
Claim 34 is rejected under 35 U.S.C. 103 as being unpatentable over Krishnamurthy in view of Tsuchida and Bhatt and further in view of Lin et al. (US 20160379091) (“Lin”).
Regarding claim 34, the rejection of claim 33 is incorporated. The combination of Krishnamurthy, Bhatt, and Tsuchida further discloses “selecting images or portions of images of the first training dataset that lie within the neighborhood of the domain of the at least one target image”, as shown in the rejection of claim 32. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Krishnamurthy/Bhatt to add data points to a dataset based on a distance measure, as disclosed by Tsuchida, for the reasons given in the rejection of claim 32.
Lin further discloses “generating the feature vectors or the combination of feature vectors for each pixel or set of pixels of each of the at least one target image of the at least one target dataset (set of semantic features (e.g., certain colors [set of pixels]) of a training image is represented using a feature vector; for example, a classifier algorithm can generate one or more feature vectors representing the semantic features 212 in class 210 and can also generate one or more additional vectors representing semantic features 216 in class 214 – Lin, paragraph 34 and Fig. 2);
generating the feature vectors or the combination of feature vectors for each pixel or set of pixels of each image of the first training dataset (set of semantic features (e.g., certain colors [set of pixels]) of a training image is represented using a feature vector; for example, a classifier algorithm can generate one or more feature vectors representing the semantic features 212 in class 210 and can also generate one or more additional vectors representing semantic features 216 in class 214 [i.e., vectors are generated for two separate datasets] – Lin, paragraph 34 and Fig. 2);
computing the distance between the feature vectors or the combination of feature vectors (classifier algorithm can generate feature vectors for the two images to determine if both images belong to one of the classes 210, 214 – Lin, paragraph 34; degree of similarity may be determined by determining a distance between feature vectors for a cluster and an input image – id. at paragraph 71); and
selecting pixels or sets of pixels of the images of the first training dataset that have a distance value lower than a threshold distance to the pixels or sets of pixels of the at least one target image of the at least one target dataset (degree of similarity between an input image and a cluster may be determined by determining a distance between feature vectors for a cluster and an input image; a sufficiently small distance (i.e., below a threshold distance) can indicate that the semantic content of the input image is similar to the semantic content of the first example tagged image; therefore, the classifier algorithm selects the first example tagged image [set of pixels] based on the determined distance – Lin, paragraph 71 [target dataset = set of example tagged images; first training dataset = set of input images]).”
Lin and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Krishnamurthy/Tsuchida/Bhatt to select images that have similarities with other images that are less than a threshold, as disclosed by Lin, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would ensure that outlier images or other images which may adversely affect the training are not added to the dataset. See Lin, paragraph 71.
Claim 35 is rejected under 35 U.S.C. 103 as being unpatentable over Krishnamurthy in view of Tsuchida, Bhatt, and Lin and further in view of Gong et al. (US 20150117767) (“Gong”).
Regarding claim 35, neither Krishnamurthy, Tsuchida, Bhatt, nor Lin appears to disclose explicitly the further limitations of the claim. However, Gong discloses that “an individual feature vector of the feature vectors is the result of combining different feature vectors comprising histograms of gradient orientations (HOG), red-green-blue (RGB) color histograms, texture histograms, response to wavelets filters, artificial neural networks, deep neural network features extracted from a pre-trained model, and their combinations (various air-quality related features are extracted from a reference clear image, various corresponding air quality related features are extracted from a training image, and differences among the features are calculated as training data in training a model; the feature comprises at least one of [i.e., possibly more than one of] luminance, chrominance, texture, and gradient density; luminance may be embodied as a histogram feature of a luminance distribution map extracted by transforming RBG color values of pixels into luminance values [RGB color histograms]; gradient may be embodied as a HOG (Histogram of Oriented Gradient) feature; the histogram of each chunk may be combined into a large vector as an overall gradient HOG feature of the image – Gong, paragraphs 40-43; see also paragraph 44 [disclosing texture features]); or wherein the feature vectors, the way the feature vectors are combined, and a function that measures the distance between the feature vectors, are selected depending on one or more image transformation invariances, wherein the one or more image transformation invariances include any combination of translations, rotations, scaling, shear, image blur, or image brightness and contrast changes.”
Gong and the instant application both relate to image processing and are analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Krishnamurthy, Tsuchida, Bhatt, and Lin to create the vector using HOG, RGB histogram, and texture features, as disclosed by Gong, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would give the system a more complete picture of the image’s features than if only a single technique were used, leading to better effects. See Gong, paragraph 45.
Claims 37, 40, and 45 are rejected under 35 U.S.C. 103 as being unpatentable over Krishnamurthy in view of Tsuchida and Bhatt and further in view of Huval (US 20180373980) (“Huval”).
Regarding claim 37, Krishnamurthy, as modified by Tsuchida, Bhatt, and Huval, discloses that “the predetermined confidence threshold is defined in relation to a level of accuracy in a prediction of an identification, classification or labeling process of the at least one target image of the at least one target dataset (neural network can calculate a confidence score for each automated label attributed to an automatically-defined location in an optical image; for example, the neural network can calculate a first confidence score that the object is a first object type represented by a first automated label; if the first confidence score is less than the preset threshold score, the system can serve the image to a human annotator [i.e., the threshold confidence represents a threshold probability that the system’s prediction that the image contains the object of the first type is accurate] – Huval, paragraph 34).” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Krishnamurthy/Tsuchida/Bhatt to define the threshold confidence in terms of prediction accuracy, as disclosed by Huval, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would ensure the accuracy of the machine learning by ensuring that only data with sufficiently high-quality labels are used. See Huval, paragraph 34.
Regarding claim 40, Krishnamurthy/Tsuchida/Bhatt appears not to disclose explicitly the further limitations of the claim. However, Huval discloses that “the second training dataset further comprises manually labeled full images or portions of images of the at least one target dataset that:
were classified by the global domain mathematical model with a level of confidence below a predetermined threshold (neural network [global domain mathematical model] can calculate a confidence score for automated labels attributed to an automatically-defined location in an optical image; if the first, highest confidence score is less than a preset threshold score, the remote computer system can serve the optical image without automated labels to a human annotator for manual labeling – Huval, paragraph 34); or
the distance between the feature vectors or the combination of feature vectors of the images or portions of target images of the at least one target dataset to the feature vectors or the combination of feature vectors of the images of the first training dataset is equal to or larger than a threshold value; or were classified by the global domain mathematical model with the level of confidence below the predetermined confidence threshold and the distance between the feature vectors or the combination of feature vectors of the target images or portions of images of the at least one target dataset to the feature vectors or the combination of feature vectors of the images of the first training dataset is equal to or larger than a threshold value.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Krishnamurthy/Tsuchida/Bhatt to have a human annotator manually label images that were not classified with sufficient confidence, as disclosed by Huval, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would increase the accuracy of the labeling by ensuring that images are not automatically assigned labels of dubious accuracy. See Huval, paragraph 34.
Regarding claim 45, Krishnamurthy, as modified by Tsuchida, Bhatt, and Huval, discloses “selecting, as selected pixels or sets of pixels, pixels or sets of pixels from the at least one target image of the at least one target dataset that have a distance value that is equal to or larger than a threshold distance value to pixels or sets of pixels of the images of the first training dataset;
manually annotating a label or assigning a value to the selected pixels or sets of pixels to obtain one or more first labeled images; and
adding the one or more first labeled images of the at least one target dataset to the second training dataset; or
selecting, as one or more selected target images, portions or full target images from the at least one target dataset that were predicted by the global domain mathematical model with a predetermined level of confidence below the predetermined confidence threshold (neural network [global domain mathematical model] can calculate a confidence score for automated labels attributed to an automatically-defined location in an optical image; if the first, highest confidence score is less than a preset threshold score, the remote computer system can serve the optical image without automated labels to a human annotator for manual labeling [i.e., the image is selected for human annotation] – Huval, paragraph 34);
manually annotating a label or assigning a value to pixels or sets of pixels of the one or more selected target images to obtain one or more second labeled images (neural network [global domain mathematical model] can calculate a confidence score for automated labels attributed to an automatically-defined location in an optical image; if the first, highest confidence score is less than a preset threshold score, the remote computer system can serve the optical image without automated labels to a human annotator for manual labeling – Huval, paragraph 34; see also paragraph 37 (disclosing that the image comprises pixels)); and
adding the one or more second labeled images to the second training dataset (Huval Fig. 3 shows that the manually labeled images are added to the training set).” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Krishnamurthy/Tsuchida/Bhatt to assign images with low-confidence labels to a manual annotator, as disclosed by Huval, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would increase the accuracy of the labeling by ensuring that images are not automatically assigned labels of dubious accuracy. See Huval, paragraph 34.
Claim 38 is rejected under 35 U.S.C. 103 as being unpatentable over Krishnamurthy in view of Huval, Bhatt, and Tsuchida and further in view of Badrinarayanan et al., “Semi-Supervised Video Segmentation using Tree Structured Graphical Models,” in 35.11 IEEE Transactions on Pattern Analysis and Machine Intelligence 2751-64 (2013) (“Badrinarayanan”).
Regarding claim 38, the rejection of claim 36 is incorporated. Huval further discloses that “a threshold value per class is predetermined, and … the prediction from the global domain mathematical model in the pixels of the portions or full target images is above the predetermined confidence threshold (neural network [global domain mathematical model] can detect an object in an optical image [comprising pixels]; calculate a first confidence score that the object is a first object type represented by a first automated label; and calculate a second confidence score that the object is a second object type represented by a second automated label [i.e., the confidence scores are determined on a per-class basis]; if the first, highest confidence score exceeds a threshold score, the system can write the first automated label to the object; if all confidence scores of object types remain below the preset threshold score, the human annotator can manually indicate an object [i.e., the threshold is determined for each of the classes] – Huval, paragraphs 53-54).” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Krishnamurthy/Tsuchida/Bhatt to select images whose classification confidence is above a threshold, as disclosed by Huval, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would ensure the accuracy of the machine learning by ensuring that only data with sufficiently high-quality labels are used. See Huval, paragraph 34.
Neither Krishnamurthy, Tsuchida, Bhatt, nor Huval appears to disclose explicitly the further limitations of the claim. However, Badrinarayanan discloses that “the portions or full target images are obtained by using a semi-supervised machine learning method and selected using their pixel-wise confidence levels (as probabilistic inference of pixel labels is performed, a family of labellings at various confidence levels is available to the user as output; the user can select one of these labellings at one or more frames [based on the confidence levels], fix or clamp them, and re-infer the labelling over the whole video; this is similar to the self-training approach used in semi-supervised learning; structural variational inference scheme infers pixel-wise labels and their confidences – Badrinarayanan, last three paragraphs before sec. 2) ….”
Badrinarayanan and the instant application both relate to semi-supervised image classification algorithms and are analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Krishnamurthy, Tsuchida, Bhatt, and Huval to select the images based on pixel-wise confidence levels using a semi-supervised machine learning method, as disclosed by Badrinarayanan, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would ensure that additional mechanisms of selecting super-pixels at the right scale are not needed, as would be the case if an unsupervised method were used. See Badrinarayanan, sec. 1, third paragraph.
Claim 43 is rejected under 35 U.S.C. 103 as being unpatentable over Krishnamurthy in view of Tsuchida and Bhatt and further in view of Dunlop et al. (US 20130259390) (“Dunlop”).
Regarding claim 43, Krishnamurthy/Tsuchida/Bhatt appears not to disclose explicitly the further limitations of the claim. However, Dunlop discloses that “the global domain mathematical model segments image contents with image content labels selected from a group comprising water bodies, rivers, lakes, dams, forests, bare lands, waste dumps, buildings, roads, crop types, crop growth, soil composition, mines, oil and gas infrastructure (image regions/segments may be labeled as including buildings – Dunlop, paragraph 161; see also paragraph 113 (disclosing that the classifier assigns classification scores to images or regions of images identifying the probability that a given image contains a particular type of content [i.e., it is a mathematical model])).”
Dunlop and the instant application both relate to image analysis using machine learning and are analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Krishnamurthy/Tsuchida/Bhatt to segment image contents with labels for buildings, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would allow a system utilizing the model to understand its environment better, thereby enabling it to draw conclusions about the environment. See Dunlop, paragraph 161.
Claim 50 is rejected under 35 U.S.C. 103 as being unpatentable over Krishnamurthy in view of Tsuchida, Bhatt, and Mithal and further in view of Huval.
Regarding claim 50, Krishnamurthy, as modified by Tsuchida, Mithal, Bhatt, and Huval, discloses that “the processor is further configured to generate the second training dataset comprising manually annotated full target images or portions of target images:
classified by the global domain mathematical model with the level of confidence below the predetermined confidence threshold (neural network [global domain mathematical model] can calculate a confidence score for automated labels attributed to an automatically-defined location in an optical image; if the first, highest confidence score is less than a preset threshold score, the remote computer system can serve the optical image without automated labels to a human annotator for manual labeling – Huval, paragraph 34); or
that the distance between the feature vectors or the combination of feature vectors of the images or portions of target images of the at least one target dataset to the feature vectors or the combination of feature vectors of the images of the first training dataset is equal to or larger than a threshold value.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Krishnamurthy/Tsuchida/Mithal/Bhatt to have a human annotator manually label images that were not classified with sufficient confidence, as disclosed by Huval, and an ordinary artisan could reasonably expect to have done so successfully. Doing so would increase the accuracy of the labeling by ensuring that images are not automatically assigned labels of dubious accuracy. See Huval, paragraph 34.
Response to Arguments
Applicant's arguments filed May 26, 2026 (“Remarks”) have been fully considered but they are, except insofar as rendered moot by the introduction of a new ground of rejection, not persuasive.
Applicant first argues that the claims as amended are eligible under 35 USC § 101 insofar as they recite a technical improvement to the field of remote sensing and domain adaptation. To summarize Applicant’s extensive remarks, Applicant alleges that: (a) any judicial exception recited is integrated into a practical application because the specification recites an improvement to remote sensing and domain adaptation algorithms that is reflected in the alleged additional elements of obtaining a target dataset, selecting second training images, generating a second training dataset, using a predictive model and an imaging system, adjusting parameters of the mathematical model, and computing the neighborhood by looking at distances between feature vectors; (b) the claim when viewed as a whole is therefore directed to these improvements in remote sensing and domain adaptation; (c) the instant claims are analogous to those at issue in Ex parte Desjardins because the selecting, generating, and retraining steps are additional elements that recite an improvement to technology; and (d) the claims contain significantly more than the abstract idea because they recite a particular way to achieve a desired outcome. Remarks at 9-19.
Regarding (a), a significant portion of the limitations that Applicant alleges to integrate the judicial exception into a practical application are in fact part of the judicial exception itself. For instance, selecting second training images and generating a training dataset could, but for the recitation that these elements are performed using machine learning, practically mentally performable. As noted in the rejection, the adjustment of parameters of a mathematical model and computation of the neighborhood are mathematical concepts. That leaves only the obtaining limitation, which amounts to insignificant extra-solution activity that is well-understood, routine, and conventional for the reasons given in the rejection. Examiner reminds Applicant that the judicial exception itself cannot provide the inventive concept. MPEP § 2106.05(I).
Regarding (b), the claims are, by their very terms, not directed to improvements in remote sensing and domain adaptation. Rather, they invoke domain adaptation and the resulting “optimiz[ed] … imaging system” purely as tools to perform the abstract idea, recited in the preamble to claim 32, of “analyz[ing] scenes within a target observation domain”. Indeed, the last substantive limitation of claim 32, other than the three wherein clauses that follow, is “analyzing one or more images captured by the imaging system to assign … continuous or discrete values to the one or more images”, which further reinforces that the claims are directed to image analysis, not domain adaptation.
Regarding (c), Desjardins is inapposite. Desjardins stands generally for the proposition that claims directed to improvements in machine learning model training can be patent eligible when the specification discloses such an improvement and that improvement is reflected in the claim language. Unlike in the instant application, the Desjardins claims specifically noted that the improvement was that the model avoids the catastrophic forgetting problem by improving performance on one task without sacrificing performance on another. No such statement is present in the claims here. While it is true that the benefit of the claimed invention need not be explicitly recited in the claims, the instant application is also distinguishable from Desjardins because the training itself recites a mathematical concept, thereby rendering the training limitations analogous to claim 2 of Example 47. Therefore, as previously noted, the training and retraining cannot provide the inventive concept, because they are part of the judicial exception itself.
Regarding (d), insofar as only one additional element of the independent claims recites insignificant extra-solution activity and that limitation was shown to be conventional in the rejection, the conventionality of the other additional elements, either standing alone or in ordered combination, is largely irrelevant. As previously noted, when evaluated in ordered combination, the claims are directed to the abstract idea of image analysis, not to an improvement in the domain-adapted models that execute the analysis.
Applicant’s equally extensive remarks with regard to the art rejection are largely moot in light of the use of newly cited reference Bhatt to teach the disputed limitations. Insofar as Applicant is arguing that Tsuchida fails to disclose determining differences between source domains and target domains, see, e.g., Remarks at 22-23, Examiner would note that source domains were not recited until the current iteration of the claims. Tsuchida does disclose “target” images insofar as any domain in which the system operates is necessarily the target domain.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN C VAUGHN whose telephone number is (571)272-4849. The examiner can normally be reached M-R 7:00a-5:00p ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar, can be reached at 571-272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN C VAUGHN/ Primary Examiner, Art Unit 2125
1 The remainder of the claim is stated as an alternative to the above three limitations; thus, it is not required by the broadest reasonable interpretation of the claim in light of the specification.
Prosecution Insights