DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “820” has been used to designate both local views of the anchor patch in Fig. 8 and local views of both the anchor patch and the rotated anchor patch in Fig. 9. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: 1205, 1210, 1220, and 1225 in ¶0061-0062 and 1300, 1305, 1310, 1315, and 1320 in ¶0065. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Color photographs and color drawings are not accepted in utility applications unless a petition filed under 37 CFR 1.84(a)(2) is granted. Any such petition must be accompanied by the appropriate fee set forth in 37 CFR 1.17(h), one set of color drawings or color photographs, as appropriate, if submitted via the USPTO patent electronic filing system or three sets of color drawings or color photographs, as appropriate, if not submitted via the via USPTO patent electronic filing system, and, unless already present, an amendment to include the following language as the first paragraph of the brief description of the drawings section of the specification:
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
Color photographs will be accepted if the conditions for accepting color drawings and black and white photographs have been satisfied. See 37 CFR 1.84(b)(2).
Specification
The disclosure is objected to because of the following informalities: the abbreviation “DBSCAN” is mentioned in ¶0051 and ¶0087 but not defined. Examiner suggests amending ¶0051 to include the definition.
Appropriate correction is required.
Claim Objections
Claims 1-20 are objected to because of the following informalities:
Regarding claims 1 and 15, the preamble should be followed by a colon. Examiner suggests amending to “comprising the steps:” in claim 1 and “imagery comprising:” in claim 15.
Regarding claims 1, 15, and 16, the limitations of the claims should be separated with semi-colons instead of commas.
Regarding claims 2-14 and 17-20, a comma should precede the start of the “wherein…” of the dependent claims. For example, Examiner suggests amending claim 2 to “The method of claim 1, wherein…”.
Regarding claim 4, “the associated centers” lacks antecedent basis. Examiner suggests amending to “the objects centered within the associated patch”.
Regarding claim 9, “the patches” lacks antecedent basis. It is introduced in claim 2 but claim 9 only claims dependency from claim 1.
Regarding claim 10, the abbreviation “DBSCAN” is not defined. Examiner suggests amending to “Density-Based Spatial Clustering of Applications with Noise (DBSCAN)”.
Regarding claim 14, the list of claim elements is recited in the alternative (“one of a group”) but uses “and” as the conjunction. For clarity, Examiner suggests amending to “or”.
Regarding claim 15, it seems the claim is not a proper system claim because it is not drafted with a clear structure that describes the component and is in a relationship with it. Further, the Examiner notes that the functions are recited as providing, training, and automating. However, it is not clear if those functions are being performed by potential recitation of a structure such as processor, data storage, and user interface (see MPEP §2114). Examiner suggests, with the first limitation as an example, amending from “in a processor and associated data storage, providing…” to “a processor and associated data storage configured to provide…”.
Regarding claim 17, “the associated centers” lacks antecedent basis. Examiner suggests amending to “their associated centers”.
Regarding claim 19, “the patches” lacks antecedent basis. It is introduced in claim 17 but claim 19 only claims dependency from claim 16.
Regarding claim 20, “a teach model” appears to be a typo. Examiner suggests amending to “a teacher model”.
Appropriate correction is required.
Positive Statement Regarding - 35 USC § 101
The Examiner’s 35 U.S.C. 101 analysis recognizes that the claimed subject matter is directed to a practical application of a technical solution. The claimed elements, taken as a whole, improve the functioning of methods for searching geospatial systems by overcoming rational invariance and differences in shape and texture, see ¶006-008. Because the claims recite specific, claimed steps and structural elements that produce a tangible technical result, they are not directed to an abstract idea absent additional inventive concept limitations. Accordingly, the record supports a positive 101 determination for the present claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 5, 13-16, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Yuan et al. (Yuan, Zhiqiang, et al. "A lightweight multi-scale crossmodal text-image retrieval method in remote sensing." IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-19) (hereafter, “Yuan2021”) in view of You et al. (You, Huaxin, Edward Y. Chang, and Beitao Li. "NNEW: Nearest Neighbor Expansion By Weighting In Image Database Retrieval." ICME. 2001) (hereafter, “You”).
Regarding claim 1, Yuan2021 discloses a self-supervised method for detecting objects of interest in geospatial imagery (Fig. 5; Fig. 7) comprising the steps providing a dataset representative of a plurality of both natural and man-made objects such as would appear in geospatial images (Fig. 5a; Fig. 7; page 10, right column, “RSICD dataset … RSITMD dataset … UCM dataset … Sydney dataset”. Examiner considers the datasets listed on page 10 to comprise geospatial imagery. Figs. 5a and 7 illustrate that these images contain various trees and buildings, which correspond to “natural” and “man-made” objects, respectively), training a teacher-student model with at least some of the dataset (Fig. 3; page 6, right column, first paragraph, we let the lightweight model (student network) learn the hidden representation of the deep model (teacher network)), providing an abstract of an object of interest (Fig. 3, Text on right of figure: some green trees are near a large piece of bare land; Page 3, §RS Crossmodal Text-Image Retrieval, RS crossmodal text-image retrieval refers to retrieve RS images by text. Examiner considers the text string as the “abstract”), the abstract comprising one or more invariant features of the object of interest but insufficient to fully characterize the object of interest (Fig. 3, Text on right of figure: some green trees are near a large piece of bare land; Fig. 6, Text queries. Examiner considers the text query examples to demonstrate features of the object such as the color of the trees but not fully characterizing the object (relative instead of exact location)), and [automating search patterns in the dataset] that actualize the abstract to cause the model to return images responsive to the abstract (Page 12, §F. RS Image Recall Analysis, The left of Fig. 6 shows two examples of image retrieval … using text to retrieve images of the two ships next to the harbor).
However, Yuan2021 fails to explicitly disclose automating search patterns in the dataset.
You teaches automating search patterns in the dataset (Page 246, right column, §1.3 Our Contributions, our method approximates the user’s queries by changing the query model dynamically).
Both Yuan2021 and You are analogous to the claimed invention because they are both directed towards image retrieval methods. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the dynamic queries of You into the model of Yuan2021. The suggestion/motivation for doing so would have been to improve performance, as suggested by You at Abstract, our method uses intelligent modeling and learning to increase query speed and accuracy.
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You to obtain the invention as specified in claim 1.
Regarding claim 5, in which claim 1 is incorporated, Yuan2021 discloses wherein the training step comprises algorithmic enhancements for extracting embeddings of at least some of the objects (Page 5, left column, second paragraph, After fusing the multi-level information, the visual extractor needs to generate self-attention to suppress the noise in the high-dimensional embedded features. Inspired by Woo et al. [38], the attention mechanism is affiliated in both channel and space to make the model dynamically focus on the salient features. Examiner considers the noise suppression as an “algorithmic enhancement”).
Regarding claim 13, in which claim 1 is incorporated, Yuan2021 discloses wherein the training step comprises self-distillation wherein a teacher model and a student model have the same architecture and are trained in tandem (Fig. 3; Page 6, knowledge distillation approach; Page 9, §E. Training Algorithm, train a teacher network AMFMN to provide dark knowledge for LW-MCR … the high-level features V h and low-level features Vl of AMFMN and LW-MCR are aligned to get the loss lmis. Fig. 3 illustrates the shared architecture between the student and teacher models).
Regarding claim 14, in which claim 1 is incorporated, Yuan2021 discloses wherein the training step comprises one of a group comprising prediction of masked image token representations, direct minimization of contrastive loss using gradient-based methods, contrastive loss minimization using cluster assignment (Page 6, The triplet function aims to shorten the distance between the intra-class samples as close as possible and make the distance between the inter-class samples as far as possible. Examiner considers triplet loss as the “contrastive loss” since it groups similar objects and separates dissimilar objects. Since the limitation is recited in the alternative, Examiner considers this citation to fully disclose the limitation), and non-parametric classification.
Regarding claim 15, Yuan2021 discloses a system for detecting objects of interest in geospatial imagery (Fig. 5; Fig. 7) comprising in a processor and associated data storage (Page 9, right column, last paragraph, All experiments are performed on a single NVIDIA GTX 2080ti GPU. Examiner considers training the model on a GPU to indicate that the training data was provided from a data storage), providing a dataset representative of a plurality of both natural and man-made objects such as would appear in geospatial images (Fig. 5a; Fig. 7; page 10, right column, “RSICD dataset … RSITMD dataset … UCM dataset … Sydney dataset”. Examiner considers the datasets listed on page 10 to comprise geospatial imagery. Figs. 5a and 7 illustrate that these images contain various trees and buildings, which correspond to “natural” and “man-made” objects, respectively), in the processor, training a teacher-student model with at least some of the dataset (Fig. 3; page 6, right column, first paragraph, we let the lightweight model (student network) learn the hidden representation of the deep model (teacher network)), by means of [a user interface], providing an abstract of an object of interest (Fig. 3, Text on right of figure: some green trees are near a large piece of bare land; Page 3, §RS Crossmodal Text-Image Retrieval, RS crossmodal text-image retrieval refers to retrieve RS images by text. Examiner considers the text string as the “abstract”), the abstract comprising one or more invariant features of the object of interest but insufficient to fully characterize the object of interest (Fig. 3, Text on right of figure: some green trees are near a large piece of bare land; Fig. 6, Text queries. Examiner considers the text query examples to demonstrate features of the object such as the color of the trees but not fully characterizing the object (relative instead of exact location)), and in the processor and associated data storage, [automating search patterns in the dataset] that actualize the abstract to cause the model to return images responsive to the abstract (Page 12, §F. RS Image Recall Analysis, The left of Fig. 6 shows two examples of image retrieval … using text to retrieve images of the two ships next to the harbor).
However, Yuan2021 fails to explicitly disclose a user interface and automating search patterns in the dataset.
You teaches a user interface (Page 245, §Introduction, A user’s query concept. Examiner considers a user’s query in indicate an interface for acquiring the query) and automating search patterns in the dataset (Page 246, right column, §1.3 Our Contributions, our method approximates the user’s queries by changing the query model dynamically).
Both Yuan2021 and You are analogous to the claimed invention because they are both directed towards image retrieval methods. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the dynamic queries of You into the model of Yuan2021. The suggestion/motivation for doing so would have been to improve performance, as suggested by You at Abstract, our method uses intelligent modeling and learning to increase query speed and accuracy.
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You to obtain the invention as specified in claim 15.
Regarding claim 16, Yuan2021 discloses one or more computer-readable non-transitory storage media embodying software (Page 1, §Abstract, We have released some codes … and made it open to access at https://github.com/xiaoyuan1996/retrievalSystem) that is operable when executed to: provide a dataset representative of a plurality of both natural and man-made objects such as would appear in geospatial images (Fig. 5a; Fig. 7; page 10, right column, “RSICD dataset … RSITMD dataset … UCM dataset … Sydney dataset”. Examiner considers the datasets listed on page 10 to comprise geospatial imagery. Figs. 5a and 7 illustrate that these images contain various trees and buildings, which correspond to “natural” and “man-made” objects, respectively), train a teacher-student model with at least some of the dataset (Fig. 3; page 6, right column, first paragraph, we let the lightweight model (student network) learn the hidden representation of the deep model (teacher network)), provide an abstract of an object of interest (Fig. 3, Text on right of figure: some green trees are near a large piece of bare land; Page 3, §RS Crossmodal Text-Image Retrieval, RS crossmodal text-image retrieval refers to retrieve RS images by text. Examiner considers the text string as the “abstract”), the abstract comprising one or more invariant features of the object of interest but insufficient to fully characterize the object of interest (Fig. 3, Text on right of figure: some green trees are near a large piece of bare land; Fig. 6, Text queries. Examiner considers the text query examples to demonstrate features of the object such as the color of the trees but not fully characterizing the object (relative instead of exact location)), and [automate search patterns in the dataset] that actualize the abstract to cause the model to return images responsive to the abstract (Page 12, §F. RS Image Recall Analysis, The left of Fig. 6 shows two examples of image retrieval … using text to retrieve images of the two ships next to the harbor).
However, Yuan2021 fails to explicitly disclose automate search patterns in the dataset.
You teaches automate search patterns in the dataset (Page 246, right column, §1.3 Our Contributions, our method approximates the user’s queries by changing the query model dynamically).
Both Yuan2021 and You are analogous to the claimed invention because they are both directed towards image retrieval methods. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the dynamic queries of You into the model of Yuan2021. The suggestion/motivation for doing so would have been to improve performance, as suggested by You at Abstract, our method uses intelligent modeling and learning to increase query speed and accuracy.
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You to obtain the invention as specified in claim 16.
Regarding claim 20, in which claim 16 is incorporated, Yuan2021 discloses wherein the software is further operable when executed to cause the training step to perform self-distillation wherein a teach model and a student model have the same architecture and are trained in tandem (Fig. 3; Page 6, knowledge distillation approach; Page 9, §E. Training Algorithm, train a teacher network AMFMN to provide dark knowledge for LW-MCR … the high-level features V h and low-level features Vl of AMFMN and LW-MCR are aligned to get the loss lmis. Fig. 3 illustrates the shared architecture between the student and teacher models).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Yuan et al. (Yuan, Zhiqiang, et al. "A lightweight multi-scale crossmodal text-image retrieval method in remote sensing." IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-19) (hereafter, “Yuan2021”) in view of You et al. (You, Huaxin, Edward Y. Chang, and Beitao Li. "NNEW: Nearest Neighbor Expansion By Weighting In Image Database Retrieval." ICME. 2001) (hereafter, “You”) as applied to claim 1 above, and further in view of Chen et al. (US 2020/0258313) (hereafter, “Chen”).
Regarding claim 2, Yuan2021 in view of You discloses the method of claim 1.
However, neither Yuan2021 nor You, whether considered individually or in combination, explicitly disclose wherein the step of providing a dataset comprises in part a data curation step comprising extracting patches of fixed size around detected objects and ensuring that the objects are centered within the associated patch.
Chen teaches wherein the step of providing a dataset comprises in part a data curation step comprising extracting patches of fixed size around detected objects (¶0042, fixed-size collection of bounding boxes. Examiner considers the bounding boxes as “patches”) and ensuring that the objects are centered within the associated patch (¶0053, a center of the bounding box is positioned at a center or approximate center of the object of interest).
Yuan2021, You, and Chen are analogous to the claimed invention because Yuan2021 and You are directed towards image retrieval methods and Chen is directed towards object detection. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the bounding boxes of Chen into the dynamic queries of You and the model of Yuan2021. The suggestion/motivation for doing so would have been to identify objects of interest, as suggested by Chen at ¶0022, identify an object of interest and generate a bounding box around the object of interest.
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You and Chen.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You and Chen to obtain the invention as specified in claim 2.
Claims 3, 4, 8, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Yuan et al. (Yuan, Zhiqiang, et al. "A lightweight multi-scale crossmodal text-image retrieval method in remote sensing." IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-19) (hereafter, “Yuan2021”) in view of You et al. (You, Huaxin, Edward Y. Chang, and Beitao Li. "NNEW: Nearest Neighbor Expansion By Weighting In Image Database Retrieval." ICME. 2001) (hereafter, “You”) and Chen et al. (US 2020/0258313) (hereafter, “Chen”) as applied to claim 2 above, and further in view of Yuan et al. (Yuan, Zhiqiang, et al. "Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval." IEEE Transactions on Geoscience and Remote Sensing, 60 (2022): 1-19) (hereafter, “Yuan2022”).
Regarding claim 3, Yuan2021, in view of You and Chen, discloses the method of claim 2.
However, Yuan2021 fails to explicitly disclose wherein each object is associated with a class and the object class is preserved.
Yuan2022 teaches wherein each object is associated with a class and the object class is preserved (Table 1 on page 9; Page 15, right column, first full paragraph, the wrong results still belong to the category of the ground truth. For example, when performing keywords retrieval, rank1 and rank2 retrieved still belong to the category of the playground. Table 1 lists various classes. Since the category is still associated with the retrieved image, Examiner considers it “preserved”).
Yuan2021, You, Chen, and Yuan2022 are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods and Chen is directed towards object detection. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the classes of Yuan2022 into the bounding boxes of Chen, the dynamic queries of You, and the model of Yuan2021. The suggestion/motivation for doing so would have been because Yuan2021 utilizes the AMFMN network introduced by Yuan2022 as the teacher network (Fig. 3 of Yuan2021).
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You, Chen, and Yuan2022.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You, Chen, and Yuan2022 to obtain the invention as specified in claim 3.
Regarding claim 4, Yuan2021, in view of You and Chen, discloses the method of claim 2.
However, neither Yuan2021 nor You, whether considered individually or in combination, explicitly disclose wherein the patches are rotated around the associated centers.
Chen teaches wherein the patches (¶0042, fixed-size collection of bounding boxes. Examiner considers the bounding boxes as “patches”) are [rotated around the associated centers].
Yuan2021, You, and Chen are analogous to the claimed invention because Yuan2021 and You are directed towards image retrieval methods and Chen is directed towards object detection. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the bounding boxes of Chen into the dynamic queries of You and the model of Yuan2021. The suggestion/motivation for doing so would have been to identify objects of interest, as suggested by Chen at ¶0022, identify an object of interest and generate a bounding box around the object of interest.
However, none of Yuan2021, You, and Chen, whether considered individually or in combination, explicitly disclose rotated around the associated centers.
Yuan2022 teaches rotated around the associated centers (Page 11, left column, first paragraph, rotate the image to enhance the training samples. Since the bounding boxes of Chen are centered on the object, Examiner considers the rotation as “around the associated center”).
Yuan2021, You, Chen, and Yuan2022 are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods and Chen is directed towards object detection. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the rotation of Yuan2022 into the bounding boxes of Chen, the dynamic queries of You, and the model of Yuan2021. The suggestion/motivation for doing so would have been because Yuan2021 utilizes the AMFMN network introduced by Yuan2022 as the teacher network (Fig. 3 of Yuan2021).
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You, Chen, and Yuan2022.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You, Chen, and Yuan2022 to obtain the invention as specified in claim 3.
Regarding claim 8, Yuan2021, in view of You and Chen, discloses the method of claim 2.
However, none of Yuan2021, You, and Chen, whether considered individually or in combination, explicitly disclose wherein an objectness score is associated with each detected object.
Yuan2022 teaches wherein an objectness score is associated with each detected object (Page 5, right column, second paragraph, the network needs to pay attention to the salient object information and suppress useless feature expression in the global features; Page 6, left column, §Redundant Feature Filter, This redundant representation may cause visual features irrelevant to other representations and even produce misleading information … generate a salient mask of the global vector. Examiner considers the values of the salient mask as an “objectness score” since it differentiates between salient object information and background information).
Yuan2021, You, Chen, and Yuan2022 are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods and Chen is directed towards object detection. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the salient mask of Yuan2022 into the bounding boxes of Chen, the dynamic queries of You, and the model of Yuan2021. The suggestion/motivation for doing so would have been because Yuan2021 utilizes the AMFMN network introduced by Yuan2022 as the teacher network (Fig. 3 of Yuan2021).
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You, Chen, and Yuan2022.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You, Chen, and Yuan2022 to obtain the invention as specified in claim 8.
Regarding claim 17, Yuan2021 in view of You discloses the storage media of claim 16.
However, neither Yuan2021 nor You, whether considered individually or in combination, explicitly disclose wherein the software is further operable when executed to perform data curation comprising extracting patches of fixed size around detected objects, ensuring that the objects are centered within associated patches, and, in at least some instances, rotating the associated patches around the respective centers.
Chen teaches wherein the software is further operable when executed to perform data curation comprising extracting patches of fixed size around detected objects (¶0042, fixed-size collection of bounding boxes. Examiner considers the bounding boxes as “patches”), ensuring that the objects are centered within associated patches (¶0053, a center of the bounding box is positioned at a center or approximate center of the object of interest).
Yuan2021, You, and Chen are analogous to the claimed invention because Yuan2021 and You are directed towards image retrieval methods and Chen is directed towards object detection. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the bounding boxes of Chen into the dynamic queries of You and the model of Yuan2021. The suggestion/motivation for doing so would have been to identify objects of interest, as suggested by Chen at ¶0022, identify an object of interest and generate a bounding box around the object of interest.
However, none of Yuan2021, You, and Chen, whether considered individually or in combination, explicitly disclose rotating the associated patches around the respective centers.
Yuan2022 teaches rotating the associated patches around the respective centers (Page 11, left column, first paragraph, rotate the image to enhance the training samples. Since the bounding boxes of Chen are centered on the object, Examiner considers the rotation as “around the associated center”).
Yuan2021, You, Chen, and Yuan2022 are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods and Chen is directed towards object detection. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the rotation of Yuan2022 into the bounding boxes of Chen, the dynamic queries of You, and the model of Yuan2021. The suggestion/motivation for doing so would have been because Yuan2021 utilizes the AMFMN network introduced by Yuan2022 as the teacher network (Fig. 3 of Yuan2021).
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You, Chen, and Yuan2022.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You, Chen, and Yuan2022 to obtain the invention as specified in claim 17.
Claims 6, 7, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Yuan et al. (Yuan, Zhiqiang, et al. "A lightweight multi-scale crossmodal text-image retrieval method in remote sensing." IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-19) (hereafter, “Yuan2021”) in view of You et al. (You, Huaxin, Edward Y. Chang, and Beitao Li. "NNEW: Nearest Neighbor Expansion By Weighting In Image Database Retrieval." ICME. 2001) (hereafter, “You”) as applied to claims 1 and 16 above, and further in view of Yuan et al. (Yuan, Zhiqiang, et al. "Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval." IEEE Transactions on Geoscience and Remote Sensing, 60 (2022): 1-19) (hereafter, “Yuan2022”).
Regarding claim 6, Yuan2021 in view of You discloses the method of claim 5.
However, neither Yuan2021 nor You, whether considered individually or in combination, explicitly disclose wherein extracting embeddings comprises the use of metric learning.
Yuan2022 teaches wherein extracting embeddings comprises the use of metric learning (Eqn. 26; Page 7, The activated s(w) is used as a dynamic weight function. Examiner considers the dynamic weight function as “metric learning”).
Yuan2021, You, and Yuan2022 are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the dynamic weights of Yuan2022 into the dynamic queries of You and the model of Yuan2021. The suggestion/motivation for doing so would have been because Yuan2021 utilizes the AMFMN network introduced by Yuan2022 as the teacher network (Fig. 3 of Yuan2021).
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You and Yuan2022.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You and Yuan2022 to obtain the invention as specified in claim 6.
Regarding claim 7, Yuan2021 in view of You discloses the method of claim 1.
However, neither Yuan2021 nor You, whether considered individually or in combination, explicitly disclose wherein the training step further comprises gathering objectness attributes from examples of generic objects.
Yuan2022 teaches wherein the training step further comprises gathering objectness attributes from examples of generic objects (Page 4, right column, second paragraph, we utilize the CNN pretrained [38]–[40] on the ImageNet dataset and transfer its parameters to our task. Specifically, the global image features … The deeper features map with a larger scale can capture high-level semantic information of salient objects, whereas the shallower features map with a smaller scale can extract fine features information. Examiner considers ImageNet to contain generic objects and the extracted features as “objectness attributes”).
Yuan2021, You, and Yuan2022 are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the features of Yuan2022 into the dynamic queries of You and the model of Yuan2021. The suggestion/motivation for doing so would have been because Yuan2021 utilizes the AMFMN network introduced by Yuan2022 as the teacher network (Fig. 3 of Yuan2021).
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You and Yuan2022.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You and Yuan2022 to obtain the invention as specified in claim 7.
Regarding claim 18, Yuan2021 in view of You discloses the storage media of claim 16.
However, neither Yuan2021 nor You, whether considered individually or in combination, explicitly disclose wherein an objectness score is associated with each detected object.
Yuan2022 teaches wherein an objectness score is associated with each detected object (Page 5, right column, second paragraph, the network needs to pay attention to the salient object information and suppress useless feature expression in the global features; Page 6, left column, §Redundant Feature Filter, This redundant representation may cause visual features irrelevant to other representations and even produce misleading information … generate a salient mask of the global vector. Examiner considers the values of the salient mask as an “objectness score” since it differentiates between salient object information and background information).
Yuan2021, You, and Yuan2022 are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the salient mask of Yuan2022 into the dynamic queries of You and the model of Yuan2021. The suggestion/motivation for doing so would have been because Yuan2021 utilizes the AMFMN network introduced by Yuan2022 as the teacher network (Fig. 3 of Yuan2021).
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You and Yuan2022.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You and Yuan2022 to obtain the invention as specified in claim 18.
Claims 9, 10, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Yuan et al. (Yuan, Zhiqiang, et al. "A lightweight multi-scale crossmodal text-image retrieval method in remote sensing." IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-19) (hereafter, “Yuan2021”) in view of You et al. (You, Huaxin, Edward Y. Chang, and Beitao Li. "NNEW: Nearest Neighbor Expansion By Weighting In Image Database Retrieval." ICME. 2001) (hereafter, “You”) as applied to claims 1 and 16 above, and further in view of Yuan et al. (Yuan, Zhiqiang, et al. "Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval." IEEE Transactions on Geoscience and Remote Sensing, 60 (2022): 1-19) (hereafter, “Yuan2022”), Aribido et al. (Aribido, Oluwaseun Joseph, Ghassan AlRegib, and Yazeed Alaudah. "Self-supervised delineation of geologic structures using orthogonal latent space projection." Geophysics 86.6 (2021): V497-V508) (hereafter, “Aribido”), and Chen et al. (US 2020/0258313) (hereafter, “Chen”).
Regarding claim 9, Yuan2021 in view of You discloses the method of claim 1.
However, neither Yuan2021 nor You, whether considered individually or in combination, explicitly disclose wherein the training step comprises post-processing steps to train a stage-1 representation model from labeled data sources, use the stage-1 model to extract vector representations of the patches from unlabeled data sources, cluster the vector representations using a clustering algorithm, and order the clusters by decreasing cluster size.
Yuan2022 teaches wherein the training step comprises post-processing steps to train a stage-1 representation model from labeled data sources (Page 4, right column, second paragraph, we utilize the CNN pretrained [38]–[40] on the ImageNet dataset and transfer its parameters to our task. Examiner considers the CNN as the “stage-1 representation model” and ImageNet to be a “labeled data source”), use the stage-1 model to extract vector representations of the [patches] from unlabeled data sources (Fig. 2(a); Page 4-5, starting at the last line of the right column of page 4, we not only extract the global features from the RS image but also extract features from each layer of the convolution network. Examiner considers the RS image as “unlabeled”).
Yuan2021, You, and Yuan2022 are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the feature network of Yuan2022 into the dynamic queries of You and the model of Yuan2021. The suggestion/motivation for doing so would have been because Yuan2021 utilizes the AMFMN network introduced by Yuan2022 as the teacher network (Fig. 3 of Yuan2021).
However, none of Yuan2021, You, and Yuan2022, whether considered individually or in combination, explicitly disclose patches, cluster the vector representations using a clustering algorithm, and order the clusters by decreasing cluster size.
Aribido teaches cluster the vector representations using a clustering algorithm (Page 3, left column, last paragraph, we propose a hierarchical clustering framework to cluster images), and order the clusters by decreasing cluster size (Page 4, §Algorithm 1, we arrange the clusters in order of cluster size).
Yuan2021, You, Yuan2022, and Aribido are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods and Aribido is directed towards object detection in images. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the clustering and sorting of Aribido into the feature network of Yuan2022, the dynamic queries of You, and the model of Yuan2021. The suggestion/motivation for doing so would have been to reduce the need for manual annotation, as suggested by Aribido at page 2, right column, second paragraph, our method does not require any annotated labels from interpreters.
However, none of Yuan2021, You, Yuan2022, and Aribido, whether considered individually or in combination, explicitly disclose patches.
Chen teaches patches (¶0042, fixed-size collection of bounding boxes. Examiner considers the bounding boxes as “patches”).
Yuan2021, You, Yuan2022, Aribido, and Chen are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods and Aribido and Chen are directed towards object detection in images. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the bounding boxes of Chen into the clustering and sorting of Aribido, the feature network of Yuan2022, the dynamic queries of You, and the model of Yuan2021. The suggestion/motivation for doing so would have been to identify objects of interest, as suggested by Chen at ¶0022, identify an object of interest and generate a bounding box around the object of interest.
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You, Yuan2022, Aribido, and Chen.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You, Yuan2022, Aribido, and Chen to obtain the invention as specified in claim 9.
Regarding claim 10, Yuan2021 in view of You, Yuan2022, Aribido, and Chen discloses the method of claim 9.
However, none of Yuan2021, You, Yuan2022, and Chen, whether considered individually or in combination, explicitly disclose wherein the clustering algorithm comprises any of a group comprising affinity propagation, mean shift, ward hierarchical clustering, agglomerative clustering, and DBSCAN.
Aribido teaches wherein the clustering algorithm comprises any of a group comprising affinity propagation, mean shift, ward hierarchical clustering, agglomerative clustering, and DBSCAN (Page 3, right column, second paragraph, DBSCAN. Since the limitation is recited in the alternative, Examiner considers this citation to fully disclose the limitation).
Yuan2021, You, Yuan2022, Aribido, and Chen are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods and Aribido and Chen are directed towards object detection in images. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the clustering and sorting of Aribido into the bounding boxes of Chen, the feature network of Yuan2022, the dynamic queries of You, and the model of Yuan2021. The suggestion/motivation for doing so would have been to reduce the need for manual annotation, as suggested by Aribido at page 2, right column, second paragraph, our method does not require any annotated labels from interpreters.
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You, Yuan2022, Aribido, and Chen.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You, Yuan2022, Aribido, and Chen to obtain the invention as specified in claim 10.
Regarding claim 19, Yuan2021 in view of You discloses the storage media of claim 16.
However, neither Yuan2021 nor You, whether considered individually or in combination, explicitly disclose wherein the software is further operable when executed to extract vector representations of the patches from unlabeled data sources, to cluster the vector representations, and to order the clusters according to size.
Yuan2022 teaches wherein the software is further operable when executed to extract vector representations of the [patches] from unlabeled data sources (Fig. 2(a); Page 4-5, starting at the last line of the right column of page 4, we not only extract the global features from the RS image but also extract features from each layer of the convolution network. Examiner considers the RS image as “unlabeled”).
Yuan2021, You, and Yuan2022 are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the feature network of Yuan2022 into the dynamic queries of You and the model of Yuan2021. The suggestion/motivation for doing so would have been because Yuan2021 utilizes the AMFMN network introduced by Yuan2022 as the teacher network (Fig. 3 of Yuan2021).
However, none of Yuan2021, You, and Yuan2022, whether considered individually or in combination, explicitly disclose patches, to cluster the vector representations, and to order the clusters according to size.
Aribido teaches to cluster the vector representations (Page 3, left column, last paragraph, we propose a hierarchical clustering framework to cluster images), and to order the clusters according to size (Page 4, §Algorithm 1, we arrange the clusters in order of cluster size).
Yuan2021, You, Yuan2022, and Aribido are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods and Aribido is directed towards object detection in images. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the clustering and sorting of Aribido into the feature network of Yuan2022, the dynamic queries of You, and the model of Yuan2021. The suggestion/motivation for doing so would have been to reduce the need for manual annotation, as suggested by Aribido at page 2, right column, second paragraph, our method does not require any annotated labels from interpreters.
However, none of Yuan2021, You, Yuan2022, and Aribido, whether considered individually or in combination, explicitly disclose patches.
Chen teaches patches (¶0042, fixed-size collection of bounding boxes. Examiner considers the bounding boxes as “patches”).
Yuan2021, You, Yuan2022, Aribido, and Chen are analogous to the claimed invention because Yuan2021, You, and Yuan2022 are directed towards image retrieval methods and Aribido and Chen are directed towards object detection in images. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the bounding boxes of Chen into the clustering and sorting of Aribido, the feature network of Yuan2022, the dynamic queries of You, and the model of Yuan2021. The suggestion/motivation for doing so would have been to identify objects of interest, as suggested by Chen at ¶0022, identify an object of interest and generate a bounding box around the object of interest.
This method of improving Yuan2021 was within the ordinary ability of one of ordinary skill in the art based on the teachings of You, Yuan2022, Aribido, and Chen.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021 with the teachings of You, Yuan2022, Aribido, and Chen to obtain the invention as specified in claim 19.
Claims 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Yuan et al. (Yuan, Zhiqiang, et al. "A lightweight multi-scale crossmodal text-image retrieval method in remote sensing." IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-19) (hereafter, “Yuan2021”) in view of You et al. (You, Huaxin, Edward Y. Chang, and Beitao Li. "NNEW: Nearest Neighbor Expansion By Weighting In Image Database Retrieval." ICME. 2001) (hereafter, “You”) as applied to claim 1 above, and further in view of Zhu et al. (Zhu, Yi, and Xiu Li. "Iterative uni-modal and cross-modal clustered contrastive learning for image-text retrieval." 2023 International Conference on Pattern Recognition, Machine Vision and Intelligent Algorithms (PRMVIA). IEEE, 2023) (hereafter, “Zhu”).
Regarding claim 11, Yuan2021 in view of You discloses the method of claim 1.
However, neither Yuan2021 nor You, whether considered individually or in combination, explicitly disclose wherein the training step comprises training on pretext tasks.
Zhu teaches wherein the training step comprises training on pretext tasks (Page 18, left column, first paragraph, we can improve the performance of the uni-modal encoders by training unpaired uni-modal data contrastively …. Given a positive image pair, which comprises two augmented images originating from the same image, we treat the other 2(A – 1) augmented examples within a minibatch as negative examples. Examiner considers the contrastive learning as on two augmented images as a “pretext task”).
Yuan2021, You, and Zhu are analogous to the claimed invention because Yuan2021, You, and Zhu are directed towards image retrieval methods. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the augmented image comparison of Zhu into the dynamic queries of You and the model of Yuan2021. The suggestion/motivation for doing so would have been to improve performance, as suggested by Zhu at Page 18, left column, first paragraph, we can improve the performance of the uni-modal encoders.
This method of improving Yuan2021was within the ordinary ability of one of ordinary skill in the art based on the teachings of You and Zhu.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021with the teachings of You and Zhu to obtain the invention as specified in claim 11.
Regarding claim 12, Yuan2021 in view of You and Zhu discloses the method of claim 11.
However, neither Yuan2021 nor You, whether considered individually or in combination, explicitly disclose wherein training on pretext tasks comprises generating one or more positive image pairs by perturbing a single instance of an image.
Zhu teaches wherein training on pretext tasks comprises generating one or more positive image pairs by perturbing a single instance of an image (Page 17, left column, first paragraph, we apply some augmentation to enhance the model’s robustness for both image and text data respectively. Besides, we form positive pairs in image and text uni-modal contrast through data augmentation. For each image, we introduce random crop at a scale of 0.2 to 1, random contrast ratio, random Gaussian blur, random grey scale and random horizontal flip).
Yuan2021, You, and Zhu are analogous to the claimed invention because Yuan2021, You, and Zhu are directed towards image retrieval methods. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the augmented images of Zhu into the dynamic queries of You and the model of Yuan2021. The suggestion/motivation for doing so would have been to improve performance, as suggested by Zhu at Page 18, left column, first paragraph, we can improve the performance of the uni-modal encoders.
This method of improving Yuan2021was within the ordinary ability of one of ordinary skill in the art based on the teachings of You and Zhu.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yuan2021with the teachings of You and Zhu to obtain the invention as specified in claim 12.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Cheng et al. (Cheng, Qimin, et al. "A deep semantic alignment network for the cross-modal image-text retrieval in remote sensing." IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 14 (2021): 4284-4297) disclose using transformer models for cross-modal image retrieval in remote sensing (Page 4285, right column, first paragraph, we proposed a cross-modal image-text retrieval network for remote sensing to deal with the complexity of semantics and refine the correspondence between remote sensing images and text).
Lv et al. (Lv, Yafei, et al. "Fusion-based correlation learning model for cross-modal remote sensing image retrieval." IEEE Geoscience and Remote Sensing Letters 19 (2021): 1-5) disclose a fusion based image-text retrieval network (Page 2, left column, first paragraph, we propose a novel correlation learning model FCLM for cross-modal RS image retrieval. A cross-modal-fusion network is designed to capture the intermodality fused information).
Tang et al. (Tang, Xu, et al. "Interacting-enhancing feature transformer for cross-modal remote-sensing image and text retrieval." IEEE Transactions on Geoscience and Remote Sensing 61 (2023): 1-15) disclose a network that simultaneously processes text and image information during retrieval tasks (Page 2, right column, first paragraph, IEFT simultaneously processes RS images and texts, so the visual and textual knowledge can be learned with small distribution misaligned).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIAOMAO DING whose telephone number is (571)272-7237. The examiner can normally be reached Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIAOMAO DING/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676
